01 March 2015
There were a lot of release this week, including the first project out of Confluent (version 1.0 of the Confluent Platform), version 1.0 of the Kite SDK, and version 1.0 of Apache HBase. And as has been the trend over the past 6 months, there's a lot of coverage of Apache Spark and Apache Kafka in this week's issue, including some details on Databricks (the company founded by Spark's creators).
Apache Flink is introducing a new feature called Flink Streaming, which is a real-time stream processing library. Flink Streaming is currently available on master but is not yet part of a release. Flink Streaming doesn't require microbatching, which allows advanced windowing techniques like sliding windows and data-driven windows. The introductory blog post has several examples of the API.
http://flink.apache.org/news/2015/02/09/streaming-example.html
This article describes how to deploy a flume master and agents to stream data from a file on the local filesystem to HDFS.
This post describes the notion of "moving the code to the data" for big data processing and how that applies to Apache Spark. It discusses Sparks' Resilient Distributed Datasets (RDD), how programs are translated to operations on RDDs (at a high level), data shuffling, execution planning, and fault tolerance.
http://horicky.blogspot.com/2015/02/big-data-processing-in-spark.html
A recent release of the Kite SDK added support for data stored as json-encoded text. This post describes how to automatically extract a schema from json data (based off of the first 20 records in a dataset) and import json data into Hive via the json-input command.
http://ingest.tips/2015/02/23/kite-adds-json-support/
Spark was a hot topic at last week's Strata+Hadoop World. The team from Databricks gave a number of presentations on topics like Spark Streaming, debugging and tuning Spark, and the future of Spark. All of the slides from these presentations have been posted on slideshare (linked to from the following post).
https://databricks.com/blog/2015/02/24/databricks-at-strata-san-jose.html
This post describes five ways to make Apache Hive queries faster. The recommendations are mostly around new features that are disabled by default or that require additional processing. These include: enabling Tez, writing data using ORCFile, enabling vectorization, enabling the cost-based optimizer (and computing statistics), and some tips for writing good SQL for Hive.
http://hortonworks.com/blog/5-ways-make-hive-queries-run-faster/
Flume has a number of plugins for interacting with Kafka. Along with the morphlines library, which can be used to parse and process data, one can build powerful data pipelines with Flume and Kafka without writing any code. This post on the Cloudera blog describes how to build one such integration: Apache httpd log data is sent to syslog, which is forward to Flume then Kafka then to Solr (via Flume/Morphlines).
This post covers rolling upgrades in YARN. Once side-by-side installations of multiple versions of YARN are available on all nodes, the Job History & Timeline Service, ResourceManagers, and NodeManagers can be restarted with the new version in a rolling fashion. Any YARN applications should survive this process, although there are some best practices to build compatible YARN applications (e.g. using the distributed cache rather than local files for config). There are many more details about YARN HA in the following post from the Hortonworks blog.
http://hortonworks.com/blog/introducing-rolling-upgrades-downgrades-apache-hadoop-yarn-cluster/
This post describes how to configure Spark to emit metrics to Graphite. There are some examples of the types of data you can collect using these metrics, and a scripts/template for working with Grafana (web front-end to Graphite) dashboards for Spark.
http://www.hammerlab.org/2015/02/27/monitoring-spark-with-graphite-and-grafana/
The Amazon Big Data blog has a post on using IPython to run a python (streaming) MapReduce job on Amazon Elastic MapReduce. The instructions are general purpose, but it's quite easy to get started with IPython on EMR with the ipython-notebook bootstrap action.
Cloudera has announced a beta version of Hive-on-Spark that includes all major functionality. For folks anxious to try it out, there is a custom CDH parcel, which supports a subset of the regular CDH components.
http://blog.cloudera.com/blog/2015/02/download-the-hive-on-spark-beta/
The Confluent blog has an excellent two-part series on building a streaming data platform with Apache Kafka. The first post is foundational—it describes a stream-centric data architecture, events and event streams, the requirements of a stream data platform, and more. The second post is more practical—it contains advice for building a Kafka-based platform, covering topics such as Kafka topologies, data formats and schemas, various types of data (event streams, application logs, machine metrics, etc), and stream processing.
http://blog.confluent.io/2015/02/25/stream-data-platform-1/
http://blog.confluent.io/2015/02/25/stream-data-platform-2/
Datanami has an article on the Open Data Platform (ODP) announcement from last week. They've collected a number of opinions from folks in industry. In particular, independent software vendors (ISVs) seem to think the goal of standardizing is important in order to ease the pain of integrating with the plethora of distributions. But the ISVs are unsure that the ODP should be the organization responsible for this—the Apache Software Foundation could also potentially provide the standard.
http://www.datanami.com/2015/02/24/making-sense-of-the-odp-where-does-hadoop-go-from-here/
VentureBeat has coverage of Hortonworks' first quarterly report since becoming a public company. In short, they announced $12.7 in revenue for Q'14 and a net-loss of $2.19/share (analysts had expected $2.04/share). The good news is that Hortonworks saw billings grow 148% year-over-year after adding 99 new customers in Q4.
ReadWrite recently spoke with Hadoop creator Doug Cutting about the growing Hadoop ecosystem (comparing Hadoop to Linux), Hadoop's warts ("It used to be a lot worse [..] It's gotten a lot better"), Hadoop's limitations (enterprise features and integrations), and how to get started with Hadoop (usually a new project lowers the barrier to entry).
http://readwrite.com/2015/02/25/hadoop-big-data-start-small-doug-cutting
Qubole has rounded up a number of articles from last week, including several on the Open Data Platform. In addition, there are a few articles based on happenings at Strata, the Hadoop skills gap, and more.
http://www.qubole.com/blog/big-data/strata-hadoop-2015/
The DBMS2 blog has a post on Databricks and Spark. The post covers several topics, including Databricks Cloud, which is the main product offering from Databricks. Databricks Cloud runs independent of Hadoop on Amazon Web Services, and it is not yet generally available. The post also discusses the company (which is over 50 people), Spark DataFrame, SparkR, and Spark Streaming.
http://www.dbms2.com/2015/02/28/databricks-and-spark-update/
Apache HBase released version 1.0 this week. The 1.0 version numbering marks the significant feature-set and stability in the release. Since version 0.98, there are several new features like: timeline consistent read replicas, online configuration changes, revamped documentation, and improvements to APIs. The Apache Software Foundation blog has a press release and a technical post on the release.
https://blogs.apache.org/foundation/entry/the_apache_software_foundation_announces72
https://blogs.apache.org/hbase/entry/start_of_a_new_era
The Kite SDK has announced the 1.0.0 release, which promises API stability in coming versions. The new release has also removed all deprecated classes/methods, improves avro schema handling, and contains some other minor changes.
https://groups.google.com/a/cloudera.org/forum/#!topic/cdk-dev/iszgEzhLFYA
Google has announced a new version of bdutil, their tool for running Hadoop clusters on Google Cloud Platform. The new release includes bug fixes, improvements for Spark, new support for Apache Flink, improvements to bdutil's Ambari plugin, a plugin for Apache Storm, a basic HBase plugin, and a basic CDH plugin.
https://groups.google.com/forum/#!topic/gcp-hadoop-announce/uVJ_6y9cGKM
Cloudera has released Cloudera Enterprise 5.2.4 and 5.3.2. The 5.2.4 release fixes an edit log corruption issue in HDFS and issues with Search, Impala, Hive, Hue, and Cloudera Navigator. The 5.3.2 release includes a fix for edit log corruption as well as fixes for nearly every component in CDH. Cloudera manager also has several fixes.
https://groups.google.com/a/cloudera.org/d/msg/cdh-user/wKO-JahzKqc/jezhCjyhj2QJ
Confluent, makers of a stream data platform powered by Apache Kafka, announced version 1.0 of the Confluent Platform. Components of the platform include: APache Kafka 0.8.2.0 (with a few patches), a schema management layer, a REST API for Kafka, and Camus for sending data from Kafka to HDFS.
http://blog.confluent.io/2015/02/25/announcing-the-confluent-platform-1-0/
Curated by Datadog ( http://www.datadoghq.com )
Bayesian Networks with R and Hadoop (San Francisco) - Monday, March 2
http://www.meetup.com/SF-Data-Science/events/220708484/
Cassandra & Hadoop Explained in Real-World Applications (Palo Alto) - Tuesday, March 3
http://www.meetup.com/Advanced-Analytics-and-Big-Data/events/220502449/
Samza Meetup Kickoff (Mountain View) - Wednesday, March 4
http://www.meetup.com/Bay-Area-Samza-Meetup/events/220354853/
Long-Running Services on YARN Using Slider (Santa Clara) - Wednesday, March 4
http://www.meetup.com/HSquared-Hadoop-Hortonworks-User-Group/events/220422211/
What Makes Machine Learning Easy to Program, and What Makes It Fast? (Palo Alto) - Friday, March 6
http://www.meetup.com/sfmachinelearning/events/219223045/
Cassandra at Instagram, Twitter, and Facebook: Industry Best Practices (Minneapolis) - Monday, March 2
http://www.meetup.com/Minneapolis-St-Paul-Cassandra-Meetup/events/220251639/
R Intro and YARN: A Game Changer (Oklahoma City) - Thursday, March 5
http://www.meetup.com/Big-Data-in-Oklahoma-City/events/220504991/
First Spark Meetup: Bringing This Group Alive (Alpharetta) - Thursday, March 5
http://www.meetup.com/Apache-Spark/events/220106017/
Apache Ignite: Introducing the Future of Fast Data (New York) - Tuesday, March 3
http://www.meetup.com/NYC-In-Memory-Computing/events/220284774/
Big Data, The Next Generation: Hands-on with Hadoop YARN, Impala, and Spark (Cambridge) - Wednesday, March 4
http://www.meetup.com/BigDataBoston/events/220730588/
Spark-as-a-Service Demo by Qubole, and Big Data Applications by DataXu (Boston) - Thursday, March 5
http://www.meetup.com/bostonhadoop/events/220329007/
ENGLAND
Spark London Meetup (London) - Thursday, March 5
http://www.meetup.com/Spark-London/events/220754934/
End-to-End Twitter Analysis with Hadoop and Watson in 10 Minutes (Stuttgart) - Friday, March 6
http://www.meetup.com/Big-Data-Developers-in-Stuttgart/events/220776443/
Cassandra Spark Connector Architecture and API (Rotterdam) - Thursday, March 5
http://www.meetup.com/Netherlands-Cassandra-Users/events/220576443/
Lessons Learned Building a Spark Distribution (Lausanne) - Tuesday, March 3
http://www.meetup.com/Scala-Romandie/events/220724991/
Introduction to Apache Spark + Error Handling in Scala (Athens) - Wednesday, March 4
http://www.meetup.com/Athens-Scala-Meetup/events/220621054/
Adam Drake of Skyscanner and Jeff Markham of Hortonworks (Singapore) - Wednesday, March 4
http://www.meetup.com/BigData-Hadoop-SG/events/220789176/