10 May 2015
HBaseCon and Strata + Hadoop World London were both this week, and presentations and announcements make up much of the content in this week's newsletter. HBaseCon content includes presentations by speakers from Pinterest, Flipboard, Adobe, and Salesforce. In terms of announcements, there were many related to HBase—Google announced Cloud Bigtable (with HBase API compatibility), Cloudera Labs added Apache Phoenix, Qubole announced their HBase-as-a-Service offering, and there's a new version of OpenTSDB. In addition to HBase coverage, there's a new version of Apache Drill, an interesting post about debugging Zookeeper failures, two articles on Apache Flink, and much more.
Apache Ambari contains "User Views" which provide tools and visualizations for a Hadoop cluster. This post on the Hortonworks blog discusses the Tez view, which provides an annotated data-flow DAG for understanding what is happening during a Tez job, and the Capacity Scheduler view for editing the queues and deploying changes (rather than mucking with XML). The post has more about using these tools (especially interesting is using the Tez view to find and fix performance issues). These views and a couple of others are available in Tech Preview.
http://hortonworks.com/blog/improving-operations-with-ambari-user-views/
The Hortonworks blog has the latest in a series of posts about rolling upgrades for Hadoop. This post describes rolling upgrades for MapReduce, which are possible because the MapReduce libraries have moved to the Distributed Cache rather than being pre-installed on nodes in the cluster.
http://hortonworks.com/blog/mapreduce-framework-deployment-for-yarn-rolling-upgrades/
PagerDuty has a very interesting post describing some low-level debugging of recent issues with Apache Zookeeper. While the symptom is a (silent) Zookeeper failure, the cause involves data corruption at the TCP layer (caused by corruption due to IPSec and the aesni-intel kernel module).
http://www.pagerduty.com/blog/the-discovery-of-apache-zookeepers-poison-packet/
The Confluent blog has a guest post that talks about real-time stream processing with Apache Flink. The post goes into details on the components and architecture of Apache Flink streaming (such as pipelining, replay, and state backup & restore). It also describes the Flink APIs, how to achieve high availability, scaling up/down, and how Flink's support for batch and streaming fits with the Lambda and Kappa architectures.
http://blog.confluent.io/2015/05/06/real-time-stream-processing-the-next-step-for-apache-flink/
The first of several posts from HBaseCon, this one describes how Flipboard is using HBase. Several Flipboard features, like user-generated magazines, likes, and comments, are powered by HBase. The slides describe their data model for magazine and their social graph—they use JSON serialization inside of HBase cells and combine with ElasticSearch for additional indexes. The post also has information on their deploy, which is in AWS across 15 clusters and 250 region servers
http://www.slideshare.net/matthewblair/hbasecon-2015-hbase-flipboard
This presentation describes HBase at Pinterest, which is deployed at pretty massive scale on AWS. There are several deployment suggestions—both on AWS (e.g. instance sizes, noisy neighbors) as well as independent of the environment (e.g. tuning JVM args such as GC and Linux file systems). The slides also describe best practices for monitoring, alerting, capacity planning, optimizing for availability (with some specifics to AWS), disaster recover in AWS, and more.
https://speakerdeck.com/phobos182/hbasecon-2015-hbase-at-scale
HBase on YARN has been available for some time (via Apache Slider), but this is among first instances I've heard of someone running HBase on Apache Mesos. The presentation describes how Adobe is running HBase via Docker containers on Mesos using the Marathon scheduler. There are details on how the Docker images are built and run as well as plans for improvement in the future (such as scheduling for data locality with HDFS).
http://www.slideshare.net/clehene/elastic-hbase-on-mesos
This presentation on HBase performance and correctness tuning is the richest collection of best practices and important configuration items that I've seen for any Hadoop ecosystem project. It covers tuning HDFS, tuning HBase RegionServers/column families, sizing RegionServer instances, tuning HBase client settings, and tuning Linux.
http://www.slideshare.net/lhofhansl/h-base-tuninghbasecon2015ok
An article on VentureBeat details estimates of Cloudera's revenue, profitability, and valuation recently completed by Manhattan Venture Partners, a private tech research firm. Among the highlights, the analysts predict $199 milliion in revenue in 2015 and profitability by 2018 at the earliest.
Apache ORC, which is a columnar storage format for Hadoop, has moved from within the Apache Hive project to be a top-level project of its own. The Hortonworks blog has more information on the project including a link to a post from a couple of months back by Facebook about some performance testing they've done with ORC.
http://hortonworks.com/blog/apache-orc-launches-as-a-top-level-project/
Cloudera and Confluent have announced an initiative to build a suite of tests to "certify API and protocol compatibility between versions and distribution" of Kafka.
http://blog.confluent.io/2015/05/06/compatibility-testing-for-apache-kafka/
Standardizing APIs has been a big topic of recent, especially as many Hadoop ecosystem projects hit version 1.0. This post highlights the work that was done for standardizing the HBase API, and how Google's contribution now makes a lot of sense (given the Google Cloud Bigtable announcement—more below).
http://blog.ubeeko.com/hbase-one-api-to-rule-them-all/
Apache Flink has been mentioned as an alternative to Apache Spark in several articles over the past few months. InfoWorld has a great summary of the differences of these two projects in areas like stream processing, memory management, and python apis (note that there is also some follow-up information in the comments).
http://www.infoworld.com/article/2919602/hadoop/flink-hadoops-new-contender-for-mapreduce-spark.html
Slides from many presentations as well as videos of keynotes from Strata + Hadoop World are available on the conference website.
http://strataconf.com/big-data-conference-uk-2015/public/schedule/proceedings
Over 200 issues were resolved as part of Apache Drill 0.9, which was released this week. Key new features include authentication for java/c++/JDBC/ODBC clients, impersonation, ownership chaining, extended json support, avro support, and enhancements to Parquet (columnar storage format) and Calcite (SQL parser).
http://drill.apache.org/blog/2015/05/04/drill-0.9-released/
Pivotal has announced a new version of their distribution, Pivotal HD, which is aligned to the Open Data Platform. In addition, the new release adopts Apache Spark, Apache Ranger, and more.
http://sdtimes.com/pivotal-announces-first-version-of-pivotal-hd-based-on-open-data-platform/
Cask has announced version 3.0 of the Cask Data Application Platform (CDAP). The release contains a major new feature called Application Templates for implementing common Hadoop workloads like ETLs from various sources to HDFS and HBase. Other features are a new role-based web UI, enhanced metrics, and support for querying CDAP Table datasets via Hive.
http://blog.cask.co/2015/05/cdap-3-0-application-templates-slick-new-ui-and-much-more/
Google has announced a beta version of Google Cloud Bigtable, which is a NoSQL-as-a-Service solution that's API-compatible with Apache HBase. Google's Bigtable is the precursor to HBase (HBase was built on the architecture described in the BigTable paper), and Google is boasting its maturity, speed, security, and cost.
http://googlecloudplatform.blogspot.com/2015/05/introducing-Google-Cloud-Bigtable.html
Cloudera has announced that Apache Phoenix, the SQL engine for Apache HBase, is now part of Cloudera Labs.
http://blog.cloudera.com/blog/2015/05/apache-phoenix-joins-cloudera-labs/
Qubole announced a new HBase-as-a-Service offering that's powered by HBase 1.0.0 and Hadoop 2.6.0. The introductory blog post has more information about the offering, such as optimization (e.g. using ephemeral nodes for compactions), cluster management, and incremental backup/restore to S3.
http://www.qubole.com/blog/product/announcing-quboles-hbase-as-a-service-for-aws/
Apache Lens (incubating), which provides a unified engine for querying data across multiple data stores including Hadoop, has released version 2.1.0-beta-incubating.
Hivemall, which is a machine learning library for Hive, has released a new stable version 0.3.1. This release includes so big changes: the license has changed from LGPLv2 to Apache v2 and Hivemall can be used with Pig 0.15 or later.
https://github.com/myui/hivemall/releases/tag/v0.3.1
OpenTSDB, the timeseries database built on HBase, released version 2.1.0 this week. The release includes a number of bug fixes.
https://github.com/OpenTSDB/opentsdb/releases/tag/v2.1.0
Ivory is a data store for facts and features to be used by a machine learning pipeline. The data store is optimized for scans, as it's backed by files in HDFS or S3.
https://github.com/ambiata/ivory
Curated by Datadog ( http://www.datadoghq.com )
Joint Seattle Spark and Graph Meetup Extravaganza (Seattle) - Wednesday, May 13
http://www.meetup.com/Seattle-Spark-Meetup/events/218764023/
Querying Multiple Distributed Storage Systems with Apache Hive Robustly (Houston) - Tuesday, May 12
http://www.meetup.com/Houston-Hadoop-Meetup-Group/events/221600442/
Using R and Hadoop for Advanced Analytics at Scale (Houston) - Tuesday, May 12
http://www.meetup.com/houstonr/events/222068164/
Special Joint Meetup (St. Louis) - Thursday, May 14
http://www.meetup.com/St-Louis-Hadoop-Users-Group/events/219706470/
Spark ETL Techniques (Chicago) - Tuesday, May 12
http://www.meetup.com/Chicago-area-Hadoop-User-Group-CHUG/
Cask: Accelerating Hadoop Projects with the Cask Data Application Platform (Boston) - Thursday, May 14
http://www.meetup.com/bostonhadoop/events/221708233/
Apache Spark (Montreal) - Thursday, May 14
http://www.meetup.com/Big-Data-Developers-in-Montreal/events/222376502/
Introduction to Apache Flink (Stockholm) - Monday, May 11
http://www.meetup.com/Apache-Flink-Stockholm/events/221701529/
Hands-on Apache Spark! (Paris) - Wednesday, May 13
http://www.meetup.com/Paris-Scala-User-Group-PSUG/events/221820153/
Hadoop Meetup on the Topic of Security (Prague) - Thursday, May 14
http://www.meetup.com/CS-HUG/events/222207024/
Cloudera Meet and Greet (Budapest) - Tuesday, May 12
http://www.meetup.com/Cloudera-Tech-Meetup/events/222203835/
Hands-on Labs: Working with Amazon Redshift, EMR & S3 (Tel Aviv-Yafo) - Tuesday, May 12
http://www.meetup.com/AWS-IL/events/222107210/
Apache Spark: How It's Being Used in Production (Tel Aviv-Yafo) - Tuesday, May 12
http://www.meetup.com/israel-spark-users/events/221051039/
Big Data Meetup (Bangalore) - Friday, May 15
http://www.meetup.com/Bangalore-Hadoop-Meetups/events/221855921/
HDP Operations: Install and Manage with Apache Ambari (Bangalore) - Saturday, May 16
http://www.meetup.com/Bangalore-HORTONWORKS-Hadoop-Meetup/events/222064799/
Deep Dive on Spark-SQL (Gurgaon) - Sunday, May 17
http://www.meetup.com/Gurgaon-Apache-Spark-Meetup/events/221010502/
Spark Meetup (Shanghai) - Saturday, May 16
http://www.meetup.com/Shanghai-Apache-Spark-Meetup/events/222286912/