14 June 2015
Hadoop Summit was this week in San Jose, and there were many new releases and partnerships announced. This week's newsletter contains many of these as well as slides from several presentations at the conference (please send along other ones I missed!). Among the highlights, Spark 1.4, MapR 5.0, and HDP 2.3 were released, Teradata is adding support for Presto, and details on how the Hammerlab is integrating bioinformatics software with Hadoop.
A future version of Apache HBase will have improved support for storing cells sized over 10K—so-called Moderate Objects (MOBs). This support is part of CDH 5.4, and this post describes the architecture of the solution. In short, MOBs are sent to a special HFile called a MOB file, which is part of a special offline region, when flushed. More details of the read and write path implementation can be found in the post.
http://blog.cloudera.com/blog/2015/06/inside-apache-hbases-new-support-for-mobs/
Apache Spark 1.4, which was released this week (more below) includes alpha-level support for accessing Spark from R. Using the SparkR integration, it's possible to get a Spark-backed DataFrame with an API similar to dplyr.
http://databricks.com/blog/2015/06/09/announcing-sparkr-r-on-spark.html
HDP 2.3 (more details below) supports transparent data encryption at rest. For this, it uses Apache Ranger as a Key Management Server, which provides an API for access encryption keys as well as a web UI for managing keys and more.
http://hortonworks.com/blog/new-in-hdp-2-3-enterprise-grade-hdfs-data-at-rest-encryption/
This post describes how Spark Streaming achieves fault tolerance with source data from Kafka. Early versions of the integration make use of a write-ahead log to recover data during a failure, and there's a newer direct API which provides exactly-once semantics. The post also describes some of the expected failure scenarios and how the implementation achieves robustness.
Members of the Core Data Libraries team at Twitter presented on their experiences with optimizing Hadoop performance. Among the topics covered include using Xprof for profiling MapReduce jobs, optimizing intermediate compression, using Scala macros to implement raw comparators, and column projection and filtering with Parquet.
http://www.slideshare.net/alexlevenson/hadoop-summit-2015-performance-optimization-at-scale
Stream processing has recently become a big topic, but it's often most useful when used in conjunction with a batch framework. This presentation describes, at a high-level, a few different frameworks for achieve this. They include Summingbird, Spark, and Flink.
http://www.slideshare.net/gwenshap/have-your-cake-and-eat-it-too-49285958
Twitter has done a lot of work to speedup the launching of and reduce startup bandwidth of MapReduce jobs. The MapReduce Distributed Cache builds on the YARN local cache to expose and API for caching job resources like jars and files. With these changes, jobs typically submit less than 2MB of data to HDFS during job startup and nodes in the cluster download less than 840KB. This presentation describes the design and implementation of this feature.
http://www.slideshare.net/ctrezzo/hadoop-summit2015-yarnsharedcachev12
The Cloudera blog has a guest post that describes the experiences at LinkedIn that led to Kafka, the high-level concepts behind Kafka, and the success of Kafka as the core of the data platform at LinkedIn.
http://vision.cloudera.com/apache-kafka-a-platform-for-real-time-data-streams-part-1/
This presentation describes a number of pitfalls and related improvements/best practices when it comes to administering a Hadoop cluster. Topics covered include configuration (instance configuration and jvm settings), issues related to metadata files, HDFS ACLs, HDFS Snapshots, and DataNode volume failures.
http://www.slideshare.net/ChrisNauroth/hadoop-operations2015hadoopsummitsanjosev5
This post describes how the folks at Hammerlab integrate bioinformatics tools with Hadoop. Since the tools expect a POSIX-compatible filesystem, they've configured NFS access to HDFS. The post describes the workflow engine that they use and one of the gotchas of the NFS setup.
http://www.hammerlab.org/2015/06/11/biokepi-on-yarn/
This presentation gives an overview of building an end-to-end implementation of the lamdba architecture (speed + batch layers) with Spark streaming, Kafka, Cassandra, and Akka (for data ingestion). It contains brief introductions to Kafka and Cassandra, snippets of code, and examples of how to hook all the pieces together.
Netflix presented at Hadoop Summit on their data pipeline which sends data to S3 to be queried by Presto. They cite the speed, scalability (it's running on >200 nodes), ANSI SQL support, and AWS support as reasons that they use Presto. Their presentation also covers some of Netflix's contributions and several upcoming features.
http://www.slideshare.net/NezihYigitbasi/prestoatnetflixhadoopsummit15
Huawei is an early adopter of Spark. This post describes some of the problems their solving with Spark (and the massive data volumes that they see when capturing raw data), summarizes some of their open-source contributions, and enumerates some features they're working on for future releases.
http://databricks.com/blog/2015/06/09/huawei-embraces-open-source-apache-spark.html
The DBMS2 blog reports that Teradata is planning on supporting Presto (the open-source SQL-on-Hadoop and more from Facebook). Several engineers from the Hadapt acquisition will work on Presto, and Teradata will sell support subscriptions. The post has more details on Presto and the short-term roadmap.
http://www.dbms2.com/2015/06/08/teradata-will-support-presto/
MapR and Microsoft have announced a partnership to bring MapR's distribution (including MapR-DB) to the Microsoft Azure cloud. Look for the offering sometime this summer.
http://finance.yahoo.com/news/mapr-offer-cloud-based-hadoop-120000319.html
KDnuggets has an interview with Beth Smith of IBM about the evolving role of analytics and analytics software. Among the topics covered are IBM's Hadoop distribution and contributions, IBM's membership in the Open Data Platform, and IBM's commitment to Spark (they recently opened a Spark Technology Center in San Francisco).
http://www.kdnuggets.com/2015/06/interview-beth-smith-ibm-analytics-hadoop-spark.html
Hortonworks announced HDP 2.3 this week, which contains a number of new features to improve operations, development, and security & data governance. Many of these improvements are made possible by a new version of Ambari, which includes support for smart configuration, a UI to manage the capacity scheduler configuration, a SQL editor, an Apache Pig Latin editor, and a new UI for Apache Falcon. For enterprise support customers, Hortonworks has introduced a new SmartSense program for proactive support.
http://hortonworks.com/blog/hortonworks-data-platform-2-3-delivering-transformational-outcomes/
DataTorrent has announced that they're open-sourcing DataTorrent RTS, which is a unified stream and batch processing system, as Project Apex. Apex is meant to compete with Spark and Storm while striving to offer more enterprise features and better performance.
A new version of BlueData's EPIC software for managing Hadoop clusters was released this week. It adds a number of new features, including integration with Apache Ambari, support for CDH 5.3 and HDP 2.2, support for Apache Spark 1.3.1, support for Kerberos, and more. The new version also supports running Hadoop or Spark within docker containers.
http://www.bluedata.com/article/bluedata-harnesses-latest-innovations-in-hadoop-and-spark-with-new-product-release/
http://www.bluedata.com/article/bluedata-announces-support-for-hadoop-and-spark-on-docker-containers/
MapR 5.0 was released this week with several new features. These include a new data replication framework to replicate data from MapR-DB to ElasticSearch, the addition of Apache Hadoop 2.7, support for Apache Drill 1.x, and new auditing capabilities that produce JSON files and integrate with Apache Drill.
https://www.mapr.com/company/press-releases/mapr-50-extends-hadoop-new-class-real-time-applications
For folks working with Apache Kafka and using Puppet for configuration, there's a new puppet module for managing the Confluent Platform Schema Registry.
https://github.com/verisign/puppet-schema_registry
Airflow is a new open-source workflow system from Airbnb. While the system is not Hadoop-specific, it includes support for Hive, Presto, HDFS, and other pieces of Hadoop infrastructure. Airflow is broken into several components including a web UI, a metadata repository, and a web CLI.
http://nerds.airbnb.com/airflow/
Version 0.3.2 of Hivemall, the machine learning UDF library for Hive, includes support for anomaly detection using Local Outlier Factor and support for polynomial features when performing non-linear regression.
https://github.com/myui/hivemall/releases/tag/v0.3.2
Cascading 3.0 was released this week with support for Apache Tez as a backend. Cascading supports all the major Hadoop distributions (CDH, MapR, HDP) as well as several Hadoop service providers (EMR, Qubole, Altiscale).
LinkedIn has open-sourced Pinot, their realtime distributed OLAP database that integrates with Kafka and Hadoop. Pinot is used to power many analytics products at LinkedIn, where it provides < 100ms of end-end latency.
https://engineering.linkedin.com/pinot/open-sourcing-pinot-scaling-wall-real-time-analytics
Apache Spark 1.4.0 was released with the new SparkR bindings, a new visualization feature, Python 3 support, and many improvements to Spark SQL, MLlib, and Spark Streaming.
http://spark.apache.org/releases/spark-release-1-4-0.html
PyKafka (formerly known as Samsa) has released a 1.0 version. The new version supports Kafka 0.8.2 and aims to have similar features to the JVM Kafka client. It doesn't yet have asynchronous producer support.
https://github.com/parsely/pykafka
Version 0.9.4 of kafka-python, another popular python Kafka client, was released. This version focusses on stability, bug fixes, documentation improvements, and cleanups.
https://github.com/mumrah/kafka-python/releases/tag/v0.9.4
Cloudera Enterprise 5.3.4 includes fixes rolling upgrades, the fair scheduler, Impala, and HiveServer2. It also contains several fixes to Cloudera Manager.
Sparkit-learn is a new project to bring together PySpark and scikit-learn. The project readme has some example usages, such as building ArrayRDD and DictRDDs and build distributed classifiers.
https://github.com/lensacom/sparkit-learn
Curated by Datadog ( http://www.datadoghq.com ) UNITED STATES
Spark Summit 2015 Live Streaming (San Francisco) - Monday, June 15
http://www.meetup.com/spark-online/events/223111627/
Breakthrough OLAP Performance on Cassandra and Spark (Santa Clara) - Monday, June 15
http://www.meetup.com/DataStax-Cassandra-South-Bay-Users/events/222799171/
Spark 1.4 Deep Dive & Spark Committers Q&A (San Francisco) - Monday, June 15
http://www.meetup.com/spark-users/events/222596215/
Elastic Meetup at MapR (San Jose) - Tuesday, June 16
http://www.meetup.com/Silicon-Valley-Elastic-Fantastics/events/222666126/
Apache Flink: Unifying Batch and Streaming Modern Data Analysis (Redwood City) - Wednesday, June 17
http://www.meetup.com/Bay-Area-Apache-Flink-Meetup/events/222879388/
Scaling with Couchbase, Kafka and Apache Spark (Pasadena) - Wednesday, June 17
http://www.meetup.com/Pasadena-Big-Data-Users-Group/events/222404600/
Working with Spark's GraphX Libraries (Mason) - Tuesday, June 16
http://www.meetup.com/Cincinnati-Apache-Spark-Meetup/events/222506183/
UNITED KINGDOM
June 2015 Hadoop User Group Meetup (London) - Tuesday, June 16
http://www.meetup.com/hadoop-users-group-uk/events/222871938/
Lighting the Spark! (London) - Wednesday, June 17
http://www.meetup.com/Big-Data-Developers-in-London/events/222961135/
Apache Flink Hands-On (Stockholm) - Wednesday, June 17
http://www.meetup.com/Apache-Flink-Stockholm/events/222474895/
Introduction to Practical Big Data with Apache Spark (Barcelona) - Monday, June 15
http://www.meetup.com/Spark-Barcelona/events/222865010/
First Spark Belgium Meetup (Mechelen) - Wednesday, June 17
http://www.meetup.com/Spark-Belgium/events/222632697/
To Hadoop or Not to Hadoop (Kongens Lyngby) - Tuesday, June 16
http://www.meetup.com/Big-Data-Denmark/events/222709594/
Introduction to Apache Flink Workshop (Berlin) - Wednesday, June 17
http://www.meetup.com/Apache-Flink-Meetup/events/222935794/
Mysteries of the Universe, Spark and DataFrames (Krakow) - Thursday, June 18
http://www.meetup.com/datakrk/events/223195346/
RUSSIA
Apache Spark (Moscow) - Thursday, June 18
http://www.meetup.com/Big-Data-Developers-in-Moscow/events/223012866/
Spark Streaming with Kafka (Bangalore) - Saturday, June 20
http://www.meetup.com/Real-Time-Data-Processing-and-Cloud-Computing/events/223123744/
If you didn't receive this email directly, and you'd like to subscribe to weekly emails please visit http://hadoopweekly.com