21 February 2016
It was a busy week of announcements and releases, many of which coincided with Spark Summit East that took place in New York. Among the highlights are the new Apache Arrow project (in-memory columnar storage) and the community edition of Databricks. Netflix and Google both wrote a bit about their big data infrastructure, and there are great articles about Cassandra, MLlib, Python & Hadoop, Kafka Connect, and more.
This post looks at why a time series is a good model for storing many types of medical data, why Cassandra is a good system for storying time series data, how to model time series data in Cassandra (including some example table definitions and row-level inserts), and more.
http://dtrapezoid.com/time-series-data-modeling-for-medical-devices.html
The Netflix blog has a post on the evolution of their data pipeline, which is currently handling over 1 petabyte of data per day. Originally built on Chukwa and Amazon EMR, the latest system (called Keystone), is based on Kafka, EMR, and is supplemented by Elastic Search and streaming consumers (Spark and others).
http://techblog.netflix.com/2016/02/evolution-of-netflix-data-pipeline.html
The Cloudera blog has an example of using Spark's MLlib to do churn analysis. The post shows how to use Spark SQL (from Python) to define the schema of a csv file, build a feature vector, use a RandomForestClassifier to train a model, and validate the generated model.
http://blog.cloudera.com/blog/2016/02/how-to-predict-telco-churn-with-apache-spark-mllib/
The IBM Hadoop blog has a presentation on Spark troubleshooting—covering topics ranging from compiling Spark to optimizing cluster utilization to collecting thread dumps for debugging production issues. The presentation covers over 10 different troubleshooting tasks.
https://developer.ibm.com/hadoop/blog/2016/02/16/beginners-guide-apache-spark-troubleshooting/
Hortonworks is starting a new weekly blog series highlight articles from their Hortonworks Community Connection. This week, the selected articles cover NiFi, Storm, Kafka, and Ambari. They also highlighted three community questions from the week.
http://hortonworks.com/blog/top-3-articles-that-ever-hadoop-developer-should-read-week-072016/
Apache Arrow is a new top-level project for columnar, in-memory data spun out of the Apache Drill project. The Apache blog has the announcement, MapR has a post about the origins and design of Arrow (aka Value Vectors), and Cloudera has a post about their plans for integrating Arrow with other big data projects.
https://blogs.apache.org/foundation/entry/the_apache_software_foundation_announces87
https://www.mapr.com/blog/apache-arrow-and-value-vectors
http://blog.cloudera.com/blog/2016/02/introducing-apache-arrow-a-fast-interoperable-in-memory-columnar-data-structure-standard/
Cloudera has an update on the state of Python and Hadoop. The past year has seen improvements to PySpark and the emergence of several Python DSLs, which have great improved the utility of Python for big data processing. The post recaps the current landscape and describes two new initiatives—efficient data interchange via Apache Arrow and Cloudera Manager integration with Continuum Analytic's Anaconda Python.
http://vision.cloudera.com/python-and-hadoop-a-state-of-the-union/
While modern tools have automated much of Hadoop configuration, many settings aren't "set it and forget it." As a cluster grows or utilization goes up, settings that made sense initially can cause major problems. This post describes one such issue (and how to resolve it)—a HDFS NameNode that became overwhelmed due to the number of HDFS blocks.
http://distcp.blogspot.com/2016/02/too-many-hdfs-blocks.html
The Confluent blog has a post on Kafka Connect, a new tool for moving data between Kafka and other systems. Part of the recently released Kafka 0.9.0, Kafka Connect ships with connectors for HDFS and JDBC. The introductory blog post has many more details on the design and implementation.
This post describes how to build a Flume Source that plays well with the JMX MBeans that keep track of message statistics for reporting to Cloudera Manager (and other systems).
http://blog.xebia.com/custom-flume-cloudera-manager/
Google has written about several large-scale sorting experiments that they've run over the past 10 years. Describing the road to sorting 50PB in 2012, the post shares a lot of interesting anecdotes (such as the use of Reed-Solomon encoding in 2010 and that their benchmark from 2012 is faster than the 2015 GraySort winner).
This post looks at the big data landscape, which the author argues has matured and is in a deployment phase. It's still difficult to start a big data system from scratch, but the ecosystem has matured (the post includes some financial numbers to back up this claim). The post also has a map of the landscape, which maps several areas like infrastructure, analytics, and open source.
http://mattturck.com/2016/02/01/big-data-landscape/
Hortonworks has a post about what they've seen with enterprise adoption of Spark and how they're helping to speed up adoption. This includes making analytics/data science easier, hardening Spark for enterprise, and innovating core Hadoop.
http://hortonworks.com/blog/spark-summit-accelerating-enterprise-spark/
Spark Summit East was this week in NYC. The Databricks blog has a post highlighting several keynotes and themes from the conference.
https://databricks.com/blog/2016/02/18/a-look-back-at-spark-summit-east-2016-thank-you-nyc.html
Qubole has announced that they're making their Qubole Data Service available to university classes at no cost. There website has more details on eligibility and the process for applying.
https://www.qubole.com/blog/big-data/university/
For the Cassandra users on the list, Planet Cassandra has a new "This Week in Cassandra." It's a collection of news and blog posts, Jira updates, upcoming events, and more.
http://www.planetcassandra.org/blog/this-week-in-cassandra-2192016/
Versions 2.1.10 and 3.1.0 of Apache Curator, the high-level framework for interacting with Apache ZooKeeper, was released this week. The releases each contain several bug fixes, improvements, and new features.
At Spark Summit in NYC this week, Databricks announced a new community edition of their Spark-as-a-Service platform and Databricks Dashboards. The community edition (currently in beta) provides free access to Databricks, and Dashboards provide a mechanism for building interactive web pages, logically separating charts and graphs from a single notebook, and more.
https://databricks.com/blog/2016/02/17/introducing-databricks-community-edition-apache-spark-for-all.html
https://databricks.com/blog/2016/02/17/introducing-databricks-dashboards.html
Apache BigTop 1.1.0 was released this week. Bigtop is a packaging, smoke/integration testing, and virtualization system for the Hadoop ecosystem. This release is built on Hadoop 2.7.1, supports five operating systems, adds support for Apache Hama and Zeppelin (incubating), adds support for producing docker images, and more.
https://blogs.apache.org/bigtop/entry/release_1_1_0_is
The Apache Hive team disclosed CVE-2015-7521, which is an authorization bug that can be used to circumvent some authorization checks. The team has provide a new jar and configuration that produce a run-time work-around. Affected versions are Hive 0.13.x through various versions of 1.2.x (see the announcement for the full list).
Cloudera and Continuum have announced an integration for deploying Anaconda Python via Cloudera Manager. This introductory post describes how to configure CM to find and install the Anaconda Parcels and demonstrates how to take advantage of the install using pyspark.
https://www.continuum.io/blog/developer-blog/making-python-hadoop-easier-anaconda-and-cloudera
Qubole has announced a new Qubole Data Service SDK for Airflow. With the SDK, it's easy to integrate Airflow workflow steps that execute via the Qubole service.
https://www.qubole.com/blog/product/open-source-integration-of-airflow-and-qubole/
Version 0.5.1 (with bug fixes to the recently announced version 0.5.0) of Apache NiFi was released this week. NiFi is a system for processing and distributing data. The new release improves S3, Hive, and encryption support, adds new extensions for Reimann, ElasticSearch, and Avro, and has improved data inspection and state management.
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.5.1
Version 1.7.0 of Apache Accumulo, the distributed key-value store, was released this week. The new version is backwards compatible with previous versions, and focusses on bug fixes and improvements in the areas of security (new Kerberos authentication), availability (improved data center replication), and extensibility (e.g.support for HTrace for distributed tracing). There are many more details about the release on the Accumulo web site.
http://accumulo.apache.org/release_notes/1.7.0.html
Apache Kafka 0.9.0.1 was released this week. The Confluent blog has a good recap of bug fixes and improvements from the release.
http://mail-archives.apache.org/mod_mbox/www-announce/201602.mbox/%3C20160219180358.6282C1880C@minotaur.apache.org%3E
http://www.confluent.io/blog/announcing-apache-kafka-0.9.0.1-and-confluent-platform-2.0.1
Apache Zookeeper 3.4.8 was released. The new version addresses several bugs.
IBM has announced the IBM Platform Conductor for Spark. The product focusses on easier deployment and optimized resource scheduling.
http://siliconangle.com/blog/2016/02/18/big-blue-touts-an-easier-way-to-deploy-apache-spark/
Curated by Datadog ( http://www.datadog.com )
How Netflix Handles Data Streams + Intro to Apache Kudu (San Francisco) - Tuesday, February 23
http://www.meetup.com/San-Francisco-Data-Engineering/events/228949298/
Building Robust, Scalable, and Adaptive Applications on Spark Streaming (San Francisco) - Tuesday, February 23
http://www.meetup.com/spark-users/events/228979867/
Enterprise Grade Streaming on Hadoop in Under 2ms (San Francisco) - Wednesday, February 24
http://www.meetup.com/SF-Data-Science/events/228111922/
The Future of Apache Storm w/ Taylor Goetz (Los Angeles) - Thursday, February 25
http://www.meetup.com/LA-Real-World-Lambda-Applications/events/228382007/
Streaming App w/ under 2ms Latency in Hadoop + Ingestion from Nifi (San Jose) - Thursday, February 25
http://www.meetup.com/Big-Data-native-Hadoop-Ingest-and-Transform-Bay-Area/events/228546474/
Hadoop and Spark: A Perfect Duo for Big Data (San Francisco) - Thursday, February 25
http://www.meetup.com/SF-Data-Science/events/228958752/
Using Cassandra with Spark (Portland) - Monday, February 22
http://www.meetup.com/DataStax-Cassandra-Portland-Users/events/228536727/
Seattle Scalability Meetup: Riak TS + Kudu (Seattle) - Wednesday, February 24
http://www.meetup.com/Seattle-Scalability-Meetup/events/225955122/
SparkCLR and Kafka+Spark (Bellevue) - Thursday, February 25
http://www.meetup.com/Seattle-Spark-Meetup/events/227398619/
February Edition of MOHUG (Dublin) - Thursday, February 25
http://www.meetup.com/MOHUG-Mid-Ohio-Hadoop-User-Group/events/227864910/
MemSQL on "Building Real-Time Data Pipelines" (Charlotte) - Wednesday, February 24
http://www.meetup.com/CharlotteHUG/events/225229241/
H2O Rains with Databricks Cloud (Washington, DC) - Wednesday, February 24
http://www.meetup.com/Washington-DC-Area-Spark-Interactive/events/228569714/
Advanced Spark Meetup (Laurel) - Monday, February 22
http://www.meetup.com/Apache-Spark-Maryland/events/227985407/
Moving from Microsoft SQL to Hive (Washington) - Tuesday, February 23
http://www.meetup.com/Washington-DC-Apache-Hive-Users-Group/events/226015860/
February Meetup (Ottawa) - Tuesday, February 23
http://www.meetup.com/Ottawa-Big-Data-Enthusiasts/events/228870361/
Hadoop Night Featuring Hortonworks and Pivotal HAWQ (Toronto) - Wednesday, February 24
http://www.meetup.com/Toronto-Pivotal-User-Group/events/227459868/
Toronto Apache Spark #6 (Toronto) - Wednesday, February 24
http://www.meetup.com/Toronto-Apache-Spark/events/227574278/
Automatic Features Generation and News from Spark Summit (Montreal) - Wednesday, February 24
http://www.meetup.com/Montreal-Apache-Spark-Meetup/events/228535534/
The Emerging Fast Data Architecture by Dean Wampler (Toronto) - Thursday, February 25
http://www.meetup.com/scalator/events/227584833/
February 2016 Meetup (London) - Tuesday, February 23
http://www.meetup.com/hadoop-users-group-uk/events/228756553/
Intro to Apache Flink: Networking (Madrid) - Thursday, February 25
http://www.meetup.com/Meetup-de-Apache-Flink-en-Madrid/events/228608831/
Kafka (Prague) - Thursday, February 25
http://www.meetup.com/CS-HUG/events/228775422/
Spark in Action: Spark Streaming (Warsaw) - Monday, February 22
http://www.meetup.com/Warsaw-Spark/events/228186010/
The First Meetup for 2016! (Singapore) - Thursday, February 25
http://www.meetup.com/Spark-Singapore/events/228253262/
Big Data on Azure Using HDInsight (Perth) - Wednesday, February 24
http://www.meetup.com/Perth-Cloud/events/228513900/