17 January 2016
Given that we're still in the first month of the year, there are still several articles this week reviewing 2015 or making predictions for 2016. And articles this week cover ecosystem projects that have seen major adoption (or are expected to) in the coming year—Kafka, Flink, and Kudu to name a few. Also of note, the program for Spark Summit East was announced, and InfoWorld has a look at the state of open-source big data software as a business.
The Cloudera blog has the third in a series on YARN. This post covers the scheduler—in particular the fair scheduler. It describes queues (including hierarchical queues), queue weights, and more. There are visual aids and xml snippets to demonstrate these main concepts.
http://blog.cloudera.com/blog/2016/01/untangling-apache-hadoop-yarn-part-3/
The Hortonworks blog has a guest post by one of its customers, Arkena. They are doing advanced analytics on massive amounts of streaming video data. The post details how they use Flume, Hive, Spark Streaming, ElasticSearch, and more.
The MapR blog has a whiteboard walkthrough (both a video and transcript) comparing Apache Spark and Apache Flink. The walkthrough covers the differences between the two—highlighting the key distinction of real time vs. microbatch. It also discusses several use cases (e.g. fraud detection, network anomaly detection) describing when microbatch or real time stream is more appropriate.
https://www.mapr.com/blog/apache-spark-vs-apache-flink-whiteboard-walkthrough
The morning paper looks at a new system for providing "distributed ACID transactions with strict serializability, high availability, high throughput and low latency." The protocols take advantage of FaRM (Fast Remote Memory) and RDMA (Remote Direct Memory Access). There's interesting discussions about the expected characteristics of data center hardware as well as how the protocol optimizes for CPU bottlenecks (given that RDMA eliminates other I/O bottlenecks).
http://blog.acolyer.org/2016/01/14/no-compromises/
The AWS blog has an example of using Spark Streaming from an Amazon EMR cluster to query data in Amazon Kinesis. Further, by microbatching the data and converting a DStream to a DataFrame, queries can be written using Spark SQL. The tutorial has a walkthrough of starting a cluster and a Python application for generating test data in Kinesis.
The IBM developer blog has an overview of some of the key benefits of using Parquet with Spark SQL. The post has a bunch of benchmarking numbers (e.g. 11x faster on Parquet than text files) as well as discussions around Spark internals (e.g. a look at how PushedFilters effect the query plan of a simple SQL query).
https://developer.ibm.com/hadoop/blog/2016/01/14/5-reasons-to-choose-parquet-for-spark-sql/
It can be frustrating to use the HBase shell with binary data, because the output from read operations is hex-encoded binary (which is difficult to understand). But HBase supports customer formatters for converting bytes to a human readable format when using the shell. This article describes how to build and use a custom formatter (with a focus on Avro data).
MapR has posted about their top-10 posts of 2015. There's a good mix of background, tutorial, and architecture posts covering topics like Spark, HBase, YARN, and Drill.
https://www.mapr.com/blog/top-10-most-popular-mapr-blog-posts-2015
Apache Kafka and Amazon Kinesis provide a similar set of APIs and guarantees. One of the considerations when choosing one or the other is performance. This post compares throughput performance of the two, varying the number of parallel producers and batch sizes to quantify the impact.
http://insightdataengineering.com/blog/ingestion-comparison/
DataInformed has a look at the potential of the Kudu project, which was open-sourced by Cloudera and submitted to the Apache incubator. In the most extreme version, Kudu might become a complete replacement for HDFS (while offering new features like updates and random-access). The post explores this idea and some of the milestones that might get the project to realize that potential.
http://data-informed.com/kudu-and-the-ongoing-evolution-of-hadoop/
In a few weeks, Hadoop will be 10 years old, and Cloudera has published an infographic celebrating the milestone. There are lots of numbers about contributions and many highlighted achievements.
http://www.cloudera.com/content/dam/www/static/images/logos/hadoop-10th-birthday-infographic.jpg
InfoWorld has an article about "16 for '16" things to know about the Hadoop and Spark ecosystem (covering hot topics like Zeppelin, security, Kafka, and Impala). The post also discusses some up-and-coming technologies to keep an eye on and some "technologies I'd rather forget."
TechRepublic has a bearish outlook on the big data market. It mentions that only one company (RedHat) has been able to make a strong go as a open source play. And while Hortonworks is public, they're not looking to turn a profit until 2017.
http://www.techrepublic.com/article/why-this-big-data-unicorn-isnt-going-to-gallop/
The agenda for Spark Summit East, which takes place in New York on February 16th-18th, is now available. The Databricks blog has highlighted several of the talks and training sessions. If you're planning on going, the post also includes discount information.
https://databricks.com/blog/2016/01/13/spark-summit-east-2016-agenda-is-now-available.html
Datameer has an article analyzing several years of big data and Hadoop news articles based on data from TechNews.IO. There are a number of interesting outputs, including a look at the top news days of the year, the top publishers of big data and Hadoop articles, and the most prolific big data/Hadoop article authors.
http://www.datameer.com/company/datameer-blog/big-data-hadoop-in-the-news-2015-analysis/
In another look at the year ahead, the Pivotal blog has a post with five predictions for 2016. Forecasted items include increased productivity of the Apache Hadoop ecosystem, an increase in adoption of open-source in corporate governance, and real-time analytics going mainstream.
https://blog.pivotal.io/big-data-pivotal/features/5-big-predictions-for-big-data-in-2016
Hortonworks is launching a new partner program called "PartnerWorks," and this article highlights some of the key provisions of the new program.
http://hortonworks.com/blog/why-hortonworks-and-why-partnerworks/
MapR announced that they've added support for Apache Drill 1.4. to their distribution. In their post announcing the support, they highlight many of the key features of the release.
https://www.mapr.com/blog/brief-overview-performance-enhancements-apache-drill-14
Curated by Datadog ( http://www.datadog.com )
Stream Processing Systems (Sunnyvale) - Wednesday, January 20
http://www.meetup.com/SF-Bay-Areas-Big-Data-Think-Tank/events/227907632/
LinkedIn’s Big Data Pipeline with Kafka, Hadoop, and Couchbase (Mountain View) - Thursday, January 21
http://www.meetup.com/Couchbase-Silicon-Valley/events/227417917/
January Hive User Group Meeting (Palo Alto) - Thursday, January 21
http://www.meetup.com/Hive-User-Group-Meeting/events/227463783/
Cassandra Data Maintenance with Spark (Santa Clara) - Thursday, January 21
http://www.meetup.com/DataStax-Cassandra-South-Bay-Users/events/228016126/
Real-Time Operational Analytics with Apache Spark (Bellevue) - Wednesday, January 20
http://www.meetup.com/Big-Data-Bellevue-BDB/events/219852705/
Deploying the Hadoop Ecosystem! (Salt Lake City) - Wednesday, January 20
http://www.meetup.com/BigDataUtah/events/226386260/
An Evening with Chris Fregly, Spark Author/Contributor (Austin) - Tuesday, January 19
http://www.meetup.com/austin-spark-meetup/events/226678379/
Flink and Nifi, 2 Stars in the Apache Big Data Constellation (Chicago) - Tuesday, January 19
http://www.meetup.com/Chicago-Apache-Flink-Meetup/events/227497510/
Hadoop 101 (Brentwood) - Thursday, January 21
http://www.meetup.com/Nashville-Hadoop-Meetup/events/227857082/
Introduction to Apache Kudu (Atlanta) - Tuesday, January 19
http://www.meetup.com/Atlanta-Hadoop-Users-Group/events/227377363/
SparkR in Big Data (Atlanta) - Wednesday, January 20
http://www.meetup.com/ADP-Atlanta-Data-Science-Meetup/events/227438694/
Solr, Spark, and Zeppelin: The Analytics Toolkit for Distributed Big Data (Richmond) - Tuesday, January 19
http://www.meetup.com/RVA-Data-Hackers/events/226915883/
Hadoop and Metron: An Introduction to Open Source Security with CapitalOne (Washington) - Wednesday, January 20
http://www.meetup.com/Washington-DC-Hortonworks-User-Group-Meetup/events/227337678/
Moving from Microsoft SQL to Hive (Washington) - Thursday, January 21
http://www.meetup.com/Washington-DC-Apache-Hive-Users-Group/events/226015860/
New Functions and Workflow Examples with Spark DataFrames (Vancouver) - Monday, January 18
http://www.meetup.com/Vancouver-Spark/events/227930158/
Northern Spark Meetup (Groningen) - January 20, 2016
http://www.meetup.com/Northern-Spark-Meetup-Data-Science-Engineering-at-Scale/events/226674345/
Real World Case: Spark and the Lifelog App by Sony Mobile (Copenhagen) - Thursday, January 21
http://www.meetup.com/CphSpark/events/226855790/
Papers We Love: Resilient Distributed Datasets (Warsaw) - Monday, January 18
http://www.meetup.com/Warsaw-Spark/events/227335322/
Second Apache Spark Workshop (Cluj-Napoca) - Wednesday, January 20
http://www.meetup.com/Big-Data-Data-Science-Meetup-Cluj-Napoca/events/228053016/
Spark Meetup (Shanghai) - Saturday, January 23
http://www.meetup.com/Shanghai-Apache-Spark-Meetup/events/227914069/
