06 December 2015
There's lots of great technical content this week covering large potions of the Hadoop ecosystem as well as distributed systems in general. In news, the CfP for Kafka Summit is open, and there's a new eBook covering "Data Munging with Hadoop." With only a couple of releases, there should be plenty of time to concentrate on the abundance of technical posts.
The Cloudera blog has a post on the past year of Apache Spark development, which has included a lot of work on Spark streaming, Hive-on-Spark, and tools for data science. Two more articles cover the DataFrames API (which enables easier development, better performance, improved interoperability, and more), MLlib (which provides implementations of popular machine learning algorithms), and the Hive-on-Spark project (there have been a number of recent improvements towards a production-ready version).
http://vision.cloudera.com/a-year-in-review-for-apache-spark/
http://blog.cloudera.com/blog/2015/11/sustained-innovation-in-apache-spark-dataframes-spark-sql-and-mllib/
http://blog.cloudera.com/blog/2015/12/progress-report-hive-on-spark-nears-production-readiness/
The IBM Hadoop Dev blog describes some of the recent security features that IBM have added to their distribution via Knox and Ranger. It also describes some plans for the future of these two projects.
https://developer.ibm.com/hadoop/blog/2015/12/01/ibm-security-current-and-near-future-work/
This tutorial describes integrating the H2O.ai machine learning libraries with Spark (and in particular Databricks). Specifically, the post shows how to tokenize data, use the TF-IDF libraries from Spark to identify important works, and build an H2O deep learning model to detect spam.
https://databricks.com/blog/2015/12/02/databricks-and-h2o-make-it-rain-with-sparkling-water.html
This post aims to highlight the key concepts of distributed systems (with links out to the relevant papers). The content is available as a presentation, a video recording, and a blog post. In total, the post covers nine topics, including timing model, failure modes, and consensus. Whether you're new to distributed systems or are looking to brush up on the main concepts, this is an important resource.
http://videlalvaro.github.io/2015/12/learning-about-distributed-systems.html
The GoDataDriven blog has a two-part series describing how to configure Cloudera CDH on the Azure cloud. In addition to the common software installation and configuration, the post describes network typology/architecture (including setting up a VPN tunnel), Azure basics, and describes some modifications to the pre-built Cloudera-Azure template.
http://blog.godatadriven.com/installing-cloudera-on-azure-1.html
Region replicas are a relatively new feature of Apache HBase. By enabling them and specifying the correct flag at query time, HBase can delivery high availability of reads. This tutorial describes how to configure HBase for HA reads and gives a quick walkthrough of using the HBase CLI to create a table with replicas and query secondary regions.
https://developer.ibm.com/hadoop/blog/2015/12/02/hbase-read-ha/
This post describes how to build a standalone Hive metastore without a Hadoop cluster, which is the scenario for running Presto with a blobstore like S3 (the instructions also mention how to setup S3 access).
http://distcp.blogspot.com/2015/12/creating-standalone-hivemetastore-not.html
Apache Flink is a streaming-first system, which means it doesn't require micro-batching like Spark streaming. But micro-batching, or windowing data, is often useful since it can be convenient to process a set of events at once. This post explores the rich windowing semantics in Flink streaming—time windows (with different notions of time), key-based partitioning of windows, count-based windows, and the base interfaces for building a new type of window function.
http://flink.apache.org/news/2015/12/04/Introducing-windows.html
The MapR blog has a short video "Whiteboard Walkthrough" about Apache Myriad (incubating), which is a system for running YARN atop of Apache Mesos. The full transcript of the walkthrough is posted, too, if you'd rather read the information.
https://www.mapr.com/blog/fine-grained-scaling-apache-myriad-whiteboard-walkthrough
Kafka Summit takes place in April in San Francisco. The call for proposals is open until January 11, and early bird registration is until January 15th. Disclosure: I'm on the program committee for the conference.
EMC Elastic Cloud Storage (ECS) has been certified for Hortonworks HDP. The integration makes use of Ambari to deploy HDP with the ECS file system instead of HDFS.
http://hortonworks.com/blog/emc-elastic-cloud-storage-now-certified-on-hdp/
The spark-sql-perf project is used to benchmark Spark SQL. Recently, the benchmark has integrated all of the queries fro the TPC-DS Benchmark, and IBM has been able to run 89 of the 99 queries when encoding data using the Parquet file format.
https://developer.ibm.com/hadoop/blog/2015/11/30/99-tpc-ds-queries-integrated-into-spark-sql-perf/
"Data Munging with Hadoop" is a new eBook by Ofer Mendelevitch and Casey Stella. It covers a range of topics, from implementing quality checks to handling time-series data.
http://www.informit.com/store/data-munging-with-hadoop-9780134435480
Datamation has a list of twenty companies associated with Big Data. In addition to the big name Hadoop vendors, there are a number of companies that hadn't yet crossed my radar.
http://www.datamation.com/data-center/20-big-data-companies-leading-the-way-1.html
Cloudera announced a new QuickStart Docker image for evaluating CDH. As always, it's interesting to see the ways in which Hadoop and Docker are integrating.
Version 4.3.2 of Apache Bookeeper, the replicated log service, has been released with an important bug fix.
http://bookkeeper.apache.org/docs/r4.3.2/releaseNotes.html
Curated by Datadog ( http://www.datadog.com )
A noETL Parallel Streaming Transformation Loader Using Spark, Kafka & Vertica (Los Angeles) - Monday, December 7
http://www.meetup.com/Los-Angeles-Apache-Spark-Users-Group/events/226595726/
Kudu: New Apache Hadoop Storage for Fast Analytics on Fast Data (Palo Alto) - Wednesday, December 9
http://www.meetup.com/SF-Bay-Areas-Big-Data-Think-Tank/events/226497046/
Hortonworks Community Celebration and Spark Meetup (San Francisco) - Wednesday, December 9
http://www.meetup.com/San-Francisco-Hortonworks-User-Group/events/226762346/
Spark, Streaming, BlinkDB, Approximate, Twitter Algebird, CountMin Sketch, HyperLogLog (San Francisco) - Thursday, December 10
http://www.meetup.com/Advanced-Apache-Spark-Meetup/events/226122226/
Case Study: Machine Learning at Scale Using Spark and Hive (Westminster) - Wednesday, December 9
http://www.meetup.com/Boulder-Denver-Big-Data/events/226843425/
Scalding: A Better Way to Write MapReduce Jobs (Addison) - Monday, December 7
http://www.meetup.com/DFW-Data-Science/events/226641213/
December 2015 Meetup: YARN (Plano) - Monday, December 7
http://www.meetup.com/DFW-Cloudera-User-Group/events/226821872/
December Edition of MOHUG (Dublin) - Tuesday, December 8
http://www.meetup.com/MOHUG-Mid-Ohio-Hadoop-User-Group/events/226564126/
December 2015 Meetup: Kafka (Atlanta) - Thursday, December 10
http://www.meetup.com/Atlanta-Cloudera-Users-Group/events/226680442/
Spark Hands-on Workshop (Laurel) - Monday, December 7
http://www.meetup.com/Apache-Spark-Maryland/events/226960737/
Continuous Data Management for Hadoop and Spark (Jersey City) - Wednesday, December 9
http://www.meetup.com/Open-Source-Analytics-New-Jersey/events/226603326/
Scaling Spark (New York) - Monday, December 7
http://www.meetup.com/Hadoop-NYC/events/227110077/
Big Data Warehousing Innovation: Introducing Kudu (New York) - Wednesday, December 9
http://www.meetup.com/Big-Data-Warehousing/events/226972822/
Database Seminar: Hadoop (Buffalo) - Thursday, December 10
http://www.meetup.com/buffalolab/events/224032630/
December Presentation Night (Boston) - Thursday, December 10
http://www.meetup.com/Boston-Apache-Spark-User-Group/events/226209947/
Apache Spark: Why Should I Care? + Spark in Production (Montreal) - Wednesday, December 9
http://www.meetup.com/Montreal-Apache-Spark-Meetup/events/227046272/
Spark Meetup (Paris) - Monday, December 7
http://www.meetup.com/Paris-Spark-Meetup/events/226967800/
Data Munging with Apache Spark (Toulouse) -
Tuesday, December 8
http://www.meetup.com/Tlse-Data-Science/events/227049634/
Saturday, December 12
http://www.meetup.com/Tlse-Data-Science/events/227049754/
Integrating Spark/Cassandra, Theory and Practice (Talence) - Thursday, December 10
http://www.meetup.com/BordeauxJUG/events/227072081/
Spark on Azure + Spark Streaming (Zaventem) - Thursday, December 10
http://www.meetup.com/Spark-Belgium/events/226416221/
Introduction to Spark Streaming and Deep Dive (Bangalore) - Saturday, December 12
http://www.meetup.com/Big-Data-Developers-in-Bangalore/events/227008581/
SOUTH KOREA
Kudu: New Apache Hadoop Storage for Fast Analytics on Fast Data (Seoul) - Tuesday, December 8
http://www.meetup.com/Korea-Big-Data-Think-Tank/events/226876563/
Spark Technical Deep Dive with Chris Fregly (Sydney) - Tuesday, December 8
http://www.meetup.com/Sydney-Apache-Spark-User-Group/events/227098916/
Spark after Dark with Chris Fregly and Jamie Engesser (Melbourne) - Wednesday, December 9
http://www.meetup.com/Melbourne-Apache-Spark-Meetup/events/226600338/