24 August 2014
This week’s edition has a lot of great technical content from prominent Hadoop vendors Hortonworks and Cloudera as well as newcomer SequenceIQ. There are also a couple of interesting articles based on real-world experience covering an A/B testing platform and Apache Zookeeper. Those types of articles tend to be quite good but more difficult to find—as always, if you have suggestions for the newsletter please send them my way!
Hortonworks has posted a video series on the most recent release of their distribution, HDP 2.1. The videos, which are recordings of several webinars, cover a large number components including YARN, HDFS, Hive, and Ambari.
http://hortonworks.com/blog/discover-hdp-2-1-series-webinar-wrap/
A guest post on the Hortonworks blog describes how SAS is working to bring their High-Performance Analytics (HPA) and LASR Analytics Server to YARN. The systems were originally built to run on as MPI applications in which SSH was used to launch processes. With YARN, HPA uses the framework for process management, and there are improvements like enforcing CPU and memory limitations.
http://hortonworks.com/blog/apache-hadoop-helps-sas/
The Hortonworks blog has a post on an in-progress feature called container delegation. Before diving into container delegation, the post gives an intro to YARN’s resource and workload management. The new feature will be used, for among other things, to provide additional per-query resources to a long-running application.
The SequenceIQ blog has a post on the YARN FairScheduler. The post has an introduction to the FairScheduler, the scheduling challenges, and some of its configuration options. Using an example test and an R-based analysis tool (which is open-sourced), the post finds that the FairScheduler is good at maintaining fairness.
http://blog.sequenceiq.com/blog/2014/08/16/fairplay/
The Hortonworks blog has had a number of security related posts in the past week. This post summarizes the coverage, which includes posts on Apache Argus and Apache Knox. It also discusses posts from some partner vendors—Protegrity, Voltage Secruity, and Dataguise. Finally, it touches on some new Hadoop features—Transparent Data Encryption for HDFS and a Key Provider API and accompanying Key Management Server.
http://hortonworks.com/blog/continued-innovation-hadoop-security/
Apache Spark ships with the spark-submit script for submitting a job to a Spark cluster. Sometimes, it’s useful or necessary to programmatically submit a job. This post describes how to write a Scala program to do so, and how to invoke the resulting binary jar.
http://blog.sequenceiq.com/blog/2014/08/22/spark-submit-in-java/
This post serves as a a good introduction to partitioning of Hive tables. It outlines the motivation and benefits of partitioning and includes several tips and best practices.
http://blog.cloudera.com/blog/2014/08/improving-query-performance-using-partitioning-in-apache-hive/
The Cloudera blog has a post with several tips and examples for writing powerful Hive queries. It includes example queries with the LAG and LEAD analytics function as well as using LATERAL VIEW and a UDTF to execute nested SQL queries. It also suggests some ways of organizing data, including the notion of a “supernova schema” which is somewhat akin to a materialized star-schema as a single table.
http://blog.cloudera.com/blog/2014/08/how-to-count-events-like-a-data-scientist/
DZone has published a cheat sheet for Apache Hadoop. It includes things like HDFS architecture, HDFS command line examples, an overview of YARN, and an introduction to MapReduce. It also covers Pig and Hive as well as providing links to several ecosystem projects.
http://refcardz.dzone.com/refcardz/getting-started-apache-hadoop
Camille Fournier, Zookeeper PMC and Rent the Runway CTO, spoke on using Zookeeper in the wild. Her talk covers a number of systems that use Zookeeper as well as a number that do not. One of her conclusions is that, while Zookeeper has a number of use-cases, it’s not always the best tool for the job.
http://www.infoq.com/presentations/zookeeper-use-case
The Pinterest engineering blog has a post on their A/B analytics platform. The post covers the implementation, which uses Kafka, Storm, MapReduce, HBase, and more. There’s an overview of the MapReduce workflow, the serving of metrics via HBase, and real-time processing via Storm. There’s also a discussion of statistical significance and group validation via chi-square.
http://engineering.pinterest.com/post/95378137929/scalable-a-b-experiments-at-pinterest
A new book on Apache Flume is in early release and available as an eBook from O’Reilly. The book is aimed at developers deploying and customizing Flume.
http://blog.cloudera.com/blog/2014/08/the-new-apache-flume-book-is-in-early-release/
Allied Market Research recently released a report on the Hadoop-as-a-Service (HaaS) market. It expects that market to growth rapidly to $16.1B by 2020. The report notes that HaaS doubled from 2012 to 2013, and it expects that HaaS will become more and more competitive with on-premises deployments.
http://channelnomics.com/2014/08/20/hadoop-service-reach-16-1-billion/
TPCx-HS is a new benchmark specification aimed at measuring the Hadoop Runtime, Hadoop Filesystem API implementations, and MapReduce layers. It is claimed to be the first “Industry Standard Big Data Benchmark,” and there are already plans for additional. The ODBMS blog has an interview with Francois Raab, the author of the TPC-C Benchmark, and Yanpei Chen of the Performance Engineering Team at Cloudera. In the interview, they discuss some plans for big data benchmarks in more detail.
http://www.odbms.org/2014/08/tpc-launches-tpcx-hs-vendor-neutral-industry-standard-big-data-benchmark/ http://www.odbms.org/blog/2014/08/big-data-benchmarks-interview-francois-raab-yanpei-chen/
Using Apache BigTop, CDH5 has been tested in conjunction with GlusterFS 3.3 (specifically its glusterfs-hadoop FileSystem). There are some more details on the implementation in a guest post on the Cloudera blog.
http://blog.cloudera.com/blog/2014/08/running-cdh-5-on-glusterfs-3-3/
The MapR blog has a transcript and video of a recent presentation by their CEO John Schroeder where he spends 5 minutes talking about several applications of Hadoop. He talks about the Aadhaar project’s biometric database, health care, advertising, music personalization, and MinuteSort.
https://www.mapr.com/blog/hadoop-5-minutes-or-less
Version 0.16.0 of the Kite SDK was released. This release adds support for Apache Spark, adds a new command-line ETL tool, fixes generation of Parquet Hive tables on Hive 0.13+, and adds a new parent pom for Kite SDK apps written for CDH5.
http://community.cloudera.com/t5/Release-Announcements/Announcing-Kite-SDK-0-16-0/m-p/17396
The folks at SequenceIQ have released a new docker image for Apache Hadoop 2.5.0. Like previous versions, their are psuedo-distributed and fully distributed variants of the image. The image uses Apache Ambari to provision a cluster.
http://blog.sequenceiq.com/blog/2014/08/18/hadoop-2-5-0-docker/
Microsoft made some announcements about their Azure cloud services this week. Among them, they announced the general availability of Apache HBase for HDInsight. The service had been in preview since June.
http://gigaom.com/2014/08/21/microsoft-unveils-documentdb-a-nosql-database-built-for-azure/
Spindle is a new analytics platform recently open-sourced by Adobe Research. It combines Apache Spark for processing, Apache Parquet for a data storage format, and a Spray-based HTTP server.
Mortar, the Hadoop/Pig as a Service system, has announced support for running jobs in local mode to improve development iteration.
http://blog.mortardata.com/post/95203524746/announcing-instant-mortar
Curated by Mortar Data ( http://www.mortardata.com )
eHarmony's Hadoop Program (Irvine) - Thursday, August 28
http://www.meetup.com/OC-HUG/
Cybersecurity & Big Data Analytics with Hadoop (Mountain View) - Thursday, August 28
http://www.meetup.com/BigDataGurus/events/202176882/
HBase Meetup @ Sift Science (San Francisco) - Thursday, August 28
http://www.meetup.com/hbaseusergroup/events/197773762/
MongoDB and Hadoop: Driving Business Insights (Austin) - Monday, August 25
http://www.meetup.com/Austin-MongoDB-User-Group/events/199019262/
Enabling Advanced Analytics & From Sandbox to Production PA (Kansas City) - Monday, August 25
http://www.meetup.com/Data-Science-KC/events/196317212/
Batch Data Processing at Spotify with Luigi (Madison) - Tuesday, August 26
http://www.meetup.com/BigDataMadison/events/194365712/
Data Governance in Big Data - Cloudera/Gazzang (Dublin) - Tuesday, August 26
http://www.meetup.com/Columbus-HUG/events/200488822/
Tresata on Omnichannel Marketing Analytics in Hadoop (Charlotte) - Wednesday, August 27
http://www.meetup.com/CharlotteHUG/events/167351712/
RTP - Big Data Developer Day (Durham) - Thursday, August 28
http://www.meetup.com/Big-Data-Developers-in-Raleigh/events/197731892/
Apache Spark Lessons Learned (McLean) - Tuesday, August 26
http://www.meetup.com/Washington-DC-Area-Spark-Interactive/events/200472002/
Storm: Real-Time Big Data Stream Processing at WebMD (Hamilton Township) - Tuesday, August 26
http://www.meetup.com/nj-hadoop/events/201348342/
Hadooping @ Prague (Prague) - Monday, August 25
http://www.meetup.com/CS-HUG/events/177483842/
Database as a Service (CouchDB, MongoDB, Cassandra, DB2, Hadoop) in the Cloud (Zurich) - Tuesday, August 26
http://www.meetup.com/Big-Data-Developers-in-Zurich/events/199553642/
PaaS and Big Data Tools (Melbourne) - Wednesday, August 27
http://www.meetup.com/Big-Data-in-Practice/events/194627392/
HDInsight: MapReduce and Beyond (Melbourne) - Thursday, August 28
http://www.meetup.com/MelbourneAzure/events/202007582/
3rd Spark London Meetup (London) - Thursday, August 28
http://www.meetup.com/Spark-London/events/200361502/
Apache Spark: In Memory Map-Reduce (Hyderabad) - Saturday, August 30
http://www.meetup.com/Hyderabad-Programming-Geeks-Group/events/197496752/
Spark Meetup (Hangzhou) - Sunday, August 31
http://www.meetup.com/Hangzhou-Apache-Spark-Meetup/events/198707512/