15 September 2013
There's a ton of great content in this week's issue riding on the wave of the recent Hadoop 2.1.0-beta. In addition, this issue contains a number of walkthroughs and tutorials -- there should be plenty to keep you buys if you're just getting started or looking to try out new things.
Gwen Shapira from Cloudera presented at Surgecon on ETL with Hadoop. The talk starts with the motivation for moving ETL to Hadoop (things like cost and flexibility), then moves onto best practices and tools for ETL on Hadoop. It covers low-level details like file system layout as well as the available tools (e.g. Sqoop and Oozie) and how to scale them.
http://www.slideshare.net/gwenshap/scaling-etl-with-hadoop-shapira-3
Parquet is a relatively new columnar file format for large datasets. The format, which is partially based-upon the format discussed in Google's Dremel paper, uses some interesting tricks to store nulls and repeated fields. A recent post walks through the low-level details of how the format serializes data efficiently. Complete with examples and illustrations, the post makes a complicated topic easily accessible.
https://blog.twitter.com/2013/dremel-made-simple-with-parquet
This post covers writing and testing simple (supporting only java primitive) UDFs and generic UDFs for Apache Hive. Generic UDFs, which are required if you want to support arguments that are complex types (e.g. List, Map), have a complicated API that takes some time to become familiar with. That also makes the API somewhat complex to test, but the post covers each of these in detail with accompanying code available on github.
http://blog.matthewrathbone.com/2013/08/10/guide-to-writing-hive-udfs.html
The Hortonworks blog has some more details on Apache Tez, the framework which aims to generalize the MapReduce paradigm. The post covers the motivation (i.e. better performance and flexibility by offering more primitives vs. MapReduce) as well as some details on the overall design. In particular, you'll learn about the logical plan modeling (akin to a database query plan), the pluggability of input/output formats (including intermediate outputs), dynamic reconfiguration (changing the query plan at runtime), and the plans for resource management.
http://hortonworks.com/blog/apache-tez-a-new-chapter-in-hadoop-data-processing/
As a follow-up to last week's release of Spring for Hadoop 2.0 M1, the Spring blog has an intro to the new YARN features in the release. The post introduces the YarnClient, YarnAppmaster, and YarnContainer interfaces and their roles in a YARN application lifecycle. There's some simple example code, a few configuration examples, and instructions for submitting code via the command-line.
Revolution R Enterprise 7 will support Hadoop as a backend for RevoScaleR, and this post gives an overview of what that interaction is going to look like. The quick summary is that there's not a whole lot to change, other than configuring connections to HDFS and the MapReduce cluster. Once that's done, the summary and logistic regression functions just work -- the latter spawning a number of MapReduce jobs.
http://blog.revolutionanalytics.com/2013/09/alpha-testing-revoscaler-running-in-hadoop.html
Lipstick, the Pig workflow visualization software, was recently integrated into the Mortar product. As part of this work, the Mortar team wrote chef recipes to configure a Lipstick service (and all dependencies, such as MySQL). Combined with Vagrant, these tools allow for a single-command startup of a VM running Lipstick. A post on the Mortar blog covers this system as well as some of the changes/features that Mortar worked on in order to bring Lipstick to their platform.
http://blog.mortardata.com/post/61026715305/using-chef-and-vagrant-to-put-lipstick-on-a-pig
The Cloudera blog has an overview of testing HBase client code. The post covers vanilla unit testing, mock testing with Mockito, integration testing with HBaseTestingUtility, and testing of HBase MapReduce code with MRUnit. There's a lot of good stuff even for someone who's been using HBase for a while, and all of the code is available on github.
http://blog.cloudera.com/blog/2013/09/how-to-test-hbase-applications-using-popular-tools/
Given the major framework and architecture changes in Hadoop 2.x due to YARN, there are a whole new set of configuration options to consider. The post considers an example hardware setup and walks through all of the YARN and MapReduce 2 config options. Rather than configuring slots-per-node, all configuration is done via slices of memory, and there are some subtle points such as configuring the java heap size to not outgrow the container memory allocation.
http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/
Before executing a Pig script, the Pig runtime converts the script to a LogicalPlan that describes the data flow. This data isn't readily exposed via an API (Pig 0.12 will include a fix), but it can be accessed with a bit of work. This posts includes an example, written in JRuby, of subclassing the PigServer class to make the LogicalPlan available programmatically.
http://thedatachef.blogspot.com/2013/09/get-pig-logicalplan.html
SAP made a number of announcements at TechCrunch Disrupt, including an expanded partnership with Intel and Hortonworks to resell and support the Intel and Hortonworks Distributions. In addition, they announced a new product aimed at manufacturers, the creation of a new Data Science organization at SAP, and the "Big Data Geek Challenge."
http://www.news-sap.com/sap-helps-customers-achieve-real-time-big-data-results/
Hadoop Summit Europe 2014 is being held in April in Amsterdam, and the call for abstracts is open through October 31st. There are two new tracks for this years conference -- a committer track and a new "unconference" track. The other tracks are "The Future of Apache Hadoop," "Data Science & Hadoop," "Hadoop Deployment & Operations," and "Hadoop for Business Applications and Development."
http://hortonworks.com/blog/hadoop-summit-europe-2014-call-for-abstracts-now-open/
Cloudera manager 4.7.1 was released to hot fix a bug preventing HUE configurations from being opened.
https://groups.google.com/a/cloudera.org/d/msg/scm-users/7a0OVnYPcG8/-xnQ5W4xoO4J
Curated by Mortar Data ( http://www.mortardata.com )
Monday, September 16
Hadoop Development Kits : Weave and Cloudera CDK (San Jose, CA)
http://www.meetup.com/BigDataGurus/events/138500182/
Tuesday, September 17
Real-time stream processing platform on Hadoop (Santa Clara, CA)
http://www.meetup.com/Silicon-Valley-NoSQL/events/138196142/
Twitter, Storm, Clojure... (London, UK)
http://www.meetup.com/Data-Science-London/events/139205302/
HBase - A Technical Introduction (Toronto)
http://www.meetup.com/TorontoHUG/events/137179092/
Wednesday, September 18
Hadoop Based Machine Learning (Austin, TX)
http://www.meetup.com/Austin-ACM-SIGKDD/events/136699592/
Data infrastructure at Spotify (Berlin)
http://www.meetup.com/Big-Data-Beers/events/138717652/
Thursday, September 19
Hadoop Ops (Sunnyvale, CA)
http://www.meetup.com/SF-Bay-Area-Large-Scale-Production-Engineering/events/129859402/