12 January 2014
This issue marks a complete year (well, 52 weeks) of Hadoop Weekly. Thanks to all the authors and writers for generating so much content—without it this newsletter couldn’t exist. And I’d also like to thank all the subscribers for spreading the word! It’s pretty awesome that over 1500 people receive this weekly email.
With that said, there’s a great issue below. SQL-on-Hadoop continues to heat up with updates on the stinger initiative, Presto-as-a-Service from Qubole, and IBM’s BigSQL. There are also some updates on writing YARN applications, and there are a handful of releases / new projects to check out.
Cloudera employee and former Oracle DBA, Gwen Shapira, has written an FAQ for Oracle DBAs thinking about learning Hadoop. The post covers four questions, including some tips for important skills if you don’t have any Hadoop experience but are looking for a Hadoop job.
http://blog.cloudera.com/blog/2014/01/the-hadoop-faq-for-oracle-dbas/
The Hortonworks blog has an update on the Stinger initiative, whose aim it is to make Hive 100x faster. Work is being wrapped up on phase 3 (out of 3) of the initiative. The most recent improvements include an improved integration with Tez, usage of the Tez Service, and a vectorized query engine. These improvements greatly increase the speed of the Hive and provide interactive querying via the Tez Service.
http://hortonworks.com/blog/3-reasons-try-stinger-phase-3-technical-preview/
HDFS needs to know about the network topology in order to place blocks across racks (to survive a rack failure). This post details how it’s implemented via a user-provided script, and it includes examples of how the script is invoked by the namenode in different scenarios.
http://ofirm.wordpress.com/2014/01/09/exploring-the-hadoop-network-topology/
Hadoop Committer Steve Loughran has a post on the (quite complex) process of reporting a bug or feature request on the Hadoop project. It covers the etiquette for reporting bugs and new features, a number of relevant tools, the lifecycle of an issue, and more.
http://www.slideshare.net/steve_l/2014-0110-reporting-a-bug
Steve Loughran also has a post this week about the pros and cons of the HOYA (HBase/Accumulo on YARN) project’s architecture as a YARN application. The post also covers its usefulness as an example/reference YARN application—in particular vs the distributed shell reference implementation. The overview is a fair assessment (including a number of pros and cons) of HOYA, and it ends with the recommendation of using the Twill Apache Incubator project if you’re starting a new YARN application from scratch.
http://steveloughran.blogspot.com/2014/01/hoya-as-architecture-for-yarn-apps.html
In a nice follow-up to the last post, LinuxInsider has an introduction to the Twill project. It covers the history of Twill (previous Weave) at Continuuity and on Github, as well as the problem that Twill is solving. In short, by making a number of assumptions about YARN applications to introduce a simpler API, Twill provides an abstraction that is much simpler than the YARN primitives.
http://www.linuxinsider.com/story/79719.html
The Hortonworks blog has a post about some upcoming features to support heterogenous storage devices. The post has a nice introduction to and comparison of the various types of storage (HDD, SSD, RAM), which motivates why you would want several types. It then talks about the architecture and API changes required, which are partially implemented in Hadoop trunk and are targeted for an upcoming 2.x release.
http://hortonworks.com/blog/heterogeneous-storages-hdfs/
IBM’s BigSQL product supports Apache HBase as a storage backend. The system has a lot of similarities with Apache Phoenix (incubating), including secondary indexes that are updated via HBase coprocessors. This presentation contains much more information about the interface.
http://www.slideshare.net/ppruski/adding-value-to-hbase-with-ibm-infosphere-biginsights-and-bigsql
The MapR blog explains their stance on the flurry of SQL-on-Hadoop projects—they are supporting several of them. In particular, they support Apache Hive (with the features from phase 2 of the stinger initiative), Impala, Apache Drill, and Presto (from Facebook).
http://www.mapr.com/blog/sql-in-hadoop-with-mapr-you-can-have-your-cake-and-eat-it-too
For folks using Cloudera software, Cloudera is putting out a new newsletter for developers. The newsletter runs monthly.
http://blog.cloudera.com/blog/2014/01/the-cloudera-developer-newsletter-its-for-you/
MapR and Savvis are partnering to offer MapR’s distribution as part of the Savvis big data solution. Savvis offers fully hosted and managed services for enterprises. Cloudera is also a Savvis partner.
http://www.mapr.com/blog/mapr-savvis-partnering-to-address-enterprise-scale-big-data-requirements
Information Age has the story of the hard work (and some luck) that went into creating Cloudera. It’s pretty wild to think that if Lehman Brothers had failed 4 days earlier, Cloudera (and thus the entire Hadoop ecosystem) as we know it might be very different today.
Intel announced an alpha release of version 2.0 of the Intel Graph Builder library. Intel Graph Builder is a library for graph processing that leverages Pig for distributed computation.
https://lists.01.org/pipermail/graphbuilder/2013-December/000022.html
Parquet, the columnar storage format for Hadoop, released version 1.3.2. The release resolves 4 issues, including a bug related to enum handling.
https://github.com/Parquet/parquet-mr/blob/471a693e39850eaa30931b765a6c99d21d656c90/CHANGES.md
Qubole has announced an alpha release of their Presto-as-a-Service offering. Presto is the low-latency big data SQL solution that Facebook open-sourced a few months ago. What makes the Qubole offering unique vs. other SQL-on-Hadoop services (including Amazon’s Redshift) is that it can directly process data stored in S3. It’s also one of the first -as-a-Service offerings in this area.
http://www.prweb.com/releases/2014/01/prweb11474296.htm
Oryx-Flume is a new project that integrates Apache Flume with Cloudera Oryx. Flume is a distributed event data transfer and collection service, and Oryx provides large-scale machine learning. The project includes support for JSON events, but the interface is extensible.
The Kiji Project has released the “Dashi” BentoBox, version 1.4.0. The bundle includes new versions of KijiExpress, the Scala DSL for analyzing data in Kiji, and KijiModeling, the modularization framework for model development. KijiExpress hits version 1.0.0, signifying a stable API. Like previous releases, this one supports CDH 4.x.
http://www.kiji.org/%3Fp%3D2473
Curated by Mortar Data (http://www.mortardata.com)
Cleveland Big Data and Hadoop User Group (Cleveland, OH)
http://www.meetup.com/Cleveland-Hadoop/events/151064312/
Get started on Hadoop: Hands-on Interactive Boot Camp (San Jose, CA)
http://www.meetup.com/BigLRN/events/159560602/
Distributed Online Algorithms for Real-time Scalable Big Data on Elastic Cloud (Mountain View, CA)
http://www.meetup.com/Frontier-Real-time-Streaming-Big-Data-Virtualization/events/143707662/
Intro to Hadoop: Hype or Reality - you decide with Kevin Crocker (San Francisco, CA)
http://www.meetup.com/Pivotal-Open-Source-Hub/events/156028172/
'The Future of Data' by Doug Cutting, Founder of Hadoop (Palo Alto, CA)
http://www.meetup.com/SF-Bay-Areas-Big-Data-Think-Tank/events/154203222/
January HUG Meetup (Pittsburgh, PA)
http://www.meetup.com/HUG-Pittsburgh/events/147625102/
Big Real-Time Learning and Applying Deep Learning (Amsterdam, Netherlands)
http://www.meetup.com/The-Amsterdam-Applied-Machine-Learning-Meetup-Group/events/155916592/
Vermont Hadoop Meetup (Burlington, VT)
http://www.meetup.com/Vermont-Hadoop-Users-Group/events/158087182/
Hadoop Based Machine Learning (Austin, TX)
http://www.meetup.com/Austin-ACM-SIGKDD/events/157569032/
Stores, Monoids and Dependency Injection - Abstractions for Spark Streaming Jobs (San Francisco, CA)
http://www.meetup.com/spark-users/events/156708582/
Tree and Graph Processing on Hadoop (Laurel, MD)
http://www.meetup.com/Data-Science-MD/events/156867282/
Top 10 considerations for running Hadoop in Production (Hyderabad, India)
http://www.meetup.com/hyderabad-hadoop/events/159805762/