19 May 2013
Both Apache Hadoop and Apache Hive crowned new releases this week, and there are a number of interesting technical articles covering YARN, NFS access to HDFS, and Apache Flume. With so much happening so quickly in the Hadoop-ecosystem, it can be a difficult to keep up -- so please let me know if I missed anything, and I'll include it next week.
Apache HDFS is getting support for the Network FileSystem (NFS) protocol. This an exciting new feature, and one of the authors working on the feature details the what, why, how, and when of Hadoop's NFS support which is being developed in trunk.
http://hortonworks.com/blog/simplifying-data-management-nfs-access-to-hdfs/
Cloudera's blog has the second in their "meet the founders" series. This post features Roman Shaposhnik who founded and works on Apache BigTop. Aside from having one of the best names of projects in the Hadoop ecosystem, BigTop is beginning to have a lot of influence in making sure that components in the stack are compatible with one another when released.
http://blog.cloudera.com/blog/2013/05/meet-the-project-founder-roman-shaposhnik/
If you've ever tried to put together a patch for Hadoop, it can be very intimidating (and a slow process) just to configure your development environment. This post provides an overview of setting up Eclipse for developing Hadoop -- covering all the major versions and flavors under development.
http://blog.cloudera.com/blog/2013/05/how-to-configure-eclipse-for-hadoop-contributions/
You might remember the "stinger initiative" which was introduced a while back by Hortonworks with the goal of making Hive 100x faster. With the release of Hive 0.11 (more below), they summarize some of the work that's already been done towards this goal, as well as some of the new features in Hive 0.11 (such as RANK and other analytical functions).
http://hortonworks.com/blog/apache-hive-0-11-stinger-phase-1-delivered/
The Manning Early Access Program (MEAP) is now available for the new book, "Pig in Action", by M. Tim Jones. With MEAP, you pre-order the book but get access to the content as the author is writing and uploading it.
http://www.manning.com/tjones/
Apache Flume is a system for transferring data from application servers or other event-generators to HDFS or HBase. In this post, the author gives an overview of the Flume architecture -- both at the component and system scale.
http://www.drdobbs.com/database/acquiring-big-data-using-apache-flume/240155029
The Natural Language Toolkit (NLTK) is a set of python libraries for natural language processing. This post describes how to tie them to Hadoop MapReduce for parallel processing using Hadoop Streaming.
http://datacommunitydc.org/blog/2013/05/nltk-hadoop/
Arun C. Murthy, one of the leads on the Apache Hadoop YARN project, gives an update on the progress of the project plus background on what YARN can enable. In particular, YARN turns Hadoop into a multi-application system, allowing more than just MapReduce to run on Hadoop. Arun highlights that we'll be able to run SQL in Hadoop rather than SQL on Hadoop (via MapReduce).
http://hortonworks.com/blog/moving-hadoop-beyond-batch-with-apache-yarn/
Hortonworks has compiled a list of links for the Hadoop on Windows developer. In particular, the .NET SDK, the Microsoft Hive ODBC driver, and HDInsight's Preview (Hadoop on Azure).
http://hortonworks.com/blog/hadoop-sdk-and-tutorials-for-microsoft-net-developers/
Platfora's product provides a combines a UI and low-latency data store to do interactive analysis on data stored in S3 or HDFS. If the data isn't already in Platfora's store, the system can generate a MapReduce job to load the data. This article gives a good overview of how all of the technology components in the Platfora system work together.
http://cloudcomputing.sys-con.com/node/2663726
HUE provides a UI for interacting with Hadoop, Hive, Pig, and more. This post describes how to leverage HUE's python API to execute queries against Hive (via HiveServer2) or Impala (which must implement the same Thrift API).
http://gethue.tumblr.com/post/49882746559/executing-hive-or-impala-queries-with-python
Storm is sometimes called the real-time version of MapReduce. With a lot of interest in getting Storm running on YARN, now's a good time to get familiar with the system. The inaugural London Storm Meetup featured an overview of Storm as well as a discussion of the presenter's use-case. This post has a summary of the event, including links out to the presentation and code examples.
http://partners.peerindex.com/london-storm-meetup-big-data/
Contexti and MapR have joined forces to provide training, consulting, and professional services for MapR's distribution in Asia-Pacific.
http://www.businesswire.com/news/home/20130512005046/en/Contexti-Expands-Hadoop-NoSQL-Portfolio-MapR
Drawn to Scale, a SQL-on-Hadoop vendor, has announced that they're closing their doors. They had an interesting system, which is built to be performant on many types of SQL operations, and they even had a compatibility layer for MongoDB. It should be interesting to see what happens to that team and their technology.
http://www.roadtofailure.com/?p=11
Concurrent and MapR announced that Concurrent's Cascading framework is now certified to run on MapR's distribution.
Hadoop 1.2.0 featuring DistCP v2 backport, web services for the JobTracker, the offline image viewer, and a bunch of other enhancements.
WibiData announced Albacore/BentoBox v1.0.4. This version has some new features, including a whole new component -- KijiREST, which provides a REST interface to KijiSchema.
Hive 0.11 was released with over 350 Jira issues closed. This is the first release since HCatalog was integrated as a subproject of Hive, and it has a bunch of new features such as HiveServer2, ORCFile, and analytics functions.
Talend's Open Studio was updated to version 5.3 a few weeks ago. This post has a quick overview of the new features, which include a new integration with Apache Pig, as well as support for Amazon's Elastic MapReduce and RedShift.
http://www.h-online.com/open/news/item/Talend-5-3-focused-on-Hadoop-usability-1864844.html
Curated by Mortar Data ( http://www.mortardata.com )
Monday, May 20 MySQL to Cassandra: Big Data, High Scale, Data Migration... Oh My! (New York, NY) http://www.meetup.com/mysqlnyc/events/114879742/
Monday, May 20 Automating the Hadoop Stack (Los Angeles, CA) http://www.meetup.com/LA-HUG/events/117428702/
Tuesday, May 21 Data & Drinks - Member Networking Meetup (New York, NY) http://www.meetup.com/Analytics-and-Data-in-Financial-Services/events/112520472/
Tuesday, May 21 Recommendation Engines & Accumulo (Denver, CO) http://www.meetup.com/Data-Science-Business-Analytics/events/116790372/
Tuesday, May 21 Thoughts on machine learning (New York, NY) http://www.meetup.com/NYC-Machine-Learning/events/119204802/
Tuesday, May 21 How we use Scala on Hadoop @ eBay (New York, NY) http://www.meetup.com/ny-scala/events/113168872/
Wednesday, May 22 Cloudera Impala: An Open Source Real-Time Query Engine for Apache Hadoop (Boulder, CO) http://www.meetup.com/Boulder-Denver-Big-Data/events/114501572/
Wednesday, May 22 Big Data, NoSQL, Now What? (New York, NY) http://www.meetup.com/mysqlnyc/events/114883642/
Saturday, May 25 Big Data Science Meetup Event (Fremont, CA) http://www.meetup.com/Big-Data-Science/events/71084702/