12 May 2013
This week's newsletter is a little lighter than normal in technical news (some fascinating articles, though!), but there are a quite a few interesting releases and upcoming events. Hope you enjoy, and please let me know if you find anything that I missed! Also, thanks to everyone that has been spreading the word about this newsletter -- the number of new subscribers each week has been really encouraging.
LinkedIn has open-sourced a number of big data projects built on or to coexist with Hadoop. In celebration of LinkedIn's 10th anniversary, this post covers 10 of those projects (such as Voldemort and DataFu), including a brief overview of each.
http://www.hadoopsphere.com/2013/05/hadoops-10-in-linkedins-10.html
Following the release of version 0.2.0 of the Cloudera Development Kit, the Cloudera blog has a new post with an overview of the project, an FAQ, and a list of future plans. They plan on having monthly releases and focusing on documentation in addition to software libraries. It should be an interesting project to watch.
http://blog.cloudera.com/blog/2013/05/cloudera-development-kit-cdk/
An overview of eBay's data warehouse, which ingests as much as 100TB/day and stores over 90PB. To power internal analytics, they use a combination of Hadoop, Teradata, and a custom built system as data stores plus front-end tools Tableau, Excel, Microstrategy (and more).
http://www.itnews.com.au/News/342615,inside-ebay8217s-90pb-data-warehouse.aspx
RCFile is a columnar format that's part of the Hive project. This post describes the motivation for RCFile as well as the benefits. In a follow-up post, the author will talk about the successor to RCFile -- ORCFile, which has similar features to the Parquet format.
http://www.bigdatarepublic.com/author.asp?section_id=2840&doc_id=262756
An honest review of a new Hadoop book, the "Hadoop Beginner's Guide" talks about both the good and the bad in the book. Overall, the review is positive but notes that there are a few technical issues that could be improved.
http://architects.dzone.com/articles/review-hadoop-beginners-guide
Russell Journey, the author of Agile Data, has posted slides to accompany his book. The slides cover a number of principles for Agile big data as well as a bunch of example code covering everything from data analysis with Pig to visualization with Bootstrap and D3.
http://www.slideshare.net/rjurney/agile-analytics-applications-on-hadoop-20839095
When an HBase RegionServer fails, it can take a few seconds or minutes for the regions owned by that RegionServer to recover. Reducing this time, known as the Mean Time to Recover (MTTR), has been the subject of a lot of work on both the HBase and HDFS projects. This post has a good overview of the technical challenges and their solutions.
http://hortonworks.com/blog/introduction-to-hbase-mean-time-to-recover-mttr/
Snakebite is a new python project that makes use of protocol buffers to talk to HDFS without going through the JVM. In addition to an API, it supplies a command line utility with similar functionality to hadoop fs (without the JVM startup overhead, it's a lot faster) and a script to startup a mini HDFS cluster, which it uses for testing.
http://labs.spotify.com/2013/05/07/snakebite/
Azkaban 2.1 was released. This is the first point release to the Hadoop workflow management software since the version 2 rewrite. It has a bunch of new features, like JMX support, auto-retries, and SLA aware notifications.
https://groups.google.com/d/msg/azkaban-dev/WN5LWbqtsxE/PhUmmcO2lZIJ
DataStax has made their ODBC driver for Hadoop/Hive free and has announced a new ODBC driver for Cassandra, currently in beta.
http://www.datastax.com/dev/blog/free-odbc-drivers-for-cassandra-and-hadoop-now-available
Apache Giraph, the computation framework for bulk synchronous parallel programming (often used for network graph algorithms), had its version 1.0 release, the first since graduating from the Apache Incubator. This release has a bunch of features, including support for running within YARN, support for accessing Hive tables, and improved performance and memory efficiency.
http://mail-archives.apache.org/mod_mbox/giraph-user/201305.mbox/%3C51888B3D.5040100@apache.org%3E
Apache Curator (incubating), is a set of Java libraries for Apache Zookeeper. The project was originally started at, and open sourced by, Netflix. This week they had their 2.0.0-incubating release, the first since joining the Apache incubator.
GigaOm notes that IBM is also an entrant into the SQL-on-Hadoop field, with a preview open to a limited number of participants.
http://gigaom.com/2013/05/06/look-ibm-is-doing-sql-on-hadoop-too/
Curated by Mortar Data ( http://www.mortardata.com )
Monday, May 13 Hackathon: Develop/MapReduce Your Dream Predictive Analytics/BigData App (Cambridge, MA) http://www.meetup.com/Predictive-Analytics/events/109954442/
Tuesday, May 14 Storm, Brickhouse, ElasticSearch (San Francisco, CA) http://www.meetup.com/Real-time-Big-Data/events/113867452/
Tuesday, May 14 Big Data is Not About The Data! (Chestnut Hill, MA) http://www.meetup.com/intelligence/events/110894352/
Tuesday, May 14 Smarter Big Data Integration for Hadoop (London, UK) http://www.meetup.com/hadoop-users-group-uk/events/115450432/
Tuesday, May 14 Introduction to Distributed Search using Cassandra with Solr and Lightning Talks (Columbia, MD) http://www.meetup.com/Hadoop-DC/events/117691352/
Wednesday, May 15 R and Hadoop (Free Webinar) (Internet) http://www.meetup.com/OC-HUG/events/113336802/
Wednesday, May 15 37th Bay Area Hadoop User Group (HUG) Monthly Meetup (Sunnyvale, CA) http://www.meetup.com/hadoop/events/69997352/
Thursday, May 16 Hackathon for Hadoop Administrators (Hyderabad, India) http://www.meetup.com/hyderabad-hadoop/events/118712352/
Thursday, May 16 STORM for streaming analytics at scale (London, UK) http://www.meetup.com/storm-london/events/118169452/
If you didn't receive this email directly, and you'd like to subscribe to weekly emails please visit http://hadoopweekly.com