23 November 2014
Lots of news out of Europe this week as both StrataConf Barcelona and ApacheCon EU took place. In addition to several interesting presentations from those conferences, this week’s issue contains several articles on Spark and YARN. Also, Stripe has open-sourced four Hadoop tools, and there were releases from open-source projects Apache Pig and Apache Chukwa as well as vendor releases by Splice Machine and HP’s Vertica. One quick editorial note—with the short week in the US for Thanksgiving, I’m going to skip next week’s issue and resume with issue #98 on December 7th.
This tutorial describes the steps needed to run a HBase cluster with Microsoft Azure’s HDInsight, and how to build a simple application to interact with it. It details how to setup a new maven application, configure hbase-site.xml, and building an executable jar for running a test against HBase.
This post looks at a number of key configuration parameters relating to YARN memory usage (which is common area to tweak). It covers configuration for map reduce, the yarn scheduler, and yarn application managers.
This post is a good overview of Spark and how it compares to MapReduce. It describes RDD, describes the computation model (and how it compares to MapReduce), and compares Spark to Cascading/Scalding. The post also explores Spark’s reputation as being in-memory focussed and what it means in terms of performance.
http://rahulkavale.github.io/blog/2014/11/16/scrap-your-map-reduce/
Hadoop’s default deployment doesn’t include any sort of authentication - the system trusts that the client is who it says it is. To enforce authentication, Hadoop has support for Kerberos. This gives a quick intro to the key concepts of Kerberos and how they integrate with Hadoop.
http://www.securityweek.com/big-data-smaller-problems-configuring-kerberos-authentication-hadoop
This presentation describes HBase’s upcoming support for richer encodings and data types. It looks at the new OrderedBytes API, the DataType API (for structs, unions, etc), and what’s upcoming. There are also examples of using the new APIs.
http://www.slideshare.net/xefyr/hbase-data-types
Twitter has posted about their system for building an index of the entire corpus of tweets. The system relies heavily on Hadoop, both for storage and for aggregating and preparing (via Pig jobs).
https://blog.twitter.com/2014/building-a-complete-tweet-index
Apache Samza is a stream processing framework built on YARN. Samza was originally open-sourced by LinkedIn, and they have written about operating Samza (and Apache Kafka) at scale. The post describes their deployment system as well as their metric collection and alerting setup.
http://engineering.linkedin.com/samza/operating-apache-samza-scale
This presentation explores the state of Hadoop and RDF. Two projects providing the must complete support are Apache Jena project, which has experimental modules for RDF in Hadoop and the Intel Graph Builder, which supports generic graph data via Pig UDFs.
http://www.slideshare.net/RobVesse/quadrupling-your-elephants-rdf-and-the-hadoop-ecosystem
SAMOA: Scalable Advanced Massive Online Analysis is a new system aiming to be the “Mahout of Streaming.” Open-sourced by Yahoo, the project includes support for several backends including Apache Storm, Apache S4, and Apache Samza. The system implements a number of machine learning algorithms, including Distributed Stream Clustering and Vertical Hoeffding Tree Classifier.
https://speakerdeck.com/gdfm/samoa-at-strata-barcelona-2014
A new (alpha) feature of the upcoming Hadoop 2.6 release will be the ability to run Docker containers as part of YARN applications. SequenceIQ has an overview of this feature, and how it can also be used when YARN is already running inside of Docker.
http://blog.sequenceiq.com/blog/2014/11/20/yarn-containers-and-docker/
The Altiscale blog has an article about the ubiquity of the Hive metastore. It notes how several new SQL-on-Hadoop systems support it (Impala, Presto, Spark SQL, Drill) and that they do this in order to be compatible with Hive. Both of these turn out to be good things for users—if your data is in HDFS and the Hive metastore, it’s easy to use in many other systems.
https://www.altiscale.com/hadoop-blog/rise-hive-metastore-semantic-repository/
The SequenceIQ blog has a post on building a hybrid Hadoop deploy in the cloud by using both a permanent cluster and adding ephemeral clusters as needed. They show how to build clusters with their Cloudbreak (cloud-agnostic provisioning built on docker and Ambari) tool and to use their other tool Periscope to autoscale the ephemeral clusters.
http://blog.sequenceiq.com/blog/2014/11/17/datalake-cloudbreak-2/
The Call For Abstracts for Hadoop Summit Europe 2015 closes on December 5th. The conference takes place in April in Brussels, Belgium.
http://2015.hadoopsummit.org/brussels/call-for-abstracts/
MapR, whose distribution is available as an option for Amazon Elastic MapReduce (EMR), has a post with some news on the integration. First, MapR is now supported via EMR on the latest instance families. Second, hourly pricing of the EMR+MapR will be lower for both the M5 and M7 enterprise versions (M3 continues to be free).
https://www.mapr.com/blog/amazon-web-services-just-made-running-mapr-even-easier
MapR and Teradata announced an expanded partnership in which Teradata will provide MapR’s distribution as part of the Teradata Unified Data Architecture.
https://www.mapr.com/blog/mapr-and-teradata-why-architecture-matters-customer-success
GigaOm has an article on eHarmony’s infrastructure ambitions around Hadoop and OpenStack. As part of the transition, they’re moving from several instances of a Hadoop appliance to a single YARN cluster as well as bringing up new technologies like Spark and Storm.
https://gigaom.com/2014/11/19/why-eharmony-is-rebuilding-itself-atop-openstack-and-hadoop/
Datanami has an article on the rise of Spark. They note that for the first time, Apache Spark has bypassed Apache Hadoop on Google Trends. The post has a look back at the highlights of Spark this year including the growing number of contributors and the recent sort workload results.
http://www.datanami.com/2014/11/21/spark-just-passed-hadoop-popularity-web-heres/
Qubole and Microsoft Azure have announced a new strategic relationship in which Qubole’s Big Data-as-a-Service platform is available on Azure.
http://www.qubole.com/qubole-partner-azure/
Splice Machine has announced general availability of their RDBMS built on Hadoop. Version 1.0 includes supports for SQL:2003 including analytics functions, native backup and recovery, and integrates with HCatalog. Splice Machine is positioned as a cheaper alternative than scaling out a traditional RDBMS.
http://www.zdnet.com/splice-machines-sql-on-hadoop-database-goes-on-general-release-7000035958/
Apache Pig 0.14.0 was released this week. Major highlights of the release include support for a Apache Tez as a backend, OrcStorage, and loader predicate push down. The release supports Hadoop 0.23.x, 1.x, and 2.x.
http://pig.apache.org/releases.html#20+November%2C+2014%3A+release+0.14.0+available
Stripe has open-sourced four new projects for Hadoop. The projects are: Timberlake, a dashboard for YARN and MRv2, Brushfire, a system for distributing learning of ensemble tree models inspired by Google’s PLANET, Sequins, a static database backed by SequenceFiles, and Herringbone, a tool for working with Parquet files and Impala/Hive.
https://stripe.com/blog/four-new-hadoop-projects
HP has announced a new version of their Vertica columnar MPP analytics engine that runs on Hadoop. It supports many major distributions, including MapR, Cloudera, Hortonworks, and Apache Hadoop.
http://www.datanami.com/2014/11/18/vertica-gets-hadoop-upgrade/
Apache Chukwa version 0.6.0 was released. Chukwa is a system for distributed monitoring and analysis from data in log files. The new release adds support for HBase, deprecates the Chukwa collector, and resolves a number of bugs.
http://chukwa.apache.org/docs/r0.6.0/releasenotes.html
Curated by Mortar Data ( http://www.mortardata.com )
Stream Processing on Hadoop (Saint Paul) - Monday, November 24
http://www.meetup.com/Twin-Cities-Hadoop-User-Group/events/218083032/
All Models Are Wrong, Some Are Useful (Plano) - Monday, November 24
http://www.meetup.com/Dallas-Big-Data-Science/events/215952012/
Options for Streaming Analytics on Azure and Azure Batch (London) - Tuesday, November 25
http://www.meetup.com/UKAzureUserGroup/events/216580082/
Hadoop Meetup on Cascading/Tez with Concurrent and Hortonworks (Paris) - Tuesday, November 25
http://www.meetup.com/Hadoop-User-Group-France/events/218753457/
November Meetup (Mannheim) - Monday, November 24
http://www.meetup.com/Big-Data-Mannheim-Rhein-Neckar/events/204416842/
Apache Spark (Hamburg) - Wednesday, November 26
http://www.meetup.com/Scala-Hamburg/events/216734832/
What's New with Apache Spark? An Evening with Paco Nathan (Amsterdam) - Monday, November 24
http://www.meetup.com/Amsterdam-Spark/events/207220772/
Offline and Real-time Click Stream Processing (Stockholm) - Wednesday, November 26
http://www.meetup.com/stockholm-hug/events/217569002/
Apache Spark: Easier and Faster Big Data (Sydney) - Thursday, November 27
http://www.meetup.com/Sydney-Apache-Spark-User-Group/events/218140722/
Spark Singapore First Meetup! (Singapore) - Wednesday, November 26
http://www.meetup.com/Spark-Singapore/events/218794905/