03 August 2014
Apache Ambari, which is cluster management software for Apache Hadoop, is a big topic this week. In addition to news that Pivotal and Hortonworks are teaming up to collaborate on Ambari, there are a string of technical articles on it. Another hot topic this week is Apache Spark—specifically its machine learning library, MLlib. Finally, it’s also worth noting that this week’s issue includes links to a few papers from the Proceedings of the VLDB Endowment.
The Databricks blog has a post showing how to write concise Spark (using the python library, pyspark) code to solve a complicated problem. Specifically, it show how to the Alternative Least Squares implementation of Spark’s machine learning library, MLlib, to build recommendations. The post also has some details on scaling the process across a 16-node cluster and how it compares to Apache Mahout’s implementation.
http://databricks.com/blog/2014/07/23/scalable-collaborative-filtering-with-spark-mllib.html
SequenceIQ’s blog also has a post on Spark’s MLlib, this time focussing on clustering with the K-Means algorithm. The post has example code to use the k-means api and for running a job (and more source code on github). The post also talks about how Apache Tez shows a lot of promise for iterative/machine learning algorithms such as k-means.
http://blog.sequenceiq.com/blog/2014/07/31/spark-mllib/
The first of several posts on Apache Ambari in this week’s newsletter covers Ambari’s tools for expanding a cluster. Specifically, it covers adding hosts to a cluster, adding components (e.g. DataNode) to an existing host, replacing hosts, and adding new services (e.g. HBase) to a cluster.
http://hortonworks.com/blog/hadoop-cluster-bursting-seams-use-apache-ambari-expanding/
The Hortonworks blog has a post on the future of Apache Ambari, for which Hortonworks is trying to rally the community around three areas: operate, integrate, and extend. This includes Ambari Stacks, the Ambari REST API, and Ambari Views—each of which is described in more detail in the post.
http://hortonworks.com/blog/future-apache-ambari/
The Kite SDK, which provides libraries, tools, and more for Hadoop development, recently released version 0.15.0. A post on the Cloudera blog highlights some of the new features and improvements in the release. Among them are enriched support for working with datasets by URI (which also cleans up the syntax used to invoke several command line tools), improved support for configuration of MR and Crunch jobs, and a new parent pom for maven projects using Kite.
http://blog.cloudera.com/blog/2014/07/whats-new-in-kite-sdk-0-15-0/
This post discusses how container virtualization is a good fit for running applications in YARN. By providing isolation, security, consistency, and more, containers can solve a lot of operational issues (e.g. not polluting the host configuration with various versions of packages or configuration). In particular, the post describes how docker, which supports distributing pre-built images, is a good solution to distributing YARN applications.
http://www.kdnuggets.com/2014/07/containers-yarn-enabler.html
A post on the Cloudera blog highlights some of the changes and new features in Apache Spark 1.0. Specifically, it covers the API incompatibilities vs previous versions, the new spark-submit command, Avro support, the Spark History Server, and Spark SQL. There’s also a discussion of the stability of Spark, which the author admits is not as stable or mature as other Hadoop ecosystem components.
http://blog.cloudera.com/blog/2014/07/new-in-cdh-5-1-apache-spark-1-0/
The August edition of the Proceedings of the VLDB Endowment was recently published online. There are several papers that could be interesting to readers of this newsletter, including an experimental analysis of graph-procesing libraries (including Apache Giraph and GraphLab), a new similarity join framework for MapReduce called ClusterJoin, and an experimental analysis of Hive on Tez vs. Cloudera Impala from IBM Research.
http://www.vldb.org/pvldb/vol7.html http://www.vldb.org/pvldb/vol7/p1047-han.pdf http://www.vldb.org/pvldb/vol7/p1059-dassarma.pdf http://www.vldb.org/pvldb/vol7/p1295-floratou.pdf
SequenceIQ recently unveiled an open-source, docker-based, Hadoop-as-a-Service framework called Cloudbreak. A guest post on the Hortonworks blog explores why they chose Ambari to provision Hadoop clusters in Cloudbreak and provides an introduction to the Ambari shell.
http://hortonworks.com/blog/searching-apache-hadoop-provisioning-swiss-army-knife/
Pivotal and Hortonworks announced that they’re teaming up to collaborate on Apache Ambari. Pivotal’s distribution includes its own Pivotal Command Center, but the announcement suggests that they’ll be moving towards Apache Ambari instead. Pivotal will allocate engineering resources to Apache Ambari to help “strengthen Hadoop as an enterprise offering."
http://blog.pivotal.io/pivotal/features/strengthening-hadoop-in-the-enterprise-with-apache-ambari
The GigaOm Structure Show has an interview with Hortonworks CEO Rob Bearden in which he discusses the recent investment by HP in Hortonworks. The linked article has some excerpts from the interview, in which Bearden explains how Hortonworks’ HDP fits into the HP Haven (which includes Vertica and Autonomy in addition to Hadoop) architecture.
http://gigaom.com/2014/08/02/heres-why-hp-invested-50m-in-the-hortonworks-approach-to-hadoop/
TechRepublic has an article about the history of Cloudera, which was (one of?) the first companies to offer commercial support for Hadoop. It talks a lot about the early days of Cloudera, including Cloudera’s early hiring practices (which put an emphasis on open-source community).
http://www.techrepublic.com/article/how-cloudera-defined-big-data-and-was-defined-by-it/
Datanami has an article on Guavus Reflex, a data processing and analytics system that’s powered by Hadoop and Spark. Reflex is used by telcos to identify and respond to network outages, among other things. The article talks about the hybrid in-memory streaming and batch processing system that powers Reflex and how it recently adopted Apache Spark.
http://www.datanami.com/2014/07/30/streaming-analytics-helps-telcos-overcome-data-deluge/
IBM has published benchmarks of its SQL-on-Hadoop system, Big SQL, against Apache Hive 0.12. While vendor benchmarks should often be taken with a grain of salt, there are some interesting details in the post such as Big SQL’s support for Parquet and ORCFile formats.
http://www.ibmbigdatahub.com/blog/blistering-fast-sql-access-your-hadoop-data
Apache Ambari 1.6.1 was released this week. The new version includes a large number of bug fixes and improvements (over 500 Jiras closed!), several of which are highlighted on the Hortonworks blog. Those include performance improvements, additional DNS configuration checks, and simplified database connection setup.
http://hortonworks.com/blog/apache-ambari-1-6-1-released/
Cloudera announced two new versions of Cloudera Manager, containing a number of bug fixes. Cloudera Manager 4.8.4 solves issues with directory quotas, activity monitors, Hive, and more. Cloudera Manager 5.1.1 fixes a bug with installing JCE policy files.
Curated by Mortar Data ( http://www.mortardata.com )
Pepperdata Meetup at SQUARE in San Francisco (Sunnyvale) - Wednesday, August 6
http://www.meetup.com/Pepperdata/events/194372282/
Making Hadoop Realtime by Dr. William Bain,CEO of Scaleout Software (Los Angeles) - Wednesday, August 6
http://www.meetup.com/Los-Angeles-Big-Data-Users-Group/events/180079492/
Spark at eBay - Troubleshooting the everyday issues (Bellevue) - Wednesday, August 6
http://www.meetup.com/Seattle-Spark-Meetup/events/169932972/
Using HBase Co-Processors to Build a Transactional SQL-on-Hadoop RDBMS (Houston) - Wednesday, August 6
http://www.meetup.com/Houston-Hadoop-Meetup-Group/events/183955282/
Introduction to Spark Course: SparkML - machine learning library (5 of 7) (Austin) - Wednesday, August 6
http://www.meetup.com/Austin-ACM-SIGKDD/events/187688782/
MPP Appliance or Hadoop/Hive? What's Your Enterprise Data Warehouse Solution? (Phoenix) - Wednesday, August 6
http://www.meetup.com/Phoenix-Hadoop-User-Group/events/191956942/
IBM - Big SQL with Hands-On Lab (Columbus) - Thursday, August 7
http://www.meetup.com/Columbus-HUG/events/197731502/
Securing Hadoop Deployments to Enterprise Compliance Regulations (Ballwin) - Saturday, August 9
http://www.meetup.com/BigData-Cloud-Meetup/events/197003582/
HIVE Evolution - Latest on HIVE, SPARK, TEZ and YARN (Saint Petersburg) - Wednesday, August 6
http://www.meetup.com/Tampa-Hadoop-Meetup-Group/events/193770252/
Rethink Data - and what your business can do (Boston) - Tuesday, August 5
http://www.meetup.com/Big-Data-Analytics-Discovery-Visualization/events/197743052/
NoSQL for SQL Professionals (Toronto) - Tuesday, August 5
http://www.meetup.com/TorontoHUG/events/174630552/
Pick and choose: Database and Big Data technologies in Bluemix (Toronto) - Wednesday, August 6
http://www.meetup.com/Toronto-SMAC-Social-Mobile-Analytics-Cloud-Meetup/events/196196142/
Introduction to Apache Hive (Kanata) - Tuesday, August 5
http://www.meetup.com/HadoopOttawa/events/196236692/
M.C. Srivas: Rethinking SQL for Big Data (Sydney) - Tuesday, August 5
http://www.meetup.com/Big-Data-Analytics/events/195216262/
'Rethinking SQL for Big Data' by another VIP speaker, M. C. Srivas (MapR) (Melbourne) - Thursday, August 7
http://www.meetup.com/Big-Data-Analytics-Meetup-Group/events/187838162/
Big Data Analytics in Scala (Hong Kong) - Monday, August 4
http://www.meetup.com/HK-Functional-programming/events/179181332/
BigData.SG/HadoopSG August Meetup (Singapore) - Thursday, August 7
http://www.meetup.com/Hadoop-SG/events/196773782/
Discuss About the Big Data and How It impacting the Industries (Chandigarh) - Saturday, August 9
http://www.meetup.com/Chandigarh-Big-Data-Meetup/events/198327542/