04 January 2015
The first issue of 2015 starts off the year relatively quiet, although there are a number good technical and industry news articles. Technical articles cover the Kite SDK, Kafka, HBase, Hive, and Sqoop; news articles include a number of year-end synopses. Based on these, it looks like 2015 will be an interesting year for the Hadoop ecosystem.
This post describes how to use the Kite SDK’s command line tools to ingest data into HDFS. Using the kite-dataset command, the author sets up the schema for datasets, creates tables in Hive, and loads the datasets. It also describes how to use the tools to build parquet files from csv.
http://captechconsulting.com/blog/ben-harden/ingesting-data-quickly-using-kite-the-command-line
Kafka has a built-in tool called MirrorMaker for replicating data between Kafka clusters. This post looks at an alternative implementation built with golang and supporting similar functionality. The tool, Go Kafka Mirror Maker, also supports a few additional features like adding a topic prefix to avoid collisions (and doesn’t run on the JVM, so it has less overhead)
http://allthingshadoop.com/2014/12/29/multi-datacenter-replication-with-apache-kafka/
This post provides a thorough overview of HBase Coprocessors, which (as the article describes) are analogous to triggers and stored procedures, MapReduce, and aspect oriented programming. The post describes the main interfaces and used to implement two categories of coprocessors, complete with a walkthrough of a coprocessor implementation. It also details the steps needed to deploy a co-processor.
http://www.3pillarglobal.com/insights/hbase-coprocessors
Recent releases of Apache Hive have a number of new optimizations, which must be enabled (possibly by reprocessing data). This post describes several of those features (Tez, cost-based optimization, vectorization) and provides instructions for enabling them.
https://www.altiscale.com/hadoop-blog/optimizing-apachehive-by-tuning-configuration-parameters/
This two-part blog series looks at several SQL-on-Hadoop engines. The first post looks at various storage backends (HBase, HDFS, etc) and file formats, while the second describes several major systems (Hive, Impala, Presto, Drill, and Spark SQL). There’s also a discussion of if these systems are appropriate for for OLTP and OLAP.
https://phdata.io/the-truth-about-sql-on-hadoop-part-1/
https://phdata.io/the-truth-about-sql-on-hadoop-part-2/
Security has been a hot topic for the Hadoop ecosystem recently, and most systems are adopting or improving their enterprise security features. This post describes a new security feature in Sqoop2: support for Kerberos. It walks through the steps necessary to enable security as part of the Sqoop2 server.
http://ingest.tips/2015/01/01/kerberos-support-in-sqoop2/
This post presents a bullish view on the future of big data. The author argues that big data provides a mechanism to measure and analyze systems that never existed before. Referencing success stories from the energy and agriculture sectors, the post describes some of the new capabilities that big data offers.
http://www.odbms.org/2014/12/underhyped-big-data-scientific-method/
This year-end retrospective piece looks at some of the advances in machine learning tools for big data in 2014 (Spark 1.0, H20, GraphLab) and makes some predictions for the industry in 2015.
https://www.linkedin.com/pulse/new-year-practical-data-kyle-napierkowski
Based on data from the recent Hortonworks IPO and reports from Wikibon and Forrester, this post speculates on the future of the Hadoop industry. The article points out some bearish observations, such as the sparsity of successful companies based on open-source and the fact that Google has moved away from MapReduce.
http://whatsthebigdata.com/2014/12/31/the-day-the-hadoop-bubble-started-to-quiver/
We’ve seen quite a few end-of-year posts, and Qubole’s list of Hadoop Happenings this week includes several articles that touch on this theme.
http://www.qubole.com/hadoop-lives-hype/
MemSQL has open-sourced a new tool for loading data from S3 and HDFS to MemSQL and MySQL. The system is somewhat similar to Sqoop, but provides additional features like deduplication and failure handling.
http://blog.memsql.com/memsql-loader/
Curated by Mortar Data ( http://www.mortardata.com )
Docker: New Approaches to Software Development (San Ramon) - Tuesday, January 6
http://www.meetup.com/Docker-Containers-Cloud-Foundry-PaaS-East-Bay-Meetup/events/219129851/
A Real-Time Streaming Implementation of Markov Chain–Based Fraud Detection (Newport Beach) - Thursday, January 8
http://www.meetup.com/Orange-County-Java-Users-Group-OCJUG/events/219321555/
Hadoop Framework and Tools, Spark Intro, plus Data Science with R and Python (Fremont) - Thursday, January 8
http://www.meetup.com/East-Bay-Big-Data-and-Data-Science-Meetup/events/219416430/
How We Used Storm Wrong, Then Right, at AdRoll (Emeryville) - Thursday, January 8
http://www.meetup.com/Bay-Area-Storm-Users/events/218816484/
Hadoop Past, Present and Future, plus NoSQL Data Modeling and Couchbase Mobile (Las Vegas) - Wednesday, January 7
http://www.meetup.com/lvbigdata/events/218825012/
BDW Meetup: Spark SQL (New York) - Wednesday, January 7
http://www.meetup.com/Big-Data-Warehousing/events/219272125/
RBelgium #9, Including “RHadoop, Using R in Hadoop” (Brussels) - Friday, January 9
http://www.meetup.com/RBelgium/events/219295488/
Spark Meetup (Shanghai) - Saturday, January 10
http://www.meetup.com/Shanghai-Apache-Spark-Meetup/events/219419191/
Streaming and Apache Spark (Bangalore) - Saturday, January 10
http://www.meetup.com/Real-Time-Data-Processing-and-Cloud-Computing/events/219490940/