02 January 2017
Lots of great content from the last two weeks, including a few year in review posts and new releases of Apache Spark, Apache Kafka, and Apache Flink. In technical posts, there is a bunch of variety, from HDFS to SparkR, including a great post on running Spark on YARN in production.
StreamSets is a tool in a similar space to Apache NiFi (and Apache Flume for some use cases) that has been gaining steam in the past year or so. This post shows how to build a custom origin for StreamSets Data Collector. The process is relatively straightforward, which makes it a powerful system for hooking into custom data sources.
https://streamsets.com/blog/building-amazon-sqs-custom-origin-streamsets-data-collector/
Hortonworks has written about an initiative to bring fine-grained access control (i.e. column-level) to Apache Spark by having it read data through the Live Long and Prosper (LLAP) daemons from Apache Hive. LLAP is integrated with Apache Ranger, which enforces the authorization on read. A post on the Hortonworks blog has more details on how this works and a link to a knowledge base article that details how to enable it for a HDP 2.5.3.
The Cloudera blog has a post about how the HDFS DataNode detects file corruption and disk failure by performing periodic scans. These include block scans (to detect corruption), directory scans (to repair inconsistencies in cached data), and disk checks (to verify permissions and other basic sanity checks).
http://blog.cloudera.com/blog/2016/12/hdfs-datanode-scanners-and-disk-checker-explained/
Inovex has shared lessons learned from running Spark streaming on YARN for over a year. Their streaming application loads data from a JMS message queue, joins with lookup data in HBase, and writes out to HBase using bulk puts. Topics covered include configuration settings (for YARN and Spark), handing of backpressure, deployments (using a marker file in HDFS with checkpointing), monitoring (including a set of custom metrics), and logging.
https://www.inovex.de/blog/247-spark-streaming-on-yarn-in-production/
Amazon EMR now supports CloudWatch events for various state changes in an EMR cluster. Coupled with SNS, Lambda, SQS, or another integration, this can be used to programmatically respond to failure, auto-scaling events, or more.
Using an EMR bootstrap action, a new cluster can come up with JupyterHub (Jupyter with multi-user support), which supports R, Python, Spark, and much more. The AWS Big Data blog has a bunch of examples of using Jupyter with several Hadoop ecosystem components (including pyhive for Hive/Presto and SparkSQL).
https://aws.amazon.com/blogs/big-data/running-jupyter-notebook-and-jupyterhub-on-amazon-emr/
Apache Apex has several mechanisms to monitor and debug a running streaming program. Among these is the ability to generate a stack trace of the application, which can be triggered from the command-line or the web interface.
https://www.datatorrent.com/blog/getting-stack-traces-apache-apex-applications/
The upcoming Hue 3.12 will include improvement to the SQL system (formatting, SQL autocomplete, data preview popups), email notifications, and more. Read about the upcoming features in the three following blog posts.
http://gethue.com/sql-improvements-with-row-counts-sample-popup-and-more/
http://gethue.com/oozie-improvements-in-3-12-with-email-notifications-and-extended-dashboard-filtering/
http://gethue.com/extract-archives-as-oozie-job/
This post, in notebook format, provides background aimed at someone with an R background who is getting started with SparkR. It provides details on the programming model, data model, execution model, and some of the machine learning and other APIs.
https://databricks.com/blog/2016/12/28/10-things-i-wish-i-knew-before-using-apache-sparkr.html
As someone who's been through multiple Hadoop upgrades, I can sympathize with this analysis of effort and risk involved in major system upgrades. It's a good analysis and describes some steps that vendors can take to get customers to upgrade sooner than later.
https://medium.com/@gwenshapira/upgrades-867afe6fb962
Big Data Tech Warsaw takes place on February 9th. This post has a description of what to expect, including a sample of the 25+ talks.
http://getindata.com/blog/post/hot-topics-at-big-data-tech-warsaw-2017/
Hortonworks recently unveiled Hortonworks Data Cloud for AWS. This post addresses some of the top questions about the service, including how it is purchased through the AWS marketplace, how to get data into a cluster, and the types of workloads it's targeting (for now, ephemeral clusters not HA).
http://hortonworks.com/blog/10-questions-hortonworks-data-cloud-aws/
InfoWorld has an interview with Hortonworks CTO Scott Gnau. Among the topics covered are HDPs release strategy/cadence and the important role of data-in-motion to the future industry.
There are several year end posts from this week. Hortonworks and Databricks covering top posts on their blogs from the year. Flink has a year in review of community growth and new features as well as a look ahead to what's underway for 2017.
http://hortonworks.com/blog/top-5-blogs-2016/
http://flink.apache.org/news/2016/12/19/2016-year-in-review.html
https://databricks.com/blog/2016/12/30/top-10-apache-spark-blog-posts-from-2016.html
IBM released the December refresh of Big SQL 4.2. It includes a number of fixes to the Hadoop integration.
https://developer.ibm.com/hadoop/2016/12/20/big-sql-4-2-december-refresh-is-now-available/
Databricks has announced that their deep learning and GPU support has hit GA. The announcement includes a tutorial and example notebook for doing image classification using the Amazon P2 instance types on a Databricks cluster.
https://databricks.com/blog/2016/12/21/deep-learning-on-databricks.html
Casks' CDAP version 4 was released. The announcement includes an overview of the major features, which include an updated UX, an app store, and new platform features and improvements.
Apache Kafka 0.10.1.1 was with a number of bug fixes and a few minor improvements.
Apache Flink 1.1.4 was released. It includes a number of bug fixes and improvements (over 75 resolved issues). The release page has more details, including a note to users using RocksDB.
http://flink.apache.org/news/2016/12/21/release-1.1.4.html
Apache Spark 2.1.0 was released with event time watermarks (for stream processing) and support for Apache Kafka 0.10. There were over 1200 issues resolved, including changes to SparkSQL, MLlib, SparkR, and GraphX.
https://spark.apache.org/releases/spark-release-2-1-0.html
Version 0.11.0 of the Apache Knox gateway was released. This release adds a basic admin UI, metrics collection, and support for new systems.
Apache NiFi announced a new bug fix release, version 1.1.1.
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.1.1
multi-hbase is a prototype client library to support both the 0.94 and 1.2 branches of Apache HBase.
https://github.com/eolivelli/multi-hbase
BigDL is a distributed deep learning library open sourced by Intel. By leveraging the Intel Math Kernel Library, it claims massive speedups over other implementations.
https://github.com/intel-analytics/BigDL
Curated by Datadog ( http://www.datadog.com )
Fault Tolerance in Spark: Lessons Learned from Production (Austin) - Thursday, January 5
https://www.meetup.com/austin-spark-meetup/events/235890083/
Introduction to Apache Kudu (Saint Louis) - Wednesday, January 4
https://www.meetup.com/St-Louis-Hadoop-Users-Group/events/236343675/
Big Data Meetup 23 (Mississauga) - Sunday, January 8
https://www.meetup.com/Mississauga-Big-Data-Analytics-Meetup/events/236415502/
A Deep Dive Into Structured Streaming/Predictive Analytics with SparkR (Istanbul) - Saturday, January 7
https://www.meetup.com/Istanbul-Spark-Meetup/events/236178394/
Meet Hadoop Family: Season 1 (Jakarta) - Thursday, January 5
https://www.meetup.com/Jakarta-Hadoop-Big-Data/events/235666619/