02 April 2017
Lots of posts this week on stream processing—frameworks are maturing and more and more people are writing about applications. There are also a number of releases this week, including the 2.8.0 Apache Hadoop release. In news, there's a new Hadoop online training course, Cloudera has filed for their IPO, and more.
This article describes stream processing (including concepts like event time, state, and windowing from the Apache Beam model) and gives a brief overview of Apache Spark Streaming, Apache Flink, and Apache Kafka Streams.
https://blog.codecentric.de/en/2017/03/distributed-stream-processing-frameworks-fast-big-data/
The Cask Data Application Platform (CDAP) has support for running Python scripts both as part of a processing pipeline and via MapReduce or Spark. This post provides an overview of how the functionality is built into CDAP.
http://blog.cask.co/2017/03/python-for-big-data-programming/
This tutorial on the Cloudera blog shows how to build a search and analytics system using open-source tools from the Hadoop ecosystem. Specifically, data is ingested via Flume's syslog source, sent to a Kafka cluster, stored to HBase for archival, and also written to Solr and OpenTSDb via a Spark Streaming job. HUE and Grafana are used for visualization.
http://blog.cloudera.com/blog/2017/03/how-to-log-analytics-with-solr-spark-opentsdb-and-grafana/
The Confluent blog has a guest post that describes how to secure the Confluent Schema Registry (exposing it over HTTPS) and configure it to connect to secure ZooKeeper and Kafka clusters (including setting up of per-topic ACLs required for the Schema Registry's data storage).
https://www.confluent.io/blog/securing-confluent-schema-registry-apache-kafka/
The Apache Flink blog has a post on Flink's Table APIs. Built on Apache Calcite, the table APIs support both batch and streaming use cases. For the latter, Flink has out of the box support for sliding window, tumbling window, and session windows (each supports both event and processing time).The post also has a look at Flink 1.2's "User-defined Table Functions" feature.
http://flink.apache.org/news/2017/03/29/table-sql-api-update.html
In a good follow up to the previous post, this article provides examples and visualizations of of how Flink's session windows work. It also covers how to reprocess data (including when data is no longer in Kafka and needs to be read from a file system) and visualizes how a join works.
http://getindata.com/blog/streaming-analytics-better-than-classic-batch-when-and-why-part-2
The Databricks blog has a guest post from the team at the Dollar Shave Club about how they connect Apache Spark (via Databricks) to Amazon Redshift for machine learning applications.
Amazon has a post on how to use their Amazon Key Management Service to encrypt data as it goes into Amazon Kinesis (and decrypt as it comes out).
https://aws.amazon.com/blogs/big-data/encrypt-and-decrypt-amazon-kinesis-records-using-aws-kms/
This post has an interesting look at the complexity in open source caused by dependencies between project communities. Hadoop hasn't been the only game in town for a while, and dependencies (and thus bug fixes or new features) can end up being several project deep.
https://www.enterprisetech.com/2017/03/24/communities-communities-next-era-open-source-software/
Early bird registration for Spark Summit, which takes place in June, ends this week. The conference is three days at the Moscone Center in San Francisco, and there are nine tracks.
https://databricks.com/blog/2017/03/30/tenth-spark-summit-with-a-terrific-agenda-for-all.html
Registration is now open for HBaseCon West, which takes place in Mountain View on June 12th. The call for Papers is open through April 24th.
The Linux Foundation, which runs the ODPi project, has announced a new free online course "Intro to Apache Hadoop" through edX.
Cloudera has filed their S-1 IPO paperwork, and their IPO is anticipated for sometime in the next few weeks. TechCrunch has looked into the filing and highlighted some details (including a bit about the partnership with Intel). The stock is set to trade under the ticker CLDR.
https://techcrunch.com/2017/03/31/cloudera-posts-long-awaited-ipo-filing/
Apache Hadoop 2.8.0 was released. It includes a number of improvements and new features across Hadoop Common (including improvements to the S3A file system and new integration with Azure Data Lake), HDFS (several enhancements to WeHDFS), YARN, and MapReduce.
http://hadoop.apache.org/docs/r2.8.0/index.html
Apache HBase 1.2.5 includes 50 resolved issues, several of which are critical bug fixes.
Apache Storm 1.1.0 is out with support for streaming SQL, improvements to the Apache Kafka integration, new integration with Druid, OpenTSDB, AWS Kinesis, and more.
http://storm.apache.org/2017/03/29/storm110-released.html
Version 0.8-incubating of Apache Atlas was released. Atlas is a governance and metadata framework for Hadoop, which provides features like centralized audiding and data lineage.
Apache Zeppelin 0.7.1 was released. The new version has a rewriten python interpreter, improved stability, and a number of others fixes.
http://zeppelin.apache.org/releases/zeppelin-release-0.7.1.html
Curated by Datadog ( http://www.datadog.com )
Apex Big Data World (Mountain View) - Tuesday, April 4
https://www.meetup.com/Apex-Users-Group-Bay-Area/events/236223984/
Apache Flink: Building a Stream Processor for Event-Driven Applications (San Francisco) - Thursday, April 6
https://www.meetup.com/San-Francisco-Bay-Area-Big-Data-and-Scalable-Systems/events/238629209/
Microservices With Hadoop (St. Louis) - Wednesday, April 5
https://www.meetup.com/St-Louis-Hadoop-Users-Group/events/237706779/
Big Data & Predictive Analytics 2017 (Arlington) - Tuesday, April 4
https://www.meetup.com/Big-Data-Ingest-Washington-DC/events/237749453/
Data Engineering London Meetup (London) - Tuesday, April 4
https://www.meetup.com/Data-Engineering-London/events/237678668/
Big Data Analytics for Small & Medium Sized Enterprises. Is It Real? (Oslo) - Monday, April 3
https://www.meetup.com/OsloBigDataDay/events/237398012/
Simple Real-Time Analytics With Spark + Build the Simplest Data Pipeline (Stockholm) - Thursday, April 6
https://www.meetup.com/Knock-Data-Stockholm/events/237620789/
The Hadoop Ecosystem (Madrid) - Monday, April 3
https://www.meetup.com/Meetup-Big-Data-DataHack-Madrid/events/238643984/
Machine Learning & PySpark With Spark Committer Holden Karau (Madrid) - Wednesday, April 5
https://www.meetup.com/Big-Data-Developers-in-Madrid/events/238739273/
Spark Meetup with Cloudera and Talend (Paris) - Monday, April 3
https://www.meetup.com/Paris-Spark-Meetup/events/238766630/
Meetup #5 (Tourcoing) - Thursday, April 6
https://www.meetup.com/Lille-Big-Data-and-Machine-Learning-Meetup/events/235858084/
Accelerating Apache Big Data and Machine Learning Workloads on OpenPOWER (Munich) - Tuesday, April 4
https://www.meetup.com/Big-Data-Developers-in-Munich/events/238408703/
HS Munich: Tensorflow and Hadoop, Accelerating HBase, Hive vs. Spark (Munich) - Tuesday, April 4
https://www.meetup.com/BDOOP-BigData-Operations-On-Perfomance-Barcelona/events/237492919/
Data Science With Spark on Hadoop (Munich) - Tuesday, April 4
https://www.meetup.com/futureofdata-munich/events/237165057/
After–Hadoop Summit Meetup: IoT, Data Science, Spark, Flink, Deep Learning (Munich) - Wednesday, April 5
https://www.meetup.com/Big-Data-Developers-in-Munich/events/238851692/
Asynchronous Queue Solution with Kafka + Advanced Data Analysis on Hadoop Clusters (Brno) - Wednesday, April 5
https://www.meetup.com/dataheads/events/238613499/
Seznam’s Euphoria API (Prague) - Thursday, April 6
https://www.meetup.com/CS-HUG/events/238741285/
GeekNight: Converged Data Platform (Hyderabad) - Wednesday, April 5
https://www.meetup.com/hyderabad-scalability/events/238830065/
IoT with Talend: Integrating Real-Time Data Streams With Spark and Kafka (Singapore) - Thursday, April 6
https://www.meetup.com/IoT_SG/events/237650455/
Kafka & Kafka Streams (Sydney) - Wednesday, April 5
https://www.meetup.com/apache-kafka-sydney/events/238737190/