06 August 2017
Short issue this week, but several high quality posts including a presentation on Spotify's data infrastructure, an in-depth look at using Kafka as a central event store, several releases, and more.
Great presentation on how Spotify uses Google BigQuery and Scio (a scala wrapper for Apache Beam/Cloud DataFlow) to power their data infrastructure. Scio integrates with BigQuery to stream results of a query, implements several join optimizations, has a REPL, and has several other advance features. The presentation also discusses a dataset diff tool for validating pipeline changes and ML models and Featran-a library for transforming features in a type safe way. It wraps up with a number of challenges that they've faced along the way.
https://www.youtube.com/watch?v=1dchSsac3T4
https://www.slideshare.net/sinisalyh/sorry-how-bieber-broke-google-cloud-at-spotify
A walkthrough of change data capture and using events stored in Apache Kafka as the source of truth rather than the relational database. It describes patterns such as writing to the database through the queue, secondary indexing in elastic search, migrating away from legacy data systems, and so-called "memory images" for serving a dataset out of memory rather than via a relational database.
https://www.confluent.io/blog/messaging-single-source-truth/
This post presents ten reasons why building your own deployment tools for Hadoop and Spark is not a good idea. It argues that rolling your own cluster is analogous to building a car.
https://www.bluedata.com/blog/2017/08/hadoop-spark-docker-ten-things-to-know/
The Cloudera blog has an overview of S3Guard, which is a new feature for Hadoop's S3A file system to make use of a metadata store to add a consistent view to data stored in S3.
http://blog.cloudera.com/blog/2017/08/introducing-s3guard-s3-consistency-for-apache-hadoop/
BigDL is a deep learning library and GraphX is a graph processing library for Apache Spark. The Qubole blog has a tutorial demonstrating how to train and evaluate a model using BigDL. There's a second post that walks through (with example code) the GraphX APIs.
https://www.qubole.com/blog/deep-learning-qubole-using-bigdl-apache-spark-part-2/
http://www.qubole.com/blog/processing-hierarchical-data-using-spark-graphx-pregel-api/
The Amazon Big Data blog describes how to enable Hive's LLAP to improve performance of Hive queries.
https://aws.amazon.com/blogs/big-data/turbocharge-your-apache-hive-queries-on-amazon-emr-using-llap/
Version 3.3 of the Confluent Platform has been released. This version includes the exactly once semantics of Apache Kafka 0.11. The post has a tour of features, labeling them by where the features live (i.e. are they open source or enterprise-only).
https://www.confluent.io/blog/we-will-say-exactly-confluent-platform-3-3-available-now/
Apache Curator 4.0.0 was released. Curator is a client library for Apache ZooKeeper, and the new version adds support for TTL Nodes, a strongly typed DSL, and more.
Apache Storm 1.0.4 and 1.1.1 were released. Both releases include a number of bug fixes and improvements.
https://lists.apache.org/thread.html/5ce900f15494807d1ff8674e5983f0dea05f5bb5c6d11557aa2217c7@%3Cannounce.apache.org%3E
https://lists.apache.org/thread.html/c453d88ac7200f2fe52d5d84d4a1bea3c8b8b8fc7adfa2a5c7d98baf@%3Cannounce.apache.org%3E
Apache Flink 1.3.2 was released. It includes a number of bug fixes and improvements. Notably, there is a fix for users of the FlinkKafkaConsumer which may take additional care when upgrading.
https://flink.apache.org/news/2017/08/05/release-1.3.2.html
Curated by Datadog ( http://www.datadog.com )
Data Engineering vs Data Science, DataOps, & Overcoming Back Office Barriers (Westlake Village) - Tuesday, August 8
https://www.meetup.com/Westlake-Village-Data-Science-Meetup/events/241795805/
Hands On: Introduction to the Hadoop Ecosystem (Kansas City) - Tuesday, August 8
https://www.meetup.com/KCLearnBigData/events/241191557/
Spark: An Intro to Crazy Fast Big Data (Orlando) - Wednesday, August 9
https://www.meetup.com/Orlando-MLDS/events/242162367/
Meet the Experts That Support Apache Kafka (Raleigh) - Wednesday, August 9
https://www.meetup.com/Raleigh-Apache-Kafka-Meetup-by-Confluent/events/242094828/
Deep Dive into HDF 3.0 (Morris Plains) - Tuesday, August 8
https://www.meetup.com/futureofdata-princeton/events/240972326/
Building a Production Data Pipeline (New York) - Thursday, August 10
https://www.meetup.com/NYC-Data-Engineering/events/242158875/
Architecting a Recommendation Engine with Scala & Spark (Singapore) - Thursday, August 10
https://www.meetup.com/Singapore-Scala-Programmers/events/241808857/
Data-in-Motion: Recent Advances in Apache Projects for Streaming Data (Sydney) - Wednesday, August 9
https://www.meetup.com/Sydney-Big-Data-and-Analytics/events/241321953/
If you didn't receive this email directly, and you'd like to subscribe to weekly emails please visit https://hadoopweekly.com