30 June 2019
This weeks issue once again covers two week's worth of content, and there are great articles covering practical advice (e.g. how to deploy Airflow on Kubernetes) to some interesting forward-looking ideas (e.g. the multiverse database). For those actively working on data engineering projects, there should be some useful stuff whether you're implementing logging systems (see Pinterest's Singer), deploying Kafka for change data capture (see Yupto's post on their CDC pipeline), or building software with Presto, Spark, or Flink (useful articles on each!).
Lots of great papers covered in the morning paper these past few weeks—I've picked one of my favorite for this issue. It covers the notion of a multiverse database, one in which each user sees a "parallel universe" containing only the data that they have permissions/access to read. To improve performance, the universes are partially pre-computed (striking a balance between read latency and disk usage).
https://blog.acolyer.org/2019/06/17/towards-multiverse-databases/
Qubole writes about an upcoming feature of Presto—Optimized Local Scheduling. By taking advantage of split locality, the new scheduler improves cache reuse and minimizes the amount of data transferred over the network. For some workloads, they see 9x improvements. If you want to read the original design doc—there's a link at the bottom of the page to the Github issue that references it.
https://www.qubole.com/blog/presto-rubix-scheduler-improves-cache-reads/
Pinterest has open sourced Singer, their logging agent that sends data to Apache Kafka for centralized logging. Singer provides Java and Python libraries for both text and thrift format data. It also supports logging in Kubernetes via a sidecar. It includes a heartbeat system and producing audit records for centralized monitoring and alerting.
Yupto writes about their change data capture pipeline for MySQL that's built with Debezium, Apache Kafka, Apache Hudi, and their Meteorikku open source ETL framework. Hudi looks neat—you can write your records and Hudi takes care of incremental updates based on key and time columns plus a partition. Th post also includes a bit on how they monitor the pipeline.
A good tutorial for running Apache Airflow on the Azure Kubernetes Service. It covers both the general Kubernetes configuration (via Helm and the Airflow Kubernetes executor) and some Azure-specific pieces (the container registry, postgres, and Azure File Share). They also use a tool called chaoskube to occasionally restart the Airflow scheduler to workaround some bugs.
https://blog.godatadriven.com/airflow-on-aks
This week's coverage of distributed transactions comes from the Fauna DB blog, where they cover time-travel anomalies under serializability and one-copy serializability guarantees. These include immortal writes, stale reads, and causal reverse—all three are described and there are some good diagrams to illustrate what happens. The post wraps up with a matrix of several types of serializable guarantees and what types of anomalies that can occur with each.
Apache Kafka 2.3 is out, and the Confluent blog has a post summarizing new features. These include improved monitoring, a maximum log compaction lag (important for GDPR), faster broker startup times, and improvements to Kafka Connect (e.g. a new cooperative rebalancing that improves performance during a reconfiguration) and to Kafka Streams (e.g. in-memory window store).
https://www.confluent.io/blog/whats-new-in-apache-kafka-2-3
Handy collection of tools for monitoring and investigating state of your postgres cluster. There are scripts for things like finding unused indexes and looking at table bloat (from deleted but not yet vacuumed records). The list of scripts describes the permissions needed for each and a bit of description about what's returned plus what you should do with that info.
https://github.com/lob/pg_insights
The Apache Flink blog has a post on Flink 1.5+'s Broadcast state, which can be used for joining a low-throughput stream with a high-throughput one. This enables a number of interesting use cases, such as using the low-throughput stream to propagate dynamic configuration. The post has good diagrams and some example code to demonstrate things.
https://flink.apache.org/2019/06/26/broadcast-state.html
cleanframes is a new library for Apache Spark that uses implicits to apply a number of data cleansing functions to your Spark data frames as they're loaded. There's a two part post introducing the library and describing how to extend it with custom transformations and more.
https://medium.com/@dawid.rutowicz/cleanframes-data-cleansing-library-for-apache-spark-eaae526ee958
If you can get past the sensational headline, this article has a lot of good coverage of the evolution of the Apache Hadoop ecosystem over the past several years—e.g. we've only had columnar storage in Hadoop since 2012 and Apache Airflow has become a more active project than Apache Oozie. There's also a set of rebuttals to common statements arguing against hadoop/big data systems (e.g. "the dataset can fit in memory" and "just use Postgres").
https://tech.marksblogg.com/is-hadoop-dead.html
Curated by Datadog ( http://www.datadog.com )
Apache Beam Meetup 1 @ Criteo (Paris) - Monday, July 1
https://www.meetup.com/Paris-Apache-Beam-Meetup/events/261775884/
From Zero to Hero with Kafka Connect (Munich) - Wednesday, July 3
https://www.meetup.com/Apache-Kafka-Germany-Munich/events/262385125/
Let's Talk about MQTT in Real Life (Hamburg) - Wednesday, July 3
https://www.meetup.com/inovex-Meetup-Hamburg/events/259699075/
Apache Flink Meetup @ Amazon (Munich) - Thursday, July 4
https://www.meetup.com/Apache-Flink-Meetup-Munich/events/261282757/
Writing Stream Processors in Kotlin (Dusseldorf) - Thursday, July 4
https://www.meetup.com/Web-Engineering-Duesseldorf/events/261588602/
Kafka Hack Night (Bern) - Thursday, July 4
https://www.meetup.com/Bern-Hack-Night/events/262197657/
Apache Kafka & New Ways of Stream Processing with KSQL (Wien) - Tuesday, July 2
https://www.meetup.com/Vienna-Kafka-meetup/events/262314643/
Demystifying Kafka (Sofia) - Thursday, July 4
https://www.meetup.com/Ocado-Technology-Events-Sofia/events/262381550/
Strimzi: Distributed Streaming with Apache Kafka in Kubernetes (Istanbul) - Thursday, July 4
https://www.meetup.com/Istanbul-Java-User-Group/events/262262908/
Services, Deployments, and Fun with Kafka War Stories (Ramat Gan) - Monday, July 1
https://www.meetup.com/Big-things-are-happening-here/events/262449737/
Links are provided for informational purposes and do not imply endorsement. All views expressed in this newsletter are my own and do not represent the opinions of current, former, or future employers.