07 April 2019
It was tough choosing from all the great articles from the last two weeks—so much great stuff. This issue covers Lyft's metadata engine, local development with Apache Airflow, optimizing queries, testing Postgres DB performance with production workloads, and more. There's also a great video on idempotence and an article on learning to build distributed systems.
It turns out that there are some well-researched formulas for generating synthetic events (spacing them out over time in a realistic way), and a small tweak makes them credible for real world simulation (where time of day is also important). This post describe the formulas and has python code to do the generation, which should be useful for realistic internal test data.
https://www.tonic.ai/post/simulating-event-pipelines/
Idempotence is often a desirable feature in distributed systems, because it makes retrying after failures much easier. This video (or the transcript if you'd prefer) covers in detail why idempotence is important, and what some common strategies are for implementing it (with email server sends for example).
https://lispcast.com/what-is-idempotence/
Apache Arrow 0.13 was released. Included for the first time is the Rust-native query engine, DataFusion. There's a separate post about the DataFusion project, which supports SQL queries of Apache Parquet files and has a new experimental DataFrame-style API.
http://arrow.apache.org/blog/2019/04/02/0.13.0-release/
https://andygrove.io/2019/04/datafusion-0.13.0/
Lyft's Amundsen is their metadata collection and discovery engine, which indexes data from data stores, dashboards, schemas, streams, and more. The post highlights a number of important things to consider for a system, like the ABCs of metadata (application context, behavior, and change), the discovery vs. curation trade-off, and compliance (e.g. for GDPR). Lots of great inspiration in this post, like the types of details they highlight for each dataset, their search-centric UI, and how they collect feedback.
https://eng.lyft.com/amundsen-lyfts-data-discovery-metadata-engine-62d27254fbb9
This post on learning to build distributed systems, describes why it's hard to get started with building them and several strategies for bootstrapping your knowledge—like reading academic papers and watching videos from practitioners. Lots of great tips inside, many of which revolve around embracing and learning from failure (which is inevitable in distributed systems).
http://brooker.co.za/blog/2019/04/03/learning.html
Whirl is a tool for standing up an Apache Airflow environment for testing using Docker containers. While you can sometimes write unit tests for your data pipeline, it's often pretty difficult to do more sophisticated testing. Perhaps Whirl with Minio and similar tools can get you most of the way there.
https://blog.godatadriven.com/open-source-airflow-local-development
A great collection of tips for speeding up your PostgreSQL queries both by optimizing (several ways) the queries themselves and by adding appropriate indexes.
https://dev.to/helenanders26/sql-201-10-ways-to-tweak-slow-running-queries-3pkb
Datadog writes about how they build a highly reliable pipeline (they have a nice elaboration of what this means) for their batch processing built on Luigi and Apache Spark. The post covers how they optimize pipelines for recoverability by breaking down jobs into smaller chunks. This goes hand in hand with monitoring, including how they track a metric for data latency to detect jobs that are falling behind.
https://www.datadoghq.com/blog/engineering/highly-reliable-data-pipelines/
Great story of how one company moved a bunch of data across cloud providers. Lots of details about how they managed to move the backend without downtime, and the types of hardening (such as a kill switch) that went into the data migration service.
https://nodecraft.com/blog/development/migrating-23tb-from-s3-to-b2-in-just-7-hours
Replaying database queries on a non-live DB for load testing or baking seems like a great idea but oftentimes doesn't happen because it's too complicated. This article introduces a new tool, written in Go, to parse logs and replay queries. Seeing how this tool is built makes query-replay less intimidating and provides valuable insight into load testing your own database.
https://blog.lawrencejones.dev/building-a-postgresql-load-tester/
Twitter writes about their investigation into a performance issue with their Redis-based caching system. They dive really deep into Redis' cache invalidation codepath, going as far as to analyze the assembly instruction-by-instruction. It's a good example of how distributed systems and traditional system optimization go hand-in-hand.
Curated by Datadog ( http://www.datadog.com )
#SDBigData Meetup #26 (San Diego) - Tuesday, April 16
https://www.meetup.com/sdbigdata/events/260052877/
Cassandra & Multi-Cloud + Ingest Data from RDBMS to Cassandra w/ StreamSets (Santa Monica) - Thursday, April 18
https://www.meetup.com/Los-Angeles-Big-Data-Users-Group/events/260278330/
Real-Time Streaming Analytics (Bellevue) - Tuesday, April 16
https://www.meetup.com/real-time-analytics/events/260098744/
Seattle Apache Kafka Meetup (Bellevue) - Thursday, April 18
https://www.meetup.com/Seattle-Apache-Kafka-Meetup/events/260096882/
Unified Next-Gen Data Platform, Presented by DataStax (Coppell) - Wednesday, April 17
https://www.meetup.com/North-Texas-DAMA-Meetup/events/260103387/
Running Kafka on k8s + Monitoring with Prometheus (Tampa) - Wednesday, April 17
https://www.meetup.com/Tampa-Kubernetes-Meetup/events/258799138/
3rd Data Engineering Meetup (Belo Horizonte) - Tuesday, April 16
https://www.meetup.com/engenharia-de-dados/events/260448474/
IRELAND
Tech Talk at Night #6: Dublin Backend (Dublin) - Monday, April 15
https://www.meetup.com/tech-talk-at-night/events/260250060/
Thorough Introduction to Kafka + Blockchain Insights with Kafka (London) - Wednesday, April 17
https://www.meetup.com/Apache-Kafka-London/events/260052405/
Apache Airflow Code Breakfast (Amsterdam) - Friday, April 19
https://www.meetup.com/Amsterdam-Airflow-meetup/events/260456937/
Kafka on Kubernetes + Pipelines with Hadoop (Munich) - Monday, April 15
https://www.meetup.com/data-engineering-munich/events/259790804/
April Meetup: Druid, Flink, Kafka for Data Analytics (Bucharest) - Thursday, April 18
https://www.meetup.com/Bucharest-Big-Data-Meetup/events/259902811/
Links are provided for informational purposes and do not imply endorsement. All views expressed in this newsletter are my own and do not represent the opinions of current, former, or future employers.