Data Eng Weekly Issue #275

29 July 2018

Lots of great content this week on topics from high-level architectural patterns for microservices, data platforms, and distributed systems to looks at systems internals (e.g. Kafka authorization and Qubole's spot instance framework) to sharing lessons from building out data platforms in adtech and for a museum. There's also news and announcements about several conferences, releases, and new projects.

Sponsor

Dremio is an open source Data-as-a-Service platform, based on SQL and Apache Arrow. Accelerate your queries up to 1000x. Self-service experience for BI and data science users. Download at https://bit.ly/2t96fa7 or visit http://dremio.com to learn more.

Technical

This post goes through examples of each of the eight fallacies of distributed systems and provides some common architectural patterns for each.

https://dzone.com/articles/understanding-the-8-fallacies-of-distributed-syste

Debezium, the change data capture system, uses the database's own log files (e.g. the transaction or bin log). This has a number of advantages over polling the database for changes—five of those advantages are highlighted in this post.

http://debezium.io/blog/2018/07/19/advantages-of-log-based-change-data-capture/

Here's a description of the bug in Kafka’s authorization layer underlying a recently disclosed vulnerability. THere's also a look at the patching/disclosure process and what other work has been done to verify the authorization implementations in the project.

https://developer.ibm.com/dwblog/2018/anatomy-kafka-cve/

As more and more systems support common formats and data stored in cloud object stores, it opens up opportunities to simplify the number of copies of data. This post provides some useful terminology to talk about the flavors of data architecture and their tradeoffs.

https://quickbooks-engineering.intuit.com/re-think-your-data-pipelines-in-the-decoupled-era-5b032bc8b779

A look at the data architecture and data engineering at Unruly, which is a company in the adtech space. Unruly is using both AWS and Google BigQuery with Apache Airflow to drive workflows. The post talks about the evolution of their platform, such as their move from bash to python and their future option of replacing BigQuery with Athena to simplify the architecture.

https://medium.com/unruly-engineering/data-engineering-tech-at-unruly-bb8b4afa2758

This post describes how to use the explode and collect_list operators in Spark to substitute the values in an array based on a lookup table.

https://medium.com/bluekiri/check-your-pyspark-abilities-by-solving-this-quick-challenge-86f563a343dd

Qubole writes about the components they've implemented to optimize performance in the face of spot instance termination in AWS. They've added functionality to respond to spot termination events to decommission the instance housing HDFS DataNodes (to ensure no new blocks written) and proactively cancel Spark executors rather than waiting for them to timeout. These features, coupled with their other modifications to work with spot instances, improve performance and reliability while keeping costs low.

https://www.qubole.com/blog/exploiting-aws-spot-instance-termination-notifications-in-qubole/

This article is about the data platform of a museum, which collects a number of data points through an interactive pen that visitors use. The data is processed using Logstack and ElasticSearch, and they've built a data warehouse using Amazon Redshift. It's an interesting look at a really different application than your normal internet company data platform.

https://medium.com/micah-walter-studio/data-warehousing-and-building-analytics-at-cooper-hewitt-smithsonian-design-museum-159ec772905e

This post recaps a talk from the recent Dataworks Summit about upgrading Hadoop 2 to Hadoop 3. It covers the main changes and gotchas that have to be considered when upgrading a cluster.

http://www.adaltas.com/en/2018/07/25/clusters-workloads-migration-hadoop-2-to-3/

Cloudera has the second part in their series on implementing Avro data serialization atop of Apache Kafka. This post looks at building a simple in-memory store as well as builds the first parts of a schema registry that stores its schemas in a Kafka topic.

http://blog.cloudera.com/blog/2018/07/robust-message-serialization-in-apache-kafka-using-apache-avro-part-2/

A great overview of microservices communication patterns and their trade-offs. These include: service-service communication, API Gateway, client-side aggregation, and event bus. There's also some advice for how to get off the ground with a new project.

https://medium.com/@deniseschlesinger/data-consistency-across-microservices-4f768b253816

Sponsor

Unravel released a step-by-step video http://bit.ly/unravel-video on how to make Kafka and Spark streaming applications fast and reliable. Watch it now.

http://bit.ly/unravel-video

News

Presentation slides from the recent Presto Summit have been published. Both some interesting use case and project internal talks.

https://www.starburstdata.com/technical-blog/presto-summit-2018-recap/

DataEngConf NYC takes place in November. The CFP is open through early September—they’re looking for 15 or 30 minutes talk proposals across data engineering and data science.

http://info.dataengconf.com/data-science-engineering-call-for-proposals-dec-nyc18

At Google Cloud Next, Google and several partners announce Knative, which is a new project to bring functions-as-a-service (a la AWS Lambda) to Kubernetes.

https://www.infoq.com/news/2018/07/knative-kubernetes-serverless

Several talks from Kafka Summit have been announced. The conference takes place in San Francisco in October, and early bird pricing ends in just a few days on August 1st.

https://www.confluent.io/blog/highlights-of-the-kafka-summit-san-francisco-2018-agenda/

Jobs

The Data Eng Weekly board has several jobs:

Netflix (Senior Data Engineer - Los Gatos, CA): https://jobs.dataengweekly.com/jobs/08f4bef5-90fd-4a11-9249-15e7ae86023d
Wooga (Data Engineer - Berlin): https://jobs.dataengweekly.com/jobs/3cbbc593-db24-48b8-8f9e-33c8ed72a0e8
Shopify (Software Eng Data Infrastructure - Ottawa/Waterloo/Montreal): https://jobs.dataengweekly.com/jobs/2471d248-f1a4-4e60-9787-eb7d2ce4b76f

Submit your own job at https://jobs.dataengweekly.com/

Releases

Astronomer has released version 0.3.0 of their managed Apache Airflow product. The release runs on Kubernetes to support major clouds and on-prem deployments, improves high availability, and has improvements to the command line utility.

https://www.astronomer.io/blog/announcing-astronomer-v0-3-0/

If you’re using Gradle, here’s a new plugin for the Confluent schema registry.

https://github.com/ImFlog/schema-registry-plugin

Apache HBase 2.1.0 was released. The new version includes over 240 changes, including improvements to replication and an experimental rolling upgrade capability to migrate from HBase 1.x to 2.x.

https://lists.apache.org/thread.html/b2ece2d74d26737bf0d90735f6df9610a114ab543cdec0f9da291989@%3Cannounce.apache.org%3E

Kafka Graphs is a new graph processing framework for Apache Kafka. This post describes the basics of the implementation, which is built using the vertex-centric, Pregel model.

https://yokota.blog/2018/07/23/kafka-graphs-graph-analytics-with-apache-kafka/

Jaquet is a new open source project for consuming JSON data from Apache Kafka and storing it as Apache Parquet in Amazon S3. This post talks through the motivation and design of the system.

https://medium.com/@eranshemesh/jaquet-saving-your-mass-of-events-b9d1d5f16c5

Version 1.2 of Reaper, the repair too for Apache Cassandra, has been released. It includes performance improvements, a new authentication layer, and several other usability improvements.

http://thelastpickle.com/blog/2018/07/24/reaper-1_2-released.html

Apache Accumulo 1.9.2 is out with critical bug fixes in recovery and concurrency. It's recommended for all users of Accumulo 1.8 and 1.9.

https://accumulo.apache.org/release/accumulo-1.9.2/

Debezium 0.8.1 and 0.9.0.Alpha1 were released. The 0.9.0 release will add connectors for SQL Server and Oracle. 0.8.1 has some bug fixes.

http://debezium.io/blog/2018/07/26/debezium-0-9-0-alpha1-released/

Confluent has released ansible playbooks for deploying Confluent Platform via systemd and for enabling encryption and authentication (including generating certificates for SASL).

https://www.confluent.io/blog/ansible-playbooks-for-confluent-platform/

Data Eng Weekly

Data Eng Weekly Issue #275

Sponsor

Technical

Sponsor

News

Jobs

Releases

Sponsors

Events

UNITED STATES

Arizona

Missouri

Florida

Iowa

GERMANY

SINGAPORE

AUSTRALIA