10 December 2017
Lots of great technical content in this weeks issue—three posts on data systems and Kubernetes, a good overview of Apache Hadoop & Kerberos, an overview of the architecture of Apache Pulsar (incubating), a look at the new Wallaroo stream processing framework, and more.
For security, Hadoop uses Kerberos' Delegation Tokens to allow distributed programs to execute on behalf of the user (and thus access data that the user has permissions for). This post looks at how this is implemented for YARN and Map/Reduce tasks, some common errors, the delegation token lifecycle, and more.
http://blog.cloudera.com/blog/2017/12/hadoop-delegation-tokens-explained/
Hue 4.2 added improved support for Impala. The SQL editor now supports computing table stats, looking at the profile, and visualizing the output of the query planner.
http://gethue.com/browsing-impala-query-execution-within-the-sql-editor/
As Kubernetes seems to be gaining lots of traction for container orchestration, it's pretty natural to try to run Spark jobs with it. This first post describes how to do so, and it describes some of the current shortcomings in the current implementation. The second looks at how to then integrate with Apache Zeppelin, which has a few gotchas.
https://banzaicloud.com/blog/scaling-spark-k8s/
https://banzaicloud.com/blog/zeppelin-spark-k8/
This deck briefly provides an overview of Kafka Streams, shows how it integrates with Spring Cloud Streams, and describes how to use Kubernetes StatefulSets to do stateful stream processing. More details about running Kafka via Kubernetes/OpenShift are provided at the readme of the barnabas github repo.
https://speakerdeck.com/mbogoevici/kafka-streams-from-the-grounds-up-to-the-cloud
https://github.com/enmasseproject/barnabas
This post describes how to use Cloud Dataflow Pipeline templates to periodically ingest data from a tweet stream. The job is triggered via App Engine cron, and the data is available for query in BigQuery once it's processed.
While Apache Pulsar (incubating) shares some similarities with Apache Kafka, it has a different architecture. Namely, it has stateless brokers and separate storage bookies (services implemented via Apache BookKeeper). Data is stored as segments, which allows scale up without rebalancing. The post describes this architecture and compares with Kafka. Also of note: Pulsar provides a Kafka-compatible API, which aims to provide drop-in compatibility.
https://streaml.io/blog/pulsar-segment-based-architecture/
On the topic of Kubernetes (k8s), this post describes some of the challenges for adopting k8s for the Hadoop stack. Fundamentally, k8s is targeting stateless apps (although there is some new support for volumes and stateful sets) which makes it a challenging fit for HDFS and other apps storing data. With that said, the BlueData team has built a prototype for indirectly deploying Hadoop with k8s via their EPIC software.
https://www.bluedata.com/blog/2017/12/big-data-container-orchestration-kubernetes-k8s/
This post demonstrates how to use StreamSets to infer a schema for a CSV file, to convert certain field types to non-string values, and to extract the schema & data from the resulting avro data file.
https://streamsets.com/blog/generate-avro-schema-automatically/
Wallaroo is an up and coming stream processing framework with first-class support for non-JVM languages. This post looks at the python Pipeline API, describes how Wallaroo partitions data, and describes its mechanism for stateful processing. The post uses a stock market (pricing and order) example to illustrate these pieces.
https://blog.wallaroolabs.com/2017/12/stateful-multi-stream-processing-in-python-with-wallaroo/
The Slack engineering blog tells the story of scaling their job queuing system. Originally built on redis task queues, there were some architectural drawbacks (particularly around memory). Rather than a big-bang switch over, they put Kafka in front of Redis to absorb some of the write throughput. In the process they implemented two new services in golang (to front-Kafka and to relay to redis), which are described in detail in the post.
https://slack.engineering/scaling-slacks-job-queue-687222e9d100
This post uses the Hail framework for analyzing genomics data. The output of that analysis is stored in S3, and with some help from AWS Glue and is then available for querying via Amazon Athena.
https://aws.amazon.com/blogs/big-data/genomic-analysis-with-hail-on-amazon-emr-and-amazon-athena/
This post describes how to ensure that data is deleted from Kafka, by null'ing out the value of a record and ensuring that compaction (which does the cleanup behind the scenes) runs periodically.
https://www.confluent.io/blog/handling-gdpr-log-forget/
Databricks is renaming Spark Summit to be Spark + AI Summit in 2018.
https://databricks.com/blog/2017/12/06/spark-summit-is-becoming-the-spark-ai-summit.html
Two podcasts of note this week: Roaring Elephant features folks from Streamlio on Apache Pulsar, and Software Engineering Daily has Martin Kleppman on conflict-free replicated data types.
https://roaringelephant.org/2017/12/05/episode-64-talking-apache-pulsar-with-matteo-and-sijie-from-streamlio/
https://softwareengineeringdaily.com/2017/12/08/decentralized-objects-with-martin-kleppman/
Version 0.3.0 of Apache MiNiFi C++ was released. It's still not considered ready for production, but the new version has new features including support to write directly to Kafka.
Druid 0.11.0 was released. Major new features include TLS support, a redis cache extension, various improvements to Druid SQL, and GroupBY performance improvements.
http://druid.io/blog/2017/12/04/druid-0-11-0.html
Curated by Datadog ( http://www.datadog.com )
Using Open Source Data Platforms to Deliver New Insights (Denver) - Wednesday, December 13
https://www.meetup.com/PyData-Denver/events/244803080/
Reactive Applications with the SMACK Stack (Kanata) - Thursday, December 14
https://www.meetup.com/Big-Data-Developers-in-Ottawa/events/245469299/
IRELAND
Airflow, Big Data, and Data Science (Dublin) - Thursday, December 14
https://www.meetup.com/Ireland-IOT-meetup/events/245288076/
Processing Streaming Data at a Large Scale with Kafka (Lisbon) - Monday, December 11
https://www.meetup.com/DevOps-Lisbon/events/242127739/
Hadoop Live (Madrid) - Tuesday, December 12
https://www.meetup.com/Meetup-Big-Data-DataHack-Madrid/events/245314264/
Streaming Architectures (Paris) - Tuesday, December 12
https://www.meetup.com/architecture-streaming/events/245462414/
Paris Fast Data (Paris) - Tuesday, December 12
https://www.meetup.com/fastdata/events/244784067/
Aggregating Online Experiments Data & Comparing Hadoop Cloud Offerings (Amsterdam) - Tuesday, December 12
https://www.meetup.com/Netherlands-Hadoop-User-Group/events/245357889/
CodeBreakfast: Apache Airflow Edition (Amsterdam) - Friday, December 15
https://www.meetup.com/PyData-NL/events/244994388/
2nd Sydney Data Engineering Meetup (Sydney) - Thursday, December 14
https://www.meetup.com/Sydney-Data-Engineering-Meetup/events/244682587/
Azure Stream Analytics (Wellington) - Thursday, December 14
https://www.meetup.com/Wellington-Data-Management-and-Analytics-Meetup/events/244195276/
Scala Taiwan #21: Introduction to Apache Kafka (Taipei) - Wednesday, December 13
https://www.meetup.com/Scala-Taiwan-Meetup/events/245388617/