22 October 2017
The theme of this week is certainly stream processing—Spotify's Scio, a couple of posts on streaming delivery semantics, KSQL, Kafka Streams, and more. The "Stream All the Things!" presentation nicely summarizes the theory and practice of the (Kafka-based) event sourcing pattern that has been a recurring topic over the past year or so. In releases, John Snow Labs has open-sourced a natural language processing library for Apache Spark.
If you're using PySpark, you've probably wanted to combine it with Pandas or other python libraries. This post describes why this is a bit of a challenge and provides some code to convert data between numpy types and PySpark-compatible types (and vice-versa) for implementing custom user defined functions. It's an in-depth article that also explains some of the internals of PySpark.
http://www.florianwilhelm.info/2017/10/efficient_udfs_with_pyspark/
Spotify uses a tool they've open-sourced, called Scio, for many of their big data jobs. This post looks at the evolution of their data infrastructure from Python MapReduce jobs with an on-prem Hadoop cluster to usage of Google BigQuery for most ad-hoc queries. The post motivates this move by describing the advantages of the Beam Model that Scio uses, describes why they've written a Scala API atop of Apache Beam, and more.
https://labs.spotify.com/2017/10/16/big-data-processing-at-spotify-the-road-to-scio-part-1/
This post from Streamlio looks at delivery semantics in stream processing. For exactly once (or effectively once) delivery, it describes two strategies—distributed snapshot/state checkpoint and at-least-once with deduplication.
https://streaml.io/blog/exactly-once/
On the topic of exactly once, this post describes how to implement deduplication with Apache Spark's Structured Streaming. In addition to deduplication based on watermark, you can also add custom logic for a stateful aggregation (using mapGroupsWithState).
Building a machine learning model that adapts in real time to new information has long been a end-goal of many ML pipelines. Kafka Streams makes this relatively easy by using the same code for offline and online training. This post walks through building out a real time evaluation and training pipeline with flight arrival data as an example.
https://www.confluent.io/blog/predicting-flight-arrivals-with-the-apache-kafka-streams-api/
The Rittman Mead blog has a good intro to KSQL for Apache Kafka. The post walks through an example of aggregating Twitter data with SQL and performing a join across streams. There are several diagrams to help communicate KSQL semantics.
https://www.rittmanmead.com/blog/2017/10/ksql-streaming-sql-for-apache-kafka/
The Confluent blog has a guest post from Zalando on how they use Apache Kafka to index and rank information from fashion websites. The system uses the Hyperlink Induced Topic Search (HITS) algorithm and is built on Kafka Streams. The post describes both HITS and how they use Kafka Streams on AWS.
https://www.confluent.io/blog/ranking-websites-real-time-apache-kafkas-streams-api/
This presentation covers a lot of ground on building a streaming-centric application and data platform. First, it motivates this event sourcing architecture by demonstrating the simplicity it brings to connecting services together. Next, it discusses the trade-offs of various streaming/messaging systems (Akka, Spark, Flink, and others) for realtime and analytics use cases. Towards the end, there are two Venn diagrams that nicely sum up the benefits of the new architecture—microservices and fast data have a lot more overlap in architecture and concerns vs. the previous world of services with big data.
https://deanwampler.github.io/polyglotprogramming/papers/StreamAllTheThings.pdf
Forbes has an interview with Confluent co-founder and CTO Neha Narkhede. In it, she talks about her experiences before and in industry, the growth of Kafka at LinkedIn, the founding of Confluent, and more.
MapR has announced a managed service for the MapR Converged Data Platform.
https://mapr.com/company/press-releases/mapr-launches-managed-services-for-improved-dataops/
The team at Spotify has interesting idea for solving data quality and reliability issues. They have gamified test coverage and quality by introducing the Test Certified for Data-Focused teams program.
https://labs.spotify.com/2017/10/17/tc4d-data-quality-by-engineers-for-engineers/
Slate has an article reflecting on how the term big data is no longer widely used. There are several interesting observations on how business and algorithmic assumptions related to data have changed over the past few years.
http://www.slate.com/articles/technology/technology/2017/10/what_happened_to_big_data.html
Amazon Redshift has announced support for a new family of nodes (dc2) that are the same price but provide up to twice the performance as the previous generation.
Version 1.20.0-incubating of Apache Pulsar has been released. Major new features include end-to-end encryption, support for event time, deduplication of messages, and more.
The John Snow Labs NLP library is a new open source framework for natural language processing on Apache Spark. The post describes it in more detail, including how it compliments Spark's ML libraries and provides better performance than other solutions.
Curated by Datadog ( http://www.datadog.com )
Tensorflow on Apache Hadoop YARN (San Francisco) - Wednesday, October 18
https://www.meetup.com/SF-Big-Analytics/events/243896061/
Apache Ignite: Building Consistent, HA Distributed Systems & Memory-Centric SQL (San Ramon) - Thursday, October 19
https://www.meetup.com/datariders/events/243704206/
The Evolving Landscape of Data Engineering & How Systems Fail (San Francisco) - Thursday, October 19
https://www.meetup.com/Data-Engineering-Club/events/243137230/
Spark MLlib (Salt Lake City) - Monday, October 16
https://www.meetup.com/apache-spark-slc/events/243934568/
KSQL and Data Management with Kafka (Minneapolis) - Wednesday, October 18
https://www.meetup.com/TwinCities-Apache-Kafka/events/243858653/
Leveraging Messaging Platforms Such as Kafka for Real-Time Streaming Transaction (Atlanta) - Thursday, October 19
https://www.meetup.com/BigData-Atlanta/events/244083390/
Real-World Deployments with Apache Kafka (Raleigh) - Tuesday, October 17
https://www.meetup.com/Raleigh-Apache-Kafka-Meetup-by-Confluent/events/243894809/
Spring for Apache Kafka (Richmond) - Wednesday, October 18
https://www.meetup.com/Richmond-Java-Users-Group/events/243370292/
Exactly-Once Semantics in Apache Kafka (New York) - Monday, October 16
https://www.meetup.com/Apache-Kafka-NYC/events/243901137/
IRELAND
Let's Talk Kafka Streams (Dublin) - Thursday, October 19
https://www.meetup.com/hadoop-user-group-ireland/events/243387159/
Stream SQL and Real-Time Applications with Apache Flink (London) - Wednesday, October 18
https://www.meetup.com/Apache-Flink-London-Meetup/events/244056097/
Big Data Analytics for Small- and Medium-Size Enterprises (Oslo) - Tuesday, October 17
https://www.meetup.com/OsloBigDataDay/events/237398012/
Helsinki Apache Kafka Meetup October (Helsinki) - Wednesday, October 18
https://www.meetup.com/Helsinki-Apache-Kafka-Meetup/events/243673560/
Stream SQL and Real-Time Applications with Apache Flink (Neuilly-Sur-Seine) - Thursday, October 19
https://www.meetup.com/Paris-Fast-Data-Meetup/events/244055726/
Stream Processing and War Stories (Berlin) - Thursday, October 19
https://www.meetup.com/fast-data-berlin/events/243706029/
Apache Kafka Streams: Building Distributed, Fault-Tolerant Processing Apps (Tel Aviv-Yafo) - Wednesday, October 18
https://www.meetup.com/Apache-Kafka-TLV/events/243455001/