18 February 2019
A bunch of posts this week on Spark and Kafka, and also some really interesting posts on speeding up database testing, AWS Glue for batch processing, defining scalability, orchestration vs. choreographing a data pipeline. There are also several must reads in news—on SQL, the best talks of 2018, and the evolution of data science.
With docker and related tools, it's quite easy to run integration tests that talk to a database. This post describes how to speed up those tests by starting up your database with fsync disabled or using a ramdisk. If you're running your DB as a docker container, it's pretty easy to do either one (see examples in the post).
https://pythonspeed.com/articles/faster-db-tests/
Smile is a binary format that is based on JSON, but is much more efficient. This post provides a quick introduction to the format, some example serialization sizes, and a Java code example.
https://medium.com/code-with-ayush/understanding-smile-a-data-format-based-on-json-29972a37d376
Term-Frequency, Inverse Document Frequency is the classic algorithm for searching for terms in a document. It was historically used in a lot of tools, but Elasticsearch and Solr both offer a lot of new strategies. Of those, this post covers several: Best Matching, Divergence from Randomness, Divergence from Independence, Information-Based, and Language Models.
https://sematext.com/blog/search-relevance-solr-elasticsearch-similarity/
Qubole writes about their dynamic filtering feature for their distribution of Spark, which improves performance of join queries by (semantically) adding additional filter predicates to the query to reduce the amount of data in the join.
https://www.qubole.com/blog/enhance-spark-performance-with-dynamic-filtering/
A two-part comparison of implementing common algorithms (finding primes and matrix multiplication) using MPI (a two decade old HPC framework) and Apache Spark. MPI wins in performance, and Spark wins in ease of algorithm implementation.
https://medium.com/@anicolaspp/mpi-workloads-performance-on-mapr-data-platform-part-1-c801f3d5fe08
This post covers the redBus Data Platform, which uses AWS Glue for coordinating batch processing & crawling datasets in S3 and Apache Drill for querying data.
https://medium.com/@dweepks/data-lake-formation-with-aws-glue-apache-drill-676133e816bc
Lots of details of the JDBC connector for Apache Kafka Connect. The article covers things like incremental ingestion, debugging common issues, custom data types, and importing from multiple tables.
https://www.confluent.io/blog/kafka-connect-jdbc-source-connector-deep-dive
This Flink blog post describes the improvements that are being merged into the Flink codebase to improve performance and features for batch processing. By taking advantage of some tradeoffs of batch processing, they can get better performance, fault tolerance semantics, scheduling, and SQL support. The post has lots more details about these changes.
https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html
Scalability is sometimes a hand-wavy term that means different things to different people. This article provides a good overview of the types of scalability (size, geographical, and administrative), and it provides several easy to follow examples.
https://medium.com/baseds/scalability-growing-a-system-in-different-directions-ae16469c4cb3
An interesting take on orchestration vs. choreography for a data pipeline, and how the tradeoffs are similar to those of a monolith vs. a microservices architecture.
https://medium.com/@imamitsehgal/big-data-pipeline-orchestration-or-choreography-325715ce4a9
This post describes the fundamentals of an event streaming architecture, including how it compares to reactive event-driven architectures (e.g. actor frameworks), how it relates to microservices, and more. This article is post 2 of a 4 part series.
A look at how one company uses the ELK stack to analyze usage of big data systems (both ad hoc and scheduled) to identify resource intensive queries.
https://medium.com/@adirmashiach/defend-your-infrastructure-handling-3-000-hungry-users-379783b24d51
ThousandEyes writes about some lessons learned working with Apache Kafka. Specifically, they cover scenarios in which Kafka topics experienced spiky disk usage and older than expected data was showing up in consumers (based on the delete or compact,delete policy set for a particular topic). They describe each of these in detail and suggest some configuration changes/lessons learned.
https://blog.thousandeyes.com/kafka-topics-pitfalls-and-insights/
State Synchronizer is component of Pravega, the streaming storage system, that provides a shared object to Pravega clients. This post describes how it compares to systems like Apache ZooKeeper, how it's used in Pravega to implement ReaderGroups, and the API/semantics it exposes to clients.
http://blog.pravega.io/2019/02/15/exploring-state-synchronizer/
A look at how Apache Pulsar implements tiered storage, which can provide significant cost savings by leveraging an object store.
https://streaml.io/blog/saving-money-with-apache-pulsar-tiered-storage
For Apache Spark applications, it's often important to understand and optimize memory usage. This post looks at the main classes used by the Spark executors to dynamically allocate memory, which are important parts of understanding the overall memory model.
https://www.waitingforcode.com/apache-spark/memory-apache-spark-classes/read
A curated list of great talks (videos and/or slides) from 2018. Lots of topics covered, and there are many on distributed systems, networking, and other relevant topics.
https://medium.com/@copyconstruct/best-of-2018-in-tech-talks-2970eb3097af
ThoughtWorks describes how data lakes are susceptible to a "build it and they will come" mentality, and they recommend instead using a bottom-up, product/use-case driven approach.
https://www.thoughtworks.com/insights/blog/curse-data-lake-monster
An argument that SQL is the the most valuable skill that a developer can have for a few reasons, like how it's valuable across many different roles and disciplines.
http://www.craigkerstiens.com/2019/02/12/sql-most-valuable-skill/
Lots of great advice for folks considering a career in data science—how the job has evolved, and how a lot the skills needed to be successful as a data scientist involve data engineering and software engineering. Great recommendations to share with anyone you might know considering a job in data.
https://veekaybee.github.io/2019/02/13/data-science-is-different/
Streams Messaging Manager 1.2 was released with topic lifecycle management features, alerting, and schema registry integration.
https://blog.cloudera.com/blog/2019/02/new-kafka-alert-management-features-in-smm-12/
Apache HBase 2.1.3 was released. It includes a number of bug fixes and improvements, most notably an upgrade to the Apache Thrift dependency to resolve a security vulnerability.
Apache Beam 2.10.0 has been announced. The release includes updates to lots of dependencies (including Apache Flink 1.6), new IOs (for Kafka, Hadoop, Mongo, Cassandra, and more). The Beam blog has more details on the these features.
https://beam.apache.org/blog/2019/02/15/beam-2.10.0.html
PostgreSQL has announced a number of new releases for the 9, 10, and 11 lines. The releases are notable in that PostgreSQL has changed its implementation of fsync() for the first time in many years to resolve potential data consistency issues.
https://www.postgresql.org/about/news/1920/
kafka-topic-manager is a new open source project providing a REST service to delete topics from a Kafka cluster. It helps to work around a bug in version 1.1.1 of Kafka by serializing the deletion of topics.
https://github.com/wushujames/kafka-topic-manager
Curated by Datadog ( http://www.datadog.com )
Kafka Is More ACID Than Your Database (Hollywood) - Wednesday, February 20
https://www.meetup.com/Los-Angeles-Apache-Kafka-Meetup/events/258680451/
Why Data-as-a-Service Is the Next-Generation Data Platform (Bellevue) - Wednesday, February 20
https://www.meetup.com/Big-Data-Bellevue-BDB/events/257879431/
Apache Flink with Hive, Tensorflow, Beam, and AthenaX (Seattle) - Thursday, February 21
https://www.meetup.com/seattle-apache-flink/events/258723322/
Big Data Transformation: Moving from Hadoop and Data Streaming to Micro-Batch (Boulder) - Thursday, February 21
https://www.meetup.com/Boulder-Denver-Big-Data/events/258528322/
Modern Data Warehouses in the Cloud: Use Cases + Live Demo (Plano) - Tuesday, February 19
https://www.meetup.com/DFW-Advanced-Analytics/events/258932811/
Event-Driven Architecture w/ Apache Kafka and Spring Cloud Stream (Indianapolis) - Wednesday, February 20
https://www.meetup.com/Indianapolis-Java-User-Group/events/258217954/
Big Data Architectures and the Data Lake (Linthicum Heights) - Wednesday, February 20
https://www.meetup.com/B-BIG-Baltimore-Business-Intelligence-Group/events/255715545/
Streaming Data Pipeline with Kafka and Druid (Boston) - Tuesday, February 19
https://www.meetup.com/Apache-Druid-Boston-Area-Meetup/events/258396474/
IRELAND
Spark and Kafka for Data Analysis and Machine Learning (Cork) - Tuesday, February 19
https://www.meetup.com/corkjug/events/257924534/
Apache Kafka and KSQL in Action! (Bristol) - Tuesday, February 19
https://www.meetup.com/BigDataBristol/events/257400740/
Updates on Stateful Stream Processing with Apache Flink 1.7+ (London) - Friday, February 22
https://www.meetup.com/Apache-Flink-London-Meetup/events/258727544/
Profiling and Caching Spark Applications with Qubole OSS (Barcelona) - Tuesday, February 19
https://www.meetup.com/Spark-Barcelona/events/258760166/
A Tour of the Kafka Environnement (Montpellier) - Wednesday, February 20
https://www.meetup.com/JUG-Montpellier/events/258321867/
Go Meets Messaging Systems (Sofia) - Tuesday, February 19
https://www.meetup.com/Uber-Engineering-Events-Sofia/events/258559973/
HONG KONG
Productionizing Apache Spark for ETL (Wan Chai) - Wednesday, February 20
https://www.meetup.com/HKSpark/events/258916721/
Apache Druid & Big Data Journey at Uber (Taguig) - Wednesday, February 20
https://www.meetup.com/Manila-BIG-DATA-Group/events/258727025/