31 July 2016
There were lots of milestones achieved this week as Apache Kudu and Apache Twill both graduated from the Apache incubator, and Apache Spark and Apache Mesos both had major releases (2.0 and 1.0 respectively). In technical posts, there's an overview of Uber's tech stack, a comparison of Apache Kafka and Google Cloud Pub/Sub, a guide for running Apache Kafka in AWS, and more.
Uber has published a detailed overview of their technology stack. The article has several sections on infrastructure (e.g. Kafka, ELK stack, Mesos) and data systems (Cassandra and Riak complement MySQL for online systems, Storm is used for real-time processing, and Spark, Hive, & Hadoop are used for batch).
https://eng.uber.com/tech-stack-part-one/
Google Cloud Pub/Sub is a large-scale messaging service that's part of Google Cloud Platform. This post provides a quick comparison of Pub/Sub and Kafka across the topics of operations (e.g. data retention, backups), application development (e.g. APIs, consumption model), cost, and architecture (e.g. ordering guarantees).
http://www.jesse-anderson.com/2016/07/apache-kafka-and-google-cloud-pubsub/
MapR has two whiteboard walkthroughs about streaming systems and micro services. The series describes why a high-performance message queue that is omni-present and easy to use is important for decoupling micro services. With a system like this (Kafka or MapR Streams), it's possible to decouple the backend data systems to power stand-alone services.
https://www.mapr.com/blog/key-requirements-streaming-platforms-micro-services-advantage-whiteboard-walkthrough-part-1
https://www.mapr.com/blog/streaming-data-how-move-state-flow-whiteboard-walkthrough-part-2
There's been a lot of innovation in open-source stream processing frameworks over the past year or so. Apache Spark 2.0 (more about the release below) introduces an alpha implementation of Structured Streaming, which aims to jockey Spark to the forefront. Structured Streaming adds new features such as event-time processing and a simplified API that aligns with DataFrames. The Databricks blog has two posts about the new framework—the first motivates Structured Streaming and gives an overview of its architectural guarantees. The second describes the API and data model in more depth.
https://databricks.com/blog/2016/07/28/continuous-applications-evolving-streaming-in-apache-spark-2-0.html
https://databricks.com/blog/2016/07/28/structured-streaming-in-apache-spark.html
Confluent has posted an overview of running Apache Kafka in AWS. As someone who searched long and hard for similar advice in the past, this guide is very welcome advice. It covers the major decisions across storage, instance types, network configuration, availability zones, and more.
http://www.confluent.io/blog/design-and-deployment-considerations-for-deploying-apache-kafka-on-aws
The AWS Big Data blog has a tutorial for configuring Amazon EMR with the Spark JobServer, which allows for REST-style Spark job submission and monitoring.
The Cloudera blog has a tutorial that demonstrates email archival to HDFS. The article describes building a pipeline with Apache James to act as a email journal recipient that writes mail out to disk, Apache Flume for sending files to Apache Kafka, and Apache Spark for combining and archiving files to HDFS.
Netflix has written about their use of Apache Mesos for services and data processing. Their Mesos cluster uses Mantis for stream processing, Titus for Docker, and Meson for workflow orchestration and scheduling.
http://techblog.netflix.com/2016/07/distributed-resource-scheduling-with.html
Two Hadoop-ecosystem projects graduated from the Apache Incubator this week. Apache Kudu, the data storage system optimized for both scans and random access, is close to a 1.0 release. Apache Twill, which is an abstraction layer for Apache Hadoop YARN, released version 0.7.0-incubating in January.
https://blogs.apache.org/foundation/entry/apache_software_foundation_announces_apache
https://blogs.apache.org/foundation/entry/apache_software_foundation_announces_apache1
Apache Spark announced the milestone 2.0.0 release this week. There are a number of changes in the release—highlights include unification of the DataFrame and Dataset APIs, support for SQL2003, substantial speedups, and Structured Streaming (alpha).
http://spark.apache.org/releases/spark-release-2-0-0.html
Apache Mesos also hit a major milestone with the project's 1.0 release. Highlights of the release include a new HTTP API, support for docker containers without the docker daemon, CPU support, and fine-grained authorization.
https://blogs.apache.org/foundation/entry/the_apache_software_foundation_announces97
Apache Kylin 1.5.3 was release this week. Kylin is an OLAP system for big data. This release adds support for Amazon EMR, improves support for MapReduce job queues, and fixes a number of bugs.
Google Cloud Dataflow has announced beta support for Python via version 0.4.0 of the Cloud Dataflow SDK for Python.
https://cloud.google.com/blog/big-data/2016/07/support-for-python-on-cloud-dataflow-is-going-beta
Curated by Datadog ( http://www.datadog.com )
Building a Real-Time Streaming Platform Using Kafka Streams and Kafka Connect (Mountain View) - Wednesday, August 3
http://www.meetup.com/BOLD-Bay-area-Organization-of-Ladies-in-big-Data/events/230825020/
Apache Spark Streaming with Apache NiFi and Apache Kafka (Santa Clara) - Wednesday, August 3
http://www.meetup.com/futureofdata-siliconvalley/events/230910905/
Better Together: Fast Data with Apache Spark and Apache Ignite (San Francisco) - Thursday, August 4
http://www.meetup.com/SF-Spark-and-Friends/events/232407316/
Technical Workshop on Spark SQL (Palo Alto) - Friday, August 5
http://www.meetup.com/Women-in-Big-Data-Meetup/events/232718819/
Building a Data Pipeline for Ad Tech Using Spark, Impala, and Zoomdata (Tempe) - Wednesday, August 3
http://www.meetup.com/Phoenix-Hadoop-User-Group/events/228165806/
Apache Zeppelin: Meet the Creator (Addison) - Monday, August 1
http://www.meetup.com/DFW-Data-Science/events/231891250/
Hadoop in High-Tech Manufacturing (Saint Paul) - Thursday, August 4
http://www.meetup.com/Twin-Cities-Hadoop-User-Group/events/232562390/
August Orlando Devs Meetup (Orlando) - Thursday, August 4
http://www.meetup.com/OrlandoDevs/events/232617857/
Solving the Big Data Security Puzzle (Atlanta) - Wednesday, August 3
http://www.meetup.com/Atlanta-Hadoop-Users-Group/events/230344617/
A View on the World of Streaming: Use Cases, Challenges and Solutions (Munich) - Wednesday, August 3
http://www.meetup.com/Apache-Flink-Meetup-Munich/events/232566197/
Scio: Scala API for Apache Beam (Warsaw) - Wednesday, August 3
http://www.meetup.com/warsaw-hug/events/232871549/
Apache Spark Meetup (Auckland) - Tuesday, August 2
http://www.meetup.com/Auckland-Apache-Spark-User-Group/events/231701714/