12 June 2016
Spark Summit was this week in San Francisco, and (as expect) this week's issue has a bunch of Apache Spark-related news, announcements, and releases. In addition to Spark coverage, this issue features articles on Kafka, Cask, Ambari, and more. Of note releases include the first release of Apache Pig in almost exactly a year, a neat new tool for distributed system design called Runway, and a new version of Apache Kudu (incubating).
Debezium is a relatively new project for capturing row-level changes to a database to an Apache Kafka topic. It currently supports MySQL, and there's a new tutorial for configuring Zookeeper, Kafka, MySQL, and more using Docker and Kubernetes.
http://debezium.io/blog/2016/05/31/Debezium-on-Kubernetes/
Some folks were surprised when the Apache Kafka project announced yet another stream processing engine, Kafka Streams. Kafka Streams has some key differentiators vs. other systems, though. This post gives a good overview of these differentiators—such as the abstractions, deployment model, and support for state-based calculations.
https://softwaremill.com/kafka-streams-how-does-it-fit-stream-landscape/
Everyone that's ever worked with MapReduce, Spark, or similar systems has run into difficult-to-debug, data-specific bugs. BigDebug is a research project/paper out of UCLA that aims to give developers access to tools found in single-machine programs: input parameters causing a crash, tracing, breakpoints, watchpoints, latency alerts, and more. The tool is available for download for Apache Spark 1.2.1.
Cask has written about running Spark inside of the open-source Cask Data Application Platform (CDAP). Spark programs running in CDAP have access to Apache Tephra (incubating) for fine-grained transaction support. This makes it easy to do a consistent copy of data from one table to another by leveraging snapshot isolation. Spark in CDAP also has access to Cask Tracker, which provides data lineage information (when it was created, used, etc). Depending on your application, the CDAP tools might add a lot of value.
http://blog.cask.co/2016/06/cdap-spark-prototype-to-production/
This tutorial from the IBM Hadoop Dev blog walks through using the Ambari REST APIs from cURL. There are examples of establishing a session both with vanilla and kerberos-enabled clusters, and reusing the session for future queries.
https://developer.ibm.com/hadoop/2016/06/07/ambari-rest-calls-for-kerberos-enabled-clusters/
The Google Cloud Platform blog has a post about debugging an Apache Beam (incubating) job running on the Google Dataflow backend. To debug the bottleneck, Dataflow has some really useful statistics and UI that allow you to dig into each of the steps.
https://cloud.google.com/blog/big-data/2016/06/understanding-timing-in-cloud-dataflow-pipelines
The Transaction Processing Performance Council (TCP) announced the TCPx-BB benchmark, which is designed for big data systems. In addition to SQL, the benchmark includes support for machine learning clustering and classification problems.
http://www.datanami.com/2016/06/01/big-data-benchmark-gauges-hadoop-platforms/
Strata + Hadoop World London was about 2 weeks ago. Videos of the keynotes and slides from many of the presentations have been posted on the conference website.
http://conferences.oreilly.com/strata/hadoop-big-data-eu/public/schedule/proceedings
Splice Machine, makers of the RDBMS built on Hadoop, have announced that they're open sourcing their software. They're currently looking for contributors/mentors/champions to help in the open sourcing effort. Splice Machine has a number of interesting features, such as ACID transactions, secondary indexing, and referential integrity.
http://www.splicemachine.com/were_going_open_source/
The Altiscale blog has compiled a number of big data use-case articles describing applications in sentiment analysis in customer service, climate change, smart cities, bias, and more. The collection also covers some articles by big data skeptics.
Spark Summit was last week in San Francisco. Conference organizer, Databricks, has posted a recap of the two day event with links to various presentations and keynotes.
https://databricks.com/blog/2016/06/08/another-record-setting-spark-summit.html
Big Data as a Service company, Qubole, has written about their customer adoption of Spark. Adoption has been fast—over half of their customers now use Spark. Qubole also supports Presto, and they've seen similar growth in adoption of that tool.
https://www.qubole.com/blog/big-data/spark-usage/
Twitter has submitted DistributedLog, their replicated log service, to the Apache Incubator.
https://wiki.apache.org/incubator/DistributedLogProposal
Big Data Day LA takes place at West Los Angeles College on July 9th. The event is free (with up-front registration) and features speakers from Confluent, Databricks, Yahoo, Netflix, and more.
Apache Spark recently released a preview of Spark 2.0. The release announcement notes that the API and functionality are not considered final.
https://spark.apache.org/news/spark-2.0.0-preview.html
JustOne has built and open-sourced a Kafka-to-PostgreSQL connector. This post introduces the connector, describes the performance, details how messages are converted to rows, describes the configuration settings, and more.
http://www.confluent.io/blog/kafka-connect-sink-for-postgresql-from-justone-database
Salesforce has open sourced Runway, which is a tool for modeling, simulating, and visualizing distributed systems. There's a live demo at runway.system with examples of "too many bananas," elevators, and the raft consensus system.
https://medium.com/salesforce-open-source/runway-intro-dc0d9578e248#.ite4u4s9y
Bloomberg recently open-sourced Presto Accumulo, a Presto connector for Apache Accumulo. In the announcement, there's a link to an 11-page paper which compares the Presto-based queries to those written with the Accumulo Java APIs and provides some benchmark results.
Microsoft Azure has announced the general availability of Apache Spark 1.6.1 for Azure HDInsight. The release contains support for the Project Livy REST job service for Spark, integration with Azure Data Lake Store (and its role-based access controls), integration with IntelliJ, support for Jupyter notebooks, and more.
https://azure.microsoft.com/en-us/blog/apache-spark-for-azure-hdinsight-now-generally-available/
LinkedIn has open source Photon ML, their library for large scale regression. Photon is built on Spark and is run on YARN at LinkedIn (it used to run on MapReduce and has seen major speed improvements since migrating).
https://engineering.linkedin.com/blog/2016/06/open-sourcing-photon-ml
Hortonworks has announced a technical preview of the Spark-HBase connector which they developed in conjunction with Bloomberg. The connector features native Avro support, support for running in a secure cluster, native Spark Datasource APIs, and optimizations such as partition pruning, column pruning, and predicate pushdown.
http://hortonworks.com/blog/spark-hbase-dataframe-based-hbase-connector/
Databricks has announced the first phase of security features for their Apache Spark platform. This phase adds cluster ACLs, SAML 2.0 support, and end-to-end audit logs.
Apache ORC 1.1.0 was released this. This release completed the migration of Java code out of the Apache Hive code base, fixes the C++ reader's handling of timestamps, and adds connectors for Hadoop MapReduce.
http://orc.apache.org/news/2016/06/10/ORC-1.1.0/
Version 0.9.0 of Apache Kudu has been released. Major features of the release are an UPSERT command, a new Spark data source that doesn't rely on the MapReduce APIs, and improvements to Tablet Server write performance.
http://getkudu.io/2016/06/10/apache-kudu-0-9-0-released.html
The Google Cloud Platform team has announced support for the Spark 2.0 preview release with Google Cloud Dataproc.
Dory (the successor to Bruce), which is a Kafka producer daemon with support for ingesting data via UNIX domain sockets or local TCP, was announced.
Apache Pig 0.16.0, the first release in over a year, was released. This version stabilizes Pig on Tez.
http://pig.apache.org/releases.html#8+June%2C+2016%3A+release+0.16.0+available
Curated by Datadog ( http://www.datadog.com )
June 2016 Meetup (San Francisco) - Tuesday, June 14
http://www.meetup.com/hadoopsf/events/230691040/
Airflow Meetup (Redwood City) - Tuesday, June 14
http://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/230236598/
Stream Processing Meetup (Mountain View) - Wednesday, June 15
http://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/231454378/
Big Data Application Meetup (Palo Alto) - Wednesday, June 15
http://www.meetup.com/BigDataApps/events/229659115/
Spark 2.0 Highlights from Spark Summit SF 2016 (Seattle) - Wednesday, June 15
http://www.meetup.com/Seattle-Spark-Meetup/events/228768247/
Taking Spark to the Clouds (Bellevue) - Thursday, June 16
http://www.meetup.com/CloudTalk/events/231204317/
Spark Hands-on 1-Day Workshop for Data Engineers, Data Scientists and Developers (Houston) - Tuesday, June 14
http://www.meetup.com/Houston-Hadoop-Meetup-Group/events/231577283/
What Is All the Hype about Apache Spark? (Houston) - Tuesday, June 14
http://www.meetup.com/Houston-Hadoop-Meetup-Group/events/228768879/
Spark 2.0 Performance Improvements & Blazegraph GPU (Laurel) - Monday, June 13
http://www.meetup.com/Apache-Spark-Maryland/events/231625107/
De-Siloing Data with Apache Drill (Baltimore) - Tuesday, June 14
http://www.meetup.com/Data-Science-MD/events/231032325/
Introduction to Spark In-Memory Computing (Philadelphia) - Tuesday, June 14
http://www.meetup.com/Big-Data-Developers-in-Philadelphia/events/231046411/
Data Driven NYC #48 (New York) - Monday, June 13
http://www.meetup.com/DataDrivenNYC/events/231348559/
Introducing Scio, a Scala API for Google Cloud Dataflow (New York) - Thursday, June 16
http://www.meetup.com/Hadoop-NYC/events/231552672/
Mike Olson, Co-Founder of Cloudera, to Present at TDWI Boston Chapter (Waltham) - Thursday, June 16
http://www.meetup.com/BigDataBoston/events/231631889/
Spark Streaming: Dealing with State (Vancouver) - Thursday, June 16
http://www.meetup.com/Vancouver-Spark/events/231605454/
IRELAND
Production Quality Data Science: Building Rapid Ingestion Data Pipelines (Dublin) - Monday, June 13
http://www.meetup.com/hadoop-user-group-ireland/events/231165491/
All about Apache Spark (Lisbon) - Thursday, June 16
http://www.meetup.com/Big-Data-Developers-in-Lisbon/events/231199476/
Apache Kafka: A High-Throughput Distributed Messaging System, by Samuel Kerrien (Geneva) - Tuesday, June 14
http://www.meetup.com/Big-Data-Romandie/events/231182818/
Spark Meetup: GraphX and DataFrames (Vienna) - Wednesday, June 15
http://www.meetup.com/Vienna-Spark-Meetup/events/230220798/
Recommendation Systems: From Scratch to a Working Spark Implementation (Tel Aviv-Yafo) - Monday, June 13
http://www.meetup.com/Big-Data-Israel/events/230511538/
Lessons Learned While Running Kafka at Scale, by Gwen Shapira (Tel Aviv) - Tuesday, June 14
http://www.meetup.com/Big-things-are-happening-here/events/231119208/
Spark Meetup (Shanghai) - Saturday, June 18
http://www.meetup.com/Shanghai-Apache-Spark-Meetup/events/231612662/