20 November 2016
Thanksgiving is this week in the US, so I've collected quite a few technical and news articles to read if you have downtime. Hortonworks also announced their "HDCloud" offering for AWS this week, and there's a great presentation detailing the work that was done to make Hadoop work better with S3.
The Paypal engineering blog has a two-part post describing their adoption of Apache Kafka and Apache Spark. The first part describes how they got from their old architecture to a new, Kafka-based one. The second part describes their lambda architecture based on Spark streaming on YARN.
https://www.paypal-engineering.com/2016/11/08/from-big-data-to-fast-data-in-four-weeks-or-how-reactive-programming-is-changing-the-world-part-1/
https://www.paypal-engineering.com/2016/11/18/from-big-data-to-fast-data-in-four-weeks-or-how-reactive-programming-is-changing-the-world-part-2/
This post is a follow-up to recent benchmarks that Yahoo and others performed to compare throughput and latency of several stream processing tools. In it, the DataTorrent team compares Apache Flink and Apache Apex. It shows Apex to have lower latency than Flink, but the main takeaway is that we need better benchmarks—there are many more use cases to take into consideration, and it seems possible to push these systems even harder than we have in previous tests.
https://www.datatorrent.com/blog/throughput-latency-and-yahoo/
This presentation from ApacheCon describes the work that was done towards the goal of making the Hortonworks Data Platform faster on AWS (with data in the S3 object store) than Amazon EMR. The slides include an introduction to object store semantics, some of the improvements that were made to make queries to S3 faster, benchmarks showing a 2.5x speedup over a version without the enhancements, future plans for s3guard to provide consistent metadata access of data in S3, and more.
http://www.slideshare.net/steve_l/hadoop-hive-spark-and-object-stores
Databricks has a post describing work to win the CloudSort benchmark, which is a measure of the cost required to sort 100TB of data. The post has an overview of the evolution of sorting benchmarks over the last several years (from DataMation Sort to TeraSort to GraySort) and a description of the record setting setup.
https://databricks.com/blog/2016/11/14/setting-new-world-record-apache-spark.html
MapR is hosting a free (behind an email-wall) eBook version of "Introduction to Apache Flink" by Ellen Friedman (MapR) and Kostas Tzoumas (data Artisans). The book has six chapters covering streaming, stateful computation, batch and more.
https://www.mapr.com/ebooks/intro-to-apache-flink/
This post provides an overview (by way of example) of 30 command-line tools from Apache Hadoop HDFS.
http://linoxide.com/file-system/hadoop-hdfs-shell-commands/
In addition to Spark Streaming, Paypal is using Apache Storm for real-time computation. This post describes how the carrier payments team uses Storm to read data from Kafka and write it out to Apache Hive.
https://www.paypal-engineering.com/2016/11/15/carrier-payments-big-data-pipeline-using-apache-storm/
Amazon Kinesis, which is often compared to Apache Kafka, has strict SLAs per shard (at most 1000 records or 1MB per second). In the case where this is exceeded, Amazon now offers a UpdateShardCount API call. This post describes how to trigger an update from CloudWatch using a Lambda function.
This tutorial, aimed a data analysts that don't yet have exposure to HDFS, describes how to use Apache Sqoop to export data from a PostgreSQL database to HDFS.
https://www.toptal.com/database/hdfs-tutorial-data-migration-from-postgresql
The Databricks blog has a tutorial that shows how to use Amazon Kinesis, Amazon Redshift, and Databricks (Spark) to model asset levels. Among other things, the post demonstrates Spark's linear regression library to fit a model to the sample data.
https://databricks.com/blog/2016/11/16/oil-gas-asset-optimization-aws-kinesis-rds-databricks.html
Amazon EMR now supports auto scaling to dynamically grow or shrink a cluster based on Cloudwatch metrics. This post has more details about how it works and how to setup auto scaling rules.
This post, which summarizes a recent paper "A Critique of the CAP Theorem" by Martin Kleppmann, provides an excellent introduction to the CAP theorem and linearizability. There are also several links for further reading.
http://jvns.ca/blog/2016/11/19/a-critique-of-the-cap-theorem/
This presentation from the recent Apache Kafka Bay Area meetup has a good overview of Kafka Streams. In particular, there are some good slides on KTable and the new interactive query support recently added to Kafka.
http://www.slideshare.net/MatthiasJSax/stream-application-development-with-apache-kafka
At ApacheCon this week, the ODPi announced version 2.0 of the runtime specification (based on Apache Hadoop 2.7) and the first version of the operations specification. For those that aren't familiar, the ODPi aims to define specifications to provide compatibility and compliance guidelines (see the press release for more details) for Hadoop distributions.
MapR has a multi-part blog series with an overview and excerpts from "BI & Analytics on a Data Lake" by Sameer Nori and Jim Scott. Part two covers the differences between data warehouses, MPP data warehouses, and Hadoop.
https://www.mapr.com/blog/bi-analytics-and-big-data-%E2%80%9C-ha%E2%80%9D-moment-part-1
https://www.mapr.com/blog/exploring-relationship-between-hadoop-and-data-warehouse-part-2
This post has a good introduction to AtScale, which is a middleware that sits between BI tools and Hadoop. This week they announced support for Teradata DW, Google Dataproc, and BigQuery.
http://www.zdnet.com/article/sql-everywhere-and-atscale/
Qubole announced this week that you can now get their Big Data-as-a-Service system through AWS Marketplace SaaS subscriptions. This makes their service available to folks with an AWS account without requiring a separate bill from Qubole.
CIOReview has a post by Hortonworks Founder and VP of Engineering Arun C Murthy. In it, he talks about how open-source is changing the governance and security model, machine/deep learning, and cloud. He also has some advice for a company working with open-source.
http://opensource.cioreview.com/cxoinsight/striving-for-innovation-nid-23386-cid-92.html
In this post, the author argues that Google BigQuery and other cloud services will do to big data what Gmail did to email (i.e. "unlimited" storage and a very easy to use system).
Apache Spark 2.0.2, a maintenance fix for the 2.0 branch, was released this week. Notably, support for Apache Kafka 0.10 was added to structured streaming.
http://spark.apache.org/releases/spark-release-2-0-2.html
The Hortonworks Data Cloud is now available on AWS marketplace.
http://hortonworks.com/blog/announcing-hortonworks-data-cloud-for-aws/
Curated by Datadog ( http://www.datadog.com )
Two Talks: DSE Graph & Event Streaming with Kinesis (New York) - Tuesday, November 22
https://www.meetup.com/Reactive-New-York/events/235488288/
Integrating Real-Time Data Streams with Spark and Kafka (Montreal) - Tuesday, November 22
https://www.meetup.com/Big-Data-Montreal/events/235658938/
Hadoop, Big Data, and Watson (Buenos Aires) - Monday, November 21
https://www.meetup.com/Buenos-Aires-Artificial-Intelligence/events/231357183/
Spark in Production at IBM (London) - Wednesday, November 23
https://www.meetup.com/Spark-London/events/235626954/
Hands on Hadoop (Oslo) - Wednesday, November 23
https://www.meetup.com/DND-Business-intelligence-Information-Management-Oslo/events/235440096/
Stockholm Erlang Meetup (Stockholm) - Wednesday, November 23
https://www.meetup.com/Erlang-User-Group-Stockholm/events/235411092/
Big Data and Docker (Malmo) - Thursday, November 24
https://www.meetup.com/Javaforum-Malmo/events/234911104/
WebTechNight Karlsruhe (Karlsruhe) - Tuesday, November 22
https://www.meetup.com/WebTechNight-Karlsruhe/events/235439539/
HBase and Elasticsearch at Socialbakers (Prague) - Thursday, November 24
https://www.meetup.com/CS-HUG/events/235437943/
Big Data Ecosystem and Apache Spark Architecture (Ankara) - Thursday, November 24
https://www.meetup.com/Ankara-Spark-Meetup/events/235350970/
Elastic Is in Town Again! (Tel Aviv-Yafo) - Thursday, November 24
https://www.meetup.com/Tel-Aviv-Elastic-Fantastics/events/235263776/
Spark Meetup November (Sydney) - Tuesday, November 22
https://www.meetup.com/Sydney-Apache-Spark-User-Group/events/234234411/
Rethink SQL for Big Data with Apache Drill (Pyrmont) - Thursday, November 24
https://www.meetup.com/Sydney-Big-Data-Converged-SQL-NoSQL-and-Real-Time/events/233462437/