26 March 2017
Lots and lots of open-source releases this week—Apache NiFi, Apache Knox, Apache Kudu, Apache Flink, and more (including a new open-source timeseries database). There are also some great technical posts on HDFS erasure encoding, Apache Phoenix, and Amazon Athena/Presto.
Sendence has written about Wallaroo, their distributed event processing framework. The team plans to open-source soon, but in the meantime this post describes what it is, the core abstractions, key features (like exactly-once processing), and future plans. Impressively, Wallaroo has median processing latencies in the microseconds and 99.99% around 1ms (their example use case is for a trading system). Currently, APIs are in C++ and Pony but support is planned for other languages too.
http://engineering.sendence.com/2017/03/hello-wallaroo/
Hortonworks has the fourth part in their "Data Lake 3.0" series. This part describes the evolution of HDFS storage—specifically the heterogenous storage system introduced in Hadoop 2.3 and the erasure coding implementation that is underway now. The post has a good description of how erasure coding is implemented, and it describes the main practical challenges (like small files and write-throughput overhead).
This post provides a brief introduction to (with examples using the Spark shell) of connecting Apache Spark to Solr.
https://developer.ibm.com/hadoop/2017/03/21/processing-solr-data-apache-spark-sql-ibm-iop-4-3/
The team at Sky Gaming and Betting has written about how they use the Confluent Schema Registry with Apache Avro and Apache Kafka to enable decentralized implementations across squads within the organization. They are using Node.js, so there's also an overview of the state of the schema registry for a Node.js client.
http://engineering.skybettingandgaming.com/2017/03/22/implementing-the-schema-registry/
The IBM Hadoop Dev blog has a look at how they've integrated Jupyter notebooks with the IBM Open Platform using Apache Knox for authentication.
https://developer.ibm.com/hadoop/2017/03/22/notebook-experience-ibm-open-platform/
The Apache Software Foundation blog has a post on the new Column Mapping and Immutable Data Encoding features of Apache Phoenix 4.10 (more below). In short, the column mapping switches Phoenix to use integers rather than strings for column names, which has a number of advantages (including both significant speedups and space savings of around 40% on a TPC-H benchmark).
https://blogs.apache.org/phoenix/entry/column-mapping-and-immutable-data
Amazon has posted performance tips for Amazon Athena (since Athena uses Presto, many of the tips are applicable outside of Athena, too). There are five tips for storing data (covering partitioning and file formats) and five tips on querying data (e.g. avoiding order by without limit and projecting columns early).
https://aws.amazon.com/blogs/big-data/top-10-performance-tuning-tips-for-amazon-athena/
Since hearing that the Strata + Hadoop World conference is being renamed Strata Data Conference, I've been curious to hear more about what the feeling was there. Datanami has some detail with a look at the "shift to real-time," the challenges due to the complexity of Hadoop, and the (perceived?) momentum due to all the companies built around Hadoop.
https://www.datanami.com/2017/03/20/hadoop-strata-not-exactly-failure-complicated/
The DBMS2 blog has a great look at the recently announced Cloudera Data Science Workbench. It adds some new details, like the fact that it's Docker-based to allow teams to install whatever software they need and that it's been beta tested by a number of big companies.
http://www.dbms2.com/2017/03/19/cloudera-data-science-workbench/
The Apache Tephra (incubating) transaction engine for Apache HBase and other distributed data stores has released version 0.11.0-incubating. The release includes a few improvements and bug fixes.
https://lists.apache.org/thread.html/95e310a1016130cee826192c2635024dbfa13aa4fa675e3f3a4d987a@%3Cannounce.apache.org%3E
http://tephra.incubator.apache.org/releases/0.11.0-incubating.html
Apache NiFi has released version 1.2.0 of the NiFi Archive bundle plugin, which can be used for class loader isolation in NiFi.
Version 0.12.0 of Apache Knox was released. There are a number of improvements and new features in the release, including improved proxy support, a YARN HA implementation of the REST API and UI, and pluggable pre-auth header provider support.
Amazon EMR has added the ability to specify "instance fleets" of up to five instance types, on which to build a cluster of mixed on-demand and spot instances.
https://aws.amazon.com/blogs/aws/new-amazon-emr-instance-fleets/
Apache Kudu 1.3.0 was released with a bunch of new featuers—Kerberos authentication, encryption in transit using TLS, coarse-grained authorization, background tasks to clean up old data, and a new crash reporter. There are also several optimizations (such as a switch to LZ4 compression) as part of the release.
http://kudu.apache.org/releases/1.3.0/docs/release_notes.html
The 1.1.5 release of Apache Flink includes fixes for high availability, fault tolerance, and Kyro serialization (among a dozen or so bug fixes).
http://flink.apache.org/news/2017/03/23/release-1.1.5.html
Apache Gora, which provides an in-memory data model for several different big data frameworks (including Avro, HBase, MongoDB, Spark, and more), has released version 0.7. The release includes over 80 issue resolutions.
Apache Phoenix 4.10 was released. This version of the SQL-on-HBase engine adds improved disk storage footprint (see separate post above), Apache Spark 2.0 integration, support for consuming data out of Apache Kafka, improved Hive integration, and more.
https://blogs.apache.org/phoenix/entry/announcing-phoenix-4-10-released
TimeScaleDB is a new, open-source time series database that's built with the Postgres engine. It's currently available in a single-node version, and there's an interesting whitepaper describing its design.
Last week, I misstated that the Microsoft announcements were made at Hadoop Summit. These were actually made at the Strata + Hadoop World conference.
Curated by Datadog ( http://www.datadog.com )
Big Data App Meetup (Palo Alto) - Wednesday, March 29
https://www.meetup.com/BigDataApps/events/237506070/
Data Science Monthly Talk: Apache Kafka (Sunnyvale) - Thursday, March 30
https://www.meetup.com/Women-Who-Code-Silicon-Valley/events/238485480/
Stream Processing With Apache Kafka and .NET (Mountain View) - Thursday, March 30
https://www.meetup.com/BayNET/events/238125420/
Robust Stream Processing with Apache Flink, with Jamie Grier (San Francisco) - Friday, March 31
https://www.meetup.com/USF-Seminar-Series-in-Analytics/events/235940713/
Fast Data With Open Source Solutions (Laurel) - Tuesday, March 28
https://www.meetup.com/Data-Science-MD/events/238057273/
Big Data All the Things (Philadelphia) - Tuesday, March 28
https://www.meetup.com/Software-as-Craft-Philadelphia/events/237732903/
Apache NiFi: Ingesting Enterprise Data at Scale (Princeton) - Tuesday, March 28
https://www.meetup.com/futureofdata-princeton/events/236924547/
Security Analytics: Securonix + Cloudera + Spark + Solr (Princeton) - Tuesday, March 28
https://www.meetup.com/nj-datascience/events/238243841/
Spark in the NHS and Cloud Object Stores (London) - Thursday, March 30
https://www.meetup.com/Spark-London/events/238613098/
How to Monitor and Optimize Spark Processes (Madrid) - Tuesday, March 28
https://www.meetup.com/Madrid-Apache-Spark-Meetup/events/237045601/
Fast Analytics on Fast Data With Apache Kudu (Madrid) - Thursday, March 30
https://www.meetup.com/Big-Data-Cloudera-Ecosystem/events/237697147/
PyData Munich March Meetup (Munich) - Tuesday, March 28
https://www.meetup.com/PyData-Munchen/events/237833175/
Hadoop Stories with Owen O'Malley (Budapest) - Thursday, March 30
https://www.meetup.com/Big-Data-Meetup-Budapest/events/238389359/
Cloud Data Analytics: Trends, Technologies, Challenges, and Opportunities (Tel Aviv-Yafo) - Tuesday, March 28
https://www.meetup.com/Cloud-BI-Tel-Aviv/events/238289829/
Real-Time Analytics with Spark Streaming by Padma Chitturi (Bangalore) - Saturday, April 1
https://www.meetup.com/Bengaluru-Insights-and-Data-Meetup/events/238459154/
Insights From Recent Strata + Hadoop World Conferences (Auckland) - Tuesday, March 28
https://www.meetup.com/NZ-Big-Data/events/238485258/