ApacheCon North America 2014

Apache Pig is a programming language and execution runtime for doing petabyte scale processing with MapReduce. One of the major recent developments in the Hadoop ecosystem is the introduction of Apache Tez, a successor to MapReduce which provides major performance enhancements and a more natural foundation for Pig. The Pig-on-Tez project aims to dramatically increase the throughput of data pipelines written in Pig by using Apache Tez as the execution engine instead of MapReduce. By running atop the Tez framework, benchmarks of representative queries have sped up 2-3x when compared to MapReduce.

In the second half of this presentation we’ll explain how LinkedIn, Netflix, Hortonworks, and Yahoo have successfully collaborated over a 6 month period to deliver a major rewrite of critical infrastructure, providing significant benefits for both themselves as well as the community at large.