Analyzing and Searching Streams of Social Media at Scale using Spark, Kafka and Elasticsearch

06/01/2015 - 17:00 to 17:30
Stage 4 / Open Stage
Short talk (30 min)

Session abstract: 

In this session we like to share our experiences from analyzing streams of Twitter data with Apache Spark Streaming in near real-time, leveraging Apache Kafka as a HA messaging backbone plus storing and searching for Tweets in Elasticsearch at a large scale. Key design aspects are short end-end processing delays, sub-second search responses and a highly available system that does not rely on hardware redundancy.


Corporate-Design: Extragestaltung, Margarethe Hausstätter
Ilustration: cyan, Berlin