Tuning Spark Streaming Applications

I wrote a blog post for Smaato on how to approach stabilizing and tuning a Spark Streaming application:

  • Start out creating a stable stream processing application before focusing on throughput.
  • You might have to make your app slower at first, then keep scaling by parallelizing processing.
  • Perform this iterative process continuously.
Stephan avatar
About Stephan
Senior Dev at Smaato Inc. - Adtech, Data Engineering, Stream Processing with Scala, Java, Python, Go, Spark & Kafka - Automation Fanatic - Father of 2
comments powered by Disqus