Tuning Spark Streaming Applications

I wrote a blog post for Smaato on how to approach stabilizing and tuning a Spark Streaming application:

  • Start out creating a stable stream processing application before focusing on throughput.
  • You might have to make your app slower at first, then keep scaling by parallelizing processing.
  • Perform this iterative process continuously.
Stephan avatar
About Stephan
Developing software since before the web existed - data engineering - devops - getting diverse teams of people to create high quality software.