Tuning Spark Streaming Applications

I wrote a blog post for Smaato on how to approach stabilizing and tuning a Spark Streaming application:

  • Start out creating a stable stream processing application before focusing on throughput.
  • You might have to make your app slower at first, then keep scaling by parallelizing processing.
  • Perform this iterative process continuously.