Creating Component Tests for Spark Applications

One of the main engineering challenges faced by the Data Team is creating robust tests for our Spark applications. Since these applications are constantly evolving, as for any application, we needed a way to ensure changes wouldn’t break the code; a guarantee that the output from our