Spark job History not showing old completed Jobs
I have Alpine Chorus connected to a CDH data source and when I run Spark Jobs, I don’t see the completed jobs being shown in the Spark Job History Server. Typically when I run spark jobs from the command line, they get submitted to the Spark Job History Server for which I can refer back to later. This does not happen when i run spark jobs from within Alpine Chorus.
Is there some configuration settings that I need enable for spark jobs to get saved to the Spark Job History Server?