modelrail.otenko Trains, computers, vintage, retro, model railways… the lot…

Apache Spark 2.4 EOL

EOL = End of Life. It's dead, Jim. No longer supported. If you get this error in Azure Synapse, then your Apache Spark Pool worker is trying to load v2.4 (INSERT_VERSON_HERE, really... as I'm sure this'll happen in the future as the window rolls) which has been deleted.

RUNTIME_CONFIGURATION_ERROR_VHDOVERRIDENOTFOUND: Livy session has failed. Session state: Error. Error code: RUNTIME_CONFIGURATION_ERROR_VHDOVERRIDENOTFOUND. Cannot find vhd based on override: Message=The system did not find a VHD copy in the system for creating the ClusterName=12341223-5555-4444-3333-63c446fe4d8b, Workspace=synws-d-e-v-d-e-v-01, Subscription=, Location=australiaeast, isVhdOverride=True Source: Dependency.

I'm putting this on the web so others can googl' for it. I had no results and it burnt 1/4 of my weekend. Re-deploy your pools with:

Update-AzSynapseSparkPool -WorkspaceName ws_name -Name pool_name -SparkVersion 3.4

And... cross your fingers. Of course, if you've tried to delete a pool with notebooks attached, then you'll have ERROR, POOL IN DELETED_FAILED STATE and you'll need to create a new temp pool, associate ALL notebooks, delete the old and then rename the new... or create another with the old name and then switch them all back and delete the new... or something. It's a monday problem.