Troubleshooting
Problem
Python in worker has different version than in driver (WSL 1.2.3.3)
Symptom
When running a script, using spark, from the script GUI, an error occurs stating that python versions in the worker and the driver are not the same. Although the worker environment is set to Python 3.7, it seems it is running in version 2.7.
Cause
Bug in 1.2.3.3.
Environment
WSL 1.2.3.3. on Linux x86_64
Diagnosing The Problem
The simple code below produces the error: from pyspark.sql import SparkSession if __name__ == '__main__': spark = (SparkSession. builder. appName('testSpark'). getOrCreate() ) df = spark.createDataFrame([(1, "a"), (2, "b"), (3, "a")], ["id", "name"]) print(df.count()) When the same script is run from a jupyter notebok (with the command %run ../scripts/{SCRIPT_NAME}), no error occurs.
Resolving The Problem
This issue is resolved in WSL 1.2.3.3 Patch 3. Link to documentation: https://www.ibm.com/support/knowledgecenter/SSHGWL_1.2.3/local/patch03-1233.html
Document Location
Worldwide
[{"Line of Business":{"code":"LOB10","label":"Data and AI"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSHGWL","label":"IBM Watson Studio Local"},"ARM Category":[{"code":"a8m0z000000bmNbAAI","label":"Admin"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)"}]
Was this topic helpful?
Document Information
Modified date:
30 November 2020
UID
ibm16373610