How specify the version of python that spark-submit uses

Hi,

I have installed python 3.7 and want to use with spark.

How specify the version of python that spark-submit uses?

Thanks

View Answers

June 25, 2018 at 7:45 AM

Hi,

You have to modify the value in conf/spark-env.sh file.

This will instruction to use the spark to use the python as mentioned in the configuration.

Open the file: conf/spark-env.sh

and then add following into it:

export PYSPARK_PYTHON=/opt/pathtopyton/python

You have to give complete path of python exectutable

This will solve problem.

Thanks

Advertisements

Add jars to a Spark Job spark-submit
Add jars to a Spark Job spark-submit Add jars to a Spark Job spark-submit I want to submit some dependency jars along with the code jar file with spark-submit. How to add jar to the Spark Job Submit process? Thanks

specify colors
specify colors How can I specify colors

ModuleNotFoundError: No module named 'specify'
ModuleNotFoundError: No module named 'specify' Hi, My Python... 'specify' How to remove the ModuleNotFoundError: No module named 'specify... to install padas library. You can install specify python with following command

kibana specify elasticsearch url
kibana specify elasticsearch url Hi, I have to connect... needs to be done in kibana for this? How we can modify kibana configuration to specify elasticsearch url? Thanks Hi, Kibana is using

how to call the parent page from child without refreshing the parent and my parent window uses the master page
how to call the parent page from child without refreshing the parent and my parent window uses the master page how to call the parent page from child without refreshing the parent and my parent window uses the master page

Uses of XML
Uses of XML In this section we are discussing the importance of XML and see the uses of XML. The full form of XML is Extensible Markup Language. XML... and vice-versa. Uses of XML Here are uses of XML in web developed and enterprise

Specify the drag indicator by using the DragManager
Specify the drag indicator by using the DragManager : In this example we must specify a size for the drag indicator. In this example we use a doDrag... and drop the image it resize. Example: <?xml version="1.0"

Uses of javascript, ActiveX in XML
Uses of javascript, ActiveX in XML hi.. Can I use JavaScript, ActiveX, etc in XML files? please anyone tell me ADS_TO_REPLACE_1 Thanks

how to grayscale an image in python
how to grayscale an image in python Hi, I have may images and it want to greyscale all with python. can it be done? how to grayscale an image in python? Thanks Hi, You can do this with opencv. here is code

How is Python used in AI?
How is Python used in AI? Hi, I am beginner in Data Science and machine learning field. I am searching for the tutorials to learn: How is Python... learn the topic "How is Python used in AI?". Also tell me which

how to create array in python
how to create array in python Hi, how to create array in python? I want to create an array in Python and add some values to it. Thanks Hi, You can create a new array with the code myArray = [] and after

Big Data and its uses
Big Data and its uses Hi, I am trying to understand the Big Data and its uses. Please tell me: What is Big Data? What is the uses of Big Data... are the uses of Big Data? Big Data provides tremendous opportunity to the business

Java application that uses the classes and
Java application that uses the classes and i have this class diagram with three classes: Curriculum, Course, and Lecture. Your class diagram...) I wish Write a Java application that uses the classes and methods above

Uses of GPS Tracking System
of operation. Here we explain how to uses of GPS Tracking system that includes... GPS Tracking System has many uses in respect to the security and information of vehicles and other devices, which can be traced using GPS system

How to install botocore library in Python
How to install botocore library in Python Hi, How to install botocore library in Python? What is the use of botocore library in Python? Thanks

How to use list in for loop in Python
How to use list in for loop in Python Hi, I have list in Python and I want to list down all the items using for loop. How to use list in for loop in Python? Thanks

How to install the Six module in Python
distribution. How to install the Six module in Python? Thanks...How to install the Six module in Python Hi, I want to install the six module in Python as my program is using this. I have checked it by importing

how to convert yyyyMMdd to date in python
how to convert yyyyMMdd to date in python Hi, I have a date string 20221231 and I want to convert it to the date in Python. how to convert... is the similar example: How to get yesterday date in Python? Thanks

How to run external command in Python?
How to run external command in Python? Hi, I am writing an application in Python and there is requirement of calling the external command from... external command. How to run external command in Python? Thanks Hi

How to print like printf in python
How to print like printf in python Hi, I am from C background and there used to be printf function. Jow to print like printf in python? The printf function is C/C++ is very useful. So, looking for some line printf in Python

how to access python dictionary elements
how to access python dictionary elements Hi, I am new to Python...? How to access python dictionary elements? Thanks Hello, Python... a collection object in my program. Now I want to access value of a key from python

how to move file in python script
how to move file in python script Hi, In my Python program I have... the file from one directory to another in Python? Thanks (adsbygoogle... of the following methods for moving a file to another directory in Python: os.rename

how to add data into dictionary in python
how to add data into dictionary in python Hi, I have a python API that returns Python dictionary. It contains many values. Here is simple example...} I want to add DOB to this dictionary object. how to add data into dictionary

How to append a dictionary to a list in Python
How to append a dictionary to a list in Python Hi, I have a loop... is to add the dictionary into the list. How I can achieve this? How to append a dictionary to a list in Python? Thanks Hi, Here is the simple example

How is Python used for data science?
How is Python used for data science? Hi, I am beginner in Data...: How is Python used for data science? Try to provide me good examples or tutorials links so that I can learn the topic "How is Python used for data science

How to read from stdin in python
How to read from stdin in python Hi, I am writing a program in python which is suppose to take input from user. In Java it was easy, but in Python how i can achieve the same? ThanksADS_TO_REPLACE_1 HI

Uses of Android Technology
interface based on touch or even gesture. Uses of android technology not just... or tablet can be just enough, so multifarious, stylish and smarter the uses of android technology eventually has become! Understanding the uses of android

How to get yesterday date in Python?
Yesterday's date in Python - How to get yesterday date in Python? Python... API. In this article we will learn how to get yesterday's date in the Python... object in Python. How to get yesterday date in Python in yyyymmdd format? Now

tensorflow 1.12 python version
tensorflow 1.12 python version Hi, The latest version of TensorFlow... the version of python to be used. What is tensorflow 1.12 supported python version? Thanks

Call the parent page(uses master page) from the child page
Call the parent page(uses master page) from the child page how to call the parent page from child without refreshing the parent and my parent window uses the master page

How to learn Python programming language?
Learn fast growing Python Programming Language - How to learn Python... describes you How to learn the Python Programming Language from scratch. So, let... the Programming language and how Python is considered as a programming

How should I start learning Python?
How should I start learning Python? Hi, I want to learn PySpark... first. How should I learn Python myself. I wish to learn Python programming as soon as possible. How should I start learning Python? Thanks

What is Machine Learning and its uses now a days?
What is Machine Learning and its uses now a days? Hi, I what... Learning and its uses now a days? Thanks What is Machine Learning and its uses now a days? In today's world, these machines and also the robots have

How do I download urllib3 for python 2.7
How do I download urllib3 for python 2.7 Hi, How do I download urllib3 for python 2.7? I want to install urllib3 on Python 2.7. How to do... on your python 2.7. You can run following command: pip install urllib3 The pip

How do you get the size of an array in Python
How do you get the size of an array in Python Hi, I want to get the size of my array in Python. How do you get the size of an array in Python? How we are doing it in Python? Thanks Hi, To find the size of any

Browser Crashes Suddenly : Application uses applet - Applet
Browser Crashes Suddenly : Application uses applet Hello All, All of sudden, when the application is running the browser IE6 gets crashed . JRE used is 1.5.0_05. Can anyone please help me out . Its urgent.. Vishal

program that uses while loops to perform the following steps :
program that uses while loops to perform the following steps : Write a program that uses while loops to perform the following steps : a. Prompt the use to input two integers : FirstNum and secondNum.( FirstNum must be less

ModuleNotFoundError: No module named 'python_version'
named 'python_version' How to remove the ModuleNotFoundError: No module named 'python_version' error? Thanks Hi, In your python...ModuleNotFoundError: No module named 'python_version' Hi, My

how to print float with 2 decimals python
how to print float with 2 decimals python Hi, I am new to Python... this value with 2 decimals places. How to print float with 2 decimals python?ADS_TO_REPLACE_1 Is there any good website to learn Python programming language

How specify the version of python that spark-submit uses

Questions by Category