ImportError: No module named pandas - Error in sparkshell

Hi,

I am new to pyspark and learning it from beginning. I have successfully installed Spark in Ubuntu and now trying to write some program.

I am trying to convert pandas array into dataframe but it giving error:

ImportError: No module named pandas

When try to import pandas:

import pandas as pd

Full error is:

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.4.0
      /_/

Using Python version 2.7.12 (default, Nov 12 2018 14:36:49)
SparkSession available as 'spark'.
>>> import pandas as pd
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named pandas
>>>

How to resolve this?

Thanks

View Answers

November 22, 2018 at 9:24 AM

Hi,

This error clearly indicates that pandas is not installed on your computer. If you install pandas then the problem will be solved.

Here is the command to install pandas:

pip install pandas

Here is simple installation process:

root@deepak-VirtualBox:/home/deepak/spark-2.4.0-bin-hadoop2.7/bin# pip install pandas
Collecting pandas
  Downloading https://files.pythonhosted.org/packages/b7/e3/f52d484244105fa3d558ce8217a5190cd3d40536076bef66d92d01566325/pandas-0.23.4-cp27-cp27mu-manylinux1_x86_64.whl (8.9MB)
    100% |????????????????????????????????| 8.9MB 125kB/s 
Collecting numpy>=1.9.0 (from pandas)
  Downloading https://files.pythonhosted.org/packages/de/37/fe7db552f4507f379d81dcb78e58e05030a8941757b1f664517d581b5553/numpy-1.15.4-cp27-cp27mu-manylinux1_x86_64.whl (13.8MB)
    100% |????????????????????????????????| 13.8MB 103kB/s 
Collecting python-dateutil>=2.5.0 (from pandas)
  Downloading https://files.pythonhosted.org/packages/74/68/d87d9b36af36f44254a8d512cbfc48369103a3b9e474be9bdfe536abfc45/python_dateutil-2.7.5-py2.py3-none-any.whl (225kB)
    100% |????????????????????????????????| 235kB 1.1MB/s 
Collecting pytz>=2011k (from pandas)
  Downloading https://files.pythonhosted.org/packages/f8/0e/2365ddc010afb3d79147f1dd544e5ee24bf4ece58ab99b16fbb465ce6dc0/pytz-2018.7-py2.py3-none-any.whl (506kB)
    100% |????????????????????????????????| 512kB 993kB/s 
Collecting six>=1.5 (from python-dateutil>=2.5.0->pandas)
  Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Installing collected packages: numpy, six, python-dateutil, pytz, pandas
Successfully installed numpy-1.15.4 pandas-0.23.4 python-dateutil-2.7.5 pytz-2018.7 six-1.11.0
You are using pip version 8.1.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
root@deepak-VirtualBox:/home/deepak/spark-2.4.0-bin-hadoop2.7/bin#

After this you should be able to import pandas in your program.

You can learn PySpark with the help of following tutorials:

Thanks

November 22, 2018 at 9:29 AM

Hi,

Now its working:

SparkSession available as 'spark'.
>>> import pandas as pd
>>>

Thanks

Advertisements

ModuleNotFoundError: No module named 'behave-pandas'
ModuleNotFoundError: No module named 'behave-pandas' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'behave-pandas' How to remove the ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'clean-pandas'
ModuleNotFoundError: No module named 'clean-pandas' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'clean-pandas' How to remove the ModuleNotFoundError: No module named '

ModuleNotFoundError: No module named 'mmal-pandas'
ModuleNotFoundError: No module named 'mmal-pandas' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'mmal-pandas' How to remove the ModuleNotFoundError: No module named '

ModuleNotFoundError: No module named 'opulent-pandas'
ModuleNotFoundError: No module named 'opulent-pandas' Hi, My... named 'opulent-pandas' How to remove the ModuleNotFoundError: No module named... the installation of opulent-pandas python library, ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'otis-pandas'
ModuleNotFoundError: No module named 'otis-pandas' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'otis-pandas' How to remove the ModuleNotFoundError: No module named '

ModuleNotFoundError: No module named 'pandas_access'
ModuleNotFoundError: No module named 'pandas_access' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas_access' How to remove the ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'pandas-anaphora'
ModuleNotFoundError: No module named 'pandas-anaphora' Hi, My... named 'pandas-anaphora' How to remove the ModuleNotFoundError: No module named... named 'pandas-anaphora' error will be solved. Thanks

ModuleNotFoundError: No module named 'pandas-bamboo'
ModuleNotFoundError: No module named 'pandas-bamboo' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas-bamboo' How to remove the ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'pandas-bj'
python library, ModuleNotFoundError: No module named 'pandas-bj' error...ModuleNotFoundError: No module named 'pandas-bj' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'pandas-bokeh'
ModuleNotFoundError: No module named 'pandas-bokeh' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas-bokeh' How to remove the ModuleNotFoundError: No module named '

ModuleNotFoundError: No module named 'pandas-cache'
ModuleNotFoundError: No module named 'pandas-cache' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas-cache' How to remove the ModuleNotFoundError: No module named '

ModuleNotFoundError: No module named 'pandas-charm'
ModuleNotFoundError: No module named 'pandas-charm' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas-charm' How to remove the ModuleNotFoundError: No module named '

ModuleNotFoundError: No module named 'pandas-compat'
ModuleNotFoundError: No module named 'pandas-compat' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas-compat' How to remove the ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'pandas_confusion'
: No module named 'pandas_confusion' error will be solved. Thanks...ModuleNotFoundError: No module named 'pandas_confusion' Hi, My... named 'pandas_confusion' How to remove the ModuleNotFoundError: No module

ModuleNotFoundError: No module named 'pandas-connect'
ModuleNotFoundError: No module named 'pandas-connect' Hi, My... named 'pandas-connect' How to remove the ModuleNotFoundError: No module named... the installation of pandas-connect python library, ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'pandas-cub'
ModuleNotFoundError: No module named 'pandas-cub' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas-cub' How to remove the ModuleNotFoundError: No module named '

ModuleNotFoundError: No module named 'pandas-datadict'
ModuleNotFoundError: No module named 'pandas-datadict' Hi, My... named 'pandas-datadict' How to remove the ModuleNotFoundError: No module named... named 'pandas-datadict' error will be solved. Thanks

ModuleNotFoundError: No module named 'pandas-datareader'
: No module named 'pandas-datareader' error will be solved. Thanks...ModuleNotFoundError: No module named 'pandas-datareader' Hi, My... named 'pandas-datareader' How to remove the ModuleNotFoundError: No module

ModuleNotFoundError: No module named 'pandas-dedupe'
ModuleNotFoundError: No module named 'pandas-dedupe' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas-dedupe' How to remove the ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'pandas_degreedays'
: No module named 'pandas_degreedays' error will be solved. Thanks...ModuleNotFoundError: No module named 'pandas_degreedays' Hi, My... named 'pandas_degreedays' How to remove the ModuleNotFoundError: No module

ModuleNotFoundError: No module named 'pandas-diligent'
ModuleNotFoundError: No module named 'pandas-diligent' Hi, My... named 'pandas-diligent' How to remove the ModuleNotFoundError: No module named... named 'pandas-diligent' error will be solved. Thanks

ModuleNotFoundError: No module named 'pandas-explode'
ModuleNotFoundError: No module named 'pandas-explode' Hi, My... named 'pandas-explode' How to remove the ModuleNotFoundError: No module named... the installation of pandas-explode python library, ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'pandas-express'
ModuleNotFoundError: No module named 'pandas-express' Hi, My... named 'pandas-express' How to remove the ModuleNotFoundError: No module named... the installation of pandas-express python library, ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'pandas-ext'
ModuleNotFoundError: No module named 'pandas-ext' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas-ext' How to remove the ModuleNotFoundError: No module named '

ModuleNotFoundError: No module named 'pandas-extensions'
: No module named 'pandas-extensions' error will be solved. Thanks...ModuleNotFoundError: No module named 'pandas-extensions' Hi, My... named 'pandas-extensions' How to remove the ModuleNotFoundError: No module

ModuleNotFoundError: No module named 'pandas-extras'
ModuleNotFoundError: No module named 'pandas-extras' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas-extras' How to remove the ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'pandas-files'
ModuleNotFoundError: No module named 'pandas-files' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas-files' How to remove the ModuleNotFoundError: No module named '

ModuleNotFoundError: No module named 'pandas-finance'
ModuleNotFoundError: No module named 'pandas-finance' Hi, My... named 'pandas-finance' How to remove the ModuleNotFoundError: No module named... the installation of pandas-finance python library, ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'pandas-flavor'
ModuleNotFoundError: No module named 'pandas-flavor' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas-flavor' How to remove the ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'pandas-fpl'
ModuleNotFoundError: No module named 'pandas-fpl' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas-fpl' How to remove the ModuleNotFoundError: No module named '

ModuleNotFoundError: No module named 'pandas-ga'
python library, ModuleNotFoundError: No module named 'pandas-ga' error...ModuleNotFoundError: No module named 'pandas-ga' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'pandas-gbq'
ModuleNotFoundError: No module named 'pandas-gbq' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas-gbq' How to remove the ModuleNotFoundError: No module named '

ModuleNotFoundError: No module named 'pandas-helpers'
ModuleNotFoundError: No module named 'pandas-helpers' Hi, My... named 'pandas-helpers' How to remove the ModuleNotFoundError: No module named... the installation of pandas-helpers python library, ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'pandas-highcharts'
: No module named 'pandas-highcharts' error will be solved. Thanks...ModuleNotFoundError: No module named 'pandas-highcharts' Hi, My... named 'pandas-highcharts' How to remove the ModuleNotFoundError: No module

ModuleNotFoundError: No module named 'pandas-htools'
ModuleNotFoundError: No module named 'pandas-htools' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas-htools' How to remove the ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'pandas-io'
python library, ModuleNotFoundError: No module named 'pandas-io' error...ModuleNotFoundError: No module named 'pandas-io' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'pandas-jalali'
ModuleNotFoundError: No module named 'pandas-jalali' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas-jalali' How to remove the ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'pandas-light'
ModuleNotFoundError: No module named 'pandas-light' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas-light' How to remove the ModuleNotFoundError: No module named '

ModuleNotFoundError: No module named 'pandas-linker'
ModuleNotFoundError: No module named 'pandas-linker' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas-linker' How to remove the ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'pandas-lite'
ModuleNotFoundError: No module named 'pandas-lite' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas-lite' How to remove the ModuleNotFoundError: No module named '

ModuleNotFoundError: No module named 'pandas-log'
ModuleNotFoundError: No module named 'pandas-log' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas-log' How to remove the ModuleNotFoundError: No module named '

ModuleNotFoundError: No module named 'pandas-lookup'
ModuleNotFoundError: No module named 'pandas-lookup' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas-lookup' How to remove the ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'pandas-mapper'
ModuleNotFoundError: No module named 'pandas-mapper' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas-mapper' How to remove the ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'pandas_mate'
ModuleNotFoundError: No module named 'pandas_mate' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named 'pandas_mate' How to remove the ModuleNotFoundError: No module named '

ModuleNotFoundError: No module named 'pandas-ml'
python library, ModuleNotFoundError: No module named 'pandas-ml' error...ModuleNotFoundError: No module named 'pandas-ml' Hi, My Python program is throwing following error: ModuleNotFoundError: No module named

ModuleNotFoundError: No module named 'pandas-montecarlo'
: No module named 'pandas-montecarlo' error will be solved. Thanks...ModuleNotFoundError: No module named 'pandas-montecarlo' Hi, My... named 'pandas-montecarlo' How to remove the ModuleNotFoundError: No module

ModuleNotFoundError: No module named 'pandas-msgpack'
ModuleNotFoundError: No module named 'pandas-msgpack' Hi, My... named 'pandas-msgpack' How to remove the ModuleNotFoundError: No module named... the installation of pandas-msgpack python library, ModuleNotFoundError: No module named

ImportError: No module named pandas - Error in sparkshell

Questions by Category