Skip to content

PySpark: Install and Run Your First DataFrame

This guide sets up PySpark locally and runs a small DataFrame job to confirm everything works.

This is the quickest setup for local learning and small experiments.

1) Create a virtual environment

If you use uv:

uv venv
source .venv/bin/activate

(You can also use python -m venv .venv.)

2) Install PySpark

uv pip install pyspark

3) Validate by launching the PySpark shell

pyspark

You should see a Spark session start.

4) Run a tiny DataFrame job

Create a file quick_check.py and run it:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("pyspark-quick-check").getOrCreate()

df = spark.createDataFrame(
    [("Ankit", 1), ("Spark", 2), ("PySpark", 3)],
    ["name", "value"],
)

df.show()
print("count:", df.count())

spark.stop()

python quick_check.py

Option B: Use PySpark with a separate Spark installation

If you already installed Apache Spark (for example under /opt/spark), you can reuse that distribution.

1) Ensure Spark is on your PATH

export SPARK_HOME=/opt/spark
export PATH="$SPARK_HOME/bin:$PATH"

2) Install PySpark

uv pip install pyspark

3) Point PySpark to your Spark (optional)

In many setups PySpark will work without this, but if you need to force a specific Spark distribution:

export PYSPARK_PYTHON=python

Then run:

pyspark

Common issues

Java not found

PySpark still needs Java.

java -version

If missing, install a JDK (commonly JDK 11 or 17).

pyspark command not found

  • Confirm you installed pyspark into the currently active environment.
  • If you’re in a venv, ensure it’s activated:
which python
which pyspark

Next: write a small ETL and run it with spark-submit.