pyspark read parquet

How to read Parquet files in PySpark Azure Databricks?

by

Are you looking to find out how to read Parquet files into PySpark DataFrame in Azure Databricks cloud or maybe you are looking for a solution, to multiple Parquet files into PySpark DataFrame in Azure Databricks using the read() method? If you are looking for any of these problem solutions, you have landed on the correct page. I will also show you how to use PySpark to read Parquet files into DataFrames in Azure Databricks. I will explain it by taking a practical example. So don’t waste time let’s start with a step-by-step guide to understanding how to read Parquet files into PySpark DataFrame.

In this blog, I will teach you the following with practical examples:

  • Different methods of reading single file
  • Read from multiple files
  • Read from multiple files using wild card
  • Read from directory
  • Write JSON file

In PySpark Azure Databricks, the read method is used to load files from an external source into a DataFrame.

Apache Spark Official Documentation Link: DataFrameReader()

Create a simple DataFrame

Let’s understand the use of the fill() function with a variety of examples. Let’s start by creating a DataFrame.

Gentle reminder:

In Databricks,

  • sparkSession made available as spark
  • sparkContext made available as sc

In case, you want to create it manually, use the below code.

from pyspark.sql.session import SparkSession

spark = SparkSession.builder 
    .master("local[*]") 
    .appName("azurelib.com") 
    .getOrCreate()

sc = spark.sparkContext

Folder Structure:

Before start learning let’s have a quick look at my folder structure and the files inside it.

The folder “read_write_parquet” has 2 files and 1 folder in it and the folder “read_directory” has three files in it.

File structure:

/mnt/practice/read_write_csv/
|– stocks_1.json
|– stocks_2.json
|– read_directory
|– stocks_3.json
|– stocks_info_1.json
|– stocks_info_2.json

Download the files and place them in the appropriate folder, as mentioned above. Each file has 20 records, excluding the header.

Download Source Files

How to read a single Parquet file in multiple ways into PySpark DataFrame in Azure Databricks?

To read a parquet file into a PySpark DataFrame, use the parquet(“path”) method provided by DataFrameReader. In this section, I will teach you how to read a single Parquet file using various practical methods with examples.

Examples:

# Varioud methods of reading file

# Replace the file_path with the source file location of yours.

# Method 1:
df = spark.read.format("parquet").load(f"{base_path}/stocks_1.parquet")
# Method 2:
df = spark.read.load(format="parquet", path=f"{base_path}/stocks_1.parquet")
# Method 3:
df = spark.read.format("parquet").option("path", f"{base_path}/stocks_1.parquet").load()
# Method 4:
df = spark.read.parquet(f"{base_path}/stocks_1.parquet")

df.printSchema()
df.show(3)

#The above four example gives the following output.

"""
Output:

root
 |-- symbol: string (nullable = true)
 |-- date: string (nullable = true)
 |-- price: string (nullable = true)

+------+----------+-----+
|symbol|      date|price|
+------+----------+-----+
|  MSFT|Jan 1 2000| null|
|  MSFT|Feb 1 2000|36.35|
|  MSFT|Mar 1 2000|43.22|
+------+----------+-----+

"""

How to read multiple Parquet files into PySpark DataFrame in Azure Databricks?

To read a Parquet file into a PySpark DataFrame, use the parquet (“path”) method provided by DataFrameReader. In this section, I will teach you how to read multiple Parquet files using practical methods with examples.

Examples:

df = spark.read.parquet(f"{base_path}/stocks_1.parquet")
print(f"First file count: {df.count()}")

df = spark.read.parquet(f"{base_path}/stocks_2.parquet")
print(f"Second file count: {df.count()}")

# Reading multiple files appends the second file to the first.
multiple_files = [f"{base_path}/stocks_1.parquet", f"{base_path}/stocks_2.parquet"]
df = spark.read.parquet(*multiple_files)
print(f"Mulitple file count: {df.count()}")

"""
Output:

First file count: 10
Second file count: 10
Mulitple file count: 20

"""

As you know, we have two files each of which has 10 records, 2 * 10 = 20 records.

How to read multiple Parquet files using wildcard into PySpark DataFrame in Azure Databricks?

To read a Parquet file into a PySpark DataFrame, use the parquet(“path”) method provided by DataFrameReader. With practical examples, I will teach you how to read multiple Parquet files using wildcards.

Examples:

df = spark.read.parquet(f"{base_path}/read_directory/stocks_info_*.parquet")
print(f"Multiple file count using wildcard(*): {df.count()}")

"""
Output:

Multiple file count using wildcard(*): 20

"""

As you know, we have two files each of which has 20 records, 2 * 20 = 40 records.

How to read Parquet files from a directory into PySpark DataFrame in Azure Databricks?

To read a Parquet file into a PySpark DataFrame, use the parquet(“path”) method provided by DataFrameReader. With examples, I will teach you how to read JSON files from a directory using various read method.

Examples:

df = spark.read.parquet(f"{base_path}/read_directory/")
print(f"Directory file count: {df.count()}")

"""
Output:

Directory file count: 30

"""

As you know, we have two files each of which has 50 records, 3 * 10 = 30 records excluding headers.

How to write Parquet files using DataFrameWriter method in Azure Databricks?

To write a Parquet file into a PySpark DataFrame, use the save(“path”) method provided by DataFrameReader. In this section, I will teach you how to write PArquet files using various practical methods with examples.

Examples:

Using mode() while writing files, There are multiple modes available and they are:

  • overwrite – mode is used to overwrite the existing file.
  • append – To add the data to the existing file.
  • ignore – Ignores write operation when the file already exists.
  • error(default) – When the file already exists, it returns an error.

df.write.mode(“overwrite”).save(“target_location”)

I have attached the complete code used in this blog in notebook format to this GitHub link. You can download and import this notebook in databricks, jupyter notebook, etc.

GitHub Link

When should you use the PySpark read in Azure Databricks?

These could be the possible reasons:

  1. To read a single parquet file
  2. To read multiple parquet files
  3. To read parquet files using wild card
  4. To read parquet files from a directory

Real World Use Case Scenarios for PySpark read function in Azure Databricks?

Assume you were given a parquet files dataset location and asked to read files using PySpark, you can use the PySpark spark.read() to fetch and convert the parquet file into a DataFrame.

What are the alternatives to the read function in PySpark Azure Databricks?

The PySpark function read() is the only one that helps in reading files from multiple locations.

Final Thoughts

In this article, we have learned about the PySpark read and write methods to read or write Parquet files into PySparks DataFrame in Azure Databricks along with the examples explained clearly. I have also covered different scenarios with practical examples that could be possible. I hope the information that was provided helped in gaining knowledge.

Please share your comments and suggestions in the comment section below and I will try to answer all your queries as time permits.

Join Telegram Group
Download Free Azure Videos
Connect on LinkedIn

Arud Seka Berne S
Arud Seka Berne S

As a big data engineer, I design and build scalable data processing systems and integrate them with various data sources and databases. I have a strong background in Python and am proficient in big data technologies such as Hadoop, Hive, Spark, Databricks, and Azure. My interest lies in working with large datasets and deriving actionable insights to support informed business decisions.