pyspark percent rank

How to find relativity rank in PySpark Azure Databricks?

by

Are you looking to find out how to relatively rank records of PySpark DataFrame in Azure Databricks cloud or maybe you are looking for a solution, to relatively rank records based on windows in PySpark Databricks using the percent_rank() function? If you are looking for any of these problem solutions, you have landed on the correct page. I will also help you how to use the PySpark percent_rank() function with multiple examples in Azure Databricks. I will explain it by taking a practical example. So please don’t waste time let’s start with a step-by-step guide to understand how to use the percent_rank() function in PySpark.

In this blog, I will teach you the following with practical examples:

  • Syntax of percent_rank() functions
  • Relativity ranking
  • Relativity ranking within a window

The PySpark function percent_rank() is a window ranking function used to rank rows relatively within a window partition in Azure Databricks.

Syntax:

percent_rank().over()

What is the syntax of the percent_rank() function in PySpark Azure Databricks?

The syntax is as follows:

percent_rank().over(window_spec)
Parameter NameRequiredDescription
window_spec(WindowSpec)YesIt represents the windowing column.
Table 1: percent_rank() Method in PySpark Databricks Parameter list with Details

Apache Spark Official documentation link: percent_rank()

Create a simple DataFrame

Let’s understand the use of the percent_rank() function with a variety of examples. Let’s start by creating a DataFrame.

Gentle reminder:

In Databricks,

  • sparkSession made available as spark
  • sparkContext made available as sc

In case, you want to create it manually, use the below code.

from pyspark.sql.session import SparkSession

spark = SparkSession.builder 
    .master("local[*]") 
    .appName("azurelib.com") 
    .getOrCreate()

sc = spark.sparkContext

a) Create manual PySpark DataFrame

data = [
    ("Lewis","McLaren",10.0),
    ("Nick","McLaren",2.0),
    ("Nico","McLaren",6.0),
    ("Fernando","McLaren",3.0),
    ("Heikki","McLaren",8.0),
    ("Kazuki","Ferrari",9.0),
    ("Sébastien","Ferrari",7.0),
    ("Kimi","Ferrari",6.0)
]

df = spark.createDataFrame(data, schema=["driver_name","team","points"])
df.printSchema()
df.show(truncate=False)

"""
root
 |-- driver_name: string (nullable = true)
 |-- team: string (nullable = true)
 |-- points: double (nullable = true)

+-----------+-------+------+
|driver_name|team   |points|
+-----------+-------+------+
|Lewis      |McLaren|10.0  |
|Nick       |McLaren|2.0   |
|Nico       |McLaren|6.0   |
|Fernando   |McLaren|3.0   |
|Heikki     |McLaren|8.0   |
|Kazuki     |Ferrari|9.0   |
|Sébastien  |Ferrari|7.0   |
|Kimi       |Ferrari|6.0   |
+-----------+-------+------+
"""

b) Creating a DataFrame by reading files

Download and use the below source file.

Download Sample Race_Results.csv
# replace the file_path with the source file location which you have downloaded.

df_2 = spark.read.format("csv").option("inferSchema", True).option("header", True).load(file_path)
df_2.printSchema()

"""
root
 |-- driver_name: string (nullable = true)
 |-- team: string (nullable = true)
 |-- points: double (nullable = true)
"""

Note: Here, I will be using the manually created DataFrame.

How to relatively rank records of PySpark DataFrame using Azure Databricks?

Let’s see how to relatively rank records based on columns of a PySpark DataFrame in Azure Databricks using various methods.

Example:

from pyspark.sql.functions import percent_rank
from pyspark.sql.window import Window

window_spec = Window.orderBy("points")
# The window orderBy() -> acts as on which order the row has be numbered

df \
.withColumn("percent_rank", percent_rank().over(window_spec)) \
.select("driver_name", "team", "points", "percent_rank").show()

"""
Output:

+-----------+-------+------+-------------------+
|driver_name|   team|points|       percent_rank|
+-----------+-------+------+-------------------+
|       Nick|McLaren|   2.0|                0.0|
|   Fernando|McLaren|   3.0|0.14285714285714285|
|       Nico|McLaren|   6.0| 0.2857142857142857|
|       Kimi|Ferrari|   6.0| 0.2857142857142857|
|  Sébastien|Ferrari|   7.0| 0.5714285714285714|
|     Heikki|McLaren|   8.0| 0.7142857142857143|
|     Kazuki|Ferrari|   9.0| 0.8571428571428571|
|      Lewis|McLaren|  10.0|                1.0|
+-----------+-------+------+-------------------+

"""

How to relatively rank records based on specific groups of PySpark DataFrame using Azure Databricks?

Let’s see how to relatively rank records based on specific groups of a PySpark DataFrame in Azure Databricks using various methods.

Example:

from pyspark.sql.functions import percent_rank
from pyspark.sql.window import Window

window_spec = Window.partitionBy("team").orderBy("points")
# The window partitionBy() -> acts as groupBy

df \
.withColumn("percent_rank", percent_rank().over(window_spec)) \
.select("driver_name", "team", "points", "percent_rank").show()

"""
Output:

+-----------+-------+------+------------+
|driver_name|   team|points|percent_rank|
+-----------+-------+------+------------+
|       Kimi|Ferrari|   6.0|         0.0|
|  Sébastien|Ferrari|   7.0|         0.5|
|     Kazuki|Ferrari|   9.0|         1.0|
|       Nick|McLaren|   2.0|         0.0|
|   Fernando|McLaren|   3.0|        0.25|
|       Nico|McLaren|   6.0|         0.5|
|     Heikki|McLaren|   8.0|        0.75|
|      Lewis|McLaren|  10.0|         1.0|
+-----------+-------+------+------------+

"""

I have attached the complete code used in this blog in a notebook format to this GitHub link. You can download and import this notebook in databricks, jupyter notebook, etc.

GitHub Link

When should you use the PySpark percent_rank() in Azure Databricks?

These could be the possible reasons:

  1. To find relative rank
  2. To find the relativity rank within a partition

Real World Use Case Scenarios for PySpark DataFrame percent_rank() in Azure Databricks?

Percentile ranks are commonly used to clarify the interpretation of scores on standardized tests. Assume that you have an aptitude test result and you want to evaluate the standing of an aptitude test score among all scores for the test. Here, the percent_rank() comes into the picture. You can percent_rank() function to find the standings.

What are the alternatives to the percent_rank() function in PySpark Azure Databricks?

There are multiple alternatives to the percent_rank() function, which are as follows:

  • rank(): The difference between rank and dense_rank is that rank leaves gaps in the ranking sequence when there are ties.
  • dense_rank(): The difference between rank and dense_rank is that dense_rank leaves no gaps in the ranking sequence when there are ties.

Final Thoughts

In this article, we have learned about the PySpark percent_rank() method of DataFrame in Azure Databricks along with the examples explained clearly. I have also covered different scenarios with practical examples that could be possible. I hope the information that was provided helped in gaining knowledge.

Please share your comments and suggestions in the comment section below and I will try to answer all your queries as time permits.

Join Telegram Group
Download Free Azure Videos
Connect on LinkedIn

PySpark in Azure Databricks, as explained by Arud Seka Berne S on azurelib.com.
Arud Seka Berne S

As a big data engineer, I design and build scalable data processing systems and integrate them with various data sources and databases. I have a strong background in Python and am proficient in big data technologies such as Hadoop, Hive, Spark, Databricks, and Azure. My interest lies in working with large datasets and deriving actionable insights to support informed business decisions.