Are you looking to find how to create user-defined functions in the Azure Databricks cloud or maybe you are looking for a solution, to make your own custom function to apply on top of Dataframe columns in PySpark Databricks using the select methods? If you are looking for any of these problem solutions, then you have landed on the correct page. I will also show you what and how to use PySpark to use udf() function in PySpark Azure Databricks. I will explain it with a practical example. So don’t waste time let’s start with a step-by-step guide to understanding how to select columns in a PySpark DataFrame.
In this blog, I will teach you the following with practical examples:
- Create and Use a UDF
- Using UDF on PySpark SQL
udf() method used to define your own custom function in PySpark Azure Databricks, which can be applied on top of Dataframe coumns.
Syntax: udf()
Contents
- 1 What is the syntax of the udf() function in PySpark Azure Databricks?
- 2 Create a simple DataFrame
- 3 First, let’s understand the DataFrame and the problem that has to be fixed.
- 4 How to create and use a User Defined Function in PySpark Azure Databricks?
- 5 How to use User-Defined Functions in PySpark SQL?
- 6 Why do we need User Defined Functions when we can solve the same problem using PySpark in-built functions in Azure Databricks?
- 7 When should you go for User Defined Functions in PySpark Azure Databricks?
- 8 Real World Use Case Scenarios for using User Defined Functions in PySpark Azure Databricks?
- 9 What are the alternative methods for User Defined Functions in PySpark Azure Databricks?
- 10 Final Thoughts
What is the syntax of the udf() function in PySpark Azure Databricks?
The syntax is as follows:
udf(f, returnType)
| Parameter Name | Required | Description |
| f (function) | Yes | It represents the custom designed function. |
| returnType (str or pyspark.sql.types.DataType) | Optional | It represents the return type of the custom function. The value can be either a pyspark.sql.types.DataType object or a DDL-formatted type string. |
Apache Spark Official Documentation Link: udf()
Create a simple DataFrame
Gentle reminder:
In Databricks,
- sparkSession made available as spark
- sparkContext made available as sc
In case, you want to create it manually, use the below code.
from pyspark.sql.session import SparkSession
spark = SparkSession.builder
.master("local[*]")
.appName("azurelib.com")
.getOrCreate()
sc = spark.sparkContext
a) Create manual PySpark DataFrame
data = [
(1,"Nitheesh",22,"m","856 324 321"),
(2,"Ranveer",21,"Male","7585543210"),
(3,"Kritisha",24,"FM","9876543210 "),
(4,"Madhu",26,"female","987 657 8954")
]
df = spark.createDataFrame(data, schema=["id","name","age","gender","mobile"])
df.printSchema()
df.show()
"""
root
|-- id: long (nullable = true)
|-- name: string (nullable = true)
|-- age: long (nullable = true)
|-- gender: string (nullable = true)
|-- mobile: string (nullable = true)
+---+--------+---+------+------------+
| id| name|age|gender| mobile|
+---+--------+---+------+------------+
| 1|Nitheesh| 22| m| 856 324 321|
| 2| Ranveer| 21| Male| 7585543210|
| 3|Kritisha| 24| FM| 9876543210 |
| 4| Madhu| 26|female|987 657 8954|
+---+--------+---+------+------------+
"""
b) Creating a DataFrame by reading files
Download and use the below source file.
# replace the file_paths with the source file location which you have downloaded.
df_2 = spark.read.format("csv").option("header", True).load(file_path)
df_2.printSchema()
"""
root
|-- id: long (nullable = true)
|-- name: string (nullable = true)
|-- age: long (nullable = true)
|-- gender: string (nullable = true)
|-- mobile: string (nullable = true)
"""
Note: Here, I will be using the manually created DataFrame.
First, let’s understand the DataFrame and the problem that has to be fixed.
Problem 1: Column “gender”
In the above DataFrame, you can see that the gender column is not in any specific format. We have to convert the value to either “Male” or “Female”.
Problem 2: Column “mobile”
In the column “mobile”, you can see that we have it doesn’t follow any pattern.
- Spaces in-between number
- Extra spaces at the end
- Some numbers don’t have 10 digits
We have to reformat the number so that it follows a defined pattern. Also return None, if the number doesn’t have a 10-digit number.
How to create and use a User Defined Function in PySpark Azure Databricks?
The most beneficial component of Spark SQL & DataFrame that is utilized to expand PySpark’s built-in capabilities is PySpark UDF, also known as a User Defined Function.
Before creating a function keep these points in mind:
- Create function
- Register function
- Apply function
Solution 1:
from pyspark.sql.functions import udf
# 1. Create function
def reformat_gender_udf(gender: str) -> str:
first_char = gender[0].lower()
output = "Male"
if first_char == "f":
output = "Female"
return output
# 2. Register function using udf() method
reformat_gender = udf(reformat_gender_udf)
# 3. Apply function on top of Dataframe columns
df.withColumn("gender", reformat_gender("gender")).show()
"""
Output:
+---+--------+---+------+------------+
| id| name|age|gender| mobile|
+---+--------+---+------+------------+
| 1|Nitheesh| 22| Male| 856 324 321|
| 2| Ranveer| 21| Male| 7585543210|
| 3|Kritisha| 24|Female| 9876543210 |
| 4| Madhu| 26|Female|987 657 8954|
+---+--------+---+------+------------+
"""
Solution 2:
from pyspark.sql.functions import udf
# 1. Create function
def reformat_mobile_udf(number: str):
number_strip = number.strip()
no_space_number = "".join(number_strip.split(" "))
output = no_space_number if len(no_space_number) == 10 else None
return output
# 2. Register function using udf() method
reformat_mobile = udf(reformat_mobile_udf)
# 3. Apply function on top of Dataframe columns
df.withColumn("mobile", reformat_mobile("mobile")).show()
"""
Output:
+---+--------+---+------+----------+
| id| name|age|gender| mobile|
+---+--------+---+------+----------+
| 1|Nitheesh| 22| m| null|
| 2| Ranveer| 21| Male|7585543210|
| 3|Kritisha| 24| FM|9876543210|
| 4| Madhu| 26|female|9876578954|
+---+--------+---+------+----------+
"""
How to use User-Defined Functions in PySpark SQL?
In order to use the custom-defined function in PySpark SQL, you have to register it before using it.
from pyspark.sql.types import StringType
# Register the user defined function
spark.udf.register("reformat_gender", reformat_gender_udf, StringType())
df.createOrReplaceTempView("sample")
spark.sql("""
SELECT id, name, age, reformat_gender(gender) AS gender, mobile
FROM sample
""").show()
"""
Output:
+---+--------+---+------+------------+
| id| name|age|gender| mobile|
+---+--------+---+------+------------+
| 1|Nitheesh| 22| Male| 856 324 321|
| 2| Ranveer| 21| Male| 7585543210|
| 3|Kritisha| 24|Female| 9876543210 |
| 4| Madhu| 26|Female|987 657 8954|
+---+--------+---+------+------------+
"""
Why do we need User Defined Functions when we can solve the same problem using PySpark in-built functions in Azure Databricks?
Let me solve the same problems using PySpark in-built functions.
Solution 1:
from pyspark.sql.functions import col, lower, when, substring
df.withColumn("gender",
when(substring(lower(col("gender")), 1, 1) == "m", "Male")\
.otherwise("Female")
).show()
"""
Output:
+---+--------+---+------+------------+
| id| name|age|gender| mobile|
+---+--------+---+------+------------+
| 1|Nitheesh| 22| Male| 856 324 321|
| 2| Ranveer| 21| Male| 7585543210|
| 3|Kritisha| 24|Female| 9876543210 |
| 4| Madhu| 26|Female|987 657 8954|
+---+--------+---+------+------------+
"""
Solution 2:
df.withColumn("mobile",
when(length(concat_ws("", split(trim(col("mobile")), " "))) != 10, None)\
.otherwise(concat_ws("", split(trim(col("mobile")), " ")))
).show()
"""
Output:
+---+--------+---+------+----------+
| id| name|age|gender| mobile|
+---+--------+---+------+----------+
| 1|Nitheesh| 22| m| null|
| 2| Ranveer| 21| Male|7585543210|
| 3|Kritisha| 24| FM|9876543210|
| 4| Madhu| 26|female|9876578954|
+---+--------+---+------+----------+
"""
You can see that we solved the same problem using PySpark’s built-in functions, but the catch is that we had to hardcode the solution. The user-defined function only needs to be written once and can be used with other DataFrames too.
I have attached the complete code used in this blog in notebook format to this GitHub link. You can download and import this notebook in databricks, jupyter notebook, etc.
When should you go for User Defined Functions in PySpark Azure Databricks?
These could be the possible reasons:
- To reduce code line
- To reformat codes
- To define your own function for a use case that PySpark functions can’t handle.
Real World Use Case Scenarios for using User Defined Functions in PySpark Azure Databricks?
Assume that you have a people dataset that contains people’s names, ages, and gender columns. But here is a twist, the gender column has unformatted text for example male was represented as “M”, “m”, “Male”, “male”, “maLe” and female was represented as “FM”, “FM”, “Female”, “FeMale”. Your goal is to convert all the males and females into “Male” and “Female” respectively. You can solve this by using conditional and string manipulation functions in PySpark. But what happens when you want to perform this on multiple Data Frames, the code will be huge right? Therefore by creating and registering our User Defined Function, we can use it wherever we want.
What are the alternative methods for User Defined Functions in PySpark Azure Databricks?
Final Thoughts
In this article, we have learned about the PySpark User Defined Functions in Azure Databricks along with the examples explained clearly. I have also covered different scenarios with practical examples that could be possible. I hope the information that was provided helped in gaining knowledge.
Please share your comments and suggestions in the comment section below and I will try to answer all your queries as time permits.
- For Azure Study material Join Telegram group : Telegram group link:
- Azure Jobs and other updates Follow me on LinkedIn: Azure Updates on LinkedIn
- Azure Tutorial Videos: Videos Link
- Azure Databricks Lesson 1
- Azure Databricks Lesson 2
- Azure Databricks Lesson 3
- Azure Databricks Lesson 4
- Azure Databricks Lesson 5
- Azure Databricks Lesson 6
- Azure Databricks Lesson 7
As a big data engineer, I design and build scalable data processing systems and integrate them with various data sources and databases. I have a strong background in Python and am proficient in big data technologies such as Hadoop, Hive, Spark, Databricks, and Azure. My interest lies in working with large datasets and deriving actionable insights to support informed business decisions.