Copy activity is one of the most popular and highly used activity in the azure data factory.
Copy activity is basically used for ETL purpose or lift and shift where you want to move the data from one data source to the other data source. While you copy the data you can also do the transformation for example you read the data from csv file which contains 10 columns however while writing to your target data source you want to keep only 5 columns. You can transform it and you can send only the required number of columns to the the destination data source.
For creating the copy activity you need to have your source and destination ready. Here destination is called as sink. Copy activity requires:
- Linked service
Assume you already have a linked service and data service created in case not you can please refer these links to create link service and datasets
Step 1 : Go to the author tab and click on Plus sign to create the pipeline.
Step 2 : Give pipeline some name
Step 3: Go to activity blade and type the ‘copy’ in the activity search box. You will get the activity , drag the copy activity to the pipeline
Now you can see ‘red 1’ on the source and sink tab in the bottom. This is indicating that these two field need to be completely filled,
Let click on the source tab now it is asking to select some data set from the drop down if you have a data set already there you can select from the drop down or else you can click on the Plus sign to create a new data set.
What is dataset in azure data factory (Follow this to create the data set for blob)
Create dynamic dataset in ADF for Azure Sql Db ( Follow this to create the dataset for the azure sql db)
We have a csv file which contains the data which we want to load it into the azure sql database. Csv file look like this :
You can download the csv file from here :
We are using the sample table comes while creating the azure sql db. If you don’t have one you can create like this :
Now coming back again to our pipeline. In source tab lets select the azure blob data set pointing to our csv file location :
Ensure that for above data set First row as header should be set at true. Because our csv file has first line as header.
Lets select the Sink now. Here sink is our dataset which point to the sql table. I already have dataset ready. You can use this link to create : Create dynamic dataset in ADF for Azure Sql Db
In the dataset select the table name
Your pipeline is almost ready now. Just click on the debug and see if its working successfully.
You can 450 rows has been inserted into the table.
Deepak Goyal is certified Azure Cloud Solution Architect. He is having around decade and half experience in designing, developing and managing enterprise cloud solutions. He is also Big data certified professional and passionate cloud advocate.