Copying multiple files from one folder to another in Azure Data Factory, you can use the Copy data activity in a loop. For looping through each file we use ForEach activity and to get the files meta data information we need to use Get Metadata Activity.
As you can see, inside the container we have two folders input and output.
Inside input folder there are two files as we can see below.
Now the requirement is that, we need to copy these two files from Input folder to Output folder.
You can see, there is only one file named test exist in Output folder as of now.
Let’s log into Azure Data Factory portal, and create a pipeline.
Now, provide the name and description for pipeline. Description is an optional.
Now, Go to Activities pane and search for Get Metadata activity and drag to the pipeline canvas.
Using Get Metadata activity we will get a list of files available in Input folder.
Now, click on Settings tab, to edit its details.
Select a dataset, or create a new one with the New button.
Once, you click on New button, it asks you to select the structure of your data.
As, data is stored in blob storage so we select Azure Blob Storage.
After that, select a file format of your data, In our case it is text file.
Now, create a linked service which defines the connection to the data source.
Click on New button to create a linked service.
Now, provide input folder path only not file name as of now.
After that, click on OK button.
So far, we have configured the dataset, now provide the details for Field list.
Click on New button.
Now, select Child items from dropdown list. Child items field list allows you to access the list of subfolders and files in the given folder.
Now, search for ForEach activity in Activities pane and drag to the pipeline canvas.
After that, go to the Settings tab of ForEach activity.
Now, select the Items field then select the Add dynamic content link to open the dynamic content editor pane.
After that, select Get Metadata activity output from Activity Outputs tab.
It adds a following expression in Pipeline expression builder.
@activity('Get Metadata1').output
@activity('Get Metadata1').output.childItems
After that, click on OK button. Now, Inside the ForEach loop we will implement the copy activity.
For this, click on Edit buttion of ForEach activity.
After that, go to Activities pane and search for Copy data activity.
Drag it to the canvas then click on Source tab to configure the source dataset.
Select the source dataset from dropdown list. In our case, we can use the dataset ds_inputfile that we created earlier.
After that, provide the file location. Note that, file names will be picked up dynamically for that we need to write expression.
Select File type path as Wildcard file path as we are using ForEach loop to get the multiple files.
In Wildcard path filed, provide the input folder name. In our case, it is Input.
For Wildcard file name, click on Add dynamic content link.
After that, click we configure Sink settings, click on Sink tab.
Select Sink dataset, we will create a new dataset.
Click on, New button then select Azure Blob Storage and click on Continue button.
Next, select the file format then click on Continue button.
Now, to define linked service, click on New button.
Provide the details for linked services and click on Create button.
Once we create linked service, now we need to provide the output folder path where we want to copy the file.
Click on browse button and provide output folder path.
After that click on Ok button.
After providing the path, it takes you back to the Set properties page.
Here, you can verify the path and click on OK.
Now, go to the pipeline and validate it. To validate the pipeline click on Validate button.
And you can see the notification that “Your pipeline has been validate”.
Now, when we run the pipeline it should copy two files Details and Customer from Input folder to Output folder. To run the pipeline click on Debug button.
You can see the status of pipeline is succeeded.
Lets, validate it. Go to container and open Output folder.
You can see Details and Customer file are copied into this folder. Ignore the test file as it was already present there.
Also Read..
Create Azure Data Factory using Azure Portal
Azure Data Factory ETL – Load Blob Storage to Azure SQL Table
Pivot Transformation Using Data Flow Activity
Wait Activity in Azure Data Factory
