Azure Data Factory, ADSL Gen2 BLOB Storage and Syncing Data from Share Point Folder

 

Photo by Manuel Geissinger

Today we are going to discuss data sync between on premisses SharePoint folder and Azure BLOB Storage. 

When we need to upload or download files from SharePoint folder within the home network to Azure. We must consider the best way to auto sync as well. Let's discuss them step by step.

Azure Data Factory (ADF) is a powerful cloud-based service provided by Microsoft Azure. Let me break it down for you:

1. Purpose and Context:

   • In the world of big data, we often deal with raw, unorganized data stored in various systems.
   • However, raw data alone lacks context and meaning for meaningful insights.
   • Azure Data Factory (ADF) steps in to orchestrate and operationalize processes, transforming massive raw data into actionable business insights.

2. What Does ADF Do?:

   • ADF is a managed cloud service designed for complex data integration projects.
   • It handles hybrid extract-transform-load (ETL) and extract-load-transform (ELT) scenarios.
   • It enables data movement and transformation at scale.

3. Usage Scenarios:

   • Imagine a gaming company collecting petabytes of game logs from cloud-based games.
   • The company wants to:
     • Analyze these logs for customer insights.
     • Combine on-premises reference data with cloud log data.
     • Process the joined data using tools like Azure HDInsight (Spark cluster).
     • Publish transformed data to Azure Synapse Analytics for reporting.
   • ADF automates this workflow, allowing daily scheduling and execution triggered by file arrivals in a blob store container.

4. Key Features:

   • Data-Driven Workflows: Create and schedule data-driven workflows (called pipelines).
   • Ingestion: Ingest data from disparate data stores.
   • Transformation: Build complex ETL processes using visual data flows or compute services like Azure HDInsight Hadoop, Azure Databricks, and Azure SQL Database.
   • Publishing: Publish transformed data to destinations like Azure Synapse Analytics for business intelligence applications.

5. Why ADF Matters:

   • It bridges the gap between raw data and actionable insights.
   • Businesses can make informed decisions based on unified data insights.

Learn more about Azure Data Factory on Microsoft Learn¹.

Azure Data Factory (ADF) can indeed sync data between on-premises SharePoint folders and Azure Blob Storage. Let’s break it down:

1. Syncing with On-Premises SharePoint Folder:

   • ADF allows you to copy data from a SharePoint Online List (which includes folders) to various supported data stores.
   • Here’s how you can set it up:
     • Prerequisites:
       • Register an application with the Microsoft identity platform.
       • Note down the Application ID, Application key, and Tenant ID.
       • Grant your registered application permission in your SharePoint Online site.
     • Configuration:
       • Create a linked service to your SharePoint Online List using the UI.
       • Use the service principal authentication method.
       • Specify the Application ID, Application key, and Tenant ID.
       • Set up the necessary permissions for your SharePoint app.
       • The connector uses OData protocol to retrieve data from SharePoint.
       • Note that it supports copying data from SharePoint Online List but not individual files¹.

2. Syncing with Azure Blob Storage:

   • ADF can also copy data to and from Azure Blob Storage.
   • You can use the Copy activity to move data.
   • Additionally, you can use the Data Flow activity to transform data within Azure Blob Storage².

3. Combining Both:

   • To sync data between an on-premises SharePoint folder and Azure Blob Storage:
     • Set up your SharePoint linked service.
     • Set up your Azure Blob Storage linked service.
     • Create a pipeline that uses the Copy activity to move data from SharePoint to Blob Storage.
     • Optionally, apply any necessary transformations using the Data Flow activity.

Remember, ADF is your orchestration tool, ensuring seamless data movement and transformation across various data sources and sinks.

On the other hand, Azure Data Lake Storage Gen2 (ADLS Gen2) is a powerful service in the Microsoft Azure ecosystem. Let’s explore how to use it effectively:

1. Overview of ADLS Gen2:

   • ADLS Gen2 combines the capabilities of a data lake with the scalability and performance of Azure Blob Storage.
   • It’s designed for handling large volumes of diverse data, making it ideal for big data analytics and data warehousing scenarios.

2. Best Practices for Using ADLS Gen2:

   • Optimize Performance:
     • Consider using a premium block blob storage account if your workloads require low latency and high I/O operations per second (IOP).
     • Premium accounts store data on solid-state drives (SSDs) optimized for low latency and high throughput.
     • While storage costs are higher, transaction costs are lower.
   • Reduce Costs:
     • Organize your data into data sets within ADLS Gen2.
     • Provision separate ADLS Gen2 accounts for different data landing zones.
     • Evaluate feature support and known issues to make informed decisions.
   • Security and Compliance:
     • Use service principals or access keys to access ADLS Gen2.
     • Understand terminology differences (e.g., blobs vs. files).
     • Review the documentation for feature-specific guidance.
   • Integration with Other Services:
     • Mount ADLS Gen2 to Azure Databricks for reading and writing data.
     • Compare ADLS Gen2 with Azure Blob Storage for different use cases.
     • Understand where ADLS Gen2 fits in the stages of analytical processing.

3. Accessing ADLS Gen2:

   • You can access ADLS Gen2 in three ways:
     • Mounting it to Azure Databricks using a service principal or OAuth 2.0.
     • Directly using a service principal.
     • Using the ADLS Gen2 storage account access key directly.

Remember, ADLS Gen2 empowers you to manage and analyze vast amounts of data efficiently. Dive into the documentation and explore its capabilities! 

Learn more about Azure Data Lake Storage Gen2 on Microsoft Learn¹.

Let’s set up a data flow that automatically copies files from an on-premises SharePoint folder to Azure Data Lake Storage Gen2 (ADLS Gen2) whenever new files are uploaded. Here are the steps:

1. Prerequisites:

   • Ensure you have the following:
     • An Azure subscription (create one if needed).
     • An Azure Storage account with ADLS Gen2 enabled.
     • An on-premises SharePoint folder containing the files you want to sync.

2. Create an Azure Data Factory (ADF):

   • If you haven’t already, create an Azure Data Factory using the Azure portal.
   • Launch the Data Integration application in ADF.

3. Set Up the Copy Data Tool:

   • In the ADF home page, select the Ingest tile to launch the Copy Data tool.
   • Configure the properties:
     • Choose Built-in copy task under Task type.
     • Select Run once now under Task cadence or task schedule.

4. Configure the Source (SharePoint):

   • Click + New connection.
   • Select SharePoint from the connector gallery.
   • Provide the necessary credentials and details for your on-premises SharePoint folder.
   • Define the source dataset.

5. Configure the Destination (ADLS Gen2):

   • Click + New connection.
   • Select Azure Data Lake Storage Gen2 from the connector gallery.
   • Choose your ADLS Gen2 capable account from the “Storage account name” drop-down list.
   • Create the connection.

6. Mapping and Transformation (Optional):

   • If needed, apply any transformations or mappings between the source and destination.
   • You can use the Data Flow activity for more complex transformations.

7. Run the Pipeline:

   • Save your configuration.
   • Execute the pipeline to copy data from SharePoint to ADLS Gen2.
   • You can schedule this pipeline to run periodically or trigger it based on events (e.g., new files in SharePoint).

8. Monitoring and Alerts:

   • Monitor the pipeline execution in the Azure portal.
   • Set up alerts for any failures or anomalies.

Remember to adjust the settings according to your specific SharePoint folder and ADLS Gen2 requirements. With this setup, your files will be automatically synced from SharePoint to ADLS Gen2 whenever new files are uploaded! 

Learn more about loading data into Azure Data Lake Storage Gen2 on Microsoft Learn¹.