Masking Data Before Ingest

Masking data before ingesting it into Azure Data Lake Storage (ADLS) Gen2 or any cloud-based data lake involves transforming sensitive data elements into a protected format to prevent unauthorized access. Here's a high-level approach to achieving this:

1. Identify Sensitive Data:

   - Determine which fields or data elements need to be masked, such as personally identifiable information (PII), financial data, or health records.

2. Choose a Masking Strategy:

   - Static Data Masking (SDM): Mask data at rest before ingestion.

   - Dynamic Data Masking (DDM): Mask data in real-time as it is being accessed.

3. Implement Masking Techniques:

   - Substitution: Replace sensitive data with fictitious but realistic data.

   - Shuffling: Randomly reorder data within a column.

   - Encryption: Encrypt sensitive data and decrypt it when needed.

   - Nulling Out: Replace sensitive data with null values.

   - Tokenization: Replace sensitive data with tokens that can be mapped back to the original data.

4. Use ETL Tools:

   - Utilize ETL (Extract, Transform, Load) tools that support data masking. Examples include Azure Data Factory, Informatica, Talend, or Apache Nifi.

5. Custom Scripts or Functions:

   - Write custom scripts in Python, Java, or other programming languages to mask data before loading it into the data lake.

Example Using Azure Data Factory:

1. Create Data Factory Pipeline:

   - Set up a pipeline in Azure Data Factory to read data from the source.

2. Use Data Flow:

   - Add a Data Flow activity to your pipeline.

   - In the Data Flow, add a transformation step to mask sensitive data.

3. Apply Masking Logic:

   - Use built-in functions or custom expressions to mask data. For example, use the `replace()` function to substitute characters in a string.

```json

{

  "name": "MaskSensitiveData",

  "activities": [

    {

      "name": "DataFlow1",

      "type": "DataFlow",

      "dependsOn": [],

      "policy": {

        "timeout": "7.00:00:00",

        "retry": 0,

        "retryIntervalInSeconds": 30,

        "secureOutput": false,

        "secureInput": false

      },

      "userProperties": [],

      "typeProperties": {

        "dataFlow": {

          "referenceName": "DataFlow1",

          "type": "DataFlowReference"

        },

        "integrationRuntime": {

          "referenceName": "AutoResolveIntegrationRuntime",

          "type": "IntegrationRuntimeReference"

        }

      }

    }

  ],

  "annotations": []

}

```

4. Load to ADLS Gen2:

   - After masking, load the transformed data into ADLS Gen2 using the Sink transformation.

By following these steps, you can ensure that sensitive data is masked before it is ingested into ADLS Gen2 or any other cloud-based data lake.

Data Masking When Ingesting Into Databricks

 

Photo by Alba Leader

Data masking is a data security technique that involves hiding data by changing its original numbers and letters. It's a way to create a fake version of data that's similar enough to the actual data, while still protecting it. This fake data can then be used as a functional alternative when the real data isn't needed. 

Unity Catalog is not a feature within Databricks. Instead, Databricks provides the Delta Lake feature, which includes data governance capabilities such as row filters and column masking.

Unity Catalog in Databricks allows you to apply data governance policies such as row filters and column masks to sensitive data. Let’s break it down:

1. Row Filters:

   • Row filters enable you to apply a filter to a table so that subsequent queries only return rows for which the filter predicate evaluates to true.
   • To create a row filter, follow these steps:
     1. Write a SQL user-defined function (UDF) to define the filter policy.
     • CREATE FUNCTION <function_name> (<parametergoog_1380099708_name> <parameter_type>, ...) RETURN {filter clause whobe a boolean};
2. Apply the row filter to an existing table using the following syntax:

   ALTER TABLE <table_name> SET ROW FILTER <function_name> ON (<column_name>, ...);

• 1. You can also specify a row filter during the initial table creation.
• Each table can have only one row filter, and it accepts input parameters that bind to specific columns of the table.

3. Column Masks:

   • Column masks allow you to transform or mask specific column values before returning them in query results.
   • To apply column masks:
     1. Create a function that defines the masking logic.
     2. Apply the masking function to a table column using an ALTER TABLE statement.
     3. Alternatively, you can apply the masking function during table creation.

4. Unity Catalog Best Practices:

5. When setting up Unity Catalog, consider assigning a location to a catalog level. For example:

   CREATE CATALOG hr_prod
   LOCATION 'abfss://mycompany-hr-prod@storage-account.dfs.core.windows.net/unity-catalog';

   
   

You can apply column masks to transform or conceal specific column values before returning them in query results. Here’s how you can achieve this:

1. Create a Masking Function:

   • Define a function that specifies the masking logic. This function will be used to transform the column values.
   • For example, let’s say you want to mask the last four digits of a credit card number. You can create a masking function that replaces the last four digits with asterisks.

2. Apply the Masking Function to a Column:

   • Use an ALTER TABLE statement to apply the masking function to a specific column.
   • For instance, if you have a column named credit_card_number, you can apply the masking function to it:

     ALTER TABLE my_table SET COLUMN MASK credit_card_number USING my_masking_function;
     

3. Example Masking Function:

   • Suppose you want to mask the last four digits of a credit card number with asterisks. You can create a masking function like this:

     CREATE FUNCTION my_masking_function AS
     BEGIN
         RETURN CONCAT('************', RIGHT(credit_card_number, 4));
     END;
     

4. Query the Table:

   • When querying the table, the masked values will be returned instead of the original values.

Let’s focus on how you can achieve column masking in Databricks using Delta Lake:

1. Column Masking:

   • Delta Lake allows you to apply column-level transformations or masks to sensitive data.
   • You can create custom masking functions to modify specific column values before returning them in query results.

2. Creating a Masking Function:

   • Define a user-defined function (UDF) that specifies the masking logic. For example, you can create a function that masks the last four digits of a credit card number.
   • Here’s an example of a masking function that replaces the last four digits with asterisks:

     def mask_credit_card(card_number):
         return "************" + card_number[-4:]
     

3. Applying the Masking Function:

   • Use the withColumn method to apply the masking function to a specific column in your DataFrame.
   • For instance, if you have a DataFrame named my_table with a column named credit_card_number, you can apply the masking function as follows:

     from pyspark.sql.functions import udf
     from pyspark.sql.types import StringType
     
     # Register the UDF
     spark.udf.register("mask_credit_card", mask_credit_card, StringType())
     
     # Apply the masking function to the column
     masked_df = my_table.withColumn("masked_credit_card", udf("credit_card_number"))
     

4. Querying the Masked Data:

   • When querying the masked_df, the transformed (masked) values will be returned for the masked_credit_card column.

You can find different related articles here kindly search.