Set up managed virtual networks for data quality scans in virtual network storage

This article outlines the steps to create protected data source connections for data profiling and data quality scans in Microsoft Purview.

Virtual networks and private endpoints enhance the security and isolation of Azure resources. Virtual network protected endpoints let you access Azure services while keeping traffic on the Azure backbone network. Private endpoints go further by assigning a private IP address in your virtual network, so all traffic stays within the network and doesn't use the public internet. Virtual network protected endpoints and private endpoints are essential when security and network isolation are critical, because they minimize exposure to threats from the public internet.

Review user permission requirements

You need the following permissions to set up managed virtual networks for data quality:

• Compute provision – Governance Domain Owner
• Managed private endpoint creation - Governance Domain Owner
• Private endpoint approval – Owner of the Azure storage source

Manage virtual network provisioning

Data Governance Administrators can provision a virtual network compute ___location in supported Azure regions. To provision a compute ___location, go to Settings > Unified Catalog > Virtual network in the Unified Catalog Admin page. Select a region from the Region dropdown. Regions that are already set up are greyed out.

Deleting a configured region removes all associated data quality connections linked to that region's virtual network across business domains. To delete a region, select X on the region's row. You can't delete a region while it's being provisioned. Provisioning has three stages: Provisioning, Completed, and Available.

Configure a data quality managed virtual network

Configure a data quality managed virtual network by creating a connection to a protected data source.

1. In the Microsoft Purview Unified Catalog, select Health Management, and then select Data quality.

2. Select a governance ___domain from the list.

3. From the Manage dropdown list, select Connections to open the connections page.

4. Select the New tab to create a new connection for the data products and data assets in your governance ___domain.

5. In the connection page, add a display name and description, and select the data source type.

6. Add other data source details like Subscription and Storage Account name or Server Name and database name, depending on the source.

7. Select the Enable managed V-Net checkbox.

8. Select the region where the data source is housed.

9. Purview Data Quality checks if a compute infrastructure already exists for the account in that region. If not, you're prompted to create a new virtual network dedicated compute.

   

   Tip

   Provisioning compute can take up to 10 minutes. After requesting compute provisioning, you can save the connection creation request in draft mode and edit it later.

10. After the compute is provisioned, data quality checks if a private endpoint connection to the asset already exists. If not, you're prompted to create a private endpoint connection.

    

11. After the private endpoint is created, or if a private endpoint already exists but wasn't approved, you're prompted to approve the private endpoint connection request.

    

12. To approve this request, go to the Networking tab in Storage Account or SQL Server. Select the Private access tab, select a pending connection, and select Approve.

    

13. Select Yes to approve the connection.

    

14. The request now shows as Approved.

    

    Tip

    After you request the private endpoint connection, you can save the connection as a draft and resume after the request is approved.

15. After the private endpoint connection is created and approved, submit the connection.

    

    Caution

    Test connection isn't currently supported for virtual network protected assets.

16. After the connection is completed, you can run data quality jobs as usual against the virtual network protected assets.

Set up data source-specific connections

To set up virtual network connections for specific data sources, see the following resources:

• Data quality for Azure Databricks UC
• Data quality for Azure Synapse serverless and data warehouse
• Data quality for Snowflake data
• Data quality for Fabric Lakehouse

For Azure SQL and Data Lake storage, follow the steps in Configure a data quality managed virtual network.

Import a schema for data quality

If the data type in a schema is undefined, incorrectly defined, or changed in the source, your data quality job might fail. If the job fails, re-import the schema by using the schema import capability. You can import schemas for data sources on both public networks and those behind private endpoints. For supported data sources, see Configure a data quality managed virtual network. To import a schema from your data sources, follow these steps:

• In Unified Catalog, under Health Management, select Data quality.
• Select a business ___domain, then select a data product, then select a data asset from that data product. You arrive at the data quality overview page.
• Select Schema, then select the Schema management toggle.
• Select Import schema to import the schema.