Azure Purview Scanning client library for Python

Azure Purview Scanning is a fully managed cloud service whose users can scan your data into your data estate (also known as your catalog). Scanning is a process by which the catalog connects directly to a data source on a user-specified schedule.

• Scan your data into your catalog
• Examine your data
• Extract schemas from your data

Please rely heavily on the service's documentation and our client docs to use this library

Source code | Package (PyPI) | API reference documentation| Product documentation

Getting started

Prerequisites

• Python 2.7, or 3.6 or later is required to use this package.
• You must have an Azure subscription and a Purview to use this package.

Create a Purview Resource

Follow these instructions to create your Purview resource

Install the package

Install the Azure Purview Scanning client library for Python with pip:

pip install azure-purview-scanning

Authenticate the client

To use an Azure Active Directory (AAD) token credential, provide an instance of the desired credential type obtained from the azure-identity library.

To authenticate with AAD, you must first pip install azure-identity and enable AAD authentication on your Purview resource

After setup, you can choose which type of credential from azure.identity to use. As an example, DefaultAzureCredential can be used to authenticate the client:

Set the values of the client ID, tenant ID, and client secret of the AAD application as environment variables: AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_CLIENT_SECRET

Use the returned token credential to authenticate the client:

from azure.purview.scanning import PurviewScanningClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
client = PurviewScanningClient(endpoint="https://<my-account-name>.scanning.purview.azure.com", credential=credential)

Key concepts

Client

This package offers request builders so you can build http requests and send these requests to the service using the send_request method. For more information on how to use request builders and our clients, see here.

Examples

The following section shows you how to initialize and authenticate your client, then list all of your data sources.

• List All Data Sources

List All Data Sources

from azure.purview.scanning import PurviewScanningClient
from azure.identity import DefaultAzureCredential
from azure.purview.scanning.rest import data_sources
from azure.core.exceptions import HttpResponseError

credential = DefaultAzureCredential()
client = PurviewScanningClient(endpoint="https://<my-account-name>.scanning.purview.azure.com", credential=credential)

request = data_sources.build_list_all_request()

response = client.send_request(request)
try:
    response.raise_for_status()
    json_response = response.json()

    assert len(json_response['value']) == json_response['count']
    for value in json_response['value']:
        print(value)

except HttpResponseError as e:
    print(e)

Troubleshooting

General

The Purview Scanning client will raise exceptions defined in [Azure Core][azure_core] if you call .raise_for_status() on your responses.

Logging

This library uses the standard logging library for logging. Basic information about HTTP sessions (URLs, headers, etc.) is logged at INFO level.

Detailed DEBUG level logging, including request/response bodies and unredacted headers, can be enabled on a client with the logging_enable keyword argument:

import sys
import logging
from azure.identity import DefaultAzureCredential
from azure.purview.scanning import PurviewScanningClient

# Create a logger for the 'azure' SDK
logger = logging.getLogger('azure')
logger.setLevel(logging.DEBUG)

# Configure a console output
handler = logging.StreamHandler(stream=sys.stdout)
logger.addHandler(handler)

endpoint = "https://<my-account-name>.scanning.purview.azure.com"
credential = DefaultAzureCredential()

# This client will log detailed information about its HTTP sessions, at DEBUG level
client = PurviewScanningClient(endpoint=endpoint, credential=credential, logging_enable=True)

Similarly, logging_enable can enable detailed logging for a single send_request call, even when it isn't enabled for the client:

result = client.send_request(request, logging_enable=True)

Next steps

For more generic samples, see our client docs.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Release History

1.0.0b1 (2021-05-11)

• This is the initial release of the Azure Purview Scanning library.