Skip to main content

Microsoft Azure Monitor Ingestion Client Library for Python

Project description

Azure Monitor Ingestion client library for Python

The Azure Monitor Ingestion client library is used to send custom logs to Azure Monitor using the Logs Ingestion API.

This library allows you to send data from virtually any source to supported built-in tables or to custom tables that you create in Log Analytics workspace. You can even extend the schema of built-in tables with custom columns.

Resources:

Getting started

Prerequisites

Install the package

Install the Azure Monitor Ingestion client library for Python with pip:

pip install azure-monitor-ingestion

Create the client

An authenticated client is required to upload Logs to Azure Monitor. The library includes both synchronous and asynchronous forms of the clients. To authenticate, create an instance of a token credential. Use that instance when creating a LogsIngestionClient. The following examples use DefaultAzureCredential from the azure-identity package.

Synchronous clients

Consider the following example, which creates synchronous clients for uploading logs:

import os
from azure.identity import DefaultAzureCredential
from azure.monitor.ingestion import LogsIngestionClient

endpoint = os.environ['DATA_COLLECTION_ENDPOINT']
credential = DefaultAzureCredential()
logs_client = LogsIngestionClient(endpoint, credential)

Asynchronous clients

The asynchronous forms of the client APIs are found in the .aio-suffixed namespace. For example:

import os
from azure.identity.aio import DefaultAzureCredential
from azure.monitor.ingestion.aio import LogsIngestionClient

endpoint = os.environ['DATA_COLLECTION_ENDPOINT']
credential = DefaultAzureCredential()
logs_client = LogsIngestionClient(endpoint, credential)

Key concepts

Data Collection Endpoint

Data Collection Endpoints (DCEs) allow you to uniquely configure ingestion settings for Azure Monitor. This article provides an overview of data collection endpoints including their contents and structure and how you can create and work with them.

Data Collection Rule

Data collection rules (DCR) define data collected by Azure Monitor and specify how and where that data should be sent or stored. The REST API call must specify a DCR to use. A single DCE can support multiple DCRs, so you can specify a different DCR for different sources and target tables.

The DCR must understand the structure of the input data and the structure of the target table. If the two don't match, it can use a transformation to convert the source data to match the target table. You may also use the transform to filter source data and perform any other calculations or conversions.

For more details, see Data collection rules in Azure Monitor. For information on how to retrieve a DCR ID, see this tutorial.

Log Analytics workspace tables

Custom logs can send data to any custom table that you create and to certain built-in tables in your Log Analytics workspace. The target table must exist before you can send data to it. The following built-in tables are currently supported:

Logs retrieval

The logs that were uploaded using this library can be queried using the Azure Monitor Query client library.

Examples

Upload custom logs

This example shows uploading logs to Azure Monitor.

import os

from azure.core.exceptions import HttpResponseError
from azure.identity import DefaultAzureCredential
from azure.monitor.ingestion import LogsIngestionClient

endpoint = os.environ['DATA_COLLECTION_ENDPOINT']
credential = DefaultAzureCredential()

client = LogsIngestionClient(endpoint=endpoint, credential=credential, logging_enable=True)

rule_id = os.environ['LOGS_DCR_RULE_ID']
body = [
      {
        "Time": "2021-12-08T23:51:14.1104269Z",
        "Computer": "Computer1",
        "AdditionalContext": "context-2"
      },
      {
        "Time": "2021-12-08T23:51:14.1104269Z",
        "Computer": "Computer2",
        "AdditionalContext": "context"
      }
    ]

try:
    client.upload(rule_id=rule_id, stream_name=os.environ['LOGS_DCR_STREAM_NAME'], logs=body)
except HttpResponseError as e:
    print(f"Upload failed: {e}")

Upload with custom error handling

To upload logs with custom error handling, you can pass a callback function to the on_error parameter of the upload method. The callback function will be called for each error that occurs during the upload and should expect one argument that corresponds to an LogsUploadError object. This object contains the error encountered and the list of logs that failed to upload.

# Example 1: Collect all logs that failed to upload.
failed_logs = []
def on_error(error):
    print("Log chunk failed to upload with error: ", error.error)
    failed_logs.extend(error.failed_logs)

# Example 2: Ignore all errors.
def on_error_pass(error):
    pass

client.upload(rule_id=rule_id, stream_name=os.environ['LOGS_DCR_STREAM_NAME'], logs=body, on_error=on_error)

Troubleshooting

Enable the azure.monitor.ingestion logger to collect traces from the library.

General

Monitor Ingestion client library will raise exceptions defined in Azure Core.

Logging

This library uses the standard logging library for logging. Basic information about HTTP sessions, such as URLs and headers, is logged at the INFO level.

Optional configuration

Optional keyword arguments can be passed in at the client and per-operation level. The azure-core reference documentation describes available configurations for retries, logging, transport protocols, and more.

Next steps

To learn more about Azure Monitor, see the Azure Monitor service documentation.

Samples

The following code samples show common scenarios with the Azure Monitor Ingestion client library.

Logs Ingestion samples

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repositories using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Release History

1.0.0 (2023-02-16)

Features Added

  • Added new on_error parameter to the upload method to allow users to handle errors in their own way.
    • An LogsUploadError class was added to encapsulate information about the error. An instance of this class is passed to the on_error callback.
  • Added IO support for upload. Now IO streams can be passed in using the logs parameter. (#28373)

Breaking Changes

  • Removed support for max_concurrency

Other Changes

  • Removed msrest dependency.
  • Added requirement for isodate>=0.6.0 (isodate was required by msrest).
  • Added requirement for typing-extensions>=4.0.1.

1.0.0b1 (2022-07-15)

Features

  • Version (1.0.0b1) is the first preview of our efforts to create a user-friendly and Pythonic client library for Azure Monitor Ingestion. For more information about this, and preview releases of other Azure SDK libraries, please visit https://azure.github.io/azure-sdk/releases/latest/python.html.
  • Added ~azure.monitor.ingestion.LogsIngestionClient to send logs to Azure Monitor along with ~azure.monitor.ingestion.aio.LogsIngestionClient.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

azure-monitor-ingestion-1.0.0.zip (76.3 kB view details)

Uploaded Source

Built Distribution

azure_monitor_ingestion-1.0.0-py3-none-any.whl (45.5 kB view details)

Uploaded Python 3

File details

Details for the file azure-monitor-ingestion-1.0.0.zip.

File metadata

  • Download URL: azure-monitor-ingestion-1.0.0.zip
  • Upload date:
  • Size: 76.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.28.2 setuptools/58.1.0 requests-toolbelt/0.10.1 tqdm/4.64.1 CPython/3.9.16

File hashes

Hashes for azure-monitor-ingestion-1.0.0.zip
Algorithm Hash digest
SHA256 e2ddf9db3f23a4987ac6fe3c14b8633c3169c421427106e117ebfa1806c30049
MD5 18c98d1ba751cdd470e4e78245bcb663
BLAKE2b-256 1efa058a618da15bca243a9c08abe10ee294fb0474ef340767a8b751801a28c6

See more details on using hashes here.

File details

Details for the file azure_monitor_ingestion-1.0.0-py3-none-any.whl.

File metadata

  • Download URL: azure_monitor_ingestion-1.0.0-py3-none-any.whl
  • Upload date:
  • Size: 45.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.28.2 setuptools/58.1.0 requests-toolbelt/0.10.1 tqdm/4.64.1 CPython/3.9.16

File hashes

Hashes for azure_monitor_ingestion-1.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 51a12b99bc8ab5caebe4bdd832d98e5ace5d7dde06ff988b2d33e9f3813e4474
MD5 cbd452fa22702f48199adde7da6a7fb2
BLAKE2b-256 d775b406b984c6221a1cf37687f1a63b75aa71d6700ecdf34271ae7b45f68945

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page