Azure Schema Registry Avro Encoder client library for Python

Azure Schema Registry is a schema repository service hosted by Azure Event Hubs, providing schema storage, versioning, and management. This package provides an Avro encoder capable of encoding and decoding payloads containing Schema Registry schema identifiers and Avro-encoded content.

Source code | Package (PyPi) | API reference documentation | Samples | Changelog

Disclaimer

Azure SDK Python packages support for Python 2.7 has ended 01 January 2022. For more information and questions, please refer to https://github.com/Azure/azure-sdk-for-python/issues/20691

Getting started

Install the package

Install the Azure Schema Registry Avro Encoder client library and Azure Identity client library for Python with pip:

pip install azure-schemaregistry-avroencoder azure-identity

Prerequisites:

To use this package, you must have:

• Azure subscription - Create a free account
• Azure Schema Registry - Here is the quickstart guide to create a Schema Registry group using the Azure portal.
• Python 3.6 or later - Install Python

Authenticate the client

Interaction with the Schema Registry Avro Encoder starts with an instance of AvroEncoder class, which takes the schema group name and the Schema Registry Client class. The client constructor takes the Event Hubs fully qualified namespace and and Azure Active Directory credential:

• The fully qualified namespace of the Schema Registry instance should follow the format: <yournamespace>.servicebus.windows.net.

• An AAD credential that implements the TokenCredential protocol should be passed to the constructor. There are implementations of the TokenCredential protocol available in the azure-identity package. To use the credential types provided by azure-identity, please install the Azure Identity client library for Python with pip:

pip install azure-identity

• Additionally, to use the async API, you must first install an async transport, such as aiohttp:

pip install aiohttp

Create AvroEncoder using the azure-schemaregistry library:

from azure.schemaregistry import SchemaRegistryClient
from azure.schemaregistry.encoder.avroencoder import AvroEncoder
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
# Namespace should be similar to: '<your-eventhub-namespace>.servicebus.windows.net'
fully_qualified_namespace = '<< FULLY QUALIFIED NAMESPACE OF THE SCHEMA REGISTRY >>'
group_name = '<< GROUP NAME OF THE SCHEMA >>'
schema_registry_client = SchemaRegistryClient(fully_qualified_namespace, credential)
encoder = AvroEncoder(client=schema_registry_client, group_name=group_name)

Key concepts

AvroEncoder

Provides API to encode to and decode from Avro Binary Encoding plus a content type with schema ID. Uses SchemaRegistryClient to get schema IDs from schema content or vice versa.

Supported message models

Support has been added to certain Azure Messaging SDK model classes for interoperability with the AvroEncoder. These models are subtypes of the MessageType protocol defined under the azure.schemaregistry.encoder.avroencoder namespace. Currently, the supported model classes are:

• azure.eventhub.EventData for azure-eventhub==5.9.0b3

Message format

If a message type that follows the MessageType protocol is provided to the encoder, it will encode the corresponding content and content type properties as follows:

• content: Avro payload (in general, format-specific payload)

  • Avro Binary Encoding
  • NOT Avro Object Container File, which includes the schema and defeats the purpose of this encoder to move the schema out of the message payload and into the schema registry.

• content type: a string of the format avro/binary+<schema ID>, where:

  • avro/binary is the format indicator
  • <schema ID> is the hexadecimal representation of GUID, same format and byte order as the string from the Schema Registry service.

If message type is not provided, and by default, the encoder will create the following dict: {"content": <Avro encoded payload>, "content_type": 'avro/binary+<schema ID>' }

Examples

The following sections provide several code snippets covering some of the most common Schema Registry tasks, including:

• Encoding
• Decoding
• Event Hubs Sending Integration
• Event Hubs Receiving Integration

Encoding

Use AvroEncoder.encode method to encode dict content with the given Avro schema. The method will use a schema previously registered to the Schema Registry service and keep the schema cached for future encoding usage. It is also possible to avoid pre-registering the schema to the service and automatically register with the encode method by instantiating the AvroEncoder with the keyword argument auto_register=True.

import os
from azure.schemaregistry import SchemaRegistryClient
from azure.schemaregistry.encoder.avroencoder import AvroEncoder
from azure.identity import DefaultAzureCredential
from azure.eventhub import EventData

token_credential = DefaultAzureCredential()
fully_qualified_namespace = os.environ['SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE']
group_name = "<your-group-name>"
name = "example.avro.User"
format = "Avro"

definition = """
{"namespace": "example.avro",
 "type": "record",
 "name": "User",
 "fields": [
     {"name": "name", "type": "string"},
     {"name": "favorite_number",  "type": ["int", "null"]},
     {"name": "favorite_color", "type": ["string", "null"]}
 ]
}"""

schema_registry_client = SchemaRegistryClient(fully_qualified_namespace, token_credential)
schema_register_client.register(group_name, name, definition, format)
encoder = AvroEncoder(client=schema_registry_client, group_name=group_name)

with encoder:
    dict_content = {"name": "Ben", "favorite_number": 7, "favorite_color": "red"}
    event_data = encoder.encode(dict_content, schema=definition, message_type=EventData)

    # OR

    message_content_dict = encoder.encode(dict_content, schema=definition)
    event_data = EventData.from_message_content(message_content_dict["content"], message_content_dict["content_type"])

Decoding

Use AvroEncoder.decode method to decode the bytes value into dict content by either:

• Passing in a message object that is a subtype of the MessageType protocol.
• Passing in a dict with keys content(type bytes) and content_type (type string). The method automatically retrieves the schema from the Schema Registry Service and keeps the schema cached for future decoding usage.

import os
from azure.schemaregistry import SchemaRegistryClient
from azure.schemaregistry.encoder.avroencoder import AvroEncoder
from azure.identity import DefaultAzureCredential

token_credential = DefaultAzureCredential()
fully_qualified_namespace = os.environ['SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE']
group_name = "<your-group-name>"

schema_registry_client = SchemaRegistryClient(fully_qualified_namespace, token_credential)
encoder = AvroEncoder(client=schema_registry_client, group_name=group_name)

with encoder:
    # event_data is an EventData object with Avro encoded body
    decoded_content = encoder.decode(event_data)

    # OR 

    encoded_bytes = b'<content_encoded_by_azure_schema_registry_avro_encoder>'
    content_type = 'avro/binary+<schema_id_of_corresponding_schema>'
    content_dict = {"content": encoded_bytes, "content_type": content_type}
    decoded_content = encoder.decode(content_dict)

Event Hubs Sending Integration

Integration with Event Hubs to send encoded Avro dict content as the body of EventData.

import os
from azure.eventhub import EventHubProducerClient, EventData
from azure.schemaregistry import SchemaRegistryClient
from azure.schemaregistry.encoder.avroencoder import AvroEncoder
from azure.identity import DefaultAzureCredential

token_credential = DefaultAzureCredential()
fully_qualified_namespace = os.environ['SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE']
group_name = "<your-group-name>"
eventhub_connection_str = os.environ['EVENT_HUB_CONN_STR']
eventhub_name = os.environ['EVENT_HUB_NAME']

definition = """
{"namespace": "example.avro",
 "type": "record",
 "name": "User",
 "fields": [
     {"name": "name", "type": "string"},
     {"name": "favorite_number",  "type": ["int", "null"]},
     {"name": "favorite_color", "type": ["string", "null"]}
 ]
}"""

schema_registry_client = SchemaRegistryClient(fully_qualified_namespace, token_credential)
avro_encoder = AvroEncoder(client=schema_registry_client, group_name=group_name, auto_register=True)

eventhub_producer = EventHubProducerClient.from_connection_string(
    conn_str=eventhub_connection_str,
    eventhub_name=eventhub_name
)

with eventhub_producer, avro_encoder:
    event_data_batch = eventhub_producer.create_batch()
    dict_content = {"name": "Bob", "favorite_number": 7, "favorite_color": "red"}
    event_data = avro_encoder.encode(dict_content, schema=definition, message_type=EventData)
    event_data_batch.add(event_data)
    eventhub_producer.send_batch(event_data_batch)

Event Hubs Receiving Integration

Integration with Event Hubs to receive EventData and decoded raw bytes into Avro dict content.

import os
from azure.eventhub import EventHubConsumerClient
from azure.schemaregistry import SchemaRegistryClient
from azure.schemaregistry.encoder.avroencoder import AvroEncoder
from azure.identity import DefaultAzureCredential

token_credential = DefaultAzureCredential()
fully_qualified_namespace = os.environ['SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE']
group_name = "<your-group-name>"
eventhub_connection_str = os.environ['EVENT_HUB_CONN_STR']
eventhub_name = os.environ['EVENT_HUB_NAME']

schema_registry_client = SchemaRegistryClient(fully_qualified_namespace, token_credential)
avro_encoder = AvroEncoder(client=schema_registry_client, group_name=group_name)

eventhub_consumer = EventHubConsumerClient.from_connection_string(
    conn_str=eventhub_connection_str,
    consumer_group='$Default',
    eventhub_name=eventhub_name,
)

def on_event(partition_context, event):
    decoded_content = avro_encoder.decode(event)

with eventhub_consumer, avro_encoder:
    eventhub_consumer.receive(on_event=on_event, starting_position="-1")

Troubleshooting

General

Azure Schema Registry Avro Encoder raises exceptions defined in Azure Core.

Logging

This library uses the standard logging library for logging. Basic information about HTTP sessions (URLs, headers, etc.) is logged at INFO level.

Detailed DEBUG level logging, including request/response bodies and unredacted headers, can be enabled on a client with the logging_enable argument:

import sys
import logging
from azure.schemaregistry import SchemaRegistryClient
from azure.schemaregistry.encoder.avroencoder import AvroEncoder
from azure.identity import DefaultAzureCredential

# Create a logger for the SDK
logger = logging.getLogger('azure.schemaregistry')
logger.setLevel(logging.DEBUG)

# Configure a console output
handler = logging.StreamHandler(stream=sys.stdout)
logger.addHandler(handler)

credential = DefaultAzureCredential()
schema_registry_client = SchemaRegistryClient("<your-fully_qualified_namespace>", credential, logging_enable=True)
# This client will log detailed information about its HTTP sessions, at DEBUG level
encoder = AvroEncoder(client=schema_registry_client, group_name="<your-group-name>")

Similarly, logging_enable can enable detailed logging for a single operation, even when it isn't enabled for the client:

encoder.encode(dict_content, schema=schema_definition, logging_enable=True)

Next steps

More sample code

Please find further examples in the samples directory demonstrating common Azure Schema Registry Avro Encoder scenarios.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Release History

1.0.0b3 (2022-04-05)

Breaking Changes

• auto_register_schemas keyword in the sync and async AvroEncoder constructors has been renamed auto_register.
• SchemaParseError, SchemaEncodeError, and SchemaDecodeError have been replaced with InvalidContentError and InvalidSchemaError. The errors have been added under the azure.schemaregistry.encoder.avroencoder namespace.
• The exceptions module in azure.schemaregistry.encoder.avroencoder has been removed.
• The encode method on the sync and async AvroEncoder only allows subtypes of the MessageType protocol as values to the message_type optional parameter, rather than any callable that has the method signature (content: bytes, content_type: str, **kwargs: Any).
• The number of hits/misses, in addition to number of entries, for the schema/schema ID caches will be logged at an info level when a new entry is added.

Other Changes

• This release and future releases will not have backward compatibility support for decoding data that was encoded with the AvroSerializer.
• The encode and decode methods on AvroEncoder support the following message models:
  • azure.eventhub.EventData in azure-eventhub==5.9.0b3

1.0.0b2 (2022-03-09)

Features Added

• request_options has been added to encode and decode on AvroEncoder as an optional parameter to be passed into client requests.
• The size of the current schema/schema ID caches will be logged at an info level when a new entry has been added.

Breaking Changes

• MessageMetadataDict has been renamed MessageContent.
• data in MessageContent has been renamed content.
• The data parameter in encode and decode on the sync and async AvroEncoder has been renamed content.
• The from_message_data method in the MessageType protocol has been renamed from_message_content. The data parameter in from_message_content has been renamed content.
• The __message_data__ method in the MessageType protocol has been renamed __message_content__.

Other Changes

• This beta release will be backward compatible for decoding data that was encoded with the AvroSerializer.
• The encode and decode methods on AvroEncoder support the following message models:
  • azure.eventhub.EventData in azure-eventhub==5.9.0b2

1.0.0b1 (2022-02-09)

This version and all future versions will require Python 3.6+. Python 2.7 is no longer supported.

Features Added

• This package is meant to replace the azure-schemaregistry-avroserializer.
• APIs have been updated to allow for encoding directly to and decoding from message type objects, where the data value is the Avro encoded payload.
• The content type of the message will hold the schema ID and record format indicator.

Other Changes

• This beta release will be backward compatible for decoding data that was encoded with the AvroSerializer.
• The encode and decode methods on AvroEncoder support the following message models:
  • azure.eventhub.EventData in azure-eventhub==5.9.0b1