Azure Storage Queues client library for Python

Azure Storage Queues is a service for storing large numbers of messages that can be accessed from anywhere in the world via authenticated calls using HTTP or HTTPS. A single queue message can be up to 64 KB in size, and a queue can contain millions of messages, up to the total capacity limit of a storage account.

Common uses of Queue storage include:

• Creating a backlog of work to process asynchronously
• Passing messages between different parts of a distributed application

Source code | Package (PyPi) | API reference documentation | Product documentation | Samples

Getting started

Install the package

Install the Azure Storage Queue client library for Python with pip:

pip install azure-storage-queue --pre

Prerequisites: You must have an Azure subscription, and a Storage Account to use this package.

To create a Storage Account, you can use the Azure Portal, Azure PowerShell or Azure CLI:

az storage account create -n MyStorageAccountName -g MyResourceGroupName

Requires Python 2.7, 3.5 or later to use this package.

Authenticate the client

Interaction with Storage Queues starts with an instance of the QueueServiceClient class. You need an existing storage account, its URL, and a credential to instantiate the client object.

Get credentials

To authenticate the client you have a few options:

1. Use a SAS token string
2. Use an account shared access key
3. Use a token credential from azure.identity

Alternatively, you can authenticate with a storage connection string using the from_connection_string method. See example: Client creation with a connection string.

You can omit the credential if your account URL already has a SAS token.

Create client

Once you have your account URL and credentials ready, you can create the QueueServiceClient:

from azure.storage.queue import QueueServiceClient

service = QueueServiceClient(account_url="https://<my-storage-account-name>.queue.core.windows.net/", credential=credential)

Key concepts

The Queue service contains the following components:

• The storage account
• A queue which contains a set of messages
• A message, in any format, of up to 64 KB

Clients

The Storage Queues SDK provides two different clients to interact with the Queues Service:

1. QueueServiceClient - this client interacts with the Queue Service at the account level. It provides operations to retrieve and configure the account properties as well as list, create, and delete queues within the account. For operations relating to a specific queue, a client for that entity can also be retrieved using the get_queue_client function.
2. QueueClient - this client represents interaction with a specific queue, although that queue need not exist yet. It provides operations to create, delete, or configure queues and includes operations to enqueue, receive, peak, delete, and update messages in the queue.

Messages

Once you've initialized a Client, you can use the following operations to work with the messages in the queue:

• Enqueue - Adds a message to the queue and optionally sets a visibility timeout for the message.
• Receive - Retrieves a message from the queue and makes it invisible to other consumers.
• Peek - Retrieves a message from the front of the queue, without changing the message visibility.
• Update - Updates the visibility timeout of a message and/or the message contents.
• Delete - Deletes a specified message from the queue.
• Clear - Clears all messages from the queue.

Examples

The following sections provide several code snippets covering some of the most common Storage Queue tasks, including:

• Client creation with a connection string
• Create a queue
• Enqueue messages
• Receive messages

Client creation with a connection string

Create the QueueServiceClient using the connection string to your Azure Storage account.

from azure.storage.queue import QueueServiceClient

service = QueueServiceClient.from_connection_string(conn_str="my_connection_string")

Create a queue

Create a queue in your storage account.

from azure.storage.queue import QueueClient

queue = QueueClient.from_connection_string(conn_str="my_connection_string", queue="myqueue")
queue.create_queue()

Create a queue asynchronously.

from azure.storage.queue.aio import QueueClient

queue = QueueClient.from_connection_string(conn_str="my_connection_string", queue="myqueue")
await queue.create_queue()

Enqueue messages

Enqueue a message in your queue.

from azure.storage.queue import QueueClient

queue = QueueClient.from_connection_string(conn_str="my_connection_string", queue="myqueue")
queue.enqueue_message("I'm using queues!")
queue.enqueue_message("This is my second message")

Enqueue messages with an async client

from azure.storage.queue.aio import QueueClient

queue = QueueClient.from_connection_string(conn_str="my_connection_string", queue="myqueue")
await asyncio.gather(
    queue.enqueue_message("I'm using queues!"),
    queue.enqueue_message("This is my second message"))

Receive messages

Receive messages from your queue.

from azure.storage.queue import QueueClient

queue = QueueClient.from_connection_string(conn_str="my_connection_string", queue="myqueue")
response = queue.receive_messages()

for message in response:
    print(message.content)
    queue.delete_message(message)

# Printed messages from the front of the queue
# >>I'm using queues!   
# >>This is my second message

Receive messages by batch.

queue = QueueClient.from_connection_string(conn_str="my_connection_string", queue="myqueue")
response = queue.receive_messages(messages_per_page=10)

for message_batch in response.by_page():
    for message in message_batch:
        print(message.content)
        queue.delete_message(message)

Receive messages asynchronously:

from azure.storage.queue.aio import QueueClient

queue = QueueClient.from_connection_string(conn_str="my_connection_string", queue="myqueue")
response = queue.receive_messages()

async for message in response:
    print(message.content)
    await queue.delete_message(message)

Troubleshooting

Storage Queue clients raise exceptions defined in Azure Core.

All Queue service operations will throw a StorageErrorException on failure with helpful error codes.

Next steps

More sample code

Get started with our Queue samples.

Several Storage Queues Python SDK samples are available to you in the SDK's GitHub repository. These samples provide example code for additional scenarios commonly encountered while working with Storage Queues:

• test_queue_samples_hello_world.py (async version) - Examples found in this article:

  • Client creation
  • Create a queue
  • Enqueue messages
  • Receive messages

• test_queue_samples_authentication.py (async version) - Examples for authenticating and creating the client:

  • From a connection string
  • From a shared access key
  • From a shared access signature token
  • From active directory

• test_queue_samples_service.py (async version) - Examples for interacting with the queue service:

  • Get and set service properties
  • List queues in a storage account
  • Create and delete a queue from the service
  • Get the QueueClient

• test_queue_samples_message.py (async version) - Examples for working with queues and messages:

  • Set an access policy
  • Get and set queue metadata
  • Enqueue and receive messages
  • Delete specified messages and clear all messages
  • Peek and update messages

Additional documentation

For more extensive documentation on the Azure Storage Queues, see the Azure Storage Queues documentation on docs.microsoft.com.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Change Log azure-storage-queue

Version 12.0.0b3:

Dependency updates

• Adopted azure-core 1.0.0b3

  • If you later want to revert to previous versions of azure-storage-queue, or another Azure SDK library requiring azure-core 1.0.0b1 or azure-core 1.0.0b2, you must explicitly install the specific version of azure-core as well. For example:

  pip install azure-core==1.0.0b2 azure-storage-queue==12.0.0b2

Version 12.0.0b2:

Breaking changes

• The behavior of listing operations has been modified:
  • The previous marker parameter has been removed.
  • The iterable response object now supports a by_page function that will return a secondary iterator of batches of results. This function supports a continuation_token parameter to replace the previous marker parameter.
• The new listing behaviour is also adopted by the receive_messages operation:
  • The receive operation returns a message iterator as before.
  • The returned iterator supports a by_page operation to receive messages in batches.

New features

• Added async APIs to subnamespace azure.storage.queue.aio.
• Distributed tracing framework OpenCensus is now supported.

Dependency updates

• Adopted azure-core 1.0.0b2

  • If you later want to revert to azure-storage-queue 12.0.0b1, or another Azure SDK library requiring azure-core 1.0.0b1, you must explicitly install azure-core 1.0.0b1 as well. For example:

  pip install azure-core==1.0.0b1 azure-storage-queue==12.0.0b1

Fixes and improvements

• General refactor of duplicate and shared code.

Version 12.0.0b1:

Version 12.0.0b1 is the first preview of our efforts to create a user-friendly and Pythonic client library for Azure Storage Queues. For more information about this, and preview releases of other Azure SDK libraries, please visit https://aka.ms/azure-sdk-preview1-python.

Breaking changes: New API design

• Operations are now scoped to a particular client:

  • QueueServiceClient: This client handles account-level operations. This includes managing service properties and listing the queues within an account.
  • QueueClient: The client handles operations within a particular queue. This includes creating or deleting that queue, as well as enqueueing and dequeueing messages.

  These clients can be accessed by navigating down the client hierarchy, or instantiated directly using URLs to the resource (account or queue). For full details on the new API, please see the reference documentation.

• New message iterator, for receiving messages from a queue in a continuous stream.

• New underlying REST pipeline implementation, based on the new azure-core library.

• Client and pipeline configuration is now available via keyword arguments at both the client level, and per-operation. See reference documentation for a full list of optional configuration arguments.

• Authentication using azure-identity credentials

  • see the Azure Identity documentation for more information

• New error hierarchy:

  • All service errors will now use the base type: azure.core.exceptions.HttpResponseError
  • The are a couple of specific exception types derived from this base type for common error scenarios:
    • ResourceNotFoundError: The resource (e.g. queue, message) could not be found. Commonly a 404 status code.
    • ResourceExistsError: A resource conflict - commonly caused when attempting to create a resource that already exists.
    • ResourceModifiedError: The resource has been modified (e.g. overwritten) and therefore the current operation is in conflict. Alternatively this may be raised if a condition on the operation is not met.
    • ClientAuthenticationError: Authentication failed.

• No longer have specific operations for get_metadata - use get_properties instead.

• No longer have specific operations for exists - use get_properties instead.

• Operations get_queue_acl and set_queue_acl have been renamed to get_queue_access_policy and set_queue_access_policy.

• Operation put_message has been renamed to enqueue_message.

• Operation get_messages has been renamed to receive_messages.

Version 2.0.1:

• Updated dependency on azure-storage-common.

Version 2.0.0:

• Support for 2018-11-09 REST version.

Version 1.4.0:

• azure-storage-nspkg is not installed anymore on Python 3 (PEP420-based namespace package)

Version 1.3.0:

• Support for 2018-03-28 REST version. Please see our REST API documentation and blog for information about the related added features.

Version 1.2.0rc1:

• Support for 2017-11-09 REST version. Please see our REST API documentation and blog for information about the related added features.
• Added support for OAuth authentication for HTTPS requests(Please note that this feature is available in preview).

Version 1.1.0:

• Support for 2017-07-29 REST version. Please see our REST API documentation and blogs for information about the related added features.
• Queue messages can now have an arbitrarily large or infinite time to live.
• Error message now contains the ErrorCode from the x-ms-error-code header value.

Version 1.0.0:

• The package has switched from Apache 2.0 to the MIT license.