Azure Cognitive Services Computer Vision SDK for Python

The Computer Vision service provides developers with access to advanced algorithms for processing images and returning information. Computer Vision algorithms analyze the content of an image in different ways, depending on the visual features you're interested in.

You can use Computer Vision in your application to:

• Analyze images for insight
• Extract text from images
• Generate thumbnails

Looking for more documentation?

• SDK reference documentation
• Cognitive Services Computer Vision documentation

Prerequisites

• Azure subscription - Create a free account
• Azure Computer Vision resource
• Python 3.6+

If you need a Computer Vision API account, you can create one with this Azure CLI command:

RES_REGION=westeurope
RES_GROUP=<resourcegroup-name>
ACCT_NAME=<computervision-account-name>

az cognitiveservices account create \
    --resource-group $RES_GROUP \
    --name $ACCT_NAME \
    --location $RES_REGION \
    --kind ComputerVision \
    --sku S1 \
    --yes

Installation

Install the Azure Cognitive Services Computer Vision SDK with pip, optionally within a virtual environment.

Configure a virtual environment (optional)

Although not required, you can keep your base system and Azure SDK environments isolated from one another if you use a virtual environment. Execute the following commands to configure and then enter a virtual environment with venv, such as cogsrv-vision-env:

python3 -m venv cogsrv-vision-env
source cogsrv-vision-env/bin/activate

Install the SDK

Install the Azure Cognitive Services Computer Vision SDK for Python package with pip:

pip install azure-cognitiveservices-vision-computervision

Authentication

Once you create your Computer Vision resource, you need its region, and one of its account keys to instantiate the client object.

Use these values when you create the instance of the ComputerVisionClient client object.

Get credentials

Use the Azure CLI snippet below to populate two environment variables with the Computer Vision account region and one of its keys (you can also find these values in the Azure portal). The snippet is formatted for the Bash shell.

RES_GROUP=<resourcegroup-name>
ACCT_NAME=<computervision-account-name>

export ACCOUNT_REGION=$(az cognitiveservices account show \
    --resource-group $RES_GROUP \
    --name $ACCT_NAME \
    --query location \
    --output tsv)

export ACCOUNT_KEY=$(az cognitiveservices account keys list \
    --resource-group $RES_GROUP \
    --name $ACCT_NAME \
    --query key1 \
    --output tsv)

Create client

Once you've populated the ACCOUNT_REGION and ACCOUNT_KEY environment variables, you can create the ComputerVisionClient client object.

from azure.cognitiveservices.vision.computervision import ComputerVisionClient
from azure.cognitiveservices.vision.computervision.models import VisualFeatureTypes
from msrest.authentication import CognitiveServicesCredentials

import os
region = os.environ['ACCOUNT_REGION']
key = os.environ['ACCOUNT_KEY']

credentials = CognitiveServicesCredentials(key)
client = ComputerVisionClient(
    endpoint="https://" + region + ".api.cognitive.microsoft.com/",
    credentials=credentials
)

Usage

Once you've initialized a ComputerVisionClient client object, you can:

• Analyze an image: You can analyze an image for certain features such as faces, colors, tags.
• Generate thumbnails: Create a custom JPEG image to use as a thumbnail of the original image.
• Get description of an image: Get a description of the image based on its subject domain.

For more information about this service, see What is Computer Vision?.

Examples

The following sections provide several code snippets covering some of the most common Computer Vision tasks, including:

• Analyze an image
• Get subject domain list
• Analyze an image by domain
• Get text description of an image
• Get handwritten text from image
• Generate thumbnail

Analyze an image

You can analyze an image for certain features with analyze_image. Use the visual_features property to set the types of analysis to perform on the image. Common values are VisualFeatureTypes.tags and VisualFeatureTypes.description.

url = "https://upload.wikimedia.org/wikipedia/commons/thumb/1/12/Broadway_and_Times_Square_by_night.jpg/450px-Broadway_and_Times_Square_by_night.jpg"

image_analysis = client.analyze_image(url,visual_features=[VisualFeatureTypes.tags])

for tag in image_analysis.tags:
    print(tag)

Get subject domain list

Review the subject domains used to analyze your image with list_models. These domain names are used when analyzing an image by domain. An example of a domain is landmarks.

models = client.list_models()

for x in models.models_property:
    print(x)

Analyze an image by domain

You can analyze an image by subject domain with analyze_image_by_domain. Get the list of supported subject domains in order to use the correct domain name.

domain = "landmarks"
url = "https://images.pexels.com/photos/338515/pexels-photo-338515.jpeg"
language = "en"

analysis = client.analyze_image_by_domain(domain, url, language)

for landmark in analysis.result["landmarks"]:
    print(landmark["name"])
    print(landmark["confidence"])

Get text description of an image

You can get a language-based text description of an image with describe_image. Request several descriptions with the max_description property if you are doing text analysis for keywords associated with the image. Examples of a text description for the following image include a train crossing a bridge over a body of water, a large bridge over a body of water, and a train crossing a bridge over a large body of water.

domain = "landmarks"
url = "http://www.public-domain-photos.com/free-stock-photos-4/travel/san-francisco/golden-gate-bridge-in-san-francisco.jpg"
language = "en"
max_descriptions = 3

analysis = client.describe_image(url, max_descriptions, language)

for caption in analysis.captions:
    print(caption.text)
    print(caption.confidence)

Get text from image

You can get any handwritten or printed text from an image. This requires two calls to the SDK: read and get_read_result. The call to read is asynchronous. In the results of the get_read_result call, you need to check if the first call completed with OperationStatusCodes before extracting the text data. The results include the text as well as the bounding box coordinates for the text.

# import models
from azure.cognitiveservices.vision.computervision.models import OperationStatusCodes

url = "https://github.com/Azure-Samples/cognitive-services-python-sdk-samples/raw/master/samples/vision/images/make_things_happen.jpg"
raw = True
numberOfCharsInOperationId = 36

# SDK call
rawHttpResponse = client.read(url, language="en", raw=True)

# Get ID from returned headers
operationLocation = rawHttpResponse.headers["Operation-Location"]
idLocation = len(operationLocation) - numberOfCharsInOperationId
operationId = operationLocation[idLocation:]

# SDK call
result = client.get_read_result(operationId)

# Get data
if result.status == OperationStatusCodes.succeeded:

    for line in result.analyze_result.read_results[0].lines:
        print(line.text)
        print(line.bounding_box)

Generate thumbnail

You can generate a thumbnail (JPG) of an image with generate_thumbnail. The thumbnail does not need to be in the same proportions as the original image.

This example uses the Pillow package to save the new thumbnail image locally.

from PIL import Image
import io

width = 50
height = 50
url = "http://www.public-domain-photos.com/free-stock-photos-4/travel/san-francisco/golden-gate-bridge-in-san-francisco.jpg"

thumbnail = client.generate_thumbnail(width, height, url)

for x in thumbnail:
    image = Image.open(io.BytesIO(x))

image.save('thumbnail.jpg')

Troubleshooting

General

When you interact with the ComputerVisionClient client object using the Python SDK, the ComputerVisionErrorException class is used to return errors. Errors returned by the service correspond to the same HTTP status codes returned for REST API requests.

For example, if you try to analyze an image with an invalid key, a 401 error is returned. In the following snippet, the error is handled gracefully by catching the exception and displaying additional information about the error.

domain = "landmarks"
url = "http://www.public-domain-photos.com/free-stock-photos-4/travel/san-francisco/golden-gate-bridge-in-san-francisco.jpg"
language = "en"
max_descriptions = 3

try:
    analysis = client.describe_image(url, max_descriptions, language)

    for caption in analysis.captions:
        print(caption.text)
        print(caption.confidence)
except HTTPFailure as e:
    if e.status_code == 401:
        print("Error unauthorized. Make sure your key and region are correct.")
    else:
        raise

Handle transient errors with retries

While working with the ComputerVisionClient client, you might encounter transient failures caused by rate limits enforced by the service, or other transient problems like network outages. For information about handling these types of failures, see Retry pattern in the Cloud Design Patterns guide, and the related Circuit Breaker pattern.

Next steps

More sample code

Several Computer Vision Python SDK samples are available to you in the SDK's GitHub repository. These samples provide example code for additional scenarios commonly encountered while working with Computer Vision:

• See sample repo

Additional documentation

For more extensive documentation on the Computer Vision service, see the Azure Computer Vision documentation on docs.microsoft.com.

Release History

0.6.0 (2020-05-18)

Features

• Model Line has a new parameter language
• Added operation ComputerVisionClientOperationsMixin.read
• Added operation ComputerVisionClientOperationsMixin.get_read_result
• Added operation ComputerVisionClientOperationsMixin.read_in_stream

Breaking changes

• Parameter words of model Line is now required
• Parameter bounding_box of model Line is now required
• Parameter text of model Line is now required
• Parameter confidence of model Word is now required
• Removed operation ComputerVisionClientOperationsMixin.get_text_operation_result
• Removed operation ComputerVisionClientOperationsMixin.get_read_operation_result
• Removed operation ComputerVisionClientOperationsMixin.recognize_text_in_stream
• Removed operation ComputerVisionClientOperationsMixin.recognize_text
• Removed operation ComputerVisionClientOperationsMixin.batch_read_file
• Removed operation ComputerVisionClientOperationsMixin.batch_read_file_in_stream
• Model ReadOperationResult has a new signature

0.5.0 (2019-10-01)

Features

• Model AdultInfo has a new parameter is_gory_content
• Model AdultInfo has a new parameter gore_score

Breaking changes

• Operation ComputerVisionClientOperationsMixin.analyze_image has a new signature
• Operation ComputerVisionClientOperationsMixin.analyze_image_in_stream has a new signature
• Operation ComputerVisionClientOperationsMixin.describe_image has a new signature
• Operation ComputerVisionClientOperationsMixin.describe_image_in_stream has a new signature

0.4.0 (2019-06-27)

Breaking changes

• "batch_read_file" and "batch_read_file_in_stream" have no "mode" parameter anymore

Bugfix

• "bounding_box" now supports float numbers
• Incorrect "Not Started" typo for state reporting

0.3.0 (2019-03-11)

Features

• Model ImageAnalysis has a new parameter brands
• Model ImageAnalysis has a new parameter objects
• Model Word has a new parameter confidence

Breaking changes

• Client ComputerVisionAPI has been renamed ComputerVisionClient
• Parameter text of model Word is now required
• Parameter bounding_box of model Word is now required

0.2.0 (2018-06-22)

Features

• analyze_image now support 'en', 'es', 'ja', 'pt', 'zh' (including "in_stream" version of these operations)
• describe_image/tag_image/analyze_image_by_domain now support the language parameter (including "in_stream" version of these operations)
• Client class can be used as a context manager to keep the underlying HTTP session open for performance

Bug fixes

• Fix several invalid JSON description, that was raising unexpected exceptions (including OCRResult from bug #2614)

Breaking changes

• recognize_text "detect_handwriting" boolean is now a "mode" str between 'Handwritten' and 'Printed'

General Breaking changes

This version uses a next-generation code generator that might introduce breaking changes.

• Model signatures now use only keyword-argument syntax. All positional arguments must be re-written as keyword-arguments. To keep auto-completion in most cases, models are now generated for Python 2 and Python 3. Python 3 uses the "*" syntax for keyword-only arguments.
• Enum types now use the "str" mixin (class AzureEnum(str, Enum)) to improve the behavior when unrecognized enum values are encountered. While this is not a breaking change, the distinctions are important, and are documented here: https://docs.python.org/3/library/enum.html#others At a glance:
  • "is" should not be used at all.
  • "format" will return the string value, where "%s" string formatting will return NameOfEnum.stringvalue. Format syntax should be prefered.
• New Long Running Operation:
  • Return type changes from msrestazure.azure_operation.AzureOperationPoller to msrest.polling.LROPoller. External API is the same.
  • Return type is now always a msrest.polling.LROPoller, regardless of the optional parameters used.
  • The behavior has changed when using raw=True. Instead of returning the initial call result as ClientRawResponse, without polling, now this returns an LROPoller. After polling, the final resource will be returned as a ClientRawResponse.
  • New polling parameter. The default behavior is Polling=True which will poll using ARM algorithm. When Polling=False, the response of the initial call will be returned without polling.
  • polling parameter accepts instances of subclasses of msrest.polling.PollingMethod.
  • add_done_callback will no longer raise if called after polling is finished, but will instead execute the callback right away.

0.1.0 (2018-01-23)

• Initial Release