Azure Text Analytics client library for Python

Text Analytics is a cloud-based service that provides advanced natural language processing over raw text, and includes six main functions:

• Sentiment Analysis
• Named Entity Recognition
• Personally Identifiable Information (PII) Entity Recognition
• Linked Entity Recognition
• Language Detection
• Key Phrase Extraction

Source code | Package (PyPI) | API reference documentation | Product documentation | Samples

Getting started

Prerequisites

• Python 2.7, or 3.5 or later is required to use this package.
• You must have an Azure subscription and a Cognitive Services or Text Analytics resource to use this package.

Install the package

Install the Azure Text Analytics client library for Python with pip:

pip install azure-ai-textanalytics --pre

Create a Cognitive Services or Text Analytics resource

Text Analytics supports both multi-service and single-service access. Create a Cognitive Services resource if you plan to access multiple cognitive services under a single endpoint/key. For Text Analytics access only, create a Text Analytics resource.

You can create either resource using the Azure Portal or Azure CLI. Below is an example of how you can create a Text Analytics resource using the CLI:

# Create a new resource group to hold the text analytics resource -
# if using an existing resource group, skip this step
az group create --name my-resource-group --location westus2

# Create text analytics
az cognitiveservices account create \
    --name text-analytics-resource \
    --resource-group my-resource-group \
    --kind TextAnalytics \
    --sku F0 \
    --location westus2 \
    --yes

Authenticate the client

Interaction with this service begins with an instance of a client. To create a client object, you will need the cognitive services or text analytics endpoint to your resource and a credential that allows you access:

from azure.ai.textanalytics import TextAnalyticsClient

text_analytics = TextAnalyticsClient(endpoint="https://westus2.api.cognitive.microsoft.com/", credential=credential)

Note that if you create a custom subdomain name for your resource the endpoint may look different than in the above code snippet. For example, https://<my-custom-subdomain>.cognitiveservices.azure.com/.

Looking up the endpoint

You can find the endpoint for your text analytics resource using the Azure Portal or Azure CLI:

# Get the endpoint for the text analytics resource
az cognitiveservices account show --name "resource-name" --resource-group "resource-group-name" --query "endpoint"

Types of credentials

The credential parameter may be provided as the subscription key to your resource or as a token from Azure Active Directory. See the full details regarding authentication of cognitive services.

1. To use a subscription key, provide the key as a string. This can be found in the Azure Portal under the "Quickstart" section or by running the following Azure CLI command:

   az cognitiveservices account keys list --name "resource-name" --resource-group "resource-group-name"

   Use the key as the credential parameter to authenticate the client:

   from azure.ai.textanalytics import TextAnalyticsClient
   text = TextAnalyticsClient(endpoint="https://westus2.api.cognitive.microsoft.com/", credential="<subscription_key>")
   

2. To use an Azure Active Directory (AAD) token credential, provide an instance of the desired credential type obtained from the azure-identity library. Note that regional endpoints do not support AAD authentication. Create a custom subdomain name for your resource in order to use this type of authentication.

   Authentication with AAD requires some initial setup:

   • Install azure-identity
   • Register a new AAD application
   • Grant access to Text Analytics by assigning the "Cognitive Services User" role to your service principal.

   After setup, you can choose which type of credential from azure.identity to use. As an example, DefaultAzureCredential can be used to authenticate the client:

   Set the values of the client ID, tenant ID, and client secret of the AAD application as environment variables: AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_CLIENT_SECRET

   Use the returned token credential to authenticate the client:

   from azure.identity import DefaultAzureCredential
   from azure.ai.textanalytics import TextAnalyticsClient
   token_credential = DefaultAzureCredential()
   
   text_analytics_client = TextAnalyticsClient(
       endpoint="https://<my-custom-subdomain>.cognitiveservices.azure.com/",
       credential=token_credential
   )
   

Key concepts

Client

The Text Analytics client library provides a TextAnalyticsClient to do analysis on batches of documents. It provides both synchronous and asynchronous operations to access a specific use of Text Analytics, such as language detection or key phrase extraction.

Single text operations

The Text Analytics client library also provides module-level operations which can be performed on a single string rather than a batch of documents. Each synchronous and asynchronous batching operation has a singular counterpart. The endpoint and credential are passed in with the desired text and other optional parameters, e.g. single_analyze_sentiment():

from azure.ai.textanalytics import single_analyze_sentiment

text = "I did not like the restaurant. The food was too spicy."
result = single_analyze_sentiment(endpoint=endpoint, credential=credential, input_text=text, language="en")

Input

A document is a single unit to be analyzed by the predictive models in the Text Analytics service. For the single text operations, the input document is simply passed as a string, e.g. "hello world". For the batched operations, the input is passed as a list of documents.

Documents can be passed as a list of strings, e.g.

docs = ["I hated the movie. It was so slow!", "The movie made it into my top ten favorites.", "What a great movie!"]

or, if you wish to pass in a per-item document id or language/country_hint, they can be passed as a list of DetectLanguageInput or TextDocumentInput, or a dict-like representation of the object:

documents = [
    {"id": "1", "language": "en", "text": "I hated the movie. It was so slow!"}, 
    {"id": "2", "language": "en", "text": "The movie made it into my top ten favorites."}, 
    {"id": "3", "language": "en", "text": "What a great movie!"}
]

Operation Result

An operation result, such as AnalyzeSentimentResult, is the result of a Text Analytics operation and contains a prediction or predictions about a document input. With a batching operation, a list is returned containing a collection of operation results and any document errors. These results/errors will be index-matched with the order of the provided documents.

Examples

The following section provides several code snippets covering some of the most common Text Analytics tasks, including:

• Analyze Sentiment
• Recognize Entities
• Recognize PII Entities
• Recognize Linked Entities
• Extract Key Phrases
• Detect Languages

Analyze sentiment

Analyze sentiment in a batch of documents.

from azure.ai.textanalytics import TextAnalyticsClient

text_analytics_client = TextAnalyticsClient(endpoint, key)

documents = [
    "I did not like the restaurant. The food was too spicy.",
    "The restaurant was decorated beautifully. The atmosphere was unlike any other restaurant I've been to.",
    "The food was yummy. :)"
]

response = text_analytics_client.analyze_sentiment(documents, language="en")
result = [doc for doc in response if not doc.is_error]

for doc in result:
    print("Overall sentiment: {}".format(doc.sentiment))
    print("Scores: positive={0:.3f}; neutral={1:.3f}; negative={2:.3f} \n".format(
        doc.document_scores.positive,
        doc.document_scores.neutral,
        doc.document_scores.negative,
    ))

Please refer to the service documentation for a conceptual discussion of sentiment analysis.

Recognize entities

Recognize entities in a batch of documents.

from azure.ai.textanalytics import TextAnalyticsClient

text_analytics_client = TextAnalyticsClient(endpoint, key)

documents = [
    "Microsoft was founded by Bill Gates and Paul Allen.",
    "Redmond is a city in King County, Washington, United States, located 15 miles east of Seattle.",
    "Jeff bought three dozen eggs because there was a 50% discount."
]

response = text_analytics_client.recognize_entities(documents, language="en")
result = [doc for doc in response if not doc.is_error]

for doc in result:
    for entity in doc.entities:
        print("Entity: \t", entity.text, "\tType: \t", entity.type,
              "\tConfidence Score: \t", entity.score)

Please refer to the service documentation for a conceptual discussion of named entity recognition and supported types.

Recognize PII entities

Recognize PII entities in a batch of documents.

from azure.ai.textanalytics import TextAnalyticsClient

text_analytics_client = TextAnalyticsClient(endpoint, key)

documents = [
    "The employee's SSN is 555-55-5555.",
    "The employee's phone number is 555-55-5555."
]

response = text_analytics_client.recognize_pii_entities(documents, language="en")
result = [doc for doc in response if not doc.is_error]

for doc in result:
    for entity in doc.entities:
        print("Entity: \t", entity.text, "\tType: \t", entity.type,
              "\tConfidence Score: \t", entity.score)

Please refer to the service documentation for supported PII entity types.

Recognize linked entities

Recognize linked entities in a batch of documents.

from azure.ai.textanalytics import TextAnalyticsClient

text_analytics_client = TextAnalyticsClient(endpoint, key)

documents = [
    "Microsoft was founded by Bill Gates and Paul Allen.",
    "Easter Island, a Chilean territory, is a remote volcanic island in Polynesia."
]

response = text_analytics_client.recognize_linked_entities(documents, language="en")
result = [doc for doc in response if not doc.is_error]

for doc in result:
    for entity in doc.entities:
        print("Entity: {}".format(entity.name))
        print("Url: {}".format(entity.url))
        print("Data Source: {}".format(entity.data_source))
        for match in entity.matches:
            print("Score: {0:.3f}".format(match.score))
            print("Offset: {}".format(match.offset))
            print("Length: {}\n".format(match.length))

Please refer to the service documentation for a conceptual discussion of entity linking and supported types.

Extract key phrases

Extract key phrases in a batch of documents.

from azure.ai.textanalytics import TextAnalyticsClient

text_analytics_client = TextAnalyticsClient(endpoint, key)

documents = [
    "Redmond is a city in King County, Washington, United States, located 15 miles east of Seattle.",
    "I need to take my cat to the veterinarian.",
    "I will travel to South America in the summer."
]

response = text_analytics_client.extract_key_phrases(documents, language="en")
result = [doc for doc in response if not doc.is_error]

for doc in result:
    print(doc.key_phrases)

Please refer to the service documentation for a conceptual discussion of key phrase extraction.

Detect languages

Detect language in a batch of documents.

from azure.ai.textanalytics import TextAnalyticsClient

text_analytics_client = TextAnalyticsClient(endpoint, key)

documents = [
    "This is written in English.",
    "Il documento scritto in italiano.",
    "Dies ist in englischer Sprache verfasst."
]

response = text_analytics_client.detect_languages(documents)
result = [doc for doc in response if not doc.is_error]

for doc in result:
    print("Language detected: {}".format(doc.primary_language.name))
    print("ISO6391 name: {}".format(doc.primary_language.iso6391_name))
    print("Confidence score: {}\n".format(doc.primary_language.score))

Please refer to the service documentation for a conceptual discussion of language detection and language and regional support.

Optional Configuration

Optional keyword arguments that can be passed in at the client and per-operation level.

Retry Policy configuration

Use the following keyword arguments when instantiating a client to configure the retry policy:

• retry_total (int): Total number of retries to allow. Takes precedence over other counts. Pass in retry_total=0 if you do not want to retry on requests. Defaults to 10.
• retry_connect (int): How many connection-related errors to retry on. Defaults to 3.
• retry_read (int): How many times to retry on read errors. Defaults to 3.
• retry_status (int): How many times to retry on bad status codes. Defaults to 3.

Other client / per-operation configuration

Other optional configuration keyword arguments that can be specified on the client or per-operation.

Client keyword arguments:

• connection_timeout (int): A single float in seconds for the connection timeout. Defaults to 300 seconds.
• read_timeout (int): A single float in seconds for the read timeout. Defaults to 300 seconds.
• transport (Any): User-provided transport to send the HTTP request.

Per-operation keyword arguments:

• response_hook (callable): The given callback uses the raw response returned from the service. Can also be passed in at the client.
• request_id (str): Optional user specified identification of the request.
• user_agent (str): Appends the custom value to the user-agent header to be sent with the request.
• logging_enable (bool): Enables logging at the DEBUG level. Defaults to False. Can also be passed in at the client level to enable it for all requests.
• headers (dict): Pass in custom headers as key, value pairs. E.g. headers={'CustomValue': value}

Troubleshooting

The Text Analytics client will raise exceptions defined in Azure Core.

Next steps

More sample code

These code samples show common scenario operations with the Azure Text Analytics client library. The async versions of the samples (the python sample files appended with _async) show asynchronous operations with Text Analytics and require Python 3.5 or later.

Authenticate the client with a Cognitive Services/Text Analytics subscription key or a token credential from azure-identity:

• sample_authentication.py (async version)

In a batch of documents:

• Detect languages: sample_detect_languages.py (async version)
• Recognize entities: sample_recognize_entities.py (async version)
• Recognize linked entities: sample_recognize_linked_entities.py (async version)
• Recognize personally identifiable information: sample_recognize_pii_entities.py (async version)
• Extract key phrases: sample_extract_key_phrases.py (async version)
• Analyze sentiment: sample_analyze_sentiment.py (async version)

In a single string of text:

• Detect language: sample_single_detect_language.py (async version)
• Recognize entities: sample_single_recognize_entities.py (async version)
• Recognize linked entities: sample_single_recognize_linked_entities.py (async version)
• Recognize personally identifiable information: sample_single_recognize_pii_entities.py (async version)
• Extract key phrases: sample_single_extract_key_phrases.py (async version)
• Analyze sentiment: sample_single_analyze_sentiment.py (async version)

Additional documentation

For more extensive documentation on Azure Cognitive Services Text Analytics, see the Text Analytics documentation on docs.microsoft.com.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Change Log azure-ai-textanalytics

1.0.0b1 (2020-01-09)

Version (1.0.0b1) is the first preview of our efforts to create a user-friendly and Pythonic client library for Azure Text Analytics. For more information about this, and preview releases of other Azure SDK libraries, please visit https://azure.github.io/azure-sdk/releases/latest/python.html.

Breaking changes: New API design

• New namespace/package name:

  • The namespace/package name for Azure Text Analytics client library has changed from azure.cognitiveservices.language.textanalytics to azure.ai.textanalytics

• New operations and naming:

  • detect_language is renamed to detect_languages
  • entities is renamed to recognize_entities
  • key_phrases is renamed to extract_key_phrases
  • sentiment is renamed to analyze_sentiment
  • New operation recognize_pii_entities finds personally identifiable information entities in text
  • New operation recognize_linked_entities provides links from a well-known knowledge base for each recognized entity
  • New module-level operations single_detect_language, single_recognize_entities, single_extract_key_phrases, single_analyze_sentiment, single_recognize_pii_entities, and single_recognize_linked_entities perform function on a single string instead of a batch of text documents and can be imported from the azure.ai.textanalytics namespace.
  • New client and module-level async APIs added to subnamespace azure.ai.textanalytics.aio.
  • MultiLanguageInput has been renamed to TextDocumentInput
  • LanguageInput has been renamed to DetectLanguageInput
  • DocumentLanguage has been renamed to DetectLanguageResult
  • DocumentEntities has been renamed to RecognizeEntitiesResult
  • DocumentLinkedEntities has been renamed to RecognizeLinkedEntitiesResult
  • DocumentKeyPhrases has been renamed to ExtractKeyPhrasesResult
  • DocumentSentiment has been renamed to AnalyzeSentimentResult
  • DocumentStatistics has been renamed to TextDocumentStatistics
  • RequestStatistics has been renamed to TextDocumentBatchStatistics
  • Entity has been renamed to NamedEntity
  • Match has been renamed to LinkedEntityMatch
  • The batching methods' documents parameter has been renamed inputs

• New input types:

  • detect_languages can take as input a list[DetectLanguageInput] or a list[str]. A list of dict-like objects in the same shape as DetectLanguageInput is still accepted as input.
  • recognize_entities, recognize_pii_entities, recognize_linked_entities, extract_key_phrases, analyze_sentiment can take as input a list[TextDocumentInput] or list[str]. A list of dict-like objects in the same shape as TextDocumentInput is still accepted as input.

• New parameters/keyword arguments:

  • All operations now take a keyword argument model_version which allows the user to specify a string referencing the desired model version to be used for analysis. If no string specified, it will default to the latest, non-preview version.
  • detect_languages now takes a parameter country_hint which allows you to specify the country hint for the entire batch. Any per-item country hints will take precedence over a whole batch hint.
  • recognize_entities, recognize_pii_entities, recognize_linked_entities, extract_key_phrases, analyze_sentiment now take a parameter language which allows you to specify the language for the entire batch. Any per-item specified language will take precedence over a whole batch hint.
  • A default_country_hint or default_language keyword argument can be passed at client instantiation to set the default values for all operations.
  • A response_hook keyword argument can be passed with a callback to use the raw response from the service. Additionally, values returned for TextDocumentBatchStatistics and model_version used must be retrieved using a response hook.
  • show_stats and model_version parameters move to keyword only arguments.

• New return types

  • The return types for the batching methods (detect_languages, recognize_entities, recognize_pii_entities, recognize_linked_entities, extract_key_phrases, analyze_sentiment) now return a heterogeneous list of result objects and document errors in the order passed in with the request. To iterate over the list and filter for result or error, a boolean property on each object called is_error can be used to determine whether the returned response object at that index is a result or an error:
  • detect_languages now returns a List[Union[DetectLanguageResult, DocumentError]]
  • recognize_entities now returns a List[Union[RecognizeEntitiesResult, DocumentError]]
  • recognize_pii_entities now returns a List[Union[RecognizePiiEntitiesResult, DocumentError]]
  • recognize_linked_entities now returns a List[Union[RecognizeLinkedEntitiesResult, DocumentError]]
  • extract_key_phrases now returns a List[Union[ExtractKeyPhrasesResult, DocumentError]]
  • analyze_sentiment now returns a List[Union[AnalyzeSentimentResult, DocumentError]]
  • The module-level, single text operations will return a single result object or raise the error found on the document:
  • single_detect_languages returns a DetectLanguageResult
  • single_recognize_entities returns a RecognizeEntitiesResult
  • single_recognize_pii_entities returns a RecognizePiiEntitiesResult
  • single_recognize_linked_entities returns a RecognizeLinkedEntitiesResult
  • single_extract_key_phrases returns a ExtractKeyPhrasesResult
  • single_analyze_sentiment returns a AnalyzeSentimentResult

• New underlying REST pipeline implementation, based on the new azure-core library.

• Client and pipeline configuration is now available via keyword arguments at both the client level, and per-operation. See README for a full list of optional configuration arguments.

• Authentication using azure-identity credentials

  • see the Azure Identity documentation for more information

• New error hierarchy:

  • All service errors will now use the base type: azure.core.exceptions.HttpResponseError
  • There is one exception type derived from this base type for authentication errors:
    • ClientAuthenticationError: Authentication failed.

0.2.0 (2019-03-12)

Features

• Client class can be used as a context manager to keep the underlying HTTP session open for performance
• New method "entities"
• Model KeyPhraseBatchResultItem has a new parameter statistics
• Model KeyPhraseBatchResult has a new parameter statistics
• Model LanguageBatchResult has a new parameter statistics
• Model LanguageBatchResultItem has a new parameter statistics
• Model SentimentBatchResult has a new parameter statistics

Breaking changes

• TextAnalyticsAPI main client has been renamed TextAnalyticsClient
• TextAnalyticsClient parameter is no longer a region but a complete endpoint

General Breaking changes

This version uses a next-generation code generator that might introduce breaking changes.

• Model signatures now use only keyword-argument syntax. All positional arguments must be re-written as keyword-arguments. To keep auto-completion in most cases, models are now generated for Python 2 and Python 3. Python 3 uses the "*" syntax for keyword-only arguments.

• Enum types now use the "str" mixin (class AzureEnum(str, Enum)) to improve the behavior when unrecognized enum values are encountered. While this is not a breaking change, the distinctions are important, and are documented here: https://docs.python.org/3/library/enum.html#others At a glance:

  • "is" should not be used at all.
  • "format" will return the string value, where "%s" string formatting will return NameOfEnum.stringvalue. Format syntax should be prefered.

Bugfixes

• Compatibility of the sdist with wheel 0.31.0

0.1.0 (2018-01-12)

• Initial Release