Provider for Apache Airflow. Implements apache-airflow-providers-microsoft-azure package
Project description
Package apache-airflow-providers-microsoft-azure
Release: 4.3.0rc1
Provider package
This is a provider package for microsoft.azure provider. All classes for this provider package are in airflow.providers.microsoft.azure python package.
You can find package information and changelog for the provider in the documentation.
Installation
You can install this package on top of an existing Airflow 2 installation (see Requirements below for the minimum Airflow version supported) via pip install apache-airflow-providers-microsoft-azure
The package supports the following python versions: 3.7,3.8,3.9,3.10
Requirements
PIP package |
Version required |
|---|---|
apache-airflow |
>=2.2.0 |
azure-batch |
>=8.0.0 |
azure-cosmos |
>=4.0.0 |
azure-datalake-store |
>=0.0.45 |
azure-identity |
>=1.3.1 |
azure-keyvault-secrets |
>=4.1.0,<5.0 |
azure-kusto-data |
>=0.0.43,<0.1 |
azure-mgmt-containerinstance |
>=1.5.0,<2.0 |
azure-mgmt-datafactory |
>=1.0.0,<2.0 |
azure-mgmt-datalake-store |
>=0.5.0 |
azure-mgmt-resource |
>=2.2.0 |
azure-storage-blob |
>=12.7.0,<12.9.0 |
azure-storage-common |
>=2.1.0 |
azure-storage-file |
>=2.1.0 |
azure-servicebus |
>=7.6.1; platform_machine != "aarch64" |
azure-synapse-spark |
Cross provider package dependencies
Those are dependencies that might be needed in order to use all the features of the package. You need to install the specified provider packages in order to use them.
You can install such cross-provider dependencies when installing from PyPI. For example:
pip install apache-airflow-providers-microsoft-azure[google]Dependent package |
Extra |
|---|---|
oracle |
|
sftp |
Changelog
4.3.0
Features
Add DataFlow operations to Azure DataFactory hook (#26345)
Add network_profile param in AzureContainerInstancesOperator (#26117)
Add Azure synapse operator (#26038)
Auto tail file logs in Web UI (#26169)
Implement Azure Service Bus Topic Create, Delete Operators (#25436)
Bug Fixes
Fix AzureBatchOperator false negative task status (#25844)
4.2.0
Features
Add 'test_connection' method to AzureContainerInstanceHook (#25362)
Add test_connection to Azure Batch hook (#25235)
Bump typing-extensions and mypy for ParamSpec (#25088)
Implement Azure Service Bus (Update and Receive) Subscription Operator (#25029)
Set default wasb Azure http logging level to warning; fixes #16224 (#18896)
4.1.0
Features
Add 'test_connection' method to AzureCosmosDBHook (#25018)
Add test_connection method to AzureFileShareHook (#24843)
Add test_connection method to Azure WasbHook (#24771)
Implement Azure service bus subscription Operators (#24625)
Implement Azure Service Bus Queue Operators (#24038)
Bug Fixes
Update providers to use functools compat for ''cached_property'' (#24582)
4.0.0
Breaking changes
This release of provider is only available for Airflow 2.2+ as explained in the Apache Airflow providers support policy https://github.com/apache/airflow/blob/main/README.md#support-for-providers
Features
Pass connection extra parameters to wasb BlobServiceClient (#24154)
Misc
Apply per-run log templates to log handlers (#24153)
Migrate Microsoft example DAGs to new design #22452 - azure (#24141)
Add typing to Azure Cosmos Client Hook (#23941)
3.9.0
Features
wasb hook: user defaultAzureCredentials instead of managedIdentity (#23394)
Misc
Replace usage of 'DummyOperator' with 'EmptyOperator' (#22974)
3.8.0
Features
Update secrets backends to use get_conn_value instead of get_conn_uri (#22348)
Misc
Docs: Fix example usage for 'AzureCosmosDocumentSensor' (#22735)
3.7.2
Bug Fixes
Fix mistakenly added install_requires for all providers (#22382)
3.7.1
Misc
Add Trove classifiers in PyPI (Framework :: Apache Airflow :: Provider)
3.7.0
Features
Add 'test_connection' method to 'AzureDataFactoryHook' (#21924)
Add pre-commit check for docstring param types (#21398)
Make container creation configurable when uploading files via WasbHook (#20510)
Misc
Support for Python 3.10
(AzureCosmosDBHook) Update to latest Cosmos API (#21514)
3.6.0
Features
Add optional features in providers. (#21074)
Misc
Refactor operator links to not create ad hoc TaskInstances (#21285)
3.5.0
Features
Azure: New sftp to wasb operator (#18877)
Removes InputRequired validation with azure extra (#20084)
Add operator link to monitor Azure Data Factory pipeline runs (#20207)
3.4.0
Features
Remove unnecessary connection form customizations in Azure (#19595)
Update Azure modules to comply with AIP-21 (#19431)
Remove 'host' from hidden fields in 'WasbHook' (#19475)
use DefaultAzureCredential if login not provided for Data Factory (#19079)
Bug Fixes
Fix argument error in AzureContainerInstancesOperator (#19668)
3.3.0
Features
update azure cosmos to latest version (#18695)
Added sas_token var to BlobServiceClient return. Updated tests (#19234)
Add pre-commit hook for common misspelling check in files (#18964)
Bug Fixes
Fix changelog for Azure Provider (#18736)
Other
Expanding docs on client auth for AzureKeyVaultBackend (#18659)
Static start_date and default arg cleanup for Microsoft providers example DAGs (#19062)
3.2.0
Features
Rename AzureDataLakeStorage to ADLS (#18493)
Creating ADF pipeline run operator, sensor + ADF custom conn fields (#17885)
Rename LocalToAzureDataLakeStorageOperator to LocalFilesystemToADLSOperator (#18168)
Rename FileToWasbOperator to LocalFilesystemToWasbOperator (#18109)
Bug Fixes
Fixed wasb hook attempting to create container when getting a blob client (#18287)
Removing redundant relabeling of password conn field (#18386)
Proper handling of Account URL custom conn field in AzureBatchHook (#18456)
Proper handling of custom conn field values in the AzureDataExplorerHook (#18203)
Main
Changes in operators names and import paths are listed in the following table This is a backward compatible change. Deprecated operators will be removed in the next major release.
Deprecated operator name |
New operator name |
Deprecated path |
New path |
AzureDataLakeStorageListOperator |
ADLSListOperator |
airflow.providers.microsoft.azure.operators.adls_list |
airflow.providers.microsoft.azure.operators.adls |
AzureDataLakeStorageDeleteOperator |
ADLSDeleteOperator |
airflow.providers.microsoft.azure.operators.adls_delete |
airflow.providers.microsoft.azure.operators.adls |
3.1.1
Misc
Optimise connection importing for Airflow 2.2.0
Adds secrets backend/logging/auth information to provider yaml (#17625)
3.1.0
Features
Add support for managed identity in WASB hook (#16628)
Reduce log messages for happy path (#16626)
Bug Fixes
Fix multiple issues in Microsoft AzureContainerInstancesOperator (#15634)
3.0.0
Breaking changes
Auto-apply apply_default decorator (#15667)
Fixes AzureFileShare connection extras (#16388)
Azure Container Volume and Azure File Share have now dedicated connection types with editable UI fields. You should not use Wasb connection type any more for those connections. Names of connection ids for those hooks/operators were changed to reflect that.
Features
add oracle connection link (#15632)
Add delimiter argument to WasbHook delete_file method (#15637)
Bug Fixes
Fix colon spacing in ``AzureDataExplorerHook docstring (#15841)``
fix wasb remote logging when blob already exists (#16280)
2.0.0
Breaking changes
Removes unnecessary AzureContainerInstance connection type (#15514)
This change removes azure_container_instance_default connection type and replaces it with the azure_default. The problem was that AzureContainerInstance was not needed as it was exactly the same as the plain “azure” connection, however it’s presence caused duplication in the field names used in the UI editor for connections and unnecessary warnings generated. This version uses plain Azure Hook and connection also for Azure Container Instance. If you already have azure_container_instance_default connection created in your DB, it will continue to work, but the first time you edit it with the UI you will have to change it’s type to azure_default.
Features
Add dynamic connection fields to Azure Connection (#15159)
Bug fixes
Fix 'logging.exception' redundancy (#14823)
1.3.0
Features
A bunch of template_fields_renderers additions (#15130)
Bug fixes
Fix attributes for AzureDataFactory hook (#14704)
1.2.0
Features
Add Azure Data Factory hook (#11015)
Bug fixes
BugFix: Fix remote log in azure storage blob displays in one line (#14313)
Fix AzureDataFactoryHook failing to instantiate its connection (#14565)
1.1.0
Updated documentation and readme files.
Features
Upgrade azure blob to v12 (#12188)
Fix Azure Data Explorer Operator (#13520)
add AzureDatalakeStorageDeleteOperator (#13206)
1.0.0
Initial version of the provider.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for apache-airflow-providers-microsoft-azure-4.3.0rc1.tar.gz
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | b38f1ce631b7cfc60959cb33059ad3651b9de931588787df05da7a54b6ab7859 |
|
| MD5 | bd787857dfbccb8bef9fea44e746a9b5 |
|
| BLAKE2b-256 | d5156cdd6427306c0bf1a01c0965ffddbd2ca03176da59fcf67facfa0e5b971c |
Hashes for apache_airflow_providers_microsoft_azure-4.3.0rc1-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 323e1506a5c930b6bc9f1db7684e6e1e58ed39670cafa043dfe6c93be7275a5a |
|
| MD5 | 50ee6c3f93da089068acd62157c1b5df |
|
| BLAKE2b-256 | 604fc24ffd637e0425210eec0fa39ea357c9e7c5150f7098dd36cb3edac15a13 |
