Skip to main content

Provider package apache-airflow-providers-amazon for Apache Airflow

Project description

Package apache-airflow-providers-amazon

Release: 2.2.0

Amazon integration (including Amazon Web Services (AWS)).

Provider package

This is a provider package for amazon provider. All classes for this provider package are in airflow.providers.amazon python package.

You can find package information and changelog for the provider in the documentation.

Installation

You can install this package on top of an existing airflow 2.1+ installation via pip install apache-airflow-providers-amazon

The package supports the following python versions: 3.6,3.7,3.8,3.9

PIP requirements

PIP package

Version required

apache-airflow

>=2.1.0

boto3

>=1.15.0,<1.18.0

jsonpath_ng

>=1.5.3

watchtower

~=1.0.6

Cross provider package dependencies

Those are dependencies that might be needed in order to use all the features of the package. You need to install the specified provider packages in order to use them.

You can install such cross-provider dependencies when installing from PyPI. For example:

pip install apache-airflow-providers-amazon[apache.hive]

Dependent package

Extra

apache-airflow-providers-apache-hive

apache.hive

apache-airflow-providers-cncf-kubernetes

cncf.kubernetes

apache-airflow-providers-exasol

exasol

apache-airflow-providers-ftp

ftp

apache-airflow-providers-google

google

apache-airflow-providers-imap

imap

apache-airflow-providers-mongo

mongo

apache-airflow-providers-mysql

mysql

apache-airflow-providers-postgres

postgres

apache-airflow-providers-salesforce

salesforce

apache-airflow-providers-ssh

ssh

Changelog

2.2.0

Features

  • Add an Amazon EMR on EKS provider package (#16766)

  • Add optional SQL parameters in ''RedshiftToS3Operator'' (#17640)

  • Add new LocalFilesystemToS3Operator under Amazon provider (#17168) (#17382)

  • Add Mongo projections to hook and transfer (#17379)

  • make platform version as independent parameter of ECSOperator (#17281)

  • Improve AWS SQS Sensor (#16880) (#16904)

  • Implemented Basic EKS Integration (#16571)

Bug Fixes

  • Fixing ParamValidationError when executing load_file in Glue hooks/operators (#16012)

  • Fixes #16972 - Slugify role session name in AWS base hook (#17210)

  • Fix broken XCOM in EKSPodOperator (#17918)

Misc

  • Optimise connection importing for Airflow 2.2.0

  • Fix provider.yaml errors due to exit(0) in test (#17858)

  • Adds secrets backend/logging/auth information to provider yaml (#17625)

2.1.0

Features

  • Allow attaching to previously launched task in ECSOperator (#16685)

  • Update AWS Base hook to use refreshable credentials (#16770) (#16771)

  • Added select_query to the templated fields in RedshiftToS3Operator (#16767)

  • AWS Hook - allow IDP HTTP retry (#12639) (#16612)

  • Update Boto3 API calls in ECSOperator (#16050)

  • Adding custom Salesforce connection type + SalesforceToS3Operator updates (#17162)

  • Adding SalesforceToS3Operator to Amazon Provider (#17094)

Bug Fixes

  • AWS DataSync default polling adjusted from 5s to 30s (#11011)

  • Fix wrong template_fields_renderers for AWS operators (#16820)

  • AWS DataSync cancel task on exception (#11011) (#16589)

  • Fixed template_fields_renderers for Amazon provider (#17087)

  • removing try-catch block (#17081)

  • ECSOperator / pass context to self.xcom_pull as it was missing (when using reattach) (#17141)

  • Made S3ToRedshiftOperator transaction safe (#17117)

2.0.0

Breaking changes

  • Auto-apply apply_default decorator (#15667)

Features

  • CloudwatchTaskHandler reads timestamp from Cloudwatch events (#15173)

  • remove retry for now (#16150)

  • Remove the 'not-allow-trailing-slash' rule on S3_hook (#15609)

  • Add support of capacity provider strategy for ECSOperator (#15848)

  • Update copy command for s3 to redshift (#16241)

  • Make job name check optional in SageMakerTrainingOperator (#16327)

  • Add AWS DMS replication task operators (#15850)

Bug Fixes

  • Fix S3 Select payload join (#16189)

  • Fix spacing in 'AwsBatchWaitersHook' docstring (#15839)

  • MongoToS3Operator failed when running with a single query (not aggregate pipeline) (#15680)

  • fix: AwsGlueJobOperator change order of args for load_file (#16216)

  • Fix S3ToFTPOperator (#13796)

1.4.0

Features

  • S3Hook.load_file should accept Path object in addition to str (#15232)

Bug fixes

  • Fix 'logging.exception' redundancy (#14823)

  • Fix AthenaSensor calling AthenaHook incorrectly (#15427)

  • Add links to new modules for deprecated modules (#15316)

  • Fixes doc for SQSSensor (#15323)

1.3.0

Features

  • A bunch of template_fields_renderers additions (#15130)

  • Send region_name into parent class of AwsGlueJobHook (#14251)

  • Added retry to ECS Operator (#14263)

  • Make script_args templated in AwsGlueJobOperator (#14925)

  • Add FTPToS3Operator (#13707)

  • Implemented S3 Bucket Tagging (#14402)

  • S3DataSource is not required (#14220)

Bug fixes

  • AWS: Do not log info when SSM & SecretsManager secret not found (#15120)

  • Cache Hook when initializing 'CloudFormationCreateStackSensor' (#14638)

1.2.0

Features

  • Avoid using threads in S3 remote logging upload (#14414)

  • Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query (#14177)

  • includes the STS token if STS credentials are used (#11227)

1.1.0

Features

  • Adding support to put extra arguments for Glue Job. (#14027)

  • Add aws ses email backend for use with EmailOperator. (#13986)

  • Add bucket_name to template fileds in S3 operators (#13973)

  • Add ExasolToS3Operator (#13847)

  • AWS Glue Crawler Integration (#13072)

  • Add acl_policy to S3CopyObjectOperator (#13773)

  • AllowDiskUse parameter and docs in MongotoS3Operator (#12033)

  • Add S3ToFTPOperator (#11747)

  • add xcom push for ECSOperator (#12096)

  • [AIRFLOW-3723] Add Gzip capability to mongo_to_S3 operator (#13187)

  • Add S3KeySizeSensor (#13049)

  • Add 'mongo_collection' to template_fields in MongoToS3Operator (#13361)

  • Allow Tags on AWS Batch Job Submission (#13396)

Bug fixes

  • Fix bug in GCSToS3Operator (#13718)

  • Fix S3KeysUnchangedSensor so that template_fields work (#13490)

1.0.0

Initial version of the provider.

Project details


Release history Release notifications | RSS feed

This version

2.2.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

apache-airflow-providers-amazon-2.2.0.tar.gz (151.1 kB view details)

Uploaded Source

Built Distribution

File details

Details for the file apache-airflow-providers-amazon-2.2.0.tar.gz.

File metadata

  • Download URL: apache-airflow-providers-amazon-2.2.0.tar.gz
  • Upload date:
  • Size: 151.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.7.9

File hashes

Hashes for apache-airflow-providers-amazon-2.2.0.tar.gz
Algorithm Hash digest
SHA256 4cd8fe5aa45f5d01b30d0da76a969c66ad99408b371940b07b58391711b1642d
MD5 977f48035a763cb6aef56cc0e265c989
BLAKE2b-256 cf8329dcf76238bbd58eac126765bb059ba5db6bf0fbe52bf9dc0ba91acb3894

See more details on using hashes here.

Provenance

File details

Details for the file apache_airflow_providers_amazon-2.2.0-py3-none-any.whl.

File metadata

  • Download URL: apache_airflow_providers_amazon-2.2.0-py3-none-any.whl
  • Upload date:
  • Size: 287.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.7.9

File hashes

Hashes for apache_airflow_providers_amazon-2.2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 37316caf7d6f8289dd055d87ac7cd6ac3965e2b47370a3ccb629dd0881b85e34
MD5 a4a00c52bd41c990a4bb5bfc50360ed0
BLAKE2b-256 12b475a3a592573a62dab6aa5ced6dcae3810d22628750ecae32cd9fefffcb44

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page