Back-ported airflow.providers.amazon.* package for Airflow 1.10.*

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for ashb from gravatar.com ashb Avatar for kaxil from gravatar.com kaxil Avatar for potiuk from gravatar.com potiuk

Unverified details

These details have not been verified by PyPI
Meta
  • License: Apache Software License (Apache License 2.0)
  • Author: Apache Airflow PMC
  • Requires: Python >=3.6
Classifiers

Project description

Package apache-airflow-backport-providers-amazon

Release: 2020.5.20

Backport package

This is a backport providers package for amazon provider. All classes for this provider package are in airflow.providers.amazon python package.

Only Python 3.6+ is supported for this backport package.

While Airflow 1.10.* continues to support Python 2.7+ - you need to upgrade python to 3.6+ if you want to use this backport package.

Installation

You can install this package on top of an existing airflow 1.10.* installation via pip install apache-airflow-backport-providers-amazon

Compatibility

For full compatibility and test status of the backport packages check Airflow Backport Package Compatibility

PIP requirements

PIP package Version required
boto3 >=1.12.0,<2.0.0
watchtower ~=0.7.3

Cross provider package dependencies

Those are dependencies that might be needed in order to use all the features of the package. You need to install the specified backport providers package in order to use them.

You can install such cross-provider dependencies when installing from PyPI. For example:

pip install apache-airflow-backport-providers-amazon[apache.hive]
Dependent package Extra
apache-airflow-backport-providers-apache-hive apache.hive
apache-airflow-backport-providers-google google
apache-airflow-backport-providers-imap imap
apache-airflow-backport-providers-mongo mongo
apache-airflow-backport-providers-postgres postgres
apache-airflow-backport-providers-ssh ssh

Provider class summary

All classes in Airflow 2.0 are in airflow.providers.amazon package.

Operators

New operators

New Airflow 2.0 operators: airflow.providers.amazon package
aws.operators.cloud_formation.CloudFormationCreateStackOperator
aws.operators.cloud_formation.CloudFormationDeleteStackOperator
aws.operators.datasync.AWSDataSyncOperator
aws.operators.ec2_start_instance.EC2StartInstanceOperator
aws.operators.ec2_stop_instance.EC2StopInstanceOperator
aws.operators.emr_modify_cluster.EmrModifyClusterOperator
aws.operators.glue.AwsGlueJobOperator
aws.operators.s3_file_transform.S3FileTransformOperator

Moved operators

Airflow 2.0 operators: airflow.providers.amazon package Airflow 1.10.* previous location (usually airflow.contrib)
aws.operators.athena.AWSAthenaOperator contrib.operators.aws_athena_operator.AWSAthenaOperator
aws.operators.batch.AwsBatchOperator contrib.operators.awsbatch_operator.AWSBatchOperator
aws.operators.dynamodb_to_s3.DynamoDBToS3Operator contrib.operators.dynamodb_to_s3.DynamoDBToS3Operator
aws.operators.ecs.ECSOperator contrib.operators.ecs_operator.ECSOperator
aws.operators.emr_add_steps.EmrAddStepsOperator contrib.operators.emr_add_steps_operator.EmrAddStepsOperator
aws.operators.emr_create_job_flow.EmrCreateJobFlowOperator contrib.operators.emr_create_job_flow_operator.EmrCreateJobFlowOperator
aws.operators.emr_terminate_job_flow.EmrTerminateJobFlowOperator contrib.operators.emr_terminate_job_flow_operator.EmrTerminateJobFlowOperator
aws.operators.gcs_to_s3.GCSToS3Operator operators.gcs_to_s3.GCSToS3Operator
aws.operators.google_api_to_s3_transfer.GoogleApiToS3Transfer operators.google_api_to_s3_transfer.GoogleApiToS3Transfer
aws.operators.hive_to_dynamodb.HiveToDynamoDBTransferOperator contrib.operators.hive_to_dynamodb.HiveToDynamoDBTransferOperator
aws.operators.imap_attachment_to_s3.ImapAttachmentToS3Operator contrib.operators.imap_attachment_to_s3_operator.ImapAttachmentToS3Operator
aws.operators.mongo_to_s3.MongoToS3Operator contrib.operators.mongo_to_s3.MongoToS3Operator
aws.operators.redshift_to_s3.RedshiftToS3Transfer operators.redshift_to_s3_operator.RedshiftToS3Transfer
aws.operators.s3_copy_object.S3CopyObjectOperator contrib.operators.s3_copy_object_operator.S3CopyObjectOperator
aws.operators.s3_delete_objects.S3DeleteObjectsOperator contrib.operators.s3_delete_objects_operator.S3DeleteObjectsOperator
aws.operators.s3_list.S3ListOperator contrib.operators.s3_list_operator.S3ListOperator
aws.operators.s3_to_redshift.S3ToRedshiftTransfer operators.s3_to_redshift_operator.S3ToRedshiftTransfer
aws.operators.s3_to_sftp.S3ToSFTPOperator contrib.operators.s3_to_sftp_operator.S3ToSFTPOperator
aws.operators.sagemaker_base.SageMakerBaseOperator contrib.operators.sagemaker_base_operator.SageMakerBaseOperator
aws.operators.sagemaker_endpoint.SageMakerEndpointOperator contrib.operators.sagemaker_endpoint_operator.SageMakerEndpointOperator
aws.operators.sagemaker_endpoint_config.SageMakerEndpointConfigOperator contrib.operators.sagemaker_endpoint_config_operator.SageMakerEndpointConfigOperator
aws.operators.sagemaker_model.SageMakerModelOperator contrib.operators.sagemaker_model_operator.SageMakerModelOperator
aws.operators.sagemaker_training.SageMakerTrainingOperator contrib.operators.sagemaker_training_operator.SageMakerTrainingOperator
aws.operators.sagemaker_transform.SageMakerTransformOperator contrib.operators.sagemaker_transform_operator.SageMakerTransformOperator
aws.operators.sagemaker_tuning.SageMakerTuningOperator contrib.operators.sagemaker_tuning_operator.SageMakerTuningOperator
aws.operators.sftp_to_s3.SFTPToS3Operator contrib.operators.sftp_to_s3_operator.SFTPToS3Operator
aws.operators.sns.SnsPublishOperator contrib.operators.sns_publish_operator.SnsPublishOperator
aws.operators.sqs.SQSPublishOperator contrib.operators.aws_sqs_publish_operator.SQSPublishOperator

Sensors

New sensors

New Airflow 2.0 sensors: airflow.providers.amazon package
aws.sensors.cloud_formation.CloudFormationCreateStackSensor
aws.sensors.cloud_formation.CloudFormationDeleteStackSensor
aws.sensors.ec2_instance_state.EC2InstanceStateSensor
aws.sensors.glue.AwsGlueJobSensor
aws.sensors.redshift.AwsRedshiftClusterSensor
aws.sensors.sagemaker_training.SageMakerTrainingSensor

Moved sensors

Airflow 2.0 sensors: airflow.providers.amazon package Airflow 1.10.* previous location (usually airflow.contrib)
aws.sensors.athena.AthenaSensor contrib.sensors.aws_athena_sensor.AthenaSensor
aws.sensors.emr_base.EmrBaseSensor contrib.sensors.emr_base_sensor.EmrBaseSensor
aws.sensors.emr_job_flow.EmrJobFlowSensor contrib.sensors.emr_job_flow_sensor.EmrJobFlowSensor
aws.sensors.emr_step.EmrStepSensor contrib.sensors.emr_step_sensor.EmrStepSensor
aws.sensors.glue_catalog_partition.AwsGlueCatalogPartitionSensor contrib.sensors.aws_glue_catalog_partition_sensor.AwsGlueCatalogPartitionSensor
aws.sensors.s3_key.S3KeySensor sensors.s3_key_sensor.S3KeySensor
aws.sensors.s3_prefix.S3PrefixSensor sensors.s3_prefix_sensor.S3PrefixSensor
aws.sensors.sagemaker_base.SageMakerBaseSensor contrib.sensors.sagemaker_base_sensor.SageMakerBaseSensor
aws.sensors.sagemaker_endpoint.SageMakerEndpointSensor contrib.sensors.sagemaker_endpoint_sensor.SageMakerEndpointSensor
aws.sensors.sagemaker_transform.SageMakerTransformSensor contrib.sensors.sagemaker_transform_sensor.SageMakerTransformSensor
aws.sensors.sagemaker_tuning.SageMakerTuningSensor contrib.sensors.sagemaker_tuning_sensor.SageMakerTuningSensor
aws.sensors.sqs.SQSSensor contrib.sensors.aws_sqs_sensor.SQSSensor

Hooks

New hooks

New Airflow 2.0 hooks: airflow.providers.amazon package
aws.hooks.batch_client.AwsBatchClient
aws.hooks.batch_waiters.AwsBatchWaiters
aws.hooks.cloud_formation.AWSCloudFormationHook
aws.hooks.ec2.EC2Hook
aws.hooks.glue.AwsGlueJobHook
aws.hooks.kinesis.AwsFirehoseHook
aws.hooks.redshift.RedshiftHook

Moved hooks

Airflow 2.0 hooks: airflow.providers.amazon package Airflow 1.10.* previous location (usually airflow.contrib)
aws.hooks.athena.AWSAthenaHook contrib.hooks.aws_athena_hook.AWSAthenaHook
aws.hooks.aws_dynamodb.AwsDynamoDBHook contrib.hooks.aws_dynamodb_hook.AwsDynamoDBHook
aws.hooks.base_aws.AwsBaseHook contrib.hooks.aws_hook.AwsHook
aws.hooks.datasync.AWSDataSyncHook contrib.hooks.aws_datasync_hook.AWSDataSyncHook
aws.hooks.emr.EmrHook contrib.hooks.emr_hook.EmrHook
aws.hooks.glue_catalog.AwsGlueCatalogHook contrib.hooks.aws_glue_catalog_hook.AwsGlueCatalogHook
aws.hooks.lambda_function.AwsLambdaHook contrib.hooks.aws_lambda_hook.AwsLambdaHook
aws.hooks.logs.AwsLogsHook contrib.hooks.aws_logs_hook.AwsLogsHook
aws.hooks.s3.S3Hook hooks.S3_hook.S3Hook
aws.hooks.sagemaker.SageMakerHook contrib.hooks.sagemaker_hook.SageMakerHook
aws.hooks.sns.AwsSnsHook contrib.hooks.aws_sns_hook.AwsSnsHook
aws.hooks.sqs.SQSHook contrib.hooks.aws_sqs_hook.SQSHook

Protocols

Moved protocols

Airflow 2.0 protocols: airflow.providers.amazon package Airflow 1.10.* previous location (usually airflow.contrib)
aws.hooks.batch_client.AwsBatchProtocol contrib.operators.awsbatch_operator.BatchProtocol
aws.operators.ecs.ECSProtocol contrib.operators.ecs_operator.ECSProtocol

Secrets

Moved secrets

Airflow 2.0 protocols: airflow.providers.amazon package Airflow 1.10.* previous location (usually airflow.contrib)
aws.secrets.secrets_manager.SecretsManagerBackend contrib.secrets.aws_secrets_manager.SecretsManagerBackend
aws.secrets.systems_manager.SystemsManagerParameterStoreBackend contrib.secrets.aws_systems_manager.SystemsManagerParameterStoreBackend

Releases

Release 2020.5.20

Commit Committed Subject
12c5e5d8a 2020-05-17 Prepare release candidate for backport packages (#8891)
f3521fb0e 2020-05-16 Regenerate readme files for backport package release (#8886)
f4edd90a9 2020-05-16 Speed up TestAwsLambdaHook by not actually running a function (#8882)
92585ca4c 2020-05-15 Added automated release notes generation for backport operators (#8807)
85bbab27d 2020-05-15 Add EMR operators howto docs (#8863)
e61b9bb9b 2020-05-13 Add AWS EMR System tests (#8618)
ed3f5131a 2020-05-13 Correctly pass sleep time from AWSAthenaOperator down to the hook. (#8845)
7236862a1 2020-05-12 [AIRFLOW-2310] Enable AWS Glue Job Integration (#6007)
d590e5e76 2020-05-11 Add option to propagate tags in ECSOperator (#8811)
0c3db84c3 2020-05-11 [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor (#7731)
cbebed2b4 2020-05-10 Allow passing backend_kwargs to AWS SSM client (#8802)
c7788a689 2020-05-10 Add imap_attachment_to_s3 example dag and system test (#8669)
ff5b70149 2020-05-07 Add google_api_to_s3_transfer example dags and system tests (#8581)
4421f011e 2020-05-01 Improve template capabilities of EMR job and step operators (#8572)
379a884d6 2020-04-28 fix: aws hook should work without conn id (#8534)
74bc316c5 2020-04-27 [AIRFLOW-4438] Add Gzip compression to S3_hook (#8571)
7ea66a1a9 2020-04-26 Add example DAG for ECSOperator (#8452)
b6434dedf 2020-04-24 [AIRFLOW-7111] Add generate_presigned_url method to S3Hook (#8441)
becedd5af 2020-04-19 Remove unrelated EC2 references in ECSOperator (#8451)
ab1290cb0 2020-04-13 Make launch_type parameter optional (#8248)
87969a350 2020-04-09 [AIRFLOW-6515] Change Log Levels from Info/Warn to Error (#8170)
b46d6c060 2020-04-08 Add support for AWS Secrets Manager as Secrets Backend (#8186)
68d1714f2 2020-04-04 [AIRFLOW-6822] AWS hooks should cache boto3 client (#7541)
8a0240257 2020-03-31 Rename CloudBaseHook to GoogleBaseHook and move it to google.common (#8011)
7239d9a82 2020-03-28 Get Airflow Variables from AWS Systems Manager Parameter Store (#7945)
eb4af4f94 2020-03-28 Make BaseSecretsBackend.build_path generic (#7948)
438da7241 2020-03-28 [AIRFLOW-5825] SageMakerEndpointOperator is not idempotent (#7891)
686d7d50b 2020-03-25 Standardize SecretBackend class names (#7846)
eef87b995 2020-03-23 [AIRFLOW-7105] Unify Secrets Backend method interfaces (#7830)
5648dfbc3 2020-03-23 Add missing call to Super class in 'amazon', 'cloudant & 'databricks' providers (#7827)
a36002412 2020-03-19 [AIRFLOW-5705] Make AwsSsmSecretsBackend consistent with VaultBackend (#7753)
2a54512d7 2020-03-17 [AIRFLOW-5705] Fix bugs in AWS SSM Secrets Backend (#7745)
a8b5fc74d 2020-03-16 [AIRFLOW-4175] S3Hook load_file should support ACL policy paramete (#7733)
e31e9ddd2 2020-03-14 [AIRFLOW-5705] Add secrets backend and support for AWS SSM (#6376)
3bb60afc7 2020-03-13 [AIRFLOW-6975] Base AWSHook AssumeRoleWithSAML (#7619)
c0c5f11ad 2020-03-12 [AIRFLOW-6884] Make SageMakerTrainingOperator idempotent (#7598)
b7cdda1c6 2020-03-10 [AIRFLOW-4438] Add Gzip compression to S3_hook (#7680)
42eef3821 2020-03-07 [AIRFLOW-6877] Add cross-provider dependencies as extras (#7506)
9a94ab246 2020-03-01 [AIRFLOW-6962] Fix compeleted to completed (#7600)
1b38f6d9b 2020-02-29 [AIRFLOW-5908] Add download_file to S3 Hook (#6577)
3ea3e1a2b 2020-02-26 [AIRFLOW-6824] EMRAddStepsOperator problem with multi-step XCom (#7443)
6eaa7e3b1 2020-02-25 [AIRFLOW-5924] Automatically unify bucket name and key in S3Hook (#6574)
3320e432a 2020-02-24 [AIRFLOW-6817] Lazy-load airflow.DAG to keep user-facing API untouched (#7517)
7d0e7122d 2020-02-24 [AIRFLOW-6830] Add Subject/MessageAttributes to SNS hook and operator (#7451)
4d03e33c1 2020-02-22 [AIRFLOW-6817] remove imports from airflow/__init__.py, replaced implicit imports with explicit imports, added entry to UPDATING.MD - squashed/rebased (#7456)
47a922b86 2020-02-21 [AIRFLOW-6854] Fix missing typing_extensions on python 3.8 (#7474)
9cbd7de6d 2020-02-18 [AIRFLOW-6792] Remove _operator/_hook/_sensor in providers package and add tests (#7412)
58c3542ed 2020-02-12 [AIRFLOW-5231] Fix S3Hook.delete_objects method (#7375)
b7aa778b3 2020-02-10 [AIRFLOW-6767] Correct name for default Athena workgroup (#7394)
9282185e6 2020-02-09 [AIRFLOW-6761] Fix WorkGroup param in AWSAthenaHook (#7386)
94fccca97 2020-02-04 [AIRFLOW-XXXX] Add pre-commit check for utf-8 file encoding (#7347)
f3ad5cf61 2020-02-03 [AIRFLOW-4681] Make sensors module pylint compatible (#7309)
88e40c714 2020-02-03 [AIRFLOW-6716] Fix AWS Datasync Example DAG (#7339)
a311d3d82 2020-02-03 [AIRFLOW-6718] Fix more occurrences of utils.dates.days_ago (#7341)
cb766b05b 2020-02-03 [AIRFLOW-XXXX] Fix Static Checks on CI (#7342)
97a429f9d 2020-02-02 [AIRFLOW-6714] Remove magic comments about UTF-8 (#7338)
7527eddc5 2020-02-02 [AIRFLOW-4364] Make all code in airflow/providers/amazon pylint compatible (#7336)
cf141506a 2020-02-02 [AIRFLOW-6708] Set unique logger names (#7330)
63aa3db88 2020-02-02 [AIRFLOW-6258] Add CloudFormation operators to AWS providers (#6824)
af4157fde 2020-02-02 [AIRFLOW-6672] AWS DataSync - better logging of error message (#7288)
373c6aa4a 2020-01-30 [AIRFLOW-6682] Move GCP classes to providers package (#7295)
83c037873 2020-01-30 [AIRFLOW-6674] Move example_dags in accordance with AIP-21 (#7287)
1988a97e8 2020-01-28 [AIRFLOW-6659] Move AWS Transfer operators to providers package (#7274)
ab10443e9 2020-01-28 [AIRFLOW-6424] Added a operator to modify EMR cluster (#7213)
40246132a 2020-01-28 [AIRFLOW-6654] AWS DataSync - bugfix when creating locations (#7270)
82c0e5aff 2020-01-28 [AIRFLOW-6655] Move AWS classes to providers (#7271)
599e4791c 2020-01-18 [AIRFLOW-6541] Use EmrJobFlowSensor for other states (#7146)
c319e81ca 2020-01-17 [AIRFLOW-6572] Move AWS classes to providers.amazon.aws package (#7178)
941a07057 2020-01-15 [AIRFLOW-6570] Add dag tag for all example dag (#7176)
78d8fe694 2020-01-08 [AIRFLOW-6245] Add custom waiters for AWS batch jobs (#6811)
e0b022725 2019-12-28 [AIRFLOW-6319] Add support for AWS Athena workgroups (#6871)
57da45685 2019-12-24 [AIRFLOW-6333] Bump Pylint to 2.4.4 & fix/disable new checks (#6888)
cf647c27e 2019-12-18 [AIRFLOW-6038] AWS DataSync reworked (#6773)
7502cad28 2019-12-17 [AIRFLOW-6206] Move and rename AWS batch operator [AIP-21] (#6764)
c4c635df6 2019-12-17 [AIRFLOW-6083] Adding ability to pass custom configuration to lambda client. (#6678)
4fb498f87 2019-12-09 [AIRFLOW-6072] aws_hook: Outbound http proxy setting and other enhancements (#6686)
a1e2f8635 2019-11-20 [AIRFLOW-6021] Replace list literal with list constructor (#6617)
baae14084 2019-11-19 [AIRFLOW-5781] AIP-21 Migrate AWS Kinesis to /providers/amazon/aws (#6588)
504cfbac1 2019-11-18 [AIRFLOW-5783] AIP-21 Move aws redshift into providers structure (#6539)
992f0e3ac 2019-11-12 AIRFLOW-5824: AWS DataSync Hook and Operators added (#6512)
c015eb2f6 2019-11-10 [AIRFLOW-5786] Migrate AWS SNS to /providers/amazon/aws (#6502)
3d76fb4bf 2019-11-09 [AIRFLOW-5782] Migrate AWS Lambda to /providers/amazon/aws [AIP-21] (#6518)

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for ashb from gravatar.com ashb Avatar for kaxil from gravatar.com kaxil Avatar for potiuk from gravatar.com potiuk

Unverified details

These details have not been verified by PyPI
Meta
  • License: Apache Software License (Apache License 2.0)
  • Author: Apache Airflow PMC
  • Requires: Python >=3.6
Classifiers

Release history Release notifications | RSS feed

2021.3.3

2021.3.3rc1 pre-release

2021.2.5

2021.2.5rc1 pre-release

2020.11.23

2020.11.23rc1 pre-release

2020.10.29 yanked

Reason this release was yanked:

The package contained other package data.

2020.10.29rc1 pre-release

2020.10.5

2020.10.5rc1 pre-release

2020.6.24

2020.6.24rc1 pre-release

2020.5.20rc3 pre-release

This version

2020.5.20rc2 pre-release

2020.5.20rc1 pre-release

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

apache-airflow-backport-providers-amazon-2020.5.20rc2.tar.gz (199.2 kB view hashes)

Uploaded Source

Built Distribution

apache_airflow_backport_providers_amazon-2020.5.20rc2-py3-none-any.whl (203.9 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page