Back-ported airflow.providers.apache.spark.* package for Airflow 1.10.*
Project description
Package apache-airflow-backport-providers-apache-spark
Release: 2020.5.20
Backport package
This is a backport providers package for apache.spark provider. All classes for this provider package
are in airflow.providers.apache.spark python package.
Only Python 3.6+ is supported for this backport package.
While Airflow 1.10.* continues to support Python 2.7+ - you need to upgrade python to 3.6+ if you want to use this backport package.
Installation
You can install this package on top of an existing airflow 1.10.* installation via
pip install apache-airflow-backport-providers-apache-spark
Compatibility
For full compatibility and test status of the backport packages check Airflow Backport Package Compatibility
PIP requirements
| PIP package | Version required |
|---|---|
| pyspark |
Provider class summary
All classes in Airflow 2.0 are in airflow.providers.apache.spark package.
Operators
Moved operators
Airflow 2.0 operators: airflow.providers.apache.spark package |
Airflow 1.10.* previous location (usually airflow.contrib) |
|---|---|
| operators.spark_jdbc.SparkJDBCOperator | contrib.operators.spark_jdbc_operator.SparkJDBCOperator |
| operators.spark_sql.SparkSqlOperator | contrib.operators.spark_sql_operator.SparkSqlOperator |
| operators.spark_submit.SparkSubmitOperator | contrib.operators.spark_submit_operator.SparkSubmitOperator |
Hooks
Moved hooks
Airflow 2.0 hooks: airflow.providers.apache.spark package |
Airflow 1.10.* previous location (usually airflow.contrib) |
|---|---|
| hooks.spark_jdbc.SparkJDBCHook | contrib.hooks.spark_jdbc_hook.SparkJDBCHook |
| hooks.spark_sql.SparkSqlHook | contrib.hooks.spark_sql_hook.SparkSqlHook |
| hooks.spark_submit.SparkSubmitHook | contrib.hooks.spark_submit_hook.SparkSubmitHook |
Releases
Release 2020.5.20
| Commit | Committed | Subject |
|---|---|---|
| 12c5e5d8a | 2020-05-17 | Prepare release candidate for backport packages (#8891) |
| f3521fb0e | 2020-05-16 | Regenerate readme files for backport package release (#8886) |
| 92585ca4c | 2020-05-15 | Added automated release notes generation for backport operators (#8807) |
| 7506c73f1 | 2020-05-10 | Add default conf parameter to Spark JDBC Hook (#8787) |
| 487b5cc50 | 2020-05-06 | Add guide for Apache Spark operators (#8305) |
| 87969a350 | 2020-04-09 | [AIRFLOW-6515] Change Log Levels from Info/Warn to Error (#8170) |
| be1451b0e | 2020-04-02 | [AIRFLOW-7026] Improve SparkSqlHook's error message (#7749) |
| 4bde99f13 | 2020-03-23 | Make airflow/providers pylint compatible (#7802) |
| 7e6372a68 | 2020-03-23 | Add call to Super call in apache providers (#7820) |
| 2327aa5a2 | 2020-03-12 | [AIRFLOW-7025] Fix SparkSqlHook.run_query to handle its parameter properly (#7677) |
| 024b4bf96 | 2020-03-10 | [AIRFLOW-7024] Add the verbose parameter support to SparkSqlOperator (#7676) |
| b59042b5a | 2020-02-28 | [AIRFLOW-6949] Respect explicit spark.kubernetes.namespace conf to SparkSubmitOperator (#7575) |
| 97a429f9d | 2020-02-02 | [AIRFLOW-6714] Remove magic comments about UTF-8 (#7338) |
| 0481b9a95 | 2020-01-12 | [AIRFLOW-6539][AIP-21] Move Apache classes to providers.apache package (#7142) |
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for apache-airflow-backport-providers-apache-spark-2020.5.20rc2.tar.gz
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 0f6ce6c030c2e0a231f80b16206820aaa3b620f13eeb39ab68f2998ad7fe9758 |
|
| MD5 | 684a6996bda3e12606c5b8219df1c600 |
|
| BLAKE2b-256 | fbef889ca2939a8595a607193729fe390e10e2cccd90288ceab6e92276c18afd |
Hashes for apache_airflow_backport_providers_apache_spark-2020.5.20rc2-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | abd154c3db92041e46655f0ea26d2b23127c590d908c45645d3ee7abb84bbe2d |
|
| MD5 | 4feda42df1728718c766255e8eaadf9f |
|
| BLAKE2b-256 | 3b697c15bedb57dfeb3487555b57ce0c4a99fd6ac51a5c5bec67a7ba962b87f5 |
