Provider for Apache Airflow. Implements apache-airflow-providers-apache-spark package
Project description
Package apache-airflow-providers-apache-spark
Release: 4.1.0rc2
Provider package
This is a provider package for apache.spark provider. All classes for this provider package are in airflow.providers.apache.spark python package.
You can find package information and changelog for the provider in the documentation.
Installation
You can install this package on top of an existing Airflow 2 installation (see Requirements below for the minimum Airflow version supported) via pip install apache-airflow-providers-apache-spark
The package supports the following python versions: 3.7,3.8,3.9,3.10
Requirements
PIP package |
Version required |
|---|---|
apache-airflow |
>=2.4.0 |
pyspark |
Changelog
4.1.0
Misc
Bump minimum Airflow version in providers (#30917)
4.0.1
Bug Fixes
Only restrict spark binary passed via extra (#30213)
Validate host and schema for Spark JDBC Hook (#30223)
Add spark3-submit to list of allowed spark-binary values (#30068)
4.0.0
Breaking changes
The spark-binary connection extra could be set to any binary, but with 4.0.0 version only two values are allowed for it spark-submit and spark2-submit.
The spark-home connection extra is not allowed any more - the binary should be available on the PATH in order to use SparkSubmitHook and SparkSubmitOperator.
Remove custom spark home and custom binaries for spark (#27646)
Misc
Move min airflow version to 2.3.0 for all providers (#27196)
3.0.0
Breaking changes
Bug Fixes
Add typing for airflow/configuration.py (#23716)
Fix backwards-compatibility introduced by fixing mypy problems (#24230)
Misc
AIP-47 - Migrate spark DAGs to new design #22439 (#24210)
chore: Refactoring and Cleaning Apache Providers (#24219)
2.1.3
Bug Fixes
Fix mistakenly added install_requires for all providers (#22382)
2.1.2
Misc
Add Trove classifiers in PyPI (Framework :: Apache Airflow :: Provider)
2.1.1
Bug Fixes
fix param rendering in docs of SparkSubmitHook (#21788)
Misc
Support for Python 3.10
2.1.0
Features
Add more SQL template fields renderers (#21237)
Add optional features in providers. (#21074)
2.0.3
Bug Fixes
Ensure Spark driver response is valid before setting UNKNOWN status (#19978)
2.0.2
Bug Fixes
fix bug of SparkSql Operator log going to infinite loop. (#19449)
2.0.1
Misc
Optimise connection importing for Airflow 2.2.0
2.0.0
Breaking changes
Auto-apply apply_default decorator (#15667)
Bug fixes
Make SparkSqlHook use Connection (#15794)
1.0.3
Bug fixes
Fix 'logging.exception' redundancy (#14823)
1.0.2
Bug fixes
Use apache.spark provider without kubernetes (#14187)
1.0.1
Updated documentation and readme files.
1.0.0
Initial version of the provider.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for apache-airflow-providers-apache-spark-4.1.0rc2.tar.gz
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | a499cb3afd258e824ebb853dde54fc74b93edcfe8293f33a69bd6bcb5d43275b |
|
| MD5 | 3b99b07e3f9c37ffeefe3da6991a35e9 |
|
| BLAKE2b-256 | 59d051f75ceffeb7cf3f5b71d7541cf3859847c59e954bd42b39d1cbe84bef73 |
Hashes for apache_airflow_providers_apache_spark-4.1.0rc2-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | bc49e9a54d4a819bdd1d5c2e00d9e29bbdb730d42c23a782c756b71b50851c03 |
|
| MD5 | a6b83d361edb2e0d6fa2e52ccf122f1e |
|
| BLAKE2b-256 | 579a2f71d2ca49c03ec591168c31cc5e9694434b22de4098069841c345607e88 |
