Great Expectations Cloud
Project description
cloud
Quick Start
Python
Install
pip install great_expectations_cloud
Optional Dependencies
pip install 'great_expectations_cloud[sql]'
$ gx-agent --help
usage: gx-agent [-h] [--log-level LOG_LEVEL] [--skip-log-file SKIP_LOG_FILE] [--log-cfg-file LOG_CFG_FILE] [--version]
optional arguments:
-h, --help show this help message and exit
--log-level LOG_LEVEL
Level of logging to use. Defaults to WARNING.
--skip-log-file SKIP_LOG_FILE
Skip writing debug logs to a file. Defaults to False. Does not affect logging to stdout/stderr.
--log-cfg-file LOG_CFG_FILE
Path to a logging configuration json file. Supersedes --log-level and --skip-log-file.
--version Show the GX Agent version.
Set env variables
GX_CLOUD_ACCESS_TOKEN
GX_CLOUD_ORGANIZATION_ID
Start the Agent
If you intend to run the Agent against local services (Cloud backend or datasources) run the Agent outside of the container.
gx-agent
Docker
Building and running the Agent with Docker
Dev Setup
See also CONTRIBUTING.md
- Install
poetry - Set up virtual environment and install dependencies
poetry install --sync
- Activate your virtual environment
poetry shell
- Set up precommit hooks
pre-commit install
Developer Tasks
Common developer tasks are available via invoke (defined in tasks.py)
invoke --list to see available tasks.
Synchronize Dependencies
To ensure you are using the latest version of the core and development dependencies run poetry install --sync.
Also available as an invoke task.
invoke deps
Updating poetry.lock dependencies
The dependencies installed in our CI and the Docker build step are determined by the poetry.lock file.
To update only a specific dependency (such as great_expectations) ...
poetry update great_expectations
To resolve and update all dependencies ...
poetry lock
In either case, the updated poetry.lock file must be committed and merged to main.
Building and Running the GX Agent Image
To build the GX Agent Docker image, run the following in the root dir:
invoke docker
Running the GX Agent:
invoke docker --run
or
docker run --env GX_CLOUD_ACCESS_TOKEN="<GX_TOKEN>" --env GX_CLOUD_ORGANIZATION_ID="<GX_ORG_ID>" gx/agent
Now go into GX Cloud and issue commands for the GX Agent to run, such as generating an Expectation Suite for a Data Source.
Note if you are pushing out a new image update the image tag version in
containerize-agent.yaml. The image will be built and pushed out via GitHub Actions.
Example Data
The contents from /examples/agent/data will be copied to /data for the Docker container.
Adding an action to the Agent
- Make a new action in
great_expectations_cloud/agent/actions/in a separate file. - Register your action in the file it was created in using
great_expectations_cloud.agent.event_handler.register_event_action(). Register for the major version of GX Core that the action applies to, e.g.register_event_action("1", RunCheckpointEvent, RunCheckpointAction)registers the action for major version 1 of GX Core (e.g. 1.0.0). - Import your action in
great_expectations_cloud/agent/actions/__init__.py
Note: The Agent is core-version specific but this registration mechanism allows us to preemptively work on actions for future versions of GX Core while still supporting the existing latest major version.
Release Process
Versioning
This is the version that will be used for the Docker image tag as well.
Standard Release:
The versioning scheme is YYYYMMDD.{release_number} where:
- the date is the date of the release
- the release number starts at 0 for the first release of the day
- the release number is incremented for each release within the same day
For example: 20240402.0
Pre-release:
The versioning scheme is YYYYMMDD.{release_number}.dev{dev_number}
- the date is the date of the release
- the dev number starts at 0 for the first pre-release of the day
- the dev number is incremented for each pre-release within the same day
- the release number is the release that this pre-release is for
For example: 20240403.0.dev0 is the first pre-release for the 20240403.0 release.
For example, imagine the following sequence of releases given for a day with two releases:
20240403.0.dev020240403.0.dev120240403.020240403.1.dev020240403.1
There can be days with no standard releases, only pre-releases or days with no pre-release or standard release at all.
Pre-releases
Pre-releases are completed automatically with each merge to the main branch.
The version is updated in pyproject.toml and a pre-release is created on PyPi.
A new Docker tag will also be generated and pushed to Docker Hub
Manual Pre-releases
NOTE: CI will automatically create pre-releases on merges to main. Instead of manually creating pre-releases, consider using the CI process. This is only for exceptional cases.
To manually create a pre-release, run the following command to update the version in pyproject.toml and then merge it to main in a standalone PR:
invoke pre-release
This will create a new pre-release version. On the next merge to main, the release will be uploaded to PyPi.
A new Docker tag will also be generated and pushed to Docker Hub
Releases
Releases will be completed on a regular basis by the maintainers of the project and with any release of GX Core
For maintainers, to create a release, run the following command to update the version in pyproject.toml and then
merge it to main in a standalone PR:
invoke release
This will create a new release version. On the next merge to main, the release will be uploaded to PyPi.
A new Docker tag will also be generated and pushed to Docker Hub. In addition, releases will be tagged with stable and latest tags.
GitHub Workflow for releasing
We use the GitHub Actions workflow to automate the release and pre-release process. There are two workflows involved:
-
CI - This workflow runs on each pull request and will update the version in
pyproject.tomlto the pre-release version if the version is not already manually updated in the PR. It will also run the tests and linting. -
Containerize Agent - This workflows runs on merge with
mainand will create a new Docker image and push it to Docker Hub and PyPi. It uses the version inpyproject.toml.
A visual representation of the workflow is shown here
Dependabot and Releases/Pre-releases
GitHub's Dependabot regularly checks our dependencies for vulnerabilty-based updates and proposes PRs to update dependency version numbers accordingly.
Dependabot may only update the poetry.lock file. If only changes to poetry.lock are made, this may be done in a pre-release.
For changes to the pyproject.toml file:
- If the version of a tool in the
[tool.poetry.group.dev.dependencies]group is updated, this may be done without any version bump.- While doing this, make sure any version references in the pre-commit config
.pre-commit-config.yamlare kept in sync (e.g., ruff).
- While doing this, make sure any version references in the pre-commit config
- For other dependency updates or package build metadata changes, a new release should be orchestrated. This includes updates in the following sections:
[tool.poetry.dependencies][tool.poetry.group.*.dependencies]where*is the name of the group (not including thedevgroup)
- To stop the auto-version bump add the
no version bumplabel to the PR. Use this when:- Only modifying dev dependencies.
- Only modifying tests that do not change functionality.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file great_expectations_cloud-20240604.0.dev2.tar.gz.
File metadata
- Download URL: great_expectations_cloud-20240604.0.dev2.tar.gz
- Upload date:
- Size: 31.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.0.0 CPython/3.12.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 8e0de68401e11f9df16dcd3077020eff4652ab22a6d718a0c4b87ff40b1a2c9d |
|
| MD5 | 0f45936b8b2859c9e980fb4129548a28 |
|
| BLAKE2b-256 | 985fbd70a30cf58b9cb5a484352f1c33b601faeb0d4774132c02dceee60e5f3e |
File details
Details for the file great_expectations_cloud-20240604.0.dev2-py3-none-any.whl.
File metadata
- Download URL: great_expectations_cloud-20240604.0.dev2-py3-none-any.whl
- Upload date:
- Size: 37.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.0.0 CPython/3.12.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | e682d300e8c33a169494d2d9f9bd34d60e56a6a3e9189e862598f7a34c7abe06 |
|
| MD5 | 829d042554839d210b5865f09dfd9c96 |
|
| BLAKE2b-256 | 6b4db3f8a553d4f37b1e5a9e0c78c048388ec82bfbc4ac067a35dfca9268292b |
