A CLI tool used to deploy ephemeral environments for testing cloud.redhat.com applications
Project description
bonfire
A CLI tool used by Red Hat engineers to deploy console.redhat.com applications into kubernetes/OpenShift. This tool is mainly used for the purpose of creating ephemeral test environments.
As an example, typing bonfire deploy host-inventory leads to the host-inventory application and all its dependencies being deployed and ready to run in under 4 minutes:
Table of Contents
- About
- Installation
- Quick Start
- Examples for Common Use Cases
- Commonly Used CLI Options
- Interactions with Ephemeral Namespace Operator
- Interactions with Clowder Operator
- Configuration Details
About
bonfire interacts with a running instance of qontract-server (the component that powers the AppSRE team's internal app-interface graphql API) to obtain applications' OpenShift templates, process them, and deploy them. A local configuration file can be defined that allows you to override an application config if you wish.
It also interacts with the ephemeral namespace operator to manage the reservation of ephemeral namespaces for testing.
It has special functionality related to the Clowder operator and Frontend operator such as auto-deploying ClowdApp dependencies.
A wide range of CLI options allow you to customize exactly what combination of components get deployed into a namespace.
NOTE: for information related to app-interface configurations, see the internal ConsoleDot Docs
Installation
Installing locally
We'd recommend setting up a virtual environment for bonfire:
VENV_DIR=~/bonfire_venv
mkdir -p $VENV_DIR
python3 -m venv $VENV_DIR
. $VENV_DIR/bin/activate
pip install crc-bonfire
To prevent GitHub rate limiting issues when bonfire reaches out to GitHub APIs, create a personal access token on your GitHub account which grants bonfire read access to your repos.
Configure bonfire to use the token with:
echo 'GITHUB_TOKEN=<your api token>' >> ~/.config/bonfire/env
Please note: When using the Github token (classic), it must have the read:project scope.
In addition, since some projects may be private, you will need to add the repo (full control of private repositories) scope too.
for the template fetching to work!
Using a container image (podman/docker)
We provide a container image at quay.io/cloudservices/bonfire. The image wraps a Python virtual environment with crc-bonfire installed on it.
In order to use Bonfire you have to provide either a valid KUBECONFIG or a URL and a TOKEN to connect to the target OpenShift cluster.
The container's entrypoint already invokes the bonfire CLI with the arguments passed to the container, so you should be able to run any bonfire command you'd typically run, for example:
podman run -it --rm quay.io/cloudservices/bonfire version
Using credentials
The Bonfire image can use the OC_LOGIN_SERVER and OC_LOGIN_TOKEN variables to connect to an Openshift cluster.
podman run -it --rm -e OC_LOGIN_SERVER=https://api.openshift.example.com:6443 \
-e OC_LOGIN_TOKEN=some-api-token \
quay.io/cloudservices/bonfire namespace list
You can also make use of the kubeconfig file on your host by mounting it into the container's $HOME directory:
podman run -it --rm --userns=keep-id:uid=1000,gid=1000 -v ${KUBECONFIG:="$HOME/.kube/config"}:/opt/bonfire/.kube/config:Z \
quay.io/cloudservices/bonfire namespace list
If you desire for the kubeconfig to live in a non-standard location in the container, you can mount the kubeconfig into another path in the container, and set the KUBECONFIG environment variable:
podman run -it --rm --userns=keep-id:uid=1000,gid=1000 -v $HOME/.kube/config:/opt/kubeconfig:Z \
-e KUBECONFIG=/opt/kubeconfig \
quay.io/cloudservices/bonfire namespace list
Quick Start
Start with bonfire --help to familiarize yourself with the basic command groups. Don't forget to use the --help flag on sub-commmands as well if you are curious to learn more about different CLI commands, options, and arguments.
Deploying
The command you are most likely to use is bonfire deploy.
If you are on the Red Hat VPN, bonfire will communicate with an internal instance of the App-SRE team's qontract-server API. You need to use oc login to log in to the ephemeral cluster.
Once logged in, you should be able to type bonfire deploy advisor
This will cause bonfire to do the following:
- reserve a namespace
- fetch the templates for all components in the 'advisor' app and process them. By default, bonfire will look up the template and commit hash using the 'main'/'master' branch of each component defined in the app. It will look up the git commit hash for this branch and automatically set the
IMAGE_TAGparameter passed to the templates. - if ClowdApp resources are found, figure out which additional ClowdApps to fetch and process
- apply all the processed template resources into your reserved namespace
- wait for all the resources to reach a 'ready' state
Run bonfire namespace list --mine and you will see your namespace.
Switch into that namespace with oc project <NAMESPACE>
Run oc get clowdapp in your namespace and you should see several ClowdApps have been deployed.
If you want to to extend the time you have to work with your namespace, use bonfire namespace extend <NAMESPACE>
When you are done working with your namespace, type bonfire namespace release <NAMESPACE> to instruct the ephemeral namespace operator to delete your reservation.
The deploy command is an 'all-in-one' command that ties together 4 steps:
bonfire namespace reserveto reserve a namespacebonfire processto fetch and process app templatesoc applyto apply the processed app templates into the namespacebonfire namespace wait-on-resources <NAMESPACE>to wait on all resources in the namespace to reach 'ready' state
Namespace Management
- Reserve a namespace with:
bonfire namespace reserve- By default, the duration is 1 hour. Increase it with
-d/--duration <time>-- example time format:48h - Different namespace pools are set up to provide different test environment configurations. Use
--pool <pool>if you need to select a non-default pool. To get a list of valid pools, usebonfire pool list
- By default, the duration is 1 hour. Increase it with
- Look up namespaces with:
bonfire namespace list- use
--mineto see only ones reserved in your name
- use
- Extend your reservation with:
bonfire namespace extend <NAMESPACE> -d <time>-- example time format:48h - Release your namespace reservation with:
bonfire namespace release <NAMESPACE>
Namespace Context
As of v4.12.0, bonfire now analyzes the namespace set in your oc/kubectl context. This means you can run commands such as namespace extend, namespace release, or deploy after you have switched into a namespace using oc project and you no longer need to specify the namespace on the CLI. bonfire will attempt to use your current namespace (it will run checks to ensure that you own it first and warn if you do not). In addition, when a new namespace is reserved using bonfire deploy or bonfire namespace reserve, bonfire will also go ahead and run oc project <NAMESPACE> for you to switch your context into that new namespace.
The deploy command can be run using the --reserve flag if you wish to ignore the current namespace and force a namespace reservation to occur.
The automatic 'oc' context switching into a newly reserved namespace is disabled when bonfire is operating in "bot" mode (i.e. when the env var BONFIRE_BOT is set to true). When using bonfire in an automated CI/CD environment such as Jenkins, you should run in bot mode to disable automatic context switching.
Examples for Common Use Cases
In these examples, we will use the 'advisor' app which has a ClowdApp component named 'advisor-backend'
- Deploy advisor-backend using a custom git branch:
bonfire deploy advisor --set-template-ref advisor-backend=custom_branch
- Deploy advisor-backend using a custom git commit:
bonfire deploy advisor --set-template-ref advisor-backend=df0d7a620b1ac02e59c5718eb22fe82b8a4cfd3d
- Use a certain branch to process the advisor-backend template, but set a different image tag on the container:
bonfire deploy advisor --set-template-ref advisor-backend=some_branch --set-parameter advisor-backend/IMAGE_TAG=some_tag
- Deploy the production version of advisor and its dependencies:
bonfire deploy advisor --ref-env insights-production
- Deploy the production version of everything but use a dev branch for advisor-backend:
bonfire deploy advisor --ref-env insights-production --set-template-ref advisor-backend=dev_branch
- Adjust the 'REPLICAS' parameter value at deploy time:
bonfire deploy advisor --set-parameter advisor-backend/REPLICAS=3
- Override the tag for all occurences of a specific container image:
bonfire deploy advisor --set-image-tag quay.io/cloudservices/advisor-backend=my_tag
Commonly Used CLI Options
Deploying/Processing
--frontends=true-- by default, bonfire will not deploy Frontend resources. Use this flag to enable the deployment of app frontends.--set-image-tag <image uri>=<tag>-- use this to change the image tag of a container image that appears in your templates. This does a search/replace for all occurences of<image uri>and sets the tag to be<tag>--target-env-- this changes which deploy targetbonfirelooks up to determine the parameters to apply when processing app templates. By default, the target env is set toinsights-ephemeral--ref-env-- this changes which deploy targetbonfirelooks up to determine the git ref to fetch for your app configuration. For example, using--ref-env insights-stagewould cause bonfire to deploy the stage git ref of your app and--ref-env insights-productionwould cause bonfire to deploy the production git ref of your app. If a target matching the environment is not found,bonfiredefaults back to deploying the 'main'/'master' branch.--set-template-ref <component>=<ref>-- use this to change the git ref deployed for just a single component.--set-parameter <component>/<name>=<value>-- use this to set a parameter value on a specific component's template.--optional-deps-method <hybrid|all|none>-- change the way that bonfire processes ClowdApp optional dependencies (see "Dependency Processing" section)--prefer PARAM_NAME=PARAM_VALUE-- in cases where bonfire finds more than one deployment target, use this to set the parameter names and values that should be used to select a "preferred" deployment target. This option can be passed in multiple times.bonfirewill select the target with the highest amount of "preferred parameters" on it. Default is currently set toENV_NAME=frontendsto select "stable" frontends in the consoledot environments.
Trusted/Untrusted Resource Configurations
When templates are processed, bonfire removes all untrusted CPU/Memory requests and limits from ClowdApp/ClowdJob/ClowdJobInvocation objects within OpenShift templates (this is because the default value for --remove-resources is all). If the resource config is unset in a ClowdApp, Clowder will set the deployments to use default values defined in the “ClowdEnvironment” to take effect.
The reason for this behavior is because many app templates tend to request far more CPU/memory in their template compared to what they actually need in a test environment.
You can use the --no-remove-resources option to turn this behavior off for certain apps or components but the preferred approach is to audit your CPU/memory configurations and set up a resource configuration that bonfire trusts.
Once you have audited your container CPU/mem usage in test environments and have determined efficient values that you want to use, you can indicate that bonfire should trust the CPU/Memory requests and limits by adhering to the following requirements:
- Any ClowdApp/ClowdJob/ClowdJobInvocation in your template asking for CPU/mem using requests or limits should use a parameter with a name formatted like this:
-
CPU_REQUEST_<NAME> -
CPU_LIMIT_<NAME> -
MEM_REQUEST_<NAME>orMEMORY_REQUEST_<NAME> -
MEM_LIMIT_<NAME>orMEMORY_LIMIT_<NAME>Where
<NAME>can be whatever arbitrary name you'd like to use containingA-Z,0-9,and_. For example, either of these would be valid:CPU_REQUEST_MQ_CONSUMERMEM_LIMIT_API
NOTE: If you only have a single component in your template, you can omit
_<NAME>from the parameter names if you desire.
- Your deployment configuration for the component must explicitly define values for these parameters. For ephemeral environments normally this will mean that your parameters are defined under the ephemeral deploy target in app-interface.
If these steps are followed then your resource config is considered "trusted" and bonfire will not remove the requests/limits configurations.
NOTE: a couple things that will change in the future:
- The "host-inventory" app is trusted by default, but once their configurations are tweaked to be trusted we will remove this
- Object types such as Deployment/StatefulSet/DaemonSet/etc. are not currently analyzed. In future once teams have audited all container resource configurations, we will begin to have bonfire check for trusted configurations on all object types.
Interactions with Ephemeral Namespace Operator
bonfire creates/modifies NamespaceReservation resources on the cluster to reserve and release namespaces. The template for the object that bonfire applies into the cluster can be found here
For more information about how the ephemeral namespace operator works, see its documentation here
Interactions with Clowder Operator
ClowdEnvironments
-
bonfireexpects a ClowderClowdEnvironmentresource to be created for a namespace before it can deploy into it. ThisClowdEnvironmentis already created by the ephemeral namespace operator. -
When
bonfire deployis executed for a namespace, it will attempt to find the ClowdEnvironment associated with that namespace and set theENV_NAMEparameter accordingly for all templates it processes. All templates that define aClowdAppresource should set theenvironmentmapping in their spec using a parameter named${ENV_NAME}.
Dependency Processing
In a ClowdApp definition, an app listed under dependencies is considered an app that MUST be present for your app to function, while an app listed under optionalDependencies is considered an app that might be absent, but your app could still come up and function without it.
When bonfire processes templates, if it finds a ClowdApp, it will do the following:
-
if the ClowdApp has
dependencies, it will try to look up the config for that ClowdApp in app-interface and also fetch its templates -
if the ClowdApp has
optionalDependencies, it will behave differently based on which optional dependencies method is selected:hybrid(default): if an app group was explicitly specified on the CLI (example: when runningbonfire deploy app-athe app group 'app-a' is being explicitly specified) then theoptionalDependencieswill be deployed for it. For any ClowdAppbonfirecomes across that is not a member of an explicitly specified app group, theoptionalDependencieswill NOT be deployed. As an example:app-ahasapp-b-clowdapplisted underoptionalDependenciesapp-b-clowdapphasapp-c-clowdapplisted under itsoptionalDependencies- You will end up with all components of
app-aandapp-b-clowdappin your namespace.app-c-clowdappwould NOT be deployed into the namespace.
all:bonfirewill deploy alloptionalDependenciesfor all ClowdApps it encounters. It will recursively process dependencies. As an example:app-ahasapp-b-clowdapplisted underoptionalDependenciesapp-b-clowdapphasapp-c-clowdapplisted under itsoptionalDependencies- You will end up with all components of
app-a,app-b-clowdapp, ANDapp-c-clowdappdeployed into the namespace.
none:bonfirewill ignore theoptionalDependencieson all ClowdApps that it encounters
Configuration Details
NOTE: for information related to app-interface configurations, see the internal ConsoleDot Docs
When running bonfire process/bonfire deploy, by default the app-sre team's internal GraphQL API server is used. bonfire will query the GraphQL API and read the application's deploy configuration.
NOTE: You need to be on the internal VPN in order to access the default GraphQL API URL.
bonfire relies on a few key pieces of info to process app configs:
- The application name. This is typically the name of the listed in
app.yamlinapp-interface - A 'target env' -- the name of the
app-interfaceenvironment that bonfire looks at to determine the parameters to apply when processing app templates. An app's config will only be processed if it has a deploy target set up that is pointing to a namespace in this environment (default: "ephemeral") - A 'ref env' -- the name of the
app-interfaceenvironment that we want to use in order to figure out which git ref of the components to deploy and whichIMAGE_TAGvalue to set. By default, none is set which causes bonfire to use the 'main'/'master' git branch for all components. - Any template refs you wish to override (optional)
- Any image tags you wish to override (optional)
- Any parameters you wish to override (optional)
By default, if app components use ClowdApp resources in their templates, then bonfire will dynamically load dependencies (see "Dependency Processing" section)
Using a local config
bonfire ships with a default config that should be enough to get started for most internal Red Hat employees.
By default, the configuration file will be stored in ~/.config/bonfire/config.yaml.
If you wish to override any app configurations, you can edit your local configuration file by typing bonfire config edit. You can then define an app under the apps key of the config. You can reset the config to default at any time using bonfire config write-default.
As of bonfire v5, there are two options for how the local configuration is loaded (controlled by the --local-config-method CLI option). Let's say we have an app configured in app-interface like this:
resourceTemplates:
- name: mycomponent1
path: /deployment.yaml
url: https://github.com/myorg/myrepo1
targets:
- namespace:
$ref: /services/insights/ephemeral/namespaces/ephemeral-base.yml
parameters:
MIN_REPLICAS: 1
- name: mycomponent2
path: /deployment.yaml
url: https://github.com/myorg/myrepo2
targets:
- namespace:
$ref: /services/insights/ephemeral/namespaces/ephemeral-base.yml
parameters:
MIN_REPLICAS: 2
SOME_OTHER_PARAM: some_other_value
and the local config file has this entry:
apps:
- name: myapp
components:
- name: mycomponent2
parameters:
MIN_REPLICAS: 10
bonfire can override the remote configuration using one of two methods:
-
--local-config-method merge(default). In this mode, the apps config in your local config is merged with the configuration that bonfire fetched remotely. With the above config, only the 'MIN_REPLICAS' parameter of 'mycomponent2' within app 'myapp' will be overridden. The 'SOME_OTHER_PARAM' parameter will still be present, and 'mycomponent1' would be unchanged. -
--local-config-method override. In this mode, the local app configuration will take precedence over the app's configuration that bonfire fetched remotely. In other words, defining a local app configuration will replace the configuration defined in app-interface for ALL components within that app. So 'mycomponent1' would be completely removed, and 'mycomponent2' would only have the parameters you defined in the local config.
Local config examples
Deploying local repo changes
If you wanted to test out some changes you're making to an app but you have not yet pushed these changes to a git repo, the local config could look similar to:
apps:
- name: my_app
components:
- name: service
host: local
repo: ~/dev/projects/my-app
path: /clowdapp.yaml
parameters:
SOME_PARAMETER: some_value
- Where host set
localindicates to look for the repo in a local directory - Where repo indicates the path to your git repo folder
- Where path specifies the relative path to your ClowdApp template file in your git repo
By default, bonfire will run git rev-parse to determine the current working commit hash in your repo folder, and this determines what IMAGE_TAG to set when the template is deployed. This means you would need to have a valid container image pushed for this commit hash. However, you can use --set-image-tag/--set-parameter or define the IMAGE_TAG parameter in your config in order to override the image you wish to use during the deployment.
Deploying changes changes in a remote git branch
If you wanted to deploy changes to your app that you were working on which have been pushed to a PR/branch, the local config could look similar to:
apps:
- name: my_app
components:
- name: service
host: github
repo: MyOrg/MyRepo
ref: my_branch
path: /clowdapp.yaml
- Where host set to
githuborgitlabindicates where the repo is hosted - Where repo indicates the
OrganizationName/RepoName - Where path specifies the relative path to your ClowdApp template file in your git repo
- Where ref specifies the branch or commit of the repo you want to deploy
By default, bonfire will use the latest commit hash found at the specified git ref to determine what IMAGE_TAG pass on to the deployment templates.
Advanced
Running a local qontract-server
bonfire is configured to point to the app-sre team's internal GraphQL API by default, but by using env variables you can point to your your own instance of qontract-server:
export QONTRACT_BASE_URL="https://localhost:4000/graphql"
export QONTRACT_USERNAME=myUsername
export QONTRACT_PASSWORD=myPassword
For testing/debug purposes, instead of committing changes directly to app-interface, you can run your own local copy of the app-interface API server.
- Clone https://github.com/app-sre/qontract-server
- Clone the internal
app-interfacerepo
In qontract-server, run:
npm install yarn
yarn install
yarn build
make bundle APP_INTERFACE_PATH=/path/to/app-interface
LOAD_METHOD=fs DATAFILES_FILE=bundle/bundle.json yarn run server
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for crc_bonfire-5.10.4-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 57da0555dae10deedbfcfba987404cbd1c55ed9bbeb95533093733521f9633af |
|
| MD5 | d9cd7aab77195ae1ccc5a41864f605eb |
|
| BLAKE2b-256 | 244fcb521b0daf77861e0d1117cab5e14fe7c54b1f76b9b42b2d48509c757e72 |
