`konfetti` provides a framework-independent way for configuration of applications or libraries written in Python.
Project description
konfetti
Description
NOTE: The documentation is in progress
konfetti
provides a framework-independent way for configuration of applications or libraries written in Python.
Key features:
- Lazy evaluation;
- Built-in environment variables support;
- Built-in Vault support;
- Helpers for tests.
The primary motivation for building this library is to unify all configuration in different projects and make it as lazy as possible. In this case, we could get these benefits:
- No need to full app configuration when running a subset of tests that don't need a full config;
- Avoid network calls during imports until necessary;
The interface design and features are heavily inspired by Django
& decouple
.
Supported Python versions: 2.7 & 3.5 - 3.8
Quickstart
To use konfetti
you need to define:
- configuration variables in a module or a class;
- an access point;
Settings module
# app_name/settings/production.py
from konfetti import env, vault
VAULT_ADDR = env("VAULT_ADDR")
VAULT_TOKEN = env("VAULT_TOKEN")
DEBUG = env("DEBUG", default=False)
DATABASE_URI = vault("path/to/db")
NOTE: The naming convention for variables names is to use upper case, other variables will be ignored.
Access point
# app_name/settings/__init__.py
from konfetti import Konfig, VaultBackend
config = Konfig(vault_backend=VaultBackend("/secret/team"))
konfetti
relies on KONFETTI_SETTINGS
environment variable to discover your settings module, in the case above:
export KONFETTI_SETTINGS=app_name.settings.production
Usage
The settings module/class with configuration options shouldn't be accessed directly, because the aforementioned features are implemented in the access point level.
from app_name.settings import config
def something():
config.DATABASE_URI
API
Table of contents:
Lazy evaluation
Until a config option is accessed it is not evaluated - it is lazy. To avoid side effects on imports accessing configuration should be avoided on a module level.
This concept allows you to choose when you actually evaluate the config. Why?
- Testing. If you need to test a small piece of code that doesn't require any configuration - you don't have to setup it;
- Faster application startup; Use only what you need at the moment
It is still possible to evaluate the config eagerly on the app startup - access the needed variables in the entry points.
It could be done either with direct accessing needed variables or with config.require(...)
/ config.asdict()
calls.
Environment
from konfetti import env
VARIABLE = env("VARIABLE_NAME", default="foo")
Since environment variables are strings, there is a cast
option to convert
given variable from a string to the desired type:
from konfetti import env
VARIABLE = env("VARIABLE_NAME", default=42, cast=int)
You can pass any callable as cast
.
If there is a need to use the environment variable immediately, it could be
evaluated via str
call (other ways could be added on demand):
from konfetti import env, vault
DATABASE_ROLE = env("DATABASE_ROLE", default="booking")
DATABASE_URI = vault(f"db/{DATABASE_ROLE}")
If cast
is specified, then it will be applied before evaluation as well.
.env
support
It is possible to specify a path to the .env
file and it will be used as a source
of data for environment variables.
dotenv_override
parameter specifies whether the .env
value should be used if
both the environment variable and the .env
record exists, False
by default.
# app_name/settings/__init__.py
from konfetti import Konfig
config = Konfig(dotenv="path/to/.env", dotenv_override=False)
Vault
Backend configuration
To use Vault as a secrets storage you need to configure the access point:
# app_name/settings/__init__.py
from konfetti import Konfig, VaultBackend
config = Konfig(vault_backend=VaultBackend("your/prefix"))
There are two Vault backends available:
konfetti.VaultBackend
konfetti.AsyncVaultBackend
The main difference is that the latter requires using await
to access
the secret value (the call will be handled asynchronously under the hood), otherwise the interfaces and capabilities are the same.
Each backend requires a prefix
to be specified, the trailing / leading slashes don't matter,
"your/prefix"
will work the same as "/your/prefix/"
.
Usage
Every Vault secret needs a path
to be used as a lookup (leading and trailing slashes don't matter as well):
# app_name/settings/production.py
from konfetti import vault
WHOLE_SECRET = vault("path/to")
In this case all key/value pairs will be loaded on evaluation:
>>> from app_name.settings import config
>>> config.WHOLE_SECRET
{'key': 'value', 'foo': 'bar'}
You can specify a specific key to be returned for a config option with []
syntax:
# app_name/settings/production.py
from konfetti import vault
KEY = vault("path/to")["key"]
>>> from app_name.settings import config
>>> config.KEY
value
Using square brackets will not trigger evaluation - you could specify as many levels as you want:
# app_name/settings/production.py
from konfetti import vault
DEEP = vault("path/to")["deeply"]["nested"]["key"]
Casting could be specified as well:
# app_name/settings/production.py
from decimal import Decimal
from konfetti import vault
DECIMAL = vault("path/to", cast=Decimal)["fee_amount"] # stored as string
>>> from app_name.settings import config
>>> config.DECIMAL
Decimal("0.15")
Sometimes you need to access to some secrets dynamically. Konfig
provides a way to do it:
>>> from app_name.settings import config
>>> config.get_secret("path/to")["key"]
value
Secret files
It is possible to get a file-like interface for vault secret.
# app_name/settings/production.py
from konfetti import vault_file
KEY = vault_file("path/to/file")["key"]
>>> from app_name.settings import config
>>> config.KEY.readlines()
[b'value']
Defaults
It is possible to specify the default value for vault variable. Value could be
any type for a key in a secret and a dict
for the whole secret.
DEFAULT = vault("path/to", default="default")["DEFAULT"]
DEFAULT_SECRET = vault("path/to", default={"DEFAULT_SECRET": "default_secret"})
>>> from app_name.settings import config
>>> config.DEFAULT
"default"
>>> config.DEFAULT_SECRET
{"DEFAULT_SECRET": "default_secret"}
Defaults could be disabled entirely if VAULT_DISABLE_DEFAULTS
is set
$ export VAULT_DISABLE_DEFAULTS="true"
Overriding Vault secrets
In some cases, secrets need to be overridden in runtime on the application level. You can define some custom values for tests or you just want to run the app with some different configuration without changing data in Vault.
There is a way to do it using environment variables or .env
records
To redefine certain config option you need to redefine the whole secret with a JSON encoded string.
Example:
# app_name/settings/production.py
from konfetti import vault
KEY = vault("path/to")["key"]
>>> from app_name.settings import config
>>> config.KEY
value
>>> import os
>>> os.environ["PATH__TO"] = '{"key": "overridden"}'
>>> config.KEY
overridden
To check how to override certain option there is a config.vault.get_override_examples()
helper:
>>> config.vault.get_override_examples()
{
"NESTED_SECRET": {
"PATH__TO__NESTED": '{"NESTED_SECRET": {"nested": "example_value"}}'
},
"SECRET": {
"PATH__TO": '{"SECRET": "example_value"}'
},
"WHOLE_SECRET": {
"PATH__TO": "{}"
},
}
By default, when the evaluation will happen on a Vault secret, the environment will be checked first.
If you don't need this behavior, it could be turned off with try_env_first=False
option to the chosen backend:
# app_name/settings/__init__.py
from konfetti import Konfig, VaultBackend
config = Konfig(vault_backend=VaultBackend("your/prefix", try_env_first=False))
Disabling access to secrets
If you want to forbid any access to Vault (e.g. in your tests) you can set KONFETTI_DISABLE_SECRETS
environment
variable with 1
/ on
/ true
/ yes
.
>>> import os
>>> from app_name.settings import config
>>> os.environ["KONFETTI_DISABLE_SECRETS"] = "1"
>>> config.get_secret("path/to")["key"]
...
RuntimeError: Access to secrets is disabled. Unset KONFETTI_DISABLE_SECRETS variable to enable it.
Caching
Vault values could be cached in memory:
config = Konfig(vault_backend=VaultBackend("your/prefix", cache_ttl=60))
By default, caching is disabled.
Lazy options
If there is a need to calculate config options dynamically (e.g., if it depends on values of other options) konfetti
provides lazy
:
from konfetti import lazy
LAZY_LAMBDA = lazy(lambda config: config.KEY + "/" + config.SECRET + "/" + config.REQUIRED)
@lazy("LAZY_PROPERTY")
def lazy_property(config):
return config.KEY + "/" + config.SECRET + "/" + config.REQUIRED
Testing
It is usually a good idea to use a slightly different configuration for tests (disabled tracing, sentry, etc.).
export KONFETTI_SETTINGS=app_name.settings.tests
It is very useful to override some config options in tests. Konfig.override
will override config options defined
in the settings module. It works as a context manager or a decorator to provide explicit setup & clean up for overridden options.
from app_name.settings import config
# DEBUG will be `True` for `test_everything`
@config.override(DEBUG=True)
def test_everything():
# DEBUG will be `False` again for this block
with config.override(DEBUG=False):
...
Overrides could be nested, and deeper level has precedence over all levels above:
from app_name.settings import config
@config.override(FOO=1, BAR=2)
def test_many_things():
with config.override(BAR=3):
assert config.FOO == 1
assert config.BAR == 3
# As it was before
assert config.BAR == 2
Also, override works for classes (including inherited from unittest.TestCase
):
@config.override(INTEGER=123)
class TestOverride:
def test_override(self):
assert config.INTEGER == 123
@config.override(INTEGER=456)
def test_another_override(self):
assert config.INTEGER == 456
def test_not_affected():
assert config.INTEGER == 1
NOTE. setup_class/setUp
and teardown_class/tearDown
methods will work with override
.
konfetti
includes a pytest integration that gives you a fixture, that allows you to override given config without
using a context manager/decorator approach and automatically rollbacks changes made:
import pytest
from app_name.settings import config
from konfetti.pytest_plugin import make_fixture
# create a fixture. the default name is "settings",
# but could be specified via `name` option
make_fixture(config)
@pytest.fixture
def global_settings(settings):
settings.INTEGER = 456
@pytest.mark.usefixtures("global_settings")
def test_something(settings):
assert settings.INTEGER == 456
assert config.INTEGER == 456
# fixture overriding
settings.INTEGER = 123
assert settings.INTEGER == 123
assert config.INTEGER == 123
# context manager should work as well
with settings.override(INTEGER=7):
assert settings.INTEGER == 7
assert config.INTEGER == 7
# Context manager changes are rolled back
assert settings.INTEGER == 123
assert config.INTEGER == 123
# This test is not affected by the fixture
def test_disable(settings):
assert config.INTEGER == 1
assert settings.INTEGER == 1
NOTE. It is forbidden to create two fixtures from the same config instances.
Extras
The environment variable name could be customized via config_variable_name
option:
config = Konfig(config_variable_name="APP_CONFIG")
Alternatively, it is possible to specify class-based settings:
from konfetti import env, vault
class ProductionSettings:
VAULT_ADDR = env("VAULT_ADDR")
VAULT_TOKEN = env("VAULT_TOKEN")
DEBUG = env("DEBUG", default=False)
DATABASE_URI = vault("path/to/db")
It possible to load the whole config and get its content as a dict:
>>> config.asdict()
{
"ENV": "env value",
"KEY": "static value",
"SECRET": "secret_value",
}
If you need to validate that certain variables are present in the config, there is require
:
>>> config.require("SECRET")
...
MissingError: Options ['SECRET'] are required
Or to check that they are defined:
>>> "SECRET" in config
True
Configuration 101
There are a couple of principles that will help you to avoid problems when you specify or use your configuration.
Do not access configuration on the module level
Do this:
from app_name.settings import config
def get_redis_client():
return StrictRedis.from_url(config.REDIS_URL)
Instead of this:
from redis import StrictRedis
from app_name.settings import config
cache_redis = StrictRedis.from_url(config.REDIS_URL)
However, if you want to have a global Redis instance, consider using python-lazy-object-proxy
:
pip install lazy-object-proxy
import lazy_object_proxy
...
cache_redis = lazy_object_proxy.Proxy(get_redis_client)
Why?
Accessing configuration on the module level leads to side-effects on imports, this fact could produce unrelated errors when you run your test suite:
- Simple unit tests will fail due to lack of configuration options or Vault unavailability;
- Slow tests due to config initialization and long network calls (they could time out as well);
Having your config access lazy will prevent many for those cases because that code branches won't be executed on imports and will not affect your test suite.
Code formatting
In order to maintain code formatting consistency we use black to format the python files. A pre-commit hook that formats the code is provided but it needs to be installed on your local git repo, so...
In order to install the pre-commit framework run pip install pre-commit
or if you prefer homebrew brew install pre-commit
Once you have installed pre-commit just run pre-commit install
on your repo folder
If you want to exclude some files from Black (e.g. automatically generated database migrations or test snapshots) please follow instructions for pyproject.toml
Testing
To run all tests:
docker run -p 8200:8200 -d --cap-add=IPC_LOCK -e 'VAULT_DEV_ROOT_TOKEN_ID=test_root_token' vault:0.9.6
tox -p all
Note that tox doesn't know when you change the requirements.txt
and won't automatically install new dependencies for test runs.
Run pip install tox-battery
to install a plugin which fixes this silliness.
It also possible to run tests via docker-compose that will start up all required environment:
$ make docker-test
or alternatively:
$ docker-compose -f docker-compose-tests.yml run konfetti
Contributing
TODO
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for konfetti-0.6.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 04bb557dda19cd3bf343d4bdc02b3212f75a2e535092260fc4bfb3908ddfc819 |
|
MD5 | 1fa2f8d0dcdd712a4a36104ac424a678 |
|
BLAKE2b-256 | 1f3597805c34f9d49872797c440903d30cc6a228dd7bbb6e308dc856aa383b80 |