No project description provided
Project description
AWS Cloud Provider for Idem
INSTALLATION
The aws idem provider can be installed via pip:
pip install "idem-aws [full]"
DEVELOPMENT
Clone the idem-aws repository and install with pip. Install extra requirements by specifying “localstack” or “full” in brackets (in ZSH you will have to escape the brackets).
git clone git@gitlab.com:saltstack/pop/idem-aws.git
pip install -e idem-aws[full] # [google_auth,localstack,full]Set UP
After installation the AWS Idem Provider execution and state modules will be accessible to the pop hub. In order to use them we need to set up our credentials.
Create a new file called credentials.yaml and populate it with credentials. If you are using localstack, then the id and key can be bogus values. The default profile will be picked up automatically by idem.
There are many ways aws providers/profiles can be stored. See acct backends for more information.
There are multiple authentication backends for idem-aws which each have their own unique set of parameters. The following examples show some of the parameters that can be used in these backends to define profiles. All backends end up creating a boto3 session under the hood and storing it in the ctx variable that gets passed to all idem exec and state functions.
All authentication backends support two optional parameters, endpoint_url and provider_tag_key. The endpoint url is used to specify an alternate destination for boto3 calls, such as a localstack server or custom dynamodb server. The provider_tag_key is used when creating new resources. idem-aws will only interact with resources that are tagged with the the customizable provider_tag_key key.
credentials.yaml:
aws:
default:
endpoint_url: http://localhost:4566
use_ssl: False
aws_access_key_id: localstack
aws_secret_access_key: _
region_name: us-west-1Additionally, you can use AWS AssumeRole with Idem
aws:
default:
endpoint_url: http://localhost:4566
use_ssl: False
aws_access_key_id: localstack
aws_secret_access_key: _
region_name: us-west-1
assume_role:
role_arn: arn:aws:iam::999999999999999:role/xacct/developer
role_session_name: IdemSessionNameYou can also authenticate with aws-google-auth if it is installed.
aws.gsuite:
my-staging-env:
username: user@gmail.com
password: this_is_available_but_avoid_it
role_arn: arn:aws:iam::999999999999999:role/xacct/developer
idp_id: 9999999
sp_id: 999999999999
region: us-east-1
duration: 36000
account: developerThe google profile example is not named default. To use it, it will need to be specified explicitly in an idem state.
ensure_resource_exists:
aws.ec2.vpc.present:
- acct_profile: my-staging-env
- name: idem_aws_vpc
- cidr_block: 10.0.0.0/24It can also be specified from the command line when executing states.
idem state --acct-profile my-staging-env my_state.slsIt can also be specified from the command line when calling an exec module directly.
idem exec --acct-profile my-staging-env boto3.client.ec2.describe_vpcsThe last step to get up and running is to encrypt the credentials file and add the encryption key and encrypted file path to the ENVIRONMENT.
The acct command should be available as acct is a requisite of idem and idem-aws. Encrypt the the credential file.
acct encrypt credentials.yamloutput:
-A9ZkiCSOjWYG_lbGmmkVh4jKLFDyOFH4e4S1HNtNwI=
Add these to your environment:
export ACCT_KEY="-A9ZkiCSOjWYG_lbGmmkVh4jKLFDyOFH4e4S1HNtNwI="
export ACCT_FILE=$PWD/credentials.yaml.fernetYou are ready to use idem-aws!!!
LOCALSTACK
Localstack can be used to test idem-aws on your local machine without needing legitimate aws credentials. It can be used for running the idem-aws tests or for testing your states locally.
Install localstack with pip:
pip install "localstack [full]"Start the localstack infrastructure:
localstack infra startTESTING
In order to run the tests you must have a profile called “test_development_idem_aws” in your acct provider information. This can use localstack (recommended) or valid production aws credentials (at your own risk).
aws:
test_development_idem_aws:
endpoint_url: http://localhost:4566
use_ssl: False
aws_access_key_id: localstack
aws_secret_access_key: _
region_name: us-west-1It’s recommended to run the tests using localstack (by specifying your localstack container address as your endpoint_url)
pytest idem-aws/testsEXECUTION MODULES
Once everything has been set up properly, execution modules can be called directly by idem. Execution modules mirror the namespacing of the boto3.client and boto3.resource modules and have the same parameters.
For example, this is how you could list Vpcs from the command line with idem:
idem exec boto3.client.ec2.describe_vpcsYou can specify parameters as well. In the case of boto3 resources, args will be passed to the resource constructor and kwargs will be passed to the operation like so:
idem exec boto3.resource.ec2.Vpc.create_subnet vpc-71d00419 CidrBlock="10.0.0.0/24"STATES
States are also accessed by their relative location in idem-aws/idem_aws/states. For example, idem-aws/idem_aws/states/aws/ec2/vpc.py contains a function absent(). In my state file I can create a state that uses the absent function like so.
my_state.sls:
idem_aws_test_vpc:
aws.ec2.vpc.absent:
- name: "idem_aws_test_vpc"This state can be executed with:
idem state my_state.slsidem state also has some flags that can significantly boost the scalability and performance of the run. Let’s use this new state which verifies that 100 vpcs are absent:
{% for i in range(100) %}
idem_aws_test_vpc_{{i}}:
aws.ec2.vpc.absent:
- name: "idem_aws_test_vpc_{{i}}"
{% endfor -%}State can be executed with –runtime parallel to make full use of idem’s async execution calls:
idem state --runtime parallel my_state.slsRemote storage for enforced state management
Idem-aws supports remote storage for Idem’s enforced state management feature. That is, Idem can store esm data on AWS S3 bucket. DynamoDB will be used as a file lock to prevent multiple users/processes access the same storage file concurrently. To use remote storage, the esm profile need to be added to the credential profile like the following:
aws:
default:
use_ssl: True
aws_access_key_id: AAAAAAAAA5CDFSDER3UQ
aws_secret_access_key: eHjPASFWERSFwVXKlsdfS4afD
region_name: eu-west-2
esm:
bucket: "idem-state-storage-bucket"
dynamodb_table: "idem-state-storage-table"
key: "/idem-state/demo-storage.json"This esm file means that Idem will use AWS S3 bucket “idem-state-storage-bucket” and DynamoDB table “idem-state-storage-table” in region eu-west-2. The “key” is the file path to which the esm data will be read and stored. Both S3 bucket and DynamoDB table need to be created before using the feature. The DynamoDB table should have the primary key as string “LockID” and nothing else.
Current Supported Resources states
ec2
elastic_ip
flow_log
instance
internet_gateway
nat_gateway
route_table
spot_instance
subnet
transit_gateway
transit_gateway_vpc_attachment
vpc
iam
instance_profile
policy
role
role_policy
role_policy_attachment
user
kms
key
organizations
organization
organization_unit
s3
bucket
autoscaling
launch_configuration
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file idem-aws-0.22.0.tar.gz.
File metadata
- Download URL: idem-aws-0.22.0.tar.gz
- Upload date:
- Size: 221.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.9 tqdm/4.64.0 importlib-metadata/4.11.3 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 7f0d1d92286434fa001f4ccc911a0227a59f145d6a5518f7c5123fc8e8a2364e |
|
| MD5 | 837ce5df255a72633de4c52610304e8b |
|
| BLAKE2b-256 | 4b5c5636e429451dbd78485246b94cc889b4272a240f2105ac43623c5813cabe |
File details
Details for the file idem_aws-0.22.0-py3-none-any.whl.
File metadata
- Download URL: idem_aws-0.22.0-py3-none-any.whl
- Upload date:
- Size: 332.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.9 tqdm/4.64.0 importlib-metadata/4.11.3 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 3c93ab67ff66793d390d699546fdd50e63d82e06b2d28f9446020d50a90a6396 |
|
| MD5 | da577ece497dda49156026d4f505d373 |
|
| BLAKE2b-256 | 8c3df56af8e94fd2b889373216c199dc6ef1d1e2bd05d298b81280ae08be40c5 |
