pyawskit package¶
Submodules¶
pyawskit.aws module¶
aws.py
pyawskit.aws_codeartifact_npm_env_config_code module¶
This script outputs the url for pip to be used when logging in to aws codeartifact.
https://docs.aws.amazon.com/codeartifact/latest/ug/npm-auth.html
- Here is an example response:
- $ aws codeartifact login –tool npm –repository “$ENV_NPM_REPOSITORY” –domain “$ENV_DOMAIN”
–domain-owner “$ENV_DOMAIN_OWNER”
pyawskit.aws_codeartifact_pip_env_config_code module¶
This script outputs the url for pip to be used when logging in to aws codeartifact.
pyawskit.aws_ecr_login_code module¶
aws_ecr_login_code.py
- pyawskit.aws_ecr_login_code.handle_data(user: str, password: str, proxyEndpoint: str) None[source]¶
really do something with the data It seems that the docker-py is broken as far as auth goes so we use the command line instead. client = docker.Client() try:
- client.login(
username=user, password=password, registry=proxyEndpoint,
)
- except requests.exceptions.ConnectionError:
print(f”ConnectionError for {user} {password} {proxyEndpoint}”, file=sys.stderr) sys.exit(1)
- pyawskit.aws_ecr_login_code.is_logged_in(proxyEndpoint: str) bool[source]¶
return if you are currently logged in to a specific server References: - https://stackoverflow.com/questions/36022892/how-to-know-if-docker-is-already-logged-in-to-a-docker-registry-server
- pyawskit.aws_ecr_login_code.strip_scheme(url: str) str[source]¶
strip scheme from url References: - https://stackoverflow.com/questions/21687408/how-to-remove-scheme-from-url-in-python
pyawskit.common module¶
common.py
- pyawskit.common.wait_net_service(server: str, port: int, timeout: int = 600) bool[source]¶
Wait for network service to appear http://code.activestate.com/recipes/576655-wait-for-network-service-to-appear/ :param port: :param server: :param timeout
pyawskit.configs module¶
configs.py
- class pyawskit.configs.ConfigAwsCodeartifactNpm[source]¶
Bases:
Config- env_npm_domain = <pytconf.param.Unique object>¶
- env_npm_domain_owner = <pytconf.param.Unique object>¶
- env_npm_repository = <pytconf.param.Unique object>¶
- class pyawskit.configs.ConfigAwsCodeartifactPip[source]¶
Bases:
Config- env_pip_domain = <pytconf.param.Unique object>¶
- env_pip_domain_owner = <pytconf.param.Unique object>¶
- env_pip_repository = <pytconf.param.Unique object>¶
- class pyawskit.configs.ConfigFilter[source]¶
Bases:
ConfigParameters to select if the use the filter or not
- filter = True¶
- class pyawskit.configs.ConfigName[source]¶
Bases:
ConfigParameters to select the name of the config to launch
- name = <pytconf.param.Unique object>¶
- class pyawskit.configs.ConfigRoleDuplicate[source]¶
Bases:
Config- from_role = <pytconf.param.Unique object>¶
- to_role = <pytconf.param.Unique object>¶
- class pyawskit.configs.ConfigWork[source]¶
Bases:
ConfigParameters to control the actualy device work
- add_line_to_fstab = False¶
- device_file = '/dev/md0'¶
- file_system_type = 'ext4'¶
- fstab_filename = '/etc/fstab'¶
- mdadm_binary = '/sbin/mdadm'¶
- mdadm_config_file = '/etc/mdadm/mdadm.conf'¶
- mount_point = '/mnt/raid0'¶
- name_of_raid_device = 'MY_RAID'¶
- start_stop_queue = False¶
- write_etc_mdadm = False¶
pyawskit.devices module¶
devices.py
- pyawskit.devices.mount_disks() None[source]¶
This script mounts all the local disks as individuals
TODO: - make this run in parallel on multiple cores and enable the user to choose (via command line option) whether to run this multi-core or not.
- pyawskit.devices.unify_disks() None[source]¶
unify disks of a machine in raid 0
This script is derived from the following bash script: https://gist.github.com/joemiller/6049831
NOTES: - python has no support for “mount” and “umount” system calls. the result is that we use the command line tools to do mounting and un mounting. References: - http://stackoverflow.com/questions/1667257/how-do-i-mount-a-filesystem-using-python
References: - http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/raid-config.html - http://www.tecmint.com/create-raid0-in-linux/ - http://dev.bizo.com/2012/07/mdadm-device-or-resource-busy.html
TODO: - make this script work in parallel. But for the dd(1) and mkfs(1) parts.
pyawskit.inet module¶
inet.py
pyawskit.main module¶
main.py
- pyawskit.main.compress_s3_folder() None[source]¶
This script accepts three parameters: bucket, in-folder, out-folder It reads every file in in-folder, gzip it, and then writes it with the suffix “.gz” to the out folder.
The credentials for this are NOT stored in this script but rather are in ~/.aws/credentials.
- pyawskit.main.generate_etc_hosts() None[source]¶
This script will update ~/.aws/config file with the names of your machines. Notice that you must hold all of your .pem files in ~/.aws/keys
- pyawskit.main.generate_ssh_config() None[source]¶
This script will update ~/.aws/config file with the names of your machines. Notice that you must hold all of your .pem files in ~/.aws/keys
- pyawskit.main.generate_tilde_hosts() None[source]¶
This script will update ~/.hosts file with the names of your machines. You must use something like libnss_homehosts to use this file for each app.
- pyawskit.main.launch_machine() None[source]¶
This script launches a new machine via boto3 with configuration from ~/.pyawskit/launch_config.json
If you want to know how to edit this file look at: http://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.Client.request_spot_instances
References: - http://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.Client.run_instances - http://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.Client.request_spot_instances - https://stackoverflow.com/questions/35721557/boto3-spot-instance-creation - https://github.com/abdul/ec2-spot-instance-launcher - https://peteris.rocks/blog/script-to-launch-amazon-ec2-spot-instances/
This is the real script where most of this one is from: - https://gist.github.com/aloysius-lim/b77f6d62a595ae329e41
- pyawskit.main.prep_account()[source]¶
These are the types of things it does: - copy ~/.aws ~/.ssh ~/.s3cfg ~/.gitconfig ~/.passwd-s3fs to it - set quiet login using:
$ touch ~/.hushlogin
configure vim to do correct python editing and save editing positions.
install fancy prompt on it.
put the git repositories you want on it.
- pyawskit.main.prep_machine(name: str) None[source]¶
A python version of the bash script here: https://gist.github.com/joemiller/6049831
sudo purge-old-kernels –keep 1 # to remove old kernels
install a list of packages on the system:
- format and mount all disks
use mdadm
- set the hostname in /etc/hosts beside
127.0.0.1 localhost [hostname] to avoid problems resolving the host name. $ touch ~/.hushlogin
configure vim to do correct python editing and save editing positions.
install fancy prompt on it.
put the git repositories you want on it.
- Parameters:
name
- Returns:
pyawskit.os_utils module¶
os_utils.py
pyawskit.roles module¶
roles.py
pyawskit.static module¶
version which can be consumed from within the module
pyawskit.utils module¶
utils.py
- pyawskit.utils.copyfileobj(source, destination, buffer_size=1048576)[source]¶
Copy a file from source to destination. source and destination must be file-like objects, i.e. any object with a read or write method, like for example StringIO.
- pyawskit.utils.object_exists(s3_connection, check_bucket_name: str, object_name: str) bool[source]¶
It seems funny we have to write this method but it seems that this is the way to check if an s3 bucket has an object.
http://stackoverflow.com/questions/33842944/check-if-a-key-exists-in-a-bucket-in-s3-using-boto3
- Parameters:
s3_connection
check_bucket_name
object_name
- Returns:
whether the object exists
Module contents¶
Initialize the module