pyawskit package

Submodules

pyawskit.aws module

aws.py

class pyawskit.aws.ProcessData(name: str)[source]

Bases: object

get_price()[source]
pyawskit.aws.attach_disk(*kw, **tw)[source]
pyawskit.aws.attach_disks(*kw, **tw)[source]
pyawskit.aws.log_func_name(func: Callable)[source]
pyawskit.aws.poll_instances_till_done(*kw, **tw)[source]
pyawskit.aws.poll_requests_till_done(*kw, **tw)[source]
pyawskit.aws.request_spot_instances(*kw, **tw)[source]
pyawskit.aws.tag_resources(*kw, **tw)[source]
pyawskit.aws.wait_for_ssh(*kw, **tw)[source]
pyawskit.aws.wait_using_waiter(*kw, **tw)[source]

pyawskit.aws_codeartifact_npm_env_config_code module

This script outputs the url for pip to be used when logging in to aws codeartifact.

https://docs.aws.amazon.com/codeartifact/latest/ug/npm-auth.html

Here is an example response:
$ aws codeartifact login –tool npm –repository “$ENV_NPM_REPOSITORY” –domain “$ENV_DOMAIN”

–domain-owner “$ENV_DOMAIN_OWNER”

pyawskit.aws_codeartifact_npm_env_config_code.handle_data(url: str, short_url: str, password: str, replace: bool) → None[source]

really handle the data

pyawskit.aws_codeartifact_npm_env_config_code.run() → None[source]

pyawskit.aws_codeartifact_pip_env_config_code module

This script outputs the url for pip to be used when logging in to aws codeartifact.

pyawskit.aws_codeartifact_pip_env_config_code.handle_data(url: str) → None[source]

really do something with the data

pyawskit.aws_codeartifact_pip_env_config_code.run() → None[source]

pyawskit.aws_ecr_login_code module

aws_ecr_login_code.py

pyawskit.aws_ecr_login_code.handle_data(user: str, password: str, proxyEndpoint: str) → None[source]

really do something with the data It seems that the docker-py is broken as far as auth goes so we use the command line instead. client = docker.Client() try:

client.login(

username=user, password=password, registry=proxyEndpoint,

)

except requests.exceptions.ConnectionError:

print(f”ConnectionError for {user} {password} {proxyEndpoint}”, file=sys.stderr) sys.exit(1)

pyawskit.aws_ecr_login_code.is_logged_in(proxyEndpoint: str) → bool[source]

return if you are currently logged in to a specific server References: - https://stackoverflow.com/questions/36022892/how-to-know-if-docker-is-already-logged-in-to-a-docker-registry-server

pyawskit.aws_ecr_login_code.logout(proxyEndpoint: str) → None[source]

logout of docker

pyawskit.aws_ecr_login_code.run() → None[source]
pyawskit.aws_ecr_login_code.strip_scheme(url: str) → str[source]

strip scheme from url References: - https://stackoverflow.com/questions/21687408/how-to-remove-scheme-from-url-in-python

pyawskit.common module

common.py

pyawskit.common.all_regions() → list[str][source]
pyawskit.common.check_root() → None[source]

check that we are running as root

pyawskit.common.do_hush_login()[source]
pyawskit.common.erase_partition_table(disk: str) → None[source]
pyawskit.common.format_device(disk: str, label: str | None = None) → None[source]
pyawskit.common.get_disks() → list[str][source]
pyawskit.common.load_json_config(name: str)[source]
pyawskit.common.mount_disk(disk: str, folder: str) → None[source]
pyawskit.common.reread_partition_table() → None[source]
pyawskit.common.run_devnull(args: list[str]) → None[source]
pyawskit.common.touch(filename: str) → None[source]
pyawskit.common.update_etc_hosts(all_hosts: bool)[source]
pyawskit.common.update_file(filename: str, pattern: str, do_all: bool = False)[source]
pyawskit.common.update_ssh_config(all_hosts: bool)[source]
pyawskit.common.wait_net_service(server: str, port: int, timeout: int = 600) → bool[source]

Wait for network service to appear http://code.activestate.com/recipes/576655-wait-for-network-service-to-appear/ :param port: :param server: :param timeout

pyawskit.configs module

configs.py

class pyawskit.configs.ConfigAwsCodeartifactNpm[source]

Bases: Config

env_npm_domain = <pytconf.param.Unique object>
env_npm_domain_owner = <pytconf.param.Unique object>
env_npm_repository = <pytconf.param.Unique object>
class pyawskit.configs.ConfigAwsCodeartifactPip[source]

Bases: Config

env_pip_domain = <pytconf.param.Unique object>
env_pip_domain_owner = <pytconf.param.Unique object>
env_pip_repository = <pytconf.param.Unique object>
class pyawskit.configs.ConfigFilter[source]

Bases: Config

Parameters to select if the use the filter or not

filter = True
class pyawskit.configs.ConfigName[source]

Bases: Config

Parameters to select the name of the config to launch

name = <pytconf.param.Unique object>
class pyawskit.configs.ConfigRole[source]

Bases: Config

role = <pytconf.param.Unique object>
class pyawskit.configs.ConfigRoleDuplicate[source]

Bases: Config

from_role = <pytconf.param.Unique object>
to_role = <pytconf.param.Unique object>
class pyawskit.configs.ConfigWork[source]

Bases: Config

Parameters to control the actualy device work

add_line_to_fstab = False
device_file = '/dev/md0'
file_system_type = 'ext4'
fstab_filename = '/etc/fstab'
mdadm_binary = '/sbin/mdadm'
mdadm_config_file = '/etc/mdadm/mdadm.conf'
mount_point = '/mnt/raid0'
name_of_raid_device = 'MY_RAID'
start_stop_queue = False
write_etc_mdadm = False

pyawskit.devices module

devices.py

pyawskit.devices.check_unmounted(logger, disks)[source]
pyawskit.devices.create_new_device(logger, disks)[source]
pyawskit.devices.mount_disks() → None[source]

This script mounts all the local disks as individuals

TODO: - make this run in parallel on multiple cores and enable the user to choose (via command line option) whether to run this multi-core or not.

pyawskit.devices.unify_disks() → None[source]

unify disks of a machine in raid 0

This script is derived from the following bash script: https://gist.github.com/joemiller/6049831

NOTES: - python has no support for “mount” and “umount” system calls. the result is that we use the command line tools to do mounting and un mounting. References: - http://stackoverflow.com/questions/1667257/how-do-i-mount-a-filesystem-using-python

References: - http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/raid-config.html - http://www.tecmint.com/create-raid0-in-linux/ - http://dev.bizo.com/2012/07/mdadm-device-or-resource-busy.html

TODO: - make this script work in parallel. But for the dd(1) and mkfs(1) parts.

pyawskit.inet module

inet.py

pyawskit.inet.check()[source]
pyawskit.inet.my_ip()[source]

pyawskit.main module

main.py

pyawskit.main.aws_codeartifact_npm_env_config() → None[source]
pyawskit.main.aws_codeartifact_pip_env_config() → None[source]
pyawskit.main.aws_ecr_login() → None[source]
pyawskit.main.compress_s3_folder() → None[source]

This script accepts three parameters: bucket, in-folder, out-folder It reads every file in in-folder, gzip it, and then writes it with the suffix “.gz” to the out folder.

The credentials for this are NOT stored in this script but rather are in ~/.aws/credentials.

pyawskit.main.copy_to_machine() → None[source]

TODO: - do not copy ~/.aws/shell (it is big)

pyawskit.main.generate_etc_hosts() → None[source]

This script will update ~/.aws/config file with the names of your machines. Notice that you must hold all of your .pem files in ~/.aws/keys

pyawskit.main.generate_ssh_config() → None[source]

This script will update ~/.aws/config file with the names of your machines. Notice that you must hold all of your .pem files in ~/.aws/keys

pyawskit.main.generate_tilde_hosts() → None[source]

This script will update ~/.hosts file with the names of your machines. You must use something like libnss_homehosts to use this file for each app.

pyawskit.main.inet_check() → None[source]
pyawskit.main.inet_my_ip() → None[source]
pyawskit.main.launch_machine() → None[source]

This script launches a new machine via boto3 with configuration from ~/.pyawskit/launch_config.json

If you want to know how to edit this file look at: http://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.Client.request_spot_instances

References: - http://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.Client.run_instances - http://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.Client.request_spot_instances - https://stackoverflow.com/questions/35721557/boto3-spot-instance-creation - https://github.com/abdul/ec2-spot-instance-launcher - https://peteris.rocks/blog/script-to-launch-amazon-ec2-spot-instances/

This is the real script where most of this one is from: - https://gist.github.com/aloysius-lim/b77f6d62a595ae329e41

pyawskit.main.main()[source]
pyawskit.main.prep_account()[source]

These are the types of things it does: - copy ~/.aws ~/.ssh ~/.s3cfg ~/.gitconfig ~/.passwd-s3fs to it - set quiet login using:

$ touch ~/.hushlogin

  • configure vim to do correct python editing and save editing positions.

  • install fancy prompt on it.

  • put the git repositories you want on it.

pyawskit.main.prep_machine(name: str) → None[source]

A python version of the bash script here: https://gist.github.com/joemiller/6049831

  • sudo purge-old-kernels –keep 1 # to remove old kernels

  • install a list of packages on the system:

  • format and mount all disks

    use mdadm

  • set the hostname in /etc/hosts beside

    127.0.0.1 localhost [hostname] to avoid problems resolving the host name. $ touch ~/.hushlogin

  • configure vim to do correct python editing and save editing positions.

  • install fancy prompt on it.

  • put the git repositories you want on it.

Parameters:

name

Returns:

pyawskit.main.role_delete() → None[source]
pyawskit.main.role_duplicate() → None[source]
pyawskit.main.role_get() → None[source]
pyawskit.main.show_disks() → None[source]

pyawskit.os_utils module

os_utils.py

class pyawskit.os_utils.OSType(*values)[source]

Bases: Enum

aml = 2
ubuntu = 1
pyawskit.os_utils.detect_os() → None[source]

first find out if we are on ubuntu or amazon linux :return:

pyawskit.os_utils.install_packages() → None[source]

install necessary package per platform :return:

pyawskit.os_utils.is_os_type(t: OSType) → bool[source]
pyawskit.os_utils.set_timezone() → None[source]
pyawskit.os_utils.update_machine() → None[source]

pyawskit.roles module

roles.py

pyawskit.roles.role_delete()[source]
pyawskit.roles.role_duplicate()[source]
pyawskit.roles.role_get()[source]

pyawskit.static module

version which can be consumed from within the module

pyawskit.utils module

utils.py

pyawskit.utils.catch_all(the_function)[source]
pyawskit.utils.compress_one_file(_list: list[Any]) → Any[source]
pyawskit.utils.copyfileobj(source, destination, buffer_size=1048576)[source]

Copy a file from source to destination. source and destination must be file-like objects, i.e. any object with a read or write method, like for example StringIO.

pyawskit.utils.gzip_file(file_in: str, file_out: str) → None[source]
pyawskit.utils.gzip_file_process(file_in: str, file_out: str) → None[source]
pyawskit.utils.object_exists(s3_connection, check_bucket_name: str, object_name: str) → bool[source]

It seems funny we have to write this method but it seems that this is the way to check if an s3 bucket has an object.

http://stackoverflow.com/questions/33842944/check-if-a-key-exists-in-a-bucket-in-s3-using-boto3

Parameters:
  • s3_connection

  • check_bucket_name

  • object_name

Returns:

whether the object exists

pyawskit.utils.object_exists_bucket(bucket_obj, object_name: str) → bool[source]

see the note above

Parameters:
  • bucket_obj

  • object_name

Returns:

whether the object exists

pyawskit.utils.print_exception(e)[source]
pyawskit.utils.process_one_file(basename, full_name, compressed_basename, full_compressed_name, bucket_name)[source]

Module contents

Initialize the module