genvid.toolbox.SDK¶

class genvid.toolbox.SDK(cluster_id='', **kwargs)¶

Bases: genvid.toolbox.aws.AWSTool, genvid.toolbox.consul_template.ConsulTemplateTool

Common commands for setting up the SDK.

This class wraps the ClusterAPI and BastionAPI for setting up a new Genvid SDK into the current cluster.

Changed in version 1.29.0: Moved load_config_template() to base class ConsulTemplateTool.

Changed in version 1.29.0: Moved merge_dict() to base class BaseTool.

Changed in version 1.30.0: Derived from ProjectTool instead of its ConsulTemplateTool ancestor.

Changed in version 1.30.0: Removed VaultTool and ConsulTool from the ancestry since they were already part of ConsulTemplateTool.

Changed in version 1.30.0: Moved load_nomad_job_template() to base class ProjectTool.

Changed in version 1.31.0: Moved update_config() to base class ProjectTool.

Changed in version 1.33.0: Loaded the binaries paths for a local cluster from ROOTDIR/local-services/init folder, instead of hard coded. Also move the initialization from the static folders from setup_jobs() to setup().

Changed in version 1.33.0: Replace ProjectTool with ConfigurationLoader, now inheriting from ConsulTemplateTool instead.

NAME = 'sdk'¶

SDK_CONFIG_VERSION = '1.7.0'¶

cluster_id = None¶: The Cluster ID.

bastionAPI = None¶: Interface to the Bastion-API instance.

clusterAPI = None¶: Interface to the cluster Cluster-API instance.

cluster_vault_secret = None¶: Key in the bastion’s Vault where the cluster vault keys are saved.

azure¶

cloud_provider¶

get_remote_cluster_conn()¶: Return the cluster IP and private key.

get_remote_cluster_secrets()¶: Returns the cluster vault initialization secrets.

setup(bucket: str = None, path: str = '/', *, region=None)¶

Starts the base SDK services.

Parameters:	bucket – The name of the bucket. Defaults to the return of `get_s3_bucket_id()`. path – The prefix key in the bucket for the images.

setup_remote(cluster_id: str, server_ip: str, ssh_private_key_pem_file: str, ssh_user: str)¶

clean()¶: Stops the base SDK services, removes jobs and clean the config.

get_sdk_config() → dict¶

Get the default SDK configuration.

The configuration is found either in ROOTDIR/local-services/config if the cluster_id is local, or ROOTDIR/cloud-services/config otherwise. The config is loaded and parsed by consul-template first, and the jobs templates are expected to be found in a sibling folder named templates.

Changed in version 1.31.0: Load config from a folder.

get_config() → dict¶: Get the current cluster configuration.

set_config(config: dict = None)¶: Merge the specified configuration with the current one. :param config: The configuration to merge.

clean_config()¶: Clean the configuration.

remove_config(config: dict)¶: Remove values from the configuration. :param config: The configuration to remove values from.

static update_dict_no_create(source, dest)¶

is_jobs_running()¶: Return True if a job is configured to be running.

start_jobs(jobs: typing.Union[typing.List[str], NoneType] = None)¶: Start all jobs or the specified job. :param jobs: The list of jobs to start.

stop_jobs(jobs: typing.Union[typing.List[str], NoneType] = None)¶

Stop all jobs or the specified job.

Parameters:	jobs – The list of jobs to stop.

default_images_sdk_path¶: Get default S3 key prefix for SDK images.

New in version 1.20.0.

upload_images_sdk(bucket=None, path=None, update_config: bool = False, cloud_provider: typing.Union[str, NoneType] = None, *, region=None)¶

Upload the images for the SDK.

Parameters:	bucket (str) – Bucket ID (default: `get_s3_bucket_id()`). path – Key prefix to use in the bucket. update_config (bool) – Update cluster images configuration. region (str) – Region containing the S3 bucket.

Warning

update_config can only be set for non-local clusters.

Changed in version 1.14.0: Default path changes from ‘/’ to ‘/images/sdk-{version}/’.

Changed in version 1.20.0: New region parameter following the conventions in AWSTool.

Changed in version 1.26.0:

Added parameter region to target another region for the S3 bucket where the image will be uploaded.
If the bucket, named after the Bastion Id, does not exist, it will be created in the targeted region.
If it already exists but in a different region, the underlying code will raise an exception.
By default, the region used to create the bucket will be the region of the current AWS profile.
It can also be overriden by the AWS_DEFAULT_REGION environment variable.
If the region parameter is None and the bucket region already exists but its region differs from the one of the profile or AWS_DEFAULT_REGION environment variable, it will simply be used.

get_s3_bucket_id()¶: Get the S3 bucket ID for images.

Note

This method makes a global STS request to AWS. We disallow setting the region for this call to avoid confusion with the region of the bucket itself.

get_s3_bucket_with_bastion_id(bastion_id: str)¶: Get the S3 bucket ID for images using the current AWS account and the Bastion ID.

Note

This method makes a global STS request to AWS. We disallow setting the region for this call to avoid confusion with the region of the bucket itself.

update_images_config(bucket: typing.Union[str, NoneType] = None, path: str = '/images/custom', *, delete: bool = False, region: typing.Union[str, NoneType] = None)¶

Update the images configuration on the cluster.

Parameters:	bucket – Bucket ID (default: `get_s3_bucket_id()`). path – Key prefix to use in the bucket. delete – Do not update configuration, delete it instead. region – Region containing the S3 bucket.

Warning

This method can only be called on a non-local cluster.

New in version 1.20.0.

upload_images(prefixes: typing.List[str], bucket: typing.Union[str, NoneType] = None, path: str = '/images/custom', update_config: bool = False, basedir: typing.Union[pathlib.Path, str, NoneType] = None, cloud_provider: typing.Union[str, NoneType] = None, *, region: typing.Union[str, NoneType] = None)¶

Upload the images with the specified prefixes.

Note

The image names must match DockerTool.RE_IMAGE_IMAGEID.

Parameters:	prefixes – Only upload the files with these prefixes. bucket (str) – Bucket ID (default: `get_s3_bucket_id()`). path – Key prefix to use in the bucket. update_config (bool) – Update cluster images configuration. basedir – Source directory (default: `ARTIFACTS_DIR`). region (str) – Region containing the S3 bucket.

Warning

update_config can only be set for non-local clusters.

Changed in version 1.14.0: Change the default path from ‘/’ to ‘/images/custom’.

Changed in version 1.20.0:

Added the basedir parameter.
Augmented error handling.
New region parameter following the conventions in AWSTool.
Factored the update_config part of the method into its own method.

Changed in version 1.26.0:

Added parameter region to target another region for the S3 bucket where the image will be uploaded.
If the bucket, named after the Bastion Id, does not exist, it will be created in the targeted region.
If it already exists but in a different region, the underlying code will raise an exception.
By default, the region used to create the bucket will be the region of the current AWS profile.
It can also be overriden by the AWS_DEFAULT_REGION environment variable.
If the region parameter is None and the bucket region already exists but its region differs from the one of the profile or AWS_DEFAULT_REGION environment variable, it will simply be used.

create_nats_window_package()¶: Creates nats-window package if it does not exist.

Changed in version 1.32.0: Make it run on Linux using the binaries file.

get_images_config(bucket: typing.Union[str, NoneType] = None, path: str = '/images/custom', *, region: typing.Union[str, NoneType] = None) → dict¶

Retrieve images configurations from an S3 bucket or azure container: based on cloud provider.

Parameters:	bucket – Bucket ID . path – Key prefix in the bucket (default: ‘/images/custom’). region (str) – Region containing the S3 bucket/Azure Container.

Important

The image names must match DockerTool.RE_IMAGE_IMAGEID to be selected.

Returns:	Images configurations as a `dict`.

Changed in version 1.14.0: Changed the default path from ‘/’ to ‘/images/custom’.

Changed in version 1.20.0: New region parameter following the conventions in AWSTool.

clean_images(bucket: typing.Union[str, NoneType] = None, path: str = '/images/custom', update_config: bool = False, *, region: typing.Union[str, NoneType] = None)¶

Remove all images from an S3 bucket.

Parameters:	bucket – Bucket ID (default: `get_s3_bucket_id()`). path – Key prefix to use in the bucket. update_config (bool) – Update cluster images configuration. region (str) – Region containing the S3 bucket.

Warning

update_config can only be set for non-local clusters.

Changed in version 1.14.0: Change the default path from ‘/’ to ‘/images/custom’.

Changed in version 1.20.0:

New region parameter following the conventions in AWSTool.
Now delegates to update_images_config() for update_config.

update_archive(archive: typing.Union[pathlib.Path, str], version: str = 'dev', destdir: typing.Union[pathlib.Path, str, NoneType] = None, clean: bool = True, keep: bool = False)¶

Update a binary archive to conform to the image format expected.

param archive: The path to the archive file. The filename should always be in the form of name.archfmt.

param version: Version of the archive.

param destdir: Destination directory. Default to the same as the original archive.

param clean: Clean old versions of the archive (the ones with the same name).

param keep: Keep the original archives. You should almost always used it with a different destdir.

New in version 1.20.0.

Deprecated since version 1.30.0: Method moved to PackageArchiver

load_config(source: typing.Union[pathlib.Path, str], *, job_template_dir: typing.Union[pathlib.Path, str, NoneType], with_consul_template: bool, dryrun: bool = False) → dict¶

Loads the configuration file(s) (all files in a directory or a specific file) from the source.

Parameters:

source – Path to the configuration file/folder.
with_consul_template – Execute consul-template on the configuration file(s) (if source is a directory) before using it only if it is set to true.
job_template_dir – Path to Folder from where nomad templates should be loaded. If not mentioned, no jobs will be loaded.
dryrun – If True, only parse the source and returns the configuration.

Returns:

The configuration as a dictionary that can be used directly with set_config().

:raises FileNotFoundError : If the job_template_dir is not None and: is not a valid directory.

Changed in version 1.26.0: Fix support for pathlib.Path for arguments. Add load parameter

Changed in version 1.30.0: Refactored to use load_config_folder() instead of creating a new instance of ProjectTool.

Changed in version 1.33.0: Use the new render_template command instead of the local consul-template.

get_output_public_ips() → list¶

Retrieves all public ip’s from output of terraform module.

Returns:	List of all public IP Addressses

get_cloud_provider() → str¶

Retrieves the cloud provider from output of terraform module.

If no cloud provider is found in terraform’s cluster output, aws is returned as default.

Returns:	Cloud provider.

load_config_template(template: typing.Union[pathlib.Path, str], env: typing.Union[typing.Mapping[str, str], NoneType] = None) → typing.Any¶: Deprecated since version 1.33.0: ConsulTemplateTool will no longer be part of the SDK ancestors. Use render_template() instead.

consul_template_once(*, use_consul: bool = False, use_vault: bool = False, env: typing.Union[typing.Mapping[str, str], NoneType] = None, **kwargs) → genvid.toolbox.consul_template.ConsulTemplateOutput¶: Deprecated since version 1.33.0: ConsulTemplateTool will no longer be part of the SDK ancestors. Use render_template() instead.

consul_template¶: Deprecated since version 1.33.0: ConsulTemplateTool will no longer be part of the SDK ancestors. Use render_template() instead.

class sdk.SDK¶: Implementation of genvid.toolbox.SDK