genvid.toolbox.SDK

class genvid.toolbox.SDK(cluster_id='', **kwargs)

Bases: AWSTool, ConsulTemplateTool

Common commands for setting up the SDK.

This class wraps the ClusterAPI and BastionAPI for setting up a new Genvid SDK into the current cluster.

Changed in version 1.29.0: Moved load_config_template() to base class ConsulTemplateTool.

Changed in version 1.29.0: Moved merge_dict() to base class BaseTool.

Changed in version 1.30.0: Derived from ProjectTool instead of its ConsulTemplateTool ancestor.

Changed in version 1.30.0: Removed VaultTool and ConsulTool from the ancestry since they were already part of ConsulTemplateTool.

Changed in version 1.30.0: Moved load_nomad_job_template() to base class ProjectTool.

Changed in version 1.31.0: Moved update_config() to base class ProjectTool.

Changed in version 1.33.0: Loaded the binaries paths for a local cluster from GENVID_SDK_FOLDER/local-services/init folder, instead of hard coded. Also move the initialization from the static folders from setup_jobs() to setup().

Changed in version 1.33.0: Replace ProjectTool with ConfigurationLoader, now inheriting from ConsulTemplateTool instead.

NAME = 'sdk'

The name of the logger for this tool. Must be redefined in children.

SDK_CONFIG_VERSION = '1.7.0'
cluster_id

The Cluster ID.

bastionAPI

Interface to the Bastion-API instance.

clusterAPI

Interface to the cluster Cluster-API instance.

cluster_vault_secret

Key in the bastion’s Vault where the cluster vault keys are saved.

GENVID_SDK_SETTINGS_SCHEMA

A mirror of GENVID_SDK_SETTINGS_SCHEMA containing the path to the default settings schema.

New in version 1.41.0.

property azure: AzureTool
property cloud_provider
get_remote_cluster_conn()

Return the cluster IP and private key.

get_remote_cluster_secrets()

Returns the cluster vault initialization secrets.

setup(bucket: Optional[str] = None, path: str = '/', *, region=None)

Starts the base SDK services.

Parameters

  • bucket – The name of the bucket. Defaults to the return of get_s3_bucket_id().

  • path – The prefix key in the bucket for the images.

clean()

Stops the base SDK services, removes jobs and clean the config.

get_sdk_config() → dict

Get the default SDK configuration.

The configuration is found either in GENVID_SDK_FOLDER/local-services/config if the cluster_id is local, or GENVID_SDK_FOLDER/cloud-services/config otherwise. The config is loaded and parsed by consul-template first, and the jobs templates are expected to be found in a sibling folder named templates.

Changed in version 1.31.0: Load config from a folder.

get_config() → dict

Get the current cluster configuration.

set_config(config: Optional[dict] = None)

Merge the specified configuration with the current one. :param config: The configuration to merge.

clean_config()

Clean the configuration.

remove_config(config: dict)

Remove values from the configuration. :param config: The configuration to remove values from.

static update_dict_no_create(source, dest)
is_jobs_running()

Return True if a job is configured to be running.

start_jobs(jobs: Optional[List[str]] = None)

Start all jobs or the specified job. :param jobs: The list of jobs to start.

stop_jobs(jobs: Optional[List[str]] = None)

Stop all jobs or the specified job.

Parameters

jobs – The list of jobs to stop.

property default_images_sdk_path

Get default S3 key prefix for SDK images.

New in version 1.20.0.

upload_images_sdk(bucket=None, path=None, update_config: bool = False, cloud_provider: Optional[str] = None, *, region=None)

Upload the images for the SDK.

Parameters

  • bucket (str) – Bucket ID (default: get_s3_bucket_id()).

  • path – Key prefix to use in the bucket.

  • update_config (bool) – Update cluster images configuration.

  • region (str) – Region containing the S3 bucket.

Warning

update_config can only be set for non-local clusters.

Changed in version 1.14.0: Default path changes from ‘/’ to ‘/images/sdk-{version}/’.

Changed in version 1.20.0: New region parameter following the conventions in AWSTool.

Changed in version 1.26.0:

  • Added parameter region to target another region for the S3 bucket where the image will be uploaded.

  • If the bucket, named after the Bastion Id, does not exist, it will be created in the targeted region.

  • If it already exists but in a different region, the underlying code will raise an exception.

  • By default, the region used to create the bucket will be the region of the current AWS profile.

  • It can also be overriden by the AWS_DEFAULT_REGION environment variable.

  • If the region parameter is None and the bucket region already exists but its region differs from the one of the profile or AWS_DEFAULT_REGION environment variable, it will simply be used.

get_s3_bucket_id()

Get the S3 bucket ID for images.

Note

This method makes a global STS request to AWS. We disallow setting the region for this call to avoid confusion with the region of the bucket itself.

get_s3_bucket_with_bastion_id(bastion_id: str)

Get the S3 bucket ID for images using the current AWS account and the Bastion ID.

Note

This method makes a global STS request to AWS. We disallow setting the region for this call to avoid confusion with the region of the bucket itself.

update_images_config(bucket: Optional[str] = None, path: str = '/images/custom', *, delete: bool = False, region: Optional[str] = None)

Update the images configuration on the cluster.

Parameters

  • bucket – Bucket ID (default: get_s3_bucket_id()).

  • path – Key prefix to use in the bucket.

  • delete – Do not update configuration, delete it instead.

  • region – Region containing the S3 bucket.

Warning

This method can only be called on a non-local cluster.

New in version 1.20.0.

upload_images(prefixes: List[str], bucket: Optional[str] = None, path: str = '/images/custom', update_config: bool = False, basedir: Optional[Union[Path, str]] = None, cloud_provider: Optional[str] = None, *, region: Optional[str] = None)

Upload the images with the specified prefixes.

Note

The image names must match DockerTool.RE_IMAGE_IMAGEID.

Parameters

  • prefixes – Only upload the files with these prefixes.

  • bucket (str) – Bucket ID (default: get_s3_bucket_id()).

  • path – Key prefix to use in the bucket.

  • update_config (bool) – Update cluster images configuration.

  • basedir – Source directory (default: GENVID_SDK_FOLDER/images).

  • region (str) – Region containing the S3 bucket.

Warning

update_config can only be set for non-local clusters.

Changed in version 1.14.0: Change the default path from ‘/’ to ‘/images/custom’.

Changed in version 1.20.0:

  • Added the basedir parameter.

  • Augmented error handling.

  • New region parameter following the conventions in AWSTool.

  • Factored the update_config part of the method into its own method.

Changed in version 1.26.0:

  • Added parameter region to target another region for the S3 bucket where the image will be uploaded.

  • If the bucket, named after the Bastion Id, does not exist, it will be created in the targeted region.

  • If it already exists but in a different region, the underlying code will raise an exception.

  • By default, the region used to create the bucket will be the region of the current AWS profile.

  • It can also be overriden by the AWS_DEFAULT_REGION environment variable.

  • If the region parameter is None and the bucket region already exists but its region differs from the one of the profile or AWS_DEFAULT_REGION environment variable, it will simply be used.

create_nats_window_package()

Creates nats-window package if it does not exist.

Changed in version 1.32.0: Make it run on Linux using the binaries file.

get_images_config(bucket: Optional[str] = None, path: str = '/images/custom', *, region: Optional[str] = None) → dict
Retrieve images configurations from an S3 bucket or azure container

based on cloud provider.

Parameters

  • bucket – Bucket ID .

  • path – Key prefix in the bucket (default: ‘/images/custom’).

  • region (str) – Region containing the S3 bucket/Azure Container.

Important

The image names must match DockerTool.RE_IMAGE_IMAGEID to be selected.

Returns

Images configurations as a dict.

Changed in version 1.14.0: Changed the default path from ‘/’ to ‘/images/custom’.

Changed in version 1.20.0: New region parameter following the conventions in AWSTool.

clean_images(bucket: Optional[str] = None, path: str = '/images/custom', update_config: bool = False, *, region: Optional[str] = None)

Remove all images from an S3 bucket.

Parameters

  • bucket – Bucket ID (default: get_s3_bucket_id()).

  • path – Key prefix to use in the bucket.

  • update_config (bool) – Update cluster images configuration.

  • region (str) – Region containing the S3 bucket.

Warning

update_config can only be set for non-local clusters.

Changed in version 1.14.0: Change the default path from ‘/’ to ‘/images/custom’.

Changed in version 1.20.0:

update_archive(archive: Union[Path, str], version: str = 'dev', destdir: Optional[Union[Path, str]] = None, clean: bool = True, keep: bool = False)

Update a binary archive to conform to the image format expected.

param archive

The path to the archive file. The filename should always be in the form of name.archfmt.

param version

Version of the archive.

param destdir

Destination directory. Default to the same as the original archive.

param clean

Clean old versions of the archive (the ones with the same name).

param keep

Keep the original archives. You should almost always used it with a different destdir.

New in version 1.20.0.

Deprecated since version 1.30.0: Method moved to PackageArchiver

load_config(source: Union[Path, str], *, job_template_dir: Optional[Union[Path, str]], with_consul_template: bool, dryrun: bool = False) → dict

Loads the configuration file(s) (all files in a directory or a specific file) from the source.

Parameters

  • source – Path to the configuration file/folder.

  • with_consul_template – Execute consul-template on the configuration file(s) (if source is a directory) before using it only if it is set to true.

  • job_template_dir – Path to Folder from where nomad templates should be loaded. If not mentioned, no jobs will be loaded.

  • dryrun – If True, only parse the source and returns the configuration.

Returns

The configuration as a dictionary that can be used directly with set_config().

:raises FileNotFoundErrorIf the job_template_dir is not None and

is not a valid directory.

Changed in version 1.26.0: Fix support for pathlib.Path for arguments. Add load parameter

Changed in version 1.30.0: Refactored to use load_config_folder() instead of creating a new instance of ProjectTool.

Changed in version 1.33.0: Use the new render_template command instead of the local consul-template.

get_output_public_ips() → list

Retrieves all public ip’s from output of terraform module.

Returns

List of all public IP Addressses

get_cloud_provider() → str

Retrieves the cloud provider from output of terraform module.

If no cloud provider is found in terraform’s cluster output, aws is returned as default.

Returns

Cloud provider.

load_config_template(template: Union[Path, str], env: Optional[Mapping[str, str]] = None) → Any

Deprecated since version 1.33.0: ConsulTemplateTool will no longer be part of the SDK ancestors. Use render_template() instead.

consul_template_once(*, use_consul: bool = False, use_vault: bool = False, env: Optional[Mapping[str, str]] = None, **kwargs) → ConsulTemplateOutput

Deprecated since version 1.33.0: ConsulTemplateTool will no longer be part of the SDK ancestors. Use render_template() instead.

property consul_template

Deprecated since version 1.33.0: ConsulTemplateTool will no longer be part of the SDK ancestors. Use render_template() instead.

class sdk.SDK

Implementation of genvid.toolbox.SDK