genvid.toolbox.BaseTool

class genvid.toolbox.BaseTool(logname=None, command_option_name='command', **kwargs)

Bases: object

Base class for all tools.

This class sets up and provides basic functionalities to all other tools. In particular, it handles the main() and get_parser() methods. See add_commands(), add_command() and run_command() for more details.

Please, note that most attributes of this class aren’t initialized properly before instantiation.

NAME

The name of the logger for this tool. Must be redefined in children.

DESCRIPTION

A description of the tool, used in the help parser. Must be redefined in children.

ENVIRONMENT_VARIABLES = {'AZURE_RESOURCE_GROUP', 'AZURE_STORAGE_ACCOUNT', 'BINDIR', 'CONSUL_HTTP_ADDR', 'CURDIR', 'GENVID_BASTION_URL', 'GENVID_CHECKIP_URL', 'GENVID_CLUSTER_ID', 'GENVID_DEFAULT_IP', 'GENVID_SDK_FOLDER', 'GENVID_SDK_VERSION', 'GENVID_TOOLBOX_FOLDER', 'GENVID_TOOLBOX_LOGFORMAT', 'GENVID_TOOLBOX_LOGLEVEL', 'GENVID_TOOLBOX_VERSION', 'GENVID_VAULT_POLICIES_FOLDER', 'GENVID_VAULT_TOKEN_ROLES_FOLDER', 'GENVID_VERBOSE_EXCEPTIONS', 'HOME', 'MSBUILD', 'PYTHON_EXECUTABLE', 'ROOTDIR', 'VAULT_ADDR', 'VAULT_KEYS', 'VAULT_KEYS_FILE', 'VAULT_TOKEN', 'VAULT_TOKEN_FILE'}

The set of environment variables used by all tools.

The list is updated each time setdefault() is called. It is used by env() and print_env().

MODE_0600 = 384

User RW file permissions.

MODE_0644 = 420

User RW, Group and Other Read file permissions.

MODE_0700 = 448

User RWX file permissions.

MODE_0755 = 493

User RWX and Group and Other RX file permissions.

static setcurdir(path: str)

Set CURDIR if not already set.

Deprecated since version 1.39.0: Variable CURDIR is ambiguous in its usage.

static setrootdir(path: str)

Set GENVID_SDK_FOLDER and GENVID_SDK_FOLDER if not already set.

static setbasescript(path: str)

Set CURDIR to path, and set GENVID_SDK_FOLDER to its parent, if not already set.

If you want to set the GENVID_SDK_FOLDER to a different path, you must call setrootdir first.

static setdefault(name: str, value: str)

Set an environment variable named name to value if not already set and returns its value.

static delenv(name: str)
classmethod merge_dict(source: dict, destination: dict) dict

Merge 2 dictionaries.

destination will be modified with the elements from source replacing them. If source contains a dict as a value, the merge will recurse on that value (possibly throwing an error if the destination is not a dict).

Parameters
  • source – The source dictionary.

  • destination – The destination dictionary.

Raises

ValueError – if original key is a dictionary but the destination already contains non-dictionary value.

Returns

destination

New in version 1.29.0.

run(*args, check=True, **kwargs)

Wrapper around subprocess.run() that passes all arguments in args and sets the default value of check to True.

For convenience, all arguments are converted to string (using str).

Changed in version 1.15.0: It’s now a method instead of staticmethod.

shrun(cmdline: str, shell=True, check=True, **kwargs)

Wrapper around subprocess.run() that pass the cmdline directly, set the default value of check and shell to True.

Changed in version 1.15.0: It’s now a method instead of staticmethod.

pyrun(*args, **kwargs)

Run a python script using the current python version.

This is roughly equivalent to:

run("python", *args, **kwargs)

Unlink a file only if it exists.

Changed in version 1.12.0: Now allow a pathlib.Path too as argument.

static rmtree(directory: ~pathlib.Path, ignore_errors: bool = False, onerror=<staticmethod(<function BaseTool._rmtree_onerror>)>)

Equivalent to shutil.rmtree() but symlink-aware on Windows.

Changed in version 1.20.1:

  • The onerror handling method is now given a default handler which will retry deleting a file for 60 seconds, using 5 seconds intervals between attempts, before raising an exception.

static remove_readonly(func, path, excinfo)

Helper for rmtree() to help remove read-only files on Windows.

install_file(src: Union[str, Path], dst: Union[str, Path], mode=420, uid=-1, gid=-1, backup=False)

Copy a file src to dst, and change its permissions.

Parameters
  • src – The file to be copied.

  • dst – The destination of the copy.

  • mode – The permission of the destination.

  • uid – The user ID of the destination. If -1, uses the user ID of the process.

  • gid – The GID of the destination. If -1, uses the uid.

  • backup – Backup the file, adding a ~ at the end of the filename.

Changed in version 1.16.0: Make it a non-staticmethod and add the backup argument.

install_tmp_file(dest: Union[str, Path], data: bytes, **kwargs)

Write data to a temporary file and copy it to dst.

The installation is done using install_file().

Parameters
  • dest – The destination of the content.

  • data – The data to write to write into the file.

Changed in version 1.16.0: Now a non-staticmethod

install_json(dest: Union[str, Path], config, **kwargs)

Dump config as a JSON file and copy it to destination.

Parameters
  • dest – The destination of the content.

  • data – The data to write to write into the file as JSON.

Changed in version 1.16.0: Now a non-staticmethod

static env()

Return the list of used environment variables and their values.

The list of variables to return is taken from ENVIRONMENT_VARIABLES.

static print_env()

Print the list of environment variables used and their values.

The list of variables to return is taken from ENVIRONMENT_VARIABLES.

logger

The logger for the tool.

PLATFORM

Name of the current platform.

Could be either linux or windows for now.

HOME

A mirror of HOME containing the user home directory name.

Taken from environment variable or os.path.expanduser().

PYTHON_EXECUTABLE

A mirror of PYTHON_EXECUTABLE containing the current python executable by default.

Taken from the environment or sys.executable if not set.

GENVID_BASTION_URL

A mirror of GENVID_BASTION_URL representing the bastion-api URL.

CURDIR

A mirror of CURDIR containing the current script directory path if set correctly, or the current working directory by default.

Warning: This variable is not very reliable. Try to avoid using it.

Deprecated since version 1.39.0: Variable is ambiguous in its usage.

GENVID_TOOLBOX_VERSION

A mirror of GENVID_TOOLBOX_VERSION containing the currently installed toolbox version.

Warning: This variable shouldn’t be overriden.

GENVID_VERBOSE_EXCEPTIONS

A mirror of GENVID_VERBOSE_EXCEPTIONS.

If the value is 1 the full exception trace is going to be displayed. Otherwise, it will be condensed to a one-liner.

AZURE_STORAGE_ACCOUNT

A mirror of AZURE_STORAGE_ACCOUNT. Name of the Azure Storage Account to access. Look in Azure Portal to access the correct name.

AZURE_RESOURCE_GROUP

A mirror of AZURE_RESOURCE_GROUP. Name of the Azure Resource Group to access.

GENVID_SDK_FOLDER

A mirror of GENVID_SDK_FOLDER which is the installation folder of the SDK.

Empty if the SDK is not present.

New in version 1.39.0.

GENVID_SDK_VERSION

A mirror of GENVID_SDK_VERSION. Empty if GENVID_SDK_FOLDER does not exist.

New in version 1.39.0.

GENVID_TOOLBOX_FOLDER

A mirror of GENVID_TOOLBOX_FOLDER which is the installation folder of the Genvid Toolbox.

New in version 1.39.0.

ROOTDIR

A mirror of ROOTDIR.

Deprecated since version 1.39.0: Please, use GENVID_SDK_FOLDER instead.

BINDIR

A mirror of BINDIR containing the folder where the binaries of the project are stored.

Deprecated since version 1.40.0: Use GENVID_SDK_FOLDER/bin or GENVID_TOOLBOX_FOLDER/bin instead.

ARTIFACTS_DIR

Where the images are saved for this MILE SDK.

Changed in version 1.20.0: Move the variable from DockerTool.

Deprecated since version 1.39.0: The code is now using GENVID_SDK_FOLDER/images or GENVID_TOOLBOX_FOLDER/images when necessary.

COMMAND_OPTION_NAME

The name used for the command arguments on the command line.

See add_command() for more details.

GENVID_TOOLBOX_LOGLEVEL

A mirror of GENVID_TOOLBOX_LOGLEVEL that set the logger level in toolbox.

GENVID_TOOLBOX_LOGFORMAT

A mirror of GENVID_TOOLBOX_LOGFORMAT that set the logger formatter in toolbox.

GENVID_CLUSTER_ID

A mirror of GENVID_CLUSTER_ID that indicates to the toolbox what cluster id to use.

commands

A dictionary of the parser for each command.

See add_command() for more details.

subparsers

The subparsers object from the argparse module used for all commands.

See add_commands() for more details.

parser

The current parser for this tool.

See add_commands() for more details.

get_sdk_version(folder: Optional[Union[Path, str]] = None)

Return the SDK version of the installed SDK.

Parameters

folder – The folder of the installed SDK. If None, use GENVID_SDK_FOLDER.

Raises
Returns

The version of the SDK installed.

which(exe, path=None)

Return the full path of the executable that would be run with exe.

If path is included, it will be used for the check. Otherwise, the following folders are checked in the following order before looking at the regular path:

Raises

ExecutableNotInPathError if the file doesn’t exist.

run_powershell(*cmds: str, interactive: bool = False, noprofile: bool = False, **kwargs)

Run cmds as a powershell script.

Parameters
  • cmds – A list of powershell commands to execute.

  • interactive – If the command is interactive.

  • noprofile – Don’t load a profile.

  • kwargs – Other keyword arguments to pass to the parser for this run().

run_elevated_win32(*args, **kwargs)

Run the command using elevated privileges on Windows.

run_elevated(*args, **kwargs)

Run the command using elevated privileges on Windows.

KILL_SIGNAL = 1

Default kill signal.

Default to signal.SIGKILL on UNIX, and signal.CTRL_BREAK_EVENT on Windows.

New in version 1.15.0.

send_signal(proc: Union[int, Process], sig: Optional[int] = None)

Send a signal to a process.

Parameters
  • proc – The process. Could be the process PID or an instance of psutil.Process.

  • sig – The signal to send. Default to KILL_SIGNAL.

New in version 1.15.0.

kill_process(proc: Union[int, Process])

Kill a process.

Force kill a process by sending a SIGKILL or call TerminateProcess.

Parameters

proc – The process. Could be the process PID or an instance of psutil.Process.

New in version 1.15.0.

kill_process_tree(proc: Union[int, Process], timeout: float = 15.0) List[Process]

Kill a process tree by sending SIGTERM/CTRL_BREAK_EVENT first, and then by pushing a SIGKILL/Terminate on all children.

Parameters
  • proc – The process. Could be the process PID or an instance of psutil.Process.

  • timeout – Timeout to wait for the process to finish (in seconds).

Returns

List of processes still alive.

New in version 1.15.0.

Changed in version 1.20.1:

  • Changed default timeout value to 15 seconds.

  • Children of children are now recursively killed.

run_daemon(*args: str, **kwargs)

Run the process in background.

Parameters
  • args – The command to run.

  • stdout – The output file for stdout.

  • stderr – The output file for stderr.

  • cwd – The current working directory.

  • env – The environment to use.

  • pidfile – The pidfile to save the PID of the process.

add_commands()

Add commands for the tool.

This method must be overridden. It allows the child class to add new commands using add_command(). The list of all existing commands and their parsers are available from commands.

add_command(name: str, description: str, **kwargs)

Add a new command to the tool.

The command will be added to the list of commands.

Parameters
  • name – The name of the command.

  • description – Help for the command.

  • kwargs – Other keyword arguments to pass to the parser for this tool subparsers.

Returns

A command parser.

Raises

AssertionError if name already exists as a command.

add_base_commands()
run_base_command(command, options)
get_parser(subparsers=None, name=None, description=None, fromfile_prefix_chars='@', **kwargs)

Returns the parser for the tool.

Parameters
  • subparsers – Custom subparsers to use.

  • name – The name of the tool (defaults to NAME).

  • description – The description of the tool (defaults to DESCRIPTION).

  • fromfile_prefix_chars – The set of characters that prefix files from which additional arguments should be read.

  • kwargs – Other keyword arguments to pass to the parser for this argparse.ArgumentParser factory.

Returns

The parser for this tool.

Changed in version 1.13.0: New default for fromfile_prefix_chars to ‘@’. Pass None to switch to old behaviour (which is None).

run_command(command, options)

Run command with those options.

You must override this method.

Parameters
  • command – The name of the command to run.

  • options – A argparse.Namespace object with the options pass in arguments.

main(options=None, *, profile=None)

Run the tool.

If options is None, it will obtained from the command line arguments. Otherwise, it must be either a namespace object or a dictionary.

Parameters

profile – Profile instance if profiles are used.

get_files(srcdir: Path)

Get all files (but not folders) under srcdir.

Equivalent to srcdir.rglob(“*”) but with folder filtered out.

Parameters

srcdir – The folder to check for new files.

Returns

an iterator of Path objects.

copy_files(srcdir: Path, dstdir: Path, files: List[Union[str, Path]], dryrun=False)

Copy all files in files``from ``srcdir to dstdir.

Parameters
  • srcdir – the base folder of the original files.

  • dstdir – the destination folder of the files.

  • files – a list of files to copy.

  • dryrun – If true, will only log the result, without doing the copy.

safe_print(text: str, fileobj=<colorama.ansitowin32.StreamWrapper object>, errors='backslashreplace')

Print a value safely to fileobj.

fileobj must have an encoding member as well as a buffer member with a write method that accept bytes.

Parameters
  • text – The text to print.

  • fileobj – The fileobj to print to.

  • errors – The error handling strategy. See str.encode() for the possible values.

class basetool.BaseTool

Implementation of genvid.toolbox.BaseTool