Setting Up the Genvid Cluster in Azure

Set Up a Bastion Server

To create a cluster, you first need a Genvid bastion server running. You can set up and manage a bastion server using the genvid-bastion script. The following command will set up one with the minimal services required.

From your shell, run:

genvid-bastion install --bastionid mybastion --checkmodules --update-global-tfvars --loadconfig
--bastionid mybastion
A unique identifier for your bastion. It must:
  • Be between 3 and 32 characters.
  • Only contain lowercase letters, numbers, or hyphens.
  • Start with a letter.
--checkmodules
Use this option to install new modules if none exist or update the ones already present.
--update-global-tfvars
Use this option to update the global Terraform variables.
--loadconfig
Use this option to load the jobs and logs.

Configure the Cluster

Important

Before configuring the cluster, make sure that azure account has been set up and you are logged in to azure. Also, the azure base images need to be created and configured. The information on how to login to Azure and configure images can be found here Creating a Cloud Environment.

The next step is to open the Bastion-UI to manage the Clusters.

  1. From your shell, run:

    genvid-bastion monitor
    
  2. Select the Terraform page.

  3. Click Add Config.

  4. Enter a unique ID.

  5. Select cluster as the category.

Create a cluster

You can select another backend if needed. For now, you can stick with the default values. Some backends may require configuring variables.

After creating a cluster, its status is EMPTY. This means that the cluster needs a module. The cluster statuses are:

Cloning a Cluster

It is a good practice to use the cloning function when you want to create different clusters with similar configurations. You can do this using the Clone button on the Terraform configuration view.

Clone a cluster

Cluster Statuses

After creating a cluster, its status is EMPTY. This means that the cluster needs a module. The cluster statuses are:

  • VOID: The cluster doesn’t exist.
  • EMPTY: The cluster exists but has no modules.
  • DOWN: The cluster is initialized but resources are empty.
  • UP: The Terraform apply procedure succeeded.
  • BUSY: A command is currently running.
  • ERROR: An error occured when checking the cluster.
  • INVALID: The cluster is reporting an invalid or unknown status.

Initialize Cluster Using a Terraform Module

Before configuring the cluster, you must initialize it with a module. This copies the module template and downloads any providers and plugins it requires.

  1. Click on the name of your chosen ID.

  2. You can either select the SDK-{version}/basic/azurerm_basic_cluster or SDK-{version}/basic/azurerm_basic_cluster_alb_ssl modules. (The azurerm_basic_cluster_alb_ssl module supports load balancing, DNS, and SSL certificates)

  3. Click Import module.

Terraform module

You will see an initialization log appear. This step should last only a few seconds.

Build the Cluster Infrastructure

To build the cluster infrastructure, you apply a Terraform execution plan.

  1. Click Plan Apply to create an execution plan.

  2. Verify the changes match what you need.

  3. Click Apply to execute the plan.

Terraform Plan Apply
Terraform Apply

You should see the log starting to appear.

If Terraform fails to build the infrastructure:
  • Check the error message.
  • Update the settings.
  • Apply again.

At the end of this step, there is a cluster running on the cloud. When you check All Configs on the Terraform page, the status is UP. This means that the infrastructure is up and ready for the Genvid SDK.

Note

When you have an existing cluster that you no longer need, there are specific commands used to clean them up. See Cleaning Up Unneeded Clusters for more information.

Terraform Settings

Basic Terraform settings

Fig. 29 Bastion-UI settings for basic Azure cluster

Terraform settings with SSL, DNS, and application gateway

Fig. 30 Bastion-UI settings for Azure cluster with SSL, DNS, and application gateway (if azurerm_basic_cluster_alb_ssl is selected)

We use Terraform to build the cluster infrastructure, so the setting values configure the cluster infrastructure. Click the Settings sub link for your cluster and edit the settings.

Note

For ease of use you can also download the settings on the page to a JSON file. You can drag and drop the edited file on the form to edit multiple settings at once or revert to a previous configuration. Please note that all unsaved configurations will be lost.

Common Azure cluster settings

Setting Description Default Value
azs Specifies the list of availability zones to use. Azure makes zones 1, 2, and 3 available for use in all regions.Instances will be evenly spanned across specified zone, sequentially, starting at the first specific zone, and rolling over back to the first one when the end of the list is reached. [1] []
datacenter The datacenter name Nomad uses to group local agents together. default
instance_encoding_count Specifies the number of encoding servers that Terraform should create. 1
instance_encoding_type Specifies the type of Azure instance that Terraform should use for encoding servers. Standard_F8s_v2
instance_game_count Specifies the number of game servers that Terraform should create. 1
instance_game_type Specifies the type of Azure instance that Terraform should use for game servers. Standard_NV6
instance_internal_count Specifies the number of internal servers that Terraform should create. 1
instance_internal_type Specifies the type of Azure instance that Terraform should use for internal servers. Standard_B2s
instance_public_count Specifies the number of public servers that Terraform should create. 1
instance_public_type Specifies the type of Azure instance that Terraform should use for public servers. Standard_B2s
instance_server_count Specifies the number of admin servers that Terraform should create. 1
instance_server_type Specifies the type of Azure instance that Terraform should use for admin servers. Standard_B2s
location The Azure region where you want to create your cluster. You can use any region that contains NV6 instances. East US
namespace The name of the current deployment project on your cluster. This value is part of the Azure Tags configuration, making it easier to group resources together in the Azure Portal. deployment
shared_gallery_image_version Which image version to use from the shared image gallery in azure. This should match the current SDK version. current SDK version
shared_gallery_name The shared image gallery in Azure where cluster VM images are stored. SDKgallery
shared_gallery_resource_group_name Name of the resource group that holds the shared image gallery. Genvid
stage The name of the stage your cluster should belong to. This value is used to tag your resources in Azure for easier searching and organization. dev
toolbox_location Genvid Toolbox Wheel file. Only required if you need to update the one from the image. An empty value indicates to not install the toolbox. null
trusted_cidr CIDR of the local machine Local CIDR
trusted_cidrs Additional CIDRs to allow access to the cluster []
trusted_security_groups List of security groups to trust []
wingame_image_prefix Prefix used for the wingame image default

Azure Cluster with SSL and DNS Settings

You can create an Azure cluster with both SSL and DNS features. To use this type of cluster, you need to have a registered domain in Azure DNS zones. It will automatically add two “A” record sets in your domain named web-{clustername} and leafs-{clustername}. For example, if your domain is example.com and your cluster name is stream, your final endpoints for leaf and web will be leafs-stream.example.com and web-stream.example.com.

You need to provide the required certificates to use the cluster. You can use either a wildcard certificate or two separate certificates for leaf and web services.

For some streaming services (like Twitch), your certificates must be valid but not self-signed. Certificates also need to have a password and be in .pfx format.

The following table shows the parameters in the cluster settings.

Setting Description Default Value
azs Specifies the list of availability zones to use. Azure makes zones 1, 2, and 3 available for use in all regions.Instances will be evenly spanned across specified zone, sequentially, starting at the first specific zone, and rolling over back to the first one when the end of the list is reached. [1] []
datacenter The datacenter name Nomad uses to group local agents together. default
domain_name A pre-created domain in Azure or any other registrar [1] example.com
gateway_autoscaling_limits The minimum and maximum number of instances for Application Gateway auto scaler [1] [1,3]
instance_encoding_count Specifies the number of encoding servers that Terraform should create. 1
instance_encoding_type Specifies the type of Azure instance that Terraform should use for encoding servers. Standard_F8s_v2
instance_game_count Specifies the number of game servers that Terraform should create. 1
instance_game_type Specifies the type of Azure instance that Terraform should use for game servers. Standard_NV6
instance_internal_count Specifies the number of internal servers that Terraform should create. 1
instance_internal_type Specifies the type of Azure instance that Terraform should use for internal servers. Standard_B2s
instance_public_count Specifies the number of public servers that Terraform should create. 1
instance_public_type Specifies the type of Azure instance that Terraform should use for public servers. Standard_B2s
instance_server_count Specifies the number of admin servers that Terraform should create. 1
instance_server_type Specifies the type of Azure instance that Terraform should use for admin servers. Standard_B2s
leaf_port The port that the Leaf service is listening on. 30001
leaf_ssl_certificate_password The password of the provided leaf certificate If you provided a wildcard SSL certificate, you do not need to provide this. [1] null
leaf_ssl_certificate_path The path of the provided valid leaf certificate If you provided a wildcard SSL certificate, you do not need to provide this. [1] null
leaf_stickiness_ttl The Leaf target group stickiness timeout [1] 20
leaf_tg_stickiness_enabled to enable/disable the Leaf target group stickiness [1] false
location The Azure region where you want to create your cluster. You can use any region that contains NV6 instances. East US
namespace The name of the current deployment project on your cluster. This value is part of the Azure Tags configuration, making it easier to group resources together in the Azure Portal. deployment
shared_gallery_image_version Which image version to use from the shared image gallery in azure. This should match the current SDK version. current SDK version
shared_gallery_name The shared image gallery in Azure where cluster VM images are stored. SDKgallery
shared_gallery_resource_group_name Name of the resource group that holds the shared image gallery. Genvid
stage The name of the stage your cluster should belong to. This value is used to tag your resources in Azure for easier searching and organization. dev
toolbox_location Genvid Toolbox Wheel file. Only required if you need to update the one from the image. An empty value indicates to not install the toolbox. null
truster_cidr CIDR of the local machine Local CIDR
trusted_cidrs Additional CIDRs to allow access to the cluster []
trusted_security_groups List of security groups to trust []
ttl_leaf_hostname TTL information of the leaf endpoint in the A record of the main domain. [1] 3600
ttl_web_hostname TTL information of the web endpoint in the A record of the main domain. [1] 3600
web_health_check_path The health check endpoint path for the web service. It should match the path defined in the web.nomad.tmpl. [1] /health
web_port The port that web service is configured to listen on [1] 30000
web_ssl_certificate_password The password of the provided web certificate If you provided a wildcard SSL certificate, you do not need to provide this. [1] null
web_ssl_certificate_path The path of the provided valid web certificate If you provided a wildcard SSL certificate, you do not need to provide this. [1] null
web_stickiness_ttl The Web target group stickiness timeout [1] 20
web_tg_stickiness_enabled To enable/disable the Leaf target group stickiness [1] false
wildcard_ssl_certificate_password The password of the provided wildcard certificate If you provided Web and Leaf SSL certificates, you do not need to provide this. [1] null
wildcard_ssl_certificate_path The path of the provided valid wildcard certificate If you provided Web and Leaf SSL certificates, you do not need to provide this. [1] null
wingame_image_prefix Prefix used for the wingame image default
[1](1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18)

New in version 1.25.0.

See the Terraform Configuration Documentation for more information on how to set up these variables globally or locally.

Availability Zone Examples:

  • 3 instances, azs: [] -> multi-az disabled
  • 3 instances, azs: [2] -> all 3 instances are created in zone 2
  • 3 instances, azs: [1, 2, 3] -> instance 1: zone 1, instance 2: zone 2, instance 3: zone 3
  • 3 instances, azs: [1, 3] -> instance 1: zone 1, instance 2: zone 2, instance 3: zone 1
  • 4 instances, azs: [1, 2] -> instance 1: zone 1, instance 2: zone 2, instance 3: zone 1, instance 4: zone 2

Note

  • Availability zones are not available for Game Instances. Azure does not support zones for GPU instances.
  • Changing the number of Server instances specifically will force a rebuild of all instances. This causes the internal IP addresses to be updated, which the other services need to be reinitialized with.

Terraform Providers

Providers are specific to a cluster and, more specifically, to the module imported in that cluster. Thus, let us start by creating a new cluster and importing some module on it.

../_images/terraform_plan_init.png

Fig. 31 On the first import, you will see the provider plugins being downloaded. It is a great way to know which providers are used by a specific module.

Every provider has its own set of arguments it recognizes. The best way to know how a specific provider can be customized is by visiting the Terraform Providers page and looking up the specific providers you are interested in.

Important

You should always specify the version to use for each provider when importing the module. If you don’t, the terraform init command will use the newest version available for each. This can lead to an unstable environment if any updates introduce a breaking change.

See the Terraform documentation for more information.

Customizing Terraform Providers

(Global) Default Configuration

There are two levels at which providers configuration can be customized: at the global (bastion) level or on the cluster directly after importing a module. The global configuration is meant to give a default configuration to new clusters (or on a reimport of the module).

../_images/bastion_ui_provider_settings.png

During the installation of the bastion, we set up such a configuration so that your clusters work out of the box with any of the modules we provide. The information is loaded from a file stored at bastion-services/terraform/providers/default.json.

When you import a module into a cluster, only the default configurations for providers that this module uses are applied. So it’s safe to add configurations for providers that aren’t used by every module.

../_images/bastion_ui_add_new_provider.png

To add a new default configuration for a provider:

  1. Open the Bastion UI page.

  2. Click Settings.

  3. Click Providers.

  4. Click + NEW PROVIDER.

  5. Enter a unique name for Provider and Alias to identify the provider you want to customize.

    • OPTIONAL: You can also set a specific default Version to use for that provider.

    Important

    The identifiers you use for Provider and Alias are checked when you import a module into a cluster. If they don’t match a provider used by that module, the default configuration won’t be applied.

  6. If you have one or more arguments you need to set for a provider, click + to add each.

  7. Click SAVE to save the provider configuration.

Per-Cluster Configuration

After importing a module on your cluster, you can go to the Provider tab (under Terraform) to see its current configuration.

../_images/bastion_ui_provider_need_module1.png

Fig. 32 Like for the Settings tab, if you did not import a module yet, you will be invited to do so by going to the Commands page.

By default, the configuration should be empty, unless some providers have a global configuration which will be automatically applied by this point. Any further change made to the cluster’s providers will only affect that cluster.

Warning

If the cluster’s module is re-imported, the global configuration will override its configuration.

../_images/bastion_ui_cluster_view_provider.png

Fig. 33 As you can see in this example, Terraform variable interpolation is supported to specify values for the providers’ arguments.

Once your cluster is created and you know which providers you want to customize, you may go to Terraform, from the top menu, and then Provider under the cluster’s name from the left menu.

Important

After modifying the version of a provider you need to execute a new terraform init command for the new version to be downloaded.

The process to customize providers is very similar to the global case. The + ADD PROVIDER button is used to add a configuration for a provider not yet listed. Otherwise, the edit button (the left Action button) can be used to customize existing configurations.

Warning

The changes to the providers configuration might not take effect until a Refresh command is issued.

Terraform Providers Arguments

You will notice that we distinguish between meta-arguments such as version and alias and other arguments.

Those meta-arguments are present on all providers and are considered during Terraform’s init phase. The other arguments are used during the plan or apply phase and vary between providers.

Note

If you notice some arguments are ignored at the plan phase, try to issue a Refresh to update Terraform’s state with the new configuration.

Provider aliases might not be useful unless you write your own Terraform module, but the version is very important as we’ve seen above. More information can be found on Terraform Provider Configuration page.

Terraform Providers Toolbox Support

We offer subcommands in the Toolbox to help managing providers. The following table summarizes the commands available.

Command Description
genvid-bastion get-default-terraform-providers Display the current global providers configuration in JSON.
genvid-bastion set-default-terraform-providers Set the global providers configuration from a JSON file matching the format from get-default-terraform-providers.
genvid-bastion delete-default-terraform-providers Delete the existing global providers configuration.
genvid-clusters get-terraform-providers Display the current providers configuration of a specific cluster.
genvid-clusters set-terraform-providers Set the providers configuration of a specific cluster from a JSON file matching the format from get-terraform-providers.
genvid-clusters delete-terraform-providers Delete the existing providers configuration of a specific cluster.

While the easiest way to create a new providers configuration is to use the bastion UI and then the toolbox to redirect the configuration into a file, here is an example in case you want to write it by hand:

[
    {
        "provider": "azurerm",
        "meta": {
            "alias": null,
            "version": "~> 2.1.0"
        },
        "arguments": {
            "region": "East US"
        }
    },
    {
        "provider": "template",
        "meta": {
            "alias": null,
            "version": "~> 2.1"
        },
        "arguments": {}
    },
    {
        "provider": "tls",
        "meta": {
            "alias": "myalias",
            "version": null,
        },
        "arguments": {}
    }
]

Applying a Terraform Infrastructure

Terraform Apply is the operation that builds the cluster infrastructure.

  1. Click Plan Apply to create an execution plan.
  2. Verify the changes match what you need.
  3. After verifying the changes, click Apply to execute the plan.
Terraform Plan Apply
Terraform Apply

You should see the log starting to appear.

If Terraform fails to build the infrastructure:
  • Check the error message.
  • Update the settings.
  • Apply again.

At the end of this step, there is a cluster running on the cloud. When you check All Configs on the Terraform page, the status is UP. This means that the infrastructure is up, but the Genvid SDK is not yet present in it.

Important

Beginning in version 1.19.0, instances provision in the background after Terraform creates them. You need to wait until an instance registers as a Nomad client in the Cluster UI before using it. In particular, Windows instances may take up to 30 minutes before they are ready.

Note

The public machine IP is not the same as the one used when setting up your AMI.

To access your game machine, use the IP from game_public_ips with the same password set during the AMI setup.

To retrieve the game_public_ips:

  1. Make sure the cluster is UP.
  2. Go to the Commands page of your cluster.
  3. Click the OUTPUT button.

You’ll find the game_public_ips listed in the JSON file.

Setting up the SDK on a Running Cluster

Once a cluster status is UP, you will need to set up the Genvid SDK on it and run your project there.

Follow the SDK in the Cloud guide to do so.

Destroying a Terraform Infrastructure

The terraform destroy command removes all resources from the cluster configuration. You should only destroy a cluster’s Terraform infrastructure when you no longer need to run your project on that cluster. To destroy the Terraform infrastructure, click Plan destroy.

If you’re sure you want to destroy the current Terraform infrastructure, click the Destroy button to confirm.

See Destroy Infrastructure for more information.

Deleting a Cluster

To delete a cluster, you have to destroy the Terraform infrastructure first. Go to All Configs and click the Delete button.

Using Custom Repositories

You can add and remove individual Terraform repositories in the bastion. Each repository contains one or more modules that can be used to instantiate a cluster. Use the following command to list the current repositories:

genvid-clusters module-list

Use the following command to add a new module:

genvid-clusters module-add -u {URL} {name}
  • {URL} can be any source compatible with go-getter, including local files.
  • {name} is the destination folder to this repository on your bastion.

After the URL is cloned in the bastion repository, it will be available as a source under modules/module. See Terraform’s Module Configuration for more details on using modules.

Bastion remembers the origin of each module, so you can update them using the following command:

genvid-clusters module-update [name]

The name is optional. If you don’t provide a name, it will update all repositories.

You can remove a module using the following command:

genvid-clusters module-remove {name}

See also

genvid-clusters
Genvid Cluster-Management script documentation.
Bastion API for Terraform
Bastion API for Terraform.
Terraform’s Module Configuration
Documentation of Modules on Terraform.
go-getter
A library for fetching URL in Go.