Running in the Cloud

The instructions here assume that you are already familiar with the Local Environment and its local.py script, explained in the the Quick Tour Guide, and that you have built and run the tutorial and the UT4 samples locally.

Building with Docker

The samples rely on Docker not only for the Genvid Services, but also for building and running the website. Although you can install a version of Docker locally (see Install Docker on Windows for more information), having access to a cloud provider can provide us with a simpler alternative [*]: We can create a Docker build server directly on Amazon EC2 using Docker Machine. The process is very straightforward now and should take less than 5 minutes.

[*]A future version of the SDK will likely support building the website directly on the servers, but it is not available right now.

The first step is to find an availability zone that you have access:

aws ec2 describe-availability-zones --query AvailabilityZones[0].ZoneName --output text

This will return a name, e.g., us-east-1b. The last letter correspond to the availability zone and everything else is the region. In the example above, the region name would be us-east-1 and the availabiility zone is b. You can now create a Docker machine on this availability zones. Using the value above, you get:

bin\docker-machine create --driver amazonec2 --amazonec2-region us-east-1 --amazonec2-zone b [docker-name]

The [docker-name] is a unique name to refer to the machine. Once you are done with the machine, you can always shut it down:

bin\docker-machine rm [docker-name]

Setup your cluster

First, let start your cluster. This operation is a bit long, since this required the windows game machine to configure itself. The whole operation takes about 25 minutes, add or remove 10 minutes, so better do it now:

py cloud.py setup

At the end of it, you should have four servers running on AWS, as well as default S3 buckets containing the services images. For a test, the current setup, as you have done previously, is enough. In production, you may need to adjust the number of servers. Take note that the current version of the Genvid Stack is not yet considered to be ready for a large production deployment. Expose it carefully and be sure you don’t leave anything important on the server should you make it public.

When the servers are up, they could be used for multiple streaming session, even with different games, as long as a single session is run at a time. Although you can reboot the server if something goes wrong, the current SDK doesn’t allow you to shutdown the server; They will loose their public IP addresses and their configuration will not be valid anymore. A future version of the SDK will fix this, allowing faster restart of your cluster.

Selecting and building your project

You can now choose the sample you want to try. Let’s start with tutorial. To select it, run:

py cloud.py load-project samples/tutorial

This will load the project settings and saved it under config/_cloud.json. Just like in the local environment, the project file will not be touched anymore if you change it. To reload it, you must run:

py cloud.py reload-project

Which will shutdown any running game to ensure no rogue process were left.

Our tutorial is a small game, with very little content. Therefore, we can simply build an archive and make it available for upload under the images subdirectory. The command also builds the website which, as stated previously, requires Docker to succeed. To advertise your Docker installation or server, you must run the following command:

# On Windows command prompt
@FOR /f "tokens=*" %i IN ('"bin\docker-machine" env [docker-name]') DO @%i

# On bash for Windows
eval $(bin/docker-machine env [docker-name])

# With Powershell
bin/docker-machine env [docker-name] | Invoke-Expression

Where [docker-name] as to be replaced with the name previously chosen, or default if you are running Docker locally. Once this is done, you can build the game using this following script command:

py cloud.py run-script build_cloud

This will automatically build two archives in the images subdirectory. Note that the script puts a checksum in the image name. This allows a trigger during the setup command to re-upload the image and remove the old ones (if necessary). A production version of the script should probably be more conservative and keep the old version, but right now, the old version is always removed.

Starting the game

Just run:

py cloud.py start

This will update the cluster with the new images on S3, and start the game. This can take a few minutes, especially if the game machine is not present. After that, you can restart and update the game as you like (using the restart command) and it should be very fast.

Sometime, some of the services timeout, or return an error. We try to handle most of those situations, but if this happen, you often only have to recall the last command. The application will do its best to progress further.

Shutting down the cluster

You can shut down the game by running the following command:

py cloud.py stop

This will keep the server alive but will shut down any streaming. You can also destroy the whole cluster by running the following command:

py cloud.py clean

This will remove all traces of the cluster including the S3 buckets, except for the AMI images. Note that we do not currently simply shut down the servers, therefore the new server instances will have different public IP adresses, which can lead to some errors. We are actively working on fixing this to allow for a fast restart of the servers.

Running the Unreal Tournament game

Unreal Tournament is a large game, and requires some preparation in order to simplify transfers to the server. To start, you need to create a package.

The first thing to do is to package your game project. Follow the Unreal Engine Documentation on Packaging Projects and create a Package for the WindowsNoEditor platform in Development mode. The [UT Folder] below corresponds to the [PackageDirectory]\\WindowsNoEditor folder of the documentation.

aws s3 mb s3://[some_unique_name]/v1/
aws s3 sync [UT Folder]/ s3://[some_unique_name]/v1/

Now, add the bucket to your configuration files (for example, inside config\ut4bucket.hcl) like this:

cloud {
  artifact {
    s3url = "s3://[some_unique_name]/v1/"
    destination = "z:\\game\\"
  }
}

The destination is the destination folder of the game on the game machine. We are using the Z:\ drive, which is a local drive to the AWS instance, to avoid hitting IO limits of an EBS volume which would otherwise stall the game process.

Now, be sure to change the set up game to ut4:

py cloud.py load-project samples/ue4/ut4

If any games are running on the servers, this will stop them.

Ensure the servers are properly setup:

py cloud.py setup

And then start the custom job for updating the game:

py cloud.py start update

This will start a job. It takes about 5 minutes, and you can see the last output and status of the job using:

py cloud.py log -t update

Or simply look it up on the hashi-ui webpage.

During this time, you can build the website as normal and upload it to S3 too:

py cloud.py run-script build_cloud
py cloud.py setup

The last command will ensure that the Docker images are all up-to-date.

Once the update job is finished, you can start the game as usual:

py cloud.py start

After a couple of seconds, the website should be ready. Open the webpage with the command:

py cloud.py open

For stopping the game after that, just run:

py cloud.py stop   # Stop the game but not the servers.
py cloud.py clean  # Terminate the servers, except for the UT4 buckets.

Available Commands

Here’s the list of the available commands for cloud.py, all of which can be consulted via the -h option:

  • check-config: Validate the current configuration.
  • clean: Uninstall services.
  • clear-config: Clear all configuration.
  • config: Configure the broadcast session.
  • copy-ami: Copy an AMI across regions.
  • env: Print the environment variable used by the script..
  • generate-keypair: Generate a key pair under config.
  • load-project: Load a project configuration.
  • log: Return the specific log for some task.
  • monitor: Open monitors pages.
  • open: Show the available links or open the specified link.
  • output: Return the output of the cluster state..
  • plan: Run plan on the infrastructure.
  • reinstall: Clean up and reinstall the services.
  • reload-config: Clear and reload the broadcast session.
  • reload-project: Reload current project.
  • rename-ami: Copy a Windows AMI from one prefix to another.
  • restart: Restart the jobs.
  • run-script: Run a project script.
  • save-ami: Create the game AMI.
  • setup: Setup the services.
  • setup-ami: Initialize a new game instance on AWS.
  • show-config: Show the configuration.
  • start: Start the game.
  • status: Give a status.
  • stop: Stop the jobs.