Unity Sample Website Integration

After completing the Unity engine integration, you will need a website able to make the proper calls to the Genvid API to display the stream and use the game data on the website. Also, if you want to interact with your game from the website, you need to do specific calls via the Genvid API.

This page will cover a step by step example on how we created the website for the Unity sample. Feel free to use any code that will be useful for your development from our sample.

  1. Files used to run the website
  2. Unity.ts - Classes and interfaces
  3. Unity.ts - Enter frame
  4. Unity.ts - User interactions
  5. Unity.ts - WebGL
  6. Unity.ts - Utility methods
  7. Admin.ts

Files used to run the website

All the files indicated in this section are located inside web/public. The other files located in web that are not inside public are used for the local server when starting the sample.

index.html
This is the main page for the website. The command genvid-sdk open web displays it. This document includes various JavaScript files and css files.
unity.ts
This is a TypeScript file which creates a JavaScript file included in the index.html and admin.html documents. It performs all calls to the Genvid API and other interactions on the index page.
genvid.d.ts

This is a TypeScript file which creates a JavaScript file included in the index.html and admin.html documents, but is used in the unity.ts file. This file is needed to make any call to the Genvid API.

genvid.d.ts only appears in the public folder after you run py unity.py build since it copies the file from the web API folder.

css folder
This folder contains icons.css and style.css. The cheer button requires icons.css and style.css defines all the other style options for index.html and admin.html.
img folder
This folder contains the images used for the website. The circle highlight made in WebGL, the Genvid logo, and the Genvid overlay-button all use images from the img folder.
js folder
This folder contains the JavaScript and map files generated for the TypeScript files: adminUnity.ts, genvid.d.ts, and unity.ts when you run the py unity.py build command. When using a TypeScript file, you make changes to the TypeScript file but include the JavaScript file in the html document.
admin.html

This page is the admin section for the website. It contains several buttons that allow direct interaction with the game objects.

The admin page requires a user name and password. The default user name is admin and the password is admin.

adminUnity.ts
This is a TypeScript file which creates a JavaScript file included in the admin.html document. It does all the calls to the Genvid API and other interactions on the admin page.
favicon.icon
This is an icon for the webpage.
tsconfig.json
This is a file needed for building the TypeScript files. Add the TypeScript files that you need to build in the include section of this file.
tslint.json
This is a file needed for building the TypeScript files. It formats the JavaScript created with the TypeScript files.
webpack.config.js
The webpack configuration tells (Webpack) how
to bundle the unity app.

Unity.ts - Classes and interfaces

For the web page, we started by creating a namespace named unitySample. This namespace contains several interfaces and classes.

interface IGameDataCube

This interface is used to properly sort the information from the game data. We normally catch the data as an array then convert it to this interface as we can have the information related to the object easily accessible.

  1. name: name of the object
  2. leader: boolean that indicate if the object is the leader
  3. popularity: current popularity value of the object
  4. mat: current matrix of the object
  5. selected: boolean that indicate if the object is selected
  6. popText: popularity value transformed into string
  7. color: current color of the object
    // Conversion from Json data into structure for a specific cube
    export interface IGameDataCube {
        name: string;
        popularity: number;
        mat: IUnityMatrix4x4;
        selected?: boolean;
        popText?: string;
        color: IUnityColor;
    }

interface ICubePopularity

This interface is used to properly sort the information related to the popularity.

  1. name: name of the object
  2. popularity: popularity value of the object
    // Conversion from Json data into structure for the popularity
    export interface ICubePopularity {
        name: string;
        popularity: number;
    }

interface IGameData

This interface is used to properly sort the information from the game data. Since we are sending information from the objects and the projection view matrix at the same time, we need this interface to sort the data.

  1. cubes: information about each object that is using the IGameDataCube interface
  2. MatProjView: the projection view matrix (useful for the WebGL portion)
    // Conversion from Json data into structure for the entire data
    export interface IGameData {
        cubes: IGameDataCube[];
        MatProjView: IUnityMatrix4x4;
        camera: number;
        scene: number;
    }

class RenderCommand

This class is used for the WebGL section and it is needed to use WebGL properly.

    // WebGL class for rendering command
    class RenderCommand {
        visible: boolean;
        vtx: [WebGLBuffer, number]; // Vertex buffer and length
        idx: [WebGLBuffer, number]; // Index buffer and length
        tex: any; // Texture ID.
        img: any; // HTML image used for the texture (async load).
    }

class unityController

This class is used to do all the interactions with the page along displaying the proper content on it. Since this class is the most important part of this help documentation, we have divided the content in several sections which we will cover in details on this page:

  1. Genvid Client initialization section
  2. Enter frame section
  3. User interactions section
  4. WebGL section
  5. Utility methods section

Each section can be found easily with the section name indicated inside a comment along symbols to make it easier to see in the unity.ts file.

Class creation process

Outside of the namespace unitySample, we proceed to create an instance of the class unityController using video_player as an argument. This argument is the id the tag that will contains the video stream. We proceed to call the onConnect() method from this object to start connecting to the stream and we are adding a keydown listener to the page to detect key press.

let unityVideo = new unitySample.UnityController("video_player");
unityVideo.onConnect();
window.addEventListener("keydown", (event) => { unityVideo.onKeyDown(event); }, true);

Unity.ts - Genvid client initialization

The Genvid Client initialization contains most of the code needed to start the stream, initialize various variables and assign several event listeners.

onConnect()

This method is starting the connection to the services. If the connection is executed properly, we proceed to the on_channel_join method with the genvid.IChannelJoinResponse information found.

        // Start the connection to the services
        onConnect() {
            let promise = $.post("/api/public/channels/join", {}, (joinRep) => {
                this.on_channel_join(<genvid.IChannelJoinResponse>joinRep);
            });
            promise.fail((err) => {
                alert("Can't get the stream info:" + err);
            });
        }

on_channel_join(info: genvid.IChannelJoinResponse)

This method creates the Genvid client after properly to the services and channel. We use the information found during this process and the video_player_id sent during the class creation process to create the client (it is an argument to the unityController class).

Afterwards, we need to associate specific events to function in this class:

  1. onVideoPlayerReady: Triggers when the video stream is ready (used to initialize content).
  2. onStreamsReceived: Triggers when the stream content is received (used to get the timecode and game data).
  3. onNotificationsReceived: Triggers when a notification is received (used for the popularity).
  4. onDraw: Triggers when drawing a new frame in the video.

We then proceed to start the client with the start() method.

        // Create the genvid Client and the function listening to it
        private on_channel_join(joinRep: genvid.IChannelJoinResponse) {
            this.streamInfo = joinRep.info;
            this.client = genvid.createGenvidClient(this.streamInfo, joinRep.uri, joinRep.token, this.video_player_id);
            this.client.onVideoPlayerReady((elem) => { this.on_video_player_ready(elem); });
            this.client.onStreamsReceived((streams) => { this.on_streams_received(streams); });
            this.client.onNotificationsReceived(this.on_notifications_received.bind(this));
            this.client.onDraw((frame) => { this.on_new_frame(frame); });
            this.client.start();
        }

on_video_player_ready(_elem: HTMLElement)

This method is used to initiate several variables, create the WebGL context and add several event listeners.

  1. We get most of the Div and Link elements in this function via JQuery.
            this.video_player = this.client.videoPlayer;

            this.timeLocalDiv = <HTMLDivElement>document.querySelector("#time_local");
            this.timeVideoDiv = <HTMLDivElement>document.querySelector("#time_video");
            this.timeVideoRawDiv = <HTMLDivElement>document.querySelector("#time_video_raw");
            this.timeComposeDiv = <HTMLDivElement>document.querySelector("#time_compose");
            this.timeComposeLastDiv = <HTMLDivElement>document.querySelector("#time_compose_last");
            this.timeStreamDiv = <HTMLDivElement>document.querySelector("#time_stream");
            this.latencyDiv = <HTMLDivElement>document.querySelector("#latency");
            this.delayOffsetDiv = <HTMLDivElement>document.querySelector("#delay_offset");
            this.controlsDiv = <HTMLDivElement>document.querySelector("#game-controls");

            this.timeCamSceneDiv = <HTMLDivElement>document.querySelector("#timeCamScene_overlay");

            this.promptOverlay = <HTMLDivElement>document.querySelector("#prompt_overlay");

            this.videoOverlay = <HTMLDivElement>document.querySelector("#video_overlay");
            this.canvas3d = <HTMLCanvasElement>document.querySelector("#canvas_overlay_3d");

            this.mouseOverlay = <HTMLDivElement>document.querySelector("#mouse_overlay");

            this.genvidOverlayButton = <HTMLLinkElement>document.querySelector("#genvid_overlay_button");
            this.genvidOverlay = <HTMLDivElement>document.querySelector("#genvid_overlay");
            this.help_overlay = <HTMLDivElement>document.querySelector("#help_overlay");
            this.helpButton = <HTMLLinkElement>document.querySelector("#help_button");
            this.fullScreenDiv = <HTMLDivElement>document.querySelector(".fullscreen-button");
            this.fullScreenElement = <HTMLElement>document.querySelector(".fa-expand");
  1. We proceed to create the webGLContext with the canvas3d selected.
  2. We are adding all the window change event listeners (fullscreen, resize).
  3. We are adding the event listeners for the click interaction on the mouseOverlay (for clicking on the video objects), the Genvid button and the help button.
            this.hideOverlay();
            this.client.videoPlayer.addEventListener(genvid.PlayerEvents.PAUSE, () => {
                this.hideOverlay();
            });

            this.client.videoPlayer.addEventListener(genvid.PlayerEvents.PLAYING, () => {
                this.showOverlay();
            });

            this.genvidWebGL = genvid.createWebGLContext(this.canvas3d); // Need to assign before any resize.

            if (this.on_video_ready_callback) {
                this.on_video_ready_callback();
            }

            document.addEventListener("fullscreenchange", () => { this.onResize(); });
            document.addEventListener("webkitfullscreenchange", () => { this.onResize(); });
            document.addEventListener("mozfullscreenchange", () => { this.onResize(); });
            _elem.addEventListener("resize", () => { this.onResize(); });

            window.addEventListener("resize", () => { this.onResize(); }, true);
            window.addEventListener("orientationchange", () => { this.onResize(); }, true);
            window.addEventListener("sizemodechange", () => { this.onResize(); }, true);
            window.setInterval(() => { this.onResize(); }, 1000); // Just a safety, in case something goes wrong.

            this.mouseOverlay.addEventListener("click", (event) => { this.clickCube(event); }, false);
            this.genvidOverlayButton.addEventListener("click", (_event) => { this.toggleGenvidOverlay(); }, false);
            this.helpButton.addEventListener("click", (_event) => { this.onHelpActivation(); }, false);
            this.fullScreenDiv.addEventListener("click", (_event) => { this.toggleFullScreen(); }, false);
  1. Finally, we are proceeding to do an onResize() to fit the overlays to the proper size of the window (this is required otherwise the overlays are too small) and we are proceeding to initialize the WebGL overlay.
            this.onResize();

            // Initialize graphics stuff.
            this.genvidWebGL.clear();
            let gl = this.genvidWebGL.gl;
            gl.disable(gl.DEPTH_TEST);

            this.gfx_initShaders();
            this.gfx_initRenderCommands();
            const muteIcon = document.getElementById("mute-button");
            muteIcon.addEventListener("click", () => this.onMute());
            if (!this.video_player.getMuted()) {
                this.onMute();
            }

            this.videoReady = true;

Unity.ts - Enter frame

The enter frame section contains mostly methods that are triggered each frame or in a similar way.

on_new_frame(frameSource: genvid.IDataFrame)

This method is called every frame for the genvid client that was created. We perform various tasks that need constant update in this method.

  1. We need to get the game data and verify that this data is valid before doing any operation with it.
  2. Once we are certain about the data validity, we proceed to get the object list from the data (cubes).
  3. We are updating the text field for the time left until the next camera and scene.
  4. We verify if we did setup the player table, if that is not the case, we call initPlayerTable.
  5. We use the objects array to create a loop that will create the circles that are made in WebGL.
  6. With the same loop, we are updating the popularity value and the position value for each object.
  7. After the loop, we draw the WebGL circles.
            // Parse the JSON from each elements
            let gameDataFrame = frameSource.streams["GameData"];
            let gameData: IGameData = null;

            if (gameDataFrame && gameDataFrame.user) {

                gameData = gameDataFrame.user;
                this.lastGameData = gameData;
                let cubes = gameData.cubes;

                // Let update the time of camera and scene
                let textForTime = "Time until next camera: " + gameData.camera.toFixed(2) + " seconds \r\n Time until next scene: " + gameData.scene.toFixed(2) + " seconds";
                this.timeCamSceneDiv.textContent = textForTime;

                // Setup the player table once the game data is ready
                if (this.playerTableSetupCompletion === false) {
                    this.playerTableSetupCompletion = true;
                    let cubes = gameData.cubes;

                    this.initPlayerTable(cubes);
                }

                // Perform the webgl update process -- Update 3d
                if (cubes) {
                    let vertices: number[] = [];
                    for (let cube of cubes) {
                        let m = cube.mat;
                        let p = genvidMath.vec3(m.e03, m.e13, m.e23);
                        let r = this.circleRadius;
                        let c = genvidMath.vec4(1, 0, 0, 1);
                        if (cube.color) {
                            c = genvidMath.vec4(cube.color.r, cube.color.g, cube.color.b, cube.color.a);
                        }
                        if (cube.selected && this.selection.length === 0) {
                            // To gather initial state.
                            this.setSelection(cube.name);
                        }
                        if (!this.isSelected(cube.name)) {
                            c = genvidMath.muls4D(c, 0.5);
                        }
                        this.makeCircleZ(vertices, p.x, p.y, p.z, r, c);

                        // Move the name tag of the cube
                        let tag = this.findOrCreateTagDiv(cube);
                        let mat = this.convertMatrix(gameData.MatProjView);
                        let pos_2d = genvidMath.projectPosition(mat, p);
                        this.center_at(tag, pos_2d, genvidMath.vec2(0, -75));

                        // Modify the popularity with the latest value
                        if (this.latestPopularity) {
                            let pCube = this.latestPopularity.cubes.find((c) => { return c.name === cube.name; });
                            if (pCube) {
                                cube.popularity = pCube.popularity;
                            }
                        }

                        cube.popText = this.popularityToText(cube.popularity);
                        let cubePopSpan = <HTMLSpanElement>document.querySelector(".cube" + cube.name + "_cheer");
                        cubePopSpan.textContent = cube.popText;

                        let cubePosXSpan = <HTMLSpanElement>document.querySelector("#cube" + cube.name + "_position_x");
                        cubePosXSpan.textContent = cube.mat.e03.toFixed(2);

                        let cubePosYSpan = <HTMLSpanElement>document.querySelector("#cube" + cube.name + "_position_y");
                        cubePosYSpan.textContent = cube.mat.e13.toFixed(2);

                        let cubePosZSpan = <HTMLSpanElement>document.querySelector("#cube" + cube.name + "_position_z");
                        cubePosZSpan.textContent = cube.mat.e23.toFixed(2);
                    }

                    let num_quads = vertices.length / (4 * 9);

                    let genvidWebGL = this.genvidWebGL;
                    let cmd: RenderCommand = this.gfx_cmd_cubes;
                    cmd.vtx = genvidWebGL.createBuffer(new Float32Array(vertices));
                    cmd.idx = genvidWebGL.createIndexBufferForQuads(num_quads);
                }

                let mat = gameData.MatProjView;
                this.gfx_prog_data_viewproj = [mat.e00, mat.e10, mat.e20, mat.e30, mat.e01, mat.e11, mat.e21, mat.e31, mat.e02, mat.e12, mat.e22, mat.e32, mat.e03, mat.e13, mat.e23, mat.e33];

                this.gfx_draw3D();
            }
  1. Afterwards, we verify if the video is ready to perform others operations.
  2. If the video is ready, we update the Genvid overlay values.
  3. We are also updating the prompt overlay visibility (used to display User changes to the sound).
  4. Finally, we verify if the video is fullscreen and update the status for it.
            if (this.videoReady) {
                // Update the Genvid information overlay
                let w = 18; // Width of the content of every line (without label).
                let localTime: Date = new Date();
                this.timeLocalDiv.textContent = `Local: ${this.msToDuration(Math.round(localTime.getTime()))}`;

                let videoTimeRawMS = 0;
                let videoPlayer = this.client.videoPlayer;
                if (videoPlayer) {
                    videoTimeRawMS = videoPlayer.getCurrentTime() * 1000;
                }
                this.timeVideoRawDiv.textContent = `Raw Video: ${this.preN(this.msToDuration(Math.round(videoTimeRawMS)), w)}`;

                this.timeVideoDiv.textContent = `Est. Video: ${this.preN(this.msToDuration(Math.round(this.client.videoTimeMS)), w)}`;

                this.timeComposeLastDiv.textContent = `Last Compose: ${this.preN(this.msToDuration(Math.round(this.client.lastComposeTimeMS)), w)}`;

                this.timeComposeDiv.textContent = `Est. Compose: ${this.preN(this.msToDuration(Math.round(this.client.composeTimeMS)), w)}`;

                this.timeStreamDiv.textContent = `Stream: ${this.preN(this.msToDuration(Math.round(this.client.streamTimeMS)), w)}`;

                this.latencyDiv.textContent = `Latency: ${this.preN(this.client.streamLatencyMS.toFixed(0), w - 3)} ms`;

                this.delayOffsetDiv.textContent = `DelayOffset: ${this.preN(this.client.delayOffset.toFixed(0), w - 3)} ms`;

                // Update the visibility on the overlay when using key press
                if (this.promptOverlay.style.visibility === "visible" && this.timeVisiblePrompt < this.timeVisibleMax) {
                    this.timeVisiblePrompt++;
                    if (this.volumeChange === 2) {
                        this.volumeChange = 0;
                        this.promptOverlay.textContent = "Volume: " + this.client.videoPlayer.getVolume().toString() + " %";
                    } else if (this.volumeChange === 1) {
                        this.volumeChange = 0;
                        this.promptOverlay.textContent = "Volume: " + this.client.videoPlayer.getVolume().toString() + " %";
                    }
                } else if (this.promptOverlay.style.visibility === "visible" && this.timeVisiblePrompt >= this.timeVisibleMax) {
                    this.promptOverlay.style.visibility = "hidden";
                }
            }
            let isFullScreen = this.checkFullScreen();
            if (isFullScreen !== this.isFullScreen) {
                this.isFullScreen = isFullScreen;
            }

initPlayerTable(cubeData: IGameDataCube[])

This method is called when the website needs to create the table for each object under the stream.

  1. By using the game data sent, we create a loop to use each object information.
  2. We are adding html code for the table in a string that we will append to an html tag afterwards.
  3. Afterwards, we are adding all the event listeners the click functionality for the table, cheer and reset.
  4. Finally, we are doing a loop for each color available and add a click event listener.
        // Method used to display the appropriate number of players with their proper buttons
        initPlayerTable(cubeData: IGameDataCube[]) {
            for (let cube of cubeData) {
                let cubeAdd = `<div class='col-md-6 col-lg-4 nopadding'>
                                        <div class='cube clickable cube` + cube.name + `'>
                                            <div>
                                                <span class='cube_name clickable'>` + cube.name + `</span>
                                                <button class='cheer' id='` + cube.name + `_cheerbutton'><i class='icon_like' aria-hidden='true'></i></button>
                                                <span class='cheer_value cube` + cube.name + `_cheer'></span>
                                            </div>
                                            <div>
                                                <span class='label clickable cube` + cube.name + `_reset'>Reset</span>
                                                <span id='cube` + cube.name + `_position_x' class='cube_position'></span>
                                                <span id='cube` + cube.name + `_position_y' class='cube_position'></span>
                                                <span id='cube` + cube.name + `_position_z' class='cube_position'></span>
                                            </div>
                                            <table class='cube_color text-center'>
                                                <tr>
                                                    <td class='clickable color` + cube.name + `_green'>Green</td>
                                                    <td class='clickable color` + cube.name + `_white'>White</td>
                                                    <td class='clickable color` + cube.name + `_yellow'>Yellow</td>
                                                </tr>
                                                <tr>
                                                    <td class='clickable color` + cube.name + `_darkblue'>Dark blue</td>
                                                    <td class='clickable color` + cube.name + `_grey'>Grey</td>
                                                    <td class='clickable color` + cube.name + `_lightblue''>Light Blue</td>
                                                </tr>
                                                <tr>
                                                    <td class='clickable color` + cube.name + `_orange'>Orange</td>
                                                    <td class='clickable color` + cube.name + `_blue'>Blue</td>
                                                    <td class='clickable color` + cube.name + `_purple'>Purple</td>
                                                </tr>
                                            </table>
                                        </div>
                                    </div>`;

                $(".gameControlsDiv").append(cubeAdd);

                let cheerButton = <HTMLButtonElement>document.querySelector("#" + cube.name + "_cheerbutton");
                cheerButton.addEventListener("click", (_event) => { this.onCheer(cube.name); }, false);

                let cubeDiv = <HTMLDivElement>document.querySelector(".cube" + cube.name);
                cubeDiv.addEventListener("click", (_event) => { this.onSelect(cube.name, true); }, false);
                this.cubeDiv.push(cubeDiv);

                let resetButton = <HTMLSpanElement>document.querySelector(".cube" + cube.name + "_reset");
                resetButton.addEventListener("click", (_event) => { this.onReset(cube.name); }, false);

                for (let colorSelect of this.tableColor) {
                    let colorButton = <HTMLSpanElement>document.querySelector(".color" + cube.name + "_" + colorSelect[0]);
                    colorButton.addEventListener("click", (_event) => { this.onColorChange(cube.name, colorSelect[1]); }, false);
                }
            }
        }

on_streams_received(dataStreams: genvid.IDataStreams)

This method is called when receiving the stream. We use loops to get the latest time code associated with the game data. We are also converting the JSON data received into IGameData.

        // Upon receving the stream, get the timecode and the data
        private on_streams_received(dataStreams: genvid.IDataStreams) {
            for (let stream of dataStreams.streams) {
                for (let frame of stream.frames) {
                    if (this.last_game_time_received < frame.timeCode) {
                        this.last_game_time_received = frame.timeCode;
                    }
                }
            }

            // Parse the JSON from each elements
            for (let stream of [...dataStreams.streams, ...dataStreams.annotations]) {
                for (let frame of stream.frames) {
                    try {
                        frame.user = <IGameData>JSON.parse(frame.data);
                    }
                    catch (err) {
                        console.info("invalid Json format for:" + frame.data + " with error :" + err);
                    }

                }
            }
        }

on_notifications_received(message: genvid.IDataNotifications)

This method is called when a notification is received. In this case, we are analyzing all the notifications received and in the case that the id is Popularity, we proceed to get the data into string and transform the JSON data into ICubePopularity.

        // Upon receiving a notification, get the notification content
        private on_notifications_received(message: genvid.IDataNotifications) {
            for (let notification of message.notifications) {
                if (notification.id === "Popularity") {
                    let datastr = genvid.UTF8ToString(notification.rawdata);
                    try {
                        // Get the latest popularity
                        let userData = <ICubePopularities>JSON.parse(datastr);
                        this.latestPopularity = userData;
                    }
                    catch (err) {
                        console.info("invalid Json format for:" + datastr + " with error :" + err);
                    }

                }
            }
        }

Unity.ts - User interactions

This section contains all the methods used for the User interactions in the index.html web page. All the key press functions and click functions are then available in this section.

onKeyDown(event: KeyboardEvent)

This method is assigned to the key down event when the window is selected. Each indicated key is triggering a specific function.

  1. changeOffset: Changes the video delay offset by changing the value.
  2. toggleGenvidOverlay: Displays or hides the Genvid overlay.
  3. client.videoPlayer.isPaused: Returns a boolean to know if the video is paused.
  4. client.videoPlayer.play: Plays the video.
  5. client.videoPlayer.pause: Pauses the video.
  6. client.videoPlayer.getMuted: Returns a boolean to know if the video sound is muted.
  7. client.videoPlayer.setMuted: Sets the sound to mute or remove mute depending of the argument.
  8. client.videoPlayer.setVolume: Sets the video volume to the sent argument.
  9. client.videoPlayer.getVolume: Gets the current video volume.
  10. onHelpActivation: Displays or hides the help overlay.
        onKeyDown(event: KeyboardEvent) {
            // Seems to be the most cross-platform way to determine keys:
            // Ref: https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent/code

            let code = event.code || this.getKeyCode(event);
            if (code === "Equal" || code === "NumpadAdd") {
                this.changeOffset(+1, event);
            } else if (code === "Minus" || code === "NumpadSubtract") {
                this.changeOffset(-1, event);
            } else if (code === "NumpadMultiply") {
                this.changeOffset(0, event);
            } else if (code === "KeyG") {
                this.toggleGenvidOverlay();
            }
            else if (code === "Space") {
                if (this.client.videoPlayer.isPaused()) {
                    this.client.videoPlayer.play();
                } else {
                    this.client.videoPlayer.pause();
                }
                event.preventDefault();
            }
            else if (code === "KeyM") {
                this.onMute();
            }
            else if (code === "KeyZ") {
                this.promptOverlay = <HTMLDivElement>document.querySelector("#prompt_overlay");
                this.client.videoPlayer.setVolume(this.client.videoPlayer.getVolume() - 20);
                this.promptOverlay.style.visibility = "visible";
                this.timeVisiblePrompt = 0;
                this.volumeChange = 2;
            }
            else if (code === "KeyX") {
                this.promptOverlay = <HTMLDivElement>document.querySelector("#prompt_overlay");
                this.client.videoPlayer.setVolume(this.client.videoPlayer.getVolume() + 20);
                this.promptOverlay.style.visibility = "visible";
                this.timeVisiblePrompt = 0;
                this.volumeChange = 1;
            }
            else if (code === "KeyH") {
                this.onHelpActivation();
            }
        }

getKeyCode(event: KeyboardEvent)

This method is used to get a proper keyCode on different browsers.

        // Compatibility code for browsers (Safari) not having KeyboardEvent.code.
        getKeyCode(event: KeyboardEvent) {
            if (event.keyCode) {
                console.log(event.keyCode, event.code);
                if (65 <= event.keyCode && event.keyCode <= 90) {
                    return "Key" + String.fromCharCode(event.keyCode);
                } else {
                    switch (event.keyCode) {
                        case 13: return "Enter";
                        case 106: return "NumpadMultiply";
                        case 107: return "NumpadAdd";
                        case 109: return "NumpadSubtract";
                        case 110: return "NumpadDecimal";
                        case 111: return "NumpadDivide";
                        case 187: return "Equal";
                        case 188: return "Comma";
                        case 189: return "Minus";
                        case 190: return "Period";
                        case 222: return "Backquote";
                    }
                }
            }
        }

onResize()

This method is used to resize the various overlays used in the web page for index.html. The overlays affected by this function are: videoOverlay, canvas3d, promptOverlay (used of sound modification) and the animated divs with the name of each object.

We get the size value of the video stream displayed in the index.html page and we are adjusting the other overlays according to this size. In this sample, resize is essential to allow the object selection since we need the proper overlay size to perform the selection properly. Also, it is needed to display the WebGL circles at their appropriate locations to follow the objects in the video.

        // Allow to adjust the various overlay when resizing the windows - needed to see the PromptOverlay and Name moving div
        onResize() {
            let refElement = this.client.videoElem; // The element to match.
            let refElementSize = refElement ? genvidMath.vec2(refElement.clientWidth, refElement.clientHeight) : genvidMath.vec2(1280, 720);
            let refElementRatio = refElementSize.x / refElementSize.y;
            let videoRatio = this.client.videoAspectRatio;
            let pos: genvidMath.IVec2;
            let size: genvidMath.IVec2;
            if (videoRatio >= refElementRatio) {
                // Top+Bottom bars, fill width fully, shrink height.
                let ey = refElementSize.x / videoRatio;
                let dy = refElementSize.y - ey;
                // Center vertically.
                let y = dy * 0.5;
                pos = genvidMath.vec2(0, Math.round(y));
                size = genvidMath.vec2(refElementSize.x, Math.round(ey));
            } else {
                // Left+Right bars, fill height fully, shrink width.
                let ex = refElementSize.y * videoRatio;
                let dx = refElementSize.x - ex;
                // Center horizontally.
                let x = dx * 0.5;
                pos = genvidMath.vec2(Math.round(x), 0);
                size = genvidMath.vec2(Math.round(ex), refElementSize.y);
            }
            let style = this.videoOverlay.style;
            let cur_pos = genvidMath.vec2(parseInt(style.left), parseInt(style.top));
            let cur_size = genvidMath.vec2(parseInt(style.width), parseInt(style.height));

            if (!genvidMath.equal2D(cur_size, size, 0.9) || !genvidMath.equal2D(cur_pos, pos, 0.9)) {
                this.videoOverlay.style.left = pos.x + "px";
                this.videoOverlay.style.width = size.x + "px";
                this.videoOverlay.style.top = pos.y + "px";
                this.videoOverlay.style.height = size.y + "px";
                this.canvas3d.width = size.x;
                this.canvas3d.height = size.y;
                this.genvidWebGL.setViewport(0, 0, size.x, size.y); // Render in the whole area.
            }

            if (this.lastSelection != null) {
                this.onSelect(this.lastSelection, true);
            }
            else {
                for (let nameSelect of this.cubeDiv) {
                    nameSelect.style.backgroundColor = "#181818";
                }
            }
        }

toggleFullScreen()

This method is used to activate or deactivate the fullscreen on the video. We use the checkFullScreen(): boolean function to verify fullscreen status and proceed to the proper condition.

If we are already in fullscreen, we cancel the fullscreen according to the proper web browser function and we also adjust the fullscreen button icon.

If we are not in fullscreen, we proceed to get the video_area element. Afterwards, we use the proper web browser function to activate the fullscreen and we are updating the fullscreen button icon.

        toggleFullScreen() {
            let doc = <any>document;
            if (this.checkFullScreen()) {
                if (doc.exitFullscreen) {
                    doc.exitFullscreen();
                } else if (doc.mozCancelFullScreen) {
                    doc.mozCancelFullScreen();
                } else if (doc.webkitExitFullscreen) {
                    doc.webkitExitFullscreen();
                } else if (doc.msExitFullscreen) {
                    doc.msExitFullscreen();
                }
                this.fullScreenElement.classList.remove("fa-compress");
                this.fullScreenElement.classList.add("fa-expand");
            } else {
                let element = <any>document.querySelector("#video_area");
                if (element.requestFullscreen) {
                    element.requestFullscreen();
                } else if (element.mozRequestFullScreen) {
                    element.mozRequestFullScreen();
                } else if (element.webkitRequestFullscreen) {
                    element.webkitRequestFullscreen();
                } else if (element.msRequestFullscreen) {
                    element.msRequestFullscreen();
                }
                this.fullScreenElement.classList.remove("fa-expand");
                this.fullScreenElement.classList.add("fa-compress");
            }
        }

clickCube(event: MouseEvent)

This function is called when clicking on the WebGL overlay. We proceed to the pickCube method to know if a cube was selected or not. Otherwise, we are handling the event propagation in this function.

        // Method used when clicking on the WebGL overlay
        clickCube(event: MouseEvent) {
            let best = this.pickCube(event);
            if (best) {
                return true;
            } else {
                // Continue propagation.
                return false;
            }
        }

pickCube(event: MouseEvent)

This method is used when we need to know if a cube is selected when clicking on the WebGL overlay.

  1. We verify if the game data is valid, otherwise we are not doing any verification.
  2. We get the clicked position in the window.
  3. Afterwards, we convert this coordinate into the projected space.
            if (this.lastGameData == null) {
                return false;
            }

            // [0, 1] coordinates in the window.
            let rect = this.canvas3d.getBoundingClientRect();
            let x = event.pageX - rect.left; // More robust to recompute from pageX/pageY.
            let y = event.pageY - rect.top;
            let p01 = genvidMath.vec2(x / rect.width, y / rect.height);
            // [-1, 1] coordinates in projection space.
            let p = genvidMath.mad2D(p01, genvidMath.vec2(2, -2), genvidMath.vec2(-1, 1));
            let mat = this.convertMatrix(this.lastGameData.MatProjView);
            let best = null;
            let bestDist = Infinity;
  1. We start a loop to verify the coordinates with each object.
  2. In this loop, we get the object coordinates and apply a radius to have a zone to search.
  3. Finally, we search to discover if the coordinates is within the object zone and keep it as a best result depending of the distance (in the event that two objects are close, the one with less distance to the zone will be selected).
            for (let cube of this.lastGameData.cubes) {
                let pos3D = genvidMath.vec3(cube.mat.e03, cube.mat.e13, cube.mat.e23);
                let radius = 1.0;
                let pos2D_rad2D = this.projectWithRadius(mat, pos3D, radius);
                let pos2D = pos2D_rad2D[0];
                let rad2D = pos2D_rad2D[1];

                let pos2D_to_p = genvidMath.sub2D(p, pos2D);
                let d = genvidMath.length2D(pos2D_to_p);
                if (d < rad2D && d < bestDist) {
                    best = cube.name;
                    bestDist = d;
                }
            }

            if (event.ctrlKey || event.metaKey) {
                this.toggleSelection(best);
                this.onSelect(best, false);
            } else if (best != null) {
                this.setSelection(best);
                this.onSelect(best, false);
            }
            return best;

toggleSelection(name)

This method add the object name sent as an object selected.

        // Select a cube via the circle
        toggleSelection(name) {
            if (!this.removeFromArray(this.selection, name)) {
                this.addToSelection(name);
            }
        }

setSelection(name)

This method is used to reset the selection list and only have one object selected.

        // Set the selection to a specific cube
        setSelection(name) {
            this.selection = [];
            if (name) {
                this.selection.push(name);
                this.client.sendEventObject({ select: name });
            }
        }

isSelected(name)

This method is used to know if an object is selected.

        // Verify if the cube is selected
        isSelected(name) {
            return this.selection.indexOf(name) !== -1;
        }

addToSelection(name)

This method is used to add an object as a selection if the object is not selected.

        // Add the cube selected to the selection list
        addToSelection(name) {
            if (name && !this.isSelected(name)) {
                this.selection.push(name);
            }
        }

changeOffset(direction, event)

This method is used to add, reduce or reset the delay offset of the video. If the argument direction is equal to 0, the delay offset of the video is reset, otherwise it is mainly changed according to the value sent.

        // Function that changed the delay offset depending of the key pressed
        private changeOffset(direction, event) {
            if (direction !== 0) {
                let delayDelta = 100 * direction;
                if (event.altKey) delayDelta *= 10;
                if (event.shiftKey) delayDelta /= 2;
                let newDelayOffset = this.client.delayOffset + delayDelta;
                this.client.delayOffset = newDelayOffset;
            } else {
                this.client.delayOffset = 0;
            }
        }

toggleGenvidOverlay()

This method is used to display or hide the Genvid overlay. The Genvid overlay is used to display various stats about the stream.

  1. Raw Video: Time when the raw video is received.
  2. Est. Video: Estimated time when the video is received.
  3. Last Compose: Last compose time.
  4. Est. Compose: Estimated compose time.
  5. Stream: Time since the streaming has started.
  6. Latency: Latency occurring when watching the stream.
  7. DelayOffset: Current delay offset for the video.
        // Display or remove the Genvid Overlay
        private toggleGenvidOverlay() {
            if (this.genvidOverlay.getAttribute("data-isHidden")) {
                this.genvidOverlay.setAttribute("data-isHidden", "");
                this.genvidOverlay.style.visibility = "visible";
                this.genvidOverlayButton.classList.remove("disabled");
            } else {
                this.genvidOverlay.setAttribute("data-isHidden", "true");
                this.genvidOverlay.style.visibility = "hidden";
                this.genvidOverlayButton.classList.add("disabled");
            }
        }

onHelpActivation()

This method is used to display or hide the help overlay.

        // Display or remove the help overlay
        private onHelpActivation() {
            if (this.help_overlay.style.visibility === "visible") {
                this.help_overlay.style.visibility = "hidden";
            }
            else {
                this.help_overlay.style.visibility = "visible";
            }
        }

onCheer(cubeName: string)

This method is used when clicking on the button to cheer a player. We send an event to the client with the name of the object.

        // Upon cheering a player
        private onCheer(cubeName: string) {
            this.client.sendEventObject({ cheer: cubeName });
        }

onReset(cubeName: string)

This method is used when clicking on the button to reset a player. We send an event to the client with the name of the object.

        // Reset the position of the cube
        private onReset(cubeName: string) {
            this.client.sendEventObject({ reset: cubeName });
        }

onColorChange(cube: string, color: string)

This method is used when clicking on the button to change the color of a player. We send an event to the client with the name of the object and the color.

        // Method used when clicking on a color to change the color of a cube
        private onColorChange(cube: string, color: string) {
            let evt = {
                key: ["changeColor", cube],
                value: color,
            };
            this.client.sendEvent([evt]);
        }

onSelect(cubeName: string, selectionInterface: boolean)

This method is used when clicking on the player table displayed under the video stream. We reset the color for each table and apply the color to the selected table only, then we select the WebGL circle afterwards.

        // Select the cube from the interface
        private onSelect(cubeName: string, selectionInterface: boolean) {

            for (let nameSelect of this.cubeDiv) {
                nameSelect.style.backgroundColor = "#181818";
            }

            let cubeDiv = <HTMLDivElement>document.querySelector(".cube" + cubeName);
            this.lastSelection = cubeName;
            cubeDiv.style.backgroundColor = "#32324e";

            if (selectionInterface) {
                this.setSelection(cubeName);
            }
        }

Unity.ts - WebGL

This section is taking care of the WebGL creation process.

gfx_draw3D()

This function is called every frame to draw the WebGL content. The circles moving along the objects are created with WebGL.

  1. We get the WebGL context and perform a clear to delete the old frame content.
  2. We setup the texture properly before doing any draw commands.
            // Perform the 3d draw process
            let genvidWebGL = this.genvidWebGL;
            let gl = genvidWebGL.gl;

            genvidWebGL.clear();

            let prog = this.gfx_prog;
            gl.useProgram(prog);

            // We prepare the program only once (good thing we have a single material).
            gl.enableVertexAttribArray(0);
            gl.enableVertexAttribArray(1);
            gl.enableVertexAttribArray(2);
            gl.activeTexture(gl.TEXTURE0);
            gl.uniform1i(gl.getUniformLocation(prog, "tex"), 0);

            if (this.gfx_prog_data_viewproj) {
                gl.uniformMatrix4fv(this.gfx_prog_loc_viewproj, false, this.gfx_prog_data_viewproj);
            }
  1. Finally, we are performing a loop for each object and draw the circle along these objects.
            // Draw commands.
            let cmds: RenderCommand[] = [this.gfx_cmd_test, this.gfx_cmd_cubes];
            for (let cmd of cmds) {
                if (cmd.visible && cmd.vtx) {
                    gl.bindBuffer(gl.ARRAY_BUFFER, cmd.vtx[0]);
                    gl.vertexAttribPointer(0, 3, gl.FLOAT, false, 9 * 4, 0 * 4); // Position.
                    gl.vertexAttribPointer(1, 2, gl.FLOAT, false, 9 * 4, 3 * 4); // TexCoord.
                    gl.vertexAttribPointer(2, 4, gl.FLOAT, false, 9 * 4, 5 * 4); // Color.
                    gl.bindBuffer(gl.ELEMENT_ARRAY_BUFFER, cmd.idx[0]);
                    gl.bindTexture(gl.TEXTURE_2D, cmd.tex);
                    genvidWebGL.checkGLError();

                    gl.drawElements(gl.TRIANGLES, cmd.idx[1], gl.UNSIGNED_SHORT, 0);
                    genvidWebGL.checkGLError();
                }
            }

gfx_initShaders()

This function is used during the initialization process to initialize the shaders.

        // Web GL initalization of shaders
        gfx_initShaders() {
            let genvidWebGL = this.genvidWebGL;

            let vShaderStr = [
                "uniform mat4 g_ViewProjMat;",
                "attribute vec3 g_Position;",
                "attribute vec2 g_TexCoord0;",
                "attribute vec4 g_Color0;",
                "varying vec2 texCoord;",
                "varying vec4 color;",
                "void main()",
                "{",
                "   gl_Position = g_ViewProjMat * vec4(g_Position, 1.0);",
                "   texCoord = g_TexCoord0;",
                "   color = g_Color0;",
                "}"
            ].join("\n");
            let fShaderStr = [
                "precision mediump float;",
                "uniform sampler2D tex;",
                "varying vec2 texCoord;",
                "varying vec4 color;",
                "void main()",
                "{",
                "   vec4 texColor = texture2D(tex, texCoord);",
                "   gl_FragColor = texColor * color;", // Texture with color.
                // " gl_FragColor = color;", // Just the color.
                // " gl_FragColor = texColor;", // Just the texture.
                // " gl_FragColor = vec4(texCoord, 0.0, 1.0);", // The texture coordinate.
                // " gl_FragColor = vec4(fract(texCoord), 0.0, 1.0);", // Repeating texture coordinate.
                "}"
            ].join("\n");

            let vsh = genvidWebGL.loadVertexShader(vShaderStr);
            let fsh = genvidWebGL.loadFragmentShader(fShaderStr);
            this.gfx_prog = genvidWebGL.loadProgram(vsh, fsh, ["g_Position", "g_TexCoord0"]);
            this.gfx_prog_loc_viewproj = genvidWebGL.gl.getUniformLocation(this.gfx_prog, "g_ViewProjMat");
        }

gfx_initRenderCommands()

This function is used during the initialization process to initialize the render commands.

        // Web gl initialization of render command
        gfx_initRenderCommands() {
            let genvidWebGL = this.genvidWebGL;
            let gl = genvidWebGL.gl;

            // Utility function.
            function handleTextureLoaded(image, texture, options) {
                options = options || {};
                gl.bindTexture(gl.TEXTURE_2D, texture);
                gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, image);
                gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
                gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR_MIPMAP_LINEAR);
                if (options.wrap) {
                    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, options.wrap);
                    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, options.wrap);
                }
                if (options.aniso) {
                    let ext = (
                        gl.getExtension("EXT_texture_filter_anisotropic") ||
                        gl.getExtension("MOZ_EXT_texture_filter_anisotropic") ||
                        gl.getExtension("WEBKIT_EXT_texture_filter_anisotropic")
                    );
                    if (ext) {
                        let max = gl.getParameter(ext.MAX_TEXTURE_MAX_ANISOTROPY_EXT);
                        // console.log("Enabling aniso filtering", max);
                        gl.texParameterf(gl.TEXTURE_2D, ext.TEXTURE_MAX_ANISOTROPY_EXT, max);
                    }
                }
                gl.generateMipmap(gl.TEXTURE_2D);
                gl.bindTexture(gl.TEXTURE_2D, null);
            }

            let onload = this.gfx_draw3D.bind(this);

            {
                let vertices: number[] = [];
                this.makeCircleZ(vertices, 0, 0, 0, 0.5, genvidMath.vec4(1, 0, 0, 1));
                let num_quads = vertices.length / (4 * 9);

                let cmd = new RenderCommand();
                let options = {
                    "wrap": gl.CLAMP_TO_EDGE,
                    "aniso": true,
                };
                cmd.vtx = genvidWebGL.createBuffer(new Float32Array(vertices));
                cmd.idx = genvidWebGL.createIndexBufferForQuads(num_quads);
                cmd.tex = gl.createTexture();
                cmd.img = new Image();
                cmd.img.onload = function () { handleTextureLoaded(cmd.img, cmd.tex, options); if (onload) onload(); };
                cmd.img.src = "img/highlight_full_pa.png";
                cmd.visible = false;
                this.gfx_cmd_test = cmd;
            }

            // Cubes.
            {
                // Only prepare textures.
                let cmd = new RenderCommand();
                let options = {
                    "wrap": gl.CLAMP_TO_EDGE,
                    "aniso": true,
                };
                cmd.tex = gl.createTexture();
                cmd.img = new Image();
                cmd.img.onload = function () { handleTextureLoaded(cmd.img, cmd.tex, options); if (onload) onload(); };
                cmd.img.src = "img/highlight_full_pa.png";
                cmd.visible = true;
                this.gfx_cmd_cubes = cmd;
            }

            this.gfx_draw3D();
        }

Unity.ts - Utility methods

This section includes all the functions that are useful to the sample, but doesn’t involve any function from the Genvid API.

popularityToText(popularity)

This method is converting a popularity value to a text value.

        // Convert popularity value to popularity text
        popularityToText(popularity) {
            let hearths = ["💔", "♡", "♥", "💕"];
            let levels = [0.1, 1.0, 5.0];
            for (let i = 0; i < hearths.length; ++i) {
                if (popularity < levels[i]) {
                    return hearths[i];
                }
            }
            return hearths[levels.length];
        }

removeFromArray(arr, item)

This method is removing a value from an array sent as parameters.

        // Removes all instances of @item in @arr, returning true if the element was removed.
        removeFromArray(arr, item) {
            let removedSomething = false;
            for (let i = arr.length - 1; i >= 0; i--) {
                if (arr[i] === item) {
                    arr.splice(i, 1);
                    removedSomething = true;
                }
            }
            return removedSomething;
        }

makeCircleZ(dst: number[], x: number, y: number, z: number, r: number, c: genvidMath.IVec4)

This method is used to prepare a circle on the XY plane centered at coordinates x,y,z of radius r and color c. All these values are sent as parameters to the method. This is used to get the appropriate locations for each WebGL circle.

        // Utility routine preparing a circle on the XY-plane
        // centered at {x,y,z}, of radius r and color c.
        makeCircleZ(dst: number[], x: number, y: number, z: number, r: number, c: genvidMath.IVec4) {
            dst.push(
                // X   Y   Z     U    V      R    G    B    A
                x - r, y - r, z, 0.0, 0.0, c.x, c.y, c.z, c.w,
                x + r, y - r, z, 1.0, 0.0, c.x, c.y, c.z, c.w,
                x + r, y + r, z, 1.0, 1.0, c.x, c.y, c.z, c.w,
                x - r, y + r, z, 0.0, 1.0, c.x, c.y, c.z, c.w
            );
        }

projectWithRadius(mat: genvidMath.IMat4, pos_3d: genvidMath.IVec3, rad_3d: number): [genvidMath.IVec3, number]

This method is used to convert a radius 3d around a 3d position using the viewProjection matrix. This is used to get a hitbox for each circle when trying to click on it.

        // Converts a @rad_3d around a 3D position @pos_3d using the viewProjection matrix @mat.
        // Returns an array [pos_2d, rad_2d].
        projectWithRadius(mat: genvidMath.IMat4, pos_3d: genvidMath.IVec3, rad_3d: number): [genvidMath.IVec3, number] {
            // There might be a more mathematically sound solution to this,
            // but I've decided to use the shotgun approach and just project
            // 8 positions (add/sub radius to every dimension), and keep
            // the one which yields the largest 2D distance.
            let pos_2d = genvidMath.projectPosition(mat, pos_3d);
            let rad_sq_2d = 0;
            let offsets = [
                genvidMath.vec3(rad_3d, 0, 0),
                genvidMath.vec3(-rad_3d, 0, 0),
                genvidMath.vec3(0, rad_3d, 0),
                genvidMath.vec3(0, -rad_3d, 0),
                genvidMath.vec3(0, 0, rad_3d),
                genvidMath.vec3(0, 0, -rad_3d),
            ];
            for (let i = 0; i < offsets.length; ++i) {
                let off = offsets[i];
                let n_pos_3d = genvidMath.add3D(pos_3d, off);
                let n_pos_2d = genvidMath.projectPosition(mat, n_pos_3d);
                let n_rad_sq_2d = genvidMath.lengthSq2D(genvidMath.sub2D(pos_2d, n_pos_2d));
                rad_sq_2d = Math.max(rad_sq_2d, n_rad_sq_2d);
            }
            return [pos_2d, Math.sqrt(rad_sq_2d)];
        }

center_at(html_element: HTMLElement, pos_2d: genvidMath.IVec2, offset_pixels: genvidMath.IVec2)

This method is used to modify an html element position to be centered at the 2d position sent. This is used to move the name div element displayed above each object.

        // Change the HTML element position to be at the center of the pos 2d sent
        center_at(html_element: HTMLElement, pos_2d: genvidMath.IVec2, offset_pixels: genvidMath.IVec2) {

            // Convert from [-1, 1] range to [0, 1].
            let vh = genvidMath.vec2(0.5, 0.5);
            let pos_2d_n = genvidMath.mad2D(pos_2d, vh, vh);

            // Convert from [0, 1] range to [0, w].
            let p = html_element.parentElement;
            let p_size = genvidMath.vec2(p.clientWidth, p.clientHeight);
            let pos_in_parent = genvidMath.mul2D(pos_2d_n, p_size);

            // Adjust for centering element.
            let e_size = genvidMath.vec2(html_element.clientWidth, html_element.clientHeight);
            let e_offset = genvidMath.muls2D(e_size, -0.5);
            let pos_centered = genvidMath.add2D(pos_in_parent, e_offset);

            // Apply user offset.
            let pos_final = genvidMath.sub2D(pos_centered, offset_pixels);
            $(html_element).css({ left: pos_final.x, bottom: pos_final.y, position: "absolute", "z-index": "1100" });
        }

preN(str: string, n: number): string

This method is used to add empty spaces to a string. This is used to display properly the data on the Genvid overlay.

        // Widens a string to at least n characters, prefixing with spaces.
        private preN(str: string, n: number): string {
            let s: number = str.length;
            if (s < n) {
                str = " ".repeat(n - s) + str;
            }
            return str;
        }

convertMatrix(rawmat)

This method is used to convert an array of numbers into a genvidMath matrix 4x4. This is used when doing operation that needs to use the proper matrix format since the data sent from the game is received as an array of numbers.

        // Convert an array of 14 entry into a genvidMath Matrix 4x4
        convertMatrix(rawmat) {
            return genvidMath.mat4(genvidMath.vec4(rawmat.e00, rawmat.e01, rawmat.e02, rawmat.e02),
                genvidMath.vec4(rawmat.e10, rawmat.e11, rawmat.e12, rawmat.e13),
                genvidMath.vec4(rawmat.e20, rawmat.e21, rawmat.e22, rawmat.e23),
                genvidMath.vec4(rawmat.e30, rawmat.e31, rawmat.e32, rawmat.e33));
        }

findOrCreateTagDiv(cube: IGameDataCube): HTMLElement

This method is used to find or create the div that is displaying the name of the object that is moving along the object.

        // Find or create the div needed for the moving name on the overlay
        findOrCreateTagDiv(cube: IGameDataCube): HTMLElement {
            let elem_id: string = "cube_tag_" + cube.name;
            let elem: HTMLElement = <HTMLDivElement>document.querySelector("#" + elem_id);
            if (elem == null) {
                elem = document.createElement("div");
                elem.id = elem_id;
                elem.textContent = cube.name;
                elem.classList.add("tag");

                let parent: HTMLElement = <HTMLDivElement>document.querySelector("#video_overlay");
                parent.appendChild(elem);

                elem.addEventListener("click", (_event) => { this.setSelection(elem.textContent); this.onSelect(elem.textContent, false); }, false);
            }
            return elem;
        }

msToDuration(duration: number): string

This method is used to convert milliseconds into a specific time.

        // Method used to convert ms to specific duration
        private msToDuration(duration: number): string {
            let res = "";
            if (duration < 0) {
                res += "-";
                duration = -duration;
            }
            let second = 1000;
            let minute = second * 60;
            let hour = minute * 60;
            let day = hour * 24;
            let started = false;
            if (duration > day) {
                started = true;
                let rest = duration % day;
                let days = (duration - rest) / day;
                duration = rest;
                res += `${days}:`;
            }
            if (started || duration > hour) {
                started = true;
                let rest = duration % hour;
                let hours = (duration - rest) / hour;
                duration = rest;
                if (hours < 10) {
                    res += "0";
                }
                res += `${hours}:`;
            }
            if (started || duration > minute) {
                started = true;
                let rest = duration % minute;
                let minutes = (duration - rest) / minute;
                duration = rest;
                if (minutes < 10) {
                    res += "0";
                }
                res += `${minutes}:`;
            }
            if (started || duration > second) {
                let rest = duration % second;
                let seconds = (duration - rest) / second;
                duration = rest;
                if (seconds < 10) {
                    res += "0";
                }
                res += `${seconds}.`;
            } else {
                res += "0.";
            }
            if (duration < 100) {
                res += "0";
            }
            if (duration < 10) {
                res += "0";
            }
            return res + `${duration}`;
        }

checkFullScreen(): boolean

This method is used to return the fullscreen status.

        checkFullScreen(): boolean {
            let doc = <any>document;
            return doc.fullscreenElement || doc.webkitFullscreenElement || doc.mozFullScreenElement || doc.msFullscreenElement;
        }

Admin.ts

The admin page can be accessed by clicking on Admin on the index.html. After entering the username and password (admin for both), the User can interact with the game content by interacting with the player tables. In the document adminUnity.ts, we have all the code used to perform these interactions.

In the file, we have one interface and one class.

interface ICommandRequest

This interface is used when sending a command to the game.

    interface ICommandRequest {
        id: string;
        value: string;
    }

class AdminController

This class contains all the methods needed to connect to the stream, display the proper information and send the commands to the game.

Outside of the class, we create an instance of this class and we perform a onConnect() on it to start the connection to the stream.

let admin = new unitySample.AdminController("video_player_hidden");
admin.onConnect();

We will cover the content of the AdminController class in the following sections.

onConnect()

This method is starting the connection to the services. If the connection is executed properly, we proceed to the on_channel_join method with the genvid.IChannelJoinResponse information found. This process is done in exactly the same way in the unity.ts file.

        // Start the connection to the services
        onConnect() {
            let promise = $.post("/api/public/channels/join", {}, (joinRep) => {
                this.on_channel_join(<genvid.IChannelJoinResponse>joinRep);
            });
            promise.fail((err) => {
                alert("Can't get the stream info:" + err);
            });
        }

on_channel_join(info: genvid.IChannelJoinResponse)

This method creates the Genvid client after properly connecting to the services and channel. We use the information found during this process and the video_player_id sent during the class creation process to create the client (it is an argument to the AdminController class). This process is done in a similar way in the unity.ts file, though we do not need some methods in this case.

Afterwards, we need to associate specific events to functions in this class:

  1. onStreamsReceived: Trigger when the stream content is received (used to get the timecode and game data).
  2. onDraw: Trigger when drawing a new frame in the video.

We then proceed to start the client with the start() method.

        // Create the genvid Client and the function listening to it
        private on_channel_join(joinRep: genvid.IChannelJoinResponse) {
            this.streamInfo = joinRep.info;
            this.client = genvid.createGenvidClient(this.streamInfo, joinRep.uri, joinRep.token, this.video_player_id);
            this.client.onStreamsReceived((streams) => { this.on_streams_received(streams); });
            this.client.onDraw((frame) => { this.on_new_frame(frame); });
            this.client.start();
        }

on_streams_received(dataStreams: genvid.IDataStreams)

This method is called when receiving the stream. We use loops to get the latest time code associated with the game data. We are also converting the JSON data received into IGameData. This process is done exactly the same way in the unity.ts file.

        // Upon receving the stream
        private on_streams_received(dataStreams: genvid.IDataStreams) {
            for (let stream of dataStreams.streams) {
                for (let frame of stream.frames) {
                    if (this.last_game_time_received < frame.timeCode) {
                        this.last_game_time_received = frame.timeCode;
                    }
                }
            }

            // Parse the JSON from each elements
            for (let stream of [...dataStreams.streams, ...dataStreams.annotations]) {
                for (let frame of stream.frames) {
                    frame.user = JSON.parse(frame.data);
                }
            }
        }

on_new_frame(frameSource: genvid.IDataFrame)

This method is called every frame for the Genvid client that was created. We perform various tasks that need constant update in this method.

  1. We need to get the game data and verify that this data is valid before doing any operation with it.
  2. Once we are certain about the data validity and that the player table is not created, we proceed to get the object list from the data (cubes).
  3. We create a loop for each object and we create the player table for each.
  4. We also add the event listeners for clicking on the buttons of the player table.
  5. After the loop we create two buttons for the scene and camera change.
  6. We also add event listeners for clicking the scene and camera change buttons.
        // During a new frame, if the game data is valid, the player table is created
        private on_new_frame(frameSource: genvid.IDataFrame) {
            // Parse the JSON from each elements
            let gameDataFrame = frameSource.streams["GameData"];
            let gameData = null;

            if (gameDataFrame && gameDataFrame.user) {

                if (this.message === "Unable to retreive data from the stream, is it still active ?") {
                    this.message = "Success: data received from the stream";
                    this.displayMessage();
                }
                if (this.playerTableSetup === false) {
                    this.playerTableSetup = true;

                    gameData = gameDataFrame.user;
                    let cubes = gameData.cubes;

                    for (let cube of cubes) {
                        let cubeAdd = "<table>" +
                                "<tr>" +
                                    "<td id='table_name'>" + cube.name + "</td>" +
                                "</tr>" +
                                "<tr>" +
                                    "<td>" +
                                        "<div id='command_button' class='" + cube.name + "_upDirection'>↑</div>" +
                                        "<div id='command_button' class='" + cube.name + "_downDirection'>↓</div>" +
                                        "<div id='command_button' class='" + cube.name + "_leftDirection'>←</div>" +
                                        "<div id='command_button' class='" + cube.name + "_rightDirection'>→</div>" +
                                    "</td>" +
                                "</tr>" +
                                "<tr>" +
                                    "<td id='command_button'><div class='" + cube.name + "_reset'>Reset</div></td>" +
                                "</tr>" +
                                "<tr>" +
                                    "<td id='command_button'><div class='" + cube.name + "_slower'>Slower</div></td>" +
                                "</tr>" +
                                "<tr>" +
                                    "<td id='command_button'><div class='" + cube.name + "_faster'>Faster</div></td>" +
                                "</tr>" +
                            "</table>";

                        $(".admin_table_section").append(cubeAdd);
                        let upButton = <HTMLButtonElement>document.querySelector("." + cube.name + "_upDirection");
                        upButton.addEventListener("click", (_event) => { this.setDirection(cube.name, 0, 1); }, false);

                        let downButton = <HTMLButtonElement>document.querySelector("." + cube.name + "_downDirection");
                        downButton.addEventListener("click", (_event) => { this.setDirection(cube.name, 0, -1); }, false);

                        let leftButton = <HTMLButtonElement>document.querySelector("." + cube.name + "_leftDirection");
                        leftButton.addEventListener("click", (_event) => { this.setDirection(cube.name, -1, 0); }, false);

                        let rightButton = <HTMLButtonElement>document.querySelector("." + cube.name + "_rightDirection");
                        rightButton.addEventListener("click", (_event) => { this.setDirection(cube.name, 1, 0); }, false);

                        let resetButton = <HTMLButtonElement>document.querySelector("." + cube.name + "_reset");
                        resetButton.addEventListener("click", (_event) => { this.reset(cube.name); }, false);

                        let slowerButton = <HTMLButtonElement>document.querySelector("." + cube.name + "_slower");
                        slowerButton.addEventListener("click", (_event) => { this.changeSpeed(cube.name, 0.8); }, false);

                        let fasterButton = <HTMLButtonElement>document.querySelector("." + cube.name + "_faster");
                        fasterButton.addEventListener("click", (_event) => { this.changeSpeed(cube.name, 1.25); }, false);
                    }
                    let sceneCameraButtons = "<table>" +
                                "<tr>" +
                                    "<td id='command_button' class='cameraChange'>Camera change</td>" +
                                "</tr>" +
                                "<tr>" +
                                    "<td>" +
                                        "<div id='command_button' class='sceneChange'>Scene change</div>" +
                                    "</td>" +
                                "</tr>" +
                            "</table>";
                    $(".admin_table_section").append(sceneCameraButtons);

                    let cameraButton = <HTMLButtonElement>document.querySelector("." + "cameraChange");
                    cameraButton.addEventListener("click", (_event) => { this.changeCamera(); }, false);

                    let sceneButton = <HTMLButtonElement>document.querySelector("." + "sceneChange");
                    sceneButton.addEventListener("click", (_event) => { this.changeScene(); }, false);

                }
            }
            else {
                this.message = "Unable to retreive data from the stream, is it still active ?";
                this.displayErrorMessage();
            }
        }

setDirection(cubeName: string, x: number, z: number)

This method is called when clicking a direction for a player. We send a command to the game to indicate which player in which direction.

        // Change the direction of the cube
        setDirection(cubeName: string, x: number, z: number) {
            this.message = this.error = "";

            let command: ICommandRequest = {
                id: "direction",
                value: `${cubeName}:${x}:0:${z}`
            };

            let promise = $.post("/api/admin/commands/game", command).then(() => {
                this.message = `Set direction to ${cubeName}:${x}:${z}`;
                this.displayMessage();
            });

            promise.fail((err) => {
                this.message = `Failed with error ${err} to do Set direction to ${cubeName}:${x}:${z}`;
                this.displayErrorMessage();
            });
        }

changeSpeed(cubeName: string, factor: number)

This method is called when clicking on Slower or Faster for a player. We send a command to the game to indicate that the object needs to go faster or slower.

        // Change the speed of the cube
        changeSpeed(cubeName: string, factor: number) {
            this.message = this.error = "";

            let command: ICommandRequest = {
                id: "speed",
                value: `${cubeName}:${factor}`
            };

            let promise = $.post("/api/admin/commands/game", command).then(() => {
                this.message = `changeSpeed ${cubeName}:${factor}`;
                this.displayMessage();
            });

            promise.fail((err) => {
                this.message = `Failed with error ${err} to do changeSpeed ${cubeName}:${factor}`;
                this.displayErrorMessage();
            });
        }

reset(cubeName: string)

This method is called when clicking on Reset for a player. We send a command to the game to indicate that the object needs to be reset at its origin position.

        // Reset the position of the cube
        reset(cubeName: string) {
            this.message = this.error = "";

            let command: ICommandRequest = {
                id: "reset",
                value: cubeName
            };

            let promise = $.post("/api/admin/commands/game", command).then(() => {
                this.message = `reset ${cubeName}`;
                this.displayMessage();
            });

            promise.fail((err) => {
                this.message = `Failed with error ${err} to do reset ${cubeName}`;
                this.displayErrorMessage();
            });
        }

changeCamera()

This method is called when clicking on the Camera change button. We send a command to the game to indicate that the camera must be changed.

        changeCamera() {
            this.message = this.error = "";

            let command: ICommandRequest = {
                id: "camera",
                value: "change"
            };

            let promise = $.post("/api/admin/commands/game", command).then(() => {
                this.message = `camera change done`;
                this.displayMessage();
            });

            promise.fail((err) => {
                this.message = `Failed with error ${err} to change the camera`;
                this.displayErrorMessage();
            });
        }

changeScene()

This method is called when clicking on the Scene change button. We send a command to the game to indicate that the scene must be changed.

        changeScene() {
            this.message = this.error = "";

            let command: ICommandRequest = {
                id: "scene",
                value: "change"
            };

            let promise = $.post("/api/admin/commands/game", command).then(() => {
                this.message = `scene change done`;
                this.displayMessage();
            });

            promise.fail((err) => {
                this.message = `Failed with error ${err} to change the scene`;
                this.displayErrorMessage();
            });
        }

displayMessage()

This method is called when we need to display a successful message about an operation that was performed. It is useful to know if we get the data from the stream properly and to indicate if a command operation is successful.

        // Display a message in the page as a sucess
        displayMessage() {
                let messageErrorDiv = <HTMLDivElement>document.querySelector("#alert_error_cube");
                messageErrorDiv.style.visibility = "hidden";

                let messageDiv = <HTMLDivElement>document.querySelector("#alert_sucess_cube");
                messageDiv.style.visibility = "visible";
                let messageSpan = <HTMLSpanElement>document.querySelector("#sucess_message_cube");
                messageSpan.textContent = this.message;
        }

displayErrorMessage()

This method is called when we need to display an error message about an operation that was performed. It is useful to know if we get the data from the stream properly and to indicate if a command operation is successful.

        // Display a message in the page as an error
        displayErrorMessage() {
                let messageDiv = <HTMLDivElement>document.querySelector("#alert_sucess_cube");
                messageDiv.style.visibility = "hidden";

                let messageErrorDiv = <HTMLDivElement>document.querySelector("#alert_error_cube");
                messageErrorDiv.style.visibility = "visible";
                let messageSpan = <HTMLSpanElement>document.querySelector("#error_message_cube");
                messageSpan.textContent = this.message;
        }