Twitch Extension Code Reference¶

For the sample Twitch extension, we opted to write the client code exclusively in JavaScript to simplify the example. In this section, we describe the Twitch extension code and the code used to connect to it.

In This Section

Server Code
Twitch Extension Code
Video Overlay
Dashboard (Live Config) Panel
Configuration Page

Server Code ¶

The server responsible for connecting public URLs to internal Genvid Services is what the Twitch documentation refers to as the Extension Backend Service (EBS).

Just like the Tutorial Sample Website, we wrote the server code using Node.js. The main difference in this application is that it uses the cross-origin resource sharing (CORS) library.

app.use(cors({ credentials: true, origin: true }));

Like the web sample, the server forwards 3 methods:

/api/streams => GET /disco/stream/info

This method gets information about the current stream (name, description, channel, etc.).

/api/channels/join => POST /disco/stream/join

This method returns a URI and a token used to connect to the Genvid EBS.

/api/commands/game => POST /commands/game

This web method sends commands to the game.

In this tutorial, you can access the method from the live configuration panel of your Twitch Dashboard. The Genvid EBS protects the API calls by validating of the user/viewer JSON Web Token (JWT) sent to it.

The JWT is decoded using the clientId of the Twitch extension. Once validated and verified, the viewer/user becomes the broadcaster where the extension is installed.

The utils/auth.ts file contains the verification method.

/**
 * Validate Twitch Authentication
 * @param req 
 * @param res 
 * @param next 
 */
export function twitchAuth(req: express.Request, res: express.Response, next: express.NextFunction) {
    let token: string = <string>req.headers['x-access-token'] || <string>req.headers['authorization'];
    console.log('token', token);
    if (token.startsWith('Bearer ')) {
        // Remove Bearer from string
        token = token.slice(7, token.length);
    }

    if (token) {
        jwt.verify(token, Buffer.from(config.twitchExtConfig.secret, 'base64'), (err: any, decoded) => {
            if (err) {
                return res.status(401).send();
            } else {
                if (decoded.role === 'broadcaster') {
                    return next();
                } else {
                    return res.status(401).send();
                }
            }
        });
    } else {
        return res.status(401).send();
    }
}

Twitch Extension Code ¶

The Twitch Extension is a simple web application showing a live video of colored cubes. It is also compatible with multistreams, so you can try using picture-in-picture composition. (See the Genvid Studio sample section for more information.) There is a video overlay showing circles around the cubes. The positions of the circles match the position of the cubes in the video.

The extension code is also written exclusively in JavaScript.

Video Overlay ¶

The video overlay (video_overlay.html) extension renders on top of the video player as a transparent overlay. It’s only viewable when the channel is live.

The overlay inlcudes the controls that allow the user to interact with the stream. For the Tutorial Sample, those controls include the following:

Toolbar Buttons:

Help = Open or close the help menu.

Video info = Open or close the video info panel.

Cube name buttons = Open or close the cube control panel.

Video:

Click on cube = circle gets brighter

Cube panels:

Cheer = Change player popularity (heart icon).

Reset = Reset cube position.

Color = Change cube color.

The Genvid overlay prevents direct interaction with the Twitch player. It also displays the colored circles and names beside each cube along with useful debug information when you click on the Video Info button.

Local = Local time of the machine displaying the webpage.

Raw Video = Current internal time of the video player.

Est. Video = Estimated video-player time.

Last Compose = Time of the last data reported by the Compose service.

Est. Compose = Estimated time of the Compose service.

Stream = Current time of the generalized stream.

Latency = Estimated latency between the game and the spectators.

DelayOffset = Optional delay offset used for minute synchronization adjustments.

viewer.js¶

The front-end code in viewer.js performs several required functions:

Validates that you are an authorized Twitch user (twitch.ext.onAuthorized).

Retrieves the URL Endpoint of the Genvid EBS from the Twitch Configuration Service.

Makes a join request on /api/public/channels/join to retrieve the stream info and authorization token.

Instantiates a GenvidClient with the stream info.

/**
 * 1- Validate that viewer is authorized to use a twitch extension
 * 2- Retrieve the endpoint of the cluster from Twitch Configuration Service
 * 3- Initiate Genvid Services...
 */
twitch.onAuthorized((auth) => {
    // save our credentials
    token = auth.token;
    tuid = auth.userId;

    // Retrieve genvid endpoint for stream info
    let config = twitch.configuration.broadcaster;
    twitch.rig.log('This is your configuration...', config);
    let configuration = JSON.parse(config.content);

    twitch.rig.log('let initiate the genvid services...');

    if (configuration && configuration['web-endpoint']) {
        const endpoint = configuration['web-endpoint'];
        getStreamInfo(endpoint);
    } else {
        twitch.rig.log('no endpoint found');
    }

});

The IGenvidClient() communicates with the Genvid services and provides a few important callbacks.

onVideoPlayerReady() is called when the video is ready.

onAuthenticated() informs about the success of the authentication.

onStreamsReceived() is triggered when stream data is received.

onDraw() is called periodically to refresh the GUI.

/**
 * Callback called regurlary to draw the overlay,
 * in sync with the video frame displayed.
 * @param {*} frameSource
 */
const onDraw = frame => {

    // update the overlays to adapt to the composition of the video stream:
    updateOverlays(frame.compositionData);

    // Parse the JSON from each elements
    let gameDataFrame = frame.streams["GameData"];
    let gameData = null;

    if (gameDataFrame && gameDataFrame.user) {
        gameData = gameDataFrame.user;
        gameData.timeCode = frame.timeCode;
        // Modify the popularity with the latest value

        if (playerTableSetupCompletion === false) {
            playerTableSetupCompletion = true;
            initPlayerTable(gameData.cubes);
        }



        for (let cube of gameData.cubes) {
            if (latestPopularity) {
                let pCube = latestPopularity.find(c => c.name === cube.name);
                if (pCube) {
                    cube.popularity = pCube.popularity;
                }
            }
        }

        // update video info component

        updateInfo();
    }

    // Send the data to the overlay and controls
    drawFrame(gameData);
    /* if (onChannelDraw) {
       onChannelDraw(gameData);
    }*/

    // Log the annotations
    const colorChangedAnnotation = frame.annotations["ColorChanged"];
    if (colorChangedAnnotation) {
        for (let annotation of colorChangedAnnotation) {
            let colorChanges = annotation.user;
            for (let colorChange of colorChanges) {
                console.info(`The cube ${colorChange.name} changed color.`);
            }
        }
    }

    if (isFullScreen !== checkFullScreen()) {
        isFullScreen = checkFullScreen();
    }
}

In this sample, we use the onDisconnect() callback (exposed by the IGenvidClient() interface) to notify us when the client’s socket closes.

We begin by binding our desired callback.

genvidClient.onDisconnect(() => {onDisconnectDetected();});

Next we instruct the script to launch reconnection attempts when we’re notified the client socket closed.

/**
 * Method triggered when a disconnected socket occured  We then initiate
 * a client reconnect
 */
const onDisconnectDetected = () => {
    twitch.rig.log('got disconnected, reconnect...');
    const fetchOptions = {
        headers: {
            'Content-Type': 'application/x-www-form-urlencoded',
        },
        method: "POST"
    };
    fetch(endpoint, fetchOptions)
        .then(res => {
            genvidClient.reconnect(res.data.info, res.data.uri, res.data.token);
            resetFibNums();
        })
        .catch(error => __awaiter(this, void 0, void 0, function* () {
            yield sleep(getNewSleepDuration());
            onDisconnectDetected();
        }));

}

Then we launch an information request asking for a new leaf address, a new token, and new streamInfo.

If the request succeeds, we call reconnect() to establish a new websocket connection using the above information.

If the request fails (for example, if there is no leaf available), we wait and retry. We determine the waiting period using the result of an incremental Fibonacci operation.

/**
 * GENVID - Start sleep
 * @param {*} ms 
 */
const sleep = (ms) => {
    return new Promise(resolve => setTimeout(resolve, ms));
}

Note that we add a ten-percent randomness factor to the result of the Fibonacci sequence. This avoids cases where two services launch connection attempts simultaneously. We reset the Fibonacci values on a successful connection.

Overlay Methods¶

The Overlay Methods section is where we read the game data.

/**
 * GENVID - Start initOverlay
 * Method that will initialize all HTML component
 * for the overlay
 */
const initOverlay = () => {
    twitch.rig.log("Init tutorial overlay...");
    try {



        genvidOverlay = document.querySelector("#genvid_overlay");
        genvidOverlayButton = document.querySelector("#genvid_overlay_button");
        videoOverlay = document.querySelector("#video_overlay");
        canvas2d = document.querySelector("#canvas_overlay_2d");
        // canvas3d = document.querySelector("#canvas_overlay_3d");
        promptOverlay = document.querySelector("#prompt_overlay");
        helpButton = document.querySelector("#help_button");
        cubeName1Div = document.querySelector("#nameCube1");
        cubeName2Div = document.querySelector("#nameCube2");
        cubeName3Div = document.querySelector("#nameCube3");
        videoPlayerTimeDiv = document.querySelector("#videoPlayerTime");
        timeLocalDiv = document.querySelector("#time_local");
        timeVideoDiv = document.querySelector("#time_video");
        timeVideoRawDiv = document.querySelector("#time_video_raw");
        timeComposeDiv = document.querySelector("#time_compose");
        timeComposeLastDiv = document.querySelector("#time_compose_last");
        timeStreamDiv = document.querySelector("#time_stream");
        timeSessionDiv = document.querySelector("#time_session");
        latencyDiv = document.querySelector("#latency");
        delayOffsetDiv = document.querySelector("#delay_offset");
        pipFrameDiv = document.querySelector("#pip_frame");
        genvidWebGL = genvid.createWebGLContext(canvas3d); // Need to assign before any resize.



        document.addEventListener("fullscreenchange", () => onResize());
        document.addEventListener("webkitfullscreenchange", () => onResize());
        document.addEventListener("mozfullscreenchange", () => onResize());
        // playerElem.addEventListener("resize", () => onResize());
        window.addEventListener("resize", () => onResize(), true);
        window.addEventListener("orientationchange", () => onResize(), true);
        window.addEventListener("sizemodechange", () => onResize(), true);
        window.setInterval(() => onResize(), 1000); // Just a safety, in case something goes wrong.
        onResize();
        ctx = canvas2d.getContext("2d");
        if (!ctx) {
            console.log("No valid 2D context!");
        }
        mouseOverlay.addEventListener("mousemove", (event) => {
          onMouseMouve(event);
        }, true);
        mouseOverlay.addEventListener("click", (event) => {
          clickCube(event);
        }, false);
        /* genvidOverlayButton.addEventListener("click", (_event) => {
            toggleGenvidOverlay();
        }, false); */

        // Add listeners to buttons
        document.getElementById('toggle-help').addEventListener('click', (e) => {
            onToggle(e);
        }, true);

        document.getElementById('toggle-video-info').addEventListener('click', (e) => {
            onToggle(e);
        }, true);

        document.querySelector('.menu-controls').classList.remove('hidden');


        // Initialize graphics stuff.
        genvidWebGL.clear();
        let gl = genvidWebGL.gl;
        gl.disable(gl.DEPTH_TEST);
        initShaders();
        initRenderCommands();
    }
    catch (e) {
        console.log("Exception during initialization:", e);
    }
    console.log("tutorial overlay initialized");
}
// GENVID - Stop initOverlay

/**
 * GENVID - Start updateInfo
 * Method that will display the video information
 */
const updateInfo = () => {

    let w = 18; // Width of the content of every line (without label).
    let localTime = new Date();
    timeLocalDiv.textContent = `Local: ${msToDuration(Math.round(localTime.getTime()))}`;

    let videoTimeRawMS = 0;
    let videoPlayer = genvidClient.videoPlayer;
    if (videoPlayer) {
        videoTimeRawMS = videoPlayer.getCurrentTime() * 1000;
    }
    timeVideoRawDiv.textContent = `Raw Video: ${preN(msToDuration(Math.round(videoTimeRawMS)), w)}`;

    timeVideoDiv.textContent = `Est. Video: ${preN(msToDuration(Math.round(genvidClient.videoTimeMS)), w)}`;

    timeComposeLastDiv.textContent = `Last Compose: ${preN(msToDuration(Math.round(genvidClient.lastComposeTimeMS)), w)}`;

    timeComposeDiv.textContent = `Est. Compose: ${preN(msToDuration(Math.round(genvidClient.composeTimeMS)), w)}`;

    timeStreamDiv.textContent = `Stream: ${preN(msToDuration(Math.round(genvidClient.streamTimeMS)), w)}`;

    if (game_data) {
        timeSessionDiv.textContent = `Session: ${preN(msToDuration(Math.round(game_data.timeCode)), w)}`;
    } else {
        timeSessionDiv.textContent = `Session: <none>`;
    }

    latencyDiv.textContent = `Latency: ${preN(genvidClient.streamLatencyMS.toFixed(0), w - 3)} ms`;

    delayOffsetDiv.textContent = `DelayOffset: ${preN(genvidClient.delayOffset.toFixed(0), w - 3)} ms`;
}
// GENVID - Stop updateInfo

/**
 * GENVID - Start initPlayerTable
 * Init all control panels for cubes
 * @param {*} cubes 
 */
const initPlayerTable = (cubes) => {
    let gameControlDiv = document.getElementById("game-controls-div");

    cubes.forEach(cube => {

        let cubeAdd = `
                                        <div id='cube_`+ cube.name + `' class='cube clickable cube` + cube.name + ` overlay-panel cube-control-container hidden'>
                                            <div>
                                                <span class='cube_name clickable'>` + cube.name + `</span>
                                                <button class='cheer' id='` + cube.name + `_cheerbutton'>Cheer</button>
                                                <span class='cheer_value cube` + cube.name + `_cheer'></span>
                                            </div>
                                            <div>
                                                <span class='label clickable cube` + cube.name + `_reset'>Reset</span>
                                                <span id='cube` + cube.name + `_position_x' class='cube_position'></span>
                                                <span id='cube` + cube.name + `_position_y' class='cube_position'></span>
                                                <span id='cube` + cube.name + `_position_z' class='cube_position'></span>
                                            </div>
                                            <table class='cube_color text-center'>
                                                <tr>
                                                    <td class='clickable color` + cube.name + `_green'>Green</td>
                                                    <td class='clickable color` + cube.name + `_white'>White</td>
                                                    <td class='clickable color` + cube.name + `_yellow'>Yellow</td>
                                                </tr>
                                                <tr>
                                                    <td class='clickable color` + cube.name + `_darkblue'>Dark blue</td>
                                                    <td class='clickable color` + cube.name + `_grey'>Grey</td>
                                                    <td class='clickable color` + cube.name + `_lightblue''>Light Blue</td>
                                                </tr>
                                                <tr>
                                                    <td class='clickable color` + cube.name + `_orange'>Orange</td>
                                                    <td class='clickable color` + cube.name + `_blue'>Blue</td>
                                                    <td class='clickable color` + cube.name + `_purple'>Purple</td>
                                                </tr>
                                            </table>
                                        </div>
                                    `;

        gameControlDiv.innerHTML += cubeAdd;

    });

    cubes.forEach(cube => {
        let cheerButton = document.querySelector("#" + cube.name + "_cheerbutton");
        cheerButton.addEventListener("click", (_event) => { _event.stopPropagation(); onCheer(cube.name); }, false);
        cheerButton.addEventListener("dblclick", (e) => { e.stopPropagation(); }, false);

        let cubeDiv = document.querySelector(".cube" + cube.name);
        cubeDiv.addEventListener("click", (_event) => { onSelect(cube.name, true); }, false);
        cubeDivMain.push(cubeDiv);

        let resetButton = document.querySelector(".cube" + cube.name + "_reset");
        resetButton.addEventListener("click", (_event) => { onReset(cube.name); }, false);

        for (let colorSelect of tableColor) {
            let colorButton = document.querySelector(".color" + cube.name + "_" + colorSelect[0]);
            colorButton.addEventListener("click", (_event) => { onColorChange(cube.name, colorSelect[1]); }, false);
        }

        let toggleBtn = document.createElement('BUTTON');
        toggleBtn.setAttribute('type', 'button');
        toggleBtn.setAttribute('data-component', `cube_${cube.name}`);
        toggleBtn.classList.add('btn-default', 'clickable');
        toggleBtn.innerHTML = cube.name;
        toggleBtn.addEventListener('click', (e) => { onToggle(e); }, false);
        let menuItem = document.createElement('li');
        menuItem.appendChild(toggleBtn);
        document.querySelector('#menu-control-items').appendChild(menuItem);


    });
}
// GENVID - Stop initPlayerTable

/**
 * GENVID - Start updateUverlays
 * @param {*} compositionData 
 */
const updateOverlays = (compositionData) => {
    // when a picture in picture type composition is sent, the first element of
    // the composition data is the background frame and the second one is the picture in picture frame
    if (compositionData && compositionData.length > 1) {
        // composition picture in picture only: update the clipping on the 3d viewport
        // to prevent the 3d overlay to overlay the secondary screen
        pipFrameDiv.style.display = "block";
        // set up the main canvas to pip affine transform matrix
        const pipMat = genvidMath.mat3FromArray(compositionData[1].affineMatrix);
        updateDomRect(pipFrameDiv, videoOverlay, pipMat);
        updateDomClipping(overlay.canvas3d, pipMat);
    }
    else {
        pipFrameDiv.style.display = "hide";
        if (canvas3d)
            canvas3d.style.removeProperty("clip-path");
    }
}
// GENVID - Stop updateOverlays

/**
 * GENVID - Start updateDomRect
 * @param {*} targetDom 
 * @param {*} referenceDom 
 * @param {*} mat 
 */
const updateDomRect = (targetDom, referenceDom, mat) => {
    // get the canvas rect
    const domQuad = genvidMath.Path2.makeQuad(0, 0, 1, 1);
    const pipQuad = new genvidMath.Path2(domQuad.getPoints());
    pipQuad.transform(mat);
    pipQuad.scale($(referenceDom).width(), $(referenceDom).height());
    const bbox = pipQuad.getBoundingBox();
    targetDom.style.left = `${bbox.x}px`;
    targetDom.style.top = `${bbox.y}px`;
    targetDom.style.width = `${bbox.width}px`;
    targetDom.style.height = `${bbox.height}px`;
}
// GENVID - Stop updateDomRect

/**
 * GENVID - Start updateDomClipping
 * @param {*} dom 
 * @param {*} mat 
 */
const updateDomClipping = (dom, mat) => {
    // get the canvas rect
    const domQuad = genvidMath.Path2.makeQuad(0, 0, 1, 1);
    const pipQuad = new genvidMath.Path2(domQuad.getPoints());
    pipQuad.transform(mat);
    pipQuad.reverse();
    domQuad.append(pipQuad);
    domQuad.scale($(dom).width(), $(dom).height());
    dom.style.clipPath = domQuad.toCssPath();
}
// GENVID - Stop updateDomClipping

/**
 * GENVID - Start drawFrame
 * 
 * @param {*} gameData 
 */
const drawFrame = (gameData) => {
    if (!ctx) {
        return;
    }
    if (gameData) {
        game_data = gameData;
    }
    if (promptOverlay.style.visibility === "visible" && timeVisiblePrompt < timeVisibleMax) {
        timeVisiblePrompt++;
        if (volumeChange === 2) {
            volumeChange = 0;
            promptOverlay.textContent = "Volume: " + genvidClient.videoPlayer.getVolume().toString() + " %";
        }
        else if (volumeChange === 1) {
            volumeChange = 0;
            promptOverlay.textContent = "Volume: " + genvidClient.videoPlayer.getVolume().toString() + " %";
        }
    }
    else if (promptOverlay.style.visibility === "visible" && timeVisiblePrompt >= timeVisibleMax) {
        promptOverlay.style.visibility = "hidden";
    }
    update3D();
    ctx.clearRect(0, 0, canvas2d.width, canvas2d.height);
    draw3D();
    updateInfo();
}
// GENVID - drawFrame stop

WebGL Methods¶

The WebGL Methods section is where we use the game data to render 3D circles in a WebGL canvas.

/**
 * GENVID - Start draw3D
 * Function used to draw the WebGL 3d
 */
const draw3D = () => {
    // let genvidWebGL = this.genvidWebGL;
    let gl = genvidWebGL.gl;
    genvidWebGL.clear();
    let prog = gfx_prog;
    gl.useProgram(prog);
    // We prepare the program only once (good thing we have a single material).
    gl.enableVertexAttribArray(0);
    gl.enableVertexAttribArray(1);
    gl.enableVertexAttribArray(2);
    gl.activeTexture(gl.TEXTURE0);
    gl.uniform1i(gl.getUniformLocation(prog, "tex"), 0);
    if (gfx_prog_data_viewproj) {
        gl.uniformMatrix4fv(gfx_prog_loc_viewproj, false, gfx_prog_data_viewproj);
    }
    // Draw commands.
    let cmds = [gfx_cmd_test, gfx_cmd_cubes];
    for (let cmd of cmds) {
        if (cmd.visible && cmd.vtx) {
            gl.bindBuffer(gl.ARRAY_BUFFER, cmd.vtx[0]);
            gl.vertexAttribPointer(0, 3, gl.FLOAT, false, 9 * 4, 0 * 4); // Position.
            gl.vertexAttribPointer(1, 2, gl.FLOAT, false, 9 * 4, 3 * 4); // TexCoord.
            gl.vertexAttribPointer(2, 4, gl.FLOAT, false, 9 * 4, 5 * 4); // Color.
            gl.bindBuffer(gl.ELEMENT_ARRAY_BUFFER, cmd.idx[0]);
            gl.bindTexture(gl.TEXTURE_2D, cmd.tex);
            genvidWebGL.checkGLError();
            gl.drawElements(gl.TRIANGLES, cmd.idx[1], gl.UNSIGNED_SHORT, 0);
            genvidWebGL.checkGLError();
        }
    }
}
// GENVID - Stop draw3D

/**
 * GENVID - Start initShaders
 * Web GL initalization of shaders
 */
const initShaders = () => {
    // let genvidWebGL = this.genvidWebGL;
    let vShaderStr = [
        "uniform mat4 g_ViewProjMat;",
        "attribute vec3 g_Position;",
        "attribute vec2 g_TexCoord0;",
        "attribute vec4 g_Color0;",
        "varying vec2 texCoord;",
        "varying vec4 color;",
        "void main()",
        "{",
        "   gl_Position = g_ViewProjMat * vec4(g_Position, 1.0);",
        "   texCoord = g_TexCoord0;",
        "   color = g_Color0;",
        "}"
    ].join("\n");
    let fShaderStr = [
        "precision mediump float;",
        "uniform sampler2D tex;",
        "varying vec2 texCoord;",
        "varying vec4 color;",
        "void main()",
        "{",
        "   vec4 texColor = texture2D(tex, texCoord);",
        "   gl_FragColor = texColor * color;",
        // " gl_FragColor = color;", // Just the color.
        // " gl_FragColor = texColor;", // Just the texture.
        // " gl_FragColor = vec4(texCoord, 0.0, 1.0);", // The texture coordinate.
        // " gl_FragColor = vec4(fract(texCoord), 0.0, 1.0);", // Repeating texture coordinate.
        "}"
    ].join("\n");
    let vsh = genvidWebGL.loadVertexShader(vShaderStr);
    let fsh = genvidWebGL.loadFragmentShader(fShaderStr);
    gfx_prog = genvidWebGL.loadProgram(vsh, fsh, ["g_Position", "g_TexCoord0"]);
    gfx_prog_loc_viewproj = genvidWebGL.gl.getUniformLocation(gfx_prog, "g_ViewProjMat");
}
// GENVID - Stop initShaders

/**
 * GENVID - Start initRenderCommands
 * Web gl initialization of render command
 */
const initRenderCommands = () => {
    //  let genvidWebGL = this.genvidWebGL;
    let gl = genvidWebGL.gl;
    // Utility function.
    const handleTextureLoaded = (image, texture, options) => {
        options = options || {};
        gl.bindTexture(gl.TEXTURE_2D, texture);
        gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, image);
        gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
        gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR_MIPMAP_LINEAR);
        if (options.wrap) {
            gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, options.wrap);
            gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, options.wrap);
        }
        if (options.aniso) {
            let ext = (gl.getExtension("EXT_texture_filter_anisotropic") ||
                gl.getExtension("MOZ_EXT_texture_filter_anisotropic") ||
                gl.getExtension("WEBKIT_EXT_texture_filter_anisotropic"));
            if (ext) {
                let max = gl.getParameter(ext.MAX_TEXTURE_MAX_ANISOTROPY_EXT);
                // console.log("Enabling aniso filtering", max);
                gl.texParameterf(gl.TEXTURE_2D, ext.TEXTURE_MAX_ANISOTROPY_EXT, max);
            }
        }
        gl.generateMipmap(gl.TEXTURE_2D);
        gl.bindTexture(gl.TEXTURE_2D, null);
    }
        ;
    let onload = draw3D.bind(this);
    // Test (TODO: remove).
    {
        let vertices = [];
        makeCircleZ(vertices, 0, 0, 0, 0.5, genvidMath.vec4(1, 0, 0, 1));
        let num_quads = vertices.length / (4 * 9);
        let cmd = new RenderCommand();
        let options = {
            "wrap": gl.CLAMP_TO_EDGE,
            "aniso": true,
        };
        cmd.vtx = genvidWebGL.createBuffer(new Float32Array(vertices));
        cmd.idx = genvidWebGL.createIndexBufferForQuads(num_quads);
        cmd.tex = gl.createTexture();
        cmd.img = new Image();
        cmd.img.onload = () => {
            handleTextureLoaded(cmd.img, cmd.tex, options); if (onload)
                onload();
        };
        cmd.img.src = "img/highlight_full_pa.png";
        cmd.visible = false;
        gfx_cmd_test = cmd;
    }
    // Cubes.
    {
        // Only prepare textures.
        let cmd = new RenderCommand();
        let options = {
            "wrap": gl.CLAMP_TO_EDGE,
            "aniso": true,
        };
        cmd.tex = gl.createTexture();
        cmd.img = new Image();
        cmd.img.onload = () => {
            handleTextureLoaded(cmd.img, cmd.tex, options); if (onload)
                onload();
        };
        cmd.img.src = "img/highlight_full_pa.png";
        cmd.visible = true;
        gfx_cmd_cubes = cmd;
    }
    draw3D();
}
// GENVID - Stop initRenderCommands

/**
 * GENVID - Start update3D
 * Updates 3D canvas
 */
const update3D = () => {
    if (!game_data) {
        return;
    }
    let cubes = game_data.cubes;
    if (cubes) {
        let vertices = [];
        cubes.forEach(cube => {
            let m = cube.mat;
            let p = genvidMath.vec3(m[12], m[13], m[14]);
            let r = circleRadius;
            let c = genvidMath.vec4(1, 0, 0, 1);
            if (cube.color) {
                c = genvidMath.vec4(cube.color[0], cube.color[1], cube.color[2], cube.color[3]);
            }
            if (cube.selected && selection.length === 0) {
                // To gather initial state.
                setSelection(cube.name);
            }
            if (isSelected(cube.name)) {
                c = genvidMath.muls4D(c, 0.5);
            }
            makeCircleZ(vertices, p.x, p.y, p.z, r, c);
            let tag = findOrCreateTagDiv(cube);
            let mat = convertMatrix(game_data.MatProjView);
            let pos_2d = genvidMath.projectPosition(mat, p);
            center_at(tag, pos_2d, genvidMath.vec2(0, -75));


            cube.popText = popularityToText(cube.popularity);
            document.querySelector(".cube" + cube.name + "_cheer").textContent = cube.popText;
            document.querySelector("#cube" + cube.name + "_position_x").textContent = cube.mat[12].toFixed(2);
            document.querySelector("#cube" + cube.name + "_position_y").textContent = cube.mat[13].toFixed(2);
            document.querySelector("#cube" + cube.name + "_position_z").textContent = cube.mat[14].toFixed(2);
        })
        let num_quads = vertices.length / (4 * 9);
        // let genvidWebGL = genvidWebGL;
        let cmd = gfx_cmd_cubes;
        cmd.vtx = genvidWebGL.createBuffer(new Float32Array(vertices));
        cmd.idx = genvidWebGL.createIndexBufferForQuads(num_quads);
    }
    let mat = game_data.MatProjView;
    gfx_prog_data_viewproj = mat;
}
// GENVID - Stop update3D

User Interaction¶

The User Interaction section defines how users can interact with the overlay and generates events, such as changing a cube’s color, cheering for a cube, or resetting it’s position.

For example, the following code demonstrates how to change the color with an event.

/**
 * Method used when clicking on a color to change the color of a cube
 * @param {*} cube 
 * @param {*} color 
 */
const onColorChange = (cube, color) => {
    let evt = {
        key: ["changeColor", cube],
        value: color,
    };
    genvidClient.sendEvent([evt]);
}

Dashboard (Live Config) Panel ¶

The Live Configuration Panel (live_config.html) is how you access the Command Channel. It appears in the panel area on the dashboard page for the installed extension.

From your Twitch Account:

Select Creator Dashboard.
In the Extensions section, activate your panel.

The panel shows a section for each cube and includes the following functions:

Direction = Click on any arrow to change the cube direction.

Reset = Reset the position of the cube.

Slower = Reduce the speed of the cube.

Faster = Increase the speed of the cube.

live_config.js¶

The live_config.js file sends commands to the game via the command channel. This is how the broadcaster changes the speed and direction of the cubes.

Because the commands control the game directly, only the broadcaster should have access to the command API. With every request, we send the Bearer Token (JWT) of the user to our EBS to validate that the user is a valid Twitch user and also the broadcaster.

For example, the following code demonstrates how to change the direction or a cube.

/**
 * Change the direction of the cube
 * @param {*} cubeName 
 * @param {*} x 
 * @param {*} z 
 */
const setDirection = (cubeName, x, z) => {

    let command = {
        id: "direction",
        value: `${cubeName}:${x}:0:${z}`
    };
    twitch.rig.log("command", command);
    const fetchOptions = {
        method: "POST",
        headers: headers, // Contains the Bearer token
        body: JSON.stringify(command)
    };
    fetch(commandApiEndpoint, fetchOptions)
        .then(data => {
            console.log('data', data);
            message = `Set direction to ${cubeName}:${x}:${z}`;
            displayMessage();
        })
        .catch(err => {
            message = `Failed with error ${err} to do Set direction to ${cubeName}:${x}:${z}`;
            console.log('err', err);
            displayErrorMessage();
        });
}

The utils/auth.ts file performs validation for our Node.js server (EBS).

/**
 * Validate Twitch Authentication
 * @param req 
 * @param res 
 * @param next 
 */
export function twitchAuth(req: express.Request, res: express.Response, next: express.NextFunction) {
    let token: string = <string>req.headers['x-access-token'] || <string>req.headers['authorization'];
    console.log('token', token);
    if (token.startsWith('Bearer ')) {
        // Remove Bearer from string
        token = token.slice(7, token.length);
    }

    if (token) {
        jwt.verify(token, Buffer.from(config.twitchExtConfig.secret, 'base64'), (err: any, decoded) => {
            if (err) {
                return res.status(401).send();
            } else {
                if (decoded.role === 'broadcaster') {
                    return next();
                } else {
                    return res.status(401).send();
                }
            }
        });
    } else {
        return res.status(401).send();
    }
}

Configuration Page ¶

The Configuration Page lets you store persistent per-channel and per-extension data. It provides that data to your frontend on extension startup.

In our case, we use it to store the API Endpoint of our Node.js server (EBS).

config.js¶

The config.js file is where we retrieve form information and save it to the Twitch Configuration Service.

/* Quick and dirty binding form to twitch extension configuration */
const GvBindTwitchConfig = (form, twitchExtConfiguration) => {

    // On Submit, serialize form values into Twitch
    form.addEventListener("submit", function (e) {
        var configuration = GvSerializeForm(form);
        twitchExtConfiguration.set('broadcaster', '1.0', configuration);

        const msg = document.getElementById("message");
        if (msg.classList.contains('gv-displaynone')) {
            msg.classList.remove('gv-displaynone');
        }

        e.preventDefault();    //stop form from submitting
    });

    // On Load, deserialize Twitch configuration into form values
    var content = twitchExtConfiguration.broadcaster ? twitchExtConfiguration.broadcaster.content : {};
    GvDeserializeForm(form, content);
}

Related Topics

Table Of Contents

Twitch Extension Code Reference¶

Server Code ¶

Twitch Extension Code ¶

Video Overlay ¶

viewer.js¶

Overlay Methods¶

WebGL Methods¶

User Interaction¶

Dashboard (Live Config) Panel ¶

live_config.js¶

Configuration Page ¶

config.js¶

Twitch Extension Code Reference¶

Server Code¶

Twitch Extension Code¶

Video Overlay¶

viewer.js¶

Overlay Methods¶

WebGL Methods¶

User Interaction¶

Dashboard (Live Config) Panel¶

live_config.js¶

Configuration Page¶

config.js¶

Server Code ¶

Twitch Extension Code ¶

Video Overlay ¶

Dashboard (Live Config) Panel ¶

Configuration Page ¶