DirectX Sample Application

To use the Genvid MILE SDK, you have to integrate it in various sections of your application. This section shows how we integrate the SDK with the DirectX Sample using code samples and line numbers to help you navigate the DirectX application.

Note

Setting up and running the DirectX Cube Sample on a Local Environment will help you better understand the integration steps in this section. See How to Run the DirectX Cube Sample for more information.

Classes

directx.cpp

This file contains the window initialization process, the DirectX rendering, the game interaction controls, and all of the Genvid MILE SDK integration.

Cube.h

Header file for the cubes displayed in the game.

Cube.cpp

Class that contains all the transformation functions for the cubes.

DDSTextureLoader.h

Header file for loading a DDS texture and creating a Direct3D 11 runtime resource for it.

DDSTextureLoader.cpp

Class that contains all the functions for loading a DDS texture and creating a Direct3D 11 runtime resource for it.

JsonConversion.h

Header file for converting various data into JSON format to send via the stream.

JsonConversion.cpp

Class that converts various data types into JSON format to send via the stream.

Header and Declarations

We start by including the Genvid Native SDK header file and declaring a few static variables.

#include <genvid.h>

// AV streams
static const std::string sStream_Audio = "Audio";
static const std::string sStream_Video = "Video";

// game data streams
static const std::string sStream_Names   = "Names";
static const std::string sStream_Colors = "Colors";
static const std::string sStream_Position  = "Positions";
static const std::string sStream_Camera = "Camera";
static const std::string sStream_Copyright = "Copyright";

// events
static const std::string sEvent_changeColor = "changeColor";
static const std::string sEvent_reset       = "reset";
static const std::string sEvent_cheer       = "cheer";

// commands
static const std::string sCommand_speed     = "speed";
static const std::string sCommand_direction = "direction";
static const std::string sCommand_reset     = "reset";

static void GenvidSubscriptionCallback(const GenvidEventSummary * summary, void * userData);

static void GenvidSubscriptionCommandCallback(const GenvidCommandResult * summary, void * userData);

int64_t gCurrentTime    = 0;
int64_t gPrevTime       = 0;
int64_t gFirstFrameTime = 0;

system_clock::time_point gStartTime        = system_clock::now();
double                   gWorldTime        = 0.0;
float                    gGenvidFramerate  = 30.0f; // uses default value
float                    gGameFramerate = 30.0f; // uses default value
uint32_t                 gGamePresentRatio = 0;     // uses default value
bool                     gEnableVSync     = true;

const float gCameraFOV    = XM_PIDIV4;
float          gCameraAspect   = 16.0f / 9.0f;
const float gCameraNear  = 0.01f;
const float gCameraFar    = 100.0f;
const XMVECTOR gCameraPosition = XMVectorSet(0.0f, 3.0f, -6.0f, 0.0f);
const XMVECTOR gCameraLookAt   = XMVectorSet(0.0f, 0.0f, 0.0f, 0.0f);
const XMVECTOR gCameraUp       = XMVectorSet(0.0f, 1.0f, 0.0f, 0.0f);

bool gSilent = false; // Global flag to kill all sounds.

XMVECTORF32 gBackgroundColor        = Colors::Blue;
XMVECTORF32 gBackgroundColorArray[] = {Colors::Blue, Colors::Green, Colors::Red, Colors::Orange, Colors::Bisque, Colors::CadetBlue, Colors::Indigo, Colors::Yellow};

XMFLOAT4 gColors[] = {
    {0.6f, 0.1f, 0.4f, 1.0f},
    {0.2f, 0.9f, 0.1f, 1.0f},
    {0.7f, 0.7f, 0.7f, 1.0f},
    {0.9f, 0.9f, 0.1f, 1.0f},
    {0.1f, 0.2f, 0.3f, 1.0f},
    {0.3f, 0.3f, 0.3f, 1.0f},
    {0.2f, 0.4f, 0.6f, 1.0f},
    {0.5f, 0.3f, 0.2f, 1.0f},
    {0.1f, 0.5f, 0.9f, 1.0f},
    {0.4f, 0.2f, 0.4f, 1.0f},
};

std::map<std::string, XMFLOAT4> sNameToColor = {
    {"rose", gColors[0]},
    {"green", gColors[1]},
    {"white", gColors[2]},
    {"yellow", gColors[3]},
    {"dark blue", gColors[4]},
    {"gray", gColors[5]},
    {"light blue", gColors[6]},
    {"orange", gColors[7]},
    {"blue", gColors[8]},
    {"purple", gColors[9]},
};

Initialization

After setting up DirectX in InitDevice(), we call InitGenvid() which does the following:

  • Initializes the Genvid library.

  • Defines all of the streams that we will need.

  • Configures all the event and command subscriptions.

In this sample we have only one video stream and one audio stream.

We send the video IDXGISwapChain via Genvid_SetParameterPointer().

We create streams to send the game data to the servers: One for the popularity change and one for the the color change.

Finally, we create all the subscribes that we need to catch the events and commands coming from the webpage.

In this case, we catch 3 actions that can occur from the webpage:

  • A color change.

  • A position reset.

  • A popularity change.

We also catch 3 actions that can occur from the webpage as a command:

  • A speed change.

  • A direction change.

  • A position reset.

The function GenvidSubscriptionCallback() manages the event-catching portion which we display later. Because we send 3 types of events from the webpage, we need to call Genvid_Subscribe() 3 times with the proper names.

Note

The names we use while calling Genvid_Subscribe() must match the ones used on the webpage and in the map-reduce rules.

HRESULT InitGenvid()
{
    GenvidStatus gs;

    // Initialize the Genvid Native SDK.
    gs = Genvid_Initialize();
    if (GENVID_FAILED(gs))
        return E_FAIL;

    // Create the audio stream.
    gs = Genvid_CreateStream(sStream_Audio.c_str());
    if (GENVID_FAILED(gs))
        return E_FAIL;

    // Set the frame.
    gs = Genvid_SetParameterFloat(sStream_Audio.c_str(), "framerate", gGameFramerate);
    if (GENVID_FAILED(gs))
        return E_FAIL;

    // Specify auto-capture video source.
    // Note: enabling WASAPI sets the audio granularity to the sampling rate.
    gs = Genvid_SetParameterInt(sStream_Audio.c_str(), "Audio.Source.WASAPI", 1);
    if (GENVID_FAILED(gs))
        return E_FAIL;

    // Create the video stream.
    gs = Genvid_CreateStream(sStream_Video.c_str());
    if (GENVID_FAILED(gs))
        return E_FAIL;

    // Set the frame.
    gs = Genvid_SetParameterFloat(sStream_Video.c_str(), "framerate", gGameFramerate);
    if (GENVID_FAILED(gs))
        return E_FAIL;

    if (gGenvidFramerate > 0)
    {
        gs = Genvid_SetParameterFloat(sStream_Video.c_str(), "framerate", gGenvidFramerate);
        if (GENVID_FAILED(gs))
            return E_FAIL;
    }

    gs = Genvid_CreateStream(sStream_Copyright.c_str());
    if (GENVID_FAILED(gs))
        return E_FAIL;

    gs = Genvid_CreateStream(sStream_Names.c_str());
    if (GENVID_FAILED(gs))
        return E_FAIL;

    gs = Genvid_CreateStream(sStream_Colors.c_str());
    if (GENVID_FAILED(gs))
        return E_FAIL;

    gs = Genvid_CreateStream(sStream_Position.c_str());
    if (GENVID_FAILED(gs))
        return E_FAIL;

    gs = Genvid_CreateStream(sStream_Camera.c_str());
    if (GENVID_FAILED(gs))
        return E_FAIL;

#if GENVID_USE_DXGISWAPCHAIN
    // Specify auto-capture video source.
    gs = Genvid_SetParameterPointer(sStream_Video.c_str(), "Video.Source.IDXGISwapChain", g_pSwapChain);
    if (GENVID_FAILED(gs))
        return E_FAIL;
#endif

    // these frames are received one time only by each viewer when the Genvid client is connected
    // we can send data that never changes once, just after the stream is created.
    std::string copyright = "Copyright 2021-2023 Genvid Technologies LLC";
    gs                    = Genvid_SubmitGameData(-1, sStream_Copyright.c_str(), copyright.c_str(), (int)copyright.size());

    if (GENVID_FAILED(gs))
        return E_FAIL;

    auto cubeNames = GetCubeNames();
    gs             = Genvid_SubmitGameData(-1, sStream_Names.c_str(), cubeNames.c_str(), (int)cubeNames.size());
    if (GENVID_FAILED(gs))
        return E_FAIL;

    char buffer[512];
    auto size = GetCameraData(buffer);
    gs        = Genvid_SubmitGameData(-1, sStream_Camera.c_str(), buffer, size);
    if (GENVID_FAILED(gs))
        return E_FAIL;

    // Subscribe to events.
    gs = Genvid_Subscribe(sEvent_changeColor.c_str(), &GenvidSubscriptionCallback, nullptr);
    if (GENVID_FAILED(gs))
        return E_FAIL;
    gs = Genvid_Subscribe(sEvent_reset.c_str(), &GenvidSubscriptionCallback, nullptr);
    if (GENVID_FAILED(gs))
        return E_FAIL;
    gs = Genvid_Subscribe(sEvent_cheer.c_str(), &GenvidSubscriptionCallback, nullptr);
    if (GENVID_FAILED(gs))
        return E_FAIL;

    // Subscribe to commands.
    gs = Genvid_SubscribeCommand(sCommand_speed.c_str(), &GenvidSubscriptionCommandCallback, nullptr);
    if (GENVID_FAILED(gs))
        return E_FAIL;
    gs = Genvid_SubscribeCommand(sCommand_direction.c_str(), &GenvidSubscriptionCommandCallback, nullptr);
    if (GENVID_FAILED(gs))
        return E_FAIL;
    gs = Genvid_SubscribeCommand(sCommand_reset.c_str(), &GenvidSubscriptionCommandCallback, nullptr);
    if (GENVID_FAILED(gs))
        return E_FAIL;

    return S_OK;
}

Remember to implement your created events and commands in your website. You can find examples on the event generation and sendCommands pages. The web.js and web-admin.js files in our Web Sample also illustrate both in more detail.

Streaming

Right before presenting the back buffer, we auto-capture the video frame (using the previously specified IDXGISwapChain) and send it to Genvid.

Next, we gather the following game data:

  • Cube position.

  • Orientation.

  • Camera matrix.

Then we format the game data into JSON for the website.

We submit the game data with the same timecode as the video data using the utility routine GetGameDataJSON().

Then we gather the popularity data and verify if there were any changes. If there was a change, we perform a Genvid_SubmitNotification() with the updated popularity data formatted into JSON.

Finally, we call Genvid_CheckForEvents() to see if we received any events from the website.

GenvidStatus   gs = GenvidStatus_Success;
GenvidTimecode tc = Genvid_GetCurrentTimecode();

auto positionData = GetCubePositions();
gs                = Genvid_SubmitGameData(tc, sStream_Position.c_str(), positionData.data(), (int)positionData.size());
assert(!GENVID_FAILED(gs));

// Submit popularity every second
if (gTimeSincePopularitySent_s >= popularitySendInterval_s)
{
    gTimeSincePopularitySent_s = 0.0f;
    auto popularityData        = GetCubePopularity();
    gs                         = Genvid_SubmitNotification("POPULARITY", popularityData.c_str(), (int)popularityData.size());
    assert(!GENVID_FAILED(gs));
}

auto colorData = GetCubeColors();
gs             = Genvid_SubmitGameData(tc, sStream_Colors.c_str(), colorData.c_str(), (int)colorData.size());
assert(!GENVID_FAILED(gs));

gs = Genvid_CheckForEvents();
assert(!GENVID_FAILED(gs));

GenvidSubscriptionCallback

Calling Genvid_CheckForEvents() might invoke the function GenvidSubscriptionCallback(). This function catches the events sent from the webpage to the game. When invoked, we:

  1. Evaluate the summary ID value to know which event was sent, then

  2. We interpret the result key fields as parameters to the event.

//--------------------------------------------------------------------------------------
// Callback invoked when Genvid Events are received.
//--------------------------------------------------------------------------------------
void GenvidSubscriptionCallback(const GenvidEventSummary * summary, void * /*userData*/)
{
    std::cout << "color change received" << std::endl;
    if (summary->id == sEvent_changeColor)
    {
        // Change the color of the cube.
        const char * cubeName  = summary->results[0].key.fields[0];
        const char * cubeColor = summary->results[0].key.fields[1];
        // Finding the cube and change the color
        std::cout << "color change received " << cubeName << std::endl;
        for (int i = 0; i < g_cubeTotal; ++i)
        {
            if (g_cubeList[i].GetName() == cubeName)
            {
                std::map<std::string, XMFLOAT4>::iterator colorToFind = sNameToColor.find(cubeColor);
                if (colorToFind != sNameToColor.end())
                {
                    g_cubeList[i].SetColor(colorToFind->second);
                    std::cout << "color changed for " << cubeName << std::endl;
                }
            }
        }
        OnUpdateColor();
    }
    else if (summary->id == sEvent_reset)
    {
        // Reset the cube position.
        const char * cubeName = summary->results[0].key.fields[0];
        for (int i = 0; i < g_cubeTotal; ++i)
        {
            if (g_cubeList[i].GetName() == cubeName)
            {
                g_cubeList[i].ResetPosition();
                return;
            }
        }
    }
    else if (summary->id == sEvent_cheer)
    {
        // Handle cube cheering.
        const char * cubeName = summary->results[0].key.fields[0];
        for (int i = 0; i < g_cubeTotal; ++i)
        {
            if (g_cubeList[i].GetName() == cubeName)
            {
                // GENVID - Get cheer min value.
                g_cubeList[i].Cheer(float(summary->results[0].values[0].value));
                // GENVID - Cheer value set.
                return;
            }
        }
    }
}

The summary structure is consistent with the map-reduce definition that we provided in config/events.json.

{
  "version": "1.7.0",
  "event": {
    "game": {
      "maps": [
        {
          "id": "changeColor",
          "source": "userinput",
          "where": {"key": ["changeColor", "<name>"], "name": "<color>", "type": "string"},
          "key": ["changeColor", "<name>", "<color>"], "value": 1
        },
        {
          "id": "reset",
          "source": "userinput",
          "where": {"key": ["reset"], "name": "<name>", "type": "string"},
          "key": ["reset", "<name>"], "value": 1
        },
        {
          "id": "cheer",
          "source": "userinput",
          "where": {"key": ["cheer"], "name": "<name>", "type": "string"},
          "key": ["cheer", "<name>"], "value": 1
        }
      ],
      "reductions": [
        {
          "id": "changeColor",
          "where": {"key": ["changeColor", "<name>", "<color>"]},
          "key": ["<name>", "<color>"],
          "value": ["$count"],
          "period": 250
        },
        {
          "id": "reset",
          "where": {"key": ["reset", "<name>"]},
          "key": ["<name>"],
          "value": ["$count"],
          "period": 250
        },
        {
          "id": "cheer",
          "where": {"key": ["cheer", "<name>"]},
          "key": ["<name>"],
          "value": ["$sum"],
          "period": 250
        }
      ]
    }
  }
}

GenvidSubscriptionCommandCallback

Calling Genvid_CheckForEvents() might also invoke the function GenvidSubscriptionCommandCallback(). This is where commands are interpreted.

The command result value contains an ID and a value which we parse into their own strings to use in the method.

void GenvidSubscriptionCommandCallback(const GenvidCommandResult * result, void * /*userData*/)
{
    std::string id(result->id);
    std::string value(result->value);
    std::cout << "Command: " << result->id << " - " << result->value << std::endl;
    if (id == "direction")
    {
        std::vector<std::string> elems;
        split(value, ':', elems);
        auto        cubeName = elems[0];
        std::string payload;
        for (int i = 0; i < g_cubeTotal; ++i)
        {
            if (g_cubeList[i].GetName() == cubeName)
            {
                float x = (float)std::atof(elems[1].c_str());
                float y = (float)std::atof(elems[2].c_str());
                float z = (float)std::atof(elems[3].c_str());
                g_cubeList[i].ChangeDirection(x, y, z);
                payload = ",\"direction\":[" + std::to_string(x) + "," + std::to_string(y) + "," + std::to_string(z) + "]";
                break;
            }
        }
        payload = WrapNotificationPayload(payload, cubeName);
        auto gs = Genvid_SubmitNotification("DIRECTION", payload.c_str(), (int)payload.size());
        if (GENVID_FAILED(gs))
        {
            std::cout << "Fail to send notification." << std::endl;
        }
    }
    else if (id == "speed")
    {
        std::vector<std::string> elems;
        split(value, ':', elems);
        auto        cubeName = elems[0];
        std::string payload;
        for (int i = 0; i < g_cubeTotal; ++i)
        {
            if (g_cubeList[i].GetName() == cubeName)
            {
                float speed = (float)std::atof(elems[1].c_str());
                auto  velocity = g_cubeList[i].GetLinearVelocity() * speed;
                g_cubeList[i].SetLinearVelocity(velocity);
                payload = ",\"speed\":" + std::to_string(speed);
                break;
            }
        }
        payload = WrapNotificationPayload(payload, cubeName);
        auto gs = Genvid_SubmitNotification("SPEED", payload.c_str(), (int)payload.size());
        if (GENVID_FAILED(gs))
        {
            std::cout << "Fail to send notification." << std::endl;
        }
    }
    else if (id == "reset")
    {
        auto cubeName = value;
        for (int i = 0; i < g_cubeTotal; ++i)
        {
            if (g_cubeList[i].GetName() == cubeName)
            {
                auto velocity = g_cubeList[i].GetInitialLinearVelocity();
                g_cubeList[i].ResetPosition();
                g_cubeList[i].ResetOrientation();
                g_cubeList[i].SetLinearVelocity(velocity);
                break;
            }
        }
        std::string payload = WrapNotificationPayload("", cubeName);
        auto        gs      = Genvid_SubmitNotification("RESET", payload.c_str(), (int)payload.size());
        if (GENVID_FAILED(gs))
        {
            std::cout << "Fail to send notification." << std::endl;
        }
    }
}

Video Streaming

The sample demonstrates two different approaches for video streaming: automatic and manual. Auto-capture is the default setting.

To activate the capture, call the Genvid_SubmitVideoData() function with null parameters.

gs = Genvid_SubmitVideoData(tc, sStream_Video.c_str(), nullptr, 0);
assert(!GENVID_FAILED(gs));

Some situations may require using manual capture, such as if you’re not using DirectX11 or you don’t want to use the rendering thread to push the data to Genvid.

To use manual capture, you must set up the video frame using the correct parameters.

assert(description.Format == DXGI_FORMAT_R8G8B8A8_UNORM);
gs = Genvid_SetParameterInt(sStream_Video.c_str(), "video.pixel_format", GenvidPixelFormat_R8G8B8A8);
assert(gs == GenvidStatus_Success);
gs = Genvid_SetParameterInt(sStream_Video.c_str(), "video.width", description.Width);
assert(gs == GenvidStatus_Success);
gs = Genvid_SetParameterInt(sStream_Video.c_str(), "video.height", description.Height);
assert(gs == GenvidStatus_Success);

Then you can capture your buffer and submit it to Genvid.

// Figure out what we need on the first run.
if (g_pVideoBuffer == nullptr)
{
    // Hardcoded for DXGI_FORMAT_R8G8B8A8_UNORM
    g_NumBytes   = description.Height * description.Width * 4;
    g_pVideoBuffer = new uint8_t[g_NumBytes];
}
// On the following runs, check just in case.
else
{
    // Hardcoded for DXGI_FORMAT_R8G8B8A8_UNORM
    uint32_t TempNumBytes = description.Height * description.Width * 4;
    if (TempNumBytes != g_NumBytes)
    {
        delete[] g_pVideoBuffer;

        g_pVideoBuffer = new uint8_t[TempNumBytes];
        g_NumBytes   = TempNumBytes;
    }
}

uint8_t * pDestination   = g_pVideoBuffer;
uint8_t * pSource        = (uint8_t *)resource.pData;
uint32_t  NumSourceBytes = resource.RowPitch;
// Hardcoded for DXGI_FORMAT_R8G8B8A8_UNORM
uint32_t NumDestinationBytes = description.Width * 4;
uint32_t NumSourceLines      = description.Height;

for (uint32_t i = 0; i < NumSourceLines; ++i)
{
    memcpy(pDestination, pSource, NumDestinationBytes);
    pSource += NumSourceBytes;
    pDestination += NumDestinationBytes;
}

gs = Genvid_SubmitVideoData(tc, sStream_Video.c_str(), g_pVideoBuffer, g_NumBytes);
assert(!GENVID_FAILED(gs));

g_pImmediateContext->Unmap(pNewTexture, subresource);

Termination

In the TermGenvid() function, we perform every action necessary to properly clean up the Genvid MILE SDK.

We need to unsubscribe every stream, event, and command we originally subscribed to. Afterwards, we destroy all the streams and finish with a Genvid_Terminate().

void TermGenvid()
{
    // Cancel command subscriptions.
    Genvid_UnsubscribeCommand(sCommand_reset.c_str(), &GenvidSubscriptionCommandCallback, nullptr);
    Genvid_UnsubscribeCommand(sCommand_direction.c_str(), &GenvidSubscriptionCommandCallback, nullptr);
    Genvid_UnsubscribeCommand(sCommand_speed.c_str(), &GenvidSubscriptionCommandCallback, nullptr);

    // Cancel event subscriptions.
    Genvid_Unsubscribe(sEvent_cheer.c_str(), &GenvidSubscriptionCallback, nullptr);
    Genvid_Unsubscribe(sEvent_reset.c_str(), &GenvidSubscriptionCallback, nullptr);
    Genvid_Unsubscribe(sEvent_changeColor.c_str(), &GenvidSubscriptionCallback, nullptr);

    // Destroy the streams.
    Genvid_DestroyStream(sStream_Position.c_str());
    Genvid_DestroyStream(sStream_Copyright.c_str());
    Genvid_DestroyStream(sStream_Camera.c_str());
    Genvid_DestroyStream(sStream_Names.c_str());
    Genvid_DestroyStream(sStream_Colors.c_str());
    Genvid_DestroyStream(sStream_Video.c_str());
    Genvid_DestroyStream(sStream_Audio.c_str());

    // Terminate the Genvid Native SDK.
    Genvid_Terminate();
}

Other Files

demo.fx

File used for the shaders.

demo.rc

File used for the resources.

demo_ps.hlsl

File used for the pixel shaders.

demo_vs.hlsl

File used for the vertex shaders.

directx.ico

Icon used for the demo executable.

directx.sln

Visual Studio project solution file.

directx.vcxproj

Visual Studio project settings file.

directx.vcxproj.filters

Visual Studio project filters file.