Skip to main content

Candera Guide to Using GStreamer in CGI Studio

Audience: Developers and integrators working with Candera CGI Studio who want to decode and display video streams (e.g., H.264) via GStreamer inside their applications.

Goal: Install and configure GStreamer, enable streaming in CGI Studio projects, add the VideoStream control, diagnose common issues, and follow best practices for reliable playback.


Overview

CGI Studio integrates a sample implementation for GStreamer to show how to use the VideoStreamer interface. The VideoStreamer sample implementation uses GStreamer to decode the specified video source and provides frames to the Video Stream behavior.

This guide walks you through setup, configuration, and common diagnostics to ensure a smooth video playback experience within your CGI Studio projects and shows how to customize or add your own implementation.

Supported Platforms
  • Windows, Android, and Linux platforms with an installed GStreamer environment are supported.

  • Android support is limited to aarch64 targets.

  • Default modules support H.264 Baseline profile.

Typical Workflow (Quick Start)
  1. Install GStreamer and verify with gst-inspect-1.0.

  2. Ensure PATH includes GStreamer.

  3. Run CMake for host build and enable/set necessary flags for streaming.

  4. Build Player for Host (includes SCHost.dll for Scene Composer) and target.

  5. Add Video Stream control in the Scene Composer Solution and set VideoSource.

  6. Enable Playback in the statemachine or (optional) hook behaviors:

    • Configure and Play Stream to start playback.

    • Check Video State to loop or pause.

  7. Debug with GST_DEBUG and pipeline DOT graphs.


Prerequisites

GStreamer Installation & Verification

GStreamer must be installed and available on your system before running CGI Studio projects that use streaming. The GStreamer version provided with the CGI Studio 3rd‑party software (3PSW) is 1.16.1.

Verify installation using:
Bash
gst-inspect-1.0 --version
gst-inspect-1.0

You should see a list of plugins and elements if GStreamer is set up correctly.

Windows

Add GStreamer’s bin directory to your PATH:

PowerShell
# Temporary (Current Session)
$env:Path = "C:\gstreamer\1.0\x86_64\bin;" + $env:Path
gst-inspect-1.0

# Persistent (User Environment Variable)
[Environment]::SetEnvironmentVariable("Path", "C:\gstreamer\1.0\x86_64\bin;" + [Environment]::GetEnvironmentVariable("Path","User"), "User")

Android Evaluation Package

Check "Install GStreamer" during setup. Even if already installed, run Repair to ensure all dependencies are met. If CGI Player reports missing DLLs, ensure GStreamer’s bin directory is in the PATH, or copy the DLLs next to the executable.

Target (Linux)

On Linux targets (e.g., Raspberry Pi OS, STM32 MPU OpenSTLinux, or Torizon), use the package manager to install GStreamer and its plugins.

  • For hardware acceleration, refer to the manufacturer’s manuals.

  • For YOCTO or Buildroot, ensure GStreamer is included in the build. Note that specific plugins like aiurdemux may have restrictions but offer hardware acceleration.

For more information, visit: https://gstreamer.freedesktop.org/


How to Build the CGI Player with GStreamer

Setup for Microsoft Visual Studio
  1. Close Microsoft Visual Studio.

  2. Install GStreamer for Windows (Developer and Runtime):

  3. Manual File Copy (if directories are missing in cgi_studio_3psw):

    • Include files: Copy glib-2.0 and gstreamer-1.0 from C:/gstreamer/1.0/msvc_x86/include to cgi_studio_3psw/src/GStreamer/Win32/include.

    • Library folders: Copy glib-2.0 and gstreamer-1.0 from C:/gstreamer/1.0/msvc_x86/lib to cgi_studio_3psw/lib/GStreamer/Win32/lib.

    • Library files: Copy gio-2.0.lib, glib-2.0.lib, gobject-2.0.lib, and gstreamer-1.0.lib to the same destination folder.

CMake Flags
  1. Open CMake GUI.

  2. Set "Where is the source code:" to /cmake/Candera/Player.

  3. Set "Where to build the binaries:" to your build folder.

  4. Go to File -> Delete Cache to ensure a clean build.

  5. Click Configure, choose Win32 (or your target) for the generator.

  6. Adjust the following CMake defines:

    • COURIER_PLATFORM: Select platform (e.g., AndroidPlatform).

    • CGIAPP_SAMPLE_STREAMING_ENABLED: ON (Links GStreamer libraries).

    • CGIAPP_SAMPLE_MIXEDREALITY_ENABLED: ON (Enables AR-related behaviors/controls).

    • CGIAPP_SAMPLE_VIDEO_ROOT_PATH: Set the base folder for video files.

  7. Click Configure again, then click Generate.

  8. Open Player.sln in Visual Studio and select Build -> Build Solution.


VideoStream Control

The VideoStream control is part of the Augmented Reality Controls (AR Controls). It is utilized in the MixedReality sample to handle video streams.

Properties
Property Type Description
TargetNode Candera::AbstractNodePointer Node where the video will be displayed. If empty, the anchor node is used.
VideoSource FeatStd::String URI to the video (local path, RTSP, or asset).
ScaleToVideoSource bool If enabled, resizes texture to match video dimensions.
FlipVertically bool Flips video vertically (2D only).
Timestamp FeatStd::UInt32 Bind video timestamp to external reference for sync.
Using Video Stream in CGI Studio
  1. Add Control: Insert the VideoStream control into a 2D or 3D scene.

  2. Assign Target: Select the mesh or rectangle for the video texture.

  3. Set VideoSource (URI Formats):

    • Assets: asset://SampleKit_MixedReality##Resources#Videos#SampleVideo_Intro.mp4

    • Network (RTSP/HTTP): http://www.cgistudio.at/mixedreality_sample.mp4

    • Local File: Provide the file name (ensure CGIAPP_SAMPLE_VIDEO_ROOT_PATH is correct in CMake).

  4. Error Handling: Set a fallback image on the RenderNode (2D) or Appearance (3D) to be displayed if the stream fails.

  5. Start Playback:

    • State Machine: Add OnEntry -> Action: Configure and Play Stream.

    • Behaviors: Use Action: Configure and Play Stream and Condition: Check Video State.


The VideoStreamer Interface

The interface header can be found at:

/cgi_studio_controls/src/Behaviors/Streaming/VideoStreamer.h

GStreamer Sample Implementation

The Player contains a sample implementation in:

/cgi_studio_player/src/Player/Features/Streamer/GStreamer.cpp

  • Protocol Selection: Automatically chooses giostreamsrc (assets), udpsrc (UDP), or uridecodebin (HTTP/RTSP/Files).

  • Android: Plugins are treated as static and must be added manually (included in the sample). Currently, prebuilt binaries are only provided for aarch64.


Debugging GStreamer

Enable GStreamer Logs (GST_DEBUG)

Set GST_DEBUG (1-9) to see console output:

Bash
# Linux/macOS
export GST_DEBUG=3
./Player AssetFileCff.bin

# Windows (PowerShell)
$env:GST_DEBUG=3
.\Player.exe AssetFileCff.bin
Level Name Description
1 ERROR Fatal errors only.
2 WARNING Non-fatal, but visible problems.
3 FIXME Incomplete code paths triggered.
4 INFO Informational messages (rare events).
5 DEBUG General debug (object lifecycle, setup).
6 LOG Steady-state streaming events.
Dumping Pipeline Graphs

To visualize the pipeline, add the following to GStreamer.cpp in InitImpl():

C++
static bool init = false;
if (!init) {
    init = true;
    g_setenv("GST_DEBUG_DUMP_DOT_DIR", "/tmp", TRUE);
}

And to trigger the dump (e.g., in StopImpl()):

C++
gst_debug_bin_to_dot_file(GST_BIN(m_pipeline), GST_DEBUG_GRAPH_SHOW_ALL, "gstreamerpipe");

Convert the .dot file to SVG using Graphviz:

Bash
dot -Tsvg /tmp/gstreamerpipe.dot -o /tmp/gstreamerpipe.svg

Troubleshooting & Best Practices

Known Issues
  • Missing DLLs: Ensure GStreamer is in PATH.

  • Android Target: Only aarch64 is supported out-of-the-box.

  • iMX8 aiurdemux: May cause issues with giostreamsrc. Try using filesrc or fragmented MP4.

Best Practices
  • Use H.264 Baseline profile for maximum compatibility.

  • Start with moderate resolutions (e.g., 720p) before scaling up.

  • Leverage Hardware Acceleration (G2D compositor, VP DSP) by customizing the pipeline in GStreamer.cpp.

Version Compatibility
Component Version
GStreamer (3PSW) 1.16.1
GStreamer (CGI Studio 3.15.1) 1.22.7

References: