API Overview

🚧

It is assumed that you have read the Framework Concepts article, so if you haven't read it yet, I recommend you do.

MediaPipe API

⚠️ MediaPipe does not publish API documentation (AFAIK), so it is possible that the usage described in this wiki is incorrect.

CalculatorGraph

mediapipe/framework/calculator_graph.h

// The class representing a DAG of calculator nodes.
//
// CalculatorGraph is the primary API for the MediaPipe Framework.
// In general, CalculatorGraph should be used if the only thing you need
// to do is run the graph (without pushing data in or extracting it as
// the graph runs).

Initialization

The simplest way is to initialize an instance with a text representing CalculatorGraphConfig.

var configText = "your favorite config";
var calculatorGraph = new CalculatorGraph(configText);

The problem with this method is that it does not raise an error if the config format is invalid.
To validate the format, initialize CalculatorGraphConfig first and then initialize CalculatorGraph with it.

var configText = "your favorite config";
var config = CalculatorGraphConfig.Parser.ParseFromTextFormat(configText);
var calculatorGraph = new CalculatorGraph(config);

💡 With the latter method, you can manipulate CalculatorGraphConfig freely, so you can modify the config dynamically.

See CalculatorGraphConfig for more details.

Enable GPU Compute

When your target platform supports GPU compute, you can make use of it. See GPU Compute for more details.

var gpuResources = GpuResources.Create().Value();
calculatorGraph.SetGpuResources(gpuResources);

Before Running

You need to do some work before running the CalculatorGraph to receive outputs from it.

Initialize OutputStreamPoller

OutputStreamPoller provides a synchronous, polling API for accessing a stream's output.

// Suppose the name of the output stream is `out` and the output type is `string`.
var statusOrPoller = calculatorGraph.AddOutputStreamPoller<string>("out");

if (statusOrPoller.Ok())
{
  var poller = statusOrPoller.Value();
}

OutputStreamPoller won't return values by default if there's no output (e.g. faces are not detected in the input image).
Because OutputStreamPoller#Next will block the thread until it gets a new value, this behavior can cause the entire application to hang.

To get an empty packet when there's no output, set observeTimestampBounds (the 2nd argument) to true.

var statusOrPoller = calculatorGraph.AddOutputStreamPoller("out", true);

Register NativePacketCallback

You can also add listeners, which will be invoked on every packet emitted by the output stream.
Note that those listeners will be called from a thread other than the main thread, so you cannot call most of the Unity APIs in them.

// Suppose the name of the output stream is `out` and the output type is `string`.
calculatorGraph.ObserveOutputStream("out", Foo.Callback).AssertOk();

class Foo
{
  // NOTE: To support IL2CPP, NativePacketCallback must be static.
  [AOT.MonoPInvokeCallback(typeof(CalculatorGraph.NativePacketCallback))]
  private static IntPtr Callback(IntPtr graphPtr, int streamId, IntPtr packetPtr)
  {
    using (var packet = new StringPacket(packetPtr, false)) // `packetPtr` is a reference to the output packet
    {
      // ...
    }    
  }
}

As with CalculatorGraph#AddOutputStreamPoller, to emit an empty packet when there's no output, set observeTimestampBounds to true.

calculatorGraph.ObserveOutputStream("out", Foo.Callback, true).AssertOk();

Start Running

calculatorGraph.StartRun().AssertOk();

If your graph receives SidePacket, you can set it here.

var sidePacket = new SidePacket();

sidePacket.Emplace("num_faces", new IntPacket(2));
sidePacket.Emplace("with_attention", new BoolPacket(true));

calculatorGraph.StartRun(sidePacket).AssertOk();

During Running

Send input packets

var timestamp = new Timestamp(0);

// Suppose the name of the input stream is `in` and the input type is `string`.
calculatorGraph.AddPacketToInputStream("in", new StringPacket("Hello World!", timestamp)).AssertOk();

See Timestamp to know how to set up Timestamp.

Fetch Output Packets

To get output packets from OutputStreamPoller, you need to call OutputStreamPoller#Next explicitly.

// NOTE: this packet can be reused in a loop
var packet = new StringPacket();

if (poller.Next(packet)) // `OutputStreamPoller#Next` will block the thread
{
  if (!packet.IsEmpty()) // if `observeTimestampBounds` is set to `true`, output packets can be empty
  {
    var value = packet.Get();
    // ...
  }
}

After Running

If your work is done, dispose of CalculatorGraph to free up resources.

First, close the input source.

// Suppose the name of the input stream is `in`.
calculatorGraph.CloseInputStream("in").AssertOk();

If there are several input sources, you may want to close them at once.

calculatorGraph.CloseAllPacketSources().AssertOk();

After that, stop CalculatorGraph.

calculatorGraph.WaitUntilDone().AssertOk();
calculatorGraph.Dispose();

CalculatorGraphConfig

mediapipe/framework/calculator.proto

// Describes the topology and function of a MediaPipe Graph.  The graph of
// Nodes must be a Directed Acyclic Graph (DAG) except as annotated by
// "back_edge" in InputStreamInfo.  Use a mediapipe::CalculatorGraph object to
// run the graph.

This class represents the configuration of a CalculatorGraph and can be used to initialize the graph, but it is not necessary since CalculatorGraph can be also initialized with an equivalent string.

However, CalculatorGraphConfig has 2 advantages over the string representations.

By converting the string to CalculatorGraphConfig, the configuration can be validated before running the CalculatorGraph, which makes debugging easier (cf. Logging).
You can modify the config easily at runtime (cf. MediaPipeVideoGraph.cs).

Initialization

var configTxt = "your favorite config"; // this must not be null
var config = CalculatorGraphConfig.ParseFromTextFormat(configTxt); // throws if the format is invalid

ValidatedGraphConfig

CalculatorGraphConfig.ParseFromTextFormat can be used to validate the format, but it doesn't validate the config itself (e.g. it can parse the config if some calculators don't exist).
To validate the config, you can use the ValidatedGraphConfig API.

var config = CalculatorGraphConfig.ParseFromTextFormat("your favorite config");

using (var validatedGraphConfig = new ValidatedGraphConfig())
{
  var status = validatedGraphConfig.Initialize(config);

  status.AssertOk(); // throws if the config is not valid
}

The beauty of ValidatedGraphConfig is that it canonicalizes the CalculatorGraphConfig, expanding all the subgraphs.
That is, it enables us to access all the nodes of the graph at runtime.

The following code shows how to modify CalculatorOptions of TensorsToDetectionsCalculator (cf. FaceDetectionGraph.cs).

using System.Linq;
using Google.Protobuf;

var config = CalculatorGraphConfig.ParseFromTextFormat("your favorite config");

using (var validatedGraphConfig = new ValidatedGraphConfig())
{
  validatedGraphConfig.Initialize(config).AssertOk();

  // NOTE: Calculator#Options is an [Extension](https://developers.google.com/protocol-buffers/docs/proto#extensions)
  //       To parse it, we need to initialize an `ExtensionRegistry`.
  var extensionRegistry = new ExtensionRegistry() { TensorsToDetectionsCalculatorOptions.Extensions.Ext };
  var canonicalizedConfig = validatedGraphConfig.Config(extensionRegistry);

  var tensorsToDetectionsCalculators = cannonicalizedConfig.Node.Where((node) => node.Calculator == "TensorsToDetectionsCalculator");

  foreach (var calculator in tensorsToDetectionsCalculators)
  {
    var options = calculator.Options.GetExtension(TensorsToDetectionsCalculatorOptions.Extensions.Ext);
    options.MinScoreThresh = 0.1; // modify `MinScoreThresh` at runtime
  }
}

ImageFrame

Status

Many APIs return Status, which is a wrapper of absl::Status.

absl/status/status.h

// absl::Status
//
// The `absl::Status` class is generally used to gracefully handle errors
// across API boundaries (and in particular across RPC boundaries). Some of
// these errors may be recoverable, but others may not. Most
// functions which can produce a recoverable error should be designed to return
// either an `absl::Status` (or the similar `absl::StatusOr<T>`, which holds
// either an object of type `T` or an error).

Initialization

// When it's successful
var okStatus = Status.Ok();

// When an error occurs
var errorStatus = Status.FailedPrecondition("the reason here");

Validation

status.AssertOk(); // throws if it's not OK

if (status.Ok()) // this line won't throw
{
  // do something
}
else
{
  Debug.Log(status.ToString()); // log the error message
}

StatusOr

StatusOr<T> is similar to Status.

absl/status/statusor.h

// absl::StatusOr<T>
//
// The `absl::StatusOr<T>` class template is a union of an `absl::Status` object
// and an object of type `T`. The `absl::StatusOr<T>` models an object that is
// either a usable object, or an error (of type `absl::Status`) explaining why
// such an object is not present. An `absl::StatusOr<T>` is typically the return
// value of a function which may fail.

Initialization

There is no case where you need to initialize StatusOr<T> instances directly.

Validation

// Some APIs return a `StatusOr<T>` instance.
// NOTE: `StatusOrGpuResources` is `StatusOr<GpuResources>`
StatusOrGpuResources statusOrGpuResources = GpuResources.Create();

// `StatusOr<T>#status` returns the internal `Status`.
statusOrGpuResources.status.AssertOk();

// `StatusOr<T>#Ok` returns true iff it's OK
if (statusOrGpuResources.Ok())
{
  // `StatusOr<T>#Value` returns the internal value if it's OK.
  var value = statusOrGpuResources.Value();
}

Packet

Timestamp

Each Packet has its Timestamp.
When sending input packets to MediaPipe, the correct timestamp must be set for each packet.

mediapipe/framework/timestamp.h

// A class which represents a timestamp in the calculator framework.
// There are several special values which can only be created with the
// static functions provided in this class.

Initialization

There are 2 things that must be observed

The underlying value is in microseconds.
The timestamp value of the new packet must be greater than the previous packets.

It is usually a good idea to use elapsed microseconds since the start as the timestamp value.

var stopwatch = new System.Diagnostics.Stopwatch();
stopwatch.Start();

var currentTimestampMicrosec = stopwatch.ElapsedTicks / (TimeSpan.TicksPerMillisecond / 1000);
var timestamp = new Timestamp(currentTimestampMicrosec);

Get its value

When working with output packets, you may want to know their timestamps.

var microsec = timestamp.Microseconds(); // Get the value in microseconds.

ResourceManager

Protocol Buffers

GPU compute

If GPU compute is supported on your target platform, you can enable it. To make use of GPU, you need to initialize GpuResources and set it to your CalculatorGraph.

var gpuResources = GpuResources.Create().Value();
calculatorGraph.SetGpuResources(gpuResources);

// `SetGpuResources` must be called before `StartRun`.
calculatorGraph.StartRun();

GpuResources

You can initialize GpuResources using GpuResources.Create.
See also mediapipe/gpu/gpu_shared_data_internal.h.

var statusOrGpuResources = GpuResources.Create();
statusOrGpuResources.status.AssertOk(); // throws if GPU computing is not supported.

var gpuResources = statusOrGpuResources.Value();

When the Graphics API is OpenGL ES, you can share the context with MediaPipe (cf. https://google.github.io/mediapipe/framework_concepts/gpu.html#opengl-es-support).

// NOTE: The following code is a bit hackish. If you know a better way, please let us know!

using Mediapipe;
using System.Collections;
using UnityEngine;

class GpuInitializer
{
  private static IntPtr _CurrentContext = IntPtr.Zero;
  private static bool _IsContextInitialized = false;

  private delegate void PluginCallback(int eventId);

  [AOT.MonoPInvokeCallback(typeof(PluginCallback))]
  private static void GetCurrentContext(int eventId) {
    _CurrentContext = Egl.GetCurrentContext(); // This API is ported by this plugin.
    _IsContextInitialized = true;
  }

  public IEnumerator Initialize()
  {
    // You need to get the current context first.
    PluginCallback callback = GetCurrentContext;

    var fp = Marshal.GetFunctionPointerForDelegate(callback);
    GL.IssuePluginEvent(fp, 1);

    yield return new WaitUntil(() => _IsContextInitialized);

    // Call `GpuResources.Create` with the current context.
    var statusOrGpuResources = GpuResources.Create(_CurrentContext);
    // ...
  }
}

GlCalculatorHelper

mediapipe/gpu/gl_calculator_helper.h

// Helper class that manages OpenGL contexts and operations.
// Calculators that implement an image filter, taking one input stream of
// frames and producing one output stream of frame, should subclass
// GlSimpleCalculatorBase instead of using GlCalculatorHelper directly.
// Direct use of this class is recommended for calculators that do not fit
// that mold (e.g. calculators that combine two video streams).

This class is useful when you'd like to manipulate a GpuBuffer instance.

Initialization

var gpuResources = GpuResources.Create().Value();
var glCalculatorHelper = new GlCalculatorHelper();
glCalculatorHelper.InitializeForTest(gpuResources);

Examples

Convert ImageFrame to GpuBuffer and send it to your CalculatorGraph[^1].

cf. mediapipe/examples/desktop/demo_run_graph_main_gpu.cc.

var timestamp = new Timestamp(0);
glCalculatorHelper.RunInGlContext(() => {
  var texture = glCalculatorHelper.CreateSourceTexture(imageFrame);
  var gpuBuffer = texture.GetGpuBufferFrame();

  Gl.Flush();
  texture.Release();

  return calculatorGraph.AddPacketToInputStream("in", new GpuBufferPacket(gpuBuffer, timestamp));
});

Build a GpuBuffer instance from a Texture.

See GpuBuffer(#gpubuffer).

[^1]: You should usually use the ImageFrameToGpuBufferCaclulator to do this.

GpuBuffer

mediapipe/gpu/gpu_buffer.h

// This class wraps a platform-specific buffer of GPU data.
// An instance of GpuBuffer acts as an opaque reference to the underlying
// data object.

In most cases, you don't need to use GpuBuffer APIs on Unity, but when Unity shares its OpenGL ES context with MediaPipe, you may want to use them for performance (see also GpuResources).

// Texture texture = your_source_texture;

var glTextureName = (uint)texture.GetNativeTexturePtr();
var glBufferFormat = GpuBufferFormat.kBGRA32; // BGRA32 is the only supported format currently.
var glContext = glCalculatorHelper.GetGlContext();
var glTextureBuffer = new GlTextureBuffer(glTextureName, texture.width, texture.height,
                                          glBufferFormat, Foo.OnRelease, glContext);
var gpuBuffer = new GpuBuffer(glTextureBuffer);

class Foo
{
  // NOTE: To support IL2CPP, DeletionCallback must be static.
  [AOT.MonoPInvokeCallback(typeof(GlTextureBuffer.DeletionCallback))]
  private static void OnRelease(uint textureName, IntPtr syncTokenPtr)
  {
    if (syncTokenPtr == IntPtr.Zero)
    {
      return;
    }
    using (var glSyncToken = new GlSyncPoint(syncTokenPtr))
    {
      glSyncToken.Wait();
    }
  }
}

GlSyncPoint

mediapipe/gpu/gl_context.h

// Generic interface for synchronizing access to a shared resource from a
// different context. This is an abstract class to keep users from
// depending on its contents. The implementation may differ depending on
// the capabilities of the GL context.

This class is rarely used, but if you'd like to initialize a GpuBuffer instance, you need to know it because GlSyncPoint is passed to the GlTextureBuffer.DeletionCallback as the 2nd argument (IntPtr).
See also GpuBuffer.

// NOTE: To support IL2CPP, DeletionCallback must be static.
[AOT.MonoPInvokeCallback(typeof(GlTextureBuffer.DeletionCallback))]
private static void OnRelease(uint textureName, IntPtr syncTokenPtr)
{
  if (syncTokenPtr == IntPtr.Zero)
  {
    return;
  }
  using (var glSyncToken = new GlSyncToken(syncTokenPtr))
  {
    // Waits until the GPU has executed all commands up to the sync point.
    // This blocks the CPU, and ensures the commands are complete from the
    // point of view of all threads and contexts.
    glSyncToken.Wait();

    // Ensures that the following commands on the current OpenGL context will
    // not be executed until the sync point has been reached.
    // This does not block the CPU, and only affects the current OpenGL context.
    glSyncToken.WaitOnGpu();

    // Returns whether the sync point has been reached. Does not block.
    if (glSyncToken.isReady())
    {
      // ...
    }
  }
}

Logging

Glog

MediaPipe uses Google Logging Library internally and this plugin ports APIs to configure it.
See also glog#setting-flags.

Glog.Logtostderr = true; // Log messages to stderr (i.e. Editor.log, Player.log) instead of log files.
Glog.MinLogLevel = 0; // default = 0
Glog.V = 0; // default = 0

Glog.Initialize("MediaPipeUnityPlugin"); // call `Initialize` after setting flags

Protobuf

Protobuf will log messages to stdout or stderr by default.
You can overwrite the google::protobuf::LogHandler so that the output appears in the Console Window.

Protobuf.SetLogHandler(Protobuf.DefaultLogHandler);

// Restore the default LogHandler before the program exits.
// Otherwise, the protobuf will retain the stale pointer, which may eventually cause SIGSEGV.
Protobuf.ResetLogHandler();

API Overview

MediaPipe API

CalculatorGraph

Initialization

Enable GPU Compute

Before Running

Initialize OutputStreamPoller

Register NativePacketCallback

Start Running

During Running

Send input packets

Fetch Output Packets

After Running

CalculatorGraphConfig

Initialization

ValidatedGraphConfig

ImageFrame

Status

Initialization

Validation

StatusOr

Initialization

Validation

Packet

Timestamp

Initialization

Get its value

ResourceManager

Protocol Buffers

GPU compute

GpuResources

GlCalculatorHelper

Initialization

Examples

GpuBuffer

GlSyncPoint

Logging

Glog

Protobuf

Clone this wiki locally