Skip to main content

Virtual Backgrounds in Standard, Hybrid, and Headless UIs

Virtual backgrounds change the video track that MediaSFU publishes. Standard UI owns the complete workflow. Hybrid UI reuses the SDK modal against the same room engine. Fully custom UI owns the controls while calling the exported processing helpers.

Availability by SDK​

SDKSupplied background surfaceFully custom boundary
ReactJS 4.3.2ModernBackgroundModal and BackgroundModalapplyVirtualBackground, clearVirtualBackground, getLocalVideoStream
Angular 2.3.2BackgroundModal and LaunchBackgroundUse the exported Angular room/background services; no React modal or React helper API
Vue 1.1.2BackgroundModal and useBackgroundModalUse Vue's current room state; no React helper API
React Native 2.4.2ModernBackgroundModal and BackgroundModalNative room parameters and processed virtualStream
Expo 2.5.2ModernBackgroundModal and BackgroundModalExpo development build and native room parameters
Flutter 2.3.1Background controls in the supplied media settingsFlutter background state and platform processor
Android/Kotlin 1.0.6Compose/platform virtual-background processingVirtualBackgroundProcessor and current room state
Swift 0.1.3, Unity preview.2, shared core 1.2.0No equivalent complete standalone background workflow in the audited public surfaceKeep the feature unavailable until the target publishes a supported contract

Examples are platform-specific: use the package's own public API reference and configuration names for the target platform.

Standard: let the room own the feature​

Mount the current complete room and open its media/background settings. The room owns camera acquisition, preview, save, apply, publish, restore, removal, and cleanup. A saved selection can be retained while the camera is off and applied when the room publishes the camera later.

import {ModernMediasfuGeneric} from 'mediasfu-reactjs';

export function StandardRoom() {
  return <ModernMediasfuGeneric />;
}

Hybrid: use the room-owned modal lifecycle​

Visibility comes from the current engine state. Opening and closing must call that engine's update function. Keep the component mounted for the room's lifetime so camera-off preview state and automatic restoration are preserved.

import type {ComponentProps} from 'react';
import {ModernBackgroundModal} from 'mediasfu-reactjs';

type BackgroundRoom = ComponentProps<typeof ModernBackgroundModal>['parameters'];

export function BackgroundPanel({parameters}: {parameters: BackgroundRoom}) {
  const live = parameters.getCurrentParams?.() ?? parameters;

  return <section aria-label="Camera background">
    <button onClick={() => live.updateIsBackgroundModalVisible(true)}>
      Choose background
    </button>
    <ModernBackgroundModal
      isVisible={live.isBackgroundModalVisible === true}
      onClose={() => live.updateIsBackgroundModalVisible(false)}
      parameters={live}
      renderMode="sidebar"
    />
  </section>;
}

ReactJS 4.3.2 supports renderMode="modal", "sidebar", or "inline" on ModernBackgroundModal. These are presentation choices, not separate modal lifecycles. Do not add a competing local isOpen state or conditionally unmount the component whenever it is hidden.

With the camera off, the modal may acquire temporary preview media. Save stores the selection for a later camera start; it does not itself prove that video is being published. Let the modal release preview resources and let the room apply the saved selection when camera publication begins.

Fully headless: own controls, not transport internals​

The React helper requires a live camera for a published replacement. Serialize actions and show every returned error.

import {useState} from 'react';
import {
  applyVirtualBackground,
  clearVirtualBackground,
  getCurrentParams,
  type HeadlessParameters,
} from 'mediasfu-reactjs';

export function BackgroundActions({parameters, image}: {
  parameters: HeadlessParameters;
  image: string;
}) {
  const [busy, setBusy] = useState(false);
  const [notice, setNotice] = useState('');

  async function change(remove: boolean) {
    if (busy) return;
    const live = getCurrentParams({parameters});
    if (!remove && !live.videoAlreadyOn) {
      setNotice('Turn on the camera before applying a background.');
      return;
    }
    setBusy(true);
    try {
      const result = remove
        ? await clearVirtualBackground({parameters: live})
        : await applyVirtualBackground({parameters: live, image, publish: true});
      setNotice(result.ok ? 'Background updated.' : result.error);
    } finally {
      setBusy(false);
    }
  }

  return <section>
    <button disabled={busy} onClick={() => void change(false)}>Apply background</button>
    <button disabled={busy} onClick={() => void change(true)}>Remove background</button>
    <p role="status">{notice}</p>
  </section>;
}

publish: true replaces the running video producer's track. publish: false is preview/local processing only. Install the optional @mediapipe/selfie_segmentation dependency and make its model assets available under your Content Security Policy when using React's background pipeline.

Always render the processed local stream first​

MediaSFU's own prepopulateUserMedia and addVideosGrid logic uses this precedence for the self view:

const selfView = keepBackground && virtualStream
  ? virtualStream
  : localStreamVideo;

The safer React headless equivalent is getLocalVideoStream({parameters: latestParameters}), which also rejects an ended or disabled local video track. A custom local tile that always renders localStreamVideo shows the raw camera even though remote users receive the processed producer track. That mismatch is a rendering bug in the custom tile, not evidence that publication failed.

On every parameter or media publication, reattach the selected stream when its identity changes. Keep the self view muted. Never feed the processed output back into the segmentation input.

Removal and teardown​

  • Clear the helper-owned pipeline before switching cameras, stopping the camera, leaving, or replacing the room session.
  • Stop temporary preview tracks and animation loops that your application created.
  • Remove the saved selection only when the person explicitly chooses to clear it; camera-off alone may preserve it for the next publish.
  • Do not call the headless clear helper against a modal-owned pipeline.
  • Test camera initially off, preview, save, later camera start, apply, remote reception, local rendering, removal, camera restart, and leave.

Continue with device and translation availability by SDK, rendered media, and leave and cleanup.