Sentiment Analysis Guide

Real-time facial emotion detection for video conferencing using machine learning.

Overview

The library includes three sentiment analysis implementations for analyzing participant emotions in video streams. All analyzers share a common API pattern and are exposed through the main package exports.

New in v1.0.1001: HumanVideoSentimentAnalyzer now includes automatic WebGPU detection for an additional 2-3x performance boost on supported browsers.

Analyzer Options

WebGPU Auto-Detection

The HumanVideoSentimentAnalyzer automatically detects and uses the best available backend:

Priority Order: WebGPU > WebGL > WASM > CPU

// Auto-detection is enabled by default
const analyzer = new HumanVideoSentimentAnalyzer();
await analyzer.initialize();

// Check which backend was selected
const backendInfo = analyzer.getBackendInfo();
console.log(backendInfo);
// { backend: 'webgpu', reason: 'WebGPU available and functional', isWebGPU: true, isGPUAccelerated: true }

// Or check WebGPU availability statically before creating an instance
const hasWebGPU = await HumanVideoSentimentAnalyzer.isWebGPUAvailable();
console.log('WebGPU available:', hasWebGPU);

Forcing a Specific Backend

// Force WebGL (skip WebGPU detection)
const analyzer = new HumanVideoSentimentAnalyzer({
  backend: 'webgl'
});

// Force WebGPU (will fail if not available)
const analyzer = new HumanVideoSentimentAnalyzer({
  backend: 'webgpu'
});

// Force WASM (for testing or compatibility)
const analyzer = new HumanVideoSentimentAnalyzer({
  backend: 'wasm'
});

Browser Support for WebGPU (2025)

Importing Analyzers

// Direct imports for single video analysis
import {
  VideoSentimentAnalyzer,
  HumanVideoSentimentAnalyzer,
  MediaPipeVideoSentimentAnalyzer
} from 'muziertcclient';

// Multi-video managers for concurrent stream analysis (20-30+ videos)
import {
  loadMultiVideoSentimentManager,      // face-api.js based
  loadHumanMultiVideoSentimentManager, // Human.js based
  loadMediaPipeMultiVideoSentimentManager // MediaPipe based
} from 'muziertcclient';

Single Video Analyzer API

All three analyzers share the same API:

constructor(options)

Creates a new sentiment analyzer instance.

Default Engagement Weights:

{
  faceDetection: 0.05,      // Face detected in frame
  eyeOpenness: 0.35,        // Eyes open (attentive)
  headPose: 0.25,           // Head facing camera
  emotionalExpression: 0.25, // Active emotional expression
  faceSize: 0.10            // Face size in frame
}

async initialize()

Loads the ML model. Must be called before analysis.

const analyzer = new HumanVideoSentimentAnalyzer({ detectionInterval: 200 });
await analyzer.initialize();

analyzeVideo(videoElement, videoId)

Starts continuous analysis of a video element.

analyzer.analyzeVideo(document.getElementById('remote-video'), 'user-123');

stopAnalysis(videoId)

Stops analysis of a specific video.

analyzer.stopAnalysis('user-123');

destroy()

Cleans up resources and stops all analysis.

analyzer.destroy();

setDebugMode(enabled)

Toggles debug logging.

analyzer.setDebugMode(true);

canDetectFaces()

Returns whether face detection is available.

if (analyzer.canDetectFaces()) {
  analyzer.analyzeVideo(videoEl, 'user-1');
}

Analyzer Events

All analyzers emit events via the standard EventTarget pattern:

Emotions Payload Example

{
  videoId: 'user-123',
  emotions: {
    angry: 0.02,
    disgusted: 0.01,
    fearful: 0.01,
    happy: 0.85,
    sad: 0.03,
    surprised: 0.05,
    neutral: 0.03
  },
  dominant: 'happy',
  engagement: 0.78,  // 0-1 scale
  timestamp: 1705432156789
}

Multi-Video Sentiment Manager

For analyzing multiple video streams simultaneously (e.g., all participants in a room):

// Lazy load the manager (recommended)
const { HumanMultiVideoSentimentManager } = await loadHumanMultiVideoSentimentManager();
// Or for face-api.js: await loadMultiVideoSentimentManager();
// Or for MediaPipe: await loadMediaPipeMultiVideoSentimentManager();

const manager = new HumanMultiVideoSentimentManager({
  maxConcurrentAnalysis: 10,
  detectionInterval: 300,
  engagementWeights: { ... }
});

await manager.initialize();

// Add videos as participants join
manager.addVideo(videoElement, 'user-123');
manager.addVideo(videoElement2, 'user-456');

// Listen for emotions from any video
manager.addEventListener('emotions', (event) => {
  const { videoId, emotions, dominant, engagement } = event.detail;
  updateParticipantEmotionUI(videoId, dominant, engagement);
});

// Remove videos as participants leave
manager.removeVideo('user-123');

// Cleanup
manager.destroy();

Complete Integration Example

import { HumanVideoSentimentAnalyzer, Client, ClientEvents } from 'muziertcclient';

const client = new Client({ roomToken });
const analyzer = new HumanVideoSentimentAnalyzer({
  detectionInterval: 250,
  smoothingFactor: 0.3
});

await analyzer.initialize();

// Analyze remote participants
client.addEventListener(ClientEvents.MEDIA_TRACK_ADDED, (event) => {
  const { userId, track, kind } = event.detail;
  if (kind === 'video') {
    const videoEl = document.getElementById(`video-${userId}`);
    videoEl.srcObject = new MediaStream([track]);
    analyzer.analyzeVideo(videoEl, userId);
  }
});

// Handle emotion updates
analyzer.addEventListener('emotions', (event) => {
  const { videoId, dominant, engagement } = event.detail;
  updateEmotionIndicator(videoId, dominant);
  updateEngagementMeter(videoId, engagement);
});

// Handle face detection
analyzer.addEventListener('faceLeft', (event) => {
  showAwayIndicator(event.detail.videoId);
});

analyzer.addEventListener('faceReturned', (event) => {
  hideAwayIndicator(event.detail.videoId);
});

// Cleanup on disconnect
client.addEventListener(ClientEvents.USER_DISCONNECTED, (event) => {
  analyzer.stopAnalysis(event.detail.userId);
});

MediaPipe Emotion Strategies

The MediaPipe analyzer supports multiple emotion detection strategies:

const analyzer = new MediaPipeVideoSentimentAnalyzer({
  emotionStrategy: 'hybrid'
});

Performance Recommendations

1. Use Human.js (HumanVideoSentimentAnalyzer) for production deployments - 5-10x faster than face-api.js
2. Increase detectionInterval to 200-300ms for many participants
3. Use Multi-Video Manager for 3+ concurrent video streams
4. Lazy load analyzers to reduce initial bundle size
5. Destroy analyzers when not needed to free resources

Engagement Score Calculation

The engagement score (0-1) is calculated using weighted components:

• Face Detection (5%): Whether a face is detected
• Eye Openness (35%): How open the eyes are (attentiveness indicator)
• Head Pose (25%): Whether the head is facing the camera
• Emotional Expression (25%): Intensity of emotional expressions
• Face Size (10%): Size of face in frame (proximity indicator)

Custom weights can be provided in the constructor:

const analyzer = new HumanVideoSentimentAnalyzer({
  engagementWeights: {
    faceDetection: 0.10,
    eyeOpenness: 0.30,
    headPose: 0.20,
    emotionalExpression: 0.30,
    faceSize: 0.10
  }
});

Browser Compatibility

All analyzers support:

• Chrome (desktop and Android)
• Firefox (desktop and Android)
• Safari (desktop and iOS)
• Edge (Chromium-based)

Note: WebGL backend (default for Human.js) provides best performance but falls back to WASM/CPU if unavailable.