# Sentiment Analysis Guide

Real-time facial emotion detection for video conferencing using machine learning.

## Overview

The library includes three sentiment analysis implementations for analyzing participant emotions in video streams. All analyzers share a common API pattern and are exposed through the main package exports.

**New in v1.0.1001:** `HumanVideoSentimentAnalyzer` now includes automatic WebGPU detection for an additional 2-3x performance boost on supported browsers.

## Analyzer Options

| Analyzer | Backend | Performance | Best For |
|----------|---------|-------------|----------|
| `VideoSentimentAnalyzer` | face-api.js | Baseline | Compatibility, proven accuracy |
| `HumanVideoSentimentAnalyzer` | Human.js + WebGPU | 5-10x faster (up to 20x with WebGPU) | Production, modern browsers |
| `MediaPipeVideoSentimentAnalyzer` | MediaPipe | 3-5x faster | WebGL environments, hybrid strategies |

## WebGPU Auto-Detection

The `HumanVideoSentimentAnalyzer` automatically detects and uses the best available backend:

**Priority Order:** WebGPU > WebGL > WASM > CPU

```javascript
// Auto-detection is enabled by default
const analyzer = new HumanVideoSentimentAnalyzer();
await analyzer.initialize();

// Check which backend was selected
const backendInfo = analyzer.getBackendInfo();
console.log(backendInfo);
// { backend: 'webgpu', reason: 'WebGPU available and functional', isWebGPU: true, isGPUAccelerated: true }

// Or check WebGPU availability statically before creating an instance
const hasWebGPU = await HumanVideoSentimentAnalyzer.isWebGPUAvailable();
console.log('WebGPU available:', hasWebGPU);
```

### Forcing a Specific Backend

```javascript
// Force WebGL (skip WebGPU detection)
const analyzer = new HumanVideoSentimentAnalyzer({
  backend: 'webgl'
});

// Force WebGPU (will fail if not available)
const analyzer = new HumanVideoSentimentAnalyzer({
  backend: 'webgpu'
});

// Force WASM (for testing or compatibility)
const analyzer = new HumanVideoSentimentAnalyzer({
  backend: 'wasm'
});
```

### Browser Support for WebGPU (2025)

| Browser | WebGPU Status |
|---------|---------------|
| Chrome 113+ | Stable |
| Edge 113+ | Stable |
| Firefox 141+ | Stable |
| Safari (macOS Sequoia) | Supported |
| Android 12+ | Chrome 121+ |

## Importing Analyzers

```javascript
// Direct imports for single video analysis
import {
  VideoSentimentAnalyzer,
  HumanVideoSentimentAnalyzer,
  MediaPipeVideoSentimentAnalyzer
} from 'muziertcclient';

// Multi-video managers for concurrent stream analysis (20-30+ videos)
import {
  loadMultiVideoSentimentManager,      // face-api.js based
  loadHumanMultiVideoSentimentManager, // Human.js based
  loadMediaPipeMultiVideoSentimentManager // MediaPipe based
} from 'muziertcclient';
```

## Single Video Analyzer API

All three analyzers share the same API:

### `constructor(options)`

Creates a new sentiment analyzer instance.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `detectionInterval` | number | 100 | Milliseconds between analysis frames |
| `minConfidence` | number | 0.5 | Minimum confidence for face detection |
| `smoothing` | boolean | true | Enable emotion smoothing |
| `smoothingFactor` | number | 0.3 | Smoothing intensity (0-1) |
| `debug` | boolean | false | Enable debug logging |
| `engagementWeights` | object | See below | Weights for engagement calculation |

**Default Engagement Weights:**
```javascript
{
  faceDetection: 0.05,      // Face detected in frame
  eyeOpenness: 0.35,        // Eyes open (attentive)
  headPose: 0.25,           // Head facing camera
  emotionalExpression: 0.25, // Active emotional expression
  faceSize: 0.10            // Face size in frame
}
```

### `async initialize()`

Loads the ML model. Must be called before analysis.

```javascript
const analyzer = new HumanVideoSentimentAnalyzer({ detectionInterval: 200 });
await analyzer.initialize();
```

### `analyzeVideo(videoElement, videoId)`

Starts continuous analysis of a video element.

| Parameter | Type | Description |
|-----------|------|-------------|
| `videoElement` | HTMLVideoElement | Video element to analyze |
| `videoId` | string | Unique identifier for the video |

```javascript
analyzer.analyzeVideo(document.getElementById('remote-video'), 'user-123');
```

### `stopAnalysis(videoId)`

Stops analysis of a specific video.

```javascript
analyzer.stopAnalysis('user-123');
```

### `destroy()`

Cleans up resources and stops all analysis.

```javascript
analyzer.destroy();
```

### `setDebugMode(enabled)`

Toggles debug logging.

```javascript
analyzer.setDebugMode(true);
```

### `canDetectFaces()`

Returns whether face detection is available.

```javascript
if (analyzer.canDetectFaces()) {
  analyzer.analyzeVideo(videoEl, 'user-1');
}
```

## Analyzer Events

All analyzers emit events via the standard EventTarget pattern:

| Event | Description | Payload |
|-------|-------------|---------|
| `emotions` | Emotion analysis result | `{ videoId, emotions, dominant, engagement, timestamp }` |
| `error` | Analysis error occurred | `{ videoId, error, message }` |
| `loading` | Model loading started | `{ progress }` |
| `ready` | Model loaded and ready | `{}` |
| `faceLeft` | Face left the frame | `{ videoId, timestamp }` |
| `faceReturned` | Face returned to frame | `{ videoId, timestamp }` |
| `userOffFrame` | User off-frame for extended period | `{ videoId, duration }` |

### Emotions Payload Example

```javascript
{
  videoId: 'user-123',
  emotions: {
    angry: 0.02,
    disgusted: 0.01,
    fearful: 0.01,
    happy: 0.85,
    sad: 0.03,
    surprised: 0.05,
    neutral: 0.03
  },
  dominant: 'happy',
  engagement: 0.78,  // 0-1 scale
  timestamp: 1705432156789
}
```

## Multi-Video Sentiment Manager

For analyzing multiple video streams simultaneously (e.g., all participants in a room):

```javascript
// Lazy load the manager (recommended)
const { HumanMultiVideoSentimentManager } = await loadHumanMultiVideoSentimentManager();
// Or for face-api.js: await loadMultiVideoSentimentManager();
// Or for MediaPipe: await loadMediaPipeMultiVideoSentimentManager();

const manager = new HumanMultiVideoSentimentManager({
  maxConcurrentAnalysis: 10,
  detectionInterval: 300,
  engagementWeights: { ... }
});

await manager.initialize();

// Add videos as participants join
manager.addVideo(videoElement, 'user-123');
manager.addVideo(videoElement2, 'user-456');

// Listen for emotions from any video
manager.addEventListener('emotions', (event) => {
  const { videoId, emotions, dominant, engagement } = event.detail;
  updateParticipantEmotionUI(videoId, dominant, engagement);
});

// Remove videos as participants leave
manager.removeVideo('user-123');

// Cleanup
manager.destroy();
```

## Complete Integration Example

```javascript
import { HumanVideoSentimentAnalyzer, Client, ClientEvents } from 'muziertcclient';

const client = new Client({ roomToken });
const analyzer = new HumanVideoSentimentAnalyzer({
  detectionInterval: 250,
  smoothingFactor: 0.3
});

await analyzer.initialize();

// Analyze remote participants
client.addEventListener(ClientEvents.MEDIA_TRACK_ADDED, (event) => {
  const { userId, track, kind } = event.detail;
  if (kind === 'video') {
    const videoEl = document.getElementById(`video-${userId}`);
    videoEl.srcObject = new MediaStream([track]);
    analyzer.analyzeVideo(videoEl, userId);
  }
});

// Handle emotion updates
analyzer.addEventListener('emotions', (event) => {
  const { videoId, dominant, engagement } = event.detail;
  updateEmotionIndicator(videoId, dominant);
  updateEngagementMeter(videoId, engagement);
});

// Handle face detection
analyzer.addEventListener('faceLeft', (event) => {
  showAwayIndicator(event.detail.videoId);
});

analyzer.addEventListener('faceReturned', (event) => {
  hideAwayIndicator(event.detail.videoId);
});

// Cleanup on disconnect
client.addEventListener(ClientEvents.USER_DISCONNECTED, (event) => {
  analyzer.stopAnalysis(event.detail.userId);
});
```

## MediaPipe Emotion Strategies

The MediaPipe analyzer supports multiple emotion detection strategies:

| Strategy | Description |
|----------|-------------|
| `landmarks` | Uses facial landmark positions (default) |
| `hybrid` | Combines landmarks with additional heuristics |
| `faceapi` | Falls back to face-api.js patterns |
| `blendshapes` | Uses ARKit-compatible blendshapes |

```javascript
const analyzer = new MediaPipeVideoSentimentAnalyzer({
  emotionStrategy: 'hybrid'
});
```

## Performance Recommendations

1. **Use Human.js** (`HumanVideoSentimentAnalyzer`) for production deployments - 5-10x faster than face-api.js
2. **Increase `detectionInterval`** to 200-300ms for many participants
3. **Use Multi-Video Manager** for 3+ concurrent video streams
4. **Lazy load** analyzers to reduce initial bundle size
5. **Destroy** analyzers when not needed to free resources

## Engagement Score Calculation

The engagement score (0-1) is calculated using weighted components:

- **Face Detection** (5%): Whether a face is detected
- **Eye Openness** (35%): How open the eyes are (attentiveness indicator)
- **Head Pose** (25%): Whether the head is facing the camera
- **Emotional Expression** (25%): Intensity of emotional expressions
- **Face Size** (10%): Size of face in frame (proximity indicator)

Custom weights can be provided in the constructor:

```javascript
const analyzer = new HumanVideoSentimentAnalyzer({
  engagementWeights: {
    faceDetection: 0.10,
    eyeOpenness: 0.30,
    headPose: 0.20,
    emotionalExpression: 0.30,
    faceSize: 0.10
  }
});
```

## Browser Compatibility

All analyzers support:
- Chrome (desktop and Android)
- Firefox (desktop and Android)
- Safari (desktop and iOS)
- Edge (Chromium-based)

**Note:** WebGL backend (default for Human.js) provides best performance but falls back to WASM/CPU if unavailable.