[ PROMPT_NODE_24861 ]
Remotion Integration
[ SKILL_DOCUMENTATION ]
# HeyGen + Remotion Integration
This guide covers workflows for generating HeyGen avatar videos and using them in Remotion compositions.
## Quick Start
```typescript
// 1. Get avatar with default voice
const avatar = await getAvatarDetails(avatarId);
// 2. Generate video (MP4 with background - most common)
const videoId = await generateVideo({
video_inputs: [{
character: { type: "avatar", avatar_id: avatar.id, avatar_style: "normal" },
voice: { type: "text", input_text: script, voice_id: avatar.default_voice_id },
background: { type: "color", value: "#1a1a2e" },
}],
dimension: { width: 1920, height: 1080 },
});
// 3. Poll for completion (10-15+ min)
// 4. Use in Remotion with motion graphics overlaid on top
```
## Overview
A typical workflow:
1. Generate avatar video with HeyGen
2. Wait for completion and get video URL
3. Download or use URL directly in Remotion
4. Compose with other elements (backgrounds, overlays, animations)
## Choosing the Right Output Format
| Your Composition | Recommended | Why |
|------------------|-------------|-----|
| Avatar as presenter with overlays | MP4 + background | Simpler, overlays go on top |
| Loom-style (avatar over screen recording) | WebM + `closeUp`, mask in Remotion | Need transparency, apply circle mask in CSS |
| Avatar overlaid ON other video/content | WebM (transparent) | Need to see through to content behind |
| Full-screen avatar | MP4 + background | Standard approach |
**Use MP4 with background for most cases.** Use WebM when you need to see content *behind* the avatar.
**Note:** WebM only supports `normal` and `closeUp` styles. For circular framing, use CSS `border-radius: 50%` in Remotion.
## Recommended: Parallel Development Workflow
HeyGen video generation takes **10-15+ minutes**. Don't wait - work in parallel:
1. **Start HeyGen generation** - save `video_id` to a file, exit immediately
2. **Build Remotion composition** - use a placeholder or the avatar's `preview_video_url` (a short loop)
3. **Check HeyGen status** periodically or when done building
4. **Swap placeholder** for real video URL once ready
**Estimate duration from script**: ~150 words/minute speech rate, so `wordCount / 150 * 60 * fps` gives approximate frames.
**Composition tip**: Design components to work with or without the avatar video, so motion graphics can be tested independently.
## Dimension Alignment
**Critical**: Match HeyGen output dimensions to your Remotion composition.
### Common Dimension Presets
```typescript
// Shared dimension constants for both HeyGen and Remotion
const DIMENSIONS = {
landscape_1080p: { width: 1920, height: 1080 },
landscape_720p: { width: 1280, height: 720 },
portrait_1080p: { width: 1080, height: 1920 },
portrait_720p: { width: 720, height: 1280 },
square_1080p: { width: 1080, height: 1080 },
square_720p: { width: 720, height: 720 },
} as const;
type DimensionPreset = keyof typeof DIMENSIONS;
```
### HeyGen Video Generation
```typescript
// Generate HeyGen video with specific dimensions
async function generateHeyGenVideo(
script: string,
avatarId: string,
voiceId: string,
preset: DimensionPreset
): Promise {
const dimension = DIMENSIONS[preset];
const response = await fetch("https://api.heygen.com/v2/video/generate", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({
video_inputs: [
{
character: {
type: "avatar",
avatar_id: avatarId,
avatar_style: "normal",
},
voice: {
type: "text",
input_text: script,
voice_id: voiceId,
},
background: {
type: "color",
value: "#00FF00", // Green screen for compositing
},
},
],
dimension,
}),
});
const { data } = await response.json();
return data.video_id;
}
```
### Remotion Composition Setup
```tsx
// remotion/src/Root.tsx
import { Composition } from "remotion";
import { AvatarComposition } from "./AvatarComposition";
const DIMENSIONS = {
landscape_1080p: { width: 1920, height: 1080 },
// ... same as above
};
export const RemotionRoot: React.FC = () => {
return (
{
const { style = "normal", backgroundColor = "#1a1a2e" } = options;
const response = await fetch("https://api.heygen.com/v2/video/generate", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({
video_inputs: [{
character: {
type: "avatar",
avatar_id: avatarId,
avatar_style: style,
},
voice: {
type: "text",
input_text: script,
voice_id: voiceId,
},
background: {
type: "color",
value: backgroundColor,
},
}],
dimension: { width: 1920, height: 1080 },
}),
});
const { data } = await response.json();
return data.video_id;
}
```
### Transparent Background (WebM)
Only use when you need to see content *behind* the avatar (e.g., avatar overlaid on screen recording):
```typescript
// Use /v1/video.webm endpoint for transparent background
// Note: Different structure than /v2/video/generate
const response = await fetch("https://api.heygen.com/v1/video.webm", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({
avatar_pose_id: avatarPoseId, // Required: avatar pose ID
avatar_style: "normal", // Required: "normal" or "closeUp" only
input_text: script, // Required (with voice_id)
voice_id: voiceId, // Required (with input_text)
dimension: { width: 1920, height: 1080 },
}),
});
```
## Using HeyGen Video in Remotion
### Important: Use OffthreadVideo for Frame-Accurate Rendering
**Always use `OffthreadVideo` instead of `Video`** for HeyGen avatar videos. The basic `Video` component uses the browser's video decoder which isn't frame-accurate, causing jitter during rendering. `OffthreadVideo` extracts frames via FFmpeg for smooth, accurate playback.
`OffthreadVideo` is included in the core `remotion` package - no additional install needed.
### Basic Usage
```tsx
// remotion/src/AvatarComposition.tsx
import { OffthreadVideo, useVideoConfig } from "remotion";
interface AvatarCompositionProps {
avatarVideoUrl: string;
}
export const AvatarComposition: React.FC = ({
avatarVideoUrl,
}) => {
return (
);
};
```
### WebM with Transparent Background (Recommended)
Using WebM from `/v1/video.webm` - no chroma keying needed:
```tsx
import { OffthreadVideo, AbsoluteFill, Sequence } from "remotion";
export const AvatarWithMotionGraphics: React.FC = ({ avatarWebmUrl }) => {
return (
{/* Layer 1: Your background/content */}
{/* Layer 2: Avatar with transparent background - use OffthreadVideo for frame-accurate rendering */}
{/* Layer 3: Overlays on top of avatar */}
);
};
```
### Loom-Style: Circle Avatar Over Screen Recording
Use `closeUp` style + WebM, then apply circular mask in Remotion:
```tsx
import { OffthreadVideo, AbsoluteFill } from "remotion";
export const LoomStyleComposition: React.FC = ({ screenRecordingUrl, avatarWebmUrl }) => {
return (
{/* Screen recording fills the frame */}
{/* Avatar with circular mask - transparent bg shows screen behind */}
);
};
```
**Note:** WebM doesn't support `circle` style - use `normal` or `closeUp` and apply circular masking via CSS.
### Legacy: Green Screen with Chroma Key
If using MP4 with green background (not recommended - use WebM instead):
```tsx
// Note: True chroma key requires WebGL or post-processing
// WebM transparent background is much simpler
```
### Layered Composition
```tsx
import { OffthreadVideo, Sequence, useVideoConfig, Img } from "remotion";
interface LayeredAvatarProps {
avatarVideoUrl: string;
backgroundUrl: string;
logoUrl: string;
title: string;
}
export const LayeredAvatarComposition: React.FC = ({
avatarVideoUrl,
backgroundUrl,
logoUrl,
title,
}) => {
const { fps } = useVideoConfig();
return (
{/* Layer 1: Background */}
{/* Layer 2: Avatar video - use OffthreadVideo to prevent jitter */}
{/* Layer 3: Title (appears after 1 second) */}
);
};
```
## Complete Workflow
### Generate and Compose
```typescript
import { bundle } from "@remotion/bundler";
import { renderMedia, selectComposition } from "@remotion/renderer";
async function generateAvatarVideoForRemotion(
script: string,
outputPath: string
) {
// 1. Generate HeyGen video
console.log("Generating HeyGen avatar video...");
const videoId = await generateHeyGenVideo(
script,
"josh_lite3_20230714",
"1bd001e7e50f421d891986aad5158bc8",
"landscape_1080p"
);
// 2. Wait for completion
console.log("Waiting for HeyGen video...");
const avatarVideoUrl = await waitForVideo(videoId);
console.log(`HeyGen video ready: ${avatarVideoUrl}`);
// 3. Get video duration for Remotion
const avatarDuration = await getVideoDuration(avatarVideoUrl);
const durationInFrames = Math.ceil(avatarDuration * 30); // 30 fps
// 4. Bundle Remotion project
console.log("Bundling Remotion project...");
const bundleLocation = await bundle({
entryPoint: "./remotion/src/index.ts",
});
// 5. Select composition
const composition = await selectComposition({
serveUrl: bundleLocation,
id: "AvatarVideo",
inputProps: {
avatarVideoUrl,
},
});
// 6. Render final video
console.log("Rendering final composition...");
await renderMedia({
composition: {
...composition,
durationInFrames,
},
serveUrl: bundleLocation,
codec: "h264",
outputLocation: outputPath,
inputProps: {
avatarVideoUrl,
},
});
console.log(`Final video rendered: ${outputPath}`);
return outputPath;
}
```
### Dynamic Duration with calculateMetadata
```tsx
// remotion/src/AvatarComposition.tsx
import { CalculateMetadataFunction } from "remotion";
export const calculateAvatarMetadata: CalculateMetadataFunction = async ({ props }) => {
// Fetch video duration from HeyGen video
const duration = await getVideoDurationInSeconds(props.avatarVideoUrl);
return {
durationInFrames: Math.ceil(duration * 30),
fps: 30,
width: 1920,
height: 1080,
};
};
// In Root.tsx
```
## Best Practices
### 1. Use Green Screen for Flexibility
Generate HeyGen videos with green screen background when you want to composite:
```typescript
background: {
type: "color",
value: "#00FF00", // Pure green for chroma key
}
```
### 2. Match Frame Rates
HeyGen default is 25 fps. Consider this when setting Remotion fps:
```typescript
// Option 1: Match HeyGen's 25 fps
fps: 25
// Option 2: Use 30 fps with playback rate adjustment
```
### 3. URL vs Download: When to Use Each
**Use URL directly** when:
- Previewing in Remotion Studio (`npm run dev`)
- URL won't expire before render completes
- You want faster iteration during development
```tsx
// Direct URL usage - simpler, faster for dev
```
**Download first** when:
- URL has expiration (HeyGen URLs expire after ~24 hours)
- Rendering will happen later or repeatedly
- Network reliability is a concern
- You need offline rendering
```typescript
// Download with retry for reliability
async function downloadVideoWithRetry(
url: string,
outputPath: string,
maxRetries = 5
): Promise {
for (let attempt = 0; attempt setTimeout(r, delay));
}
}
throw new Error("Download failed after retries");
}
// Use local file in Remotion
const localPath = await downloadVideoWithRetry(avatarVideoUrl, "./public/avatar.mp4");
```
**Hybrid approach** (recommended for production):
```typescript
// Save both URL and local path in metadata
const metadata = {
videoUrl: result.video_url, // For quick preview
localPath: "./public/avatar.mp4", // For reliable rendering
expiresAt: Date.now() + 24 * 60 * 60 * 1000, // URL expiration
};
// In Remotion component, prefer local if available
const videoSrc = fs.existsSync(localPath) ? staticFile("avatar.mp4") : avatarVideoUrl;
```
### 4. Handle Avatar Positioning
Common avatar positions in compositions:
```typescript
const AVATAR_POSITIONS = {
fullscreen: { width: "100%", height: "100%", position: "center" },
bottomRight: { width: "40%", bottom: 0, right: 0 },
bottomLeft: { width: "40%", bottom: 0, left: 0 },
pictureInPicture: { width: "25%", bottom: 20, right: 20 },
leftThird: { width: "33%", left: 0, height: "100%" },
};
```
## Output Formats
### HeyGen Output
- Format: MP4 (H.264)
- Audio: AAC
- Resolution: As specified in request
### Remotion Output
- Codec: H.264 (default), VP8, VP9, ProRes
- Match or exceed HeyGen quality settings
```typescript
await renderMedia({
codec: "h264",
crf: 18, // High quality
// ...
});
```
## Troubleshooting
### Video Not Playing in Remotion
1. Check URL accessibility (CORS issues)
2. Verify video format compatibility
3. Try downloading locally first
### Dimension Mismatch
Ensure both HeyGen and Remotion use identical dimensions:
```typescript
// Shared config
const VIDEO_CONFIG = {
width: 1920,
height: 1080,
fps: 30,
};
// HeyGen
dimension: { width: VIDEO_CONFIG.width, height: VIDEO_CONFIG.height }
// Remotion
```
### Video Jitter During Rendering
If avatar video appears jittery or stuttery in rendered output:
1. **Use `OffthreadVideo` instead of `Video`** - The basic `Video` component uses the browser's video decoder which isn't frame-accurate
2. Update imports (no additional install needed - it's in core `remotion`):
```tsx
// Before (causes jitter)
import { Video } from "remotion";
// After (frame-accurate)
import { OffthreadVideo } from "remotion";
```
3. For WebM with transparency, add the `transparent` prop:
```tsx
```
### Audio Sync Issues
If avatar audio drifts:
- Verify source video frame rate
- Check for encoding issues
- Consider re-encoding with consistent settings
{title}
{/* Layer 4: Logo */}
Source: claude-code-templates (MIT). See About Us for full credits.