Once face replacement is integrated into a real-time engine, the challenge goes beyond simply pasting one face onto another. The character's face must understand its position in 3D space; how microphones, hands, and hair will occlude it; and whether performance input maintains consistent timing for the same person—all of which must be validated in the prototype.
2D Replacement and In-Engine Replacement Are Not the Same
Public paGAN 2 demos integrate synthetic faces into Unreal characters and scenes. By training on known digital assets, the system leverages facial and 3D scene relationships to handle occlusion, differing fundamentally from simply overlaying a face onto camera footage.
For branded short films, 3D spatial data preserves the relationship between camera angles, lighting, and props. However, this adds workload: character assets must be stabilized, training and performance data require authorization, and final footage needs shot-by-shot refinement.
Test Occlusion Before Discussing Real-Time Performance
Test clips should intentionally include microphones, hands, hair, glasses, and rapid head turns. Passing a frontal talking-head shot does not guarantee success with side profiles or product interactions. Whether the system maintains correct depth ordering when the face is occluded is often more telling than a polished still frame.
The image below is from an ONCE lifestyle video, used here to discuss acceptance criteria for subjects, environments, and camera distance. It does not imply that AI face replacement or neural rendering was used in this project.
Brand projects must manage performance and identity separately
Character performance can be handled by actors and motion capture systems, but identity and likeness rights must be documented separately by the brand and production team. Voice data, facial training data, endorsement term, distribution regions, and revocation mechanisms must all be specified in project documentation rather than relying solely on post-production staff memory.
Test footage must include at least the following:
- Frontal and profile direct-to-camera delivery
- Rapid speech, pauses, and emotional shifts
- Occlusion by hands, props, and hair
- Various focal lengths, shutter speeds, and on-set lighting conditions
- Differences between real-time preview and final render
The current MetaHuman documentation explains how real-time and offline performance data feeds into character animation, while research papers can supplement the neural rendering background. Brands should still include licensing, manual review, and rollback plans in the same table as technical testing. To plan facial content samples, start from the ONCE Project Brief portal .
Material Verification
- Research Seed 0931 New real-time hybrid AI and CGI face replacement
- Epic Games MetaHuman Animator
- Research Paper Deep Video Portraits
- Research Paper Neural Rendering
