Rotoscoping — isolating subjects from their backgrounds frame-by-frame — has always been one of the most labour-intensive tasks in any VFX pipeline.
Over the past year, our AI Labs team embedded directly with the VFX department to understand exactly where time was being lost. The answer was unsurprising: edges. Hair, motion blur, translucent fabric — all the things that make human subjects interesting visually are also the things that break automated segmentation models trained on clean studio data.
The dataset problem
Every off-the-shelf segmentation model we evaluated failed in the same ways on real production footage. We spent three months cleaning and labelling roughly 40,000 frames from internal productions, using a custom version of Segment Anything as a starting point, then had senior roto artists correct every mask at the pixel level.
Architecture and training
We fine-tuned a transformer-based architecture with a temporal attention component. After six months of iteration, we reduced average artist correction time by 70% on internal benchmarks. The model is now running in production on three active shows.