How Does AI Cut You Out of a Video Background? Person Masking Explained

Chroma key, segmentation and matting are different jobs

Chroma key, segmentation and matting all separate you from the room, but each one decides what counts as you in a different way.

Chroma key is the green screen. Software removes every pixel close to one chosen colour, usually green or blue, which Wikipedia’s article describes as the colours furthest from skin tones. It knows nothing about people, only about colour. That makes it predictable on a well-lit screen, and the same article names even lighting and avoiding shadows as the biggest challenge. Green light bouncing onto you, known as spill, and clothes close to the screen colour are the classic problems.

Segmentation gives every pixel a label. MediaPipe’s Image Segmenter, the fallback in this site’s setup, divides an image into regions by category, and its selfie model has two: person and background. It can return a category mask, where each pixel is one or the other, or confidence masks, where each pixel carries a score. The Robust Video Matting authors note that using a plain segmentation mask directly to replace a background produces strong artefacts.

Matting answers a finer question. The Robust Video Matting (RVM) paper describes every frame as a mix of a foreground and a background:

I = αF + (1 − α)B

Alpha (α) runs from 0 to 1 for each pixel. At 1 the pixel is all you, at 0 it is all room, and in between it is a blend, which is exactly what a strand of hair or a blurred hand looks like. A matting model predicts alpha and your own colours, so you can be placed on any new background. RVM does this for people in video without a trimap, which is a hand-drawn map of the uncertain areas, and without a photo of the empty room.

Manual rotoscoping is the traditional answer. Wikipedia describes it, in visual effects, as manually creating a matte for an element on live-action footage, traced frame by frame. Software helps with motion tracking, but a person still draws and checks the matte.

Why hair and fast motion are the hard parts

Hair and fast motion are hard because their pixels really are part you and part background. A strand of hair can be thinner than a pixel, so that pixel’s colour is a blend of hair and wall. A hand moving quickly smears across the frame in the same way. Any method that must answer yes or no for each pixel will either chop those edges off or drag a fringe of the old room along with you.

Resolution makes it harder. MediaPipe’s selfie model works on a 256 × 256 input, or 144 × 256 for its landscape version, far smaller than a 1080 × 1920 frame. RVM also shrinks the frame for its first pass, and its documentation suggests a working size between 256 and 512 pixels. It then refines the result at full resolution. In the paper’s comparisons on real videos, it predicted fine details such as hair strands more accurately than the other methods tested.

Fast motion is a timing problem as well as a blur problem. The paper notes that an earlier trick, comparing each prediction with neighbouring frames to suppress flicker, cannot handle fast-moving body parts, and the authors report that their model handled them better in cellphone and webcam tests. Even so, a quick gesture is where you are most likely to spot a halo or a clipped fingertip.

Green screens have their own version of the hair problem. Wikipedia’s chroma key article warns that fine details such as frizzy hair may not resolve properly.

Why masks flicker, and how video models keep them steady

A mask flickers when each frame is cut out on its own, with no memory of the frames before it. Small differences in where each frame’s edge falls add up to a shimmer around your outline, most visible at your hair and hands.

The RVM authors point out that most earlier methods, though built for video, still processed frames as independent images. Their model uses a recurrent network instead, so each prediction draws on what came before. They give three reasons this helps. Results are more coherent and flicker less. When a single frame is ambiguous, for example when your clothing resembles something passing behind you, earlier frames help place the boundary. And as you move, parts of the room you were hiding come into view, so the model learns more about the background over time.

That memory has practical consequences. RVM’s documentation says frames must be processed in order while the model’s recurrent state, its memory, is carried forward. In the paper’s measurements, the error fell sharply over the first 15 frames and then held steady, so the opening moments of a clip may be slightly rougher than the rest.

On speed, the paper reports 4K at 76 FPS and HD at 104 FPS on an Nvidia GTX 1080 Ti, measured as the model’s throughput on that GPU. The repository adds that its bundled conversion script is expected to be much slower, because it does not use hardware video decoding and encoding.

The setup prompt on this site adds two safeguards: it softens the mask edge by 2–3 px and smooths the mask over time. Smoothing has a cost, though. Average across too many frames and the edge of a fast-moving hand starts to trail behind the hand itself.

What footage gives the cleanest mask

Footage with one clear subject, a simple background and a steady camera gives the cleanest masks. The RVM paper is direct about this in its limitations: the model prefers videos with a clear target subject, people in the background make the subject ambiguous, and simpler backgrounds give more accurate matting.

• Film alone. Someone walking behind you is the textbook failure case.
• Keep the background simple, not empty. Effects such as Vertigo need some room to show, so tidy shelves beat both a cluttered desk and a bare wall.
• Use a tripod. The authors found the model did slightly better on static backgrounds than on dynamic ones, where objects or the camera move.
• Contrast with what is behind you. An AI mask does not key out a colour, so a green shirt is fine. What matters is that your hair and clothes differ from the wall directly behind them.
• Stay close to the camera. MediaPipe described its older selfie segmentation solution as intended for people within about 2 m of the camera, which matters if the setup falls back to it.
• Light the scene well. More light allows a shorter exposure, which means less motion blur on your hands.
• Think about what you hold. A mug or a phone is not a person, so a person mask may cut it away or include it inconsistently.

How a mask puts graphics behind you

A mask lets the software slide a graphic between the room and you. Each frame is rebuilt in layers: the original room at the back, then the graphic, then you, cut out with the mask, then anything that belongs in front, such as captions.

Alpha is what makes the join believable. The final pixel is your colour weighted by alpha plus the layers behind you weighted by one minus alpha, the same equation matting starts from. Where alpha is 0.5 along a hair edge, half of the graphic shows through, so the strand blends into the new layer instead of ending in a hard cut line.

This is how Orbit sends cards around your back, how Couch Wall fills the wall with thumbnails while you stay in front, and how Chart Grow draws a line behind you while your hand stays on top. Frame Break turns the idea around: the video is clipped to a frame, but the masked version of you is drawn outside it. Green Screen replaces the room entirely, which is why it also pulls leftover room colour out of your edges, the same spill problem chroma keying has.

There are two catches. A mask is flat: it knows where you are on screen, not how far away the wall or the shelf is, so graphics can go behind you but not behind your furniture. And lifting you out leaves a person-shaped hole in the room layer, so effects that move the background, such as the Vertigo dolly zoom, have to fill it. The setup prompt makes the masks once per video, and the guide to adding effects to talking-head videos shows which effects use them.