I Built an Invisibility Cloak in the Browser No Green Screen, No Model Training

What hand tracking and a little geometry can do that a segmentation model can’t

A few weeks ago I saw a demo that stuck with me: someone waving their hand in front of a camera, and wherever their fingers passed, their body just… disappeared. Not blurred. Not replaced with a green screen. Gone, replaced by whatever was behind them a moment ago.

My first assumption was that it had to involve some kind of segmentation model — something trained to recognize “person” vs. “background” and mask one out. That’s how most of these effects work.

It wasn’t. And once I understood the actual trick, I wanted to rebuild it myself.

This post walks through how it works, why it’s simpler than it looks, and includes both a browser version and a Python version you can run yourself.

The illusion, explained

Here’s the whole trick in one sentence: capture a photo of the empty background, then use your fingertips to paint a window that shows that old photo instead of the live feed.

That’s it. There’s no understanding of “you” as a person anywhere in the pipeline. The system doesn’t know what a body is. It just knows:

1. Here’s what the room looked like before you stepped in.
2. Here’s a shape, defined by your fingertips, that moves every frame.
3. Inside that shape, show the old pixels instead of the new ones.

Because the background doesn’t change (assuming the camera stays still), swapping “old” background pixels for “new” background pixels in that region is seamless. The illusion isn’t computer vision understanding you — it’s computer vision ignoring you in a very precise, moving shape.

Breaking down the pipeline

1. Capture the background

Before anything else happens, the app grabs a single frame of the empty scene and holds onto it. This is the “before” photo everything else gets compared against — a straightforward canvas draw operation, no processing involved.

If the camera or the room changes after this point, the illusion breaks — because the frozen photo no longer matches reality behind you.

2. Track the hands

This is where MediaPipe’s hand landmark model comes in. It detects 21 points per hand in real time — knuckles, fingertips, the works. For this effect, only two of those 21 points actually matter per hand:

• Landmark 4 — thumb tip
• Landmark 8 — index fingertip

That’s it. The other 19 points exist in the model’s output, but this effect doesn’t need them.

3. Turn four points into a polygon

With both hands in frame, you get up to four points: two thumb tips, two index tips. To turn those into a clean shape (rather than a self-intersecting mess), they get sorted by angle around their shared centroid — essentially finding the “average center” of all four points, then ordering each point by its angle relative to that center, so connecting them in sequence traces a clean loop instead of a shape that crosses itself.

That sorting step is doing a lot of work — without it, the four points could connect in an order that crosses itself, producing a broken, pinched shape instead of a clean polygon.

4. Smooth it, and keep left/right stable

Raw landmark data is noisy. Frame to frame, points jitter slightly even when your hand is still. The fix is a simple exponential moving average — each new point is blended mostly with its previous smoothed position, so a single noisy frame doesn’t cause a visible jump.

There’s a subtler bug this also has to handle: MediaPipe doesn’t always report “Left hand, then Right hand” in a consistent order between frames. If you smooth blindly by array index, your polygon can flicker or twist whenever detection order changes. The fix is keying the smoothing state by the handedness label itself (“Left” / “Right”), not by array position — so each hand’s points are only ever averaged against their own history.

5. Composite

The final step is where the “invisibility” actually happens. The polygon becomes a mask — filled white, then blurred slightly for a soft edge instead of a hard geometric outline. That blurred mask is then used to cut the matching region out of the frozen background, and the result is drawn directly over the live video feed.

Because the mask edge is blurred rather than sharp, the drawing operation blends automatically at the boundary — no manual per-pixel alpha math required. That soft edge is the difference between “obviously a cardboard cutout” and something that actually looks like a window into the past.

Two versions, same idea

Browser (JavaScript) — a single HTML file using MediaPipe Tasks Vision and the Canvas API. GPU-accelerated, runs entirely client-side, nothing to install beyond opening the file in Chrome.

Python (OpenCV + MediaPipe) — same pipeline, different tools. OpenCV builds the polygon, applies a Gaussian blur to soften the mask, and the compositing step is a manual weighted blend between the background and the live frame, since OpenCV doesn’t have a built-in equivalent to the browser’s masking operation.

Both versions are on GitHub, linked at the bottom.

Why this matters beyond the party trick

It’s easy to look at this and think “cute demo,” and it is. But it’s also a good example of a broader idea worth sitting with: you don’t need a heavyweight model for every visual effect that looks like it requires “AI.”

No object detection. No segmentation network. No training data. Just:

• A snapshot
• 21 tracked points (of which only 2 per hand are used)
• Some angle sorting
• A blurred mask
• Standard alpha compositing

The “smart” part of this system — the hand tracking — is a small, well-defined piece doing one job. Everything else is geometry and image compositing that’s existed in graphics programming for decades. The lesson isn’t “don’t use ML,” it’s use it for the part that actually needs it, and plain code for everything else.

Try it yourself

Both versions — HTML and Python — are available here: **https://github.com/AnuragNagare/Ghost-frame**

If you build on it, I’d genuinely like to see what you make. A gesture-triggered version (only activate the reveal when both hands form a specific shape) is the obvious next step, and MediaPipe already gives you everything needed to detect that — you’re just checking distances between the same landmarks you’re already tracking.

If this was useful, a clap or a follow helps more than you’d think. I’ll be writing more on lightweight computer vision projects like this one — things that look like “AI magic” but are really just careful, well-applied fundamentals.