RealityFrame

RealityFrame is a Python project that applies real-time visual effects to your webcam feed using hand gestures as the controls. You do not need a keyboard to switch between effects once the app is running. Instead, the camera watches your hands and recognizes specific poses to trigger actions.

The effects include a vintage film look with sepia and grain, a ghost mode that makes you semi-transparent against your background, a thermal camera simulation with heat-map colors, a sketch filter that turns the video into something resembling a pencil drawing, and a clone mode that cuts your body out of the background using a machine learning model and duplicates you across the screen. You switch between effects by pinching your thumb and index finger together. Making a fist activates and deactivates clone mode.

A distinctive feature is the portal gesture. If you form a photographer's frame with both hands (thumbs and index fingers extended at right angles to form a box), a glowing boundary appears between your fingers. After that, whichever effect is active renders only inside that box rather than across the whole screen. Dropping the gesture closes the portal.

When the app first starts, it asks you to step out of the camera frame for a few seconds. This calibration step captures a clean image of your background without you in it, which the ghost and clone effects rely on to separate you from the scene.

The project uses two main libraries: MediaPipe for tracking 21 points on each hand in real time, and OpenCV for capturing video and rendering the effects. Python 3.11 or later is required. Keyboard shortcuts are available as fallbacks for quitting, recording video, and manually switching effects.

The README notes this is a side project and lists a few planned improvements, including hooking in an experimental finger-snap detector and separating the rendering step from the machine learning inference to improve frame rates.