Netflix has developed a technology for realistically removing objects from video

Researchers from Netflix, in collaboration with INSAIT, have introduced a new artificial intelligence model called VOID (Video Object and Interaction Deletion), designed to remove objects from videos while accounting for their interactions with the surrounding environment.

Existing solutions in the field of video object removal typically handle the basic task of "painting over" an object and restoring the background. However, such models do not account for physical interactions. For example, if an object collided with others or influenced the movement of the scene, the resulting video appears unrealistic.

VOID addresses this issue by modeling so-called "counterfactual" scenarios—essentially recalculating how the scene would have unfolded if the object had never existed. This allows for the correct modification of other objects' movements, trajectories, and the consequences of interactions.

The approach is based on a combination of a vision-language model and diffusion-based generation. First, the system identifies areas of the scene affected by the object being removed. Then, a diffusion model generates a new version of the video that maintains physical consistency.

To train the model, developers created a specialized dataset containing both synthetic and real-world scenes where removing an object requires altering the entire logic of interactions. This enabled the model to better handle complex scenarios, including collisions, falls, and changes in object motion.

Testing results show that VOID demonstrates higher realism compared to previous methods, particularly in scenes involving dynamic interactions.

The technology may find applications in the film industry, post-production, advertising, and digital content creation, where precise and realistic video editing is required.

Source: void-model