Meta has unveiled its new AI model, Segment Anything Model 2, or SAM 2. It can track a particular object or subject in a video in real-time and enable users to add visual effects. The company has made this AI model open-source, allowing developers to create video editing tools. Here are the details of Meta’s SAM 2 AI model.
Meta SAM 2 AI Model: Video Editing Made Easy
The SAM 2 AI model uses object segmentation to identify the pixels of the main subject in a video. It automatically creates a different layer for that subject and allows users to add special visual effects while keeping the background of the video unchanged. This concept is also known as masking in traditional video editing.
Here’s an example of how Meta’s SAM 2 AI model works. The sample video shows a person riding a skateboard. The SAM 2 AI model can identify the person’s exact dimensions and differentiate it from the background.
Special VFX can be added to highlight only the person or create an object-tracking pattern around the subject. The entire process happens within seconds using AI, in almost real-time.
Emulating the same visual effects in legacy manual video editing takes a lot more effort and professional-grade skills. Advanced software is needed for such masking techniques. The SAM 2 AI model simplifies the process to a large extent by automating it. It is similar to background removal tools, which were extremely difficult before AI but have now become easier with readily available tools like remove.bg.
Meta’s SAM 2 AI model will be integrated across Meta’s family apps, such as Instagram. Features such as backdrops and cutouts in Instagram previously used the first version of SAM and will soon be upgraded to SAM 2.
It is possible that users could soon be able to edit videos in the Instagram app using AI directly, but Meta has not confirmed any details yet.
Meta CEO Mark Zuckerberg also mentions that the SAM 2 AI model also has applications in the medical field. It can be used to identify bacteria, viruses, and other biological elements to improve medical diagnosis. However, Meta has not disclosed any additional details about the concept yet, possibly due to regulatory approvals.
Meta SAM 2 is Open Source
Meta has made the SAM 2 AI model open source under an Apache 2.0 license. Software developers can implement this AI model to create video editing tools using all of SAM 2’s object tracking and segmentation functions.
This also includes provisional access to the SA-V dataset on which SAM 2 is trained. It contains 4.5 times more videos and 53 times higher annotations than the current largest video segmentation dataset. It has over 6,00,000 masklets that serve as training data, helping the AI model to identify the subject in a video in a more efficient manner.
Meta has not mentioned the availability of SAM 2 in its Meta AI chatbot. We can expect it to be implemented in Instagram and Facebook video and editing tools first.
