Introduction
In the realm of photo editing, the secret to transforming average images into stunning visuals often lies in selective adjustments. Photographers and enthusiasts alike may find themselves frustrated by existing mobile tools that complicate isolating and enhancing specific image elements. Enter Snapseed, with its innovative interactive on-device segmentation that takes the photo editing experience to the next level.
Challenges in Mobile Photo Editing
Mobile devices have long posed challenges for precise image editing. The limitations surrounding touch input accuracy and computational power make it difficult for users to achieve detailed edits. Users often grapple with traditional tools that offer basic subject or background selections but lack the flexibility and intuitiveness needed for efficient edits.
Introducing the Object Brush
With Snapseed’s new Object Brush, located in the ‘Adjust’ tool, editing specific objects in your photos has become quick and intuitive. Users can now simply draw strokes over the desired object, allowing for targeted adjustments without disrupting the overall image aesthetics. This tool is set to roll out widely, and its initial features are already turning heads in the editing community.
How Interactive Segmentation Works
The heart of the Object Brush is the Interactive Segmenter, a sophisticated AI model that functions entirely on-device. It operates via a simple gesture — a tap or a line trace — enabling users to select any object or person within the frame. Impressively, the model detects and selects the complete object or face in under 20 milliseconds, generating precise segmentation masks that conform to the object’s boundaries. This rapid feedback loop empowers users to refine selections effortlessly, ensuring a seamless editing experience propelled by technologies like MediaPipe and LiteRT.
Training the Interactive Segmenter Model
To foster flexibility in recognizing various objects and scenes, the Interactive Segmenter model incorporates a Big Transfer approach. By utilizing a pre-trained image encoder for pseudo-annotation, the model avoids the labor-intensive process of annotating vast datasets while still achieving a high accuracy in segmentation.
Distillation and Knowledge Transfer
The Interactive Segmenter comprises two key iterations: the Teacher model, which provides high-quality segmentation, and Edge, a streamlined model adept for on-device applications. To train Edge efficiently, knowledge distillation techniques leverage a large dataset of over 2 million weakly annotated images, crafting effective prompts for diverse object categories. This two-model strategy effectively balances performance with mobile processing constraints.
Balancing Quality and Latency
A critical challenge lies in balancing the quality of segmentation versus the need for real-time responsiveness. Snapseed achieves this by separating image understanding from prompt processing. The heavyweight image encoder works independently from the lightweight interactive encoder-decoder to ensure snappy performance, operating within the necessary latency constraints and providing a smooth user experience.
Importance of High-Resolution Masks
For high-quality edits, particularly on large images, it is vital to maintain high-resolution segmentation masks. Snapseed addresses this by training the model to produce masks at a 768×768 resolution which are then upsampled to fit the original image dimensions. This method maintains optimal image quality while keeping latency low by applying high-resolution techniques only after user gestures are complete.
Conclusion
Snapseed’s new interactive segmentation tools have profoundly transformed the way users engage with photo editing. By turning simple touch gestures into detailed selections, Snapseed enables users to see their creative visions come to life. As the Object Brush feature continues to roll out and expand, its applications promise to empower users across a variety of devices, including its recent introduction in the Chromebook Plus 14. Dive into the new features of Snapseed and elevate your photo editing prowess. Download Snapseed for iOS here and stay tuned for future updates.
