Generative edit
Date
2024/2025
Role
Product design, Interaction design, Visual design
Description
Integrating breakthrough generative AI capabilities into Microsoft Paint.
Introduction
Overview
At a high level, this track of work was a process of integrating breakthrough generative AI capabilities into Paint. The shipped editing features were Generative fill, Object select, and Generative erase. It was part of larger efforts to revitalize Paint as a creative tool and incorporate powerful NPU-powered AI tools across surfaces in Windows as part of Copilot.
Team & Collaboration
I served as the primary designer, working closely with product managers and engineers from the initial exploratory phase through to launch. Our tight collaboration was essential, especially since these features were central to Windows 11 & Copilot+ PCs’ holiday campaign under an aggressive timeline.
Context
Paint is a widely used digital art and image editing app cherished by Windows users for its simple interface and approachable creative tools. To be competitive in the landscape of creative tools and cater to the evolving needs of our users, we embarked on a project to integrate cutting-edge generative AI features into Paint. Our aim with these updates is to enhance the creative capabilities of Paint, making powerful image editing more accessible and efficient for users of all skill levels.
Generative fill and Generative erase updates came in fall 2024 with the major constraint that there was a tight time frame to deliver Generative fill in time for October. Generative fill was one of the “holiday 8,” a selection of eight features marketed as part of a wider effort to market Windows 11 and Copilot+ PCs for the 2024 holiday season. This also entailed working with a tight scope for the feature. These updates came as part of a series of updates adding generative AI to Paint with the aim of differentiating with AI. Object select went live in April of 2025.
Working timeline for generative edit features
History
Paint is 40 years old and has a lot of goodwill. It has a well-defined perception as an approachable basic digital art app. However it also has a reputation as too basic/limited for serious digital art.
Windows 1
Windows 3
Windows 95
Windows 98
Windows XP
Windows Vista
Windows 7
Windows 11
Goals & Objectives
Enhance creativity
Improve Paint’s digital art creation experience by leveraging modern generative AI capabilities.
Shift perceptions
Improve the perception of Paint to be seen as a more capable digital creation tool.
Showcase Innovation
Demonstrate the power of Copilot+ and NPU-enabled devices through a high-impact feature set.
Shipped solution
Our solution shipped as three features. Object select allows users to instantly select elements with a single click, Generative erase allows users to remove elements from an image, blending with the background, and Generative fill allows users to seamlessly add elements to an image using a prompt.
Research & Discovery
High level
We had been aware of the power of AI tools for editing for a while. Compared to image generation, AI editing tools have been around for some time. Versions of spot removal have been in mainstream image editing apps since the late 2000s. However, these new versions of the tools brought more power and capabilities. Inpainting functionality present in diffusion models allowed for categorically different content creation tasks.
Competitive Analysis
We evaluated platforms like Adobe Photoshop/Firefly, Canva, Krea.ai, DALL·E, Pixlr, and Clipdrop. Two major approaches emerged:
- Integrated Experience: Embedding AI capabilities within existing selection tools.
- Dedicated Experience: Creating standalone, fluid AI-driven experiences.
Above: Screenshot of comparable experiences for existing inpainting/Generative-fill implementations
Research & Discovery
Features
Generative edit
Within our purview, three editing-focused features allowed for the greatest range of editing tasks: Object select, Generative erase, and Generative fill. Combined, these would allow a user to precisely add, remove, or edit elements in an image.
Implementing Generative Fill
Above: Recorded prototype of primary flow for Generative fill in Paint
One of the biggest implementation details was that Generative fill would live in the existing selection tool. This had a lot of benefits. One major constraint was that we were not able to make any updates to the selection tool functionality.
Basic user flow
The design centered on a streamlined two-step process:
- Prompt Entry: Allow users to easily articulate their edit vision.
- Decision: Allow the user to review outputs and decide with minimal distraction.
Above: Diagram of user flow for Generative fill.
Exploration: Control placement
One point of consideration for Generative fill was which pattern to employ: would the experience make sense as a control overlaid on the canvas, or in a panel? We had used a panel for Cocreator so there was precedent and a consideration of re-using the control and matching the interaction pattern we used for text to image generation.
Above: Explorations for Generative fill as an on-canvas control and incorporated into the Cocreator experience.
After consideration I went with a contextual on-canvas control, as an experience built into Cocreator would bring additional complexity. An on-canvas contextual control also meets users where they are, since selection is one of our most used tools. The on-canvas contextual control also tested well in a past Paint research study.
Output preview
One design detail is how I went about handling generated outputs. Many inpainting/generative fill implementations that I reviewed favored preview image tiles for output selection. I felt that they were not very effective in practice when testing them. To start, for most types of generation it was impossible to discern details/differences between the outputs. Also the output is inherently contextual to the content around it. The thumbnail treatment does not indicate how well the generated content blends or contrasts in the actual image.
Above: Animated demo showing cycling through multiple generative fill outputs
Layout
We had two directions for the layout of the floating control.
- Single Story Design: A wider, dynamic prompt box that remained unobstructive and revealed options only as needed.
- Double Decker Design: Taller, less contextual, based on the control used for AI rewrite in Notepad.
Above: Comparison of proposed “Single story” design and shipped “Double decker” design
Ultimately the double decker version was what shipped. The main reason was consistency with Notepad; the thought was that we could build on the control and have a standard implementation across inbox apps. The main reason I advocated for the single story design was that it showed options more contextually and the reduced height was much less obstructive.
Floating contextual action menu
The floating action menu was an update that I had been pushing for us to adopt in Paint for a while, so I was happy to see it make it in for Generative fill. It had tested very well in research studies that I ran the year before. With this control in our arsenal we had a more contextual control instead of relying on the toolbar as we had historically.
Above: Defined behavior for contextual action menu when space is constrained.
I successfully advocated for a floating action menu, including defining how it would behave when constrained to ensure engineering alignment.
Designing Generative Erase
Overview
Generative Erase offers an AI-powered solution for seamlessly removing unwanted elements, intelligently blending the removed area with its surroundings. It landed before Generative Fill, allowing us to validate our AI approach early.
Above: Screen recording of shipped experience for Generative erase
Overlay treatment
Another aspect was the selection state. Historically, Paint has used a dashed outline pattern for selection, but this circumstance was very different. The new selection style needed to be distinct to indicate the intended use as well as readable when overlaid on an image.
Above: Animated overlay used in Paint’s Generative erase feature
Collaboration
Partnering with the Microsoft Photos team and our branding/design systems experts brought additional insights, ensuring our selection overlays aligned with emerging Windows AI patterns.
Object select
Hover overlay
The selection overlay that we went with was an evolution of the existing pattern we used for Generative erase. On-image overlays are difficult in that they need to work on a wide array of backgrounds while allowing for comparison of the mask, foreground, and background simultaneously. For instance, while making the foreground opaque would contrast well and vice versa for the background, it would only be doing part of the job since both need to be evaluated simultaneously to be effective. The overlay is also contextual in that it is only shown while the user is creating the initial selection.
Above: Parts of the updated hover overlay for object select hover
Touch interaction
One interesting aspect of design for automatic selection is the interaction with touch. For touch there is no hover indicator that previews the selectable element. Other implementations address this in a few ways; the most common pattern is a "glimmer" style animated state when the initial selection is made to indicate the created mask. For example, Apple’s background removal animates to emphasize the edge of the selection. Other implementations of this pattern indicate more aspects of the selection such as convex areas within the mask as well as partial blended transparency.
Above: Animated demonstration of touch drag interaction for object select
The solution that I landed on was tap and drag to select with the selection being created on pointer up. This had a lot of benefits in that it maintained the simple tap selection and did not add any additional controls while still allowing control over the selected elements.
Sub-element selection
One interesting aspect of designing for this interaction was what I called sub-element selection. That is, a valid selection target can be composed of valid selection targets. For instance, in this image the wrap may be the intended target or all the food on the plate, or the plate and all the food on it.
Above: Animated example of sub-element selection of a plate of food
Conclusion
Impact
These updates to Paint have been successful in changing Paint’s perception, moving it towards a more capable creative tool. These updates have been featured in major tech publications like The Verge, Forbes, Creative Bloq, Fast Company, TechSpot, and Digital Trends.
“Microsoft Paint, once a joke, could be the future of image editing - Fast Company”
“2023's most improved app - Laptop Mag”
“Microsoft Paint is back from the dead, and we’re as surprised as you - Creative Bloq”
Above: Select feedback from users on the new features
Beyond the new interest and major capabilities, even with some of the biggest changes in the app’s history, Paint has the highest Net Promoter Score among inbox apps, maintaining its reputation as the classic accessible painting app.
Final thoughts
Looking ahead, I think the generative edit features have built on past updates to make a compelling experience in Paint. With these more ambitious capabilities, a new benchmark for Paint is being set.