Generative edit

Context

Paint is a widely used digital art and image editing app cherished by Windows users for its simple interface and approachable creative tools. To be competitive in the landscape of creative tools and cater to the evolving needs of our users, we embarked on a project to integrate cutting-edge generative AI features into Paint. Our aim with these updates is to enhance the creative capabilities of Paint, making powerful image editing more accessible and efficient for users of all skill levels.

Generative fill and Generative erase updates came in Fall of 2024 with the major constraint that there was a tight time frame to deliver Generative fill in time for October. Generative fill had been put as one of the holiday 8 a selection of 8 features to be marketed for Windows as part a wider effort to market Windows 11 and Copilot+ PCs for the 2024 holiday season. This also entailed working with a tight scope for the feature. These updates came as part of a series of updates adding Generative AI updates to Paint with the aim of differentiating with AI. Object select went live in April of 2025.

Working timeline for generative edit features

History

Paint is 40 years old and has a lot of good will. It has a well defined perception as an approachable basic digital art app. However it also has a reputation as too basic/limited for serious digital art.

Windows 1

Windows 3

Windows 95

Windows 98

Windows XP

Windows Vista

Windows 7

Windows 11

Goals & Objectives

Enhance creativity

Improve Paint’s digital art creation experience by leveraging modern generative AI capabilities.

Shift perceptions

Improve the perception of Paint to be seen as a more capable digital creation tool.

Showcase Innovation

Demonstrate the power of Copilot+ and NPU-enabled devices through a high-impact feature set.

Shipped solution

Our solution shipped as 3 features. Object select allows users to instantly select elements with a single click, Generative erase allows users to remove elements from a image blending with the background, and Generative fill allows users to seamlessly add elements to an image using a prompt.

Research & Discovery

High level

We had been aware of the power of AI tools for editing for a while. Compared to image generation AI editing tools have been around for a while. Versions of spot removal have been in mainstream image editing apps since the late 2000s. However these new versions of these tools brought more power and capabilities. Inpainting functionality present in diffusion models allowed for categorically differennt content creation tasks.

Competitive Analysis

We evaluated platforms like Adobe Photoshop/Firefly, Canva, Krea.ai, DALL·E, Pixlr, and Clipdrop. Two major approaches emerged:

• Integrated Experience: Embedding AI capabilities within existing selection tools.
• Dedicated Experience: Creating standalone, fluid AI-driven experiences.

Research & Discovery

Features

Generative edit

With our purview 3 editing focused features allowed for the greatest range of editing tasks; Object select, Generative erase, and Generative fill. Combined these would allow a user to precisely add remove or elements from an image.

Implimentation Generative Fill

One of the biggest key implementation details was that while Generative fill would exist in the current selection tool. This had a lot of benefits. One major constraint was that we were not able to make any updates to the selection tool functionality.

Basic user flow

The design centered on a streamlined two-step process:

• Prompt Entry: Allow users to easily articulate their edit vision.
• Decision: Allow the user to review outputs and decide with minimal distraction.

Exploration: Control placement

One point of consideration for Gen-fill was the pattern that we would employ for the experience would the experience make sense as a control overlayed on the canvas or in a panel. We had used a panel for Cocreator so there was precedent and a consideration of re-using the control and matching the interaction pattern we used text to image generation.

Above: Explorations for generative fill as an on canvas control and incorporated into the Cocreator experience.

After consideration I went with a contextual on canvas control as an experience built into Cocreator would bring additional complexity. Also an on canvas contextual control meets user’s where they are, since selection is one of our most used tools. The on canvas contextual control also tested well in a past Paint research study.

Layout

We had to directions for the design for the layout of the floating control.

• Single Story Design: A wider, dynamic prompt box that remained unobstructive and revealed options only as needed.
• Double Decker Design: Taller, less contextual, based on control used for AI rewrite in notepad.

Ultimately the double decker version was what shipped. The main onus for why was consistentcy with notepad. The thought being that we could build on the control and have a standard implementation across inbox apps. The main reason I advocated for the single story design was that it showed options more contextually the reduced height was much less obstructive.

Floating contextual action menu

The floating action menu was an update that I had been pushing for us to adopt in Paint for a while, so I was happy to see it make it in for Generative fill. It had tested very well in research studies that I ran the year before. With this control in our arsenal we had a more contextual control instead of relying on the toolbar as we had historically.

I successfully advocated for a floating action menu, including define the behavior for how it would behave when constrained to ensure engineering alignment.

Designing Generative Erase

Overview

Generative Erase offers an AI-powered solution for seamlessly removing unwanted elements intelligently blending the removed area with its surroundings. It landed before Generative Fill, allowing us to validate our AI approach early.

Overlay treatment

With that another aspect was the selection state. Historically in Paint we have utilized a dashed outline pattern for selection but this circumstance was very different. Given that the new selection style needed to be distinct to indicate the intended use as well as readable when overlayed over an image.

Collaboration

Partnering with the Microsoft Photos team and our branding/design systems experts brought additional insights, ensuring our selection overlays aligned with emerging Windows AI patterns.

Object select

Hover overlay

The selection overlay that we went with was an evolution of the existing pattern we used for generative erase. On image overlays are difficult in that they need to work on a wide array of backgrounds while allowing for comparison of the mask, foreground, and background simultaneously. So for instance while making the foreground opaque would contrast well and vice versa for the background, it would only be doing part of the job since both need to be evaluated simultaneously to be effective. The overlay is also contextual in that it only temporary for while the user is creating the initial selection.

Touch interaction

One interesting aspect of design for automatic selection is the interaction with touch. For touch there is no hover indicator that previews the selectable element. There are a few ways to other implementations attempt to address this, the most common pattern is a "glimmer" style animated state when the initial selection is made to indicate the created mask. For example this is what apple does when their background removal where they animate to emphasize the edge of the selection. Other implementations of this pattern indicate more aspects of the selection such as convex areas within the mask as well as partial blended transparency.

The solution that I landed on was tap and drag to select with the selection being created on pointer up. This had a lot of benefit in that it maintained the simple tap selection and did not add any additional controls while still allowing control over the selected elements.