Generative edit

Context & Constraints

Paint’s enduring appeal comes from its simplicity. Integrating cutting-edge generative AI meant balancing innovation with this signature ease-of-use:

• Aggressive Timeline: We had to deliver Generative Fill by October 2024 to support the holiday marketing push.
• Existing Framework: We built the AI features into the current selection tool, allowing us to maintain the established Paint experience while working within stringent scope limitations.
• Consistency: Our design decisions also aligned with wider Windows AI patterns, ensuring future scalability and a cohesive experience.

The Challenge

Digital art is booming, yet traditional creation tools often demand expertise and time—barriers that can stifle spontaneity. Our challenge was to Empower users to edit images quickly with minimal friction and ensure that advanced AI tools are approachable for users of all skill levels without altering the familiar Paint interface.

Goals & Objectives

• Enhance Creativity: Improve Paint’s digital art creation experience by leveraging modern generative AI capabilities.
• Shift Perceptions: Improve the perception of Paint to be seen as a more capable digital creation tool.
• Showcase Innovation: Demonstrate the power of Copilot+ and NPU-enabled devices through a high-impact feature set.

Research & Discovery

High level

We had been aware of the power of AI tools for editing for a while. Compared to image generation AI editing tools have been around for a while. Versions of spot removal have been in mainstream image editing apps since the late 2000s. However these new versions of these tools brought more power and capabilities. Inpainting functionality present in diffusion models allowed for categorically differennt content creation tasks.

Competitive Analysis

We evaluated platforms like Adobe Photoshop/Firefly, Canva, Krea.ai, DALL·E, Pixlr, and Clipdrop. Two major approaches emerged:

• Integrated Experience: Embedding AI capabilities within existing selection tools.
• Dedicated Experience: Creating standalone, fluid AI-driven experiences.

Implimentation Generative Fill

One of the biggest key implementation details was that while Generative fill would exist in the current selection tool. This had a lot of benefits. One major constraint was that we were not able to make any updates to the selection tool functionality.

Basic user flow

The design centered on a streamlined two-step process:

• Prompt Entry: Allow users to easily articulate their edit vision.
• Decision: Allow the user to review outputs and decide with minimal distraction.

Exploration: Control placement

One point of consideration for Gen-fill was the pattern that we would employ for the experience would the experience make sense as a control overlayed on the canvas or in a panel. We had used a panel for Cocreator so there was precedent and a consideration of re-using the control and matching the interaction pattern we used text to image generation.

Above: Explorations for generative fill as an on canvas control and incorporated into the Cocreator experience.

After consideration I went with a contextual on canvas control as an experience built into Cocreator would bring additional complexity. Also an on canvas contextual control meets user’s where they are, since selection is one of our most used tools. The on canvas contextual control also tested well in a past Paint research study.

Layout

We had to directions for the design for the layout of the floating control.

• Single Story Design: A wider, dynamic prompt box that remained unobstructive and revealed options only as needed.
• Double Decker Design: Taller, less contextual, based on control used for AI rewrite in notepad.

Ultimately the double decker version was what shipped. The main onus for why was consistentcy with notepad. The thought being that we could build on the control and have a standard implementation across inbox apps. The main reason I advocated for the single story design was that it showed options more contextually the reduced height was much less obstructive.

Floating contextual action menu

The floating action menu was an update that I had been pushing for us to adopt in Paint for a while, so I was happy to see it make it in for Generative fill. It had tested very well in research studies that I ran the year before. With this control in our arsenal we had a more contextual control instead of relying on the toolbar as we had historically.

I successfully advocated for a floating action menu, including define the behavior for how it would behave when constrained to ensure engineering alignment.

Designing Generative Erase

Overview

Generative Erase offers an AI-powered solution for seamlessly removing unwanted elements intelligently blending the removed area with its surroundings. It landed before Generative Fill, allowing us to validate our AI approach early.

Overlay treatment

With that another aspect was the selection state. Historically in Paint we have utilized a dashed outline pattern for selection but this circumstance was very different. Given that the new selection style needed to be distinct to indicate the intended use as well as readable when overlayed over an image.

Collaboration

Partnering with the Microsoft Photos team and our branding/design systems experts brought additional insights, ensuring our selection overlays aligned with emerging Windows AI patterns.