Cocreator
Date
2022-2024
Role
Product design, Interaction design, Visual design
Description
Building a Gen-AI assisted creation experience in Paint.
Introduction
Overview
Conceptually, Cocreator is an experience that allows users to create visually at a high level without an advanced skillset within Paint’s approachable experience. It uses Stable Diffusion to take an initial sketch and a prompt to produce an output image.
Above: Screen recording of the current Cocreator experience in Paint
Team & Collaboration
For the major Cocreator work I worked with a Principal designer for Paint and inbox apps. We worked together through iteration, exploration, and refinement of what would become Cocreator, and with a Senior designer on follow-up updates to Cocreator.
Goals & Objectives
Enhance creativity
Improve Paint’s digital art creation experience by leveraging modern generative AI capabilities.
Shift perceptions
Improve the perception of Paint to be seen as a more capable digital creation tool.
Showcase Innovation
Demonstrate the power of Copilot+ and NPU-enabled devices through a high-impact feature set.
Challenge
Digital art is as big as it has ever been, and users are constantly seeking tools that enable them to create quickly and creatively without a steep learning curve. Traditional digital creation methods can be time-consuming and require a level of expertise that casual users might not possess. These perceived and experienced obstacles can prevent would-be creators from accomplishing their creative goals.
Context
This track of work was part of a broader effort to improve Paint. It was also at the forefront of efforts to showcase generative AI capabilities in Windows. It was also an example of a feature built for Copilot+ PCs, the next generation of Windows devices built to run powerful local models.
Copilot menu
The Copilot menu update was mainly a branding update focused on aligning with the Microsoft-wide direction of having all Generative AI features folded into Copilot. It ended up testing well and improving discoverability of Generative erase and Cocreator.
Research & Discovery
Research came in multiple forms over the course of the project. Text based image generation was at the start of the work a very new pattern. So while normally I am able to leverage more competitive analysis and existing heuristics, for Cocreator it felt like we were very much learning as we go at a fast pace.
Competitive Analysis
Competitive Analysis
Where possible, we evaluated platforms like Adobe Photoshop/Firefly, Canva, Krea.ai, DALL·E, Pixlr, and Clipdrop. Two major approaches emerged:
Integrated Experience:
Embedding AI capabilities within existing tools and experiences.
Dedicated Experience:
A standalone fully contextual flow AI-driven experience built around a specific task.
Magic Paint
One interesting flashpoint was a viral trend showing Stable Diffusion being used in combination with crude MS Paint drawings. Beyond being a technical demo it showed how Paint’s un-intimidating simplicity lent itself towards these new creators coming from a non-digital art background.
With these exciting developments and the release of the stable diffusion model as open source I was able to work with a design engineering colleague in a hackathon on a prototype derived from the concept. And with that “Magic Paint” was born.
Above: An early viral image to image demo using stable diffusion
The Magic Paint demo became very popular and was shared with colleagues across Windows. With an overall momentum building for AI at Microsoft I was able to explore further what these new experiences could look like in Paint.
Above: An early exploration of Magic Paint which would become Cocreator done during an internal Hackathon
Magic Paint had another breakthrough when generation speed began to improve; it soon became clear that “real-time”(continuous generation) was something that was going to be possible. With that I hacked together a demo to show a potential real-time experience for an internal audience.
Users responded well to demos and prototypes of Paint updated with AI capabilities. Existing research had also shown that users were interested in doing more creative tasks in Paint.
Implementation
There were two big stages for launching Cocreator. Stage one when Cocreator first launched it was text to image powered by DALL·E. Stage two was when Cocreator was updated to be image to image and run on Copilot+ devices. The text to image feature was then rebranded as Image Creator. There was friction but some insights that we were able to derive from this unorthodox release.
Image Creator
Image Creator
First steps with Gen AI
Our text to image experience launched first as Cocreator though it would be renamed Image Creator. We had hoped to launch with the full image to image experience but this ended up being impossible by the deadline. However there was a lot of insight derived from the experience shipping to a general audience.
Style selection
Style selection was an area where the experience was able to go beyond the prompt box based experience. One thing that was an interesting facet of image generation was that describing an image style was useful for better and more consistent results and it was particularly difficult for people to describe a visual style in writing. Given that, I was a big fan of a style selection menu as a way to simplify the process with a visual cue, without limiting the creative potential of the prompt box.
Above: Style selection in Cocreator
Preview and selection
I felt that true active co-creation would require minimal active management. A single action to confirm and add to the canvas. While the user is creating what they are interested in on the canvas they should not have to worry about managing the panel.
Above: Selection of wireframes made during explorations of panel experience for iterative cocreation
Design iteration
There were several points of iteration for the Cocreator design across development but one of the most notable was the V2 version which was built off of user feedback, and telemetry from the V1 experience.
Above: Version comparison of Cocreator V1 and V2
Cocreator V1 experience
With that I felt that true active co-creation would require minimal active management. A single action to confirm and add to the canvas. While the user is creating what they are interested in on the canvas they should not have to worry about managing the panel.
Above: Cocreator panel preview for design September 2024 release
The final control put a higher emphasis on the preview above all other elements to allow passive review while painting. The hover preview interaction would allow the user to quickly view the generation on the larger canvas without having to switch context.
Problems V1
Problems V1
After launch Cocreator had some issues that we identified via telemetry and customer feedback.
Issue 1: Apply image drop off
After launch, we discovered that there was a huge drop-off (50%) of users who were not applying their generated images to the canvas.
Issue 2: Iterate
We also received feedback that users wanted to more quickly iterate their output without drawing.
V2 Solution
V2 Solution
A refined slider control
Offers a larger range of sizes while giving precision in smaller sizes and wider increments in larger sizes, adapting to the natural sensitivity required at different scales.
A preview indicator
Offers immediate feedback on brush size in the context of the image.
Updated experience results
The V2 experience was a big success. It addressed the issues identified in V1 and maintained the core appeal while improving consistency within the generative AI experiences.
Discoverability and user education
One area where we saw a potential need and opportunity was the area of user education and discovery within Paint. Generative AI patterns are new to most users and there were a few aspects of the interaction that were useful to communicate. Also we had a desire to drive discovery of these new capabilities.
Above: Cocreator first run experience dialog
This took us towards implementing a first run experience. While there are many avenues for addressing our core goals a conventional modal view aligned with our initial needs of discovery and education of the feature. For the FRE modal we knew that we would have limited user bandwidth so we focused on communicating the need for a prompt, the function of selecting a style, and the iterative process. Beyond the content we also wanted to use visuals to build excitement for the experience.
Conclusion
Impact of updates
Image Creator, the text to image experience that first launched as Cocreator, has generated over 10 billion images. There was a lot more talk about Paint than in recent history, and one aspect became clear. Paint Cocreator along with the Creator updates, helped build an improved impression of the app. Colleagues on the Surface team spoke about people using Cocreator and connecting with the Cocreator feature and giving feedback. Paint Cocreator was called out by CEO Satya Nadella during an interview with the Wall Street Journal covering the launch of Copilot+ PCs. The functionality and Paint are featured prominently across all Windows social media channels.
Above: Select feedback from users on the new features
Paint Cocreator was called out by CEO Satya Nadella during an interview with the Wall Street Journal covering the launch of Copilot+ PCs.
Above: CEO Satya Nadella talking about Paint Cocreator during an interview with the Wall Street Journal
Final thoughts
Paint’s superpower is how un-intimidating it is. This approachability allows it to be a tool for creativity for those who would otherwise have difficulty with an advanced creative tool. With powerful AI capabilities, what was once an advanced workflow is now within reach. Patterns for Generative AI tools have come a long way but are ultimately still very new. The extent to which people navigate these patterns needs to be better understood. As Cocreator has started to be delivered to larger audiences we are focusing on next steps for Cocreator to refine the experience, drive feature discoverability, and build stickiness.