Muse AR


For this capstone project we were tasked with creating an augmented reality system and incorporating it into a museum space. Our solution was ultimately to create a system that could work across different museums


7 weeks at the time of project


Figma, Pencil and Paper, Blender, Adobe Photoshop, and E-Cycles Renderer


Team Project working with Kamaria Daniel
UXUI, product design, user testing, prototyping
Framing the problem
"How might we build a framework that can improve the museum experience across a diverse range of ambitious pieces?"
How often have you been to a museum, looked at an interesting piece and wanted to know more context about it? If you are like 80% of the people we surveyed then you have had this experience. Through this and other insights we built the Muse AR system as a means of utilizing augmented reality to enhance the museum experience to make it more immersive, more informative, and more accessible.
It was important to start with the museum's core goals. What is their mission? What do they hope to achieve? What difficulties do they have? How do they define these?
For our process we followed the Design Thinking framework when approaching the our solution. Given the less defined status of AR design we hoped a somewhat fundamental process could deliver fundamental results. We broke our time down to roughly 1-2 weeks per step in the process.
We dedicated time in the define stage to build a better understanding with personas to define aspects of the audience and created a customer journey that could reflect the AR experience and show us opportunities.
Key design decisions
Hardware component
Overall the UI design for the Main TV was dictated by to main motivations. First to make the TV as functional as possible of course. The other main motivator in the visual design was to move the design closer to the Fluent Design System. In my reading of the landscape of current TV UIs they for the most part seemed very homogenous. Given that I opted to stay close to the existing patterns to make sure that it was able to match user’s expectations and mental models. So that meant; organizing content into rows of content tiles and top bar navigation.My next goal involved using fluent design for the UI. While the design resources for fluent are very expansive they did not seem to me as having much coverage for designs with a very large display. I used the UI for xbox as a reference however that seemed somewhat divergent from other versions of Fluent I found in my research
When defining our gestures we elected to go with existing patterns for interaction that mapped to existing experiences like using a touchscreen. For example patterns like pinch to zoom and scrolling were used when defining our gestures.
Way finding
Another key design decision was how way finding was handled. Our solution was the illuminated path which unobtrusively creates a visual path on the floor showing the way. A highlight is also given to the destination to avoid confusion and provide a above eye level option if preferred or otherwise needed,
UI Background treatment
 One key design decision was a “blurred glass” style background for any major interface elements. This was done for the ability to at least partially show the background environment while a menu or other interface element is present and still maintain contrast for text. The blurred effect allows drastically more contrast with complex high contrast backgrounds than can be done with just            opacity. This can help to minimize any risk from the rare time where an interface can significantly obstruct the user's vision. With this treatment we can still allow the user to perceive elements of the environment and moving objects through in the background. The UI also self adjusts to maintain contrast in different lighting/background conditions.
We also sought to prototype on-boarding experience. Specifically we designed for a dedicated area at the entrance of the museum. After receiving the headset the user is prompted to complete 3 simple tasks in the staging area. The first step is to demonstrate navigation. The user is prompted and shown how to get directions to the bathroom. For the second step the user is prompted to get more information about an exhibit. And finally the user is prompted to collect a virtual trophy they are then advised that more trophies are available through out the museum at different locations. The staging area was designed to accomplish several important tasks. Introduce the headset in a location near the start where an attendant could advise the patron, and assist if needed in the early stages.
For testing we wanted to focus on our visual interface and hoped to get an understanding of any issues with navigation. For the format we did task based user testing where we provided the respondent with a task and asked how they would complete the task using the given interface. After this round of testing the results showed that overall the interface was able to be navigated without major issue. Feedback we got indicated that the interface elements built using a “touch screen inspired mental model were particularly effective.

Contextual modal

For the slide in controls I tried a few variants that did not work in a mobile format and eventually decided to make the menu contextual.

Example of a contextual modal that shows up when a show is being played left.
Also an example of a dial-pad variant for selecting a channel on traditional over the air/cable tv right.

Onboarding flow

I was also tasked with creating an onboarding flow for the Smart TV experience. It was expected to be led by our research.

Sign In

The first task for this project would be sign in. The tasks, make sign in easy, incentivize use of a Microsoft account for those with one already, incentivize creation of a Microsoft account.


In this step we have privacy selection. All are checked by default the user can choose to opt out of using the camera, microphone, and analytics.

Connected Home

The third step is connecting the user’s IoT connected home devices. This is a key feature but also particularly hypothetical. Currently for the most part there is not a uniform system to connect and manage across IoT devices.  
Given that after research I found that most devices activate via connection links and the most reasonable way to have the user parse that would have the user enter their links and manage the linking aspect using the given information.

Account Auto Sign-in

The forth step for on-boarding is having the user the option to sign into any services that they would like at that time or sign in later as needed since they would be given an option to skip sign-in to proceed later as they need.


Looking back at this project I enjoyed stepping out of my immediate sphere of reference and other ways to approach.

I found my research to be interesting and illuminating. I have been aware of voice design in the past but the more in depth research really shed light on the subject.

I found the problem itself very challenging. I think that any level of trust can be difficult to establish. For a large company like Microsoft building trust is hard under normal circumstances. But with a voice activated listening device that only makes things more challenging.  

As to the design work and my solution it is hard to evaluate completely since a voice based TV UI is not something that I can really user test. However the parts of the design process that were able to worked through I was happy with the results.


For me the first part of research for the project was around  figuring out what makes a successful voice based assistant. Understanding this would be key to how I would approach my solution.

What problems does a voice assistant solve? What is a voice assistant ill-equipped to solve? What are people’s expectations of a voice assistant are they radically different from the capabilities?  What is the current market and what approaches are other’s taking?

After some table top research I was able to get a more defined idea of this and was able to boil down the information into the following...

Some common patterns I incorporated into my design were;

  • The ability to understand unique queries.
  • The ability to interconnect/access to user’s existing services and devices.
  • The handling of failstates/impossible requests.
  • The perceived speed in response.
  • Privacy


For the ideation phase one area that I found challenging was the aspect of designing for user trust in a voice activated smart device. Thinking about trust I made a few key design decisions...

Key design decisions

Camera and microphone array

Camera and Microphone arrays. In the pursuit of designing for trust I was think of ways that trust could be shown or built in to a device or service. I thought of a rumor that I had read that the small green indicator light was hard wired to the camera on macbooks and that the camera would only be active if the light was on. After looking it up that is evidently only partially true, but the idea of a hardwired solution with a built in indicator interested me.

So I thought what would that look like on a TV? Could I make it more clear or fit better?

So my idea came towards a retractable Camera/Microphone array. When the array is down in the chassis it’s power circuit is physically disconnected from the TV showing and reinforcing a commitment to privacy into the physical interaction of the design.

UI Name Indicators

All portions of the on screen interface have labels indicating them. The format (65” High resolution screen) allows plenty of real-estate as well as flexibility of sizing while maintaining readability depending on distance from the screen.

Connected  home

To address the problems of not using connected home features, I elected to add connected home indicators consisting of icons and text showing devices that are connected. My rational being that having the name and a standing in picture of the device would remind them and “put the words in their mouth” so that they could easily make a quire.

Fail State response

In my initial research as to what made a good voice Ai the ability to handle unique queries was always mentioned as a primary concern.

However when looking at available voice services it seemed as though the fail state question was to an extent a solved problem. When looking at all of the major options an unintelligible query returned a response of “I am sorry I didn’t quite get that” almost verbatim across the platforms.

Given that my solution goes that route however I felt like this would also potentially be a good chance to emphasize the tasks feature.

When faced with  a query that is not able to be parsed the assistant will try to detect key words and search against the library of tasks in the event that it is not able to help with the request.


Physical design iteration

I felt that the design could be improved after review, so after reviewing a bit of relevant information on consumer industrial design and other Microsoft products I decided to make a few changes.

First off I decided to incorporate Microsoft’s existing design language into the design, especially considering that this would existing with other devices like the xbox out in the real world.

Second and importantly I decided to make the design serve the goal of designing for trust.
The previous iteration had a lot of harsh hardened edges that were counter productive towards the goal of trust. In commercial product design rounder softer shapes read as more friendly.

The other change was coloration. While the previous charcoal colored design has it’s own appeal it read as a bit intimidating.

The new eggshell color made the design read more like a consumer product that lives in the home.

TV UI Design

Overall the UI design for the Main TV was dictated by to main motivations. First to make the TV as functional as possible, since I would not be able to do testing I would have to rely completely on patterns in other products.

Some common patterns I incorporated into my design were;

  • Top bar navigation with large type
  • Dark Background for comfort while watching at night/creating a theater analog
  • Organizing content into rows of content tiles
  • Top first hierarchy of content tiles, with a Hero show/movie or carousel

The other main motivator in the visual design was to move the design closer to the Fluent Design System. While the design resources for fluent are very expansive they did not seem to me as having much coverage for designs with a very large display. I used the UI for xbox as a reference however that seemed somewhat divergent from other versions of Fluent I found in my research.

Mobile Design/Remote App

For the mobile app I broke the app into three main pages via the tab bar.

I added a remote button that when tapped reveals the standard controls. For ease of one handed use the remote is able to be accessed by sliding in from the right edge and sliding to the right to conceal. I decided to keep both options for accessing the remote since it is a core function of the app and discoverability and ease of one handed use were important to maintain.

Project Goal

To design a voice enabled Smart TV experience that is considerate of the brand's goals.

Framing the problem

"How might I design a Smart TV that builds trust and accomplishes the brands goals?"


For this project the goal was to design a voice enabled smart TV experience and pitch it. The designed product would be made for Microsoft. The target market was younger early adopting tech enthusiast.

Assignment considerations

  1. Design should use the Fluent Design System
  2. Design for user trust in mind
  3. Include user onboarding flow
  4. Create a video showing and pitching the solution

Microsoft TV

User Experience/Interface Design, Speculative Design, Product Design, Motion Design.