Microsoft Mesh - Immersive Meetings

Mesh is a mixed reality platform built to connect people through 2D/3D Immersive meetings for collaboration and communication. Below is a Short video celebrating Microsoft Mesh’s GTM created by Tendril Studio and Zelig sound

Our mission when building Mesh was to empower people to come together in shared immersive experiences across VR, AR and 2D endpoints

We created Mesh to enable users to establish deeper and higher quality connection with others in hybrid workforces through immersive interaction models.

This was achieved by building a platform that makes it easy to create compelling immersive spaces where people can interact with each other in more authentic and natural ways across all their devices.

Goal:

The main goal was to create and launch engaging activities in Microsoft Teams, Standalone, and Mesh Toolkit to help developers, creators, and businesses improve interaction in and deep co-Prescence within virtual spaces.

My Role:

as a Technical Designer on the Developer Experience team, my role was to work across multiple teams and drive the creation of immersive activities in Mesh. The purpose for the social gatherings environment is to help remote workers feel more connected, reduce social anxiety and lead to increased productivity. My responsibilities was to design, develop and maintain immersive activities for Microsoft Mesh’s Lakehouse Environment in Microsoft Teams. I was also responsible for owning and leading the Toybox package project for the 3rd party Mesh Toolkit, and supporting both the developer community and Microsoft Learn platform.

North Star

Empower hybrid work and remote teams by creating immersive social experiences that make virtual connections feel personal and engaging, The aim is to enhance customer onboarding, increase monthly active usage, and improve customer satisfaction based on feedback.

KPIs

  • Immersive Meetings: deliver immersive experiences on Microsoft Teams at adds differentiated value for meetings, 3D collaboration and social experiences.

  • Developer Experience: deliver efficiency, scalability and quality to Mesh developers and creators via the Mesh Toolkit.

  • Toybox Package: Empower internal and external creators to compose and integrate Mesh activities into 1st & 3rd party environments.

  • Integrate Mesh into M365 Teams for immersive meetings and social gatherings

  • Drive customer engagement by designing interactive tutorial based samples via the Mesh Toolkit

  • Gather customer insights via Social Gatherings and the Toolkit to gauge CSATs and plan for next steps

I worked between the intersections of design and engineering to develop a set of immersive activities aimed to foster social gatherings. Key area workstreams (UX Engineering, Prototyping, and Shipping GTM features).

Technical Challenges

  • While developing immersive activities, I encountered endless technical problems, such as platform breaking bugs, feature regression, conflicting priorities, limitations of the codebase, managing multiple dependencies, and working within constraints while sim shipping different versions to Early Adopters and Technical Adopter customers. However we we’re able to work efficiently across multiple channels to find stop-gap solutions and workarounds.

  • Also as a large group of teams working together on the same repository and branch, we we’re confronted with the challenges around developing an environment that allows 16 or more users in each session, with activities that are synchronized across clients which run smoothly at 72 frames per second across different devices (PC, Web, and Meta Quest headsets).

  • While developing I had to ensure each visual script was no larger than 3ms in size to retain optimal performance. I needed to improve workarounds and refactorization of each activity multiple times due to the platform's limits.

Design Challenges

  • Define what it means to enrich remote connections within virtual environments, and how might we create memorable experiences in Mesh that are fun?

  • Cross-team collaboration when polishing each activity for production ready assets.

  • How to support multiple versions of activities and users via the Mesh Toolkit.

We needed to design each activity with the "Participant," "Creator," and "Organizer" in mind to ensure they were relevant for the Lakehouse. However these categories didn't exist at the start of the project, only after prototyping and adjusting, it became clear that each activity had to change to suit the customer needs.

Early Prototyping

Before diving deep into the activities that landed inside Teams, I created a series of immersive games that promotes team building exercises, icebreakers and subtle environment interactions. Each activity was designed for 2D and 3D endpoint’s (PC, Web and Meta Quest VR Headsets).

Social Gathering activities were integrated within MS Teams and the 3rd party Mesh Toolkit via the Toybox package in Github. Utilizing the Toybox package, we were able to support both 1st party environments and 3rd party developers simultaneously.

Throughout this project (and afterwards) I was in charge of building, shipping, and maintaining all activities in the Toybox package while collaborating with different teams to provide ready-to-use assets. All activities are designed for PC, Web, Quest, and Apple Vision Pro (AVP).

After gaining the greenlight from the creative director to run with 5 immersive activities, The environment’s team began creating the Lakehouse environment for activities. During the process, the Havok Physics team began migrating multiple technical systems simultaneously. This created additional challenges due to limitations on current interactions and physics codebase. Our team had to develop various solutions to build functional prototypes and maintain their usability while the systems changed.

Iterations & Production

While this added additional complexity during testing and development, it helped us find issues in the current code. Ultimately empowering the Physics team to improve the code, fix major bugs, and focus on core physics logic. This also encouraged the engineering team to migrate their current system and integrate Unity’s Visual Scripting and allow-list methods, which made it possible to use visual scripting for safer development across the 1st & 3rd party toolkit ecosystem.

Customer Feedback & Insights

Final Solution

Outcome & Success Metrics

  • Globally shipped Immersive Activities in Mesh For Teams & 3rd Party Toolkit

  • Immersive activities highlighted directly in Satya Keynote - Microsoft Ignite 2023

  • Partnered with M365 Developer Channel to empower world building & immersive experiences

  • Immersive activities and Toybox package featured in Microsoft Build 2023 / 2024

  • Project fueled Microsoft Mesh & Meta Partnership to deliver Immersive Experiences

  • Successfully onboarded 90+ customers through Early Adoption, Private Preview, Public Preview and GA

  • Gathered customer insights to improve Mesh Toolkit samples and packages

  • Successfully maintained and shipped new features on a 2 week release cadence in the Mesh toolkit after public release. Targeting new sample features, tooling and bug fixes

Since our team was prototyping within the constraints of predefined component classes built within the Shared Physics system and code exclusion enabled, we had to use what was readily available without writing custom C# logic, otherwise the code would become stripped after getting compiled at runtime.

  • Social Gathering’s helps our teams feel more connected

  • Would like “low code, no code” solution for object based experiences

  • 3rd party Toolkit is useful for Unity developers and technical artists, but enterprises find it costly to hire a studio for customization instead of having 1st party tools integrated within the Mesh App itself.

Next

Microsoft Mixed Reality - Content Design