Frontline workers are the backbone of many industries, such as manufacturing, heavy industrial, healthcare, retail and more. They operate and maintain complex equipment on the field and perform essential services to keep things operational and running. However, they often face challenges such as skill gaps, missing details or having difficulty finding or getting the right information and procedures when they need it. To help them overcome these challenges, Microsoft introduced Copilot in Dynamics 365 Guides, a new tool that combines generative AI with Mixed Reality to provide interactive guidance and on-demand information to frontline workers. Based on the information available, let me share with you what Copilot in Guides is, how it works, and why it is important for frontline workers and businesses.
Microsoft introduced Copilot in Dynamics 365 Guides during Microsoft Ignite 2023 -event.
- What is Copilot in Dynamics 365 Guides?
- How can Copilot in Dynamics 365 Guides help frontline workers?
- How about businesses?
- Conclusion
What is Copilot in Dynamics 365 Guides?
Copilot in Dynamics 365 Guides is, or will be since it isn’t available yet, a smart AI tool that is designed to assist frontline workers in industrial settings with complex equipment and processes. Copilot enables workers to ask for relevant information using natural language and human gestures, such as looking, pointing or touching. Copilot uses Artificial Intelligence to search for information from various sources it has, like technical documentation, service records, training content, and any other data sources that has been curated to Guides Copilot. It then summarizes the necessary information and offers interactive guidance through content and holograms overlaid on the equipment / asset in operation or maintenance. All this is powered securely by Microsoft Azure OpenAI Service, which uses GPT large language models to understand and generate natural language. Guides Copilot will be available on HoloLens 2 first and on mobile devices in the future.
How can Copilot in Dynamics 365 Guides help frontline workers?
Copilot in Dynamics 365 Guides is important for frontline workers because it helps them complete complex tasks and resolve issues faster, more reliably and with less disruption to the flow of work. Copilot can help workers in various ways, such as:
- Point and ask guidance: Workers can point to a component or piece of machine and ask questions, and Copilot recognizes, with HoloLens 2 capabilities, what the user is pointing at and provides answers. For example, workers can ask What this component does? or What are oil pressure limits for these valves? Tell me how I can take this panel off.
- Step-by-step assistance: As a maintenance person, think about the situation where you can have progressive conversations and receive step-by-step assistance when you are alone in the field. Instead of trying to call to a colleague or wake up expert (through some support lines, of course) who lives on a different time zone, workers could ask Walk me through the steps to troubleshoot flow regulator problems or I removed the circuit board, what is the next step? Copilot follows along with the process and conversation using customer’s curated knowledge base and articulates answers with speech, text, or 3D holograms overlayed on top of the physical asset.
- On-demand information: Workers will be able to access rich information for troubleshooting or repair through connections to IoT (Internet of Things) sensors, operational data, and service records. For example: What is the current oil pressure? When was this last checked? Were there any concerns mentioned in the last service visit”
Copilot can pinpoint specific assets, identify & show them, and access relevant information sources in real time. This is what I classify as part of the Industrial Metaverse.
- Expert knowledge integration: There will be a possibility to leverage notes or call transcripts from previous expert activity to guide new workers through the same process. For example: How was this problem fixed the last time? What does Steve recommend in fixing this?
Copilot can really help workers learn from each other, make “silent knowledge” usable and automate internal knowledge transfer. - Streamlined documentation: Enabling workers to talk to Copilot, in natural language, about what they are doing, and let Copilot help them breeze through service tasks, notes, and product updates. Copilot can offer ideas to quickly tick off tasks, jot down notes, change product amounts and conditions, and speed up data entry.
- Remote collaboration: Workers can use Guides to communicate with remote experts or colleagues who can provide guidance and feedback through holograms and voice. This is not something that is the future – Guides and Remote Assist are already in use and have been for some time. This tech enables real-time collaboration and knowledge sharing across distances – something that is very much needed in various industrial environments.
- Preventive maintenance: How about using Copilot to proactively check and maintain equipment before it breaks down or causes safety issues? Expanding On-Demand information with questions & information about “When is the next scheduled maintenance for this machine?” or “What are the common signs of wear and tear for this part?”. In industrial processes it is always better (and cheaper) to have scheduled maintenance than an unexpected breakdown. Copilot in Guides can help prevent costly downtime and improve operational efficiency.
- Quality assurance while saving time: Did I install that part right? Did I perform installation steps correctly? Copilot opens possibilities to verify that work meets the required standards and specifications – and the job is done right the first time. For example: Did I complete all the steps correctly?
Quality and compliance with AI-powered validation and inspection is the future.
How about businesses?
Copilot in Dynamics 365 Guides will benefit businesses by increasing efficiency and effectiveness in their operations. As the world is changing, the work and needs also change all the time.
Copilot can help businesses in several ways, a few being:
- Improving worker productivity and quality is not just about speed. It is about making fixes the right way on the first try. AI can help to complete tasks faster, more accurately, and with less rework, reducing operational costs.
- Enhancing worker safety and satisfaction: Using guides and instructions it is possible to avoid errors and injuries, improving worker safety and satisfaction.
- Enabling workforce agility and scalability: This is already there with Guides and it is easy to see that Copilot helps workers ramp up quickly, learn new skills, and adapt to changing demands, enabling businesses to respond faster to market opportunities and challenges.
- Reducing the upfront investment for spatial content creation by simplifying the creation, management, and delivery of mixed reality content using customer-defined data sources, enabling faster adoption of mixed reality by businesses.
Conclusion
Copilot in Dynamics 365 Guides is a tool that transforms the real world into a prompt and an anchor for AI, keeping humans at the center. It helps frontline workers act in the real world, increasing efficiency and effectiveness in business operations. I am looking forward to seeing this in action, as it not just brings AI to frontline, but is a practical application that benefits both employees and businesses. This is technology enabling the Future of Work .
Copilot in Dynamics 365 Guides, currently in private preview, holds significant potential for businesses seeking greater value from Mixed Reality and AI. This is truly Copilot x AI x Mixed Reality x Metaverse x People. I will be keeping a close eye on this tech 100%,. .
How can your organization get started? By deploying Dynamics 365 Guides and Remote Assist on either HoloLens 2 or mobile devices as the first step. And let us not forget how Dynamics Field Services and connect tasks to frontline workers and support them with correct Guides. That can be done already today, for Copilot we need to wait until it is in public preview.
Read Microsoft’s introduce blog post and Guides Copilot AI x Metaverse introduction video. I used pictures from Microsoft’s blog post to illustrate this article.
Yes, I did use Copilot to help to write the first article draft and continued manually from there. This helped me to create the structure and draft content first and then I continued editing the text, added more content and so on. Copilot is a tool for me, just like I use Microsoft Editor (spell & grammar checks etc.) to make the text better.
Vesa Nopanen - My Metaverse Day 