Microsoft’s Copilot Vision can now see your entire desktop – but it probably won’t judge you for the clutter
- Microsoft has updated its Copilot Vision tool to let the AI assistant view your desktop
- The feature will let Copilot respond to any content you’re viewing
- To give users control, Copilot Vision must be enabled, so it will only look at what you want the AI to see
Microsoft has opened the eyes of its Copilot AI on desktops to see the entire screen. The feature, which enables Copilot to look at and analyze things on your screen, can see any window or browser you want, as a feature Microsoft calls “Desktop Share.”
When Copilot Vision first launched, it was limited to just spotting what was happening in Edge. You’d ask a question about a webpage, it’d scan that page, and the AI would help explain what was there. With this update, it can now zoom out and see your whole desktop or any specific app window you choose to share
You can pick what the AI assistant looks at by clicking on the eyeglasses icon in the Copilot app. Then you select the area you want the AI to look at, and you can discuss what’s on the screen with the AI without having to describe what you’re looking at endlessly.
Let’s say you’re revising a resume and want it to sound more like a confident human and less like a panicked grad student. You can open the Word document, share the window with Copilot, and ask it to highlight specific types of experiences. Copilot sees the document itself and suggests edits.
The same goes for PowerPoint, Photoshop, and even some games. You can ask Copilot for help with the settings menu in a game you’ve never played before, and it’ll walk you through options with tips and suggestions for when you get stuck.
Visionary planning
In case you’re worried about Copilot snooping on your secret romance novel or something else you’d rather it not see, you can always turn it off, too. For now, the update is only available to Windows Insiders who already have Copilot Vision. It will likely come to the general Windows 11 population soon.
It might not seem too big a deal compared to other generative AI features, but it’s a subtle way for Microsoft to embed its AI into quite literally anything on your computer. Instead of writing an email in one tab and having to describe it to the AI in another tab for help, it’s all right there. Copilot Vision makes the process more intuitive. You digitally point to something you want help with, ask a basic question, and get an answer that makes sense in context.
It’s hard to overstate how much friction this could eliminate. It’s not perfect, of course. Some specificity is needed since the AI can’t read your mind. But an extra set of eyes to help you over the hump of designing a PowerPoint or shortening a meeting summary is not nothing.
The dream of AI as a personal assistant has always been about context, the idea of an AI that could know what we mean, not just what we say. Copilot Vision is Microsoft’s clearest step in that direction yet. Copilot Vision could quietly become one of the most useful AI tools Microsoft has ever made. And unlike Clippy, it will only pass judgment if you ask.
Add comment