AI agents are supposedly the replacement for business apps. If that were already the reality today, the productivity of M365 business users would be shattered by the constant issues of agents not being found. For the fifth day in a row, this is the experience that users get when clicking on “Get agents” in the Microsoft 365 Copilot UI:

None of the buttons in there work, nor do I understand what they should do. I don’t even have a recollection of what’s supposed to be presented on this screen as the UI changes all the time. That in itself is a major issue as it leads to me constantly questioning myself: “am I doing it wrong?” Maybe I missed some exciting product announcement and am now trying to perform something that hasn’t been valid for many weeks anymore?

This time, I get comfort from checking the M365 Admin Center and discovering an open issue CP1053707 in the Service Health menu. Of course, 99%+ of Copilot users wouldn’t be able to view this, due to the tenant admin rights required for seeing such info not distributed on the public web.

So, it’s just one issue. Is that a big deal? I would say yes, since this is exactly the experience I expect from Copilot agents. Based on the experiences of being a M365 Copilot subscriber in my company’s tenant for a year now, I’ve only learned to not trust it to do real work for me. Copilot and AI agents remain an experiment that I spend time with primarily because I have to keep up with what Microsoft is doing. If I wasn’t working inside this ecosystem, I would surely look elsewhere.

In this week’s newsletter issue, I’ll share some of the reasons for concern over the reliability of Copilot and AI agents as part of Microsoft’s business productivity toolkit in the year 2025.

Now you see it, now you don’t

I have worked with Power Apps long before they were called that. Ever since the Dynamics CRM apps went into the public cloud 14 years ago (with global availability), there’s been times when the service has been unavailable. Not often, but it happens. Rather than a complete failure for any UI to render, the typical issues only concern some parts of the application. Also, while the app UI performance is hit & miss still - it loads when you just give it enough time.

With the new UI for AI, the entire UI goes away incredibly often. This is because unlike Power Apps that have direct URL we can bookmark, the agents are accessed through the experience embedded within Microsoft’s Copilot. (And when I say “Copilot”, I’m referring to the chat UI that is outside individual apps like Word or Dynamics 365. By the time you read this, it may have already been reimagined to some new name.)

What happens is that on any day when you come to work and open up your PC to access Copilot, there’s a possibility that the agents you have built are not there. These virtual assistants just refused to wake up and stayed home instead - and there’s absolutely nothing you can do about it. If the Copilot UI refuses to show the agents, you can’t use them. Because it is “the UI for AI”.

I don’t know about you but this drives me up the walls. It’s the absolutely worst thing - a “404 not found” screen but for the age of AI, meaning the service pretends it’s available. It’s just gaslighting you, like LLMs so often do to us. And it makes you feel powerless in front of the machine.

I should be able to work in this Copilot domain with ease. I’m used to things moving around in Microsoft product UIs all the time without warning. I usually know which exciting update is coming that will change the names of things. I even have a level of understanding about the admin controls that impact the visibility of features and elements in the service.

Or do I really? That’s something I keep asking myself as surprising things show up inside Copilot. Back in October when they were still called “Copilot extensions” and not “agents” (almost 6 months ago, ancient history, I know…), AI Builder prompts and Dynamics 365 1st-party features just began appearing as things to directly chat with on M365 side. Some multiple times, from any Power Platform environment here I had deployed solutions provided by Microsoft.

I’m pretty sure the playbook for Responsible AI doesn’t include unintentionally exposing agent/prompt to every user in the tenant without admin control. Yet this is the sort of thing that has become business as usual when living with Microsoft Copilot.

Sure, there is the Copilot Control “System” that Microsoft launched at Ignite. Now we just need to wait for it to turn from a slide of capabilities to actual platform features for the admins.

In UNIX SharePoint, everything is a file

Copilot Studio agents build on the relatively robust governance capabilities of Power Platform. It’s the new UI layer that makes it painfully obvious Copilot is a cross-platform play that includes actors who’ve not performed on the same stage before. No one seems to know quite into which direction to move to.

Let’s look at a domain outside Power Platform then. SharePoint agents are, not surprisingly, completely different from the types of agents you create with Copilot Studio. Despite being called “agents”, the SharePoint version doesn’t have any agency to perform actions apart from browsing the contents of specific sites or documents. Yet that can be quite handy for real information work where you need to search for answers from internal docs.

The beauty of SharePoints agents is that they are just files that you can search anywhere, using the “*.agent” search term. Not a fancy UI, but at least you are in (nearly) full control of discovering the agents you need - unlike inside Copilot. You could add these to your bookmarks for quick access anywhere.

Opening a SharePoint site, when you click on the Copilot icon, there’s a dedicated UI for surfacing the agents available on the site. It includes an “approved for this site” section that suggests there’s a level of admin controls to govern what agents users are recommended to leverage.

Everything looks nice on the surface, ready for keynotes and sales demos. It’s only when you learn how the system has been actually designed and implemented that the cracks start to appear. This comment under the SharePoint agents GA blog post does an amazing job of revealing the issues that organizations will face once they want to take the Copilot features into production use with live business data:

I almost couldn’t believe what I read here. The fact that Microsoft would design a feature for promoting approved agents on the UI level and then completely neglect any back-end capabilities needed to enforce the approval concept is… I don’t even know what to call it. “Insult to all the SharePoint admins” would perhaps describe my feelings in a polite way.

Agents without agency or strategy

Ever since Microsoft decided to rebrand “copilots” as “agents” in the latter part of 2024, there’s been pressure to turn the AI chatbots into tools that can do more than just talk to you. After all, calling natural language search an “agent” may eventually cause the general audience to dismiss Microsoft’s Copilot efforts as just a reincarnation of Clippy (which is also on the cards for 2025, by the way - literally).

To address this dilemma of all talk and no actions, Microsoft announced Copilot Actions in November. Aimed at the Microsoft 365 Copilot end users, not to be confused with actions in Copilot Studio agents, these were supposed to be help everyone automate their repetitive tasks by leveraging AI.

Sounds pretty useful. Guess what happened next, though? After a long wait, the Copilot Actions showed up in my tenant a couple of weeks ago. Then, last week, I read the announcement from M365 Admin center that the Copilot Actions feature is actually going to be removed.

Subscribe to Plus to read the rest.