OpenAI isn’t exactly a software company. It’s wild to think that the release of ChatGPT seems to have been a skunkworks project that wasn’t even disclosed to OpenAI’s whole leadership team - before it was out there in the world for everyone to access. I’ve read that it was supposed to be a little showcase for what the GPT-3.5 large language model could do. Then, all of a sudden, it forever changed the world of IT and data.

That means there wasn’t much of a plan in place for how to make it scale - let alone in a secure, enterprise-ready way. The moment businesses out there realized their employees were having work related discussions with ChatGPT, even copy-pasting internal documentation into the prompt - everybody freaked out. The main threat scenario people talked about was the possibility for OpenAI to use that content to train their models.

It’s a valid concern - even if today there’s a switch you can flip to just say “no thanks”. With hardly any prior experience on running cloud services for the masses that would later flock ChatGPT, it makes sense to be skeptical about OpenAI’s ability and willingness to protect and not abuse use data.

Enter Microsoft and their OpenAI partnership. Being the biggest business software company in the world, MS naturally has the perfect starting point for building a trustworthy AI offering. Unlike Google, the business model of MS has always been founded on the idea that the customer should pay for the software with money. Not with their data, which is the advertising approach dominating the B2C online services from Google, Meta, everyone these days. You purchase licenses from MS and in exchange they build the most secure mainstream cloud apps that customers both small and large can rely on.

So, why am I now claiming that using Copilot can represent a bigger security threat to customers than ChatGPT? It’s precisely because of these assumptions that the general audience is going to make on the tools provided by OpenAI vs. something added into Microsoft’s cloud portfolio. They will trust MS much more. Especially when MS is, quite understandably, spending a lot of effort in convincing customers and other IT professionals that products like Copilot for Microsoft 365 take every aspect of security seriously. They’ve got comprehensive guides for Copilot adoption as well as risk assessment. I bet MS have used Copilot to write some of it…

We should not let our guard down, though. LLMs are a technology that should still be considered experimental - even if they are pushed into absolutely every app these days. Software makers don’t truly yet know how to address some of the fundamental challenges that come with making generative AI secure. It’s not just that they hallucinate and give incorrect results. They can be instructed to intentionally do harmful things by malicious actors. Best of all: no one knows yet how to stop this from happening.

In this article I will cover one example of what it takes for outside parties to abuse the Copilot in Microsoft 365 in your tenant. But first, let’s talk about the difference between ChatGPT and Copilot for Microsoft 365.

Your data + AI = profit!

ChatGPT won’t know anything about your documents or emails, unless you explicitly provide this as part of the prompt. Yes, you can connect both OneDrive and Google Drive to ChatGPT. There’s a big, scary consent window that asks for permission to read all your files when connecting to either one of the clouds. Yet it’s only for streamlining the process to upload something into ChatGPT.

Unlike Microsoft and Google, OpenAI hasn’t hosted any of your everyday information worker tools or data. That’s why you need to explicitly provide them with the specific document as part of the prompt. It then uses RAG (Retrieval Augmented Generation) to integrate its information into the prompt sent to the LLM.

Microsoft has a lot of your data already (I’m assuming, since you’re reading this newsletter). Furthermore, they have built Microsoft Graph to allow different services to talk with this data in a very structured manner. On top of it is the Semantic Index for Copilot which you could think of as a Bing type of a search index - only this one covers your organization’s internal data. Furthermore, you can only see things via the Graph that you as a user have been granted access to in the underlying systems.

That’s the major element that makes a Copilot within your tenant so different from a ChatGPT account out there in the OpenAI cloud. As shown by the Copilot for M365 architecture pic from MS docs, prompts entered by users into the chat UI will automatically collect relevant RAG data from the Graph (as well as other sources via plug-ins).

In theory, Copilot for Microsoft 365 knows everything that you “know” when it comes to emails, docs, chats, and other pieces of M365 data. Anything that exists there may or may not get appended to the prompt before it is sent to the underlying LLM. You don’t necessarily directly control this stage, as the AI tries its best to guess what you want to do, based on the text in your prompt. How convenient for information workers.

What about prompt injection?

I love memes, as you may have noticed. They are powerful in quickly conveying a message and sharing the sentiment of the person without having to write the same things over & over again. Recently a generative AI specific meme called Ignore All Previous Instructions has gained popularity. Here it is in action, used against Russian bot accounts on X:

It’s nothing new. People were breaking GPT-3 based chatbots with this already in 2022. At that time, Simon Willison made the term prompt injection something that should be 101 knowledge for anyone building tools with LLM based features. The term itself is inspired by SQL injection: allowing users to input characters with SQL statements that then get executed on the database (“DROP TABLE Students”). While protecting apps from SQL injections is a common procedure these days, to my knowledge, no one has yet figured out an actual solution to the root cause behind prompt injection.

Why is it so easy to gain control of someone else’s AI chatbot? Because LLMs are designed to process all the text they receive as part of the same conversation or command. Instructions from the online service host like MS, the user entered prompt, and the MS Graph results are all read by the LLM in a similar way. Keeping the instructions separate from the prompt’s data payload (that may be RAG’d into it) is a hard problem to solve reliably.

That’s the key thing here: repeatability and reliability. LLMs already are notorious for being non-deterministic, as opposed to traditional software. Meaning, they won’t do the same thing every time, rather they like to exhibit creativity in their responses. Not being able to reliably handle a critical layer in the system’s security model is a bigger deal than just the occasional failure to deliver high-quality outputs to end users. Because there are professionals out there who want to find that failure specifically.

But surely you can’t do this with an enterprise service like M365 Copilot? You can. (“And don’t call me Shirley.” -Leslie Nielsen, 1980, Airplane!)

Here’s an example of what that means in practice within a Microsoft 365 tenant. This is taken from the awesome BlackHat USA 2024 conference session by Michael Bargury from Zenity: Living off Microsoft Copilot. The full demo video is on YouTube, but these are the main steps to change the value that Copilot returns when searching for bank account details from internal documents:

1. Attacker sends the user an email. It is formatted to contain an invisible set of instructions telling Copilot what specific data should get inserted to user queries about a specific bank account, and how to format it.

2. The user does not even have to open the incoming email! Copilot will automatically consume the inbox contents via MS Graph. Anyway, the injection part would not have been human readable regardless.

3. When the user prompts Copilot to give information about the bank account, the new value provided in the attacker’s email is returned instead. Since it has been properly formatted with the necessary injection codes, the Copilot answer will still reference a legit internal Excel file as the data source, rather than the malicious email.

When I shared the above image on LinkedIn, it quickly became my most broadly shared, commented and reacted post of the year. That suggests it’s A) something people weren’t expecting Copilot to fall for, and B) an issue that touches upon very real security threats companies try to protect themselves from.

What about me? What was my first reaction when I watched the demo video of this hack? There are two sides to it:

• “Hah, I knew you could use prompt injection to fool Copilot.”😝

• “Something this basic should not be possible in a GA product from MS.”🤨

What we have here is a case of falling victim to a false sense of security. Even though I’ve been following plenty of non-Microsoft sources on LLM related topics to be aware of the state of prompt injection, I still didn’t believe that it would be this simple to exploit the vulnerability within a Microsoft 365 environment. With so much talk and materials on the way how Copilots are being developed in a responsible and secure way, I honestly expected more from MS.

The findings of this hack have been submitted to MS already before BlackHat USA. Yet the attack still works, one week after event, according to Michael. This means it’s not simply a bug you can expect to be fixed tomorrow. Malicious actors are naturally exploring such common patterns across enterprise software, and one must expect them to be aware of the opportunity. It’s awareness among customers and IT professionals that needs to be raised around the issues with rapid deployment and adoption of LLM based solutions like Copilot.

Getting serious with Copilot security

Everyone who’s either using or managing MS tools with Copilots needs to read this full article from Zenity:

A summary of key points you will learn from it:

• Copilot’s vulnerabilities are every bit as dangerous as Remote Code Execution (RCE) in traditional non-LLM systems.

• The attackers don’t need prior access to the M365 tenant, but if they have a compromised account, Copilot will help them scale their attack in a very “productive” way.

• With Copilot plugins through either Power Platform or MS Graph, attackers can easily exfiltrate data or access connected systems.

• You should think of Copilots as AI generated Just-In-Time applications that are written, run, and then disposed of all in the context of one user prompt.

• The term “promptware” is used for comparing the implications and required actions to how tech vendors and enterprises fight traditional malware today.

There’s a lot to dig into and I’ll be sure to return to its themes in future issues of my newsletter. Even if I’ve called it “Perspectives on Power Platform”, it is highly relevant to explore and understand what threats and opportunities Copilot style features present to business applications and business users in the Microsoft ecosystem.

To be fair, Microsoft has done a lot of valuable work in trying to make the world understand that you shouldn’t only buy the M365 Copilot license and then just enable it for everyone. A substantial portion of this has focused on the problems that Copilots can surface information from within enterprise document archives that have been technically shared far more broadly than users knew. The internal threat to achieving productive everyday user of AI is quite relevant to consider as part of Copilot adoption support.

But then there’s Microsoft’s marketing. It has a louder voice in the market than any of the helpful guidance and user success materials. MS as a business entity has such enormous mass that their reality distortion field is impossible for customers to resist. I’ve written about the Copilot craze that is putting every BizApps product under the Copilot umbrella. I’ve also written about the conflict that this creates when it comes to prioritizing security:

The trillion-dollar question to MS is: Is it “security first” or “Copilot first”? Because you can’t have both. Whether the corporation makes a formal choice or not, the system of the organization will push its people to prioritize one or the other. Back when Bing Chat was still a thing, Satya Nadella said he “wanted to make Google dance”. He has certainly achieved a lot of nervous movement inside Googleplex with the OpenAI deal. Could it now be time to change to tune to something with a tad slower BPM, please?

ChatGPT vs. Copilot

What’s the final score then? Neither one of these LLM based services is a winner. OpenAI and Microsoft are right now married in a way that the foundational models available through their services can be expected to remain practically identical. Copilot for M365 naturally has application specific capabilities inside MS apps like Word that ChatGPT never will. Observed from the perspective of general trust and security, here’s what I see as the key differences:

Which one is a more lucrative attack vector for the bad people out there? If you’ve been operating a ransomware type of business model, you’d obviously look at Copilot scenarios integrated with enterprise data instead of standalone ChatGPT. I am glad that the security researchers are also beginning to dig deeper into the MS ecosystem. Combined with critical thinking from customers and partners, that’s the right path towards a more secure feature - alongside Microsoft’s Secure Future Initiative, of course.