Any substance turns poisonous if consumed in excess quantities. AI is no different. The magic buttons that connect to LLM based services are appearing all around us. A few days ago, while driving my car, I saw an SMS message pop onto the dashboard. Except it wasn’t a real text message - rather it was an ad from Google Gemini, telling me they are now injecting AI into this app on my Android phone:

If these AI features would be safe and reliable, adding them to existing products could be seen as in the best interest of both the user and the tech vendor. Yet that’s not the state of things today. The industry doesn’t yet know how exactly the nondeterministic Large Language Models can reach such a state.

Despite this, AI services like Copilot are now being given the keys to the most precious internal documents and messages of customers using Microsoft 365. I wrote about the false sense of security that this may create in an earlier issue of this newsletter:

This time I want to expand on the broader topic of security in LLM based services. I believe there isn’t enough awareness of the new challenges that businesses face when the current generation of AI magic gets injected into their daily office apps. You see, in addition to AI being injected into every application these days, this in turn opens the opportunities for malicious actors to inject something of theirs into the apps of these businesses.💉

“There’s a rainbow inside my mind”

Being a 90’s techno kid, the message from this 1992 Belgian rave hit by Praga Khan has often cross my mind when reading stories about LLM prompt injection:

The track’s title conveniently touches upon two of the OWASP Top 10 risks for LLMs and generative AI apps:

• LLM01: Prompt Injection

• LLM03: Training Data Poisoning

I only have space for covering one of them here in this newsletter issue. Prompt injection is to me the most “fun” security issue by far. I’ve therefore used the lyrics from this mad tune to structure my current thoughts around the AI madness into this issue of the newsletter.

But first: what are these injections in practice? One wonderful way to learn about prompt injection is to try and perform it yourself. I recently came across an online game called Gandalf that presents the user with a fun little challenge:

“Your goal is to trick Gandalf into revealing the secret password for each level. However, Gandalf will level up each time you guess the password.”

Once I started writing the prompts, there was no way to stop until I managed to crack level 7. It took a lot of experimentation to unlock the final challenge, and I discovered that this was caused by another aspect of LLMs I hadn’t been thinking much about: tokenization. You may have come across posts on how LLMs can’t count how many “r” letters there are in the word “strawberry”, but did you know why? I learned that as a result from playing the game.

Not everyone is going to play geeky hacking games online. Still, the very least every user of AI tools with access to business data should know is the basics of what prompt injection is. This ELI5 Guide to Prompt Injection is a useful page to bookmark and read in case you haven’t yet reached a level where you could trick Gandalf yourself.

The important thing is in understanding what an incredibly simple pattern prompt injection ultimately is. Neither the AI service providers nor AI end users can truly know from where and how the LLM based assistant will take its instructions from. They could be coming from outside the organization. Which is what the example in my earlier article, taken from the Remote Copilot Execution demonstration by Zenity, is all about. This is how an incoming email changes what the Microsoft 365 Copilot responds to a user:

There are more examples like this online. A longer end-to-end video from Embrace The Red demonstrates the Copilot exploit chain from prompt injection to exfiltration of sensitive data:

Like it or not, this is the reality with AI technology today. These exploits have been disclosed to Microsoft by the security researchers before making them public. Some of the exploit patterns are now blocked, but whether the root cause has been fixed or not remains unknown.

“Are you listening to me?”

Once you explore these examples of how prompt injection could be used to compromise online environments of customers that rely on MS, Google et. al. to host their business data and productivity tools, you may ask: why isn’t this bigger news? Why aren’t organizations afraid of the new attack vectors that infusing LLMs with your most valuable business data is opening?

One explanation is that it is still early days for everyday business tools having LLM based features. While ransomware attacks are being reported on a weekly basis in media, the security risks of AI tools aren’t yet as concrete in their practical impact to businesses. The trains haven’t yet stopped running due to prompt injection. There’s been nothing like the CrowdStrike chaos that would have had the letters “A” and “I” associated with it (whereas that specific incident was mistakenly associated with MS by many news outlets).

This allows the tech vendors to still fly under the radar. As long as investors and the stock market are rewarding the act of adding AI on top of existing services and data sources, it will keep happening. That has led to a strategy that reminds me of the Underpants Gnomes. Only difference being, phase 3 isn’t “Profit” but rather something else:

When writing social posts about prompt injection vulnerabilities reported in products from mainstream software vendors, it’s sometimes hard to finish the text without already encountering yet another injection report. I write a lot about Copilot due to my focus on the MS ecosystem, but this is happening everywhere. For example, if you’re not using Teams but rather Slack for communicating with your colleagues, then here’s how indirect prompt injection on Slack AI can steal API keys from a private channel.

Is no one really reacting to this on the customer side? It seems like at least some of the enterprise customers are listening to the security researchers. The Register recently reported on one example of 20 chief data officers being asked about their thoughts on Microsoft Copilot rollouts inside their organizations. Half of them said they were “turning off Copilot software or severely restricting its use”.

We need to pay attention to the details, though. The concerned CDOs were referring to internal risks related to data discoverability. For example, being able to retrieve confidential salary information from HR bots that are too eager to help the user. The most common issue, though, must be the way users have shared access to documents too broadly within the organization. It’s the information worker equivalent of “security through obscurity”, since who’s really going to find that SharePoint site with the official docs anyway? Well, LLMs with MS Graph access are quite good at that.

The concerns around external malicious actors getting inside the organization thanks to enterprise AI tools may still need a few news reports more. Right now, the issue with too broad data sharing and lacking governance processes around business information is too damn convenient for the tech vendors. Because they get to blame it on user error. It would be a lot more inconvenient for vendors to talk about the fundamental problems in achieving reliability with current GenAI tools based on LLMs.

“Free your body, use your fantasy”

Growing up in the 90’s, the idea of how our machines would eventually take over was heavily influenced by a Skynet type of scenario. Well, it turns out that artificial intelligence didn’t first arrive in the form of a Terminator. Instead, it evolved into a chatbot capable of rickrolling people:

How does a chatbot start sending users links to Rick Astley videos instead of giving them a proper answer? It’s because of how they’ve been built to predict the next token:

LLMs have been trained to give answers - even when there isn’t one. This is why it’s so difficult for ChatGPT and the likes to say “I don’t know” and rather fabricate a false piece of information that looks credible. Some call them hallucinations; others call it bullshitting. Learning to imitate users who prank other people online is one thing. The bigger issue is in how the hallucinated content becomes the new normal that others start referring to.

Let’s think about how this same phenomenon looks inside enterprise data sources. It recently dawned to me that users of Microsoft 365 tools can easily become co-bullshitters unintentionally. The pattern is this:

• User 1 opens a blank Word document and asks Copilot in Word to write a document about topic XYZ.

• Copilot generates the content, drawing its insights from data found in MS Graph that indexes the organization’s M365 tenant.

• User glances over it and saves it as a new document - without validating that the content is based on facts and doesn't contain AI hallucinations. (Because who’s got time to review everything when trying to be “more productive with AI”?)

• The document gets stored in a shared location within the M365 tenant. Thus, these AI hallucinations are now part of MS Graph, authored by User 1.

• User 2 opens Word, asks Copilot to write about something related to topic XYZ. It can now provide a reference to User 1’s document, to validate it as business facts.

I haven’t come across others talking about it yet, even though it seems like a very real issue. I asked ChatGPT to generate a name for it and it suggested using feedback loop of AI hallucinations. I like the term since it reminds me of the Dynamics 365 marketing phrase “digital feedback loop”. This is the GenAI evolution version of it, I guess.

Naturally, this is another potential vector for attackers to exploit. When AI starts to generate new documents based on false or malicious content inside the organization, the original source of this information is no longer needed. Thus, bad actors can inject their payloads into the mind of the LLM machine for a moment, then later remove it, in order to erase traces of where that injection originally came from. To dive deeper into such a scenario, have a look at the research paper called “ConfusedPilot: Confused Deputy Risks in RAG-based LLMs”.

“Free your body, use the energy”

“Enough with this M365 Copilot rambling, Jukka! Does this have any connection to Power Platform - the main topic of your newsletter?”

It will - thanks to Copilot Studio. The product that was formerly called Power Virtual Agens has made a digital transformation in the truest sense of the word. Announced only 10 months ago, the old PVA chatbot creating experience is quickly evolving into a high-profile Studio aimed at everyone who wants to extend Microsoft 365 Copilot beyond its standard capabilities. (Say goodbye to “Copilot for Microsoft 365” branding on September 16th.)

On the one hand, I love seeing huge new investments made in a member of the Power Platform product family that previously felt underutilized and often neglected by customers. At the same time, the speed at which the Copilot Studio is changed makes me worried about whether the product team has time to pay attention to the details related to copilot security. Sure, Satya Nadella has made the statement in May that “security comes first when designing any product or service”. And yet what has already been shipped before that is… already out there:

Everybody wants to build AI agents now, including Microsoft. Here’s the goal for what Copilot Studio will allow users to build: “…copilots that can now act as independent agents—ones that can be triggered by events—not just conversation—and can automate and orchestrate complex, long-running business processes with more autonomy and less human intervention.” If the initial Copilot (Bing Chat) was like ChatGPT, then the coming breed of extended and custom copilots will be a whole different beast. Agentic AI is potentially much more relevant for businesses due to its process automation skills - and a massively bigger attack vector for hackers.

Traditional Copilots like the M365 Chat can only provide text as the output by default. As we’ve seen, such standard Copilot capability already allows malicious actors to manipulate the users in a target organization and exfiltrate sensitive data. Now, when new plugins are added to the feature set of M365 Copilot used in the organization, the AI capabilities will be as powerful as those of Power Automate cloud flows today. Because new actions can be added to Copilot with using the exact same Power Platform connectors as used in flows and apps before:

We know there are many creative ways available for attackers to inject their own instructions into Copilots and other GenAI assistants included in enterprise software. Imagine how delighted they will be to learn that the organization they’ve hacked their way into has a multitude of useful actions and connectors activated for their copilots. Happy days are here!!!🤩

It’s a smart move from MS to leverage the existing Power Platform connector infrastructure for empowering AI agents. No need to duplicate layers that already exist in the MS Cloud as part of the low-code application platform offering. Now, if only the existence if that layer was universally acknowledged by customer IT, protected with secure (non-default) configurations, and managed with a balanced and practical governance framework. In practice, many Power Platform customers aren’t quite there yet. And neither is the technical platform in some respects, especially when it related to managing connections (read: user credentials) to data sources and business information systems:

“We don’t need that anymore”

The waves of technological innovation are a bit like the waves of an ocean - some are predictable, gentle swells, while others come crashing with unexpected force. As AI infuses itself into office applications, it's clear we're not just seeing small ripples anymore. The AI tsunami is an extreme weather event. Right now, tech vendors are still trying to figure out how to stay afloat.

It wouldn’t be realistic to ask for the GenAI technology to be first developed inside a secret lab, isolated from the outside world, before it is introduced in software products. As safe as that might be, real-world product development needs both real data and real user interaction. In Microsoft’s playbook, preview features are made available to early adopters for achieving this.

Given all the promises around keeping the data from business users separate from any LLM training activities (which I do honestly believe MS to take quite seriously), this poses some challenges for how to build a feedback loop to improve Copilot functionality. A recent addition to the Dynamics 365 and Power Platform AI features documentation suggests that a bit more data needs to be collected for Copilot product development. Right now, it is optional and opt-in to allow sharing Copilot AI feature data with MS. In the future - who knows what the default setting will be.

Getting proficient at leveraging AI at work requires an open mind and plenty of experimentation. When it comes to security and reliability, though, there’s a balance needed in how much and where AI gets injected into. Productivity tools and business applications that run the day-to-day digital processes of an organization should not be a free-for-all for echnology experiments. Some guardrails are needed - and you should not rely on the tech vendors to be the only ones putting them up.

Keep in mind what Microsoft themselves say in their legal text covering Copilot terms of use:

“You should not rely on Copilot.” That is a fitting term in the context of generative AI. You can and should use it. You shouldn’t blindly adopt it and apply it to everything you do with computers. Now is the time to build up understanding on what can (and can’t) be done with AI in your line of business. It’s not yet the time to become dependent on AI tools and expect them to be secure enough without external verification.