Your AI obeys anyone who knows how to ask.The list of the ten biggest AI security risks, explained with real cases and what you can do to avoid them.
In December 2023, a user needed just a couple of messages to get a Chevrolet dealership’s chatbot in California to agree to sell him a Tahoe for one dollar. It was a prank, but the flaw that made it possible is the same one now used to pull data out of companies through their AI assistants.
The ten AI security risks
The audio version of this article, in about seven minutes.
The sale never went through, but Air Canada did lose money: its chatbot promised a customer a refund the airline did not actually offer, and a tribunal made the airline pay it, because everything on its website, chatbot included, is its responsibility.
The OWASP foundation, known for its list of web security risks, publishes another one with the ten most important risks in applications powered by a language model, such as ChatGPT or Copilot. It is called the OWASP Top 10 for LLMs (LLM stands for large language model), each risk has a code from LLM01 to LLM10. You can read it in full on its website; here is our summary.
Why a chatbot obeys the wrong person
A language model receives everything as text: the company’s instructions, the user’s question and the documents or emails it reads along the way. It cannot reliably tell an order from a piece of data, so an instruction hidden in an email can outweigh the company’s own. This is called prompt injection.
Prompt injection, much like scams, is unlikely ever to be fully solved.
OpenAI itself admits it, with an example: a user asks their AI agent to draft an out-of-office reply, the agent reads an email with hidden instructions along the way and ends up sending a resignation letter to the boss on the user’s behalf.
The ten risks, one by one
LLM01 · Prompt injection
This is the Tahoe case. The instruction can be typed by the user or hidden in a PDF, a web page or an email the assistant reads, and the second kind is the more dangerous because nobody sees it. In 2025 Microsoft had to fix a flaw like this in Copilot that let attackers extract company information.
LLM02 · Sensitive information disclosure
Samsung employees pasted internal information into ChatGPT and the company ended up banning its use. Something similar happens inside a company: if the assistant searches all the documentation without checking who is asking, anyone can end up seeing the payroll. The fix is for the assistant to search only what that user already has permission to see.
LLM03 · Supply chain
On Hugging Face, the main site for downloading AI models, around a hundred models carrying malicious code turned up, giving remote access to the computer of whoever opened them. A downloaded model is third-party software: get it from trusted sources and in formats that do not run code when opened.
LLM04 · Data poisoning
Anyone who slips manipulated text into the data a model is trained on can change how it behaves. And it takes very little: Anthropic showed that 250 documents are enough to make a model start producing gibberish whenever it reads a specific phrase. You need to know where every document comes from.
LLM05 · Improper output handling
This happens when the model’s answer is used without checking it, for example by running it as code. A LangChain function, a widely used tool for building AI applications, did exactly that, and a carefully crafted question could make the server do anything. A model’s answer deserves the same suspicion as a form filled in by a stranger.
LLM06 · Excessive agency
This means giving an agent more permissions than it needs. In Notion it was shown that a PDF with hidden instructions could get its agent to send confidential company data out, without any software bug involved. Each agent should be able to do only what is essential, and sending, deleting or paying should always go through a person.
LLM07 · System prompt leakage
The day after Bing’s chat was unveiled, a student got the chat to reveal its internal instructions, including the codename it was not supposed to disclose. Assume your assistant’s instructions will be seen and keep passwords and internal data out of them.
LLM08 · Vector and embedding weaknesses
Many assistants store documentation in a special database that lets them quickly find the passages closest to each question. If documents from different customers or departments are mixed in it, or anyone can add text, the assistant will end up showing what it should not. Keep the information separated and control who can add content.
LLM09 · Misinformation
This is the Air Canada case. Models are wrong with great confidence and sometimes make data up. It helps if the assistant says where each answer comes from and replies ‘I don’t know’ when it cannot find one.
LLM10 · Unbounded consumption
Every query costs money or computing power, and abuse or an agent stuck in a loop can send the bill soaring or take the service down. Per-user usage limits and alerts when consumption spikes prevent it.
manipulated documents were enough to alter models of up to 13 billion parameters.
Anthropic, October 2025Protecting yourself from something with no fix
Filtering suspicious phrases is not enough, because the same instruction can be written a thousand ways. Tools such as Prompt Guard stop the crudest attempts, but the sensible approach is to assume the attack will come and set up the application so that it does as little damage as possible.
Five measures get you there:
- Keep content coming from outside apart from the instructions, so the system knows what is data and what is an order.
- Give each tool the minimum permission. An agent that summarises emails does not need to send them.
- Check permissions outside the model, in the system that performs the action and with the real user’s identity.
- Ask a person to confirm before sending, deleting or paying.
- Log everything and review it, because an attack that works produces no errors.
And test it with known attacks every time you change something.
And if the model runs on your servers
Many companies run the model on their own servers so that data stays in-house (here we explain which server you need). It is a good idea, but it does not protect you from everything: some risks go down and others become your job.
| Risk | With an external provider | With the model on your servers |
|---|---|---|
| LLM02 Sensitive information | Data goes to the provider and you depend on their contract. | It stays in-house, but access permissions are still up to you. |
| LLM03 Supply chain | The provider chooses and protects the model. | You download it, so checking where it comes from is your job. |
| LLM04 Poisoning | You do not control the training. | If you fine-tune it with your data, watching that data is your job. |
| LLM07 Internal instructions | They travel to the provider with every query. | They stay inside, although the model can still reveal them. |
| LLM10 Unbounded consumption | Abuse shows up on the bill. | There is no bill, but if the server is overloaded the service goes down for everyone. |
Prompt injection and excessive agency, however, are just as dangerous on your servers as in the cloud.
What to log so you find out in time
These attacks produce no errors: the model does what it is told and nobody notices. That is why you should record who made each query, which documents the assistant read and which actions it took, and send it to your security alerting system (a SIEM such as QRadar or Wazuh). With that, the system can alert you when:
- an agent sends or changes something right after reading an external document, as in the resignation letter case;
- an answer includes documents from a department the user does not belong to;
- a user or an agent suddenly consumes far more than usual.
Where to start
Take stock of what data your application uses, what it can do and where it keeps its keys. Then review permissions, set usage limits and prepare the tests and alerts. And since many of these applications work like any other API, go through the OWASP API Security Top 10 too.
All of this is practised in our AI security course: OWASP for LLMs and red teaming, and if you would rather design your agents with us, there is our secure AI agents service.
Frequently asked questions
What is the OWASP Top 10 for LLMs?
It is the list of the ten most important security risks in applications that use language models, published by the OWASP foundation, which updates it periodically.
Can prompt injection be prevented completely?
Not today. You can limit the damage with minimum permissions, by checking the model’s answers, by asking for human confirmation on important actions and by monitoring what happens.
Is it safer to run the model on your own servers?
It keeps data from going to an external provider, but it does not protect against prompt injection or excessive permissions, and it means you have to watch where you download the model from and how much it is used.
Got an assistant or an agent up and running?
Tell us what data it uses, what it can do and with what permissions, and we will tell you where we would start protecting it. If you want your team to learn how to attack and defend it, that is what our AI security course: OWASP for LLMs and red teaming is for.
AI security training and services · sixe.euBlog!
Our mission
SIXE ensures that your infrastructures based on IBM®, Red Hat®, Canonical®, and SUSE® technologies remain stable, secure, and optimized for as long as you need them.
At SIXE, we are committed to sustainability. That’s why we help you design, implement, and maintain durable and efficient technological solutions.
