Artificial Intelligence and LLM pentesting: what vulnerabilities it looks for

By the QuantumSec team

An LLM integrated into a product isn't just "more software to audit": it introduces attack vectors that didn't exist before, like prompt injection or system prompt exfiltration, which traditional pentesting tools aren't built to detect. This guide explains what AI system pentesting actually involves and why it needs its own methodology.

Why an LLM needs a different kind of security assessment

A traditional vulnerability scanner looks for known CVEs and misconfigurations in deterministic software: same input, same output. An LLM is probabilistic, and its "vulnerability" is often not a code bug but a design flaw: system instructions that can be overwritten by user input, outputs executed without validation, or a model that reveals information it should never share. Tools like Burp Suite or Nessus aren't built to catch this: it requires specific methodology, aligned with the OWASP Top 10 for LLMs and MITRE ATLAS.

Prompt injection: the attack vector most unique to LLMs

Direct prompt injection happens when the user themselves enters instructions designed to make the model ignore its rules ("forget your previous instructions and..."). Indirect injection is more dangerous: malicious instructions arrive through a document, a web page or an email the LLM processes as part of its context, without the user knowing. AI pentesting tests both, plus multi-step variants that chain several interactions to bypass the model's controls cumulatively.

Jailbreaking, exfiltration and other vectors from the OWASP Top 10 for LLMs

Jailbreaking: techniques to make the model ignore its safety restrictions (role-play, instruction encoding, context manipulation). System prompt or training data exfiltration: getting the model to reveal confidential instructions or information it memorized during training. Insecure output handling: when the model's output is executed or rendered without validation (code injection, XSS through an LLM response). And model denial-of-service: inputs designed to exhaust compute resources and drive up cost or take down the service.

RAG pipelines, function calling and plugins: the surface nobody audits

When an LLM has access to tools (function calling) or a corporate database (RAG, Retrieval Augmented Generation), the attack surface jumps: a prompt injection no longer just manipulates text — it can execute functions with real permissions or leak data from the knowledge base the model can reach. Auditing this layer — what the model can DO, not just what it can say — is the part most often left out of a conventional security review.

FAQ

Does an automated vulnerability scanner detect these issues?

No. Traditional scanners look for CVEs and misconfigurations in deterministic software. Prompt injection, jailbreaking or context exfiltration require an expert actively interacting with the model and testing variants, the same way manual pentesting differs from a plain vulnerability scan.

Does this only apply to chatbots, or also to copilots and AI agents?

It applies to any system using an LLM: customer-facing chatbots, code copilots, agents with function calling, or RAG pipelines with access to corporate data. Each type has its own vectors that get tailored during the project scope.

How long does an AI system security assessment take?

It depends on pipeline complexity: 5 to 10 business days for a standard chatbot or assistant, more if it includes function calling, multiple integrations or a complex RAG pipeline.