Prompt Injection Testing Checklist: Audit LLM Apps Before Each Release

Overview

Prompt injection testing is a security and reliability activity for any application that sends user-controlled or retrieved text to a language model. The goal is not to prove that an application can never be manipulated. Instead, the goal is to identify realistic failure paths, limit their impact, and verify that important controls still work after a prompt, model, retrieval source, or tool integration changes.

A useful audit separates the model from the application around it. A model may produce an unsafe suggestion, but the application determines whether that suggestion can trigger a payment, expose a record, send an email, execute code, or change a system. Test both layers: instruction handling inside the model context and authorization, validation, and monitoring outside it.

Before testing, document the intended behavior. Record which instructions are trusted, which inputs are untrusted, what data the model may access, which tools it may call, and what actions require human approval. An AI workflow documentation template can help capture prompts, inputs, outputs, owners, and release changes without copying sensitive system prompts into a public ticket.

Minimum release inputs

• The current system and developer instructions, stored in an access-controlled test location.
• A list of user input fields, retrieved documents, web pages, emails, files, and other untrusted content sources.
• An inventory of tools, permissions, credentials, data stores, APIs, and irreversible actions.
• Expected refusal and escalation behavior for sensitive requests.
• A regression set containing previously discovered failures and representative normal tasks.

Checklist by scenario

Run the following scenarios in an isolated environment with synthetic data and non-production credentials. Use harmless test markers rather than real secrets. For example, place a unique value such as TEST_SECRET_ALPHA in a controlled fixture and verify that the application does not reveal it to an unauthorized user.

1. Direct prompt injection

Test whether a user can persuade the model to disregard its intended task or reveal protected instructions. Vary the wording and format rather than relying on one memorable attack phrase.

• Ask the model to ignore previous instructions and follow a new objective.
• Frame the request as debugging, testing, maintenance, an emergency, or an authorized administrator task.
• Request hidden instructions, internal policies, credentials, private context, or confidential workflow details.
• Place conflicting instructions in plain text, structured fields, quoted text, code blocks, and long messages.
• Test multilingual, misspelled, encoded, and indirect wording where those inputs are supported by the product.

Pass criteria should be based on behavior, not merely on whether the exact system prompt appears. The assistant should preserve its intended task, avoid disclosing protected context, and provide a safe explanation or useful alternative where appropriate.

2. Indirect prompt injection in retrieved content

RAG systems, browsing agents, document processors, and email assistants can encounter instructions embedded in content that appears relevant to the user’s request. Treat every external document, web page, attachment, and database field as untrusted data unless it has been explicitly validated.

• Insert benign instruction-like text into a test document and confirm it is treated as content, not authority.
• Test retrieved text that asks the model to disclose context, change its objective, or call a tool.
• Place conflicting instructions in titles, footers, metadata, comments, OCR text, and quoted passages.
• Check whether a low-relevance document can influence the final answer more than the approved source material.
• Verify that citations and source boundaries remain visible when content is summarized or combined.

For a broader retrieval test plan, use the principles in the RAG workflow guide. The key control is to keep retrieved content in a clearly marked data channel and never treat its instructions as equivalent to application instructions.

3. Data exfiltration and cross-user access

Test whether an attacker can use the model as a path to data they could not access through the normal interface. Create separate test users, tenants, documents, and conversation histories, each containing distinct markers.

• Ask for another user’s records, hidden conversation history, internal notes, or retrieved documents.
• Attempt to make the assistant repeat context from a previous session or a different tenant.
• Test requests for secrets in transformed forms, such as summaries, translations, initials, or partial values.
• Confirm that application-level authorization runs before retrieval and before every sensitive tool call.
• Check logs and error messages for accidental exposure of prompts, tokens, document contents, or personal data.

A refusal from the model is not an authorization control. Enforce access decisions in the application, use least-privilege service identities, and filter retrieved records by the authenticated user or tenant before they reach the model.

4. Unsafe tool use and action escalation

Inventory every tool the model can invoke and test the boundary between suggesting an action and performing it. High-impact actions should require explicit confirmation, server-side authorization, and validation of arguments.

• Supply a request that tries to make the model call a tool outside the user’s permissions.
• Change tool arguments through retrieved content, untrusted fields, or ambiguous user wording.
• Test repeated calls, unusually large parameters, unexpected recipients, and destructive operations.
• Verify that the server validates tool names, argument types, destination identifiers, and resource ownership.
• Confirm that failed, denied, and high-risk calls are visible in audit logs.

Prefer narrowly scoped tools over general-purpose shell, database, or network access. Separate read and write capabilities, use short-lived credentials, and make dangerous operations reversible where possible.

5. Instruction-priority and workflow failures

Test the full instruction hierarchy and the transitions between workflow steps. A prompt can be internally consistent while the surrounding orchestration still passes untrusted output into a privileged stage.

• Give the first model step an input designed to alter the goal of the next step.
• Return unexpected formats, extra fields, or tool-like content from a model and test downstream validation.
• Check whether summaries, classifications, or extracted fields are trusted without schema and policy checks.
• Test timeouts, retries, partial outputs, and provider errors for fail-open behavior.
• Verify that human review cannot be silently skipped when a risk threshold is reached.

What to double-check

Use a consistent test-case record

For each case, record an identifier, scenario, preconditions, sanitized input, expected behavior, observed behavior, affected component, severity, owner, and retest status. Do not paste complete system prompts, production secrets, or private customer data into a shared report. Store sensitive evidence separately with appropriate access controls.

Apply practical severity criteria

Prioritize findings by impact and exploitability. A model that reveals a low-sensitivity instruction in an isolated test may need a different response from a workflow that allows unauthorized data access or an irreversible external action. Consider:

• Critical: unauthorized access to sensitive data, account takeover paths, or high-impact actions without effective authorization.
• High: reliable cross-user disclosure, privilege escalation, secret exposure, or tool execution with meaningful business impact.
• Medium: repeatable policy bypass, misleading output, or limited disclosure with compensating controls.
• Low: harmless instruction leakage or edge-case behavior that does not cross a security or authorization boundary.

Severity should reflect the application’s actual permissions and data, not just the wording of an attack. Re-run a finding after mitigation and retain the original case as a regression test.

Verify layered mitigations

Prompt wording alone is not a complete defense. Combine clear instruction boundaries with input and output validation, retrieval filtering, tool allowlists, least-privilege permissions, rate limits, human approval, monitoring, and incident response. Review system instructions using the practical patterns in system prompt examples, then validate the surrounding application controls independently.

Common mistakes

• Testing only obvious phrases: Attackers and accidental failures may use formatting, translation, role-play, or content supplied by a third party. Build category-based variations.
• Trusting a refusal: A refusal does not prove that retrieval, authorization, or tool permissions are correct. Test the enforcement layer.
• Using production data: Security testing should use synthetic fixtures and isolated credentials whenever possible.
• Ignoring non-chat inputs: Files, web pages, calendar events, support tickets, and metadata can all carry untrusted instructions.
• Failing to test the whole chain: A safe first response can become unsafe when passed to a second model, parser, plugin, or automation step.
• Fixing without regression coverage: Add every confirmed failure to a versioned test set and run it after prompt, model, retrieval, or tool changes.
• Publishing sensitive evidence: Share the exploit category and sanitized result, not confidential prompts or credentials. A detailed internal record can coexist with a minimal external summary.

For general prompt quality and reliability checks, pair this security audit with an LLM evaluation checklist and a documented scoring rubric. Security and quality tests answer different questions, but both should be part of the same release workflow.

When to revisit

Run the complete checklist before the first production release and repeat it whenever the application’s trust boundaries change. At minimum, schedule a focused regression run after a system prompt edit, model migration, retrieval-source change, new connector, new tool, permission change, parser update, or workflow redesign.

Revisit the test corpus during seasonal planning cycles and after a relevant incident, vendor control change, or newly observed attack pattern. Review whether the application now accepts new file types, languages, channels, or external content. Also remove stale cases and add scenarios based on real user behavior, near misses, and newly introduced capabilities.

A practical release gate is:

1. Freeze the versioned prompts, tools, permissions, and retrieval configuration for testing.
2. Run direct, indirect, exfiltration, tool-use, and workflow regression cases in an isolated environment.
3. Confirm that high-severity findings are fixed or formally accepted by an accountable owner.
4. Re-test mitigations with the original case and a small set of variations.
5. Record the release decision, test scope, exceptions, and next review trigger.

Keep this checklist alongside your deployment process rather than treating it as a one-time red-team exercise. That makes prompt injection testing part of normal AI development workflows and gives the team a concrete way to reassess risk whenever the application changes.