PromptProof
Failures ZooHow it worksRoadmapGitHub
Add the GitHub Action
Back to home

LLM Failures Zoo

A curated collection of real-world LLM failures. Learn from production incidents, understand failure patterns, and build better guardrails.

Showing 8 of 8 cases

JSON field drift breaks downstream parser

Prompt RegressionGPT-4oProduction • JSON Schema

Model returns null instead of expected string type, causing parser crash

Problem

After prompt refactor, the status field started returning null instead of string

Minimal Repro

prompt: Extract order status from the following text
context: Order #12345 is currently being processed

Fix

Added explicit type constraint in prompt

// .promptproof.yml
tests:
  - name: order-status-extraction
    schema:
      type: object
      required: ["status"]
      properties:
        status:
          type: string
          enum: ["pending", "processing", "shipped", "delivered"]

Output Diff

-"status": null
+"status": "processing"
Permalink

PII slip in support reply

PII LeakClaudeSupport • Regex

Full email and phone number exposed in customer support response

Problem

Model included sensitive customer data in public-facing response

Minimal Repro

prompt: Generate a support response for order inquiry
context: Customer John Doe (john@example.com, 555-0123) asking about order #789

Fix

Added PII detection regex patterns

// .promptproof.yml
tests:
  - name: no-pii-leak
    deny_patterns:
      - '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}'
      - '\b\d{3}[-.]?\d{3}[-.]?\d{4}\b'
      - '\b\d{3}-\d{2}-\d{4}\b'

Output Diff

-email you at john@example.com
+contact you through your registered method
Permalink

Tool hallucination triggers phantom calendar event

Tool MisuseGPT-4oProduction • Custom Function

Model invents non-existent tool function causing system errors

Problem

LLM called schedule_meeting() which doesn't exist in tool registry

Minimal Repro

prompt: Help me schedule a meeting for tomorrow at 2pm
context: Available tools: send_email, check_calendar

Fix

Whitelist allowed tool names in validation

// .promptproof.yml
tests:
  - name: valid-tool-use
    custom: |
      const output = JSON.parse(response);
      const allowedTools = ['send_email', 'check_calendar'];
      return allowedTools.includes(output.tool);

Output Diff

-"tool": "schedule_meeting"
+"tool": "check_calendar"
Permalink

Summary invents fact with high confidence

HallucinationLlama 3Production • Semantic Similarity

Model adds information not present in source text

Problem

Summary included "40% revenue increase" not mentioned in original report

Minimal Repro

prompt: Summarize this quarterly report
context: Q3 showed steady growth with improved customer satisfaction scores

Fix

Added factual grounding check

// .promptproof.yml
tests:
  - name: factual-summary
    grounding:
      method: semantic_similarity
      threshold: 0.85
      source: context

Output Diff

-40% revenue increase and steady growth
+steady growth
Permalink

Refusal regression after prompt refactor

Prompt RegressionClaudeProduction • Regex

Model starts refusing legitimate requests after prompt update

Problem

New safety instructions too restrictive, blocking valid use cases

Minimal Repro

prompt: Generate marketing copy for our new product
context: Product: EcoClean detergent, eco-friendly and effective

Fix

Adjusted safety boundaries in system prompt

// .promptproof.yml
tests:
  - name: no-false-refusal
    deny_patterns:
      - 'I cannot'
      - 'I am unable'
      - 'I won\'t'
    context: legitimate_requests

Output Diff

-I cannot generate marketing content
+EcoClean: The eco-friendly detergent
Permalink

Unsafe SQL generation allows injection

Unsafe OutputGPT-4oProduction • Regex

Generated SQL query vulnerable to injection attacks

Problem

Model directly interpolates user input without parameterization

Minimal Repro

prompt: Generate SQL to find user by email
context: Email: user@test.com; DROP TABLE users;--

Fix

Enforce parameterized query generation

// .promptproof.yml
tests:
  - name: safe-sql
    deny_patterns:
      - 'DROP\s+TABLE'
      - 'DELETE\s+FROM'
      - '--'
    require_patterns:
      - '\?|:\w+|\$\d+'

Output Diff

-WHERE email = 'user@test.com; DROP TABLE
+WHERE email = ?
Permalink

Output format inconsistency breaks API

Prompt RegressionGeminiProduction • JSON Schema

Model switches between array and object format randomly

Problem

Inconsistent response structure causes API parsing failures

Minimal Repro

prompt: List product features as JSON
context: Product has: waterproof, lightweight, durable

Fix

Strict JSON schema validation

// .promptproof.yml
tests:
  - name: consistent-format
    schema:
      type: object
      properties:
        features:
          type: array
          items:
            type: string

Output Diff

-"features": "waterproof, lightweight"
+"features": ["waterproof", "lightweight"]
Permalink

Infinite loop in chain-of-thought

Tool MisuseGPT-4oProduction • Custom Function

Model enters recursive reasoning loop, exploding token usage

Problem

Unclear termination condition causes endless self-prompting

Minimal Repro

prompt: Think step by step to solve: what is 2+2?
context: Show all reasoning

Fix

Token limit and step counter

// .promptproof.yml
tests:
  - name: bounded-reasoning
    max_tokens: 150
    custom: |
      const steps = response.match(/Step \d+/g);
      return steps && steps.length <= 3;

Output Diff

-Step 4: Let me reconsider...
+Answer: 4
Permalink