Guardrails & Safety

The two-sided model

user input ──▶ [INPUT GUARDRAILS] ──▶ LLM / agent ──▶ [OUTPUT GUARDRAILS] ──▶ user / tool
                     │                                       │
                     ▼                                       ▼
                block · sanitize · rewrite           block · redact · re-ask · escalate

Input guardrails

• Prompt-injection detection — flag "ignore previous instructions", role-override attempts, or instructions hidden in retrieved docs / tool outputs.
• Jailbreak classifiers — Llama Guard, ProtectAI, Lakera Guard, Azure Prompt Shields.
• PII / secrets scanning — strip emails, phone numbers, card numbers, API keys before they hit a third-party model.
• Topic & scope checks — reject off-topic queries to keep an HR bot from giving medical advice.
• Rate limiting & abuse — per-user quotas, anomaly detection on prompt length/entropy.
• Language & locale gates — only accept supported languages.

Output guardrails

• Schema / structured output validation — Pydantic, Zod, JSON Schema; re-ask on failure.
• Toxicity, hate, self-harm, sexual content — moderation APIs (OpenAI, Perspective, Azure Content Safety).
• PII leakage checks — verify the model didn't echo private data from context.
• Groundedness / hallucination checks — ensure cited facts appear in retrieved sources.
• Competitor & brand mentions — block disallowed terms.
• Executable-content sanitization — strip <script>, suspicious URLs, shell commands.
• Tool-call gating — require human approval for destructive actions (payments, deletes, emails).

Implementation patterns

Schema validation with retry

from pydantic import BaseModel, ValidationError

class Answer(BaseModel):
    summary: str
    citations: list[str]

for attempt in range(2):
    raw = llm.invoke(prompt)
    try:
        return Answer.model_validate_json(raw)
    except ValidationError as e:
        prompt = f"{prompt}\n\nYour last reply failed: {e}. Reply ONLY valid JSON."
raise RuntimeError("model could not produce valid output")

Layered moderation

async function safeAnswer(userInput: string) {
  if (await isPromptInjection(userInput)) return refuse("injection");
  if (await hasPII(userInput))            userInput = redactPII(userInput);
  if (!inScope(userInput))                return refuse("scope");

  const draft = await llm.complete(userInput);

  if (await isToxic(draft))               return refuse("toxic");
  if (!isGrounded(draft, sources))        return reAsk();
  return draft;
}

Frameworks & tools

• NVIDIA NeMo Guardrails — Colang policy DSL, dialog rails.
• Guardrails AI — Python validators, RAIL specs, Guardrails Hub.
• Llama Guard 3 / Prompt Guard — open classifiers from Meta.
• OpenAI Moderation, Azure AI Content Safety, AWS Bedrock Guardrails, Google Vertex Safety Filters.
• Lakera Guard, ProtectAI Rebuff, Promptfoo red-teaming.
• Pydantic / Zod / Instructor / Outlines for structured-output safety.

Responsible AI principles

Governance program

• AI inventory — every model, prompt, agent, dataset, and use case registered.
• Risk tiering — classify by impact (informational ↔ autonomous decision) and gate accordingly.
• Model cards & system cards — purpose, training data, limitations, evals.
• Policy as code — guardrail configs in version control, peer reviewed.
• Pre-launch review — privacy, security, legal, accessibility sign-off.
• Continuous evaluation — LangSmith, Promptfoo, custom dashboards; alert on drift.
• Incident response — runbooks for prompt-injection, data leak, harmful output.

Regulatory landscape (informational)

• EU AI Act — risk-based obligations; high-risk systems require risk mgmt, data governance, transparency, human oversight.
• NIST AI RMF — Govern, Map, Measure, Manage functions.
• ISO/IEC 42001 — AI management system standard.
• GDPR / CCPA — lawful basis, DSARs, automated-decision rights.
• Sector rules — HIPAA (health), GLBA/SR 11-7 (finance), FERPA (education).

Production checklist

• Default-deny: guardrail failures block, not warn.
• Log every guardrail decision with input hash, rule, and outcome.
• Separate system instructions from data; never trust retrieved text as instructions.
• Treat tool outputs and web content as untrusted input — re-run input rails.
• Red-team continuously; keep a regression suite of past jailbreaks.
• Provide users a clear refusal message and a feedback channel.