Generative AI & LLM Security (Apps, Data, and Ops)3 items

Threats & Controls Overview

Primary risks: prompt injection, data exfiltration via outputs, training or retrieval poisoning, latent policy bypass, and model supply-chain risk.

Controls: input/output filtering, allowlisted tool use, context isolation per tenant, retrieval guardrails, and robust audit logging.

Isolation: separate embeddings per tenant; encrypt RAG indices; never mix contexts.
Filtering: pre- and post- processing (PII, secrets, safety policies).
Tool use: allowlist function calls, data-economy budget per session, and human-in-the-loop on high-risk actions.
Telemetry: log prompts, tool calls, and red-team prompts for detection; protect logs (they may contain sensitive text).

Retrieval-Augmented Generation (RAG) Hardening

Index governance: redact secrets and PII at ingest; classification tags drive access filters.
Query-time policy: restrict corpus by tenant + purpose; limit chunk size and number.
Poisoning defense: dedupe near-duplicates; verify provenance; quarantine untrusted sources.
Response guardrails: block data exfil indicators (e.g., large base64, secrets patterns).

Model & Dependency Supply Chain

Use signed model artifacts; verify checksums; pin versions.
Scan vector DBs/embeddings for policy violations (e.g., personal data)
Document provenance of datasets and fine-tuning corpora; maintain SBOM for AI stack.