AI analysis governance
How to Keep AI Crosstab Analysis Evidence-Linked, Approvable, and Undoable
Give the model only a bounded, disclosed analytical context; bind every factual cell claim to a stable evidence ID and independently resolve it against the computed table; fail closed when a citation does not match; keep answers read-only; express requested project changes as typed, previewable proposals; require a named human to approve or reject them; recheck the project hash before atomic apply; record the evidence, model, prompt version, actors, decisions, and hashes; and allow exact undo only while it cannot erase later work.
Published by crosstabs.com · Named statistical review pending · Last updated
Run this on your own data — free, no signup
Upload a CSV or XLSX. Everything runs in your browser; your file never leaves your device.
Open the workspace →Treat evidence, decisions, and state transitions as different objects
A polished answer is not evidence, and an evidence-linked answer is not permission to change a project. Keep the computed result, generated prose, proposed operations, human decision, applied state, and recovery record separate so each can be verified.
| Stage | Controlled object | Enforcement | Retained evidence |
|---|---|---|---|
| Bound context | Computed table, filters, weight, wave, table ID | No respondent rows for aggregate analysis | Context hash and disclosed payload |
| Generate | Model receives the bounded analytical context | Read-only output | Provider/model boundary and prompt contract |
| Validate | Resolve every cell pointer against the computed result | Fail closed on invalid link, label, row, or column | Accepted evidence IDs or withheld issues |
| Propose | Convert a requested mutation into allowed typed operations | No direct project write | Evidence-linked before/after diff and dependencies |
| Approve | A named human selects or rejects operations | Human decision required | Reviewer, time, selected and rejected operations |
| Apply | Recompute the approved base hash and apply atomically | Reject a stale approval | Before hash, after hash, actor, time |
| Undo | Restore the recorded pre-apply snapshot | Refuse if later work changed the post-apply state | Undo actor, time, and retained plan history |
Use application-enforced evidence pointers, not decorative citations
A cell pointer should resolve to row and column coordinates plus labels, counts, percentages, expected values, residuals, base, warnings, filters, weight, wave, and table identity under a stable context hash. Validate the claim beside each pointer so one valid citation cannot lend credibility to a different sentence.
NIST's Generative AI Profile recommends reviewing and verifying sources and citations during pre-deployment measurement and ongoing monitoring. For crosstabs, that review can be executable: a bad pointer, wrong label, wrong row, or wrong column causes the model wording to be withheld rather than merely flagged.
Worked example
Worked example
An analyst asks the AI to explain a Region × Satisfaction table and add three demographic cuts. The answer claims that North × Satisfied is high and cites R1C2. The application accepts the prose only if the same claim names the resolved row and column labels and the pointer maps to that computed cell. Otherwise it withholds the prose and renders a deterministic summary from verified cells.
Adding cuts is a separate operation. The model can propose a multibanner and tables, but the proposal changes nothing. The analyst previews the typed diff, approves the banner and two tables, rejects the third, and applies the approved dependency-complete set. If the project changed after approval, apply is blocked until a new preview and approval are recorded.
The applied plan retains its evidence, provider, served model, prompt version, actors, timestamps, before and after hashes, and pre-apply snapshot. The engine refuses exact undo when later edits would be overwritten; the safe recovery is another reviewed forward plan.
How to interpret it
Rule of thumb
Governance is strongest when the model has the least authority needed. Provider failover can keep a request available, but every provider remains behind the same deterministic evidence and approval contract. Model agreement, confidence, fluency, or brand reputation is not a substitute for statistical evidence or a human decision.
Make approval selective, dependency-safe, and stale-state aware
Every proposed operation should name its evidence, risk, affected object, before state, after state, and dependencies. Partial approval is legitimate only when every required dependency is also approved; otherwise the preview must refuse the selection.
Record the approved base hash. At apply time, recompute it. If the project changed after approval, the previous decision does not authorize the new state: generate a fresh preview and obtain a new approval. Apply an accepted set atomically so the project cannot be left half-mutated.
Preserve recourse without overwriting later work
Store a pre-apply snapshot and the exact post-apply integrity hash. Exact undo is allowed only while the current project matches that post-apply hash. If another edit occurred, the engine refuses exact undo because restoring the snapshot would silently destroy work.
This guard implements recourse without pretending every state can be rewound safely. Retain the original plan and its decisions, then use a new evidence-linked forward change when later work exists.
Current Crosstabs product boundary
In the current Workspace, aggregate AI analysis sends the disclosed table context—not source rows or the filename—through Vercel AI Gateway. Cell citations are resolved against the computed table. A mismatch withholds the model prose and substitutes a verified deterministic summary. These answers are read-only.
The visible Governed AI changes panel sends project metadata without respondent rows and supports typed proposals to create sections, multibanners, and tables or set table status. It exposes preview, dependency-safe selection, approve, reject, atomic apply, and conflict-checked undo. That operation set does not authorize arbitrary recodes, filters, weights, row edits, or exports.
The project change plan is persisted inside the local project graph with evidence, provider, served model, prompt version, actors, timestamps, statuses, hashes, and the guarded undo snapshot. By contrast, the AI answer activity list is session-level and is not a durable project audit log. Open-end coding has its own reviewed, evidence-linked approval and undo workflow.
Measure the control system, not only model quality
Track citation-integrity failures, withheld answers, proposal validation failures, approvals, rejections, partial approvals, stale-state conflicts, applies, undo attempts, refused undos, user corrections, and provider/model versions. Evaluate representative tables and risky edge cases whenever prompts, models, routing, or statistical output contracts change.
NIST's AI RMF makes human oversight, systematic documentation, monitoring, and risk response lifecycle activities rather than a one-time checklist. Its Playbook also recommends histories, audit logs, override statistics, and documented go/no-go decisions.
Frequently asked questions
- What makes an AI crosstab answer evidence-linked?
- Each factual cell claim should point to a stable evidence record containing the table context, row and column coordinates, labels, count, percentages, base, warnings, filters, weight, wave, and table identity. The application—not the model—must resolve that pointer back to the computed result and reject mismatches.
- Should an AI answer be allowed to edit a research project directly?
- No. Read-only analysis and project mutation should be separate capabilities. A requested change should become a typed proposal with a previewable before/after diff, evidence, dependencies, risk, and allowed operation set. A human can approve selected operations, reject the plan, or leave it unapplied.
- How do conflict checks prevent stale AI changes?
- Hash the governed project scope when the proposal is created and again at preview, approval, and apply time. If the project changed after approval, the approved diff is stale and must be previewed and approved again instead of being applied to a different state.
- When is an AI-created change safely undoable?
- An exact snapshot restore is safe only while the current governed project state still matches the recorded post-apply hash. If later work changed that state, exact undo must be refused because it would erase subsequent edits; use a new forward change instead.
- Does using several model providers make an answer more trustworthy?
- No. Provider routing and fallbacks can improve availability, cost, or latency, but they do not validate statistical claims. Evidence resolution, deterministic checks, human review, audit history, and product-specific evaluation remain necessary regardless of which model served the request.
References & further reading
Try it on your own data — free, no signup
Upload a CSV or XLSX. Everything runs in your browser; your file never leaves your device.
Open the workspace →