Response Provenance: Tracing Agent Claims Back to Their Causes

Backstory: When me and my colleague igaurab started adding artifact/spec-driven design of agent development in AKD Labs — the Agent Co-Design Studio [source] — where agent specs (roles, scope, output, guardrails, etc.) are guided entirely based on artifacts (a bunch of text files as artifacts), we thought of adding agent response traceability as a feature.

The idea was initially: since multiple SMEs/users would collaborate and “co-design” an agent’s specs, an agent response could maybe theoretically be traced back to which artifact from which user contributed to the agent response. By “response”, I mean: the agent produces a response, and a specific segment of that response would maybe be traced back and pinpointed directly to (artifact, user). This would help a user interacting with the final agent get thorough transparency of the chats.

But it remained in our backlog. The idea kept lingering throughout, for so many months, in the back of my head, as background white noise. But since right now coding agents like Claude Code have become so good at implementation, I started studying the current literature and such — and finally found that this problem is unique in itself: that I would want to use a black-box, post-hoc approach to try to solve this, as a proxy.

Hence, this technical report.

1. Introduction

Language agents are increasingly used for work whose output is expected to be defensible rather than merely plausible: literature synthesis, data discovery, analysis over curated document collections. In these settings the agent’s answer is not the end of the task but the beginning of one, since somebody must decide whether to act on it. That decision requires knowing where the answer came from.

This report is concerned with a specific and, we argue, under-served part of that requirement. Not whether an agent’s answer is correct, which is a question about the world, but where each part of it originated, which is a question about the agent’s context and is answerable from records that agent systems already keep. We call this response provenance, formalise it in §2, and spend the remainder of the report on the observation that the deployment setting most agents occupy makes the standard formulation of the problem unobservable, and on what can be measured in its place.

1.1 Background

A language agent answering a technical question does not draw on a single source. Over the course of one turn it may retrieve documents, call tools whose returns enter its context, operate under a configuration that constrains what it is permitted to assert, and condition on several turns of prior conversation. Its output is a single stretch of prose in which contributions from all of these are interleaved, and in which the model’s own parametric knowledge is interleaved with them indistinguishably.

Current agent interfaces expose the execution trace: which tools ran, with which arguments, and what they returned. This is a record of the agent’s actions. It is not a record of what the agent’s assertions rest on, and the two are not recoverable from one another. A tool may execute and its return may go unused; a claim may be shaped decisively by a configuration rule that appears in no tool call at all. Between the execution log and the generated text there is a gap that the interface does not cross.

1.2 Motivation

The consequence of that gap is not primarily a matter of hallucination. It is that an assertion whose origin is unknown cannot be audited, and an assertion that cannot be audited cannot be corrected, cited, or built upon.

Consider what a domain expert must do when an agent returns a value they believe to be wrong. Three distinct conditions are consistent with that observation, and each requires a different intervention: the governing document was absent from the agent’s working set, so it must be supplied; the document was present but structured such that the agent misread it, so its structure must be revised; or the document was present and legible and the model answered from parametric knowledge regardless, so a constraint must be added. These are not variations on one repair. They are three separate repairs, and choosing between them requires knowing which of the three occurred. Without provenance the expert cannot distinguish them and must proceed by guesswork.

The same gap has a second consequence, at the layer of the artefact rather than the error. A claim accompanied by a verified pointer to its origin can be reproduced by a third party and entered into a methods section. A claim without one cannot, irrespective of whether it happens to be correct. Provenance does not establish that an answer is right; it establishes what would have to be examined in order to find out.

1.3 The gap in existing approaches

The natural response to the foregoing is to require the agent to cite its own sources inline. This is what OpenAI’s citation-formatting convention supports [1], and what deep-research products do.

A 2026 audit of exactly this shows why it is insufficient. Onweller et al. [2] parsed and fact-checked the citations that frontier deep-research agents produce. Link validity exceeded 94% and topical relevance exceeded 80%, so the citations present as sound. Factual support was 39–77%. Worse, fact-check accuracy degraded by roughly 42% as tool calls scaled from 2 to 150: doing more research produced less reliable citation.

Two conclusions follow. First, a citation the model declares is itself a generated claim about provenance, produced by the same process and subject to the same failure modes as the prose surrounding it; it is a hypothesis about origin rather than evidence of one. Second, declared citations cover only the text the model elected to mark. An unmarked sentence, whether a judgement, a caveat, or an unsupported quantity, remains unexplained; and those are frequently the sentences most in need of examination.

Declared citation is also structurally incapable of reaching two of the three sources that drive an agent’s output. It can point to a retrieved document. It cannot point to the agent’s own configuration, nor to an earlier turn of the conversation, and no citation convention has an agent annotate a sentence with the rule that required it.

1.4 Contribution

This report makes the following contributions:

1. A formalisation of response provenance, and its separation from declared-citation auditing and from corroborative fact-checking (§2, §3).
2. An argument that the contributive question is structurally unavailable on hosted APIs, with the measurements that establish it (§4, §5).
3. An estimator for that setting: a forced-choice judged scalar in place of the unobservable probability, exact leave-one-out over coarse sources rather than sparse regression, a layer-first hierarchical search, and span localisation with mechanical verification (§6), given as complete pseudocode with every parameter (§7).
4. An accounting of what the output licenses, including the finding that the central quantity is unfalsifiable in this setting, plus a live worked example (§8) and a statement of limits (§10).

2. Problem formulation

This section fixes the objects the rest of the report reasons about. We define a turn and the partition of its context, state what a segment is and what an attribution over segments returns, and introduce ablation as the single intervention available to a black-box method. We then separate three questions that are routinely treated as one: whether a declared citation holds, whether a source supports a claim, and which source caused it. The separation matters because a method answering one of them is blind to the failure modes of the others, and much of the design in §6 follows from taking it seriously.

Throughout, we assume only what a persisted agent turn ordinarily records: the query, the tool calls with their arguments and returns, the agent’s configuration, and the prior messages. No assumption is made about the model beyond the ability to sample from it.

2.1 The turn

A single agent turn is a triple

where $q$ is the user’s query, $C$ the context the model conditioned on, and $r$ the generated response. The context is an ordered collection of sources

Critically, $C$ is not just retrieved documents. In an agent setting it partitions into exactly three layers of provenance:

Layer 1 — tools and artifacts ($C_{\text{tool}}$). This turn’s tool calls, each paired with its arguments and its return. One structural point here is easy to get wrong: a workspace artifact is not a fourth layer. An agent does not receive its artifacts through a separate channel; it reads them on demand through a console toolset — read_file, ls, grep, glob. An artifact load is therefore literally a tool call whose arguments carry the path and whose return carries the file content. Modelling artifacts as their own layer would double-count them and leave every attribution ambiguous between the two. They live here, and ClassifyTool (§7.1) merely labels them artifact_read so the interface can name them to a user. The same holds for MCP tools, web fetches and web searches: different kinds within one layer, because they are all “something the agent went and got during this turn”.

Layer 2 — the agent’s own instructions ($C_{\text{instr}}$). The system prompt: role, scope limits, refusal rules, output conventions. This is the layer no citation convention reaches, and it is frequently the true cause of a sentence — a hedge, a refusal, an “I can’t certify that” is more often mandated by a prompt rule than derived from any artifact. It is treated as one indivisible source: ablation assumes sources are independent, but a system prompt is a single instruction set, so deleting one section leaves a prompt the model has never seen the like of and the resulting score change reports incoherence rather than attribution.

Layer 3 — the conversation so far ($C_{\text{hist}}$). Prior user messages and agent replies. A constraint the user set three turns ago — “only ever use Baldwin County” — can drive a claim in the current turn with no artifact and no instruction involved.

Two things are deliberately not layers:

• The current user message $q$. It is the query, held fixed. Ablating it asks “what if the user had asked nothing”, which is a different question; ContextCite ablates the context, not the query. Treating it as a source hands the history layer a spurious effect on every claim.
• Attached files whose content the trace does not retain. The record keeps their names but not their bytes, so their text cannot be recovered post-hoc. Rather than omit them silently — which would make a claim grounded in one read as unsupported for no visible reason — they are returned separately as unscorable context, and the interface says so.

2.2 Segmentation and the attribution target

The response is segmented into claim-sized units

Sentence granularity is the practical choice: it is the smallest unit that carries a complete assertion, and it is what a reader can select. Following ContextCite [3], a target may in fact be any contiguous token span $r_i \dots r_j$, so arbitrary user selections cost no extra formalism.

The object we want is an attribution map

assigning each source a responsibility score for segment $r_k$.

2.3 Ablation as the intervention

All ablation-based attribution shares one primitive. For a mask $v \in {0,1}^d$,

is the context with the masked-out sources removed. Two masks are distinguished throughout: $\mathbf{1}$ (everything present) and $\mathbf{0}$ (everything removed). Write $\mathbf{1} \ominus i$ for the all-ones mask with position $i$ zeroed, and $\mathbf{1} \ominus L$ for the all-ones mask with every source in layer $L$ zeroed.

2.4 Three questions, routinely conflated

The literature separates two questions and practice adds a third. Conflating them produces mush, so:

The distinction is not academic. Suppose the agent misreads a file and states 1 km where the file says 5 km. A corroborative method finds nothing — no source supports “1 km”, so the claim looks unsupported and unexplained. A contributive method points straight at the misread passage. The failure mode of greatest consequence is the one corroboration is blind to.

Conversely, contributive attribution alone yields a fact about model mechanics (“source 5 caused this, weight 0.62”) that a scientist cannot act on. Knowing which artifact caused a claim does not tell you whether the claim or the artifact is right.

So the useful output needs both axes: find the responsible source interventionally, then interrogate that source directly.

3. Related work

Work bearing on response provenance divides along the distinction drawn in §2.4. One line asks which part of the context caused a generation and answers it by intervening on the context; a second asks whether a claim is supported by a given source and answers it by comparison; a third, more recent, audits the citations that agents emit of their own accord. The three have different requirements and different blind spots. The method proposed here borrows its protocol from the first, while being unable to use any of that line’s instruments directly.

We review each line in turn, with attention throughout to a property that is rarely stated explicitly in this literature but is decisive in practice: what access to the model each method presumes.

3.1 Contributive attribution

ContextCite [3] is the reference formulation. Sources are context sentences. It samples $m \approx 32$ random masks, and for each computes the logit-scaled probability that the model would produce the target statement given the ablated context:

then fits a sparse linear surrogate by Lasso [4]:

The weights $\hat w$ are the attribution scores. It is elegant, well-validated, and — as §4 shows — depends entirely on a quantity a hosted chat API will not return.

AttriBoT [5] attacks the cost of leave-one-out attribution with cached activations, hierarchical coarse-to-fine attribution, and small proxy models standing in for the large target, reporting

300× speedup while tracking the target’s LOO error more faithfully than prior methods. The hierarchical idea transfers to our setting even though the caching and proxy tricks do not.

ARC-JSD [6] is the current mechanistic state of the art: it identifies essential context sentences via Jensen–Shannon divergence without fine-tuning, gradient computation, or surrogate modelling. Being a mechanistic study of internal behaviour, it presumes access to the model’s distributions — the opposite of the constraint we operate under.

TokenShapley [7] pushes granularity to individual tokens by combining Shapley-value data attribution [8] with KNN retrieval, reporting 11–23% accuracy gains for keyword-level attribution — numbers, years, names. It is the right granularity for the claims scientists care about, and it is white-box and expensive.

The frontier is moving toward mechanistic, white-box methods. Under an API-only, no-new-infrastructure constraint, there is no state-of-the-art method to adopt.

3.2 Corroborative attribution and its evaluation

Automatic attribution evaluation — given a claim and a cited source, is the claim supported? — is harder than it looks. AttributionBench [9] finds that even a fine-tuned GPT-3.5 reaches only ~80% macro-F1 on the binary formulation, with errors concentrated in nuanced cases and in disagreements between model knowledge and annotator judgement. Related benchmark work on knowledge-aware attribution [10] reaches similar conclusions about headroom.

This sets a hard ceiling on any LLM-judged verdict, and it is the single most important number for interface design: a verdict from this family is a reading, not a proof, and must never be rendered as one.

3.3 Claim decomposition

FActScore [11] decomposes long-form generations into atomic facts before scoring. That would split our example sentence into a checkable claim (“1 km resolution”) and a judgement (“too coarse for urban work”) and let them receive separate verdicts. It is a real improvement over sentence granularity and it costs a model call per segment; we treat it as deferred, not dismissed.

3.4 Position of the present work

Against §2.4, the 2026 audit [2] answers the declared-citation question: parse what the agent emitted, then check it. It cannot attribute an unmarked segment, and it has nothing to say about instructions or conversation history. ContextCite and ARC-JSD answer the contributive question but need access we lack. The corroborative line answers a different question and is blind to misreads.

The gap is segment-to-source tracing across all context layers, from generation access alone — which is what the rest of this report builds.

4. Why the state of the art is unavailable

The constraint is: a hosted chat completions API. No weights, no activations, no teacher-forced scoring, no new inference infrastructure.

Every method in §3.1 needs one of:

The blocking requirement is easily missed, and bears stating exactly. ContextCite needs the probability that the model would have produced a specific string it did not just produce. That is a teacher-forced quantity. Chat completions endpoints return log-probabilities only for tokens they generate. There is no echo. And the natural workaround — put the target in a trailing assistant message and read the logprobs of its continuation — fails because the API restarts rather than continues a trailing assistant turn.

We verified this directly: prompting with a trailing assistant message "The three primary colors are red," produced a restarted sentence, not a continuation.

So the scalar at the heart of the best-validated method is not merely expensive here. It is unobservable.

5. Negative results

Before the method, the three designs that failed, because each failure constrains what remains. All were measured against real stored agent turns, at a total API cost of roughly $0.44.

5.1 Assistant prefill leaks the evidence

To localise attribution to segment $r_k$ rather than the whole response, the natural move is to supply the response prefix $r_1 \dots r_{k-1}$ and score only $r_k$. Since prefill is unavailable (§4), the prefix must be supplied by instruction — “continue this text”.

This destroys the experiment. The prefix contains the evidence. Ablating all six sources of a real turn moved the score by 0.004 — no signal at all, because the answer was already in the prompt.

Consequence. The claim must be scored standalone. Prefix-based localisation is unavailable, so segment identity has to come from the segmenter, not from conditioning.

5.2 Generated-text similarity is too noisy for a regression

If probability is unobtainable, an obvious substitute is: regenerate under the ablated context and measure similarity between the regeneration and the original claim.

Measured on identical masks, similarity came out 0.165 / 0.743 / 0.870. Variance exceeded signal. A Lasso fit on this scalar assigned a negative weight to the single most relevant source — a confidently wrong attribution, which is worse than an absent one.

5.3 First-token log-probability is uninformative

Scoring the first token of the claim is cheap and defensible in principle. In practice agent responses begin with markdown (**), which is perfectly predictable regardless of context. The quantity carries no attribution signal. Separately, top_logprobs caps at 5 and support is per-model.

5.4 The epistemic problem

The deepest issue is not noise. It is that in this setting contributive attribution has no ground truth. Its ground truth is the log-probability of a supplied target, which the API will not return. Validating an approximate scalar against leave-one-out of the same approximate scalar is self-consistency, not validation.

This is worth saying plainly because it is a real epistemic limit, not a temporary engineering gap: on a hosted API you can build something useful, but you cannot show it is faithful in the sense a probability-based method can.

6. Method

Two constraints established above determine the shape of what follows. From §4, the scalar that contributive attribution is defined over cannot be observed through a hosted inference API, so any method operating there must substitute something else for it. From §5, three candidate substitutes fail for measured reasons: supplying the response prefix leaks the evidence into the prompt, similarity between regenerations is too noisy to regress on, and the first-token log-probability of a claim carries no signal. What remains is a judged scalar, which is not a probability and does not behave like one.

We therefore describe a method that retains the interventional structure of ablation-based attribution while accepting a weaker measurement inside it, and which is organised so that the weakness is contained rather than propagated. It proceeds in four tiers of increasing cost. The first resolves or discards most of a response without calling a model at all (§6.1). The second introduces the judged scalar and states what it does and does not measure (§6.2). The third performs the attribution itself, by exact leave-one-out over coarse sources under a hierarchical search (§6.3, §6.4), together with the test that separates parametric knowledge from redundancy across sources (§6.5). The fourth localises a claim to a span of text and verifies that span mechanically (§6.6), which is the only step in the method whose output can be checked without trusting a model.

One consequence belongs before the details, because it inverts the emphasis a reader might expect. Because the scalar is coarse and corroborative, the ablation results are reported as an ordering over sources rather than as weights, and the verified span, not the ablation, carries the answer that a reader can act on. §6.7 then describes an optional probe that addresses the contributive question directly, at higher cost and with a strictly weaker conclusion.

6.1 Tier 0 — free gating

Most sentences should never get a provenance badge. Forcing citations onto greetings, questions, and offers — or worse, flagging them “ungrounded” — is a category error that trains users to ignore every badge, destroying the one signal that matters.

So each segment is first classified by cheap rules, at zero model cost:

• question / recommendation → agent voice: no citation expected, no warning.
• self-report (“I read the three config files”) → action: attributed structurally to the turn’s actual tool calls. Exact and free.
• disclaimer (“I can’t certify an official rate”) → traceable, because these are precisely the sentences the system prompt tends to mandate.
• verbatim overlap with a source → quoted: cited to path and line span, certain and free.

For the verbatim test, let $b$ be the longest common contiguous block between segment $r_k$ and source $c$. It counts as a quote iff

with $\ell_q = 40$ characters and $\rho = 0.6$. Both conditions are required. Length alone fails badly in scientific domains: MODIS_Combined_L3_IGBP_Land_Cover_Type_Annual is 45 characters, so any sentence merely mentioning a layer ID cleared an absolute threshold and was filed as “quoted”, resolving it for free — while its actual claim (“…at the 2015 timestamp”) went unchecked.

Only segments asserting something specific and unresolved reach the metered tiers. The affordance is earned.

6.2 The scalar: forced-choice LLM-as-judge

Since $p_\theta(r_k \mid \cdot)$ is unobservable, we substitute a judged quantity. A judge model $J$ receives the query, the candidate sentence, and the ablated sources, and is required to answer with a single digit 1–9: 9 = fully and specifically supported, 5 = partial, 1 = nothing relevant. Define

Measured properties. On real trace rows this was perfectly stable — $\Delta = 0.000$ across repeated identical calls — with a strong context effect of $+4.000$ between all-sources and no-sources, and it correctly isolated the single responsible source. The distribution is degenerate in practice, landing on exactly 1.000 / 5.000 / 9.000, so the expectation equals the argmax equals a plain parse of the digit; reading token log-probabilities buys nothing and would tie the method to specific providers.

What it is. This substitution changes the semantics of the measurement: the judge scores support, not generation probability. Applying it under ablation therefore yields corroboration measured interventionally, which is neither a probability estimate nor a static entailment judgement, and which has no established name. It is also coarse. Use it to rank sources; never present it as a fine-grained weight.

6.3 Estimator: exact leave-one-out, not Lasso

Sparse-regression approaches use Lasso because the scalar is noisy and $d$ is large, sources being individual context sentences. Neither holds here: the scalar is stable (§6.2), and sources are whole evidence units — typically $d \approx 3\text{–}10$ rather than ~100. Under those conditions exact leave-one-out is both cheaper and assumption-free. For source $i$:

at $d+2$ calls, against 26–34 for a random-mask regression. Lasso is the right choice when one cannot afford $d$ measurements; here you can.

6.4 Hierarchical search: layers before sources

Following AttriBoT’s coarse-to-fine idea [5], round 1 ablates whole layers:

costing $|\mathcal{L}| + 2$ calls, and answering what kind of context drove the claim. Round 2 then ablates individual sources only within contributing layers. Sources outside them stay present in every mask — they remain context the model sees, just not candidates under test.

This is what keeps $d$ small without arbitrary narrowing: a long conversation carries dozens of prior turns, and cutting them to fit a budget makes which sources survived an arbitrary choice.

Total cost:

where $d^{\prime}$ counts sources in contributing layers. Also note $\Delta_L$ is reported for every contributing layer, not a single winner: instructions commonly shape a sentence’s form while a tool return supplies its fact, and collapsing to one winner discards half the finding.

6.5 Internal knowledge and the redundancy confound

Parametric knowledge requires no separate mechanism, and this has been observed before [3]: a claim the model produced from training rather than from context is indicated by attribution to nothing.

What that observation leaves unresolved is that near-zero attribution is ambiguous. It arises equally when a fact is redundant across several sources, so that removing any one of them changes nothing. Distinguishing the two cases is necessary for the verdict to mean anything, and it is not obtained from per-source scores at all. The $\mathbf{0}$ anchor separates the cases. Define the total context effect

Then, with thresholds $\tau_\Gamma = 0.75$ and $\tau = 0.5$ (the latter set at the scorer’s quantisation floor):

The first row says the claim survives with the entire workspace removed — internal knowledge, with no redundancy confound. The second says removing everything mattered but removing any one layer did not: several layers carry the claim independently, so none is individually responsible. Reporting that as “nothing contributed” would be a wrong conclusion, and ablating further would only produce another page of zeros.

Per-source roles use the same threshold:

A negative drop means removing the source raised apparent support — it was diluting the evidence. This was observed once on a real turn (dropping a 4.6k README raised the score by 4.000) but did not replicate on a rerun of the same claim. It is reported as an observation the mechanism permits, not a validated capability.

6.6 Span localisation with verified quotes

Ranking sources is rarely the whole question. “Which artifact” matters less than “which line”. So for each contributing source, one structured call returns a verdict in {supported, contradicted, silent}, a confidence, and — critically — the verbatim span justifying it.

The span is then verified mechanically before display. With $\nu$ a normalisation (decode literal JSON escapes, collapse whitespace, casefold), the predicate is

where $\sqsubseteq$ is substring containment. If verification fails the verdict is downgraded to silent and the quote discarded. A fabricated quote inside a provenance UI is worse than no quote: it manufactures the false confidence the method exists to prevent.

Two details earned by measurement. The escape decoding is functional rather than cosmetic: tool returns frequently arrive as JSON whose payload is itself a JSON string, so stored text contains \"layergroup\": \"Aerosol Albedo\" while any readable quote says "layergroup": "Aerosol Albedo". Without decoding, verification could never succeed against most MCP tools or any web fetch, and every honest quote was discarded as fabricated. And a single retry is warranted, because the same claim against the same source has produced a clean citation on one run and an unverifiable paraphrase on the next — so a verification failure and “this source says nothing” must not look alike.

The instructions layer needs a different question. A system prompt does not state a response, it mandates one. Asking whether the prompt “states” the claim finds nothing; asking whether any instruction directs this response resolves a refusal to the exact governing rule.

6.7 The contributive probe (opt-in, directional)

Everything above is corroborative. For the contributive question — would the agent still make this claim without that source? — there is one intervention the API permits: remove the source, regenerate the answer from the query alone, and test whether the claim reappears.

Presence is detected via rare anchors. Let $\mathcal{A}(r_k)$ be the claim’s discriminative anchors — identifiers, paths, numbers with units, dates — retaining only those not carried by nearly every source. With $G$ the generator and $\mathrm{cov}$ anchor coverage,

Run $k$ samples with everything present and $k$ without source $i$:

Why direction only, never a weight. Measured on a real turn, discrimination was stable — the responsible source scored 0.000 every time — but the baseline was not, with spread 0.500 even at $k=3$, because anchors can include tokens the agent invented rather than read. The baseline-stability guard ($B = k$) is therefore mandatory: an unstable baseline is reported as inconclusive rather than subtracted into a number that looks more precise than it is.

Both methods independently agreed on the same responsible source. That agreement is the main reason to trust the cheap tier as the default.

6.8 Cost model and refusal

Cost is computed from the actual corpus, never a constant, since a turn with ten large files costs an order of magnitude more than one with two small ones. With per-source token cap $\kappa$,

Note $|r_k|$ enters multiplied by call count — a long selection costs proportionally more, which the interface must disclose rather than quietly under-quote.

Over a hard ceiling, a run is refused with a reason rather than trimmed. A silently reduced corpus produces a confidently wrong attribution, which is worse than no answer.

7. The algorithm, as implemented

§6 gives the reasoning. This section gives the procedure, at the level of detail needed to reimplement it. Every constant is stated in §7.8.

7.1 Algorithm 1 — Corpus construction

Two input paths, in priority order. In practice the fallback dominates: most stored turns predate the migration that persists native message lists, so the flattened payload carries the corpus for the majority of history. Treating it as an edge case is a mistake.

ALGORITHM 1  BuildLayeredCorpus(model_messages, response_payload,
                                request_payload, system_prompt) -> [Unit]

 1  raw <- []
 2  if model_messages is non-empty:                       # primary path
 3      order <- [];  calls <- {}
 4      for each message, for each part in message.parts:
 5          if part.kind in {tool-call, builtin-tool-call}:
 6              key <- part.tool_call_id or synthesise("_anon" + |order|)
 7              if key not in calls: append key to order
 8              calls[key] <- {name: part.tool_name, args: AsDict(part.args), text: ""}
 9          else if part.kind in {tool-return, builtin-tool-return}
10                  and part.tool_call_id in calls:
11              if part.outcome not in {null, "success"}: continue   # failure is not evidence
12              calls[part.tool_call_id].text <- Flatten(part.content)
13      raw <- [calls[k] for k in order]                   # preserves call order -> ordinals
14  if raw is empty and response_payload is non-empty:     # fallback path (dominant)
15      raw <- [{name: e.name, args: AsDict(e.arguments), text: Flatten(e.response)}
16              for e in response_payload.tool_calls]
17
18  units <- []
19  for (name, args, text) in raw:
20      if Trim(text) is empty: continue        # a call with no result grounds nothing;
21                                             # keeping it adds a permanently-zero column
22      header <- name + "(" + json(args) + ")"          # the call is scorable, not decoration
23      body   <- header + "\n" + text
24      append Unit(id: "s" + |units|, kind: ClassifyTool(name), tool: name,
25                  label: Label(name, args), text: body, ordinal: |units|,
26                  layer: TOOLS)
27
28  # --- instructions layer: exactly one source, never split by heading ---
29  if Trim(system_prompt) is non-empty:
30      append Unit(id: "s" + |units|, kind: "instructions", tool: "system_prompt",
31                  label: "agent instructions (system prompt)",
32                  text: system_prompt,          # stored IN FULL; each tier truncates its own
33                  ordinal: |units|, layer: INSTRUCTIONS, unversioned: true)
34
35  # --- history layer: newest-first, current query excluded ---
36  usable <- [m in request_payload.messages
37             where m.role in {user, assistant} and m.content is a non-empty string]
38  if |usable| > 1:
39      prior <- usable[:-1] if usable[-1].role == "user" else usable   # drop the query
40      prior <- last HISTORY_MAX_MESSAGES of prior
41      for offset, m in enumerate(prior):
42          turns_ago <- |prior| - offset                  # counts back from the present
43          body <- Truncate(Trim(m.content), HISTORY_TOKENS_PER_MESSAGE)
44          append Unit(id: "s" + |units|, kind: "history", tool: m.role,
45                      label: RoleLabel(m.role) + " · " + turns_ago + " turn(s) ago",
46                      text: "[" + RoleLabel(m.role) + ", " + turns_ago + " turn(s) ago]\n" + body,
47                      ordinal: |units|, layer: HISTORY)
48  return units

Three details determine correctness:

• Ids are globally sequential across layers (s0, s1, …). A mask index must map to exactly one unit regardless of which layer it came from.
• Tool arguments are folded into the scorable text, not just the label. The agent’s own call is in its context, and the query it chose shapes everything that came back.
• ClassifyTool maps a name to a kind the interface can name to a user: artifact_read for {read_file, ls, glob, grep}, then web_fetch, web_search, mcp_tool if the name is dotted, else user_tool. A fetched URL and a search-results page are different things to go verify.

Label prefers the identity a scientist recognises — path, file_path, url, query, q — then falls back to summarising scalar arguments as k=v, then to naming argument keys without dumping their contents. Dumping raw JSON produced labels like worldview_permalink_tool({"layers": [{"id": "MODIS_Combined_L3_IGBP_Land_Co… — a truncated dump that named the tool twice and hid the one informative argument (t=2015-01-01).

7.2 Algorithm 2 — Anchor extraction

Anchors are what make a claim checkable, and they serve double duty: a gating signal (“does this sentence assert anything specific?”) and a presence detector for the causal probe.

ALGORITHM 2  Anchors(text) -> [token]

 1  patterns (case-SENSITIVE except the unit group, which scopes its own (?i:)):
 2      `backticked span`            `[^`\n]{2,80}`
 3      path                        \b[A-Za-z0-9_]+(/[A-Za-z0-9_.*+-]+)+\b
 4      ALLCAPS / SNAKE_CASE        \b[A-Z][A-Z0-9_]{2,}\b
 5      underscore identifier       \b[A-Za-z][A-Za-z0-9]*(_[A-Za-z0-9]+)+\b
 6      camelCase                   \b[a-z]+([A-Z][a-z0-9]+)+\b
 7      quantity + unit             \b\d+(\.\d+)?\s?(?i:km|nm|mm|cm|deg|%|kb|mb|gb|tb|hz|px)\b
 8      degrees                     \d+(\.\d+)?\s?°
 9      ISO date                    \b\d{4}-\d{2}-\d{2}\b
10      version / decimal           \bv?\d+\.\d+(\.\d+)?\b
11
12  found <- []
13  for match in scan(text):
14      stripped <- strip(match, "`*_.,;:()[]{}\"'")
15      token    <- lower(stripped)
16      if |token| < MIN_ANCHOR_LEN:            continue
17      if token in STOPWORDS:                  continue     # json, id, name, path, file, …
18      if not PathLike(token):                 continue
19      if isalpha(token) and |token| < 6
20             and not isupper(stripped):       continue     # word, not identifier — but
21                                                           # ALLCAPS acronyms survive
22      if token not in found: append token
23  return found

    PathLike(t): true if "/" not in t, else
                 t has a file extension, or >1 "/", or a digit/underscore in a segment

Every guard here was forced by a measured failure:

• Case sensitivity is required. An earlier version applied IGNORECASE to the identifier pattern, turning [A-Z][A-Z0-9_]{2,} into “any word of 3+ letters” and harvesting load, all, and, each. Presence detection then measured nothing.
• The acronym exemption (line 20) exists because the short-alpha guard, meant to reject load, also dropped NASA, MODIS, LST, IGBP. That left a refusal — “there isn’t a single NASA Worldview layer that provides an official deforestation rate” — with no anchors at all, so the one claim most worth tracing was filed as generic.
• PathLike exists because county/watershed and legal/administrative are ordinary prose that happens to use a slash, and counting them as paths made throwaway sentences look checkable while substantive ones went unmarked.
• Mixed-case underscore identifiers (line 5) needed their own pattern: ALLCAPS and camelCase both missed MODIS_Combined_L3_IGBP_Land_Cover_Type_Annual and QC_Day, the dominant convention in Earth-observation layer naming.

Two derived functions:

Discriminative filtering matters only for the causal probe: an anchor every source carries cannot distinguish between them, so testing it measures nothing.

7.3 Algorithm 3 — Free gating

ALGORITHM 3  GateTurn(evidence, segments) -> [Verdict]

 1  has_evidence <- |evidence| > 0
 2  for (id, text) in segments:                 # segments come from the RENDERER, never re-split
 3      (klass, anchors) <- Classify(text)
 4      if klass in {question, recommendation}:
 5          emit(id, klass, status: AGENT_VOICE, traceable: false);  continue
 6      if klass == disclaimer:
 7          emit(id, klass, status: TRACEABLE, traceable: true);     continue
 8      if klass == self_report:
 9          emit(id, klass, status: ACTION, traceable: false,
10               citations: [every tool call this turn]);            continue
11      if not has_evidence:
12          emit(id, klass, status: UNANCHORED, traceable: false);   continue
13      q <- BestVerbatim(text, evidence)
14      if q is not null:
15          emit(id, klass, status: QUOTED, traceable: false, citations: [q]); continue
16      if anchors is non-empty:
17          emit(id, klass, status: TRACEABLE, traceable: true);     continue
18      emit(id, klass, status: UNANCHORED, traceable: false)

    Classify(text):                              # order matters; see notes
 1      anchors <- Anchors(text)
 2      if text ends with "?":                      return (question, anchors)
 3      if SELF_REPORT matches:                     return (self_report, anchors)
 4      if DISCLAIMER matches:                      return (disclaimer, anchors)
 5      if RECOMMEND matches and anchors is empty:  return (recommendation, anchors)
 6      return (anchors ? anchored_assertion : generic_assertion, anchors)

The ordering of Classify encodes three corrections:

1. Self-report before recommendation. “I can read the file for you” is an offer; “I read the file and found X” reports work attributable to real calls. The SELF_REPORT pattern therefore carries a negative lookahead rejecting a modal right after the pronoun — otherwise “I can check” would be attributed to calls that never happened.
2. Disclaimer before recommendation. A mandated limitation usually also reads as an offer: “I can help you locate datasets … but I can’t certify an official rate” is both, and the limitation is the part with provenance. Disclaimers are traceable regardless of anchors, because their provenance is a directive, not a fact — there is nothing to anchor on but there is a rule worth citing.
3. Anchors beat modal phrasing. RECOMMEND matches a modal anywhere in the sentence, so “In practice, I would start with an Alabama bounding box: west −88.6, south 30.1, east −84.9, north 35.1” — four coordinates, the most checkable content in its response — was dismissed as agent voice because of “I would”. Requiring the absence of anchors separates a genuine offer from a fact phrased conditionally.

BestVerbatim takes the longest contiguous common block over all sources (difflib, autojunk disabled) and applies the two-condition test from §6.1, returning the citation with its line span.

7.4 Algorithm 4 — Hierarchical attribution (the main procedure)

ALGORITHM 4  TraceClaim(model, claim, query, C) -> Result

    # ---- round 1: which LAYER ----------------------------------------------
 1  L <- layers present in C, in display order (tools, instructions, history)
 2  masks <- [ 1^d , 0^d ] ++ [ mask_off(layer) for layer in L ]
 3  scores <- Measure(model, claim, query, C, masks)        # concurrent, capped
 4  s1 <- scores[0];  s0 <- scores[1];  per_layer <- scores[2:]
 5  Gamma <- s1 - s0
 6  for (layer, s_without) in zip(L, per_layer):
 7      Delta_layer <- Drop(s1, s_without)
 8      role_layer  <- Role(Delta_layer)
 9  contributing <- [layer : role_layer in {grounds, distractor}] ranked by |Delta_layer|
10  internal_knowledge <- (Gamma < TAU_GAMMA)
11  redundant <- (contributing is empty) and not internal_knowledge

    # ---- round 2: which SOURCE, within contributing layers only -------------
12  grounding <- null
13  if contributing is non-empty:
14      candidates <- [c in C : c.layer in contributing]  or  C
15      (S, omitted) <- SelectSources(candidates)         # layer-balanced, cap MAX_SOURCES
16      masks <- [ 1^d , mask_off(S) ] ++ [ mask_off({c}) for c in S ]
17      scores <- Measure(model, claim, query, C, masks)
18      for (c, s_without) in zip(S, scores[2:]):
19          Delta_c <- Drop(scores[0], s_without);   role_c <- Role(Delta_c)

    # ---- span localisation: ALWAYS, not gated on ablation -------------------
20  (targets, span_omitted) <- SelectSources(C)
21  supports <- concurrent[ CheckSupport(model, claim, c) for c in targets ]   # Algorithm 5

22  return Result(Gamma, internal_knowledge, redundant, layer scores,
23                per-source scores each carrying its span,
24                omitted: omitted ∪ span_omitted,
25                calls: actual count including retries, cost: provider-reported usage)

    Drop(full, without): 0 if either is NaN else full - without    # NaN = no information
    Role(d):  grounds if d >= TAU;  distractor if d <= -TAU;  irrelevant otherwise
    mask_off(X): all-ones with every position in X zeroed

Why span localisation runs unconditionally (line 20) is the single most important design decision in this algorithm, and it took a bug to find. Ablation and quoting answer different questions, and only one of them always works:

• Ablation answers did removing this change the answer — it cannot separate sources that each carry the claim, and it says nothing about where inside a source the claim lives.
• Quoting answers which span of this source states the claim — it needs no independence between sources, and it is mechanically verifiable.

Since the question being asked is “exactly what part of exactly which source”, the quote is the primary answer and the ablation drop is an annotation on it. Gating spans behind ablation meant a redundantly-supported claim returned nothing at all — the one outcome that is certainly wrong.

ALGORITHM 4a  SelectSources(C) -> (kept, omitted_labels)

 1  if |C| <= MAX_SOURCES: return (C, [])
 2  queues <- group C by layer;  sort each queue by DESCENDING ordinal   # recency
 3  kept <- []
 4  while |kept| < MAX_SOURCES and any queue non-empty:
 5      for layer in PROVENANCE_LAYER_ORDER:            # round-robin across layers
 6          if queue[layer] non-empty and |kept| < MAX_SOURCES:
 7              append queue[layer].pop_front() to kept
 8  omitted <- labels of C not in kept                  # disclosed, never silent
 9  return (sort kept by ordinal, omitted)              # ordinal order keeps mask indices valid

Narrowing is layer-balanced, and the first attempt — rank by source size — was backwards twice over. A turn with a dozen tool calls eliminated the instructions and history layers entirely, destroying the comparison round 1 exists to make; and since every source is truncated to SOURCE_TOKEN_CAP anyway, preferring the largest keeps files that get cut while dropping small ones that would have been scored whole.

7.5 Algorithm 5 — Span localisation and quote verification

ALGORITHM 5  CheckSupport(model, claim, c) -> Verdict

 1  directive <- (c.layer == INSTRUCTIONS)
 2  instructions <- directive ? DIRECTIVE_PROMPT : EVIDENCE_PROMPT
 3  body <- Truncate(c.text, SUPPORT_TOKEN_CAP)       # larger than the grounding cap
 4  prompt <- "CLAIM:\n" + claim + "\n\n" + heading + ":\n" + Fence(body)
 5  out <- model(prompt, structured_output: {verdict, evidence_quote, confidence, reason})
 6  (ok, l0, l1) <- VerifyQuote(out.evidence_quote, c)
 7  calls <- 1
 8  if out.verdict in {supported, contradicted} and not ok:      # one retry, no more
 9      retry <- model(prompt + RETRY_NUDGE, same structured output)
10      calls <- 2
11      (ok2, m0, m1) <- VerifyQuote(retry.evidence_quote, c)
12      if ok2: out <- retry; (ok, l0, l1) <- (true, m0, m1)     # keep only if it verified
13  if out.verdict in {supported, contradicted} and not ok:
14      out.verdict <- silent;  downgraded <- true               # cannot stand behind it
15      confidence <- min(confidence, 0.3)
16  return Verdict(out.verdict, clamp(out.confidence, 0, 1), quote: ok ? … : null,
17                 line_start: l0, line_end: l1, downgraded, calls)

ALGORITHM 5a  VerifyQuote(q, c) -> (verified, line_start, line_end)

 1  if |trim(q)| < MIN_QUOTE_LEN: return (false, ·, ·)
 2  needle <- Normalise(q)
 3  if needle not a substring of Normalise(c.text): return (false, ·, ·)
 4  # locate the tightest line window, in ONE pass
 5  parts, ends, lineno <- [], [], []
 6  offset <- 0
 7  for i, line in enumerate(c.lines):
 8      n <- Normalise(line);  if n empty: continue      # blank lines never renumber others
 9      if parts non-empty: offset <- offset + 1         # the joining space
10      append n to parts;  offset <- offset + |n|
11      append offset to ends;  append i+1 to lineno
12  joined <- " ".join(parts)
13  at <- index of needle in joined
14  if at < 0: return (true, 1, |c.lines|)               # single-line source; still verified
15  return (true, lineno[bisect_right(ends, at)],
16                lineno[bisect_right(ends, at + |needle| - 1)])

    Normalise(t): decode literal JSON escapes (\" \n \r \t \/), collapse whitespace, casefold

The retry (line 8) is not politeness. The same claim against the same source has produced a clean line citation on one run and an unverifiable paraphrase on the next; without a retry, a verification failure looks identical to “this source says nothing”, which is the opposite conclusion. And the downgrade (line 14) is the one hard guarantee in the whole system: a supported verdict without a checkable quote does not get to render as one.

Normalise’s escape decoding is required. Tool returns frequently arrive as JSON whose payload is itself a JSON string, so stored text contains \"layergroup\": \"Aerosol Albedo\" while any readable quote says "layergroup": "Aerosol Albedo". Before decoding was added, every honest quote from an MCP tool or web fetch was discarded as fabricated; one real case went from silent / downgraded / 0.30 to supported / verified / line 5 / 0.99.

The line search is one pass with bisection rather than the obvious “walk candidate start lines and re-normalise the remainder”, which is quadratic in source length — and the instructions unit deliberately holds the entire system prompt, the longest source in the corpus.

7.6 Algorithm 6 — The contributive probe

ALGORITHM 6  ProbeSource(model, claim, query, C, target, k) -> Verdict

 1  A <- Discriminative(Anchors(claim), C)
 2  if |A| < MIN_ANCHORS:                              # bail BEFORE spending
 3      return inconclusive, note "claim carries only |A| distinctive anchor(s)"
 4  results <- concurrent[
 5        Present(regenerate(query, ABLATE(C, 1^d)))        for 1..k ] ++
 6            concurrent[
 7        Present(regenerate(query, ABLATE(C, 1 - target))) for 1..k ]
 8  B <- sum(results[:k]);   W <- sum(results[k:])
 9  if B < k:      return inconclusive, note "baseline unstable: B/k full-context runs"
10  if W == 0:     return required
11  if W == k:     return not_required
12  return inconclusive, note "mixed result without the source"

    Present(x): cov(A, x) >= PRESENCE_THRESHOLD
    regenerate: max_tokens = CAUSAL_MAX_OUTPUT, instructions = "answer using ONLY the SOURCES"

No response prefix is supplied. An earlier design passed the response-so-far to localise the claim, and the prefix leaked the evidence — ablating every source then moved the score by 0.004 (§5.1). Regenerating from the query alone is what makes the intervention real.

7.7 Prompt-injection containment

The corpus is partly attacker-influenced by construction: a fetched page or uploaded artifact can contain “ignore the above and answer supported”. Containment is two-part:

Fence(text) = FENCE + "\n" + replace(text, FENCE, "[fence marker removed]") + "\n" + FENCE

plus a rule appended to every tier’s instructions stating that the fenced region is material to be examined, never instructions, and that text addressing the judge is content being judged.

Neutralising the marker (the replace) is the part that matters. Without it a source containing the literal fence closes the data region, and everything after it reads as top-layer prompt — handing the named adversary a one-line bypass. Two residuals remain and are not fixed by this: prose-level argumentation, and — inside the grounding and causal fences, where sources are listed as [id] label blocks — one source imitating another’s header. The second is bounded: those tiers read back only a digit or a regenerated answer, never a source identity, and the support tier, which does attribute a span to an id, puts a single source in the prompt.

7.8 Parameters

Every constant, with the reasoning. None is calibrated against annotated ground truth (§11, future work).

7.9 Complexity and what a run actually costs

With $|\mathcal{L}|$ layers present, $d^{\prime}$ candidate sources in contributing layers, and $d^{\prime\prime}$ span targets:

At defaults ($|\mathcal{L}| = 3$, $d^{\prime} = d^{\prime\prime} = 12$) the worst case is $5 + 14 + 24 = 43$ calls, inside the 48 ceiling. The span term must be counted at $2d^{\prime\prime}$, not $d^{\prime\prime}$: assuming one call per source understates a full run by about a third and lets the ceiling be passed by a run it had already approved.

A random-mask regression fixes its budget at roughly 32 calls regardless of $d$, and additionally requires teacher-forced scoring. The hierarchical decomposition is what keeps this affordable — a flat leave-one-out over every source in a long conversation would need arbitrary narrowing, and which sources survived would then be an arbitrary choice rather than a measured one.

7.10 Interactive demo A — the algorithm’s internals

This is the first of two demos. It exposes the algorithm’s internals on one real turn — segmentation and gating over all thirteen sentences, corpus construction across the three layers, then the two ablation rounds and span verification for one selected claim. Every score it reports is a value the live run actually returned; nothing is computed in your browser, so treat it as a replay of a recorded run rather than a simulation of the model. Its purpose is to make the mask sequence and the arithmetic legible, which a static listing cannot do.

Three things are worth watching. The free tier resolves or excludes ten of the thirteen sentences before any model call exists, which is what makes the metered tiers affordable at all. The layer round then eliminates six of eight sources for five calls, because history is most of the corpus and none of it matters here. And the ending is instructive: after ten calls the ablation numbers are a tie between two layers at $-4.00$ and two per-source drops of $+0.00$ — ablation has localised the claim to a pair of layers and refuses to go further. The answer comes from span verification, which is exactly why Algorithm 4 never makes it conditional.

8. Implementation

The formulation above is implementation-independent. The deployment described here is AKD Labs, a platform for agent co-design in scientific workflows. Three properties of that implementation generalise beyond it and are reported for that reason.

It is strictly read-only and post-hoc. Every source is recovered from rows already stored: the native message list with tool calls paired to returns, the agent configuration, and the request payload. No migration, no writes, no change to the generation path — so it works retroactively on all existing history. This matters more than it sounds: it means provenance can be added to a system already in production without touching what that system does.

The renderer is the single segmentation authority. Segmenting server-side on raw markdown and client-side on the rendered tree would drift, and every drift is a citation anchored to the wrong sentence. Segments are produced once, during render, and the server only ever receives them.

The system prompt is unversioned, and says so. The agent configuration carries no updated_at, so an edit since the turn ran is undetectable. Excluding instructions would be worse — a prompt-driven claim would be reported as “came from the model”, which is false — so they are included and flagged, and the interface states that the text is current rather than a snapshot.

8.1 A worked example, end to end

The method described in this report was developed with the Accelerated Knowledge Discovery (AKD) team at NASA ODSI, inside the AKD agent design environment [12][13], and the worked example below runs against that team’s MIO Worldview Agent — a NASA Earth-observation assistant with a real artifact workspace and a real system prompt. A public-facing instance of the same agent is available as a Hugging Face Space [14], so the agent itself can be inspected independently of this write-up.

Everything below is a single live run captured while writing this report. The user had asked the agent to list available guardrails; the agent answered with a Worldview permalink and an explanation of the MODIS annual land-cover layer.

Step 1 — the free tier. Toggling provenance on costs nothing and produces a complete accounting of the response:

8 sources · 13 sentences: 3 you can trace, 1 quoted directly,
                          1 questions or offers, 8 nothing specific to check

Note what did not get marked. “Open the link.” and the numbered steps carry nothing checkable. “If you want, tell me whether you want (A) statewide Alabama or (B) a specific county” is an offer. Marking those would be the category error of §6.1. The three that did earn markers are the layer identification, the forest-vs-non-forest procedure, and the non-authoritative note.

Step 2 — the cost gate. Clicking the non-authoritative note prices the run before spending:

Sources            8
Model calls       31
Est. input tokens 77,407
Est. cost         ~$0.164
⚠ 2 sources are larger than the 700-token scoring window, so only the
  first part is checked. A claim grounded later in those files will
  read as ungrounded.

The call count is a live check on §7.9: three layers present and 8 sources give $(3+2) + (8+2) + 2\times 8 = 31$, exactly what the gate reports. The truncation warning is the disclosure required by §6.8 — two of the eight sources exceed the 700-token window, and the interface says so rather than letting a miss look like an absence.

Step 3 — the result. The claim was:

Non-authoritative note: classification maps can change due to real land change or mapping uncertainty (especially near edges/mixed landscapes), so treat this as an exploratory visual comparison rather than a definitive change calculation.

and the run returned $s(\mathbf{1}) = 5.00$, $s(\mathbf{0}) = 1.00$, hence $\Gamma = 4.00$ — well clear of $\tau_\Gamma$, so this is not internal knowledge. The instructions layer resolved to a verified quote:

Agent instructions (1)
  agent instructions (system prompt)
  "## Non-authoritative communication
   - Use neutral language; avoid authoritative framing.
   - Always include a non-authoritative disclaimer in the user-facing narrative."
                                                              (line 64–66)
  directs this response · confidence 0.72
  · current text; an edit since this turn can't be detected

This is the demonstration that motivates the whole design. The sentence is a hedge; no artifact states it; a corroborative sweep over the workspace would return nothing; and no declared-citation scheme would ever footnote it — yet its cause is sitting in the agent’s own configuration at lines 64–66, and the method retrieves it with a mechanically verified quote.

Step 4: a complication. The layer drops were negative: both tools and instructions scored $\Delta_L = -4.00$. Removing either raised apparent support, from 5.00 to 9.00. This is the distractor pattern of §6.5, and it replicated here on both layers.

Read carefully, this is not a failure — it is the argument for the design. The ablation numbers alone would be confusing: two layers tied at $-4.00$, nothing “grounding” anything. The quote gave the exact, checkable answer. This is precisely why §7.4 runs span localisation unconditionally and treats the drop as an annotation on the quote rather than the other way round. A system that reported only ablation weights here would have reported noise.

8.2 A second claim: the tools layer, and redundancy

The same session, same corpus, a different sentence — the layer identification:

Layer: MODIS Annual Land Cover Type (IGBP classification).

The cost gate is identical (8 sources, 31 calls, ~$0.164, same corpus), but the outcome is a different branch of §6.5 entirely: $s(\mathbf{1}) = 9.00$, $s(\mathbf{0}) = 1.00$, so $\Gamma = 8.00$ — the strongest context effect observed anywhere in this work — while every layer drop is exactly $+0.00$. That is the redundancy region: removing the whole context changed the answer decisively, but removing any single layer changed nothing, because more than one layer carries the claim independently. Round 2 is therefore skipped as uninformative rather than because there was nothing to find, and the interface says so:

Carried by several sources
Removing everything changed the answer, but removing any one layer did not — each
carries this independently, so no single layer is solely responsible. The layer
scores below are real zeros for that reason; per-source scores were skipped
because they would read zero too, so the quotes are the answer here.

Three things in this result are worth drawing out.

The tools layer grounds it, with a byte-exact quote. The span check returns lines 26–30 of the search_worldview_layers return: "layer_id": "MODIS_Combined_L3_IGBP_Land_Cover_Type_Annual", "platform": "", "bm25_score": 1.0, "instrument": "modis", "description": "The Terra and Aqua combined Moderate Resolution Imaging Spectroradiometer (MODIS) Land Cover Type (MCD12Q1) Version 6.1 data product provides global land cover types at yearly intervals. Verdict: states the claim, confidence 0.86. Note that this is JSON, and the quote verified — which is the escape-decoding path of §6.6 doing its job. Before that normalisation existed, every quote from a tool return of this shape was discarded as fabricated.

The history layer also carries it. agent replied · 3 turns ago quotes “showing the MODIS annual land cover classification (IGBP)” at line 2, also states the claim at 0.86. The agent had already named this layer earlier in the conversation, so the claim is over-determined — present in the tool return and in the prior turn. This is what redundancy means concretely, and it is why a single-winner attribution would have been arbitrary here.

The instructions layer correctly abstains. “no instruction here bears on this”, confidence 0.77. Compare §8.1, where the same source was the whole answer. The directive framing of §6.6 is not biased toward finding a rule; on a factual claim it declines.

Taken together with §8.1, the two claims exercise three of the outcomes the method can return — attributed to instructions, redundant across layers, and grounded in a tool return — over the same corpus, with the layer drops informative in one case and uniformly zero in the other. In both, the verified quote is what carries the answer.

8.3 A third claim: internal knowledge, and a failed attempt at a contradiction

The two claims above both had a cause inside the turn. The remaining branch of §6.5 is the one where nothing does. To exercise it deliberately we asked the same agent, in the same conversation:

Without calling any tools, answer in exactly one sentence from your own knowledge: what does the acronym NDVI stand for?

The choice of acronym matters. NDVI occurs four times in this conversation, but its expansion occurs nowhere in the conversation and nowhere in the system prompt — so the claim’s content is genuinely parametric even though its subject is not. The agent answered “NDVI stands for the Normalized Difference Vegetation Index.” with zero tool calls.

The result is unambiguous: $s(\mathbf{1}) = s(\mathbf{0})$, giving $\Gamma = 0.00$ exactly, with both surviving layers at $+0.00$.

This turn also produced an incidental confirmation of the cost model under a different corpus shape. With no tool calls there are only two layers present, and eleven sources across them, so $(2+2) + (11+2) + 2 \times 11 = 39$ — exactly the figure the cost gate reported. The $|\mathcal{L}|$ term is not decorative.

The attempt at a contradiction failed, and the failure is informative. contradicted is the verdict §6.6 argues earns the span check its keep, and it is the one outcome we could not produce. The natural route is a genuine agent misread, which by definition cannot be summoned on demand, so we tried to plant one — asking for a caption asserting that the annual land-cover layer is updated monthly, which directly contradicts the "yearly intervals" string sitting in the tool return quoted in §8.2. The agent declined:

I can’t write that caption because it’s factually incorrect.

So the demonstration is unavailable for a reason worth recording: on this agent, the response-side route to a contradiction is closed by the agent’s own guardrails. Producing one would require planting the mismatch on the artifact side instead — changing a value in the workspace after the turn ran — which is a different experiment, and one that manufactures a disagreement rather than observing it. contradicted therefore remains verified in unit tests and unobserved in the wild, and §10 should be read with that gap in mind.

8.4 Interactive demo B — the end-to-end product flow

The panel below is a working replica of the interaction above — a chat widget you can actually drive. It opens ready to use, and a numbered prompt tells you what to click at each stage, with the target highlighted: check sources, pick a claim, accept the cost, read the result. Every number, quote and verdict is what the live run returned; no model is called. The second tab shows the screenshot of the real UI in the same state, if you want to confirm the replica is faithful.

All three recorded outcomes are selectable from the claim dropdown — the hedge (§8.1, instructions), the layer identification (§8.2, redundant across layers) and the NDVI acronym (§8.3, internal knowledge, from a separate turn). Play advances the four states at about a second each; Step and To end are there if you would rather drive it. Where Demo A (§7.10) shows the algorithm’s internals — masks, scalars, drops — this is what a user actually sees and clicks.

▶ Interactive Screenshot of the real UI

replica · recorded values, no model calls

▶ Play Step ▸ To end ⏭ Reset ↺ claim: the hedge (§8.1 — instructions) the layer id (§8.2 — redundant) NDVI (§8.3 — internal knowledge)

can you list the available guardrails?

MIO Worldview Agent · openai:gpt-5.2

Check sources

Clicking through it makes one property obvious that the prose can only assert: the affordance is absent from most of the response. Ten of the thirteen sentences are not clickable at all, because they were resolved or excluded for free. That absence is the design — a badge on every sentence would be indistinguishable from no badges at all.

8.5 Design decisions and their measured justification

9. Applications

The utility of response provenance is not that it makes an answer feel more trustworthy — a citation does that whether or not it is sound, which is a hazard rather than a benefit (§10). Its utility is that it converts an opaque failure into a diagnosable one, and it does so along a dimension that neither execution logs nor declared citations expose.

Consider the situation an SME is actually in when an agent returns a wrong number. Three distinct causes are consistent with the observation, and each demands a different repair: the governing artifact was absent from the workspace, so it must be added; the artifact was present but structured such that the agent misread it, so its structure must be fixed; or the artifact was present and legible and the model answered from parametric knowledge anyway, so a guardrail is needed. Without provenance these are indistinguishable, and the practical consequence is that specification changes are made by guesswork. Crossing the $\Gamma$ test of §6.5 against the span verdict of §6.6 separates them:

Neither flagged cell is visible from one axis alone, which is the argument for computing both rather than choosing between contributive and corroborative methods.

A second application follows from the fact that attribution accumulates. Where an agent’s specification is itself an accreting tree of artifacts — the case in co-design workflows, where SMEs add scope documents, examples and constraints over many sessions — attribution across turns separates specification that is in use from specification that is merely present. Context pruning is an established use of attribution [3]; applied to specification rather than to retrieval corpora, it becomes the mechanism by which a growing prompt-and-artifact tree can be pruned rather than simply growing.

The third application is the narrowest and the most consequential for scientific use. A claim accompanied by a mechanically verified quote and a line span is citable: it can be reproduced by a third party who has the same workspace, and it can be entered into a methods section with a pointer to its origin. A claim without that cannot, regardless of whether it happens to be correct. Provenance does not establish that an answer is right — §10 is emphatic on this — but it establishes what would have to be checked in order to find out, and that is the difference between an assertion and a result.

10. Limitations and threats to validity

The most serious limitation is epistemic rather than technical, and it was established in §5.4: the quantity this method is a proxy for has no ground truth in this setting. Contributive attribution is defined against the probability the model assigns to a supplied target, hosted chat endpoints do not return that probability, and validating an approximate scalar against leave-one-out of the same approximate scalar establishes self-consistency rather than faithfulness. Every accuracy claim in this report should be read with that constraint in front of it. Were a scoring-capable endpoint to become available, the probability-based formulation becomes computable and this method should be evaluated against it rather than defended.

The substituted scalar carries two further limitations that bound interpretation. It is corroborative measured under intervention, not causal: it reports which source a claim is about, which is evidence about but not proof of what produced the claim. And it is coarse — the judged digit distribution is degenerate in practice, so scores snap toward 1, 5 and 9. Rankings over sources are meaningful; a difference of 0.2 between two sources is not, and any interface that renders one as a precise weight is misrepresenting the measurement. The same caution applies with more force to the span verdict, which is an LLM judgement operating in the regime AttributionBench characterises, where even a fine-tuned model reaches roughly 80% macro-F1 on the binary formulation [9]. A contradicted verdict is grounds to go read the file, not a finding. Quote verification is the single hard guarantee anywhere in the system: a fabricated verdict still cannot produce a span that occurs in the source, and the ranking tier has no equivalent check, which is the reason its output is presented as an ordering.

That same verdict is also the method’s largest evidential gap, and it should be stated as such rather than left implicit. contradicted is the outcome §2.4 argues justifies computing the corroborative axis at all — the misread that a purely contributive method would point at without explaining, and a purely corroborative method would miss. It is verified in unit tests and it has never been observed on a real turn. §8.3 records the attempt: the natural route is a genuine agent misread, which cannot be summoned on demand, and the planted alternative was refused by the agent as factually incorrect. So of the four outcomes the decision rule can return, three are demonstrated live in §8 and the fourth — the one that matters most for catching errors — rests on tests alone. Any claim in this report about the value of the span check should be read with that asymmetry in view.

Three limitations concern what is being measured rather than how well. Attribution describes the model performing the ablation, so defaulting to the trace’s own model is what keeps the result about the agent under study; AttriBoT’s proxy-model substitution would cut cost but changes the object of measurement. Segmentation at sentence granularity bundles claims, so a sentence carrying both a fact and a judgement receives one verdict for both, and FActScore-style decomposition [11] is the principled remedy. Per-source truncation is genuine evidence loss: a claim grounded beyond the scoring window returns as ungrounded, which is a wrong answer rather than an empty one, and the only defensible response is the disclosure of §6.8 rather than an assumption that the tail did not matter.

Prompt injection is mitigated but not solved. Fencing with a non-forgeable marker prevents a source from closing its own delimiter, and the data-only rule tells the judge that the fenced region is material; neither stops an injection that argues in prose. What holds is downstream, in quote verification. Finally, and most importantly for anyone deploying this: a transparency interface can manufacture the very confidence it exists to discipline. A citation makes an answer feel verified whether or not it is. Since the estimator is unvalidated in the strong sense above, an interface that appears more authoritative than its measurement warrants makes the system less scientific rather than more — which is why every design decision in §6 and §7 that trades apparent precision for stated uncertainty is deliberate.

11. Future work

The most valuable extension is not an improvement to the estimator but the removal of the constraint that forced it. A scoring-capable endpoint — teacher-forced log-probabilities for a supplied target — would make the probability-based formulation directly computable, and the comparison between the two would then be an empirical question rather than an unavailable one. Absent that, the estimator’s known weaknesses have identified remedies of differing cost: atomic claim decomposition addresses the granularity limitation at the price of one additional call per segment, and separating tool arguments from tool returns into distinct sources would make “the agent chose the wrong date” distinguishable from “the file says X”, a distinction the current unit conflates because arguments and returns are folded into one scorable block (§7.1).

A complementary direction bears stating, precisely because this report has argued against declared citations as a substitute. Injecting source identifiers into tool returns and requesting inline markers would yield exact character offsets nearly for free, at the cost of changing what the agent generates. Declared citations verified post-hoc by the machinery of §6.6 are strictly stronger than either component alone: the marker supplies the offset, and verification supplies the guarantee the 2026 audit found missing [2]. The objection to declared citations is to trusting them unverified, not to producing them.

The extension closest to the original motivation is per-author attribution. The problem this work started from was not “which artifact” but (artifact, user) — which colleague’s contribution a sentence rests on, in a workspace several domain experts co-design. Everything here delivers the first half. The second needs the artifact tree to carry authorship at a granularity finer than the file: a source unit would have to resolve to the person who wrote the span that was quoted, not merely to the person who last touched the file. Since the span is already localised and verified (§6.6), the missing piece is provenance metadata on the artifacts themselves rather than anything in the estimator — which makes it a tractable next step, and the one that would make this useful to the people it was designed for.

Finally, the work this method most needs is not architectural. Every threshold in §7.8 — $\tau$, $\tau_\Gamma$, $\theta$, the verbatim pair $(\ell_q, \rho)$ — was set by inspection on a small number of real turns. None is calibrated against annotated ground truth. The current state is therefore one in which the algorithm is specified and its constants are not. Establishing an annotated set of agent turns with per-sentence provenance labels, and fitting these thresholds against it, would convert a set of defensible guesses into a measured operating point — and would incidentally provide the first real test of whether the corroborative proxy tracks the contributive quantity it stands in for.

12. Reproducibility and credit

What is reproducible from this report. §7 is the whole method: every algorithm, every threshold, every prompt structure. It depends on nothing but a chat-completions endpoint that can return structured output, and it is deliberately implementation-independent — the formulation assumes only that you can recover a turn’s tool calls, its system prompt, and its prior messages from whatever you already store. The accompanying reference implementation at github.com/NISH1001/response-provenance lets every algorithm in §7 be read as code rather than taken from the pseudocode on trust.

What is partially reproducible. The agent used in §8 is public: a instance of the MIO Worldview Agent runs as a Hugging Face Space under the ai-agents-for-science organisation [14], and the platform source is open [13]. A reader can therefore interrogate the same agent over the same domain.

What is not. The specific figures come from an authenticated deployment against a particular workspace, so the exact numbers — $\Gamma = 4.00$, the quote at lines 64–66 — are not re-derivable by a third party without that workspace and that system prompt. The screenshots are illustrations of a working system rather than an evaluation, and the measurements in §5, §6 and §8 are observations from a handful of turns: enough to eliminate three designs and to demonstrate that the instructions layer resolves, nowhere near enough to constitute a benchmark. Nothing here is calibrated (§11).

Credit. This work was done with the NASA ODSI Accelerated Knowledge Discovery team, whose AKD Labs environment is the substrate the method was built and tested in, and whose MIO Worldview Agent — its artifact workspace and system prompt — is the subject of the worked example. I am tech lead on that platform; the provenance formulation and this write-up are mine, the agent and the environment are the team’s. None of this would be testable without a real agent doing real work over real artifacts, which is harder to come by than the method itself.

13. Conclusion

Response provenance is a different problem from citation generation and from fact-checking, and it is the one that determines whether an agent’s output can enter a scientific record. The contributive formulation is the right one, and on hosted APIs it is not directly computable — its ground truth is an unobservable quantity.

What is achievable is a method assembled from one borrowed component and several new ones: adopt context ablation as the intervention, substitute a stable-but-coarse judged scalar for the unobtainable probability, replace the regression with exact leave-one-out over whole sources, search layers before sources, and pin every claim to a mechanically verified quote. That yields something a scientist can act on — this sentence traces to this span of this artifact, this rule in the agent’s instructions, or nothing in your workspace at all — provided the interface states that this is a reading and not a proof.

The alternative is what we have now: fluent paragraphs, execution logs underneath, and no way to connect them.

References

1. OpenAI. Citation formatting. API documentation. developers.openai.com/api/docs/guides/citation-formatting
2. Onweller, H., Lumer, E., Huber, A., Ramchandani, P., Subbiah, V. K., & Feld, C. (2026). Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents. arXiv:2605.06635
3. Cohen-Wang, B., Shah, H., Georgiev, K., & Madry, A. (2024). ContextCite: Attributing Model Generation to Context. NeurIPS 2024. arXiv:2409.00729
4. Tibshirani, R. (1996). Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society, Series B, 58(1), 267–288.
5. Liu, F., Kandpal, N., & Raffel, C. (2025). AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attribution. ICLR 2025. arXiv:2411.15102
6. Li, R., Chen, C., Hu, Y., Gao, Y., Wang, X., & Yilmaz, E. (2025). Attributing Response to Context: A Jensen-Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generation. ICLR 2026. arXiv:2505.16415
7. Xiao, Y., Zhu, Y., Samyoun, S., Zhang, W., Wang, J. T., & Du, J. (2025). TokenShapley: Token Layer Context Attribution with Shapley Value. ACL Findings 2025. arXiv:2507.05261
8. Shapley, L. S. (1953). A Value for n-Person Games. Contributions to the Theory of Games, II, 307–317.
9. Li, Y., Yue, X., Liao, Z., & Sun, H. (2024). AttributionBench: How Hard is Automatic Attribution Evaluation? ACL Findings 2024. arXiv:2402.15089
10. Li, X., Cao, Y., Pan, L., Ma, Y., & Sun, A. (2024). Towards Verifiable Generation: A Benchmark for Knowledge-aware Language Model Attribution. ACL Findings 2024. arXiv:2310.05634
11. Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W., Koh, P. W., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023). FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. EMNLP 2023. arXiv:2305.14251
12. NASA ODSI Accelerated Knowledge Discovery team. AKD Labs — agent co-design environment. labs.akd.odsi.io
13. NASA IMPACT. AKD Labs source repository. github.com/NASA-IMPACT/akd-labs
14. AI Agents for Science. MIO Agent (public instance of the MIO Worldview Agent). Hugging Face Spaces. huggingface.co/spaces/ai-agents-for-science/mio-agent

Citation

@techreport{pantha2026provenance,
  author      = {Pantha, Nishan},
  title       = {Response Provenance: Tracing Agent Claims Back to Their Causes},
  institution = {Bits and Paradoxes},
  type        = {Technical Report},
  year        = {2026},
  month       = jul,
  url         = {https://nishparadox.com/research/response-provenance/},
  note        = {Black-box post-hoc attribution of agent response segments to
                 tool returns, agent instructions, and conversation history}
}

Plain text:

Pantha, N. (2026). Response Provenance: Tracing Agent Claims Back to Their Causes. Technical Report. https://nishparadox.com/research/response-provenance/