Which LLM observability tools see what matters?

An observability product cannot tell you whether an LLM feature is healthy unless you first define what "healthy" means. Most tools can draw an attractive trace and total the tokens. Far fewer can connect a bad answer to the retrieval step that caused it, assign the waste to the customer and feature that created it, then alert on a change in task success before support tickets arrive.

That gap matters because LLM failures rarely look like ordinary software failures. The request returns 200. Latency stays under the service target. The model produces fluent text. Yet the agent chooses the wrong tool, a retriever feeds it stale policy, or a retry loop spends six dollars completing a task worth fifty cents. A green infrastructure dashboard can sit beside a broken product.

I compare LLM observability tools by the evidence they capture, not by the number of dashboard tiles they advertise. You need three separate capabilities: a causal trace for debugging, business dimensions for cost attribution, and a scored quality signal for drift alerts. No tool manufactures missing context after ingestion. Instrumentation choices decide what the dashboard can ever know.

A trace records execution, not correctness

A trace answers what ran, in what order, with which inputs, outputs, timings, and errors. It does not answer whether the outcome helped the user unless you attach an outcome or evaluation to that same trace.

This distinction gets blurred because LLM tracing interfaces expose prompts and responses, which feel richer than ordinary spans. Richer evidence is still evidence, not judgment. A trace can show that the agent called search_orders, received three records, and generated a polite reply. It cannot know that the user asked about a refund, that the agent should have called refund_status, or that the reply contradicted company policy.

There are three useful levels of trace depth. A gateway trace sees the exchange with the model provider: model, tokens, latency, response, and often cost. A framework trace adds agent nodes, tool calls, retrieval, and relationships between parent and child operations. A complete application trace connects those spans to the API request, database query, queue job, browser action, and downstream failure. Each level answers a different incident question.

Gateway visibility is enough when the feature is a single prompt followed by a response. It becomes misleading when an agent retries, calls paid tools, retrieves documents, or delegates work. A model call that costs $0.10 may belong to a workflow that costs $2. Worse, the failed first attempt may disappear from the final response while still appearing on the invoice.

Framework visibility makes the decision path legible, but only for code the instrumentation touches. A custom function that silently truncates context will remain invisible unless you wrap it in a span and record the relevant attributes. Complete application visibility catches the surrounding system, but generic application performance monitoring rarely knows that a retrieved passage was irrelevant or a groundedness score fell.

Do not ask whether a product has tracing. Ask where its root span begins, which child operations it creates automatically, how context crosses queues and services, and which payloads it drops or redacts. The missing span is usually where the expensive mystery lives.

Trace depth decides which failure you can explain

The best tracing tool is the one that covers your actual execution boundary with the least custom glue. Tool names matter less than whether your trace preserves causality across model calls, tools, retrieval, services, and asynchronous work.

LangSmith models an operation as a trace made of runs and groups traces into projects and threads. Its integrations are especially natural for LangChain and LangGraph, while wrappers and manual instrumentation cover other code. It is strong when an engineer needs to inspect an agent path, compare runs, attach feedback, and move a production example into an evaluation dataset. OpenTelemetry ingestion reduces dependence on one framework, but the richest automatic structure still follows the integrations you use.

Langfuse records traces, spans, generations, events, sessions, scores, model usage, and prompt versions. Its documentation says each observation can include timing, input, output, and cost information. That data model fits teams that want one place for production tracing, prompt management, evaluation scores, and experiments. Open source deployment and current OpenTelemetry based SDKs give you more control over storage and export. You still have to propagate trace context through your custom jobs and label business dimensions yourself.

Phoenix accepts OpenTelemetry traces and uses OpenInference conventions for model calls, retrieval, tool use, and other AI operations. That makes Phoenix a sensible choice when you want an open instrumentation layer and local or self hosted analysis. It is particularly good for investigating retrieval and attaching evaluations to spans. Phoenix shows only the operations your instrumentors or manual spans emit, so a clean trace tree does not prove that every decision entered the tree.

Helicone can sit in the provider request path as a gateway or receive data through an SDK. That placement gives fast visibility into model requests, costs, latency, errors, sessions, and custom properties without wrapping every application function. It is a good fit when provider traffic and spend are the first problem. The gateway cannot infer an internal router decision, an unlogged tool result, or the business outcome of a workflow. Add application spans if those questions matter.

Datadog connects LLM spans with application performance data, logs, infrastructure, and service traces. It has LLM cost calculation and trace level evaluations, but its structural advantage is correlation: an agent timeout, database saturation, queue delay, and model call can share an operational view. Teams already running Datadog may get faster incident coverage by extending it than by introducing a separate island. Dedicated LLM products often offer a tighter prompt, dataset, and annotation workflow.

This boundary map is a starting point, not a permanent feature verdict. LangSmith is strongest around agent runs and evaluation. Langfuse keeps an open LLM engineering record that joins generations, prompt versions, scores, and cost dimensions. Phoenix favors open telemetry, retrieval analysis, and experiments. Helicone owns the provider boundary. Datadog connects the AI call to the wider production system. Product capabilities change, and your architecture changes too. Validate the path with your own trace before signing a long contract.

Streaming deserves a specific test. Some integrations record time to first token, total generation time, and cancellation correctly; others finish the span only after a stream closes normally. If a user abandons a slow response, confirm that the trace records the cancellation, emitted tokens, billed usage, and final outcome. Otherwise latency percentiles describe completed requests while the worst user experiences vanish.

Also inspect nested agents and remote tools. A trace that stops at an MCP request or an agent handoff tells you that delegation happened, but not what the delegate did. You need shared context across that boundary and a policy for which arguments and results can enter telemetry. A vendor may support both technologies while leaving context propagation to your code.

Cost attribution starts with your dimensions

No LLM observability tool can allocate spend to a customer, feature, or outcome if the application sends only a model name and token count. Accurate price calculation and useful cost attribution are separate jobs.

Most specialist tools calculate model cost from provider usage and a pricing table. LangSmith automatically records token usage and costs for major providers and accepts custom cost data for model and nonmodel runs. Langfuse can ingest usage and cost directly or infer them from a model definition; its documentation correctly prefers provider supplied usage when available. Phoenix tracks token, latency, and model cost when the required attributes and pricing are present. Helicone calculates cost at the gateway and can break traffic down with custom properties. Datadog also calculates supported provider costs and marks traces as partially costed when it lacks prices for some LLM spans.

Those totals can still be wrong for management decisions. Provider tables change. Cached input, reasoning tokens, image tokens, regional pricing, batch discounts, fine tuned models, and negotiated contracts complicate inference. Tool calls, search APIs, vector databases, sandbox minutes, and human review may cost more than the model. If a product reports only model cost, call it model cost rather than task cost.

Attribution requires stable dimensions at ingestion time. I use tenant.id, feature.name, environment, release, prompt.version, model, workflow.name, and a terminal outcome. For an agent that runs many steps, I also record attempt count and whether a human intervened. User identifiers should be pseudonymous when operators do not need the identity. Prompts and retrieved documents need explicit redaction rules, not a vague promise that engineers will avoid sensitive data.

The following payload is the minimum useful contract I expect on a completed workflow. The names can change, but the relationships should not:

{
  "trace_id": "01J...",
  "tenant.id": "tenant_42",
  "feature.name": "invoice_reconciliation",
  "workflow.name": "match_and_explain",
  "release": "2026.08.3",
  "prompt.version": "invoice-v17",
  "model": "provider/model-version",
  "attempt.count": 2,
  "usage.input_tokens": 18420,
  "usage.output_tokens": 912,
  "cost.model_usd": 0.184,
  "cost.tools_usd": 0.031,
  "outcome": "human_corrected",
  "quality.task_success": 0
}

With that contract, finance can ask cost per tenant, product can compare prompt versions, and engineering can isolate retries. Without it, the observability product gives you a provider bill with prettier filters. Tag at the root and propagate the context to every child span. Reconstructing business ownership from prompt text later is slow, brittle, and risky.

Cost per request is often the wrong denominator. An agent that costs less per run can cost more per resolved task if it retries, escalates to a person, or produces work that needs repair. Record both the requested action and the terminal state, then calculate model cost, tool cost, and human correction cost by successful outcome. The observability platform can aggregate those fields, but product and finance must agree on their meaning.

Reconcile a sample against provider invoices before trusting the chart. Pick requests that use caching, reasoning tokens, images, retries, and more than one provider. Check the exact model identifier and the effective date of the price entry. When calculated and billed amounts differ, preserve both values rather than silently overwriting one. The difference tells you whether the problem sits in instrumentation, pricing data, or contract terms.

Quality drift needs a scored production signal

A drift alert is useful only when it watches a metric tied to user success and compares like with like. Token, latency, and error alerts detect operational changes. They do not detect a model that becomes more confident and less correct.

The word "drift" covers several different events. Input drift means the mix of requests changed, perhaps because a new customer sends longer documents. Retrieval drift means the corpus, ranking, or chunk distribution changed. Behavior drift means the same workflow now chooses different tools or produces a different format. Quality drift means a stable cohort scores worse on a task measure. Cost drift means spend per successful outcome rises. An alert that says "average tokens increased" cannot tell these apart.

LangSmith can run online evaluators against production traces, store feedback, chart qualitative metrics, and alert through channels such as webhooks or PagerDuty. Langfuse stores numeric, categorical, boolean, and text scores from users, code, humans, or model judges. Its monitors can alert on observations or aggregate score thresholds, including the share of boolean passes. Phoenix supports code, human, and model based evaluations attached to traces and spans; Phoenix itself is strongest as the evidence and evaluation workbench, while managed monitoring depth varies between Phoenix and the broader Arize offering. Helicone alerts cover operational measures such as error rate, cost, latency, tokens, and request count. If you need semantic quality drift there, feed a quality score into a separate monitor or confirm the current evaluation alert path in your edition. Datadog offers evaluations and monitors in the same broader operations system, which helps when quality and service health must page the same team.

Model judges are useful but they are not ground truth. They can change when the judge model changes, reward verbosity, miss domain rules, and leak cost into every sampled trace. Pin the judge model and prompt version. Keep a reviewed calibration set. Track judge disagreement with human labels. Use deterministic checks for facts a program can decide, such as JSON validity, forbidden fields, citation presence, tool choice, and arithmetic.

Alert on cohorts rather than a global average. A global score can stay flat while one enterprise workflow collapses and a high volume consumer workflow improves slightly. Segment by feature, prompt version, model, language, customer tier, and release, but avoid tiny groups that page on noise. Require a minimum sample count and use a sustained window. Send the alert to a trace sample that shows the failed score, not merely to a dashboard.

Quality alerts also need a baseline policy. A fixed threshold works for a contractual measure such as format validity. A relative comparison works for an established feature with seasonal traffic. A release comparison works when a prompt or model changes. Decide which question you mean before configuring the monitor.

Treat evaluator changes as production releases. If you edit a judge prompt or replace its model, old and new scores may no longer share a scale. Run both versions on the calibration set, store the evaluator version on every score, and avoid stitching them into one trend line until you understand the difference. A sudden quality improvement after a judge change is usually a measurement event, not a product victory.

Delayed outcomes need a second pass. A support answer may look grounded now but prove wrong when the customer reopens the case tomorrow. An extraction may pass schema validation but fail when accounting rejects the record. Update or append the outcome when that evidence arrives, and keep the trace identifier with the business record. Immediate model based scores and later operational outcomes answer different questions.

Every platform misses context you chose not to record

The largest blind spots come from instrumentation and governance decisions, not missing dashboard features. Teams routinely blame the product after sending incomplete, sampled, or unsafe data.

Sampling is the first trap. Ordinary distributed tracing often keeps errors and drops routine successes. LLM quality analysis needs a representative slice of both because a bad answer usually has no exception. Cost attribution needs totals or a mathematically sound estimator. If you sample after an expensive retry, you may retain the final success and discard the waste that caused it. Head sampling also decides before the outcome exists.

Payload capture creates the opposite problem. Recording full prompts, retrieved passages, tool arguments, and outputs makes debugging much easier. It can also copy personal data, secrets, source code, or customer documents into another system with different access and retention rules. Redact before export where possible. Separate metadata retention from payload retention. Test whether deletion by user or tenant reaches traces, annotations, datasets, and exports.

Async context is another common failure. An API request enqueues a job, the worker starts a new root trace, and the final model call looks unrelated to the user action. Cost splits across two traces and the latency chart excludes queue time. Pass W3C trace context or an explicit correlation identifier through the message. Confirm in the UI that the chain survives, rather than trusting an SDK configuration flag.

Evaluation coverage can lie too. If an online judge scores only short English answers, the dashboard says quality is stable while long multilingual sessions remain unmeasured. Track evaluation coverage as its own metric: eligible traces, sampled traces, completed scores, judge errors, and score latency. A quality chart without its denominator is decoration.

Finally, none of these products automatically knows task economics. It may show that a workflow cost $1.40 and scored 0.82. Only your application knows whether it reconciled a $20 invoice, prevented an hour of support work, or created a case that a person had to repair. Record the outcome close to the business transaction, then join it to the trace.

The cheapest tool can produce the most expensive telemetry

Observability cost depends on event volume, payload size, retention, evaluations, and engineering effort, so a free tier tells you almost nothing about production economics. Instrumentation granularity itself creates billable data.

A single chat turn might create one trace, several framework spans, two generations, five tool spans, a session record, and multiple scores. Some hosted products bill by traces or spans, some by events or ingested units, and broader monitoring suites may charge for indexed logs, hosts, or data volume. Langfuse documentation, for example, defines an ingested unit as a trace, observation, or score. An online evaluation can add a score and another model call. The feature that improves visibility also increases both telemetry and inference spend.

Retention has two prices. The vendor charges to store and query data, and your team accepts the privacy exposure of keeping payloads. Keep searchable metadata longer than raw prompts when incident patterns do not require the text. Preserve selected failed traces and evaluated examples for regression datasets. Do not retain every customer document forever because storage looked cheap during a pilot.

Self hosting changes the invoice rather than removing it. Langfuse and Phoenix can give teams more control, but somebody still runs databases, object storage, upgrades, backups, access control, and capacity planning. For a small team, a managed service may cost less than the interruption load. For regulated data or high event volume, operating the stack may be justified. Price both the platform and the owner.

Generic application monitoring can be economical if you already pay for it and need correlation with production services. A dedicated LLM product can save engineer time during prompt and evaluation work. Running both is reasonable when each has a clear job: use OpenTelemetry as the shared transport, route selected AI spans to the specialist, and keep service health in the main operations system. Duplicating every payload into two long retention stores is rarely necessary.

Estimate with a representative trace, not a marketing request count. Multiply spans, observations, scores, payload bytes, and evaluation calls by real traffic. Add retries and long agent sessions. Then run the same query your team will use during an incident and measure how much custom work it takes to answer.

A candidate test should include a deliberately bad release

You can choose an LLM observability tool in a week if you test evidence rather than feature checkboxes. Send the same instrumented failure through each serious candidate and ask operators to diagnose it without help from the person who built the workflow.

Use one production shaped workflow with retrieval, a tool call, a retry, an asynchronous boundary, and a known business outcome. Include sensitive fields that the exporter must redact. Create two tenants and two prompt versions so attribution and comparison are real. Keep the traffic small, but preserve the structure that usually breaks context propagation.

1. Run a healthy baseline and confirm the trace contains every expected parent and child span.
2. Release a retriever change that returns a plausible but outdated document, then verify that a quality evaluator fails while HTTP and latency stay healthy.
3. Trigger one model retry and one paid tool retry, then reconcile tool and model cost against raw provider records.
4. Send the workflow through a queue and confirm tenant, release, prompt version, and trace context reach the worker.
5. Ask an engineer unfamiliar with the setup to find the regression, affected cohort, wasted spend, and source payload under a time limit.

Score the candidates on answers, not UI polish. Can the engineer move from alert to cohort to failed trace to responsible span? Can finance group cost by tenant and successful outcome? Can a privacy owner find and delete a payload? Can you export traces and scores in a usable format? What becomes invisible when sampling begins?

Also test failure in the observability system. Block its exporter, exceed a payload limit, send an unknown model name, and break the evaluator. Your application should keep serving according to an explicit policy, and telemetry loss should create its own alert. Silent loss is dangerous because the dashboard looks calm precisely when it has stopped receiving evidence.

The winner may be two products with a narrow boundary. Helicone plus application spans can cover gateway economics and internal logic. Phoenix or Langfuse can hold open traces and evaluations while Datadog owns infrastructure incidents. LangSmith can be the shortest path for a LangGraph heavy team that wants tracing and evaluation in one workflow. Architecture fit beats a universal ranking.

Ownership matters more than another dashboard

An observability deployment works only when one person owns the trace contract, cost definitions, evaluator calibration, alert routing, retention, and telemetry loss. Buying software does not assign that responsibility.

Engineering should own instrumentation and trace continuity. Product should define task success and important cohorts. Finance should approve cost definitions. Security or privacy should set payload and retention rules. One accountable technical owner must resolve the gaps between them, because otherwise each dashboard can be locally correct and the overall answer wrong.

Set a small operating contract. Every production LLM feature needs a versioned trace schema, a cost owner, at least one outcome metric, an evaluation coverage measure, and an alert destination. A release that changes a model, prompt, retriever, tool, or judge should carry a version attribute. Reviews should reject an untraceable asynchronous boundary just as they reject missing error handling.

Write down which system holds the authoritative copy of each fact. Provider usage may own billed tokens, the trace may own execution context, the product database may own the final outcome, and the evaluation store may own reviewed labels. Your dashboard can join them, but it should not quietly redefine them. This prevents two teams from reporting different cost per success while both believe the observability tool supports their number.

Do not page on every interesting metric. Page when a person can act now: a cost surge, a sustained task failure, an exporter outage, or a safety check regression. Send slower changes to a weekly review where product and engineering can inspect cohorts and add failed examples to regression datasets. Alert fatigue will teach the team to ignore the one signal you needed.

In the Team & AI Audit I run through oleg.is, I look for this ownership gap because expensive AI work often hides in retries, duplicated review, and tools nobody trusts enough to use. The observability choice should reduce that uncertainty, not create another system that needs a committee to interpret.

Choose the visibility boundary first. If model traffic and spend dominate, start at the gateway. If agent decisions and evaluation dominate, start with an LLM engineering platform. If incidents cross databases, queues, and models, extend the operations platform or connect both with OpenTelemetry. Whichever route you take, require one demonstrated chain from business outcome to trace, cost, score, and alert. If a candidate cannot produce that chain on your broken release, its feature list does not matter.