LLM Observability in 2026: Tracing, Evals & the OTel GenAI Spec

Most LLM observability roundups are feature lists that quietly get three things wrong: they call source-available and proprietary tools open source, they treat the OpenTelemetry GenAI conventions as a frozen standard, and — as of July 2026 — they still recommend Helicone without mentioning that it has been in maintenance mode since its March 2026 acquisition. This guide fixes all three, gives you the actual span and attribute names to instrument, and reframes the buying decision around observed cost per completed task rather than per-token pricing.

What LLM observability actually means

LLM observability is three things bolted together, and vendors blur them constantly:

• Tracing — a hierarchical span tree recording every LLM call, tool call and retrieval step in a request, with the inputs and outputs attached.
• Evaluation — scoring those outputs, via LLM-as-judge, code and heuristic checks, or human labeling.
• Monitoring — cost, latency and quality over time, with dashboards and alerts.

The reason you cannot just point Datadog APM at it and go home is that the primitives are different. Langfuse's own documentation makes the point plainly: these platforms are purpose-built to understand LLM-specific concepts — tokens, model parameters, prompt and completion pairs — rather than HTTP status codes and DB query times. A 200 OK that returned a confidently wrong answer at 4x the expected token count is, to a normal APM, a perfectly healthy request.

That gap gets worse the moment you move from single calls to agents. A one-shot chat completion is easy to log. An eleven-step agent loop that calls a search tool twice, retries after a malformed tool argument, and silently pads its context to 90k tokens is a debugging problem you cannot solve from flat logs. If you are building in that territory, read this alongside our AI agent harness guide — the harness decides what happens, the trace tells you what happened.

What to instrument: the trace tree

The single most useful mental model is the agent trace tree described by the OpenTelemetry GenAI conventions: a top-level invoke_agent span, with chat spans for each LLM call and execute_tool spans for each tool invocation nested beneath it. One user task, one trace.

Notice where the money and the mystery live: in the second and third chat spans, where input tokens climbed from 1,240 to 11,050 because every tool result got appended to the context. That is the single most common cost surprise in agent systems, and it is invisible without nested tracing. We break the arithmetic down in how to estimate LLM API costs.

The minimum viable instrumentation set, in priority order:

1. One span per LLM call, with request and response model, input and output token counts, and finish reason.
2. Provider name — and note the rename below, because this one silently breaks dashboards.
3. Tool call spans with the tool name and an error status when the call fails or returns malformed arguments.
4. A stable session or conversation identifier, so multi-turn behaviour is reconstructable.
5. Prompt version or template identifier, so you can attribute a quality change to a specific edit.

The OpenTelemetry GenAI conventions

This is the part most articles get wrong. The OpenTelemetry GenAI semantic conventions are the emerging industry standard, and they are not stable. The official status badge on the GenAI span documentation reads Status: Development as of mid-2026. The conventions have also moved out of the main semantic-conventions repository into a dedicated one, open-telemetry/semantic-conventions-genai.

The core span attributes, for a span of kind CLIENT:

The metrics convention defines a dozen instruments — client, server, workflow, agent and tool timings — but two carry almost all the early value: gen_ai.client.operation.duration for latency and gen_ai.client.token.usage for token consumption. Together they give you per-request cost estimation, a way to catch token-hungry prompts before production, and latency regression detection, which is most of what a first observability rollout needs to deliver. Everything else on that list (gen_ai.invoke_agent.tool_calls, gen_ai.execute_tool.duration and friends) is worth adding once the basics are landing cleanly.

The rename is the practical landmine. If you built dashboards against gen_ai.system and later upgraded instrumentation, your provider breakdown quietly empties out rather than erroring. Instrumentation libraries manage the transition through the OTEL_SEMCONV_STABILITY_OPT_IN environment variable. During the changeover you may see both attributes on the same span, so filter on gen_ai.provider.name where it exists and fall back to gen_ai.system for older data. Pin your instrumentation versions, and re-check every saved query after an upgrade. Ecosystem support from general APM vendors is arriving but partial as of mid-2026 — treat any specific vendor version number you read as something to confirm against that vendor's own changelog before you depend on it.

The tools, license-accurate

Here is the matrix competitors keep getting wrong. Licenses are stated explicitly and taken from each project's own LICENSE file or docs; nothing here is a feature we could not confirm from the tool's own docs or GitHub repository. Where we could not verify a vendor's adoption or feature-count claim, we left it out rather than repeat it.

Four clarifications the roundups skip:

Helicone. Its acquisition by Mintlify was announced on 3 March 2026, with founders Justin Torre and Cole Gottdank joining Mintlify. Per both companies' announcements the service stays live in maintenance mode — security patches, new-model support and bug fixes continue, but there is no new feature roadmap. Its GitHub release history stops at v2025.08.21-1 on 21 August 2025. That is a perfectly reasonable thing to keep running if you already depend on the proxy for caching, rate limiting and key management. It is a poor thing to newly standardise on for the next two years.

Phoenix is not Apache-2.0. This is the single most repeated error in observability roundups. Arize's own self-hosting documentation states that Phoenix is released under the Elastic License 2.0. ELv2 is source-available, not OSI-approved: you can self-host it on your own infrastructure for free with no feature gates, but you cannot offer it to third parties as a hosted or managed service. For the overwhelming majority of teams that distinction changes nothing — but if you are an agency or a platform planning to resell a hosted Phoenix, it changes everything, and it is worth reading the license before you build.

Phoenix vs Arize AX. Phoenix is the free, self-hostable, ELv2 project; Arize AX is the separate commercial platform. Any article that lists them as one product has not read either set of docs.

LangSmith is not open source. It is genuinely framework-agnostic — you do not need LangChain to use it — but it is proprietary, and self-hosting is listed only under Enterprise. Verified on LangChain's pricing page in July 2026: Developer is free up to 5,000 base traces per month then pay-as-you-go; Plus is $39 per seat per month up to 10,000 base traces; Enterprise is custom with self-hosted and hybrid options. Base traces are retained 14 days, with 180-day extended retention for a fee. Billing runs through LCUs at $1.50 and LSUs at $1.00. Pricing pages move — date-stamp anything you put in a budget.

Self-host vs SaaS

The decision rule we use, in order:

1. Do prompts and completions contain data you cannot send to a third party? If yes, you are self-hosting, and your shortlist is Phoenix, Opik or Langfuse — all three are free to run on your own infrastructure, with the caveat that Phoenix's ELv2 licence bars you from reselling it as a managed service. LangSmith self-host exists but only at Enterprise pricing.
2. Do you have someone who will own a Postgres and an object store at 2am? If no, take the SaaS tier of the same tool. All four self-hostable options also sell hosted versions, so this is a deployment choice, not a lock-in choice.
3. Is your instrumentation OTel-native? If you want the freedom to switch backends later without re-instrumenting, emit standard GenAI spans — Phoenix is built directly on OpenTelemetry with OpenInference instrumentation, and Langfuse ingests OTel too. That is the cheapest insurance against a vendor going into maintenance mode or changing hands, which, after two acquisitions in this category inside three months, is not a hypothetical.

A cheap starting move: run Phoenix locally for a week. It needs no API key, has no feature gating, and will tell you within a day whether your token distribution looks anything like what you assumed.

How evals catch regressions

Tracing tells you what happened. Evals tell you whether it was any good. The pattern that all four platforms' first-party documentation converges on:

• Offline, pre-deploy. Freeze a dataset of representative inputs with expected properties. Run it against the candidate prompt or model. Compare scores to the current production baseline. This is where a prompt edit that helps your three favourite examples and hurts forty others gets caught.
• Online, post-deploy. Sample live production traces and score them with LLM-as-judge or cheap heuristic checks — did it cite a source, did it stay in JSON, did it refuse when it should have. Alert on drift in the score distribution, not on individual failures.
• Alerting on the boring signals too. A jump in finish_reasons=length, a jump in mean input tokens, a new error rate on one tool. These often precede the quality drop by a day.

Evals are also the honest way to measure whether an anti-hallucination intervention worked — see how to reduce LLM hallucinations, where the grounding techniques only pay off if you can score before and after on the same dataset.

Cost per completed task

Here is the reframe that makes observability pay for itself, and it is the number almost nobody reports: cost per completed task, not cost per token or per call.

Per-token pricing tells you nothing about an agent, because the interesting variable is how many times it loops. Two models with a 3x gap in list price can land far closer than that on cost per resolved ticket, if the cheaper one retries more, pads more context, or fails a tool call and starts over. Your trace tree already has every ingredient for the real number: sum gen_ai.usage.input_tokens and gen_ai.usage.output_tokens across every span in a trace, price them against gen_ai.response.model, and divide by the count of traces that reached a successful terminal state.

Compute that per model and per prompt version and three things usually fall out at once: one prompt is silently several times the cost of its peers, one tool fails often enough that its retries dominate the bill, and your model choice was optimised against the wrong metric. Once you have the number, the levers are in how to cut LLM API costs, and for coding agents specifically, cutting token costs on coding agents. If you are still selecting the model itself, the best LLMs for AI agents in 2026 is the companion piece — but measure the loop before you trust any leaderboard.

Do the instrumentation first. Every optimisation after it is guesswork otherwise.

FAQ