AI workflow observability helps teams explain failures instead of guessing at them. This guide shows what to log, how to trace multi-step LLM workflows, how to estimate token and infrastructure costs, and how to connect operational data with answer-quality signals. Use the formulas, field template, and review checklist as a baseline for prompt deployments, incident reviews, and model changes.
Overview
An AI application can appear to work while gradually becoming more expensive, slower, or less reliable. A retrieval step may return irrelevant documents, a prompt may grow after a template change, or a fallback model may handle more traffic than expected. Without structured telemetry, these problems are difficult to separate.
AI workflow observability combines three related practices:
- Logs record what happened in a request, including inputs, outputs, errors, model settings, and timing.
- Traces show the sequence of operations in a workflow, such as classification, retrieval, tool use, generation, validation, and fallback.
- Evaluation signals indicate whether the result was useful, safe, complete, and consistent with the task.
These signals should be reviewed together. A rise in latency might come from retrieval, a tool timeout, or a larger completion. A rise in cost might reflect token growth, retries, or a model-routing change. A fall in user ratings might be caused by answer quality, missing context, or a broken downstream action.
Operational metrics tell you whether the system is running. Quality metrics help establish whether it is doing the right thing. Monitoring only one category leaves important failure modes hidden.
How to estimate observability metrics
Start by defining one measurement unit. For most applications, the unit is a workflow run, such as one customer question, document-generation job, or support-ticket classification. Assign a unique trace_id to that run and a unique span_id to each step.
Estimate request cost
Use the pricing inputs supplied by your model provider or internal chargeback system. Keep the inputs configurable rather than embedding them in application code.
input_cost = input_tokens / billing_unit × input_rate
output_cost = output_tokens / billing_unit × output_rate
request_cost = input_cost + output_cost + tool_costs + infrastructure_costs
If a workflow can retry, include retries in the estimate:
expected_cost_per_run = base_cost × average_attempts + average_tool_costs
For a planning estimate, multiply by expected volume:
period_cost = expected_runs × expected_cost_per_run
Use actual usage when it becomes available. Planning assumptions are useful for capacity decisions, but they should not be presented as an invoice forecast unless the inputs and exclusions are clear.
Estimate latency and failure rates
For a sequential workflow, total latency is approximately the sum of each span’s duration plus orchestration overhead:
workflow_latency = span_1 + span_2 + ... + span_n + overhead
Parallel steps change the calculation. If retrieval and metadata lookup run concurrently, the combined duration is closer to the slower branch rather than the sum of both branches. Record both span duration and parent trace duration so this distinction remains visible.
Calculate failure rate using a consistent denominator:
failure_rate = failed_runs / total_runs × 100
Define what counts as a failure. A provider error, timeout, malformed JSON response, failed tool call, unsafe output, and low-quality answer may need separate categories. Combining them into one number can hide the best remediation.
Estimate quality signals
Quality usually requires a combination of automated checks and sampled human review. Useful signals include:
- Schema-valid output rate.
- Grounding or citation-support rate for retrieval workflows.
- Task completion rate.
- Refusal or escalation rate.
- User correction, regeneration, or negative-feedback rate.
- Evaluation score on a fixed test set.
Do not treat a single score as a complete measure of quality. A response can be fluent and schema-valid while still answering the wrong question. For a broader testing process, use the LLM Evaluation Checklist and compare quality signals before and after prompt or model changes.
Inputs and assumptions
A useful telemetry plan begins with a written input contract. The following fields give an AI development team enough context to investigate most cost, latency, and quality changes without storing every sensitive payload.
Recommended log-field template
| Category | Example fields | Why it matters |
|---|---|---|
| Identity | trace_id, span_id, workflow_name, environment, tenant_id | Connects steps and separates production from testing. |
| Versioning | prompt_version, model_identifier, application_version, retriever_version | Links changes to observed outcomes. |
| Usage | input_tokens, output_tokens, cached_tokens if available, tool_calls | Supports LLM cost tracking and token-growth analysis. |
| Timing | started_at, duration_ms, queue_ms, provider_ms | Shows where latency is introduced. |
| Outcome | status, error_type, retry_count, fallback_used | Distinguishes transient errors from workflow defects. |
| Quality | schema_valid, evaluator_version, quality_label, user_feedback | Connects runtime behavior with answer quality. |
For privacy and security, log references, hashes, classifications, or redacted excerpts where possible instead of unrestricted prompts and completions. Establish retention periods, access controls, and redaction rules before sending data to a third-party observability system. Never assume that a log destination is appropriate for secrets, credentials, personal data, or confidential source documents.
Trace map for a multi-step workflow
A typical retrieval-augmented workflow can be represented as:
request → authentication → input normalization → query rewrite → retrieval → reranking → prompt assembly → generation → validation → response
Each arrow should have a trace span when the step can fail, add meaningful latency, consume tokens, or affect quality. Record counts and decisions as well as durations. For example, retrieval telemetry might include the number of candidates, selected documents, filter conditions, and a quality check for context relevance. Generation telemetry might include finish status, token counts, structured-output validity, and whether a retry occurred.
For more detail on documenting inputs, prompts, outputs, and ownership, see the AI Workflow Documentation Template. If the workflow handles untrusted retrieved content, pair observability with the Prompt Injection Prevention Guide.
Worked examples
Example 1: Monthly cost estimate
Assume a workflow is expected to process 20,000 runs during a planning period. A test sample shows an average of 2,400 input tokens and 700 output tokens per run. The model’s current input and output rates are treated as editable variables, represented here as R_in and R_out per billing unit.
The estimated model cost is:
20,000 × ((2,400 / billing_unit × R_in) + (700 / billing_unit × R_out))
Next, review the trace data. If 8% of runs retry once, the expected number of model attempts is approximately:
20,000 × (1 + 0.08) = 21,600 attempts
Apply the same attempt factor to the relevant token and tool estimates. Do not apply it to fixed costs automatically; hosting, storage, and monitoring may follow different billing rules.
Example 2: Finding a quality regression
Suppose an evaluation set shows that schema-valid output is stable, but grounded-answer checks decline after a prompt deployment. The trace comparison shows that average retrieved-document count is unchanged, while the prompt assembly span has grown substantially.
This pattern suggests a focused investigation: compare prompt versions, inspect truncation behavior, check whether instructions are being placed after retrieved content, and review whether irrelevant documents are entering the context. It does not justify concluding that the model itself is worse. Versioned spans and evaluation labels make the next test measurable.
Example 3: Building an alert worksheet
Create a worksheet with one row per signal:
| Signal | Baseline | Review trigger | First investigation |
|---|---|---|---|
| Error rate | Established from recent stable runs | Meaningful deviation from baseline | Group by provider, workflow, and error type. |
| Cost per run | Median and high-percentile sample | Sustained increase | Compare tokens, retries, routing, and tool calls. |
| Latency | Parent and span percentiles | Workflow or critical-span increase | Locate the slowest span and queue time. |
| Quality score | Fixed evaluation-set result | Regression beyond review tolerance | Compare prompt, model, retrieval, and output changes. |
Use thresholds as review triggers, not automatic proof of an incident. Traffic mix, evaluation-set composition, and seasonal workload changes can affect the numbers.
When to recalculate
Recalculate your AI workflow baseline whenever an input to the estimate changes. At minimum, revisit it after:
- A provider changes pricing, token accounting, limits, or model behavior.
- A prompt, system instruction, retrieval strategy, or output schema changes.
- A model, embedding model, reranker, tool, or fallback route changes.
- Traffic volume, request length, document size, or user mix changes.
- Retry, timeout, caching, batching, or concurrency settings change.
- A quality benchmark, human-review rubric, or evaluation dataset changes.
- An incident reveals an unlogged span, missing version field, or misleading metric.
Set a recurring review as well as event-based reviews. A monthly cost audit can catch token drift, while a release review can compare prompt and model versions before rollout. Keep the calculation inputs, assumptions, and date of review beside the results so another engineer can reproduce them.
To put this into practice, begin with one critical workflow. Add a trace ID, version fields, token counts, span timing, retry status, and one quality signal. Establish a baseline from representative runs, then add alerts only after the measurements are stable enough to interpret. Finally, connect every alert to an owner and a runbook. Observability is most valuable when it shortens the path from an unusual signal to a concrete engineering decision.