---
title: 'AI agent observability: what to log on every tool call'
description: >-
  We read the log of a six-hour session with a coding agent: 886 tool calls.
  What it recorded, what we had to rebuild, and the record we now want for every
  call.
author: elroot_labs
date: '2026-11-18'
categories:
  - artificial-intelligence
---
AI agent observability comes down to one record per tool call that answers seven questions: which agent and model made the call, which tool it called with which arguments, which task the call served, whether it was allowed and by whom, what came back, how long it took, and what it cost. We checked those questions against a real log: one six-and-a-half-hour working session with a coding agent on this website's repository, 886 tool calls in all. The log answered four of the seven well. We had to reconstruct timing from timestamps, could not attribute cost to individual calls, and found no link from a call to the task it served. Below is what the log contained, what we had to rebuild, and the record we now want for every call.

## What one long agent session looks like in its log

The session ran from the evening of 13 November 2026 into the early hours of the next day, and its log is a file of JSON lines, one record per event. Counting the records gave us this picture:

| Measure                                              | Count                     |
| ---------------------------------------------------- | ------------------------- |
| Model responses                                      | 1,003, all from one model |
| Tool calls                                           | 886                       |
| Shell commands                                       | 305                       |
| File edits and writes                                | 155                       |
| Browser actions (batched)                            | 83                        |
| Web page fetches                                     | 70                        |
| Results flagged as errors                            | 21                        |
| Calls approved automatically by a configuration rule | 573                       |
| Calls approved by the person                         | 2                         |

![Bar chart of an AI agent session log: 1,003 model responses and 886 tool calls, of which 573 were approved automatically by a configuration rule and 2 by the person; 305 shell commands, 155 file edits and 21 results flagged as errors.](/images/blog/elroot/figures/ai-agent-session-log-tool-calls-and-approvals.png)

Even before any analysis, those numbers say something about risk. A third of the calls were shell commands, and almost every call that needed approval was approved by a rule written in advance, not by a person in the moment. That is normal for a productive session and it is exactly why the log matters: the rules made hundreds of decisions, and the log is the only record of them.

## What the log answered well

Four of our seven questions had direct answers.

Who made the call: every model response carries the session, the model name, the working directory and the git branch. In a system with several agents, that is the minimum for telling their actions apart.

What was called: each tool call carries the tool's name, its full input and a call id, and each result carries the same id back, so calls and results can be paired.

Whether it was allowed, and by whom: most results carry a permission decision with its source, which is how we know that 573 calls were let through by configuration and 2 by the person. Recording the source of an approval is the detail most homemade agent logs leave out, and it is the first thing an incident review asks for.

What came back: results carry their content and an error flag. The 21 flagged errors were mostly shell commands that failed, which is useful but coarse. A command that ran successfully and did the wrong thing is not an error by this measure.

## What we had to rebuild, and what we could not

Timing was the first gap. Only 38 of the 886 results carried an explicit duration. We rebuilt latency from the timestamps on each call and its result, for the 576 calls we could pair within the file: a median of 0.42 seconds, a 90th percentile of 6.8 seconds and a maximum of 33 seconds. The rest could not be paired from this file alone, which is its own lesson: if pairing depends on records surviving in one place, some will not.

Cost was the second. Token usage is recorded per model response, not per tool call, so the cost of a single web fetch, including the tokens spent reading its result on later turns, cannot be separated out. The totals were still revealing. The session's responses read just under 397 million tokens from the prompt cache, against about 1.16 million output tokens. Almost all of the work, and so most of the bill, was the model re-reading its own growing context. A log that records only input and output tokens would hide that entirely. We covered the budgeting side of this in [AI agent cost control](/blog/ai-agent-cost-control).

The third gap was purpose. Nothing in a record says which task a call belonged to. The session worked on several things in turn, and attributing a shell command to one of them meant reading the surrounding conversation. For a single person's session that is tolerable. For an agent working on a customer's records, "which job was this action part of?" has to be answerable from the log alone.

## The record we now want for every tool call

OpenTelemetry, the open standard most teams already use for traces, has a set of [generative AI attributes](https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/) that covers part of this. They are marked as still in development, so expect names to move, but using them now saves a migration later. Here is the record we recommend, with the standard name where one exists.

| Field                          | OpenTelemetry attribute                                                                           | Why it matters                                          |
| ------------------------------ | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------- |
| Operation                      | `gen_ai.operation.name` = `execute_tool`                                                          | Separates tool calls from model calls in the same trace |
| Agent and session              | `gen_ai.agent.id`, `gen_ai.conversation.id`                                                       | Tells agents and runs apart                             |
| Model                          | `gen_ai.request.model`                                                                            | A behaviour change often turns out to be a model change |
| Tool and call id               | `gen_ai.tool.name`, `gen_ai.tool.call.id`                                                         | Pairs each call with its result                         |
| Arguments                      | `gen_ai.tool.call.arguments`                                                                      | What was actually requested, after redaction            |
| Result                         | `gen_ai.tool.call.result`                                                                         | Or its size and a hash, when it is large or sensitive   |
| Tokens                         | `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.cache_read.input_tokens` | The cache reads are where the money went in our session |
| Start time and duration        | span timing                                                                                       | Recorded directly, never rebuilt                        |
| Approval decision and approver | none yet                                                                                          | Rule, person or policy engine, and which one            |
| Policy rule                    | none yet                                                                                          | The specific rule that allowed or denied the call       |
| Task or business object        | none yet                                                                                          | The order, ticket or document the call served           |
| Effect class                   | none yet                                                                                          | Read, write inside the system, or action outside it     |
| Estimated cost                 | none yet                                                                                          | Tokens times price, attributed to the call              |

The last five rows are the ones the standard does not define. They are also the ones that answer the questions people ask after something goes wrong: who allowed this, under which rule, for which customer, and did it change anything outside our own systems?

## Rules for logging agent tool calls

1. Write the record before the tool runs and complete it after. A call that crashes the agent should still leave a trace that it was attempted.
1. Log denied calls with the rule that denied them. A rising rate of denials is often the first sign of a prompt regression or an injection attempt.
1. Redact arguments before they are stored. Tool inputs carry credentials, personal data and file contents more often than people expect.
1. Store large results as a size and a hash, with the full content kept separately under the same retention rules as the data it came from.
1. Never sample writes. Sampling reads to save storage is reasonable; every action that changed something must be logged.
1. Give every call a task id at the moment the task starts, and pass it to every tool call the task makes.
1. Alert on three patterns: the same call repeated in a loop, an error rate climbing within a run, and cost per task jumping above its usual range.

Logs are also what make oversight possible after the fact, which is the subject of our post on [guardian agents and runtime oversight](/blog/guardian-agents-runtime-oversight). If you are putting an agent into production and want the logging designed in from the start, [talk to us](/contact).
