Skip to main content
Version: 2.0

Create and verify a custom metric

Create a count of recorded top-level invocations for one agent, then check the metric against that agent's activity. Author it under Dev tools > Custom metrics and view it under Build > Dashboards. The custom metric reference is the canonical list of supported fields, aggregations, and limits.

What you are creating​

A custom metric is a standing definition, not a query you run. You describe which recorded operations to select and how to summarize them. The platform then computes that summary as new activity arrives, stores the result as a time series, and serves the stored series to the dashboard and the API. Reading a metric never reads your traces.

This is what makes a metric useful for a question you ask repeatedly. A built-in metric already answers the common operational questions. A custom metric answers the ones specific to your deployment, which the platform cannot anticipate: how often one tool is reached, how many model calls a workflow spends, how much of your traffic a particular caller type drives. You define that question once, and it keeps answering.

Two properties of the computation shape everything else on this page.

A filter reads one span at a time. The wizard calls these rows. A row is a single recorded operation with its own attributes: one agent invocation, one model call, one tool call. Your filter is evaluated against each row in isolation. It cannot compare one row with another, read the rest of the session the row belongs to, or notice that something happened twice. Measurements that depend on those relationships are not expressible here, and the filter limits say so explicitly rather than leaving you to discover it by experiment.

What a metric measures is permanent, by design. If a filter could be edited, a stored series would change meaning partway along, and a single chart would silently mix two populations across the edit. Fixing the measurement is what lets a metric mean one thing for its entire life, which is the property that makes it safe to compare one month against another. The cost is that the thinking happens before creation rather than after. To change a measurement, create a second metric and keep the first until you have compared them.

Create metrics before you need them​

Backfill is bounded by the configured lookback, available traces, and the metric's retention. Creating a metric can compute data from before its creation, but cannot recover activity outside those bounds.

Define important measurements early rather than relying on backfill to reconstruct an incident. This improves the opportunity to retain the history you need, but is not a guarantee of complete historical coverage.

Decide up front what you intend to compare​

Some questions need two metrics, not one. A tool error rate needs its errors and its calls. Tokens per session needs its tokens and its sessions. The platform computes each metric independently and does not divide one by another, so you create both and divide them when you read.

Creating such a pair together can simplify verification, but creation dates do not establish comparable coverage. A later-created metric can backfill earlier activity, and two metrics created together can progress or fail independently.

Before dividing, verify matching computed coverage, compatible retention, the same agent scope and depth choice, and aligned read boundaries. Choose a window covered by both metrics and distinguish the numerator's condition from the denominator intentionally. An all-agent numerator over a single-agent denominator measures different populations. Treat a zero denominator as undefined.

Name for the question, not the implementation​

The name is fixed at creation and is the identifier your application passes to the API, so it outlives the Console session in which you chose it. Name it after the question it answers, not after how it currently computes. A metric named for its filter becomes misleading the moment you build a better one alongside it.

Before you begin​

You need an account with custom metrics enabled and permission to manage them. See metrics permissions. To inspect the source activity, you also need session and trace access.

Choose an agent that you are authorized to inspect. Use an existing completed top-level interaction with a known time, or, if authorized to run it, use its Preview tab to send a test message and wait for a response. See Create a session. Avoid an agent with side-effecting tools unless those effects are intended.

Record the agent key and a time window containing the completed activity. Use View traces to identify recorded invoke_agent activity for that agent. A transcript message alone does not prove that a matching span is available. Do not assume a test message always adds exactly one to a metric.

Plan before you create​

These settings are fixed after creation: Name, Agent scope, filter, aggregation, dimensions, and the Top-level sessions only choice. For why, see what you are creating. A replacement starts its own computation and backfill, and does not inherit the old metric's computed history.

This walkthrough uses one agent, Count, top-level sessions only, and no dimensions. Its result is not a count of unique sessions or people. If you need an all-agent metric that the dashboard can narrow later, read dimensions before creating a different metric.

1. Describe and scope the metric​

  1. Open Dev tools > Custom metrics and select Create metric. The wizard opens at Details.
  2. Set Title to Agent invocations. Enter an unused name such as usage.invocations in Name. The Console adds custom., producing custom.usage.invocations.
  3. Enter a Description, such as Recorded top-level invocations for the selected agent. Set Category to usage, choose the agent you identified above under Agent, and set Unit to invocations.
  4. Keep the default Retention for this walkthrough, then select Next. You reach Filter.

Check: the selected agent is the one whose activity you recorded. Unit is a label, not a conversion or calculation.

2. Select invocations​

Enter this expression in Expression (optional):

SELECT AGENT INVOCATIONS

Code example with sql syntax.
1

Leave Top-level sessions only on and select Next.

Check: the saved filter will also require depth zero. This excludes sub-agent sessions and spans without a recorded depth. It is not a filter for all spans inside a trace. See session depth.

3. Count and save​

  1. In Measure, choose Count under Aggregation. Count needs no value field. Leave Dimensions empty for this single-agent example.
  2. Select Create metric. You return to the custom-metrics list.
  3. Find the exact metric name you chose and check its Agent, Aggregation, and Status.

Check: the stored scope names your agent and the aggregation is Count. Creation can succeed before any chart data is available.

4. Verify the result​

  1. Open Build > Dashboards and choose a window containing the recorded activity. The window must comply with the query-window limits.
  2. Find the chart for your metric. Check the same agent, time window, and top-level population you used above. Do not compare it with an all-agent or all-depth total.
  3. If the chart has no observations, follow the diagnostic sequence below. If it has data, inspect a contributing run to confirm that the selected operation and agent match the intended measurement.
  4. If you need to reconcile a numeric total, use the metrics read workflow and recorded spans, account for returned bin boundaries, and include only the matching population. A backfilling metric or unavailable trace history is not sufficient evidence for an exact reconciliation.

Success means: the metric exists with the intended immutable settings, its selected window has computed observations, and the inspected activity matches the population you intended. A nonzero chart alone is not proof that the filter answers the right question.

If the chart is empty​

Work through these checks in order rather than deleting and recreating the metric:

  1. Confirm identity and scope. Match the metric name and agent. Make sure the chosen window contains the activity's end time.
  2. Inspect Status. For Error, read the detail beside the status. For Disabled, an authorized editor can resume computation. For Backfilling, historical bins can still be incomplete. There is no guaranteed completion time stated here.
  3. Verify matching observations. Inspect recorded activity for the chosen agent and operation. The depth-zero condition excludes missing depth and sub-agent sessions. A transcript or a built-in total can cover a different population.
  4. Check coverage. Historical traces might no longer be available for backfill. Review coverage and retention. Do not interpret No data yet in isolation.
  5. Escalate with evidence if unresolved. Provide the metric name, agent scope, UTC window, status/error detail, and relevant trace IDs to your account administrator or support contact. Do not include API keys or sensitive span content unnecessarily.

Manage the metric​

In the custom-metrics list, Backfilling means historical computation is in progress, Active means new data is being computed, Disabled means computation is paused, and Error means computation failed. The lifecycle reference explains retained data and Data since.

When editing, Computation > Compute this metric pauses or resumes computation. Resuming starts backfill from traces still available. Descriptive settings and retention can be changed, but changing the measurement requires a new metric.

Edit metric JSON​

Select Edit JSON in the wizard header to open Edit metric JSON. Applying JSON replaces values across the wizard steps. Review those steps before saving. This does not bypass immutable settings. The documented application workflow is reading metrics, not a public programmatic authoring contract.

Reference topics​

Choose an aggregation​

See Supported aggregations.

Filter and field vocabulary​

See Field vocabulary.

Top-level sessions only​

See Session depth.

Dimensions​

See Dimensions.

Retention and query windows​

See Coverage and retention.

Other measurements​

See Measurement recipes.

Limits to account for​

See Filter limits.

Last updated on by Ayesha Saddiqua