Horizon UI
1.1.0
Breaking changes and migration
Breaking: Each of the six Dashboard setup pages now has its own read/write permission pair, and three permissions are retired. dashboard:read / dashboard:write become layer-template:read / layer-template:write; overview:write splits into overview-template:write, translation:write, alarm-setup:write, infra-3d-setup:write and setup:write; overview:read keeps only its original meaning, the rendered overview dashboards. The built-in roles are updated for you, and a role still naming a retired permission keeps working — the old names, and the dashboard:* / overview:* wildcards, are accepted for this release and expand to what they used to gate. Rename them in your horizon.yaml before the next one; Horizon now logs a warning at startup naming any permission it does not recognise, which it previously ignored in silence. Two consequences worth checking: viewer and maintainer no longer read stored template configuration (they keep every dashboard they could see), and a role granted overview:read alone no longer reaches the template-administration pages.
Breaking: Turning the Cold pill on no longer re-reads the page. It changes what the NEXT read asks for — the following auto-refresh round, or whatever you do first, typically choosing the time range the cold data lives in. Flipping it used to start a round of its own, which on a cold tier routinely ran past the sixty-second cap; the sweep that followed then re-queued exactly the queries the cap had just cancelled, so the cap was at its most useless during the outage it exists for. And because a cold read REPLACES the hot one rather than widening it, an operator who had not yet moved the time range watched the whole page empty on the click. Anything already in flight is cancelled, so no batch is left half hot and half cold — the stage is read when a request goes out, and a call still queued behind the concurrency limiter would otherwise leave under the new stage while belonging to the old one. Nothing to migrate: if you want the page re-read immediately after flipping, press Refresh.
AI agent conversations
Features
Horizon can now read the AI agent conversations an OAP stores. OAP 11.1.0 keeps the conversations that the SkyWalking AI Sessionizer pushes under the AI_AGENT layer, one row per conversation and one document per conversation. This release wires Horizon to both reads, behind a new permission, ai-conversation:read, that the built-in viewer, maintainer and operator roles carry. It is its own permission rather than part of ai:read, because that one is the AI assistant, where Horizon sends text to a model provider; this one reads stored transcripts of other agents, and a deployment may allow either without the other. Two settings under performance.aiConversation tune the reads: listLimit, how many of the newest rounds the list folds into rows (default the OAP ceiling, 10,000, because a smaller number lets one long conversation hide the short ones), and viewTimeoutMs, how long to wait for a conversation document, whose first byte arrives only once OAP has folded the whole chain (default 120 s, matching OAP). See Configuration File and Roles and Permissions.
A new bundled layer, AI Agents, with a Conversations tab. When OAP reports the AI_AGENT layer, the sidebar gains an AI Agents entry whose one tab lists the conversations of one agent: pick the agent in the layer header and a time range (a day to 90 days), optionally one agent runtime, click Run query, and each row shows the conversation’s title, which Sessionizer sent it, its talks and unresolved references, its span and its last activity, newest first. A title text and a conversation id are query conditions too, applied by OAP. The line above the table states the round budget the list was folded from, because OAP builds the list from the newest rounds rather than from conversations and cannot say whether the budget cut anything. Layer templates gain the aiConversations component flag behind this tab. See AI Agent Conversations and AI Agents.
A conversation opens in its own tab, as one shareable page. Clicking a row on the Conversations tab opens /ai-conversation/<id> in a new browser tab: the conversation’s transcript, with the agent’s work folded under each input; a flow timeline of one execution stream with lanes for input, responses, context, model calls, tools, agent activity and nested child streams, relation lines for the selected step, and a way to dive into a child agent and back; and an inspector with the selected step’s details, relations and landed evidence. The address carries the conversation, its agent and agent runtime, and the reader’s position (talk, step, stream), updated in place as you move, so it can be shared and lands on the same step. Each read follows the current Cold selection, requesting cold storage only when enabled and otherwise using OAP’s default stages. The page reads the whole document from OAP and shows the wait in phases — OAP assembling the conversation with the seconds counting, the bytes arriving as n of m MB with a percentage, rate and time left, then the parse and the draw — states the document’s integrity (verified, incomplete, mismatch) and the Sessionizer’s recorded problems, and follows your Horizon theme, the kinds of step in their own colours per theme. The renderer behind it is a framework-free module the SkyWalking AI Sessionizer’s own viewer embeds too, built from a pinned Horizon commit, so both show a conversation identically. See AI Agent Conversations.
The AI Agents layer gains metric dashboards for the agent. OAP now turns Claude Code’s own OpenTelemetry metric family into per-agent and per-agent runtime metrics under the AI_AGENT layer, whether Claude Code’s exporter sends them or the AI Sessionizer derives the token metric from the transcripts it lands. The layer’s landing page lists agents with two columns, Tokens and Cache read share, and the layer gains an Agents dashboard and a Agent runtimes dashboard. Four widgets are always shown, Tokens by type, Main agent and subagents, Tokens by model and Cache read share; seven more, Cost by model, Active time, Sessions started, Lines of code, Commits, Pull requests and Edit permission decisions, appear only when Claude Code’s own exporter reports, because a Sessionizer-only deployment never has them. Every point is the total of its bucket at the minute a call ended, so no title says “per minute”. An OAP that already holds the earlier, conversation-only AI Agents template keeps it: adopt the dashboards from Dashboard setup → Layer dashboards → AI Agents with Reset to → Bundled and Check diff & push. See AI Agents.
The conversation page shows which files each tool call changed. When the AI Sessionizer recorded a call’s workspace changes, the call’s card in the transcript wears a mark beside the tool name, 2 files · +3 −1, or an eye reading read-only for a shell command the asz Claude Code plugin classed as read-only and did not scan, which is not the same as no changes. Clicking the mark opens the changed files under the card, each opening to its diff, without moving the selection; the same mark sits on the step’s clip in the flow timeline. The inspector gains a Changes tab with each record’s provenance: who recorded it (the agent’s own patch on Edit, Write and NotebookEdit, or the plugin’s observation of shell commands and subagent edits), the scans it ran between, the command’s outcome, the policy sets, a warning when a scan stopped early, and the landed position it was read from. Two records of one call are kept apart, the agent’s first, and one command that touched several configured workspace roots is listed by root. The status strip counts the conversation’s change records and opens a panel of every changed file by workspace root, each record a link that opens the step that made it on its Changes tab, with changes no observed tool window made listed at the end. See AI Agent Conversations.
The Conversations tab says what each conversation did. Its columns are now Talks, Model calls, Subagents, Bash runs and Changes: the change records the Sessionizer captured as of the newest round, one per observation (a watched tool call per workspace root, the agent’s own patch of an edit, a scan that found nothing, a change seen between calls that no call made), with the plus-over-minus mark and the lines those records added and removed, summed over the records, beside a count above zero. A dash in any of the four means the Sessionizer that pushed the conversation predates the count. They need an OAP that carries the counts in its conversation list.
The inspector says where a step sits in words. Its Details tab opens with the step’s place: the activity window and its number, the agent, the talk by its opening line and its place in the stream, the run’s place in the talk, and the step by kind and name, instead of a chain of clipped identifiers; the identifiers stay in the tooltip.
Tool inputs and results read as fields. On the conversation page, a tool call’s card and its Details tab draw the input the agent recorded as the fields it holds instead of one escaped JSON string: a shell command as the lines that were run, an edit as its path and the text before and after, a result’s output and error streams apart. A long text is cut short with a show all link that opens it in place, and where the document clipped the text a note says how much of it is there. Every field has a copy button beside its name, a result that is one text has one on its result line, an edit’s old_string and new_string are drawn as one diff with removed lines red and added lines green, and the inspector pops out into a panel over the whole page, four fifths of it, with the ⤢ button in its header, docked again with ⤡, Escape or a click outside it.
GenAI evaluation
Features
-
The Virtual GenAI layer gains an Evaluation records tab: the LLM-as-Judge results OAP stores for a provider, one row per judged call with its task, score or verdict, level, reason and judge model, over a level histogram of the page and a detail popout. Provider, model and the calling service are picked in the page with type-to-filter pickers; the conditions cover the value — a score range, a true/false verdict, or a string or JSON result — plus task, calling service, judge model, trace id and a time range up to seven days, and as on the Logs tab they are staged until Run query. The trace link on a row opens the trace the call was judged from — the native waterfall for a SkyWalking-agent trace, the Zipkin renderer for an OTLP one — looked up at the record’s own time, so an older trace still opens with the query cold stage on. The trace-id condition narrows to one span of that trace: Pick span… lists the trace’s spans with the ones the judge samples marked LLM and shown first, and picking one fills the segment and span index (native) or the span id (OTLP); the same fields can be typed in from ids copied out of logs. Reading the tab needs
logs:read; its provider picker works for a role withoutmetrics:read. -
The Virtual GenAI Models dashboard gains an Evaluation Score chart, and a point on it drills into the Evaluation records tab scoped to that provider, model, task and hour.
Sign-in and access control
Features
A custom role can now be added without restating the built-in ones. Naming rbac.roles has always replaced the block outright, so adding one role meant copying viewer, maintainer, operator and admin into your config and keeping those copies current forever. Set builtinRoles: keep and the built-ins become the base instead: a role name you list overrides that one role, a new name is added, and everything you did not mention stays as it shipped — landingByRole merges the same way. The default is unchanged (replace), because a deployment that trimmed its block to remove a role must not have it handed back by an upgrade, least of all admin. Under keep a built-in can no longer be dropped by omission; grant it nothing (admin: []) instead. The effective role names are logged at startup so you can confirm what a merge produced. See Access Control Configuration.
The Dashboard setup pages open read-only when you may look but not publish. Each page — Overview templates, Layer dashboards, Translations, Alert page, 3D Infra Map, Global defaults — is reached with its own read permission and published with its own write permission. Granting the read half alone shows the page in the sidebar and opens it with the whole configuration visible, every editing control disabled, and a banner naming the permission publishing needs; granting the write half turns the page back on. Previously three of these pages could not be opened at all without edit rights, and the other three rendered every button enabled and failed at the click. Refresh from remote is now treated as the read it is, so it works for a read-only viewer instead of being refused. A read-only page shows what OAP holds (or the shipped default) rather than an unpublished draft saved in your browser: the draft is kept and the page marks it local, but it is not displayed while you have no way to discard or publish it.
The login audit can now be stored in BanyanDB instead of PostgreSQL. Set audit.provider: banyandb and point audit.banyandb.address at a liaison’s gRPC port — including the BanyanDB SkyWalking already stores its telemetry in, since Horizon keeps the records in groups of its own (horizon_audit and horizon_audit_metrics, prefixed by namespace where two deployments share one server). The page is the same on either backend: the same rows, the same three filters, the same hourly summary and token-usage tab. Retention is the groups’ own lifetime rather than a job Horizon runs, so there is no sweep interval to tune; on a cluster, set shards and replicas, because the single-node defaults would keep the whole audit log on one data node. As with PostgreSQL, an address that is not loopback must use TLS unless allowCleartext says otherwise. See Login audit.
Fixes
A malformed permission no longer grants more than it says. A grant is at most three segments, but a longer one used to be silently truncated to its first three — so rule:*:typo was read as rule:* and handed over every rule permission, and rule:write:structural:extra matched rule:write:structural. Anything outside the documented forms now matches nothing and is named in the startup warning alongside other unrecognised grants.
The BanyanDB audit configuration is only what a deployment actually differs on — where the server is, how to authenticate to it, a namespace prefix, and how long to keep records. The groups’ layout is bundled rather than exposed: one shard, which a sign-in rate never outgrows. That is also the safe default, because BanyanDB fixes a group’s shard count at creation and does not move data when it changes — a shard count edited after records exist would route new ones away from the old ones rather than resharding. Horizon now reports such a difference instead of applying it. Replication is left to you: Horizon sets no replica count, so a deployment that wants the audit log replicated can create the two groups with the count it wants and Horizon will keep them that way.
The login audit list pages the way every other SkyWalking list does. It asks for a page number rather than carrying a position between requests, which is the arrangement OAP uses for traces, logs, alarms and events alike. The page still shows 50 rows at a time and still reports only whether more exist; paging is bounded at 500 pages deep, and a request past that is refused rather than quietly answered with something else.
The hourly summary and token-usage totals now refuse a window they cannot read completely. Both are sums, so a row that did not fit did not look missing — it looked like a smaller number. A window holding more rows than one read may return now reports that it cannot be shown, instead of drawing a total that is quietly low.
A database outage no longer costs the hourly sign-in counts it spanned. Each Horizon process now records a running total for the hour rather than a count of what happened since its last write, so the first write after the database comes back restores the figures for the hour in progress — and a write that is retried after an uncertain outcome leaves the same number instead of counting twice. The sign-in list is unchanged: rows from an outage are still not recorded.
Refreshing
Features
Refreshing is one coordinated round. A page used to refresh in pieces, on clocks that drifted apart: the header and the service roster followed the topbar timer, while a dashboard’s widgets and the alarms card each polled on intervals of their own. A refresh is now a single round — header, roster, every widget, the alarms card and the maps — all asking about one time window, and the round is not finished until the last of them lands. They still appear as they arrive, so a large comparison fills in over a second or two; what changed is that they are all answering the same question. The countdown measures the gap between rounds rather than between starts: while the readings are out it reads Refreshing, and it begins again when the last of them lands, so time spent loading is never charged against the interval. On a slow backend the interval genuinely stretches and two rounds can no longer overlap. A round that has started always finishes, even if you switch refreshing off or move to a page that pauses it — the next one simply does not begin. One that runs longer than a minute is given up on, so a single wedged screen cannot stop everything else refreshing.
Auto-refresh has its own on/off, separate from the interval. Turning it off and back on returns to the interval you last chose rather than forgetting it, and off means off everywhere: passing through a page that pauses refreshing and coming back no longer triggers a refresh you had switched off. Switching it on while a page is paused or the tab is in the background saves the setting without refreshing then and there, and the interval menu stays usable on pages that pause refreshing — how often to refresh outlives the page you happen to be on. Pages and overlays pause refreshing independently of each other, so opening the Smartscape hierarchy overlay freezes the background while you pan through it and changing the time range no longer unfreezes it underneath you.
The refresh control says whether anything is actually loading. A download arrow beside the countdown appears while requests are in flight and goes when they land — the icon used to spin whenever auto-refresh was merely enabled, which told you nothing about whether data was arriving. While a round is out the refresh buttons are disabled rather than accepting a click that could only ask for what is already being fetched.
A refresh that fails is recorded where you can go and read it. Failures from the timer collect beside the refresh control, newest five first, with a count of the ones you have not seen — nobody asked for that round, so it waits rather than interrupting. Each entry names the screen, what it was trying to do, the request and the server’s answer; secrets in the URL or the response are removed before it is shown. Failures from something you just did — expanding a node, for instance — appear immediately instead, as a message that pauses while you read it and can be opened for the same detail.
Fixes
Overview widget grids and layer dashboards emptied on every refresh. Both treated a moving time window as a new question, so each tick blanked the grid and replaced it with a loading line — over widgets that were usually about to be filled with the same numbers. They keep the previous values while the next reading is out and replace them in one go when it lands.
The Alarms card on a dashboard refreshed on a clock of its own. It polled every minute regardless of the page’s own cadence, so a dashboard set to refresh every fifteen seconds showed an alarm list up to a minute older than the metrics beside it. It moves with the rest of the page now.
A read you walked away from went on costing OAP work. Navigating away mid-load, or a refresh that gave up, stopped the browser waiting but not the query behind it — so an abandoned page went on costing OAP the whole fan-out, multiplied by the node count on a cluster. Those reads are now cancelled all the way through. Anything that CHANGES something — creating a profiling task, pushing a template — still runs to completion, so a closed tab cannot leave it half applied.
Clicking Refresh against a dead backend appeared to do nothing. The failure was filed away and the screen said nothing, though you had just pressed the button and were watching. A refresh you asked for now answers on screen when part of it fails — once, however many widgets failed — and points at the list beside the control, where the detail is.
One outage was listed several times. Two screens sharing the same reading each recorded it, so the failure list showed one problem as several. Each is recorded once now, and a comparison entity that fails on its own is recorded at all, which it was not before.
A failed read was drawn as a page of zeroes. When a layer read could not reach OAP the answer came back empty, and an overview’s KPIs rendered that as 0 and its service counts as none — indistinguishable on screen from a system that genuinely had nothing running. Overviews and the layer service list now keep the last values they read and report the failure instead. Layer dashboards keep their widgets through the same failure rather than emptying them.
Query cold stage could leave hot and cold answers on screen together. Flipping it re-read each screen on its own, so the page showed both stages while it settled, and a screen that had not caught up yet could file its answer under the wrong one. The whole page is re-read as one round now.
A refresh that gave up left its heaviest requests running. The sixty-second cap cancelled the query but not the request behind it for the layer landing, the dashboard batches and the overview — the browser stopped waiting while OAP finished a fan-out nobody would read, multiplied by the node count on a cluster. Those now stop with the round.
Two reads racing could leave every request an hour wrong. The server timezone and the capability probe were each fetched per request, so a round’s dozen reads probed in parallel; behind a load balancer a fast success could be overwritten by a slower timeout falling back to UTC, and that answer then served every request for the next minute. Each is now read once per expiry and shared.
A brief storage hiccup emptied the sidebar for a minute. A failed service-catalog read replaced the known-good roster with an empty one and cached it for the full minute, so service counts fell to zero and group navigation collapsed long after OAP was healthy. The last good roster is kept and marked stale instead, and a failed read is retried within seconds — for as long as the outage lasts, rather than surviving only the first failure.
A landing metric batch that timed out silently reordered the layer. Its services were left without a value, sorted to the bottom as though idle, and could fall out of the top-N — choosing a different busiest service and a different default — while the response still reported success. Services whose metric could not be READ are now ranked above those that genuinely reported nothing, and the layer header says so on screen — naming how many batches failed — rather than leaving the blanks to be read as zeroes. An overview whose widgets aggregate across services says the same thing, because there a lost batch leaves a total quietly low rather than visibly absent. It keeps saying it for as long as the incomplete reading is on screen, not only on the refresh that hit the timeout — and an incomplete reading is re-read once, behind the values already shown, so a momentary backend failure costs a few seconds of a partial header rather than the rest of the hour.
Topology, deployment and dependency maps
Features
Changing what you are looking at says what it is loading. Picking a different service, endpoint, depth or time range names its target — Loading topology for “checkout”… — instead of leaving the previous answer under the new heading while the next one is fetched.
Fixes
The 3D Infra Map follows the theme: its canvas, tier slabs, rims, cube edges, panels and cards took their colours from a fixed dark palette, so a light theme such as Daybreak showed dark panels with unreadable text. They now come from the theme tokens and change with the theme.
The 3D Infra Map opens framed on the tiers that hold services. A deployment with one populated tier had its slab at the top edge of the view, because the default pose aimed between empty planes; the default and Reset views now frame what is there.
An expansion on the API dependency graph could land in a graph you had left. Nothing stopped you switching endpoints while one was loading, and the branch then arrived in whatever graph had replaced it. Expanding now shows its pending state and holds the endpoint picker and the other expand handles until it lands. An expansion started while an admin preview is open also resolves against the draft being previewed, rather than against the published template the rest of the graph is not showing.
A map that failed its FIRST read showed a loading line for ever. With nothing cached to fall back on there was no way to tell a read still in progress from one that had already failed, so the graph sat on “Reading data…” indefinitely. It now says the read failed and offers Retry.
The maps blanked on every refresh. The Topology, Deployment, API dependency and instance-relationship graphs dropped to a loading line and came back on each tick, taking your zoom, pan and any nodes you had dragged with them. They redraw in place now, and are re-framed only when the question changes — a different layer, service, endpoint, focus, depth or time range. A service appearing or disappearing no longer re-frames the canvas or discards placements.
A failed reading erased the map you were looking at. When a round could not reach OAP the graph was replaced by an empty one, so an empty map could mean either that there was nothing to show or that the read had failed. The previous picture stays and the failure is reported beside it. A layer whose template an administrator has disabled is the deliberate exception — that is an answer rather than a failure, so the map is cleared and says so, instead of telling you to check a backend that is fine.
A failed expansion on the API dependency graph was reported as “no further dependencies”. A read that could not be completed marked the branch exhausted and faded its handle — a claim about your system made from a failure to read it. It now says the expansion failed and leaves the handle live, so the click can simply be repeated.
The API dependency columns re-sorted themselves on every refresh. Rows were ranked by the centre metric, which moves every cycle, so an endpoint you were watching had to be found again after each tick. Which endpoints get a row is still decided live — one that becomes busy enough still appears — but an endpoint already on screen keeps its place and an arrival takes the next free one.
Pods drifted on the Deployment map, and dragging could stop working. A placement was held as an offset from the packed position, so a refresh that added a pod carried yours along with the re-pack; and the map only re-armed its drag handles when a count changed, so a refresh that swapped one instance for another silently left them unbound. Placements are absolute now, and a node on the API dependency graph keeps its place the same way.
Dashboard templates
Features
Cluster Status shows what the template store has actually loaded. A new Dashboard templates pane reports how many layer templates, overviews, alert pages and translation overlays are being served from OAP, and when they were last read — and, when a read fails, the message it failed with, so “unreachable” is no longer the only thing you are told about a 404, a 401 and a timeout alike. Its badge separates a store that cannot be read while your dashboards keep rendering from one that has never been read at all.
A template published elsewhere reaches an open browser on its own. Horizon re-reads the template store on a slow cycle of its own, so a dashboard pushed from another Horizon, from swctl, or from anything else writing the same OAP store appears within about a minute — without anyone reloading the page.
Every MQE field in the layer template editor can now be run where you type it. A run button beside each expression opens a panel that picks a service (and, where the metric needs one, an instance, endpoint or process), picks a time range with the same precision-tabbed picker the top of the page uses (custom ranges included), fires the expression against your OAP and shows exactly what came back — a graph of the series it returned, aligned on one time axis with a legend per label, the entity each ranked row belongs to, and OAP’s own error message when the expression is wrong. Until now the editor fired no queries at all, so the only way to find out whether an expression worked was to push the template, open the layer and look. Every field is covered: the service-list columns, the topology, deployment, process and API-dependency metrics, the dashboard widget expressions and their visibleWhen gates. Relation metrics ask for both a source and a destination, since that is what they measure. A blank service-list column is runnable too — it reports the catalog default that will run in its place.
Fixes
A template store that could not be read emptied every dashboard, overview and map. They are rendered from the last successful read now, so a brief outage of OAP’s admin port leaves the console up, with the banner saying how stale it is. Horizon still never substitutes the templates bundled in the release — showing shipped defaults in place of your own configuration would misrepresent what is on screen — so a Horizon that has never read the store still blocks those pages rather than inventing content for them.
The unreachable banner never cleared once the store came back. Nothing re-read the store’s status after the page had loaded, so the warning stayed up — and the pages behind it stayed blocked — however healthy OAP had become. Both recover on their own now.
Overview dashboards
Features
A calendar heatmap widget for overview dashboards. The new calendar-heatmap widget type draws a fixed window of the last 7 to 93 days (30 by default), and lays it out to fit: up to 14 days by the hour, one row per day and one column per hour of the OAP’s clock, so ten days read as 240 hour cells over the calendar days they touch; a longer window by the day, one row per week and one column per weekday, Monday first; a Resolution control in the editor forces either. Each cell is shaded by its value, with the window total in its footer; clicking a cell keeps it picked and reads its date and value out beside the total. A line comparing the total to the text of a well-known book can be turned on per widget. It is the one overview widget that does not follow the time picker: its window is part of the template, so the grid reads the same whatever range the rest of the page is on. Give it a plain per-service metric such as meter_ai_agent_tokens, a unit, and whether to sum or average the layer’s busiest services into each cell; the shades are cut at the quantiles of the days that saw traffic, so one heavy day does not flatten the rest, today’s cell is outlined as the day in progress, and each cell’s tooltip carries its date and value. The grid fits its card: cells stretch to the width and height available, a card too narrow for the grid’s long axis turns it the other way, and a day cell wide enough shows its day number. It refreshes with the page and keeps its cells while it reads. Add one from Dashboard setup → Overview templates → Add widget; the drawer edits its metric, unit, aggregation, window and comparison line. See Overview Widgets.
A ranking widget, and range totals for page-side KPIs. The new ranking widget type lists a layer’s services ranked by one per-service metric over the picked time range, busiest first, each with its value and a bar against the top one, in as many columns as its height needs past five rows; it lists up to twenty. Page-side KPI rows and ranking widgets can also ask for a range total (rangeTotal): a service’s buckets are summed over the picked range instead of averaged, so a counter such as tokens reads as the total for the range rather than a value per interval. Add one from Dashboard setup → Overview templates → Add widget; the drawer edits its metric, unit, row count and the range-total switch, and the same switch sits on each KPI row of a page-side tile.
A bundled AI Agent Overview page. When OAP reports the AI_AGENT layer, the sidebar gains an AI Agent Overview with the agents reporting on the layer and their tokens over the picked time range, in total, by type (input, output, read from cache, written to cache), the part spent by subagents, and their cache read share, a Top 20 agents list ranking them by that total in two columns, an Hourly tokens grid of the last 10 days by the hour, and a Daily tokens calendar heatmap of the last 30 days, the busiest agents summed (the footer says when a layer has more than one read covers). It carries only the metrics every metrics source produces; the exporter-only families (cost, sessions, lines of code) stay on the layer’s own dashboards. Like every overview it is editable under Dashboard setup → Overview templates. See AI Agent Overview.
Operating Horizon
Fixes
The sidebar uses less left padding and makes layer grouping clearer. AI Agents, Browser and other standalone layers align with group headings. A group with only one visible layer shows that layer directly at the same level; groups with multiple visible layers can be folded and unfolded by clicking their heading. Group headings have the same row height as layer entries. Navigating to a layer opens its group, while a menu refresh preserves groups you folded. Satellite and other direct layer links under Platform monitoring no longer show an extra service-count badge.
Data retention now reads the same on every storage backend, and says which setting governs each figure. The page used to mirror OAP’s wire shape — a Records / Metrics split, a Minute / Hour / Day trio, and a class called Normal — which on ElasticSearch, MySQL, PostgreSQL, H2 and TiDB meant nine numbers that were really two, presented as nine knobs you could turn apart. Those backends now show the five things retention is actually decided for — Metadata, Metrics, Logs, Traces and Others — each naming core.recordDataTTL or core.metricsDataTTL beneath it, with a note saying the two settings are all there is. BanyanDB shows the same five in the same order, expanding Metrics into its minute / hour / day groups and adding Zipkin traces and Browser error logs, because there each one is a separate bydb.yml group that genuinely can be tuned apart.
The retention chart hid data classes whose retention happened to match. Classes sharing a figure were merged into a single All records (5) bar — which reads as a fact about the backend when it is only a fact about today’s configuration, and left an operator unable to see what they could change. Every class now has its own row. Others is the one row named for a group rather than a data type: BanyanDB keeps alarms, events, sampled traces, top-N and every profiling record together in records, so they share one retention and no name for a single one of them would be true.
On the live debugger’s Capture history page, an active session’s start time ran into the rule name beside it. The left column of an Active row carries the DSL badge, the LIVE marker and the start time, and had no room for all three, so the time overlapped the catalog · file · rule text of the same row. The time now sits under the badges.
Traces, logs and events
Features
A trace can be looked up by its id. The Traces tab, native and Zipkin, and Trace inspect gain a Filter / Trace ID switch at the top of the query form. Trace ID trades the conditions for an id field and a time range that starts at No time range: Run query reads the id within the range you pick — or, with none, wherever the hot and warm stages keep it — with no service and no other condition, and refuses to run without an id. Switching back finds the filter as you left it. The Zipkin side takes several ids at once, each locked in with Enter, and names the ids that matched nothing. This replaces the Zipkin tab’s Open trace ID field, which Run query ignored and Enter turned into a popout, and gives Trace inspect’s Zipkin mode the lookup by id it never had. With no time range the cold stage is not read — pick a range with Cold on for a trace that old — and on BanyanDB with OAP 11.0.0 and earlier the lookup covers the last 24 hours only.
Trace queries run only from Run query. On the Traces tab, native and Zipkin, and on Trace inspect, neither switching modes nor pressing Enter runs a query — Enter in an id field locks the id in, and Enter in the Zipkin tab’s Annotations or custom-range fields, which used to run the query, now does nothing.
Trace inspect’s Zipkin annotation query suggests keys and values, as the Zipkin Traces tab’s does: the keys OAP is configured to index, and after key=, that key’s values.
Fixes
A trace’s age reads as days and hours. The trace list on the Traces tab, native and Zipkin, wrote an age past a day as a count of hours, 288h ago; it now reads 12d 0h ago, hours and minutes under a day, minutes under an hour. The Auth status admin page’s time of the last LDAP probe reads the same way.
Annotation suggestions on the Zipkin Traces tab vanished once the query held more than one term. They completed only the last term, while the browser matches and replaces the whole Annotations field, so after error nothing was offered, and picking a value would have dropped error. Each suggestion now carries the terms already typed. Trace inspect’s Zipkin annotation query, which gains the same suggestions in this release, behaves the same way.
Switching the Zipkin service on Trace inspect kept the previous service’s span name and remote service. They were cleared only when the service field was emptied, so going from one service straight to another ran the new service with the old one’s picks and matched nothing. They are cleared on every switch now, as on the Zipkin Traces tab.
Running a new query left the previous trace open on the Traces tab. With a trace open beside the result list, clicking Run query again refreshed the list but kept the old waterfall on screen — so a query that matched nothing still showed a trace, as though it were the answer. A new run now closes the open trace and clears any pick on the Distribution chart, on the native and Zipkin tabs alike, and a list waiting on its first rows says Reading data… instead of reporting an empty window before the answer has arrived.
Query cold stage
Features
Cold is asked for only where cold data is kept. Traces, logs and metrics are the classes a deployment is advised to age into cold storage, and they are now the only ones Horizon sends the flag for. Alarms, the instance and endpoint pickers, events and everything from profiling stay hot — they are small, and they are the first things you reach for during an incident, so a Cold toggle emptying them bought nothing. Those pages keep answering while Cold is on.
Fixes
Zipkin traces ignored the Query cold stage toggle. The Zipkin Traces tab and its trace detail read the hot stage whatever the topbar toggle said, and said nothing about it. Both now send the toggle to OAP with the query, so with it on a Zipkin search reads the cold stage like a native one; a detail opened from the list is looked up in the window the list was read with, since a cold trace is older than the day OAP would otherwise search.
The cold-stage warning told you to pick a window that does not exist. On a deployment with no cold stage configured — the BanyanDB default — it still said “pick a window older than N days”, which cannot work, and following it moved the range out of hot+warm so the warning hid itself: a blank page with the one sentence explaining it gone. It now says plainly that no cold stage is configured and stays up until you turn Cold off. Where cold IS configured, the suggested window is the one that clears the deepest class rather than the shallowest — it used to name the records boundary (3 days by default) while every metric widget needed 7, so following it brought traces back and left the metrics exactly as empty. Both the warning and its advice are now decided by the classes Horizon actually reads under Cold — traces, logs and metrics. Alarms, events and profiling retention no longer count: they are usually kept for the shortest time of anything, and reading them here hid the warning for exactly the windows an operator turning Cold on is most likely to be looking at.
Dashboards
Features
The layer header’s KPIs describe one completed hour, and say which one. They used to be read over whatever window the time picker held, which meant reading the sort metric for every service in the layer on every refresh — hundreds of requests on a large layer, repeated every thirty seconds, for figures that only move by the hour. The header now reads one completed hour and holds it, and the hour it covers is named above the table (09:00–10:00) so the numbers are never older than they appear. Ten minutes are left for the backend to finish aggregating that hour, so the header moves on shortly after ten past. While the next hour is being read the previous one stays on screen with a star against each figure, instead of the table emptying. A deployment too new to have any completed hour shows the hour in progress instead — a finished hour is preferred to it whenever one is available. Whenever no hour can answer at all, the header reads the time picker’s window exactly as it always did: on a deployment young enough that the backend has written no hour-level figures yet (they are aggregated on a longer cycle than minute ones), and on the first visit to a large layer while the hour is still being read. Either way the page opens with real numbers rather than a table of dashes. Everything below the header — the trend lines, the dashboards, the maps — still follows the time picker as before.
The BanyanDB layer has a Trace Sampling page. BanyanDB 0.11 can run a chain of sampler plugins during trace merge and finalization, dropping whole traces after their fragments have landed — tail sampling inside the storage tier. The new page is an extension page under Cluster, with its own sidebar row, and it reads the meter_banyandb_trace_sampling_* families OAP collects from the plugin chain: the committed drop ratio and the trace outcomes behind it, per-plugin Decide rate and latency, chain batching, what the first-party samplers proposed by rule, every fail-open guard and drop-set capacity counter, finalization state, and the host’s bounds on a plugin’s own telemetry. It is cluster-wide with the storage group as a series rather than a per-group dashboard, so one page charts every group at once.
The Envoy AI Gateway layer is now called Agent Router. Envoy AI Gateway joined the Agentic AI Foundation as Agent Router, with the same code and APIs, so the bundled layer template now shows Agent Router in the sidebar and Agent Routers as its service label, in every language. Only the display names changed: the layer key ENVOY_AI_GATEWAY, the meter_envoy_ai_gw_* metrics and every widget stay as they were. An OAP that already stores this layer’s template and translations keeps the old name; to adopt the new one, open Dashboard setup → Layer dashboards → Agent Router (listed as Envoy AI Gateway until then), choose Reset to → Bundled and Check diff & push, then do the same for each language on the Translations page. See Agent Router.
Fixes
A compared service that failed to refresh was drawn as though it had. In Compare, each entity keeps its last good reading when its own read fails — deliberately, so one failure does not blank its siblings — but the retained series was then plotted across the CURRENT axis and ran to the right-hand edge beside the fresh ones. Ten-minute-old latency sat next to live latency with nothing to tell them apart. Each series is now placed at the buckets it was actually read for, so a stale one stops where its data does and the gap is visible.
A layer whose dashboard an administrator removed no longer reports an outage. The page said the template was disabled and pointed at an admin screen to re-enable it — a control most operators cannot reach, on a page that does not exist under that name — while the refresh history recorded “OAP could not be reached” for what was an administrative decision on a healthy server. The page now simply says it is not available, and the layer leaves the sidebar on the next menu read.
Reactivating a layer brings back its own dashboard, not the shipped default. OAP keeps a disabled template’s configuration; Reactivate was re-pushing the bundled default over it, discarding every edit the layer carried. It now restores what was there, falling back to the bundled default only when there is nothing to restore.