1.0.0

AI assistant (new)

• Ask about your system in plain language and get answers built from the real dashboard widgets — not just text. A new floating AI Assistant launcher (right edge, after login) opens a chat: describe what you want to know (“what’s unhealthy right now?”, “investigate latency for a service”) and the assistant reads live OAP data through the same query path the dashboards use, then streams back an ordered narrative with inline charts, top-N lists and tables drawn by the same widget components you see across the UI. Figures are numbered so the prose can reference them. Open it as a side drawer, expand to a full page (/ai), or open in a new tab; past conversations are kept locally in your browser.
• Read-only, and it inherits your permissions. The assistant can list layers/services, read active alarms (the “what’s unhealthy” signal), browse the per-layer metric catalog (the curated MQE for each metric), drill a service down to its instances/endpoints, and render those metrics — every data tool checks the same read verb you already hold, so the assistant never sees more than you can, and it never changes configuration, rules, or dashboards.
• It renders the real feature views inline — same components, focused on the service. Rather than a link you open, the assistant mounts the actual product views inside the chat, read-only and scoped to the service: ask for topology and it embeds the real service map focused one hop (hex nodes, edges, RPM/latency, zoom controls); ask how a service maps across layers and it shows the real Smartscape hierarchy fan; ask for traces and it embeds the Traces view — the trace list and the span waterfall, with the click-a-trace → open-its-spans interaction intact — for both trace modes: a native SkyWalking layer embeds by service, and a Zipkin-tracing layer (mesh / k8s) is embedded too by first listing the Zipkin services and matching yours to a Zipkin-side name (they key on their own service universe); ask for logs and it embeds the layer Logs view — the stored log stream with its row → detail popout; for a browser app, ask for browser errors and it embeds the client-side JS error list with its row → stack-trace detail; ask for deployment and it embeds the real Deployment view — the instance-to-instance call graph within the service, with pan/zoom and node/edge detail; ask for API dependencies and it embeds the real API-dependency view — the service’s busiest endpoint and its upstream/downstream dependency chain, where you can expand any node; name a source and destination service and it embeds the real Instance map — the instances of each service and the calls between them. One renderer per surface across the product, so what you see in the chat is exactly what the layer tab shows.
• Read live Kubernetes pod logs, right in the chat. For a k8s workload the assistant pulls a pod container’s on-demand logs (the error stack) and shows the fetched lines inline as a read-only result — the same on-demand-log path as the Pod Logs tab, so nothing is stored and it inherits your logs:read permission. It’s a result, not a console: no tail or refresh controls (operate a live tail in the Pod Logs tab); when a content filter was applied the block shows it, so an empty result reads as “nothing matched this filter” rather than a silent pod. On-demand logs are gated off by default on OAP; when they’re disabled the assistant says so instead of failing. K8S (cluster/node) and K8S_SERVICE (service/pod) metrics read through the normal catalog like any other layer.
• It can propose profiling — the right kind for the target — and only you start it. When metrics and traces can’t localise a cause, the assistant presents a decision card explaining what it found, why profiling would help, and what it expects to reveal; nothing runs until you approve it in the popout, and only if you hold the profiling permission. It picks the profiling flavour that fits the target — trace sampling, async-profiler for a JVM service, pprof for a Go service, eBPF on/off-CPU, or network profiling — by reading which types the layer is set up for and the service’s runtime language, and resolves the target instances for you. On approve it starts that task; once it has collected, ask the assistant to analyze it and it renders the result inline, captured so it replays identically on reload: for a code profile, the flame graph the Profiling tab draws (with the hottest frames called out in the units that profiler actually reports), and for a trace profile the profiled trace’s span waterfall beside the flame — the trace+profiling combination; for network profiling, the captured process-conversation graph. When there’s nothing to show — no Rover/eBPF agent, or a task that ran but collected nothing — it says so in plain text rather than pointing you at a live view that would drift from what it read. It never triggers anything on its own.
• Your backend decides what profiling is allowed — the assistant no longer refuses on its own guesses. Previously the assistant could decline to even show a profiling card: because the layer’s template didn’t list that profiling type, because the instances reported a runtime the profiler doesn’t match, because no process had advertised eBPF support recently, or because no process had reported in the last 30 minutes — and it phrased those as facts about your deployment (“the GENERAL layer does not support network profiling”). None of those are checks OAP performs when a task is created, so operators were blocked from tasks their backend would have accepted, by Horizon’s own configuration. The assistant now shows the card and tells you what it’s unsure about — that the layer isn’t set up for this type, that the runtime looks like a mismatch so the profiler will probably collect nothing, that no Rover agent seems to be reporting — and lets you decide. If the backend really does refuse, you get OAP’s own reason (“The instance doesn’t have processes.”), which is both accurate and actionable, instead of a guess. The assistant is also now honest about what a window means per flavour: OAP runs every network profiling task for a fixed 10 minutes and ignores any duration, and an empty process graph shortly after approval means “too early”, not “unsupported”.
• Off-CPU eBPF profiles are ranked by time blocked, not by how often a thread yielded. An OFF_CPU profile is what you reach for when the question is “what is this service waiting on”, but its frames were ranked by scheduler switch count — so a method that yields thousands of times for microseconds outranked the one that blocks for a second, inverting the answer. Off-CPU results now aggregate by duration. Relatedly, the assistant now names each profiler’s real unit when it reads a flame back to you instead of assuming one: trace self time is milliseconds; pprof is a sample count; async-profiler depends on what was captured — CPU-family events (CPU, WALL, CTIMER, ITIMER) are a sample count, but ALLOC is bytes allocated and LOCK is nanoseconds of contention, since those two record a value rather than a count. A task that captured more than one distinct event (say CPU + ALLOC) renders one at a time and names the others.
• An approved profiling task starts immediately, timed by your backend rather than your laptop. The task’s start time was sent as the browser’s wall clock, but OAP treats it as “begin after this moment” and only dispatches tasks starting within ±5 minutes of its own clock — so on a workstation whose clock had drifted (a suspended VM, a container host without NTP) the task was accepted, returned an id, the card said “started”, and nothing ever ran. Both trace and eBPF tasks now use OAP’s “start as soon as possible” semantics, so there is no clock to disagree about.
• An event the assistant doesn’t recognise is no longer silently swapped for CPU. Asking for a profiling event outside its known list produced a card that read as requested but fired a CPU profile instead. The event is now passed through for OAP to accept or reject by name, and the assistant flags that it didn’t recognise it — a clear rejection beats a wrong profile. Relatedly, when a HEAP or ALLOCS pprof profile is read back, the assistant now states that the ranking is a count of sample records rather than bytes, so it can’t be misreported as memory held.
• Long eBPF profiles say when only part of the range was analyzed. A wide eBPF task could exceed what OAP fetches in a single analysis, and the excess was dropped server-side with nothing on screen to indicate it — a partial flame graph presented as complete. The assistant now bounds the request to the most recent profiling schedules and tells you when it did.
• An empty network process graph is no longer reported as a missing agent. Because a network task runs for a fixed 10 minutes, analyzing shortly after approval legitimately finds nothing yet; that now reads as “still collecting”, and a missing Rover agent is offered as the likely cause only once the window has actually elapsed.
• A profiling task that is still collecting is no longer reported as one that found nothing. Asking the assistant to analyze a task shortly after you approved it — the obvious thing to do — could come back with “the task ran but produced no analyzable stacks; do not retry”, because any progress log was read as “it finished”. The first log a task gets means only that it was handed to the agent. The assistant now distinguishes the three real states: still collecting (wait and ask again), finished with nothing above the sampling threshold (don’t keep retrying), and failed on the agent — an execution error, or a profile too large for OAP to accept, which it now surfaces with the instance and what to do about it instead of silently reporting an empty profile.
• Allocation and multi-event profiles are read whole. An async-profiler ALLOC profile was showing only half its data: OAP splits allocations into in-TLAB and outside-TLAB trees, and only the first was read — so the large-object allocation paths, usually the reason for opening an allocation profile, were missing. A task started with several events (say CPU + ALLOC + LOCK) likewise rendered only the first. Both now render every tree the task produced.
• Network profiling results no longer come back empty on a wide time range. The process-conversation graph is built from metrics OAP only keeps at minute granularity, so reading it against an hour- or day-scale chat window returned nothing — which the assistant then reported as “no Rover agent / network profiling unavailable”. The read is now always minute-granular.
• Trace profile analysis stopped asking OAP the same question many times over. One analysis query was issued per profiled span, though OAP marks profiling per segment — so a segment with 20 spans cost 20 near-identical queries, exhausted the analysis budget 20× faster, and could trip OAP’s “only part of the snapshots were analyzed” warning, which was then relayed to you as a truncated profile that was never truncated. One query per segment now.
• A point-in-time pprof profile (HEAP, GOROUTINE, ALLOCS, THREADCREATE) is labelled as one instead of showing a “0 min” collection window, which read as a task that collected for no time. Relatedly, the process list used to confirm an instance can be network-profiled no longer counts virtual processes, which OAP’s own check excludes — it was possible to be offered an instance OAP would then reject.
• pprof collection windows are validated in the unit OAP actually uses. The pprof task duration is measured in minutes and capped at 15 by OAP, but Horizon was bounding it as if it were 600 seconds — so an over-long request sailed through and came back as an opaque backend error after you approved it. It is now bounded at OAP’s real limit. Likewise, an async or pprof task targeting more instances than a single task allows is now rejected with a clear reason instead of quietly profiling only the first 32 while the card said the whole fleet, and an eBPF task shorter than OAP’s 60-second minimum is refused up front with that reason rather than after approval.
• Guided root-cause analysis. Ask “what’s the root cause?” and the assistant follows built-in investigation playbooks — a master method (locate the root service → calling chain → error stack; walk the dependency topology upstream and fix a sick upstream first; a remote / Virtual_* dependency exposes only its client-side edge metric) plus latency, error-rate/SLA, saturation, middleware, Kubernetes-workload, and service-mesh specializations. It can also follow the cross-layer hierarchy from a service down into its backing infra/database layer, where memory / disk / connection causes live. Because a database / cache / MQ is a topology leaf with nothing downstream to walk, the middleware playbook now bottoms out there and pivots to its logs + Kubernetes hierarchy (its pods’ memory / disk / connection pressure) + the network edge; the Kubernetes-workload playbook works pod restarts / OOMKilled / CrashLoopBackOff → pod logs → limits-vs-usage headroom → node pressure.
• Bring your own LLM — vendor-neutral, and off by default. Enable it with the new ai: config block (HORIZON_AI_*). The default transport is OpenAI-compatible (any OpenAI-shaped endpoint — a hosted model, a local model, or an AI gateway; set model + base URL + API key); Amazon Bedrock is also supported (provider: bedrock). The API key is a secret, env-only, redacted from logs and excluded from the audit trail. The launcher shows for every signed-in user so the AI-powered APM is discoverable; until an admin points it at a model the panel opens read-only with a short “ask your administrator to set it up” notice, and who may actually use it once configured is controlled by RBAC (ai:read, granted to every role by default).
• Both prompts are yours. The assistant’s system prompt (ai.systemPrompt) and the starter example chips shown in an empty chat (ai.starters) ship with sensible defaults and can be replaced entirely in horizon.yaml.
• Starter chips can ask for a service first. A starter that names a <service> (e.g. “Investigate latency for <service>”) opens a one-field fill-in on click: type the service free-form — a partial name or a description is fine — and it’s dropped into the prompt before sending. You don’t have to know the exact name or its layer; the assistant matches what you typed to a real service through its own cross-layer search. Plain starters without a placeholder still send in one click. The bundled starters include these service-scoped prompts; operator-supplied ai.starters can use the same <service> / <layer> convention.
• Your question stays in view while the answer streams. The message you just sent pins to the top of the chat and stays there as a sticky header — while its answer streams in and as you scroll through it — so you never lose sight of what you asked; opening or switching to a past conversation pins that chat’s last question the same way. Each turn now carries a timestamp: when you sent the question, and when its answer finished.
• The assistant orients on a layer’s template before reading it — so it answers in the layer’s own terms. A new capability lookup surfaces, per layer: what a Service / Instance / Endpoint is CALLED there (a K8S_SERVICE instance is a Pod, a mesh instance a Sidecar, a K8S service a Cluster), which components the layer carries, whether its traces are native or Zipkin (so the assistant picks the right trace tool instead of guessing from the layer name), and each relationship’s metric legend — which metric is the HEALTH signal vs the LOAD, with its threshold bands. So a topology answer reads “success rate 98.3% → below the 99% warn band” against the operator’s configured thresholds rather than a guess, and narrates in the layer’s vocabulary.
• Embedded maps are captured evidence now — they replay exactly on reload, offline, with their edge sparklines. When the assistant draws a topology, deployment, instance map, API-dependency chain or cross-layer hierarchy, it now captures the whole graph the moment it reads it — every node and edge WITH its metric values and the per-edge trend series — and the chat block carries that snapshot. So reopening the conversation later re-renders the identical point-in-time map (each stamped “captured <time>”) without going back to OAP: click an edge and its sparkline part-graphs are right there, even if the window has rolled past or OAP is unreachable. The API-dependency chain also freezes which endpoint it drew, so a reload shows the same chain rather than whatever is busiest now. (Previously these views silently re-queried live on reload, quietly showing current data under an old question.) The assistant also now reads the real relationship metrics — a topology answer cites each dependency’s actual traffic, latency and success rate instead of just naming the neighbours.
• Embedded traces, logs and browser errors are captured evidence too — the list AND its detail replay offline on reload. When the assistant shows a service’s traces, Zipkin traces, logs or browser errors, it now freezes the list the moment it reads it and the chat block carries that snapshot — so reopening the conversation replays the exact same rows (each stamped “captured <time>”) with no fresh query, and clicking a row opens its detail (the span waterfall, the log record, the stack trace) straight from the frozen data, even offline. Native SkyWalking v2 traces and Zipkin freeze 30 with their spans inline; native v1 traces freeze 10 and pull each one’s spans up front so its waterfall still opens offline; logs and browser errors freeze up to 100 rows. (Previously these re-queried live on reload, showing current data under an old question.)
• Every captured block now wears a replay badge. Each frozen point-in-time block — figures, maps, the profiling flame, traces/logs/browser errors, and pod logs — carries a small replay icon next to its “captured <time>” stamp, so it’s unmistakable at a glance that you’re looking at a snapshot the assistant took, not live data. Because the assistant captures every answer’s data server-side and feeds the UI that snapshot, these blocks are always replaying — the badge shows from the first render, not just after a reload.
• A charted metric is the real dashboard widget — with its explanation, exact MQE and capture time. When the assistant shows a metric from a layer’s catalog it now renders that dashboard widget directly, carrying its full configuration (unit, number formatting, value thresholds and legends) instead of a rebuilt approximation — so the chart in the chat matches the dashboard exactly. Each figure documents what it is: the metric’s plain-language explanation, the precise MQE expression that produced it, and a “captured <time>” stamp. That last part matters on reload — a figure you reopen from history shows the frozen data from when it was read (a point-in-time snapshot, not a fresh query), and the capture time makes that unambiguous.
• Conversation history now persists per user in your browser, with controls. Past chats move to the browser’s IndexedDB — far larger than before, so long conversations with embedded charts survive — and are scoped to your username, so a shared browser keeps each person’s history separate. A Save history toggle (on by default) turns persistence off entirely; a usage meter shows how much of the client budget (default 500 MB, HORIZON_AI_HISTORY_MAX_MB) is in use; Clear all (with a confirm step) wipes it, and a single conversation still deletes from the History sidebar. History is stored unencrypted in the browser — a note says so — so turn it off on shared workstations; a more secure server-side history mode is planned.
• Two tabs can’t quietly overwrite each other’s chats. History is read once when the assistant opens, and each conversation is stored under its own id, so saving one never touches another. Before every save the assistant checks whether that same conversation was continued somewhere else — in a second tab — and if the two have diverged nothing is overwritten: the conversation is marked Not saved in the History sidebar and the chat asks which version you want to keep, this one or the other tab’s. A conversation whose turn is still in flight shows In progress until it’s written.

Profiling

• Continuous-profiling saves are now consistent under a failed confirmation, a mid-save edit, or a mid-save service switch. A save that succeeded but whose confirmation read failed no longer reseeds the editor from stale, pre-save data — the applied badge now stays honest rather than showing an old policy as current. An edit made while Apply is still in flight is no longer discarded by that same reseed, and neither is switching to a different service mid-save: the confirmation now always checks the exact service that was saved, not whatever the picker currently shows. The service picker’s labels and filter counts also refresh right after Apply, instead of only after a remount.

• A profiling request that cannot be honoured is refused, not repaired. pprof’s dumpPeriod is a sampling rate — lower means more samples — and an invalid value used to be replaced with 1, turning a bad request into the most expensive profile the agent can produce. Invalid values are now rejected with the reason.

• Async profiling accepts the full 15 minutes it offers. The New Task dialog has always listed a 15-minute option, but the task was quietly shortened to 10. The cap now matches what the dialog offers, and a request above any cap is refused with the limit named rather than silently trimmed — the same for target instances, where asking for more than a task accepts is an error, not a silent slice of the fleet.

• Network profiling no longer refuses a task OAP would accept. Create was disabled whenever no process had reported on the instance in the last 30 minutes, but OAP’s own check has no time limit — so an idle-but-healthy instance was blocked for a task the backend would have run. The process list is now advice: it warns, and lets you create.

• Profiling tasks that a continuous policy started are now visible. The eBPF and Network Profiling tabs — and the assistant’s profile reads — asked OAP only for tasks somebody started by hand, so a policy could fire and its profile appeared nowhere. Both kinds are now listed together, newest first.

• Continuous profiling now has a home: a per-layer Continuous Profiling tab. Every profiling surface until now started a task on demand. Continuous profiling is the opposite: arm a policy once and the task starts itself when a process crosses a threshold, with nobody present — which is how you catch the problem that only shows up at 3 a.m. It is eBPF profiling only and needs a Rover agent, which both evaluates the thresholds and runs the task; trace, async-profiler and pprof stay on-demand. Horizon could already show the results of auto-triggered tasks but had no way to see or change the rules behind them. A policy is one or more targets (ON_CPU, OFF_CPU, NETWORK), each with conditions built from a measurement (process CPU, thread count, system load, HTTP error rate, HTTP response time), a threshold, an evaluation period and how many matches must occur before it fires; the two HTTP measurements can be narrowed to a URI list or a URI regex. Inside each target, Where it runs lists the instances and processes OAP is actually evaluating and how often each has fired — the difference between a policy that is stored and one that is working. Nothing is gated on the agent being there: arming a policy before deploying Rover is a valid order of work. Note that saving replaces that service’s whole policy, since that is how OAP stores it, so keep every rule you want to survive. Reading needs profile:read; saving needs profile:enable.

• Profiling task creation is consistent and tells you upfront what it needs. Across all five task types (Trace / eBPF / Network / pprof / Async) the New Task button enables as soon as the basic entity is chosen and always carries a tooltip, and inside the create box a missing target is explained next to Create rather than by a silently greyed-out button — a hard requirement (no instances on the service) disables Create with the reason, while an advisory one (no rover-reported processes on a Network target) warns and lets you create, since OAP itself would accept the task.

• Network profiling picks its target instance in the create box and checks it has processes before you submit. The create modal selects the instance inline and lists that instance’s rover-monitored processes; if it has none, a warning explains the task will likely collect nothing — Create stays available, since OAP itself accepts a task on an idle-but-healthy instance.

• pprof and async-profiling tasks open a detail modal with their captured logs.

• K8S_SERVICE gains a Network Profiling tab. Kubernetes services, already observed by SkyWalking Rover’s eBPF probes, now expose Network Profiling — pick a pod and capture the process-to-process network conversations as a topology, the same capability the Mesh layer offers.

• Profiling create dialogs are clearer and harder to misuse. After a create the hint counts down to its single list refresh (refreshing in Ns), Escape closes the Async and pprof create dialogs, and Analyze stays disabled until at least one instance is selected.

• The Async result’s event-type picker shows only what the task captured. It lists just the JFR trees the selected task’s events produce (EXECUTION_SAMPLE, LOCK, OBJECT_ALLOCATION_*), dropping options like PROFILER_LIVE_OBJECT that no Horizon-created task can produce — so you can’t pick a type that renders an empty graph.

• A profiling task list that can’t scope to its service now says so, instead of looking empty. The Trace, eBPF, Network, async-profiler and pprof task reads returned an empty list when they couldn’t establish which service to ask about, which reads as “this service has never been profiled”. Each now reports the reason in the tab’s error line. The Network task read went further and dropped the service from the request entirely, answering with every service’s network-profiling tasks — it now refuses instead, while an instance-scoped read with no service stays a legitimate query.

Traces & logs

• Custom time ranges on the Traces and Logs tabs now return results when your browser and the OAP server sit in different timezones. A custom range (and the metric→trace drill’s centered window) was sent as a browser-local wall-clock string that the server re-read in its own timezone — so on a UTC-container deployment the window shifted by your UTC offset and came back empty, while the rolling presets kept working. Both paths now send absolute timestamps and the server applies the OAP timezone offset, consistent with every other query surface.

• Opening a Zipkin trace’s spans is now instant — no second query to OAP. On a Zipkin-tracing layer (mesh / Kubernetes), clicking a trace in the results list rendered its span waterfall by re-fetching the trace by id — even though Zipkin’s list response already contained every span. The list now carries those spans through, so the waterfall opens immediately from what’s already on screen, one round-trip lighter. Opening a trace by a pasted id or a shared deep link still fetches it directly.

• A one-service query can no longer widen into a whole-layer one. Traces, Logs and Browser errors are scoped to the service you picked, and when that scoping couldn’t be established the service filter was simply left out of the query. OAP reads a missing service as all services, so the tab filled with other services’ traces / log lines / JS errors under a page that still named yours. Those reads now come back empty, stating that the service identity didn’t arrive intact, and the Logs level/service facet counts refuse the same way instead of reporting the whole layer’s volume as this service’s. A request that deliberately carries no service at all is unchanged — it still returns every service in the window.

• A service-scoped tab no longer queries the whole layer in the moment before it knows which service you picked. The service is resolved after the tab opens — from the layer’s sampled rollup, or from the full service roster when the pick sits outside that sample (a deep link, a service in the long tail) — and a query started inside that window carried no service, which OAP answers with every service in the layer. Traces, Logs (its level facet counts included) and Browser errors now wait for their service: Run query is held until the name lands, and the panel states which kind of wait it is — Resolving service… while the lookup is in flight, or The selected service is not in this layer — pick another one to query. once the lookup has finished without finding it. That second case is a refusal now, not a silent whole-layer read.

• Switching service on the Traces tab clears the previous service’s results. The committed query survived the switch, so the trace list, the duration distribution and an open span waterfall stayed on screen under the new service’s name — read as its data — and a distribution pick left behind could filter the next query’s results down to nothing. A switch now resets the tab to Pick your conditions, then click Run query., dropping what the previous service returned, the way the Logs and Browser errors tabs already do.

• The Traces tab prints why a read failed, in the page. The reason — an unreachable OAP, or the refusal above naming the service and the layer — was only reachable by hovering the small unreachable marker on the results header. It now appears above the results, the way the Logs tab already reports a failed read.

• The service you picked is now queried by its exact identity, not by its display name. The pickers select a service by SkyWalking’s own service id, but several reads then threw that id away and sent the name they were showing, leaving the backend to look it up again. That round-trip can land somewhere else: SkyWalking holds an agent-detected service and a conjectured (virtual) one of the same name as two different entities, and a service name can equally read like another service’s id. Traces (the trace list and its own Instance / Endpoint pickers), Logs (its level/service facet counts included), Browser errors, the layer dashboards’ instance and endpoint pickers, the profiling tabs’ instance and endpoint pickers, the API-dependency endpoint list, the Pod-logs pod list and Explore’s instance/endpoint dropdowns all now carry the picked id end to end — exact, and one lookup lighter. A request that arrives with a name but no id is refused, rather than answered from whichever service that name happened to match first; the views the AI assistant embeds are seeded from the same roster row, so they carry both halves too.

• A service whose name looks like an internal id — api.1, orders.2026 — is queried correctly again. Service names ending in a dot and digits happen to match the shape of SkyWalking’s own service ids, and several reads guessed by that shape: the name was forwarded as if it were an id, matched nothing, and the tab came back empty for a service sitting right there in the picker. Traces, Logs, Browser errors, every profiling task list, and the Instance and Endpoint pickers now carry the id and the name as two separate fields, so nothing is inferred from how a name is spelled.

Result counts and paging

• A last page that happens to be exactly full no longer offers a Next that lands on an empty screen. The Logs and Browser Logs pagers decided whether another page existed by looking at how many rows came back: a page that filled the page size was assumed to have more behind it. When the window’s row count happened to be a multiple of the page size — a window holding 24 rows read 12 at a time — the final page was full, Next stayed enabled, and clicking it showed nothing with no explanation. Horizon now asks whether another page exists rather than inferring it, so Next is offered only when there is a page with rows on it, on both tabs.

• The Logs pager stopped claiming a total it never had. The footer read page 3 · showing 50 of 50 total — where “total” was the count of rows on the page you were looking at, so it always equalled what was shown and told you nothing. OAP’s log query does not report a cross-page total (no list query in the query protocol does), so the footer now states the page and what is on it, and the honest “there is more” signal lives on the Next button.

• Changing the Logs page size restarts at page 1. Page size feeds the query’s offset, so sitting on page 5 at 20 rows and switching to 100 jumped from row 80 to row 400 — usually past the end of the window, leaving a blank page with nothing to explain it. The Browser Logs tab already did this; the Logs tab now matches.

• “Truncated” and “capped” notices are now facts instead of guesses at the boundary. A result set whose size landed exactly on the cap was reported as truncated when it was complete: the topbar alarm badge showed 200+ for exactly 200 events, the alarms page warned that the window might be clipped when it wasn’t, the events popout offered “more available — narrow the range” for a window that held nothing more, and the Logs level counts couldn’t say whether their sample covered the window. All of these now reflect what the backend actually holds, and the alarms notice says plainly that there are more alarms than were fetched rather than restating a count.

• Trace, log and Explore result lists say when they were capped — a signal they never had. A Traces, Zipkin-traces or Explore/Inspect result that exactly filled the Limit you chose was indistinguishable from a complete one: there was no total, no indicator, nothing. Those lists now carry a capped at N — narrow the window note when the backend held more than the limit returned. Same for the endpoint search behind the Endpoint / Page pickers, which now says more matches — narrow the keyword instead of silently cutting the list.

• A next-page read no longer risks skipping rows. OAP derives a query’s offset from its page size, so asking for “one row more” past the first page shifts the offset too and silently pages in the wrong stride. Horizon asks for the extra row only where that is exact, and everywhere else reads the next page’s first row as a separate one-row probe carried in the same request against the same time window — so the page you see and the answer about what follows it always describe the same window.

Deployment & configuration

• Serving Horizon under a path prefix is a first-class build option now, and the docs say how. Set HORIZON_UI_BASE=/horizon/ when building from source and the prefix is baked into the artifact — page URLs, static assets and every API call resolve under it — for a reverse proxy that strips the same prefix before forwarding. Server Listener documents both halves and what each one looks like when only one of them is in place. Previously this needed a hand-passed build flag and the setup docs stated, wrongly, that no prefix support existed at all.

• The runtime-rule dump download works under a path prefix. Downloading a runtime-rule dump (all catalogs or one) was the single link in the app that ignored the deploy prefix: it aimed at the site root, so behind a gateway serving Horizon at /horizon/ it hit whatever else owns /api there — usually a 404 instead of the archive. It now resolves against the same base as the rest of the UI.

• Running against an OAP whose template store Horizon cannot read? templates.mode: readonly is required. In the default live mode Horizon renders only what is published to OAP, and an unreachable template store is a deliberate block: layer-driven pages stay empty behind the connectivity banner instead of quietly rendering a bundled copy that may not be what you published. The most visible casualty is Traces, which reads its query configuration from the layer template. That is the permanent state on an OAP 10.x — it has no /ui-management/templates* endpoint at all (template management lives on the query port’s GraphQL there, which Horizon does not speak), so the store never answers. readonly is the supported answer: it renders every dashboard from the templates bundled in the release, stops probing an endpoint that will never reply, and makes the configuration surface honestly display-only.

• Run on the bundled templates, read-only — no OAP ui_template API needed. A new templates.mode setting (HORIZON_TEMPLATES_MODE) adds a readonly mode: Horizon renders every dashboard / overview / alert-page / 3D-map / translation from the local bundle and never calls OAP’s ui_template admin API. The whole config surface goes read-only — the admin pages still open and show the bundled config, but editing and publishing are disabled (and the BFF rejects a write even if it’s fired directly). OAP’s query API is still used and health-checked, so metrics / traces / logs / topology work exactly as before; only the config-template store is local. Default stays live (seed-to-OAP, editable). The Cluster Status page shows the active mode and ui_template availability.

• The container image runs with environment variables only — no mounted config file. There is now one committed, env-driven horizon.yaml (the former horizon.example.yaml / local-copy split is gone): every field is a ${HORIZON_…:default} token, and the image bakes that same file. So docker run -e HORIZON_OAP_QUERY_URL=… -e HORIZON_AUTH_LOCAL_USERS='[…]' … is enough — no -v mount, no repackaging. Previously oap.*, auth.*, users, LDAP, RBAC, and performance tuning were YAML-only. Lists and secrets (users, LDAP, OAP auth) are set as single-line JSON-string env vars; precedence is env > file > built-in default. The config file itself is the complete, self-documenting env-var reference, mirrored in the container-image docs. Mounting your own horizon.yaml still works and overrides the baked one.

• Per-layer and alarm-page setup live only in OAP now — two local state files retired. Horizon no longer writes horizon-setup.json or horizon-alarms.json: the setup.file / HORIZON_SETUP_FILE config is gone (the per-layer override it backed was unused — per-layer config is authored in the layer-dashboards template), and the alarm-page setup (pinned layers, default window, overview fetch cap) now reads and writes only OAP’s horizon.alert.page-setup template, exactly like every other template. One less pair of files to mount for durability.

• Audit logging can be turned off. audit.enabled (env HORIZON_AUDIT_ENABLED, default true) — the audit trail stays on by default (it is a security record) but can be disabled entirely, mirroring debugLog’s enabled + file shape.

• Cluster Status now reports admin-feature reachability, not just config-presence. The admin-host pane fires a safe GET at the real REST path each feature calls on OAP — dashboard templates → /ui-management/templates, DSL management → /runtime/rule/list, live debugger → /dsl-debugging/status, Inspect → /inspect/metrics — and colors each row by whether that path actually responds. A feature whose module is loaded but whose endpoint 404s (a renamed or forked module, a selector that’s on-but-broken) now reads unreachable instead of a misleading green; the config-dump selector check is kept only as an informational “selector detected” footnote. Dashboard templates (ui_template) join the same table as a feature — shown as “readonly · bundled” when running in readonly mode — each row shows how long ago it was last checked, and the page can force a fresh re-check on demand.

• An invalid horizon.yaml edit at runtime is rejected out loud, not silently ignored. horizon.yaml is hot-reloaded, but a reload that failed validation used to be swallowed with no trace — the file said one thing while the server kept running another. A rejected reload is now logged with the exact field path(s) that failed (the same per-field shape a boot failure prints), and the previous valid config keeps serving until the file is fixed.

• The role editor no longer offers two permissions that did nothing. setup:write and alarm-setup:write appeared as Edit per-layer setup and Edit alarm-page setup in the role editor, but no request ever checked them — publishing a Global-defaults or Alert-page-setup change goes through the same template-write path as every other template and is gated by overview:write. Withholding them fenced nothing off, so they are gone; the two read verbs stay (they decide whether the sidebar entry appears) and are now labelled with the pages they actually govern.

• The release process now gates the exact commit it tags, and stable images wait for the vote. The release script pushed the tag before it installed, packaged or license-checked anything, ran no tests / type-check / lint / i18n at all, and validated the caller’s working tree while tagging a freshly cloned one — so it could green-light one tree and release another. Repository CI never covered the gap either: it runs on pull requests and pushes to main, never on a tag. The full battery (install, type-check, both builds, unit tests, lint with the i18n gates, license headers, packaging, and the binary LICENSE/NOTICE check) now runs inside the release clone on the release commit itself, before the tag is created. And because a tag is only a release candidate until the Apache vote passes, pushing it no longer publishes the :<version>, :<major.minor> and :latest images — the tag build publishes the immutable digest tag only, and those human-facing tags (plus the Docker Hub mirror) are attached by an explicit promotion run after the vote.

• A release is verified before it is published, and a half-finished finalize can be re-run. Finalizing a passed vote moved the candidate into the official dist/release location FIRST and only downloaded it afterwards, and it never checked the .asc detached signatures at all — so a candidate that was corrupt, truncated or signed by the wrong key became the official Apache release before anything looked at it. The candidate is now fetched and verified before a single byte is promoted: the .sha512 checksums, then both detached signatures against a throwaway keyring built only from the project’s published KEYS file — which is what makes “signed by a SkyWalking release key” a property that is actually checked rather than assumed — plus an @apache.org identity on that key and the requirement that both artifacts carry the same signer. Any failure stops the run with nothing published. The run is also resumable now: an interrupted finalize used to be impossible to retry, because the candidate it looked for had already been moved out of the staging area, and a GitHub release left half-uploaded stayed half-uploaded. A re-run picks the candidate up from wherever it actually is, skips what already landed, and checks the GitHub release asset by asset against the bytes it just verified — size first, then sha512 on the attached copy — because a file sitting under the expected NAME is no evidence that its content is the voted one. A truncated upload from an interrupted run, or a leftover from an earlier candidate, is replaced rather than trusted, and the run prints exactly which assets it replaced and why.

• The key a release is signed with is the key the script showed you. On a workstation holding more than one GPG secret key the release script could report one key, validate a different key’s @apache.org address, and then hand the signing over to whichever key GPG picks by default — three keys, no guarantee any two were the same. It now resolves exactly one signing key (HORIZON_RELEASE_GPG_KEY, else git config user.signingkey, else the sole key in the keyring) and refuses to guess when that choice is ambiguous, checks the @apache.org identity on that key alone, pins it explicitly on every signature, and re-reads each .asc afterwards to confirm the signature really came from it. The generated vote email now carries the signer’s full fingerprint so voters can check it against KEYS.

• The official release directory holds exactly what was voted on — nothing rides along. Promotion verifies six artifacts but moves the whole candidate directory in one step, so anything else left in it (a stale tarball from an earlier candidate, a hand-uploaded file) would have landed in the official Apache release location without ever being verified or voted on. The candidate’s contents are now compared against the six expected artifacts before anything is published; an unexpected entry stops the run, lists every offender with its full URL and the exact svn rm command to remove it, and publishes nothing.

• Cleaning up the superseded release can no longer delete a newer one. After promoting, finalize offers to remove the release this one supersedes — ASF keeps only the current release live and older ones fall to the archive — but it picked the highest OTHER version on the release path, so finalizing an older maintenance line (1.0.1 while 1.2.0 was already out) proposed deleting the NEWER release. Only versions below the one being finalized are considered now; any newer release is reported as deliberately untouched, and the prompt states the exact SVN path being deleted, what it lists, what is kept, and where the deleted version stays downloadable on archive.apache.org.

• The CHANGELOG is validated on the commit being tagged, and the published notes are read from the tag. The release section check runs inside the release clone with the rest of the pre-tag battery, so a section that exists only in the release manager’s local checkout — or one still carrying the (In development …) stub on the branch being released — fails the release instead of passing it while a different tree gets tagged. The GitHub release body finalize publishes is likewise extracted from the CHANGELOG at the tag rather than from the local checkout, so the notes always describe the bytes that were actually released. LICENSE / NOTICE / HEADER are asserted on that same commit for the same reason.

• Finalize expects the moving latest image tag only when the release really is the newest. The promotion run deliberately leaves apache/skywalking-ui:latest where it is when the promoted tag is not the highest, so that publishing a patch on a superseded line never drags latest backwards — but finalize announced latest as expected for every release. It now applies the same rule: latest is checked only when the release is the highest tag, and then it is also confirmed to point at the same image digest as horizon-<version>, rather than merely existing.

• Dependencies refreshed — the shipped image and tarball now carry no known-vulnerable package. The bundled runtime tree was upgraded (notably the static-file server the BFF serves the UI with, and the charting library), clearing every advisory pnpm audit reported against the production dependency tree, and the binary LICENSE/NOTICE were regenerated to match.

• The production default log level is now warn (was error). The misconfiguration and security warnings Horizon emits — break-glass logins, LDAP failures, rejected config reloads — were discarded by the old default, so none of them reached stdout unless the operator had already raised LOG_LEVEL. They now show by default; set LOG_LEVEL=error to restore the old quietness.

• The debugLog wire log is implemented — enabling it now actually captures OAP traffic. The config block was documented (see the wire debug log docs) but nothing wrote the file; with debugLog.enabled: true every outbound OAP request/response pair (GraphQL, admin REST, Zipkin) is now appended to debugLog.file as one JSON line — Authorization headers redacted by default, bodies truncated at maxBodyChars — and every setting, including on/off and the file path, hot-reloads without a restart. LLM-provider traffic is deliberately excluded so the wire log can never see the AI API key.

• A duplicated template row on OAP is now reported, never resolved behind your back. When the same dashboard / overview / alert-page template ends up on two OAP rows (two row ids for one template name — residue of an OAP release that minted a random id per create), earlier versions quietly disabled one of them at every BFF start, and picked the untouched bundled copy over the one you had edited. Retiring a row does not bring its content back — OAP has no delete, only disable, and Reactivate restores the bundled default rather than the copy you retired — and doing it unattended at boot meant a rolling upgrade could retire a dashboard nobody agreed to lose. Horizon now only detects: it draws the lowest-id copy, names the affected templates on the Dashboard-templates banner and in the log, and changes nothing. Deciding which copy survives is a deliberate OAP-side cleanup, not something a restart or a button does for you. A disable that stalls waiting for OAP to confirm it also now says so, instead of reporting itself as a stalled update.

• A duplicated template is now impossible to miss — and impossible to open by accident. A layer or overview dashboard whose template name sits on more than one enabled OAP record, where the copies carry different content, is hidden from the sidebar rather than rendered from whichever copy happened to win, so nobody navigates to a dashboard whose definition is ambiguous (the server log names every duplicate it finds, and the hidden layers specifically). Opening such a layer by URL — a bookmark, a reload, a shared link — now says the dashboard is duplicated and, for operators who may edit templates, links straight to Dashboard setup → Layer dashboards, instead of claiming the layer is inactive or unknown. Records that are byte-identical are not hidden: either copy renders the same dashboard, so the page keeps working, and only the cleanup is outstanding. The template shows a Duplicate marker in the admin browse list, and selecting it raises a banner naming the record ids and stating that Horizon renders the lowest-id copy and changes nothing on its own. Deleting a duplicated template is refused with the same detail rather than half-applied — it would otherwise disable only the copy on screen and leave its twin rendering. If the template status cannot be read at all, every layer stays visible: hiding requires a positive signal.

• Publishing a layer dashboard now refuses malformed content before anything reaches OAP. Overview and 3D-map templates were already checked on the way out; layer dashboards were not, so a hand-edited or imported template could be published and break that layer for everyone until someone pushed a good copy over it. Publishing — Check diff & push, Reset to bundled, or a bulk sync — now names the offending field (an unknown key, a misspelled component flag or dashboard scope, a widget the grid cannot run, a service-list column the service list cannot query) and writes nothing; in a bulk sync the other templates still go through and the rejected one is listed. Half-authored work still publishes normally, including widgets and metric rows whose MQE is still empty, and your local draft is unaffected either way — it never leaves the browser until you push it.

General Service — PHP runtime (PHM)

• Six instance dashboard line widgets for PHP Health Metrics — process CPU utilization, memory used/peak, virtual memory, thread count, and open file descriptors (meter_instance_php_*). Each line widget uses visibleWhen so widgets render only when the PHP agent reports PHM data (Linux /proc sampling of the parent PHP process via getppid()).

General Service — Node.js runtime

• Six instance dashboard line widgets for Node.js runtime metrics — process CPU, V8 heap used/total/limit, RSS, and external memory (meter_instance_nodejs_*). Each line widget uses visibleWhen so widgets render only when the Node.js agent reports runtime data.

General Service — metric-to-trace drill-down

• Removed the deprecated “Layer-scoped” widget toggle from the dashboard editor. It emitted OAP’s scope: All entity — deprecated in the query-protocol since OAP 9.4.0 — and no bundled dashboard used it. Any legacy stored dashboard that still carries the flag now renders at normal service scope (the unknown key is ignored). Layer-wide rollups belong on the Overview dashboards.

• Click a latency or error point on a General-service chart to open the matching traces. Drill-capable line widgets — Avg Response Time, Response Time Percentile, Error Rate, Apdex, Success Rate, and MQ consume latency, across the Service, Instance, and Endpoint scopes — carry a traces flag in their header, and their datapoints are clickable. Clicking a point opens the native Traces list in a new browser tab, pre-filtered to that service (and the selected instance / endpoint) and centered on the clicked bucket’s time window. A latency widget opens slowest-first with a minimum trace duration of the clicked value; an error-rate / success-rate widget opens with the trace status set to Error. Throughput (Traffic) and runtime/resource widgets offer no drill — only latency and success/error metrics have a natural trace criterion. Dashboard authors enable the drill per line widget with a new Trace drill option in the widget editor (none / latency / error). It’s config-driven and not bound to a specific layer — it works on any dashboard whose layer has native traces (the Traces component on in native or both mode), with the bundled General service widgets shipped enabled.

Access control & permissions

• The Roles & Permissions board no longer offers permissions that do nothing. Four entries on it — “Edit alarm rules”, “Add / remove local users”, “Change role grants”, “Read the audit log” — read as capabilities you could delegate, but nothing in Horizon checks them: alarm rules are read-only upstream (they change in OAP’s YAML), the user list and the role definitions are edited in horizon.yaml, and the audit trail is a file you ship to your SIEM rather than something Horizon serves. All four are now marked Reserved on the board, with a line saying that granting them has no effect, and the built-in operator role no longer grants alarm-rule:write — a grant that bought nothing today and would have handed every operator an alarm-rule write the day one existed. Custom roles that name a reserved verb keep working and still show their check mark on the board, so you can see the grant you wrote and that it buys nothing. The RBAC documentation now marks the same four, and the corrections run the other way too: the board’s five previously undescribed permissions (browser error logs, source maps, the AI assistant, data retention, OAP configuration) are grouped and named, and the reference table now says what rule:write:structural and the live-debug verbs really gate.

• The Live Debugger is live-debug:read / live-debug:write and nothing else — the rule:debug permission is removed. Every Live Debugger call except the per-node status read demanded rule:debug on top of its own permission: watching a capture was granted by live-debug:read and then refused unless you also held rule:debug, so the read-only role the documentation describes could not open the page. Starting and stopping a capture required live-debug:write and rule:debug together — the same pair for both, so it never separated the two. Downloading a runtime-rule dump asked for rule:debug outright, while the button, the handler and the docs all said rule:read, which showed the download as enabled and then answered 403. rule:debug is now gone from the permission vocabulary, the built-in operator role, the Roles board and the documentation: watching is live-debug:read, starting and stopping are live-debug:write, and the dump is the read it always was, rule:read.

• Read this if you hand-wrote a role that reached the Live Debugger through rule:debug — including by granting the rule:* wildcard. rule:* matched rule:debug, so a role holding the wildcard held it too. Removing a permission from a two-permission requirement loosens what remains: a role granted live-debug:write without rule:debug could previously neither start nor stop a capture, and now can do both. Conversely a role that reached the Live Debugger only through rule:debug or rule:*, without any live-debug: grant, loses that reach entirely — rule:* no longer touches the Live Debugger at all. There is no start-versus-stop distinction to fall back on: a role that should not run captures holds live-debug:read without live-debug:write. The four built-in roles are unaffected — operator was the only one holding either permission and it held both. A horizon.yaml that still names rule:debug keeps loading with no warning; the grant is inert and the row simply disappears from the Roles board.

• The 3D infrastructure map entry is hidden from roles that cannot open it. The topbar entry to the map showed for everyone, but the map’s configuration and traffic reads require infra-3d:read — so a role without it followed the entry to a page that could only tell it that it lacked access. The entry is now removed for those roles, and the Roles board’s menu-visibility matrix — which already claimed the entry was hidden — is now telling the truth. The map’s own route still loads for anyone who reaches it by URL; the page reports the missing access, as it did before.

• The Roles board’s menu-visibility matrix lists every navigation entry — four were missing. The matrix presented itself as exactly what each role sees in the menu, but Trace inspect, Log inspect, Translations and the 3D Infra Map setup page had no row, so a role planned from the table reached pages the table never mentioned. All four now have one. The sub-entries under DSL management are covered by its row and need the same permission it does; the row for the 3D infrastructure map itself stays, now beside a separate row for its setup page. The matrix reads as every navigation entry rather than the sidebar alone — the map is reached from the topbar — and the rows now follow menu order.

• What one operator’s session read no longer reaches the next person to sign in on that browser tab. Horizon caches fetched responses so a page you return to paints instantly — but that cache outlived the session: after a sign-out and a fresh sign-in in the same tab, the new user could be served the previous user’s already-fetched data (service lists, alarms, traces, dashboards), and the reads Horizon deliberately holds for the life of a page (alarms, events, the layer service list) would have handed theirs to the next session without so much as a refetch. Every identity change — signing out, signing in, and a session the server ends mid-use (an expired or revoked login, which bounces you to the login page) — now discards every cached response instead of merely marking it old, and a read that was still in flight when it happened can no longer paint: where the cache owns that read it is cancelled outright, and everywhere else its answer is thrown away on arrival instead of being written back. Dropped with it: the dashboard-configuration preload, the debug ticker’s event history, the assistant’s on-screen conversation, the layer picker’s service / instance / endpoint selection (including a locked comparison set), and what the 3D infrastructure map carries between sessions — which services are alarming, the per-cube traffic figures, the map’s own configuration, and its loading timeline. Because the 3D map re-reads its configuration instead of reusing the previous session’s copy, an account without 3D map access is now told so rather than being shown the map the last person left on screen. A page you already had open at the moment the session ended keeps whatever it had finished drawing until you navigate or reload — there is nothing behind it any more, so the next read goes to the server. Your saved AI history is per-user on disk and untouched — signing back in re-reads yours.

• Shortening the session timeout now actually shortens the sessions. session.ttlMinutes hot-reloads, but only half of it was applying: the cookie handed to the browser carried the new timeout while the server went on honouring session ids for whatever timeout was in force when the BFF started. Tightening the window — the thing you do right after an incident — was the damaging direction, because it looked like it had worked: the browser was told to forget the cookie sooner, while anyone still holding the session id (a copied cookie, a script, an API client) could keep using it for the full original window, and the background sweeper kept the old clock too. Lengthening it failed the other way — the cookie promised the longer window but the server signed you out at the old, shorter one. Both halves now read the live value, and a change applies to sessions that are already signed in, not just to new logins, so a shortened timeout reaches the sessions you are actually worried about. Sessions stay sliding either way: the window is measured from each session’s last request.

• A custom role no longer loses the org theme, the default time window and the alarm badge on sign-in. The three org-wide settings an admin publishes on Global defaults and Alert page setup — the default theme, the default time window for dashboards and overviews, and the alert page setup that also sizes the topbar alarm badge and the overview “Active alarms” widget — were read through the template-administration endpoint, which requires overview:read and returns every dashboard template with both its OAP-stored and its shipped copy. A role that was not granted overview:read was refused all three at sign-in and silently fell back to defaults, and reading a single theme id handed the caller the whole template store along with it. These settings now load from their own read that needs nothing beyond a signed-in session and returns only those three values. The four bundled roles are unaffected — they all carry overview:read — and template administration keeps its existing permission.

• When OAP’s template store can’t be read, org settings fall back to Horizon’s built-in defaults instead of the files shipped in the image. In the default live template mode the copy stored on OAP is the only source of truth for configuration, but if the template store was unreachable — or a setting’s row was missing or had been disabled — Horizon quietly rendered the theme, time window and alert page setup that ship inside the image as though they were your organization’s published settings. They now resolve to Horizon’s built-in defaults, which is what these pages already document, so an unreadable store can no longer present shipped files as your configuration. In templates.mode: readonly the shipped files remain the declared source and are served exactly as before.

Alarms

• The alarm timeline reads more clearly — a clearer selection band and legend, and the detail sidebar reflows cleanly on narrow windows. Hovering the timeline now hints both affordances — click a minute to filter, or drag across the timeline to select a range — so range-selection is no longer hidden.

• Filtering alarms by a virtual service now finds them. The databases, caches, message queues and gateways SkyWalking infers from an agent’s calls (VIRTUAL_DATABASE, VIRTUAL_CACHE, VIRTUAL_MQ, VIRTUAL_GATEWAY, …) are conjectural services, and OAP distinguishes them from agent-reporting ones by a flag that is part of the service’s identity. The alarms filter dropped that flag and always asked as if the service were agent-reporting, so picking a virtual service — on its own or narrowed further to an instance or endpoint — filtered against an identity that isn’t its own, and the page came back empty however many alarms had fired. Horizon now carries the flag that came with the service you picked, exactly as the layer’s service list reported it, so the filter asks for the entity you chose. The dropdown is unchanged — you still just pick a service — and filtering a normal service behaves exactly as before.

Events

• A per-service events popout on the service banner. Every layer drill-down’s service banner gains an Events button that opens a modal for that one service’s lifecycle events — agent restarts, Kubernetes events, and other point-in-time records from OAP queryEvents — without leaving the page you’re on. The service is fixed, so the view is a swimlane of instance × time: one row per service instance in its own color, each event a bar on a time axis (an event with no end time is an instant marker), Error events ringed red. Overlapping events on one instance stack into sub-lanes; the time axis marks the date at day boundaries; the popout owns its own window (6h / 1d / 2d plus a custom range up to 7 days) and queries at second precision.

• Built for scale and honest about limits. A rolling restart of a large service is one bar per instance stacked at the same moment — the granularity is the point, and a search box filters the instance rows by name for services that run hundreds of them. Scrolling is fully internal (sticky time-axis header + sticky instance column, horizontal scroll for long ranges, opened scrolled to the newest events). The newest events are fetched up to a configurable cap (200 by default); the popout shows “newest N · all in range shown” or, when the window holds more, “more available — narrow the range”. Clicking a bar opens a detail panel with the instance, Started / Ended / Duration, message, and reported parameters. The button is permission-gated; viewer / maintainer / operator roles gain events:read. Events are lifecycle facts, not alerts — for threshold breaches, the Alarms page is unchanged.

User experience

• The runtime-rule page is a dump, and now says only that. Under DSL management it was titled Dump & restore and carried a dimmed second panel headed restore, tagged “later release”, promising an upload affordance once an upstream API shipped. There is no restore and none is planned — reverting a rule to its bundled version is the reset Horizon offers — so the placeholder panel is gone and the page and its sidebar entry are now Runtime-rule dump. Exporting the rules, all catalogs at once or one at a time, is unchanged.

• Cluster Status describes the admin-server row in your language, under the product’s name. The row’s “what this affects” line named the project by an internal working name and had drifted out of the translation catalogs, so every locale silently fell back to English. It now reads correctly in all 8 locales, and no longer claims a fixed count of dependent modules.

• The whole console now speaks your language. Large surfaces that previously rendered English in every locale — the per-layer sidebar sub-navigation, the Logs tab filter bar, all five profiling tabs and their task dialogs, the admin template editors (layer dashboards, overview, topology, deployment, 3D map, global defaults), the 3D Infrastructure Map, the service-map detail panels, and the remaining topbar chips — are now fully wired into the translation catalogs and ship translated in all 8 locales, alongside a sweep that restored every stranded catalog key.

• The page’s <html lang> attribute now follows the active locale — screen readers, spell checkers, and translation tooling see the language you actually picked instead of a hardcoded en.

• Escape closes any dismissible panel — modals, row popouts, and the topology focus / node-filter dropdowns all dismiss on Esc.

• Switching service clears the dependent filters (log level / tags, browser-error category) back to a clean state, so a stale filter never silently hides the new service’s data.

• Denser Kubernetes dashboard tables — the K8s layer’s table widgets show more rows without scrolling.

• Live debugger reads cleanly on tall and wide results. The LAL pipeline matrix’s frozen first column now stays pinned when you scroll the grid sideways (it used to drift off with the rest of the matrix), clicking a source line flashes the whole matching step row — not just its label — and the MAL / LAL / OAL debugger pages now scroll as one page for tall captures instead of trapping the result in a fixed-height inner box.

• The LAL pipeline matrix renders Envoy access-log (ALS) and any non-generic log format. Each cell now shows whatever fields OAP serialized for the record rather than a fixed LogData subset — so an EnvoyAccessLogBuilder snapshot displays its service / endpoint / response data where it used to render blank, a record whose raw proto input OAP couldn’t serialize surfaces the reason (jsonformat-failed …) instead of an empty cell, and each cell names its payload class. The free-text search and the cell popout follow the same format-agnostic rendering.

• The LAL matrix gains per-row filtering and is correct across cluster nodes. A filter on each step row — shown only when that row has gaps — narrows the grid to the records that actually produced data for that step (e.g. just the records that emitted output), and the row counts now reflect the whole capture rather than the visible page. Each OAP node’s matrix filters and column-pins independently, so acting on one node’s grid no longer changes another’s. Oversized cells are height-capped so one huge record can’t blow out the grid, and statement-mode step labels read function @7 rather than a raw template.

• A LAL step that dropped a record now says why. When a pipeline step stops a record, the matrix cell and its popout show OAP’s human-readable drop reason — a parse exception, a non-matching regexp, an input-type mismatch — instead of leaving you to reconstruct the cause from the payload. An OAP version that predates the reason field simply shows no reason, as before.

• Inspect a LAL cell’s full data and diff pipeline stages. Every cell carries a persistent button — VIEW on the input row, DIFF on the builder rows — that opens the complete payload in a syntax-highlighted JSON viewer with the nested log content inlined as real JSON. For the builder snapshots a compare picker renders the captured DSL itself, with per-statement steps marked on their line and the extractor / sink block snapshots drawn as selectable ranges; picking one shows a side-by-side diff, so you can see exactly what a statement or stage changed in the built log.

• A failed live-debug or OAL request now names OAP’s own error code. Those endpoints report a failure as a code plus a message — rule_not_found when no node has the rule loaded, too_many_sessions when every node is at its capture ceiling, injection_disabled when debug capture is switched off — but Horizon read the field the rule-apply endpoints use instead, so the code was dropped and the failure rendered as undefined: <message> in the server log and in the per-node live-debug status. The code now leads the message.

Dashboards

• A dashboard can no longer be published under a name that isn’t what it holds. Pushing a layer dashboard to a name the runtime never reads — horizon.layer.general instead of horizon.layer.GENERAL, or an OAP legacy alias like horizon.layer.CACHE where the layer is read as VIRTUAL_CACHE — used to succeed and then change nothing on screen, with no explanation anywhere: the record existed but no page ever asked for it. Publishing an overview dashboard under one id while its content declared another used to succeed too, filing it under the wrong identity: it appeared in the dashboard list as the other dashboard — two entries carrying one name — while opening it by the name it was published under rendered that other dashboard. Both are now refused before anything is written, naming the one name that template is read under. Records that already ended up in that state — written by an older Horizon, by another tool, or by hand — are listed on the affected admin page’s banner with their OAP record ids, and nothing is served from them any more: no sidebar entry, no overview in the dashboard menu, no layer dashboard. They stay reachable in the admin editor, deliberately — a record stored under a name that page addresses still opens as that name’s live copy, because repairing it means pushing corrected content over that same record. That matters most for the second shape, a record stored under a name Horizon does read whose content names a different dashboard or layer — until now it was served under the name it sat under, so a layer could quietly show another layer’s dashboard. A layer whose only record is unreadable falls back to Horizon’s built-in defaults, exactly as a layer with no published template does.

• Overview dashboards are checked against the same bar as everything else when you publish them. The push previously ran a weaker check than the one the shipped dashboards pass in CI, so a misspelled field, a widget type the renderer has no case for, or an out-of-range width was stored as dead config and simply never rendered. Half-authored work still publishes — a blank MQE, an empty KPI list, a cleared title, a brand-new dashboard with no widgets yet are all normal states of an unfinished draft, and none of them are refused.

• A refused push now tells you why on the layer and overview editors. The reason (which field, or the name to publish under) was already in the response but the pages showed only “failed (400)”.

• Cards can render values as colored status chips. A card widget with format: enum now takes an optional chip color per value-map entry — ok (green), warn (amber), err (red), info (blue), neutral (grey) — and renders each matched value, or metric label, as a colored chip instead of a bare number. Set it in the layer-dashboard admin’s value-map editor, next to the existing value → label mapping.

• The Kubernetes Node Status card now reads as a status, not a number. Instead of a raw 1, it shows the node’s active conditions as colored chips — Ready in green, the *Pressure / NetworkUnavailable conditions in amber/red — so node health is legible at a glance.

• The Kubernetes Node dashboard gains a Pod Total card. A compact card now sits directly under Node Status showing the current count of pods scheduled on the selected node (all phases) — the latest value of the same metric the “Pods on Node” trend already charts — so the space beside the status card is no longer blank.

• Satellite event and queue widgets break out per pipeline. The SO11Y_SATELLITE Receive Events, Fetch Events, Queue Input / Output, and Queue Used widgets now label each series by its Satellite pipeline (tracingpipe, jvmpipe, logpipe, …) instead of collapsing every line onto a single all, so you can see which collection pipeline drives the rate.

• HOUR-step line charts no longer collide their x-axis labels. An hourly label carries the date (07-02 05:00), so on a narrow widget several wide labels overlapped into an unreadable smear; overlapping labels are now hidden (the label density target is unchanged — only the colliding ones drop), so any dashboard viewed at HOUR step keeps a legible axis.

Overview dashboards

• The per-layer KPI tiles now report the whole layer, not a single service. A tile such as “General services · RPM / Latency / SLA” was aggregating just one service (an internal top-1 cap), so a busy layer read far too low. Each tile now rolls up every service in its layer — throughput summed, latency and SLA averaged — for General, Mesh, and the Virtual Database / Cache / MQ / GenAI layers. Mesh’s second tile is now Latency (average response time) in place of the old P95.
• New per-widget aggregation control for Overview composites. An Overview data widget defaults to a self-aggregating expression that OAP rolls up server-side; a widget can instead pick page-side aggregation (an Aggregation mode choice in the Overview templates admin, with a top-limit services window) for metrics that can’t be expressed that way — the Kubernetes cluster-capacity and Istio pilot composites use it (pilot over up to 5 control-plane services). A page-side widget also chooses how those top services are ranked — by one of its KPIs (default the first) or by a separate ranking metric — so a labeled first metric no longer skews the ranking. The self-aggregating window is bounded by a new HORIZON_QUERY_OVERVIEW_TOPN setting (default 100).

Performance & behavior tuning

• New performance section in horizon.yaml. Tune how hard the BFF fans metric queries out to OAP — per-route bulk (request) sizes and concurrency for the topology, 3D-map, landing, and dashboard fan-outs — plus protective caps: the service-map render valve (topologyMaxNodes / topologyMaxEdges) and per-request record caps for traces / logs / browser logs. Operational, hot-reloaded, per-deployment; defaults match the previous built-in values, so the whole block is optional. Raise it for a beefy OAP + storage backend, lower it to protect a modest deployment; every value clamps to a hard ceiling.
• 3D-map fan-out tuning moved out of the dashboard template into horizon.yaml (performance.bulk.infra3d). These metric concurrency / batch knobs were operational settings misplaced in a published-to-OAP dashboard template (not even surfaced in the admin editor); a stale template still carrying the old pipeline block is accepted and stripped on save, so a 3D config that was synced before the move converges back to synced after one re-push (instead of showing diverged forever).
• Unified page-size pickers across the event lists. Traces, Logs, and Browser Logs share a 20 / 30 / 50 / 100 page-size dropdown — and Browser Logs gains a picker it never had (it had a fixed 100). Each picker’s max matches the server-side fetch cap in performance.limits.maxPageSize.
• Node memory sizing guidance. The container image now sets a default NODE_OPTIONS=--max-old-space-size, and the docs cover sizing the Node heap to your container memory limit and the in-memory source-map budget.

Trace explorer

• Zipkin traces now render with the full native trace experience. The Zipkin trace detail and popout gained the KPI strip, service legend, the duration-distribution scatter (drag to filter, click to open), time-positioned waterfall bars (service · operation inside the bar, kind/status affordances), and a centered span-detail modal — matching the SkyWalking-native trace view. Zipkin annotation codes (cr / cs / sr / ss …) show an inline plain-language hint.
• Shareable trace links are unified. Native and Zipkin traces both open from a single ?traceId= link under the layer’s trace tab; the viewer auto-selects native vs Zipkin by the trace-ID shape, so /layer/<layer>/trace?traceId=… always opens the right one.
• Trace filters are searchable, on-theme dropdowns. The native Service / Instance / Endpoint pickers and the Zipkin Service / Remote service / Span name pickers use a dark type-to-filter dropdown that reopens correctly after a pick.

Logs

• Log and browser-error lists query on demand, not on every edit. The per-layer Logs tab, cross-layer Log inspect, and the Browser Errors tab now stage condition changes and fetch only when you press Run query — a fresh tab shows a “Pick your conditions, then click Run query” prompt, and switching service resets to that prompt (clearing the level / tag / category filters), so the previous service’s data never lingers under the new one.
• Log inspect uses the full width. The cross-layer Log inspect form (Target + Tags / Trace ID / Time / Limit conditions) now spans the whole page instead of sharing a two-column strip with empty space.
• Clicking a log row opens a centered popout. Both the cross-layer Log inspect and the per-layer Logs tab now open the same full-payload popout on row click — format-aware pretty-print (JSON pretty-printed by content type), the tags table, service / instance / endpoint / time meta, a copy button, and the trace link. Escape or the close button dismisses it.
• Log inspect can now query Browser errors across the page. A new Browser source on Log inspect (beside Raw) queries the BROWSER layer’s JS error logs from anywhere — pick a browser service or type a service name (or leave it blank for all services), then narrow by category (AJAX / RESOURCE / VUE / PROMISE / JS / UNKNOWN), version, page, and time window, and read the error list (message, category, page path, app version, time, minified line:col). Upload and manage source maps inline, then click a row to open a popout with the error meta, the raw stack, and the source-map de-obfuscation control — resolve the minified stack back to the original frames + source snippet.
• Tag fields autocomplete on theme. The Tags filter on Trace inspect, Log inspect, and the per-layer Traces / Logs tabs now suggests tag keys (before =) and per-key values (after) in a dark, dense dropdown anchored under the field, replacing the browser’s native <datalist> popup. On Trace / Log inspect, pressing Enter commits the current tag and starts the next, mirroring the per-layer chip tabs.
• Log inspect can now read Kubernetes Pod logs across the page. A new Kubernetes Pod logs source on Log inspect (beside Raw and Browser) tails a specific pod’s container logs on demand from the K8s API through OAP — pick a Kubernetes layer, then pick or type a service, choose a pod and container, set a trailing window (30s … 30m) and optional keyword filters, and read the dense, read-only log lines (timestamp + content). The Layer field lists only Kubernetes-deployed layers (the ones that actually carry pods) and auto-selects the single one when there’s exactly one; the pod and container auto-select when there’s only one to choose, so the common single-replica case is one click. These logs are streamed live and never persisted, so there is no cold-stage. When on-demand pod-log tailing is disabled on OAP, or the pod can’t be resolved, the page surfaces OAP’s reason as a hint instead of an empty pane.

Metrics Inspect

• Inspect can now chart a “foreign” metric — one the connected OAP doesn’t define. When a metric is written into shared storage by another OAP (an older version, or a different distribution, that the OAP behind Horizon doesn’t carry the analysis rule for), it never shows up in the catalog drawer. The + add metric drawer now has a Foreign metric tab (beside the catalog browse): type the metric name, pick its scope (Service / ServiceInstance / Endpoint / the three relations), and give its storage value column (default value) and value type (LONG / INT / DOUBLE / LABELED) — those last two come from the catalog of the OAP that does define the metric. Stage several with + add to list and add them in one go — the drawer’s shared footer counts the pending selection and respects the board cap, exactly like the catalog tab. Each resulting widget behaves like any other: it enumerates the entities holding data for the metric, defaults to the top one, and plots the value series — stepping or multi-selecting entities, switching chart type, and refetching all work. The connected OAP can’t evaluate a foreign metric through normal MQE, so the values are read through its admin surface with the column + type you supplied. Marked with a FOREIGN pill and the value type; persists on the board across refreshes like any other widget.

Bundled layer dashboards

• Single-value metrics now render as cards, not flat lines, on several layer dashboards. Widgets whose expression collapses the window to one number (a latest(...) total) had been mis-ported as line charts — drawn as a lone dot that misreads as a time series and shares one axis with an unrelated average trend. Each is now split into a proper single-value card (the total) plus a trend line (the average), matching the metric’s shape, the way booster-ui rendered them. Affects the Virtual GenAI (Input / Output Tokens, Estimated Cost — provider and model scopes), Elasticsearch (deleted documents), ClickHouse (Zookeeper sessions / watches), RabbitMQ (connection / publisher / consumer / channel / queue totals, allocated memory), RocketMQ (max CommitLog disk ratio, max producer / consumer message size), and APISIX (etcd reachability) dashboards; every changed dashboard row still tiles to full width.

• The Kubernetes Service dashboard tiles without a gap. The Service and Pod views left an empty half-row where a run of same-width widgets didn’t divide evenly into the grid, and the layout back-filled the hole by pulling a later chart up out of order. The CPU / Memory Resources and Pod HTTP RPM / Response Time widgets are re-sized so every row fills the grid in order — no gap, and the charts stay in reading order.

• The Kafka layer’s cluster metrics are back. Its Max Lag column asked for a roll-up the service-list query does not offer (max), and one unusable column rejects the whole request — so the Kafka page came back with no cluster measured at all: every metric column blank, an empty KPI strip, and no cluster selected for you (the clusters themselves stayed listed and selectable). Max Lag now rolls up as the average across the listed clusters, matching how the same metric is charted on the Kafka cluster dashboard.

• The AWS DynamoDB account list sorts by throttling again. Its default sort named a throttled column that does not exist — the columns are Read / Write Throttled and Read / Write Sys Err — so accounts came back in name order instead of worst-first, and on a deployment with more accounts than the metric-probe cap the accounts that got measured were not the throttled ones. It now sorts by Read Throttled.

• Every bundled layer and overview template is checked before release. The templates Horizon ships are validated on each build against what the pages actually accept — roll-up values, widget kinds, required fields, and every internal reference (a default sort naming a real column, an overview widget naming a real layer) — so a template defect like the two above fails the build instead of reaching your screen as an empty page.

Layer dashboard editor

• A layer template that would break the layer is refused at publish instead of being stored. Publishing a layer template — from the editor, or from a file you imported or hand-edited — is now structurally checked before anything reaches OAP, the same way overview dashboards and the 3D-map config already were. A template carrying an unknown field, an unknown component flag or dashboard scope, a widget the dashboard grid cannot run (unknown kind, or no expression at all), or a service-list column the service list cannot query (a roll-up other than sum / avg, or more columns than it accepts) is rejected, with nothing written to OAP — previously it was stored as-is and took the layer down for everyone, because one unusable widget empties the whole widget grid and one unusable column empties the whole service list. Work in progress still publishes: a config section you opened but haven’t filled in, a metric row whose expression is still empty, or a cleared alias are all accepted — at worst that one metric reads —.

• New tab widget — a sized slot with named tab panels, each holding its own widgets, edited inline. A layer dashboard widget can now be a tab container: a grid slot you size with span / row span, holding any number of named tabs, where each tab is its own little dashboard — its own set of widgets (card / line / top / record / table) in a sub-grid. Switching a tab swaps the whole set, and only the active tab’s widgets are queried (lazy) — an unopened tab costs nothing, and a previously-viewed tab stays warm. Author it in the Layer dashboards admin: add a widget, set its type to tab, then on the tile a segmented tab bar switches the active panel and a per-tab + widget drops a widget into it — you build each tab’s layout right where it sits, no separate screen: click a widget to edit it in the drawer, drag its corner to resize it, or drag it out of the tab to move it back to the top level. Manage the tabs (add / rename / reorder / delete) from the drawer’s Tabs list. The tab slot is framed by an open top/bottom rule with rounded corners so its inner widgets stay full-width. Drag a top-level widget onto a tab to move it in; a tab can’t nest a tab. Useful for packing related views — traffic / latency / Apdex, or one panel per subsystem — into one dense slot.

• + Add widget now lets you pick the kind. The add button opens a grouped menu — Tab group first, then the five widget kinds (card, line, top, record, table) — each with a one-line description, instead of always dropping in a card you then have to retype. The editor header stays pinned so the button is always reachable, and selecting a widget reserves a right-hand column for the edit drawer (the canvas re-shrinks to fit) with the drawer pinned in view as you scroll.

• The widget editor pins in place beside the canvas and always opens complete. On the Layer dashboards admin, clicking a widget — anywhere on the board, including the bottom rows — now opens the per-widget editor pinned next to the canvas and fully visible, without scrolling the page (a sticky panel used to get clipped past the bottom of a tall board, hiding the editor’s top or its Up / Down / Delete row). The move / delete controls sit in a pinned footer, and the editor tucks away when you scroll up to the scope config above. Adding a widget scrolls the new widget into view, next to the editor that opens for it.

Compare on a layer dashboard

• One-click exit from comparison. The compare bar gains a Clear all button that drops every locked entity — including the current one, whose chip has no per-entity × — and returns the page to the single-entity view. Previously, when your only lock was the current entity, there was no way to leave compare from the bar.