[{"body":"\nBanyanDB 0.11.0 is out, and it\u0026rsquo;s a dense one: vectorized queries move from opt-in to the default query path, a new pluggable pipeline handles trace-retention sampling, cluster-wide schema-consistency barriers close a real correctness gap, coding agents get two new ways to query BanyanDB in natural language, and etcd support is fully removed in favor of the property-based schema registry. We went through the 229 commits behind the release — not just the changelog — to pull out what actually matters if you operate a cluster.\nKey Takeaways\nVectorized query paths for measure, stream, and trace are now on by default, cutting allocations for scan-heavy queries — but it flips the rolling-upgrade order: liaison nodes first, then data nodes. A new in-merge and finalize-time trace-retention sampling pipeline drops unwanted spans with pluggable .so sampler plugins, with bounded memory even under multi-million-trace merges. Coding agents can query BanyanDB in natural language two ways now: a Claude Code/Codex MCP plugin with a bydbql skill, and a standalone bydbctl agent terminal UI. etcd support is fully removed and the API version bumps to 0.11 — plan a maintenance-window upgrade. Queue and lifecycle metrics were also redesigned. Below: the features worth trying, the performance work worth knowing about, the API surface that grew, and the changes that will break your upgrade if you\u0026rsquo;re not ready for them.\nVectorized Queries Are Now the Default The columnar (vectorized) query path — which replaces per-row protobuf serialization with a batch, columnar pipeline — was already on by default for measure queries since 0.10. In 0.11, stream and trace queries join it: --stream-vectorized-enabled, --trace-vectorized-enabled, and --measure-vectorized-enabled all default to true.\nFor measure queries specifically, coverage is now complete on a single node: scan, GroupBy+Agg via BatchAggregation, scalar reduce, raw GroupBy, TopN/BottomN, order_by, and boundary-error parity all resolve through the vectorized dispatch with row-path-equivalent semantics. The gRPC wire format is byte-identical to the row path\u0026rsquo;s output, and the team validated it with a 6-hour production soak showing zero divergences. Distributed Map-mode partial aggregation and multi-group requests still flow through the row path pending follow-up work.\nThis is also the release\u0026rsquo;s headline breaking change for rolling upgrades — see Breaking Changes below for the required upgrade order.\nA Pluggable Pipeline for Trace Retention Sampling Trace volume is the classic observability-backend problem: keep everything and pay for it, or drop data and hope you kept the traces that mattered. 0.11 gives BanyanDB an answer: a storage-node in-merge trace-retention filter that evaluates per-group sampler chains and safely drops non-retained traces from both core and secondary-index parts, configured dynamically per group with runtime register/update/remove support.\nTwo design properties make this production-shaped rather than a one-off filter:\nFinalization sampling is a best-effort backstop. A single node-wide, concurrency-1 scanner periodically sweeps cooled segments and force-merges each shard\u0026rsquo;s un-finalized parts through the group\u0026rsquo;s sampler chain, reusing the existing hot-merge path so it never contends with the hot-merge semaphore. A per-part finalizeGen stamp — written to disk before the part metadata — means a crash can\u0026rsquo;t double-sample on replay. The drop-set is bounded by design. A shard\u0026rsquo;s first finalize round can select every cooled part into one merge. An 18-million-entry drop set would otherwise reach roughly 1.3 GiB live and 2.6 GiB reserved heap, in a process also serving queries. So the pipeline bounds the sampling decision instead, not the pruning predicate. Once a merge\u0026rsquo;s drop set is full, every further proposed drop is retained instead of recorded. The set stays complete with respect to drops actually performed — no orphaned entries, none missing. The ceiling itself comes from the memory protector, as limit/(16×CPUs), so the aggregate across concurrent merges stays near limit/16. How the trace-retention sampling pipeline decides what to keep Two inputs feed the sampler chain: new parts evaluated during in-merge filtering, and cooled segments swept by the finalization backstop scanner for parts that missed the in-merge pass. The sampler chain evaluates per-group rules and routes each trace to retained or dropped. New parts (in-merge filter) Cooled segments (finalization backstop) Sampler chain (per-group rules) Retained Dropped Source: BanyanDB CHANGES.md and docs/design/trace-drop-set-bounding.md, 0.11.0 How the trace-retention sampling pipeline decides what to keep. Original diagram. See the trace drop-set bounding design doc for the full derivation.\nFirst-party sampler plugins ship for SkyWalking\u0026rsquo;s own trace schema and for Zipkin (sw-trace-sampler.so, zipkin-trace-sampler.so), plus a bounded telemetry SDK so a sampler plugin can emit its own metered metrics and logs without the host process\u0026rsquo;s cardinality or log budget going unbounded.\nSee the trace-pipeline plugin SDK and sampler config reference for the full config schema.\nCluster-Wide Schema Consistency Before 0.11, a schema change (create a stream, add an index rule, delete a group) could return success from the metadata service before every node in the cluster had actually applied it — a query against a node that hadn\u0026rsquo;t caught up could see stale or missing schema. 0.11 introduces client-observable revision tracking and a barrier RPC to close that gap. See API Changes below for the concrete fields and RPCs, and the schema-consistency client surface and SchemaBarrierService reference for the full RPC contract.\nPhase 2 extends the barrier cluster-wide: it fans the same calls out across every liaison and data node through a new NodeSchemaStatusService, handling mixed-version and membership-change cases safely. All of it is opt-in — zero-valued requests preserve prior behavior, so existing clients that don\u0026rsquo;t pass a revision see no change.\nNatural-Language Querying for Coding Agents 0.11 gives coding agents two independent ways to query BanyanDB without hand-writing BydbQL.\nThe first is a Claude Code / Codex plugin that packages the BanyanDB MCP server with a bydbql skill for natural-language-to-BydbQL generation over STREAM, MEASURE, TRACE, and PROPERTY resources. Install it directly from the repository (/plugin install apache/skywalking-banyandb in Claude Code, or the equivalent codex plugin add flow), and any Claude Code or Codex session gains four MCP tools: list_groups_schemas for schema discovery, get_generate_bydbql_prompt for generation (it\u0026rsquo;s the only tool that injects the live indexed-field list and enforces ORDER BY index-rule substitution), validate_bydbql for parse-only syntax and safety validation via a prebuilt Go binary, and list_resources_bydbql to execute a validated, read-only statement.\nThe second is bydbctl agent, a standalone two-pane terminal UI that drives a Codex or Claude Code CLI process directly for interactive natural-language BanyanDB querying: it discovers your schema, proposes typed query plans, and runs read-only queries. It owns none of your AI provider\u0026rsquo;s credentials — you authenticate the CLI it wraps, separately. Where the MCP plugin adds BanyanDB querying to any Claude Code/Codex session, bydbctl agent is a dedicated interactive tool for the same job.\nTwo ways to query BanyanDB in natural language Path one: a Claude Code or Codex session uses the MCP plugin bydbql skill to query BanyanDB directly. Path two: the standalone bydbctl agent terminal UI drives a separate Codex or Claude Code CLI process, which queries BanyanDB. Both paths are independent and read-only. Claude Code / Codex session MCP plugin (bydbql skill) bydbctl agent (terminal UI) Codex / Claude Code CLI process BanyanDB Source: docs/operation/mcp/plugin.md, skills/bydbql/SKILL.md, docs/interacting/bydbctl/agent.md Two independent, read-only paths to query BanyanDB in natural language. Original diagram. See the bydbctl agent setup and usage doc to get started with the terminal UI.\nAlso Shipped: Canopy, Migration Tooling, and More Four more additions worth knowing about:\nCanopy is a brand-new admin UI — a standalone React SPA with a Fastify BFF, not embedded in the existing ui/ — covering metadata CRUD for Group/Stream/Measure/Trace/IndexRule, a query console with WHERE-clause coverage across distributed clusters, Property collection CRUD, and TopN aggregation management. It shipped with its own Docker image, CI, and E2E suite. (These specifics come from the canopy/ commits and design docs themselves, not from CHANGES.md, which covers Canopy in less detail.) A migration tool with copy, verify, and analyze subcommands now covers measure and stream data (index-mode measures included), building on the trace/lifecycle migration work from earlier releases. Tags can change type across schema changes without breaking old parts. If a tag\u0026rsquo;s type changes (say, int to string), BanyanDB now persists each type variant in its own file ({tag_name}.{tag_type}.tf) instead of overwriting, and query/merge logic resolves by the (name, type) pair. This covers measure, stream, trace, and sidx parts. Fair fast/slow lane scheduling for trace part merges, so short merges no longer queue behind long-running ones; queue wait time is now exposed as total_merge_queue_latency. See the Canopy setup and architecture guide for how to run it.\nPerformance Improvements Beyond the vectorized-by-default query engine above, a handful of targeted optimizations landed in 0.11:\nFaster point-lookup queries for trace and stream, via lazy block-metadata decode — queries that only need a handful of rows no longer pay for decoding metadata upfront. Trace sampler decode path optimized: deferred decoding, zero-copy string and tag handling, early tag rejection, direct scalar reads, and cached rule prefixes roughly halve sampler decision cost and tag-rule cost for the SkyWalking and Zipkin samplers. Faster GCS backup uploads — each object and its checksum metadata now write in one request, dropping the per-object Update round-trip. Lifecycle migration is dramatically more memory-efficient. Streaming the dump reader and pooling size-classed marshal buffers, instead of reading a large measure part entirely into memory, cuts peak heap for row-replay by roughly 80% on the same workload: Lifecycle row-replay peak heap: before vs. after 0.11.0 Peak heap during large-measure-part row replay dropped from approximately 1.5 GB to approximately 296 MB, roughly an 80% reduction, via a streaming dump reader, pooled size-classed marshal buffers, and a bounded in-flight batch (default 32 MiB). Source: BanyanDB CHANGES.md, 0.11.0. Lifecycle migration: peak heap, before \u0026#8594; after Row-replay of large measure parts, same workload ~1.5 GB Before ~296 MB After \u0026#8595; ~80% less peak heap Source: BanyanDB CHANGES.md, 0.11.0 (streaming dump reader + pooled buffers + 32 MiB bounded batch) Source: BanyanDB CHANGES.md, 0.11.0 — streaming dump reader, pooled size-classed marshal buffers, and a 32 MiB default bound on in-flight batch bytes. API Changes The API version itself also moved to 0.11 — that\u0026rsquo;s covered under Breaking Changes below, since it\u0026rsquo;s upgrade-blocking rather than additive. The changes here are all additive and opt-in:\nmod_revision added to Group/IndexRule/IndexRuleBinding/TopNAggregation create/update responses; delete_time added to all delete responses; created_at added and preserved across updates. New STATUS_SCHEMA_NOT_APPLIED status code for writes and queries whose revision is ahead of the server\u0026rsquo;s cache. New SchemaBarrierService RPC — AwaitRevisionApplied, AwaitSchemaApplied, AwaitSchemaDeleted — so a client can block until a schema change has actually propagated, cluster-wide, before proceeding. QueryRequest.group_mod_revisions / QueryResponse.group_statuses added for per-group query-path revision gating. BydbQL gains ? positional parameter binding — bind values instead of string-interpolating them into the query text, closing off QL injection the same way parameterized SQL does elsewhere. A reusable Prepared binding type layers prepared-statement caching on top, on the gRPC query path, with a bounded cache, top-K retention, cache and slow-query observability, and bound parameters redacted in the slow-query log. New validation: Measure\u0026rsquo;s ShardingKey must now contain all Entity tags, to guarantee entity locality. Breaking Changes and How to Upgrade Safely Straight from the project\u0026rsquo;s own upgrade guide, in order:\n1. API version 0.11. A cluster containing both 0.10 and 0.11 nodes is not supported. This one needs a maintenance window: stop writes and all API clients, stop all 0.10 nodes, upgrade and start all nodes at 0.11, upgrade API clients to require version 0.11, then verify schema initialization and ingestion before restoring traffic. Rollback means stopping all clients and nodes first — never run a mixed 0.10/0.11 cluster, in either direction.\n2. Vectorized query paths, liaison before data. A distributed data node with a vectorized path enabled emits a native columnar frame instead of protobuf on the liaison↔data wire. A 0.11 liaison decodes both formats — it dispatches per message on the frame\u0026rsquo;s leading magic byte — but an older liaison has no frame decoder at all and fails to deserialize the response. That flips the normal rolling-upgrade order:\nUpgrade order Result Liaison first, then data Safe. New liaisons decode both frames and protobuf; old data nodes keep sending protobuf until upgraded. Data first, then liaison Queries fail for the duration of the rollout. Standalone deployments are unaffected — the frame is only emitted on a distributed data node. If you can\u0026rsquo;t control node ordering, start new data nodes with --stream-vectorized-enabled=false --trace-vectorized-enabled=false --measure-vectorized-enabled=false and flip them on only after every liaison is upgraded. Rollback is the same three flags; no data migration is involved, since the flags affect only the query and wire paths, never the on-disk format. This is the one most likely to bite an automated rolling-upgrade pipeline that assumes \u0026ldquo;data nodes first\u0026rdquo; from every previous release.\n3. etcd is gone. The property-based schema registry is the only supported mode now. Every --etcd-* flag is gone, --namespace is gone, and --node-discovery-mode no longer accepts etcd (use dns, file, or none). If --schema-registry-mode or --node-discovery-mode still reference etcd, you need to migrate to the property-based registry before you can run 0.11 at all.\n4. Queue and lifecycle metrics were redesigned. queue_pub/queue_sub metrics moved to a uniform operation/group-labeled model (the old topic label and chunk-ordering metric families are gone), and lifecycle health metrics gained remote_node/remote_role/remote_tier/group labels while banyandb_lifecycle_self_identity_resolution_total was removed outright. Update dashboards and alerts before you upgrade, not after.\nSee the complete \u0026ldquo;Upgrading to 0.11\u0026rdquo; walkthrough for the full maintenance-window checklist.\nBehind the Release 14 people contributed non-merge commits between v0.10.3 and v0.11.0 — a reminder that a release this dense is a team effort, not a single push.\nTop contributors to BanyanDB 0.11.0 Commit counts, v0.10.3 to v0.11.0, non-merge commits, 14 total contributors. Gao Hongtao 131, mrproliu 48, eight other contributors combined 18, Owen Willison 11, Huang Youliang 10, OmCheeLin 6, Tanay Paul 5. Top contributors to 0.11.0 Commits per author, v0.10.3\u0026#8594;v0.11.0 (14 contributors total) Gao Hongtao Gao Hongtao: 131 commits 131 mrproliu mrproliu: 48 commits 48 8 other contributors 8 other contributors: 18 commits 18 Owen Willison Owen Willison: 11 commits 11 Huang Youliang Huang Youliang: 10 commits 10 OmCheeLin OmCheeLin: 6 commits 6 Tanay Paul Tanay Paul: 5 commits 5 Source: BanyanDB git history, v0.10.3…v0.11.0 (229 non-merge commits, 14 authors) Source: BanyanDB git history, v0.10.3…v0.11.0 (229 non-merge commits, 14 authors). Original analysis. What\u0026rsquo;s Next The vectorized engine\u0026rsquo;s own release notes flag what\u0026rsquo;s still pending: distributed Map-mode partial aggregation and multi-group (multi-measure) requests still run through the row path. Expect that gap to close in a follow-up release rather than this one — and expect the trace-sampling pipeline\u0026rsquo;s plugin ecosystem to grow past the two first-party samplers now that the SDK and dev toolkit are stable.\nFrequently Asked Questions Do I have to reorder my upgrade automation for 0.11? Yes, if you run a distributed cluster with any of the vectorized flags enabled (the default). Upgrade liaison nodes before data nodes — the reverse of every prior release\u0026rsquo;s guidance — or disable the vectorized flags on new data nodes until every liaison is upgraded.\nCan I keep running etcd for schema discovery? No. --schema-registry-mode only accepts property in 0.11, and every --etcd-* flag has been removed. Migrate to the property-based registry before upgrading.\nIs the vectorized query path safe to trust for correctness, not just speed? The measure path was validated by a 6-hour production soak with byte-identical parity and zero divergences against the row path, plus per-workload bench gates. All three engines (measure, stream, trace) keep a rollback flag (--{measure,stream,trace}-vectorized-enabled=false) that reverts to the row path immediately with no data migration required, if you do hit a discrepancy.\nDoes BydbQL support parameterized queries now? Yes. 0.11 adds positional ? parameter binding to BydbQL, so you bind values instead of string-interpolating them into the query text, closing off QL injection. A reusable Prepared binding type also adds prepared-statement caching on the gRPC query path, with a bounded cache, top-K retention, cache and slow-query observability, and bound parameters redacted in the slow-query log.\nWhat\u0026rsquo;s the difference between the BydbQL MCP plugin and bydbctl agent? The MCP plugin adds four BanyanDB query tools (schema discovery, generation, validation, execution) to any Claude Code or Codex session you\u0026rsquo;re already running — install it once and it\u0026rsquo;s available alongside whatever else you\u0026rsquo;re doing. bydbctl agent is a separate, dedicated two-pane terminal UI purpose-built for interactive BanyanDB querying. Use the plugin if you want BanyanDB querying inside your existing agent workflow; use bydbctl agent if you want a standalone querying tool.\nConclusion 0.11.0 is the release where BanyanDB\u0026rsquo;s columnar query engine graduates from opt-in to default, trace retention gets a real pluggable answer instead of a blunt TTL, and coding agents get first-class natural-language access to your data. Read the full 0.11.0 release notes for the complete list, and work through the \u0026ldquo;Upgrading to 0.11\u0026rdquo; checklist before you touch a production cluster.\n","excerpt":"\u003cp\u003e\u003cimg src=\"banner.jpg\" alt=\"BanyanDB 0.11.0 release cover showing 229 commits, 14 contributors, and three query engines vectorized by default\"\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/apache/skywalking-banyandb\"\u003eBanyanDB\u003c/a\u003e 0.11.0 is out, and it\u0026rsquo;s a dense one: vectorized queries move from opt-in to the …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/banyandb-0-11-0-vectorized-queries-by-default-explained/","title":"BanyanDB 0.11.0: What's New and How to Upgrade"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/tags/release/","title":"Release"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/tags/storage/","title":"Storage"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/tags/","title":"Tags"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/tags/ai/","title":"AI"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/tags/community/","title":"Community"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/tags/engineering/","title":"Engineering"},{"body":"\nAfter Claude Code has worked through a long task, how do you revisit what happened? You may want to follow the conversation, inspect a tool call and its result, or see what a child agent contributed. The records already exist on your machine, but a task that spans hours, tools, context compactions, and child agents can be spread across many files.\nToday we are sharing Apache SkyWalking AI Sessionizer, a new pre-alpha project offering an early local replay view for Claude Code sessions. It reads the native evidence Claude Code has already written, connects the conversation with tool and child-agent activity, and lets you explore that execution in your browser. No plugin installation or configuration changes are needed in Claude Code, and you can start with history recorded before you began using Sessionizer.\nWhat does replay mean? Replay means reconstructing and navigating a recorded session from the evidence it left behind. Open a conversation, follow its inputs and responses, inspect recorded tool requests and results, and explore parent and child-agent activity on a time axis. The aim is to make a long agent session understandable after it happened.\nFor example, when Claude Code delegates part of a task, you can inspect the parent\u0026rsquo;s request, explore the child\u0026rsquo;s recorded work in its own execution stream, and follow the relationship back to the parent where the evidence supports that connection. This brings the conversation and the activity behind it into one view.\nReplay here does not rerun the model or execute tools again. Claude Code\u0026rsquo;s local files do not expose every exact provider request, system instruction, cache annotation, timing boundary, or retry identity. The view reconstructs the recorded activity and makes gaps and uncertain connections visible; it cannot reproduce hidden reasoning.\nCollection and assembly are implemented, and the local replay view is still developing. Static export and remote telemetry integration are future work. We are opening the project at pre-alpha so developers can try the experience and help shape the conversation model and evidence rules.\nNothing to install or configure in Claude Code Sessionizer reads existing local files. There is no Claude Code plugin to install, hook script to register, process wrapper to use, or environment and runtime configuration to change. You do not need to start a new Claude Code session to capture data: retained history can already be explored.\nSessionizer itself runs as a separate program. The walkthrough below builds it from source and starts the local view.\nStart the local replay view To build Sessionizer from source, use Go 1.27 or later. Clone the project, build it, and start the local replay view:\ngit clone https://github.com/apache/skywalking-ai-sessionizer.git cd skywalking-ai-sessionizer make build ./bin/asz view Then open http://127.0.0.1:8787. asz view is the command that gets the local experience up and running: it starts the page, discovers existing Claude Code data, collects and parses it in the same process, and continues refreshing it on the configured interval.\nTwo additional commands are useful for inspecting the pipeline, but they are not required before asz view:\n./bin/asz sources # list discovered sessions and source files ./bin/asz collect -once # collect the current local evidence once, without the UI Run these commands from the repository root so Sessionizer reads the included asz.yaml. The defaults discover existing Claude Code data and serve the local view without any configuration edits.\nFigure 1: The conversation index summarizes talks, steps, streams, segments, active span, and last activity across the assembled Claude Code sessions.\nThe browser view begins with conversations and talks, then exposes the parent and child execution streams behind them. Transcript content and a time-axis flow can be inspected alongside model calls, tools, agent activity, relations, and the source evidence used to build those relations.\nFigure 2: The talk view connects readable input and output with parent and child streams, an evidence inspector, and a time-axis flow of context, model, and tool activity.\nFrom a transcript to a session A transcript is useful, but it is not the whole execution.\nClaude Code may place the main transcript, child-agent transcripts, child metadata, workflow journals, manifests, and scripts in different locations. Some records are a readable conversation. Others describe tool execution, model calls, delegation, or workflow state. A single source does not necessarily establish how every piece relates to the others.\nBefore assembly, the Claude Code adapter maps those runtime-specific artifacts into Session Data, a common evidence format. Each record keeps its session and stream, native identifiers and parent references, call, run, tool, and child-agent keys, and a stable source location. This separates the meaning of the evidence from the file format in which one runtime happened to write it.\nHow Sessionizer links the evidence Assembly follows eight ordered stages because each establishes facts required by the next. Repeated records are removed first, keeping the first landed copy. Records are then partitioned into independent parent and child execution streams. Assistant fragments are grouped into model calls by message ID; tool requests and results are joined by tool-use ID; and agent calls are connected to child streams through the available agent and run identities.\nOnly an explicit reset record can open a new context epoch. Talks and Runs are then built by following triggers and parent ancestry, rather than assuming that nearby lines belong together. The fetching period provides the window in which Segments are determined. A Segment groups several Talks from that period, but it does not necessarily contain every Talk fetched in the period. Landed order remains authoritative throughout this process because timestamps from different records can move backward.\nFigure 3: Fragmented native artifacts become a durable conversation through ordered, evidence-based assembly. A Segment groups several—but not necessarily all—Talks from one fetching period, while unresolved records and the quality of every join remain visible.\nThe assembled result separates ownership from relationship. A node has at most one containment parent: Session → Execution Stream → Context Epoch → Talk → Run → Step. Cross-stream flow and other causal claims are represented as sparse, typed relations such as starts, reports, ends_with, follows, summarizes, and in_segment. Every such relation carries both its source evidence and correlation quality.\nA Conversation supplies the durable chain identity, while a Session preserves source-runtime provenance. Parent and child agents remain in distinct Execution Streams so the child\u0026rsquo;s messages and tools are not copied into the parent. A Segment is not another containment parent. It relates several Talks found within a fetching period without implying that all Talks in that period belong to the Segment.\nA Segment groups several Talks from a fetching period The fetching period determines the window used to form Segments. Figure 4 shows four Talks collected in one period. Segment 3 groups Talk 12 and Talk 13; Talk 11 and Talk 14 remain outside it. Talk 12 is expanded to expose the agent activity behind that readable interaction.\nOne Talk can contain an entire agent loop A Talk is the readable interaction that begins with input from outside the agent. It is not necessarily one prompt followed by one reply. More human input can arrive while work is running, the agent can speak between tool calls, and a child-completion notification can start another Run while the original Talk continues.\nA Run is therefore an agent loop, not a single model call. Inside a Run, one model response can request a tool, the tool result can lead to another model call, and delegation can open an independent child stream. Sessionizer joins a tool request and its result into one Tool step. Child output stays owned by the child stream; the parent receives a qualified relation to that activity instead of absorbing it.\nFigure 4: A fetching period contains four Talks, but Segment 3 groups only two of them. The expanded Talk shows how one readable interaction can span multiple parent Runs and an independent child-agent stream; solid, dashed, and unresolved links preserve what the evidence can establish.\nThis process is what we mean by sessionizing: turning fragmented runtime evidence into a coherent, durable session without discarding its source or pretending that every relationship is certain.\nThe complete rules and object definitions are documented in Conversation Assembly and the Unified Conversation Model.\nEvidence has to remain evidence An attractive timeline can easily look more certain than the underlying records justify. Sessionizer therefore treats evidence quality as part of the data rather than a footnote in the UI.\nEvery claim is qualified as observed and replayable, observed but report-only, proposed, or unavailable. Correlations are also given resolution states such as exact and unique, exact but ambiguous, strongly inferred, unresolved, or conflicting. If one identifier matches several candidates, the assembler keeps the ambiguity instead of silently selecting the most convenient one.\nThat discipline applies throughout the model. Session identity is not guessed from a username or timestamp proximity. Parent and child streams retain their own continuity. Information absent from the source stays unavailable rather than being approximated into a fact.\nThe result is useful for more than rendering a page. It is a committed session-data foundation on which future views, exports, measurements, and evaluations can operate with the same understanding of what was observed.\nWhere we want to take it The current Claude Code path gives us something concrete to examine, but the project is intended to grow beyond a local viewer and beyond one agent runtime. Our agenda has three directions.\n1. Unify local and remote collection with one data format Local evidence is valuable because it provides immediate access to current and historical sessions without requiring runtime integration. Remote collection is necessary when sessions need to be observed across machines, teams, or managed environments.\nOur direction is to make these complementary collection paths feed the same session-data format. A conversation collected from local artifacts and one assembled with remote telemetry should retain the same core hierarchy, provenance rules, evidence qualifications, and execution boundaries. Local reconstruction, portable export, remote ingestion, and centralized storage should not create separate meanings for the same agent behavior.\nThis also creates a path from today\u0026rsquo;s local inspection toward near-real-time monitoring and shareable session evidence. The data model comes first; transport and presentation can then evolve without redefining the session each time.\n2. Move beyond Claude Code to Codex and LangChain/LangGraph agents Claude Code is a useful starting point because it already leaves rich local evidence and exercises many of the difficult cases: long conversations, tool calls, context changes, subagents, workflows, and activity distributed across files.\nIt is not intended to be the boundary of the project. We plan to explore adapters for Codex and for agents built in the LangChain and LangGraph ecosystems. Each runtime has its own vocabulary and exposes different evidence, so an adapter should preserve those runtime-specific facts while mapping them into the common conversation structure.\nThe goal is not to force every agent into a lowest-common-denominator transcript. It is to give different agent ecosystems a shared session boundary while retaining the evidence needed to understand each one accurately.\n3. Derive metrics and analysis dashboards as a bonus Once session data is collected and normalized, useful measurements follow naturally. Session duration, model and tool activity, token usage when available, child-agent participation, errors, context discontinuities, and other behavioral signals can be derived from the same committed structure.\nThis makes metrics analysis and dashboards a valuable bonus of sessionization. They can help operators understand activity and trends across many sessions, while a click into an individual session can retain the detailed evidence needed for investigation. Over time, the same foundation could also support whole-session evaluation: judging the trajectory and outcome of an agent conversation rather than scoring isolated inputs and outputs.\nThese dashboards and evaluations are a direction, not a feature we are announcing as complete today. The immediate work is to make the underlying session data trustworthy enough that future analysis has a sound basis.\nWhy share it this early? The most important decisions in Sessionizer are not page colors or chart types. They are decisions about identity, continuity, causality, provenance, privacy, and uncertainty. Those decisions become much harder to change after adapters and stored data depend on them.\nThat is why we are sharing the project at pre-alpha. We want developers of agents, observability systems, and evaluation tools to inspect the model, try the Claude Code adapter against real sessions, and challenge the cases we have not yet seen. Early contributions are especially valuable around schemas, privacy-safe fixtures, deterministic assembly, qualification rules, and golden tests.\nIf you would like to share your thoughts or discuss the project\u0026rsquo;s direction with us, join the conversation in Apache SkyWalking Discussions.\nExplore the source and documentation in the Apache SkyWalking AI Sessionizer repository. Start with a Claude Code session already on your machine, explore its recorded execution in the local replay view, and help us build toward broader agent replay, monitoring, and analysis.\n","excerpt":"\u003cp\u003e\u003cimg src=\"sessionizer-replay-featured.png\" alt=\"Replay Claude Code sessions without plugins or configuration changes with Apache SkyWalking AI Sessionizer.\"\u003e\u003c/p\u003e\n\u003cp\u003eAfter Claude Code has worked through a long task, how do you revisit what happened? You may want to …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-09-03-ai-sessionizer-first-look/","title":"Replay Claude Code Sessions Without Plugins or Configuration Changes"},{"body":"SkyWalking Cloud on Kubernetes 0.11.0 is released. Go to downloads page to find release tars.\n0.11.0 Features Release the skywalking-swck Helm chart from this repository. One chart installs the operator and, behind a values flag, the custom metrics adapter. The CRDs, the operator\u0026rsquo;s ClusterRole and the admission webhook configurations it ships are generated from the operator sources by make chart-manifests, and CI fails on any drift. Wire BanyanDB storage. A Storage of type: banyandb now yields SW_STORAGE=banyandb and SW_STORAGE_BANYANDB_TARGETS. Previously the operator could only configure Elasticsearch, so an OAPServer on BanyanDB had to carry the storage environment by hand — and the variable it needed changed name between SkyWalking 9.x and 11.x. Without this SWCK cannot deploy a working OAP at all across the supported range, since SkyWalking removed H2 permanently in 10.2.0. Wire BanyanDB TLS. security.tls with security.tlsSecretName mounts the CA at /skywalking/bydb-tls and sets SW_STORAGE_BANYANDB_SSL_TRUST_CA_PATH. Previously tls: true on a banyandb Storage was accepted, wired no TLS, and mounted the Elasticsearch keystore secret, leaving the OAP pod waiting on a secret nothing creates. Pass BanyanDB credentials from a Storage\u0026rsquo;s security.user.secretName, as the Elasticsearch path already did. Configure the Horizon UI with environment variables instead of a generated file. UI.spec.env and UI.spec.envFrom now carry them, the operator sets only what it derives, and the ConfigMap is mounted only when spec.config supplies a whole file — so a setting added in a future Horizon release works without an SWCK release. Add UI.spec.templatesMode, emitted as HORIZON_TEMPLATES_MODE. Left unset it follows the admin address: live reads OAP\u0026rsquo;s template store over the OAP admin host, so it is chosen only when spec.OAPServerAdminAddress is set, and readonly — which renders the templates bundled in the image — otherwise. Add envFrom to OAPServer and Satellite, which are configured entirely through environment variables and previously had no way to take one from a Secret. Reference storage credentials instead of copying them. The operator used to read the Storage\u0026rsquo;s user secret and write the username and password in as literal env values, so they appeared in both the OAPServer and its Deployment for anyone with read access. They are now secretKeyRefs resolved by the kubelet. Support OAP 10.4.0 and later, with 11.0.0 recommended, matching skywalking-helm. An OAPServer below that is admitted with a warning rather than rejected. Ship the Helm chart tarball as a signed, voted artifact on dist.apache.org, alongside the source and binary tarballs. Breaking changes Only the Horizon UI is deployed. spec.kind on the UI resource now accepts horizon alone — apache/skywalking removed the legacy Booster UI in 11.0.0 and no longer builds an image for it. A UI with kind: booster is rejected, with a message saying what to use instead. An OAPServer with no storage is refused. SkyWalking removed the embedded H2 permanently in 10.2.0, so there is nothing to fall back to: an OAPServer with nowhere to write starts, dials a BanyanDB on 127.0.0.1:17912 and never becomes ready. The webhook now says so at admission. Setting SW_STORAGE directly in spec.config still counts as having chosen a storage. OAPServerConfig and OAPServerDynamicConfig default to version 11.0.0, was 9.5.0. These match an OAPServer by exact version string, so a config that omits version previously only attached to an OAP explicitly pinned at 9.5.0. Set spec.version explicitly if you run an older OAP. Bugs Stop reconciling a UI whose kind is no longer supported. Narrowing the CRD enum to horizon only rejects new resources — schema validation runs on admission, never on read — so a UI stored as kind: booster by an earlier operator survives the upgrade and still reconciles. With the templates now unconditionally Horizon\u0026rsquo;s, reconciling one rewrote a running Booster Deployment into a shape its image cannot serve and took the UI down on the first pass after upgrade. Such a resource is now left untouched, with an UnsupportedKind event saying what to do. Stop applying an OAPServer Deployment when the Storage it names cannot be read. Every lookup error was logged and ignored, and the Deployment applied anyway — without SW_STORAGE, targets, credentials or TLS volumes — so a Storage briefly deleted and recreated replaced a working OAP with one that never becomes ready. The reconcile now leaves the running Deployment alone, emits a StorageUnresolved event and requeues. Keep the storage TLS volume when an OAPServerConfig mounts static files. The overlay assigned over the pod\u0026rsquo;s volume and mount lists, and ApplyOverlay is an RFC 7386 merge patch under which an array replaces rather than merges — so the certificate volume disappeared and SW_STORAGE_BANYANDB_SSL_TRUST_CA_PATH pointed at nothing. Roll the OAP when its credential Secret is rotated. Environment variables taken from a Secret are resolved once, when the container starts, so a rotation went unnoticed until something restarted the pod. The controller now watches Secrets and carries the referenced Secret\u0026rsquo;s resourceVersion in a pod-template annotation. Create certificate signing requests through certificates.k8s.io/v1. The v1beta1 API this used was removed in Kubernetes 1.22, so internal Elasticsearch TLS could not obtain a certificate on any cluster newer than that and the workload waited on a Secret nothing produced. The wait loop is also bounded now, and sleeps. Fix the default image for kind: horizon UIs. It was apache/skywalking-horizon-ui:\u0026lt;version\u0026gt;, a Docker Hub repository that does not exist — Horizon releases share apache/skywalking-ui with the legacy Booster UI and are told apart by a horizon- tag prefix. Since horizon is the default kind, every UI created without an explicit image could never pull. Build genuinely multi-architecture images. The Dockerfiles hardcoded GOARCH=amd64 while the publish workflow advertised linux/arm64, so apache/skywalking-swck:0.10.0 shipped an arm64 manifest holding x86-64 binaries and an arm64 node got exec format error. The release now builds a binary per architecture, and the publish workflow pulls every advertised platform back and checks the ELF machine type before the release completes. Raise the OAP startup probe budget from 110 seconds to 10 minutes. SkyWalking 11 has no embedded storage, so every start installs a schema into BanyanDB or Elasticsearch — work that overran the old probe on a cold cluster, and being killed mid-schema turned a slow first boot into a crash loop. Stop deriving the Horizon admin and Zipkin URLs. The OAP admin host arrived in 11.x, and on 10.x port 17128 is the AI-pipeline URI-recognition server, so a derived oap.adminUrl pointed Horizon at the wrong service; the OAPServer this operator deploys exposes no Zipkin port at all. Both are now emitted only when spec.OAPServerAdminAddress / spec.OAPServerZipkinAddress are set. Stop maintaining a copy of Horizon\u0026rsquo;s configuration schema. The generated config restated Horizon\u0026rsquo;s own defaults and had drifted: viewer was granted 6 of the 12 permissions Horizon gives that role, and the admin landing route was /admin/cluster, which Horizon has no route for. It also carried keys Horizon 1.0.0\u0026rsquo;s schema does not have, whose presence stops the BFF booting at all. Stop truncating rendered manifests at the first #. Every manifest was cut line-by-line at its first hash with no awareness of YAML quoting, so any value containing one — a password, an AI prompt, a URL fragment — was severed mid-string and the resulting manifest no longer parsed. Only whole-line comments are dropped now. Require SW_STORAGE to carry a value. The mandatory-storage check accepted an entry named SW_STORAGE with nothing behind it, which reaches the OAP as an empty selector and produces exactly the never-ready state the check exists to prevent. Render an OAPServer whose Storage cannot be read yet, rather than reaching through the nil and failing to render at all. Reference the Elasticsearch credentials from the Storage controller too, which still copied them out of the Secret and into the resource. Deep-copy the new env and envFrom fields; zz_generated.deepcopy.go had not been regenerated, so those slices were shared with the objects controller-runtime\u0026rsquo;s cache hands out. Ship the eventexporter admission webhook in the chart, and drop the duplicate meventexporter.kb.io entry that the API server rejects. Documentation Document BanyanDB storage: the endpoint format and its gRPC port, cluster targets, authentication, persistence, and the flags BanyanDB 0.11 renamed. See docs/en/setup/banyandb.md. Document that Horizon ships with no users, so a UI refuses every login until one is seeded through HORIZON_AUTH_LOCAL_USERS. Restructure the documentation into docs/en/{concepts-and-designs,setup,examples,guides,changes}, following the layout of apache/skywalking, and move the changelog into docs/en/changes/. Full Changelog: https://github.com/apache/skywalking-swck/compare/v0.10.0...v0.11.0\n","excerpt":"\u003cp\u003eSkyWalking Cloud on Kubernetes 0.11.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"0110\"\u003e0.11.0 …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cloud-on-kubernetes-0-11-0/","title":"Release Apache SkyWalking Cloud on Kubernetes 0.11.0"},{"body":"SkyWalking Kubernetes Helm Chart 5.0.0 is released. Go to downloads page to find release tars.\nThis release targets SkyWalking OAP 11.0.0, Horizon UI 1.0.0 and BanyanDB 0.11.0. See Upgrade for the migration steps.\nBreaking changes OAP 11 requires BanyanDB 0.11.x. OAP pins the BanyanDB server API versions it accepts (SW_STORAGE_BANYANDB_COMPATIBLE_SERVER_API_VERSIONS, 0.11 in 11.0.0) and checks them with string equality, so pairing OAP 11 with BanyanDB 0.10.x makes OAP refuse to start. The three versions move together — see Version Compatibility. oap.ports.admin is required. OAP 11 enables every admin feature module by default and serves /status/* and /debugging/* on the admin port only; they are no longer mirrored on oap.ports.rest. Horizon UI reads status, inspect, DSL debugging and the dashboard template store from it. The legacy booster UI is no longer supported. OAP 11 deleted apm-webapp and the skywalking-booster-ui submodule along with the docker.ui build target, so apache/skywalking-ui publishes no 11.x tag — only horizon-* tags. Replace ui.image.tag=\u0026lt;oap-version\u0026gt; with ui.image.tag=horizon-1.0.0. oap.config.ui-initialized-templates does nothing. OAP 11 removed the on-disk dashboard seed files and UITemplateInitializer, along with the sidebar menu storage, the UIConfigurationManagement GraphQL mutations and SW_ENABLE_UPDATE_UI_TEMPLATE. Horizon UI ships its own dashboard library and manages templates over the admin REST port. Horizon is configured by environment variable, and no ConfigMap is mounted by default. The image ships a complete env-tokenized /app/horizon.yaml; the chart sets only what it computes (HORIZON_SERVER_PORT, HORIZON_OAP_QUERY_URL, and the admin, Zipkin and public URLs when configured) and leaves the rest to ui.extraEnv / ui.envFromSecret. ui.config is now opt-in: setting it renders a ConfigMap and mounts it over the image\u0026rsquo;s file, so fields you do not write fall back to Horizon\u0026rsquo;s defaults. If you carried a ui.config block from the pre-release main values, move it to environment variables — see Configure Horizon. The SWCK charts are removed. chart/operator and chart/adapter packaged apache/skywalking-swck — its image, its CRDs and its version — and had no relationship to chart/skywalking. They were never released to Docker Hub, so no released artifact disappears, but installs from source or from the ghcr.io snapshot channel will break. They belong with the operator, where the CRDs are generated alongside the code that consumes them. The UI no longer proxies /graphql. Callers that talked to the UI\u0026rsquo;s GraphQL endpoint (for example swctl --base-url=http://\u0026lt;ui\u0026gt;/graphql) must target the OAP service directly on oap.ports.rest. Features ui.extraVolumes / ui.extraVolumeMounts, for the two Horizon settings that take a filesystem path: auth.tokensFile and sourceMaps.bootMountDir. server.publicUrl is derived from the first ui.ingress.hosts entry when an ingress is enabled, so single sign-on callbacks and the OAuth issuer are built from the address operators actually reach — see UI Service and Ingress. server.port is derived from ui.service.internalPort, so the BFF binds the port the container exposes. oap.extraEnv (a list, so entries can carry valueFrom) and oap.envFromSecret, applied to the OAP Deployment and the init Job. Note Kubernetes gives an explicit env entry precedence over envFrom, and the chart sets SW_ES_PASSWORD / SW_DATA_SOURCE_PASSWORD itself — so sourcing those from a Secret needs oap.extraEnv. Horizon\u0026rsquo;s config hot-reload works again. The chart previously mounted horizon.yaml with subPath, which Kubernetes never updates in place, so the file watcher could not fire. Documentation moved into docs/ and is published at skywalking.apache.org/docs/skywalking-helm, together with the release guide for a process that was previously unwritten. Corrections Horizon UI does not refuse to start without configured users. It boots, serves the login page, and answers /api/auth/health with 200 — which is this chart\u0026rsquo;s readiness probe — so the pod reports Ready and nobody can sign in. Earlier documentation claimed a CrashLoopBackOff. See Set Up Logins. Full Changelog: https://github.com/apache/skywalking-helm/compare/v4.9.0...v5.0.0\n","excerpt":"\u003cp\u003eSkyWalking Kubernetes Helm Chart 5.0.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cp\u003eThis …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-kubernetes-helm-chart-5.0.0/","title":"Release Apache SkyWalking Kubernetes Helm Chart 5.0.0"},{"body":"SkyWalking APM 11.0.0 is released. Go to downloads page to find release tars.\nHorizon UI is now the official UI This release no longer ships a bundled web UI. apm-webapp and the skywalking-booster-ui submodule are removed, along with the skywalking/ui Docker image, the on-disk dashboard seed templates, and the UI-related GraphQL mutations and sidebar-menu storage.\nThe official UI is Horizon UI, a SkyWalking sub-project that releases independently of the OAP backend, with images on Docker Hub at apache/skywalking-ui. There is no 1:1 mapping between OAP versions and Horizon UI versions — pin the UI image tag in your deployment and upgrade the two on separate cadences.\nTo upgrade: replace skywalking/ui:\u0026lt;tag\u0026gt; with apache/skywalking-ui:latest (or a horizon-\u0026lt;version\u0026gt; tag), expose port 17128 from the OAP container, and move any script that called the legacy GraphQL UI mutations to the UI Management API.\nA new admin server, on by default Admin and operator-facing endpoints now live on a dedicated admin-server host — HTTP on 17128, plus an admin-internal gRPC bus on 17129 for peer-to-peer cluster RPCs, kept separate from the public agent gRPC port. Status, inspect, UI management, DSL debugging and runtime rules all mount on it, and all of them default to enabled.\nThe host has no built-in authentication and must be gateway-protected with IP allow-lists — never expose it to the public internet. See the Admin API security notice.\nTwo capabilities land on that host in this release:\nRuntime rule hot-update for MAL and LAL — ship metric and log rule changes without restarting OAP. Rules persist to the storage backend, the cluster converges within ~30 seconds, and hot-updates survive restart. Live debugger for MAL / LAL / OAL (SWIP-13) — a sample-based runtime debugger that captures per-stage inputs and outputs as the three DSLs process live ingest, fanning out to every cluster peer. New monitoring targets iOS/iPadOS apps (SWIP-11), Apache Airflow (SWIP-7), WeChat and Alipay Mini Programs (SWIP-12), Node.js and PHP runtime metrics, MCP observability for Envoy AI Gateway, and a rebuilt BanyanDB self-observability model (SWIP-15).\nProject Move the DSL class-loading machinery under core/dsl, and collapse the three copies of the \u0026ldquo;define a generated class into the right loader\u0026rdquo; dispatch into a static BytecodeClassDefiner.define. No behaviour change. Extend the GET /inspect/entities admin API to inspect a metric persisted by any OAP, even one this node does not define locally — the caller supplies valueColumn + valueType and the storage backend resolves the physical location from its own running config. Scope is no longer required. Add the POST /inspect/values admin API — read the value series of a metric persisted by another OAP by supplying its {valueColumn, valueType}. The real MQE engine runs over a request-scoped overlay, so the read returns a native MQE ExpressionResult. Admin-only; not mirrored onto the public REST / GraphQL surface. Remove the always-on alarm-to-event conversion (EventHookCallback). Events now originate only from real event sources (agents, SkyWalking CLI, Kubernetes Event Exporter); alarms remain available through the alarm store and the configured alarm hooks. TLS for all OAP HTTP/REST servers, with cert hot-reload. Adds restSSLEnabled / restSSLKeyPath / restSSLCertChainPath to every OAP HTTP server — core REST, sharing-server, admin, PromQL, LogQL, TraceQL and Zipkin query/receiver — each with its own environment variables. Refreshed certificates are picked up without restarting OAP. HTTP TLS is server-side only (no mTLS). New queryAlarms GraphQL query — entity / layer / rule-name filters for alarms, plus keyword, tags, duration and paging. Legacy getAlarm is deprecated but still routes to the same DAO. Adds a layer column on AlarmRecord and makes id0/id1 indexed; IAlarmQueryDAO.queryAlarms is a new abstract method, so 3rd-party storage backends fail at compile if they miss the override. The new filters apply only to alarms written after the upgrade. Breaking Change — apm-webapp and the skywalking-booster-ui submodule are removed; this distribution no longer ships a bundled web UI. The skywalking/ui Docker image, the apm-dist/ webapp packaging, the ui-initialized-templates/ dashboard seeds, the UIConfigurationManagement GraphQL mutations and queries, the SW_ENABLE_UPDATE_UI_TEMPLATE flag, and the server-side sidebar menu (UIMenuManagementService, UIMenu, MenuItem and their storage impls) are all retired. The official UI is Horizon UI, released independently on its own schedule. New ui-management admin module — five REST operations for dashboard templates on admin-server, replacing the retired GraphQL template resolver. The sidebar menu is intentionally not served; Horizon UI owns it client-side. All admin feature modules default-on — admin-server, status, inspect, ui-management, dsl-debugging and receiver-runtime-rule. Set the matching SW_* env var to empty to disable a feature. Status API moved to the admin host. /status/* and /debugging/* register on admin-server (default 17128) and no longer mirror on core.restPort. URIs and payloads are unchanged; only the host moved. One exception: /status/config/ttl stays bound on the public REST host so ecosystem tools can discover TTL bounds without learning the admin port. New admin-server module — shared host for admin and on-demand write APIs, running an HTTP REST surface (default 17128) and an admin-internal gRPC bus (default 17129) for peer-to-peer cluster RPCs. Enabled by default; it has no built-in authentication and must be gateway-protected. The runtime-rule config block loses its own restHost/restPort/etc. keys, which move under the new admin-server block. Runtime rule hot-update for MAL and LAL. addOrUpdate creates or replaces a rule, inactivate soft-pauses one while preserving the backend measure and its history, and delete removes an inactive row (with ?mode=revertToBundled to fall back to the on-disk YAML). Read-side endpoints cover get / bundled / list / dump. Every node converges within ~30 s (receiver-runtime-rule.refreshRulesPeriod), hot-updates survive restart, and all writes serialize on a deterministic \u0026ldquo;main\u0026rdquo; peer with transparent forwarding, so an L7 load balancer can route any operator request to any OAP. Live debugger for MAL / LAL / OAL — implements SWIP-13. Idle-path cost is one volatile-bool read per probe that JIT eliminates after warm-up; active sessions fan out to every cluster peer. Disabled by default (SW_DSL_DEBUGGING=default). Per-session hard caps: recordCap ≤ 10000, retentionMillis ≤ 1 hour. LAL sessions accept granularity=block|statement. Capture payloads include raw log bodies, so treat the admin port as authenticated infrastructure. BanyanDB schema mismatches are now visible at boot, not silent. A resource whose backend shape doesn\u0026rsquo;t match the current rule is skipped with an ERROR logging the declared-vs-backend diff, and OAP continues booting — previously the mismatch was accepted and its samples quietly dropped. Bump infra-e2e to testcontainers-go v0.42.0, which uses the Docker Compose v2 plugin natively. Remove the deprecated version field from all docker-compose files for Compose v2 compatibility. Best-effort schema-cutover fence for BanyanDB. After firing a schema install or drop, OAP waits up to a bounded window (default 2s) for every data node to apply the change before resuming dispatch, logging a warning and proceeding on laggard timeout. Bump dependencies: gRPC 1.70.0 → 1.80.0, protobuf-java 3.25.5 → 4.33.1, Netty 4.2.10.Final → 4.2.12.Final, Netty-tcnative 2.0.75 → 2.0.77, pgv 1.2.1 → 1.3.0. Inspect API on admin-server. GET /inspect/metrics lists every registered metric with its type / scope / catalog / value column / downsamplings; GET /inspect/entities scans the entities emitting values for a metric over a time range and returns an mqeEntity block ready to paste into execExpression. Adds IMetricsQueryDAO.listEntityIdsInRange as an abstract method. Enabled by default. Status feature module relocation, finalized. status-query-plugin is replaced by a status feature module under server-admin/ with URIs and payloads unchanged; the selector renames from SW_QUERY=…,status-query-plugin to a top-level SW_STATUS=default. Drop six unused test-scoped dependencies from runtime-rule; that coverage now lives in test/e2e-v2/cases/runtime-rule/. Declare server-testing at test scope everywhere, keeping its org.junit stubs off the runtime classpath that server-starter copies into oap-libs. library-banyandb-client gains the direct library-util dependency it always needed. Add ThreadPolicy.ioBound(N) to library-batch-queue for queues whose consumers spend most of their time blocked — virtual threads on JDK 25+, N platform threads otherwise, with identical semantics on both paths. Also fixes BatchQueue.shutdown(), which ran its final drain on the caller\u0026rsquo;s thread while drain loops could still be inside consume(), breaking the single-drain-thread invariant. OAP Server Add component IDs for the Spring LDAP Java agent plugin (spring-ldap: 179) and LDAP server (LDAP: 180), including their server mapping. Fix LAL\u0026rsquo;s segmentId and spanId extractor statements, which the grammar accepted and the parser never implemented — a rule writing segmentId ... failed at boot with a NullPointerException naming IfStatementContext. An unhandled extractor statement now reports its own rule line instead of throwing. Remove dead code from the DSL subsystem and correct the shared kernel\u0026rsquo;s own documentation. Unify source attribution for every generated DSL class, so a stack frame from OAL, MAL, LAL or Hierarchy code leads back to the rule that produced it. Support runtime rule hot-update and DSL debugging for the meter-analyzer-config catalog, bringing native meter rules to parity with otel-rules. Support Elasticsearch 9.x as storage. Add Node.js runtime metrics via the Node.js agent MeterReportService pipeline (meter_instance_nodejs_*, default 20s sample/report), analyzed through nodejs-runtime.yaml. Add PHP runtime PHM meter analyzer (php-runtime.yaml) for the SkyWalking PHP agent process. Batch the BanyanDB schema fence per runtime-rule apply, so a rule file that changes dozens of rules no longer does K×M sequential fences and overruns the apply\u0026rsquo;s REST budget. Add a runtime-rule apply-status query — the cluster main tracks each structural apply through a phase machine (pending → DDL → fencing → rolling-out → applied), with degraded and surfaced fenceLaggards for a committed-but-unconfirmed apply. Push runtime-rule convergence to peers on commit via a NotifyApplied admin-internal RPC, instead of waiting up to one refresh tick (~30s). Fix BanyanDB peer nodes permanently flooding \u0026lt;metric\u0026gt; is not registered, and a follow-on case where a peer kept translating writes with a stale schema shape after a runtime-rule reshape. Support LAL json {} parsing JSON content delivered in a plain-text log body, so previously-aborting rules on OTLP-fed layers now work without any receiver or protocol change. Surface the drop reason in LAL live-debugging when a rule stops a log at a parse step. Fix a v2 MAL CounterWindow key collision: rate() / increase() / irate() keyed each counter\u0026rsquo;s sliding window on the rule\u0026rsquo;s output metric name instead of the counter\u0026rsquo;s own name, so counters sharing a label set computed rates against each other\u0026rsquo;s values. Fix the v2 MAL Elvis operator ?: to honor Groovy-falsy semantics — the fallback now applies to an empty-string primary, not only to null. SWIP-15: rebuild BanyanDB self-observability around the cluster / container / group model (requires BanyanDB 0.11+) — a cluster is one Service, each container a ServiceInstance, and each storage group an Endpoint. Runtime MAL/LAL hot-update rules can declare layerDefinitions: to introduce new layers. Fix: runtime-rule schema changes now work in no-init mode — the deployment mode every production cluster runs. Previously a runtime addOrUpdate introducing a new metric blocked forever in the storage installer\u0026rsquo;s init-node poll loop. Fix: runtime-rule cross-node writes no longer fail with HTTP 400 forward_self_loop on a multi-replica Kubernetes cluster, where every replica shared the 0.0.0.0_11800 self node id. Fix: remove the redundant tags from the envoy-ai-gateway.yaml LAL configuration. Add a Zipkin Virtual GenAI e2e test, using the zipkin_json exporter to avoid a protobuf dependency conflict. Fix missing taskId filter and incorrect IN clause parameter binding in JDBCJFRDataQueryDAO and JDBCPprofDataQueryDAO. Remove deprecated GroupBy.field_name from BanyanDB MeasureQuery request building. Push the taskId filter down to the storage layer in IAsyncProfilerTaskLogQueryDAO, removing in-memory filtering from AsyncProfilerQueryService. Fix missing parentheses around OR conditions in JDBCZipkinQueryDAO.getTraces(), which bypassed the table filter for all but the first trace ID. Replaced with a proper IN clause. Fix missing and keyword in JDBCEBPFProfilingTaskDAO.getTaskRecord(), which caused a syntax error on every invocation. Fix storage layer bugs in profiling DAOs and add unit test coverage for JDBC query DAOs. Optimize TraceQueryService.sortSpans from O(N^2) to O(N) by pre-indexing spans by segmentSpanId, so trace detail queries scale linearly with span count. Support MCP (Model Context Protocol) observability for Envoy AI Gateway: MCP metrics (request CPM/latency, method breakdown, backend breakdown, initialization latency, capabilities), MCP access log sampling (errors only), the ai_route_type searchable log tag, and MCP dashboard tabs. Add weighted handler support to BatchQueue adaptive partitioning — MAL metrics use weight 0.05 at L1 (vs 1.0 for OAL), reducing partition count and memory overhead when many MAL metric types are registered. Fix missing taskId filter in pprof task log query and its JDBC / BanyanDB / Elasticsearch implementations. Fix duplicate calls in EndpointTopologyBuilder, which unlike ServiceTopologyBuilder did not deduplicate when storage returns multiple records for the same relation. Use containsOnce and noDuplicates for topology dependency e2e expected files to enforce no-duplicate verification. Bump infra-e2e to ef073ad to include noDuplicates pipe function support. PromQL: support querying Zipkin metadata (service name, remote service name, span name). TraceQL: support more tags and variables in Grafana for querying. LAL: add sourceAttribute() for non-persistent OTLP resource attribute access in scripts. LAL: add layer: auto mode for dynamic layer assignment when service.layer is absent. Add a two-phase SpanListener SPI mechanism for extensible trace span processing, and refactor GenAI from a hardcoded SpanForward.processGenAILogic() to GenAISpanListener. Add OTLP/HTTP receiver support for traces, logs, and metrics (/v1/traces, /v1/logs, /v1/metrics), for both application/x-protobuf and application/json. Fix: TTL query add metadata TTL. Fix: PersistentWorker used the wrong TTL for the metrics cache when the storage is BanyanDB. Add iOS/iPadOS app monitoring via the OpenTelemetry Swift SDK (SWIP-11) — the IOS layer, IOSHTTPSpanListener for outbound HTTP client metrics across OTel Swift\u0026rsquo;s .old/.stable/.httpDup semantic-convention modes, and IOSMetricKitSpanListener for daily MetricKit metrics. Add Apache Airflow monitoring via native OpenTelemetry metrics (SWIP-7) — a new AIRFLOW layer with Service (cluster) and Instance (host) dimensions, and 27 metrics under otel-rules/airflow/. Fix LAL layer: auto mode dropping logs after an extractor set the layer — codegen now propagates layer \u0026quot;...\u0026quot; assignments to LogMetadata.layer. Fix MetricKit histogram percentile metrics being reported at 1000× their true value, by marking the SampleFamily with defaultHistogramBucketUnit(MILLISECONDS). Add WeChat and Alipay Mini Program monitoring via the SkyAPM mini-program-monitor SDK (SWIP-12) — two new layers (WECHAT_MINI_PROGRAM, ALIPAY_MINI_PROGRAM) and two new JavaScript componentIds. Fix: remove VirtualServiceAnalysisListener\u0026rsquo;s dependency on GenAIAnalyzerModule if it is disabled. MAL: register TimeUnit in MALCodegenHelper.ENUM_FQCN so rule YAML can write .histogram(\u0026quot;le\u0026quot;, TimeUnit.MILLISECONDS) for SDKs that emit bucket bounds in ms. Fix: potential unexpected current directory inclusion in the Docker OAP classpath. MAL: add safeDiv(divisor) on SampleFamily, yielding 0 rather than Infinity/NaN when the divisor is 0, and use it in the Envoy AI Gateway latency-average rules. Fix envoy-ai-gateway metrics rules to return 0 when the divisor is 0. Custom Layers can now be declared without modifying the OAP source — via an operator-managed layer-extensions.yml, an inline layerDefinitions: block in a MAL or LAL rule file, or a plugin extension. The recommended ordinal range for external layers is \u0026gt;= 1000. LAL: support full arithmetic (+, -, *, /) on numeric operands, fixing the bug where (tag(\u0026quot;x\u0026quot;) as Integer) + (tag(\u0026quot;y\u0026quot;) as Integer) was treated as string concatenation, so token-threshold conditions never triggered abort {}. Fix: avgHistogramPercentile / sumHistogramPercentile reported the smallest finite bucket boundary for every rank when no samples were observed in any bucket. Fix: MAL expPrefix now applies to every metric source in exp, not just the leading one — previously secondary metrics inside arguments silently skipped the prefix. Add @Stream(allowBootReshape = true) opt-in for additive boot-time reshape of BanyanDB streams / measures, so a new @Column on a code-defined stream is appended to the live schema instead of being rejected as SKIPPED_SHAPE_MISMATCH. Mask keywords trustStorePass and keyStorePass by default. Bump dependencies to clear CVE alerts on shipped OAP jars: log4j 2.25.3 → 2.25.4, jackson 2.18.5 → 2.18.6, kafka-clients 3.4.0 → 3.9.2, postgresql 42.4.4 → 42.7.11, commons-compress 1.21 → 1.26.2. Bump more dependencies to clear CVE alerts: netty 4.2.12.Final → 4.2.15.Final, jackson 2.18.6 → 2.18.8, commons-codec 1.11 → 1.13, and realign jackson-databind 2.16.0 → 2.18.8 so the whole jackson family is managed at a single version. Bump Apache Curator 4.3.0 → 5.9.0 and Apache ZooKeeper 3.5.7 → 3.9.5 to clear CVE-2023-44981. No source changes were required. Migrate the Consul cluster and configuration client from the abandoned com.orbitz.consul:consul-client 1.5.3 to the maintained fork org.kiwiproject:consul-client 0.9.0, clearing CVE-2021-0341; the BOM now pins okhttp to 4.12.0. Bump test-scope assertj-core 3.20.2 → 3.27.7 to clear CVE-2026-24400. Clear three security alerts in the Airflow e2e mock (CI-only, never shipped): protobuf 4.25.8 → 5.29.6, opentelemetry-proto 1.24.0 → 1.28.0, grpcio 1.62.2 → 1.63.2. Clear Dependabot CVE alerts in the e2e Go test fixtures (CI-only, never shipped): golang.org/x/net 0.48.0 → 0.55.0 and the Go toolchain 1.24 → 1.26.5. Fix: continuous profiling policy validation now rejects a threshold / count of 0, matching rover\u0026rsquo;s value \u0026gt;= threshold trigger semantics. CPU percent and HTTP error rate are tightened from [0-100] to (0-100]. Fix wrong BanyanDB resource options in record data. Align the default BanyanDB stage segmentInterval values so each coarser stage is an integer multiple of the finer one, keeping hot → warm → cold lifecycle migration on the cheap whole-segment fast path. Fix: layer-extensions.yml is now excluded from the skywalking-oap jar and shipped to the distribution config/ directory, so an operator-edited copy is no longer shadowed by the empty template bundled in the jar. Fix: the v2 MAL compiler now resolves custom layers referenced as Layer.NAME in an expression, which previously failed code generation because a custom layer has no generated Layer.* static field. Fix Envoy ALS rendering for the LAL live-debugger and the persisted log content — an Istio metadata-exchange peer in filter_state_objects is now decoded into readable peer metadata instead of an opaque jsonformat-failed envelope. Surface the effective BanyanDB configuration (bydb.yml / bydb-topn.yml) in the /debugging/config/dump admin API, which previously showed an empty storage.banyandb block. Fix: an MQE top_n(metric, N, order, attrX='value') query whose attribute is not a column of the target metric now returns a descriptive MQE error instead of a raw storage IOException. Migrate all BanyanDB storage read queries from the typed query-builder API to BydbQL. Fix: BanyanDB queries no longer silently truncate at the storage engine\u0026rsquo;s implicit row cap (100 rows for measures, 20 for streams/traces), which was applied after GROUP BY on read paths where OAP sent no limit. Support BanyanDB\u0026rsquo;s group-scoped trace retention pipeline in bydb.yml — the trace and zipkinTrace groups gain a pipeline block that OAP pushes onto the BanyanDB group as a TracePipelineConfig, letting a sampler plugin drop traces inside the data node during Hot-phase compaction. Fix: a blank value in bydb.yml (key: with nothing after it) aborted OAP startup with an opaque NullPointerException; the loader now skips blank entries and leaves the field at its default. Route LAL rules within a layer by their input type, so a single layer can host rules over different proto inputs and LogFilterListener skips any rule whose type doesn\u0026rsquo;t match the incoming log. Fix the PagerDuty alarm hook to default its Events API v2 endpoint to https://events.pagerduty.com/v2/enqueue. Fix HttpAlarmCallback logging a successful alarm delivery as a failure — the shared HTTP hook helper treated only 200 and 204 as success, so the 202 Accepted returned by asynchronous intake APIs produced an ERROR on every delivered alarm. Make the PagerDuty Events API v2 endpoint configurable through a new optional events-api-url setting on each pagerduty hook, so an EU-region account can point the hook straight at its own endpoint. Bump the default BanyanDB compatible server API version (SW_STORAGE_BANYANDB_COMPATIBLE_SERVER_API_VERSIONS) from 0.10 to 0.11. UI Add Airflow layer dashboards and menu i18n under Workflow Scheduler in Horizon UI (SWIP-7). Add the mobile menu icon and i18n labels for the iOS layer. Fix metric label rendering in multi-expression dashboard widgets. Add i18n menu labels for WeChat Mini Program and Alipay Mini Program (en / zh / es). Support the trace V1 view in the trace single page. Documentation Document the meter-analyzer-config catalog in the runtime-rule hot-update and DSL-debugging references, and add the optional layerDefinitions block and the active-files startup-failure behaviour. Update the LAL documentation with the sourceAttribute() function and layer: auto mode. Add Airflow monitoring setup documentation (SWIP-7). Add iOS app monitoring setup documentation. Add WeChat / Alipay Mini Program monitoring setup documentation, plus a client-side-monitoring section in the security guide covering public-internet ingress for mobile / browser / mini-program SDKs. Improve the downsampling documentation. Fix the docker-compose quickstart: the OAP healthcheck no longer calls curl (absent from the JRE image), and the Horizon UI service maps the correct container port. Add PHP runtime metrics (PHM) dashboard documentation. Add Node.js runtime metrics dashboard documentation. Add a BanyanDB trace tail sampling guide under \u0026ldquo;BanyanDB Exclusive Setup\u0026rdquo;. Correct the APISIX monitoring guide to align its Collector configuration and metric names with the current APISIX MAL rules and Horizon UI dashboard. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking APM 11.0.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"horizon-ui-is-now-the-official-ui\"\u003eHorizon UI is now the …\u003c/h3\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-11.0.0/","title":"Release Apache SkyWalking APM 11.0.0"},{"body":"SkyWalking BanyanDB 0.11.0 is released. Go to downloads page to find release tars.\nFeatures Breaking Change: Upgrade the API version to 0.11 for the schema revision fields and SchemaBarrierService operations added in this release. External clients and internal node communication must use API 0.11; mixed BanyanDB 0.10 and 0.11 clusters are not supported. Add batch-write phase metrics for client-held-open time, post-seal admission time, and admission deadlines, plus per-part liaison-wqueue-to-data-node file-sync latency for measure, stream, and trace. Add bounded trace plugin overhead metrics: chain decision batches by result, per-plugin execution count and duration, batch trace-count distribution, and per-plugin fail-open bypasses by reason. The configured plugin_name distinguishes links in a multi-plugin chain; segment, shard, and schema labels remain absent. The catalog works with pipeline_traces_evaluated for per-plugin time/calls per trace, latency percentiles, timeout/circuit-open rates, and link-health monitoring. The Phase 10 merge benchmark mirrors chain batches and named-link timing into JSON and HTML and rejects missing, failed, bypassed, timed-out, or circuit-open execution records. Add trace finalization sampling (PIPELINE_EVENT_FINALIZE), a best-effort background backstop for the in-merge trace retention filter. A single node-wide, concurrency-1 scanner periodically sweeps cooled segments (segEnd \u0026lt; now − finalize_grace, reopening idle-closed segments via SelectSegments(reopenClosed=true)) and force-merges each shard\u0026rsquo;s un-finalized parts through the group\u0026rsquo;s registered sampler chain, reusing the hot merge path (mergeParts + per-sidx Merge + the shard\u0026rsquo;s introducer loop) so it never touches the hot merge semaphore. Selection uses a per-part finalizeGen stamp (min-propagated on ordinary merges, written to disk before the part metadata so a crash cannot double-sample on replay); per-shard state (finalize.json) holds the generation, cooldown clock, round count, terminal flag, and an arrival-based unsampled-bytes counter incremented O(1) on the flush path. Re-rounds are bounded by an absolute FLOOR (8 MiB), a RATIO (0.10 of segment size), a cooldown (= finalize_grace), and a hard max_finalize_rounds (8). Coverage is best-effort — misses (e.g. a segment TTL-deleted before finalization) are accepted, not guaranteed. Enabled per group via enabled_events containing PIPELINE_EVENT_FINALIZE; the settling window is finalize_grace with a fixed 5m engine fallback. Finalize rounds emit the merge metrics under type=\u0026quot;finalize\u0026quot;/lane=\u0026quot;finalize\u0026quot;. Bound the trace merge drop-set so a large sampled merge cannot exhaust the heap. The merge-wide dropped-trace-ID collector was unbounded: because a shard\u0026rsquo;s first finalize round selects every cooled part into one merge, an 18M-drop set reaches ~1.3 GiB live and ~2.6 GiB reserved heap (append doubling plus a single contiguous lookup index), inside a process also serving queries. The bound is applied to the sampling decision, not to the pruning predicate: once a merge\u0026rsquo;s drop set is full every further proposed drop is retained instead of recorded, so the set stays complete with respect to the drops actually performed and each sidx still contains exactly its part\u0026rsquo;s retained traces — no orphan entries, none missing. The fragment guard is skipped for a trace that will be retained, since it exists only to confirm drops. Every merge is charged the same ceiling, finalize included, resolved from the memory protector as limit/(16*CPUs) so the aggregate across concurrent merges stays near limit/16; bytes-per-entry is derived from the observed trace-ID length rather than a constant, so a deployment with long service-prefixed IDs cannot overshoot the budget. A node whose protector is disabled (no --allowed-bytes and no valid cgroup limit) gets a flat fallback ceiling regardless of RAM — there the lever is the protector limit, not the machine\u0026rsquo;s memory. The accepted cost is bounded deletion: a capped finalize round leaves traces undeleted and they are only revisited by the existing re-round path, which is capped at max_finalize_rounds (8) per shard for its lifetime, so a very large shard can retain traces the sampler wanted dropped. Finalize I/O is unchanged, because the round count is unchanged. New group-level metrics make the loss visible without segment or shard cardinality: pipeline_drop_set_budget_bytes, a lane-split pipeline_drop_set_entries histogram emitted whether or not a merge caps, pipeline_traces_retained_by_ceiling, pipeline_merges_ceiling_reached{lane}, and scan-level finalize_rounds/finalize_terminal worst-case gauges. Design: docs/design/trace-drop-set-bounding.md. Redesign queue (queue_pub/queue_sub) metrics around a uniform model with per-message and per-batch-stream catalogs. Use operation and group labels, add error_type and remote-endpoint labels, and propagate sender identity and group on the wire. [Breaking Change] Previous queue metric families, chunk-ordering metrics, and the topic label are removed; update dashboards and alerts. Stamp the lifecycle\u0026rsquo;s tier-migration publisher\u0026rsquo;s identity onto the wire so the receiving data node records a non-empty remote_node/remote_role/remote_tier on its banyandb_queue_sub_total_finished series. The lifecycle\u0026rsquo;s parseGroup resolves the lifecycle\u0026rsquo;s self identity by matching the lifecycle pod\u0026rsquo;s hostname (POD_NAME via the K8s downward API, falling back to os.Hostname() — same precedence as nativeNodeContext at banyand/backup/lifecycle/service.go) against the data-node registry\u0026rsquo;s GrpcAddress (host-portion match with loopback-alias and IP-literal normalization, via hostMatches at banyand/backup/lifecycle/steps.go) — Metadata.Name becomes remote_node, Labels[\u0026quot;type\u0026quot;] becomes remote_tier — and calls SetSelfNode(senderNode, \u0026quot;lifecycle\u0026quot;, senderTier) on the migration publisher. The previous --grpc-addr address-match (Pass 1) and --node-labels superset-match (Pass 2-3) fallbacks are replaced by this single host-based match because they failed on the production cluster where the data pod\u0026rsquo;s GrpcAddress is a headless-service FQDN but the lifecycle\u0026rsquo;s --grpc-addr is the loopback. Mirrors the liaison\u0026rsquo;s existing SetSelfNode(node.NodeID, \u0026quot;liaison\u0026quot;, liaisonTier) call in pkg/cmdsetup/liaison.go:170-171; no new CLI flags are introduced. Add labeled lifecycle health metrics (banyandb_lifecycle_cycles_total, banyandb_lifecycle_last_run_timestamp_seconds, banyandb_lifecycle_last_run_success) keyed by remote_node, remote_role, remote_tier, and group; stamp the last-run gauges atomically at cycle end and remove their previous label tuple. [Breaking Change] Update dashboards and alerts for the labeled series. Remove banyandb_lifecycle_self_identity_resolution_total. The regression-detection role moves to the now-labeled banyandb_lifecycle_cycles_total{remote_node!=\u0026quot;\u0026quot;} (an empty remote_node series means the registry match failed for every group, the bug the old counter caught), plus the existing receiver-side count of empty remote_node on lifecycle banyandb_queue_sub_total_finished series. The wire-level cluster.v1.SendRequest sender-identity fields are unchanged. Vectorized measure query path is now enabled by default. The columnar pipeline replaces per-row protobuf serialization in NewMIterator, cutting allocations and ns/op for scan-heavy measure queries; gRPC wire format (*measurev1.InternalDataPoint) is byte-identical. Single-node coverage is complete: scan, GroupBy+Agg via BatchAggregation, scalar reduce (Agg without GroupBy), raw GroupBy (without Agg), implicit projection coverage for GroupBy/Agg fields, TopN/BottomN, order_by (via logical.ParseOrderBy, mirroring the row path\u0026rsquo;s PushDownOrder rule), queries with hidden criteria tags, and boundary-error parity (nil time range, unknown projection, empty result) all resolve through the vec dispatch with row-path-equivalent semantics (SUM/COUNT/MIN/MAX/MEAN type semantics, first-seen carry-forward of non-key projected tags, canonical validation errors). Validated by a 6h production soak (byte-identical parity, zero divergences) and the per-workload bench gates. Distributed Map-mode partial aggregation (multi-node GroupBy+Agg / TopN), multi-measure (multi-group) requests, and non-vectorized backends continue to flow through the row path pending the distributed vectorized query work. Rollback: pass --measure-vectorized-enabled=false on the standalone or data-node command line and restart; the row path resumes immediately. Enable vectorized stream and trace queries by default. Stream uses the columnar pipeline and native liaison-to-data frame for supported timestamp-, index-, criteria-, and multi-group queries; unsupported shapes and traced queries fall back to row or protobuf execution. Roll back with --stream-vectorized-enabled=false or --trace-vectorized-enabled=false. Add banyandb_vec_frame_encoded{engine} and banyandb_vec_frame_decoded{engine} gauges (engine: measure/stream/trace), reporting how many query responses this process has put on the wire as a native columnar frame and how many it has decoded from one. Which wire format a node is actually speaking was otherwise invisible from outside the process, and the 0.11 upgrade-ordering rule depends on exactly that: a data node emits frames only when it is distributed AND its engine flag is on, so a standalone server reports zero encodes however the flag is set. Both series are cumulative and, being labelled, appear on /metrics only after the first collection tick. A run whose encode count stays flat on the data nodes is falling back to protobuf. [Breaking Change] Upgrade liaison nodes before data nodes when rolling out vectorized query paths. A 0.11 liaison accepts both columnar frames and protobuf, but an older liaison cannot decode frames emitted by an upgraded data node. Standalone deployments are unaffected. Add validation to ensure Measure\u0026rsquo;s ShardingKey contains all Entity tags to guarantee entity locality. Organize access logs under a dedicated \u0026ldquo;accesslog\u0026rdquo; subdirectory to improve log organization and separation from other application data. Collect BanyanDB data on e2e test failure for CI debugging. Add log query e2e test. Sync lifecycle e2e test from SkyWalking stages test. Add noDuplicates verification to all e2e expected files to detect duplicate data in query results. Add a program-generated trace query integration-test framework under test/cases/trace/cmd/{generate,capture}: layered case generation (criteria leaves, AND/OR trees, and feature-pairwise across the traceID-lookup and order-based query modes), env-gated golden capture, and a shared SeedAll seeder — mirroring the measure test-case framework. Persist segment end time in per-segment metadata so boundaries don\u0026rsquo;t shift across restarts or config changes. Introduce fair fast/slow lane scheduling for trace part merges to prevent short merges from being blocked by long-running merges; expose queue wait time as total_merge_queue_latency. [Breaking Change] Remove etcd components. The property-based schema registry is now the only supported mode. All --etcd-* CLI flags have been removed. The --namespace CLI flag has been removed (it previously configured the etcd key prefix). The --node-discovery-mode flag no longer accepts etcd (supported values: none, dns, file). The --schema-registry-mode flag only accepts property. Implement panic diagnostics and the FODC crash reporting pipeline. Schema consistency (Phase 1): introduce client-observable revision and propagation primitives. All gates are opt-in and zero-valued requests preserve prior behavior. Add mod_revision to Group / IndexRule / IndexRuleBinding / TopNAggregation Create and Update responses. Add delete_time to all *ServiceDeleteResponse messages so clients can observe tombstones. Add created_at to Stream / Measure / Trace / Property / IndexRule / IndexRuleBinding / TopNAggregation / Group; preserved across updates. Add STATUS_SCHEMA_NOT_APPLIED (10) for writes/queries whose mod_revision is ahead of the server cache. Add three-way write-path ModRevision gate on Stream and Measure write RPCs (\u0026lt; expired, == succeed, \u0026gt; not-applied, 0 skipped). Add per-group query-path gate via QueryRequest.group_mod_revisions and QueryResponse.group_statuses for Stream / Measure / Trace queries. Add automatic time-range clamp max(time_range.start, schema.created_at); multi-group uses the maximum across queried groups; nil for pre-upgrade schemas is a no-op. Add SchemaBarrierService with AwaitRevisionApplied, AwaitSchemaApplied, AwaitSchemaDeleted (10 000-key cap, timeout-bounded, returns laggards on expiry). Add tombstone retention/GC (default 7 days, configurable via --schema-server-tombstone-retention) with a per-cache count cap to bound memory under bulk deletes. Reject Create with updated_at \u0026lt;= tombstone.delete_time to prevent replayed creates from overwriting newer deletes. Guard pkg/schema/cache against out-of-order EventDelete events; expose monotonic LatestModRevision watermark. Schema consistency (Phase 2): add cluster-wide barriers that fan out AwaitRevisionApplied, AwaitSchemaApplied, and AwaitSchemaDeleted across liaison and data nodes through NodeSchemaStatusService, while safely handling mixed-version, membership, and timeout cases. Align schema barriers and write/query revision gates with each node\u0026rsquo;s executor cache, including correct watermark advancement and no-op update revisions. Add schema-barrier metrics, structured access logs, integration coverage, and a load-test harness. Introduce a migration tool with copy, verify, and analyze subcommands for measure and stream data. Support displaying a measure\u0026rsquo;s indexed tags in the dump tool, resolved per part so peak memory is bounded by the part rather than a segment-wide series map. Make segment lifecycle safe for snapshots, backups, and inspection: segments retain a dormant state while idle, and these operations no longer reopen closed segments or race idle cleanup. Add opt-in vectorized measure query tracing over raw-frame distributed queries, including a trace envelope and fixed trace-label vocabulary. Speed up GCS backup uploads: write each object and its checksum metadata in one request, dropping the per-object Update round-trip. Lifecycle migration now archives rows whose measure/stream schema was deleted from the registry, instead of aborting the group. Canopy M3 metadata CRUD: add IndexRule + IndexRuleBinding create/edit/delete forms with admin-only edit/delete, pure validate*() module covering 8 input groups (group, stream, measure, trace, index rule, index rule binding, interval, lifecycle stages), useFocusTrap + useDirtyGuard modal hooks, live m3-indexrule e2e (CRUD round-trip + MF2 server-authority negative), and opt-in m3-handoff e2e + zero-dep static server for the design-bundle comparison. Prevent lifecycle migration receiver OOMs with memory-efficient external series-index deduplication, receiver load shedding and bounded memory waits, and sender retries for transient failures; incomplete parts resume in the next cycle. Add a storage-node in-merge trace-retention filter (PIPELINE_EVENT_MERGE) that evaluates per-group sampler chains and safely drops non-retained traces from core and secondary-index parts. Configure samplers dynamically through each group\u0026rsquo;s pipeline, with bounded staging, fail-open error handling, and runtime register, update, and removal support. Add FODC memory-pressure pprof capture: agent grabs heap/goroutine profiles when RSS nears the cgroup limit, served via the proxy. Add fuzzy resource search to the From row: type a name and jump to any measure / stream / trace / Top-N across every group, with contiguous-run + word-boundary scoring, highlighted matches, arrow/Enter/Escape navigation, and click-outside dismiss. Add bounded plugin telemetry (meter + logger) to the trace-pipeline sampler SDK. A sampler plugin opts in by implementing sdk.HostAware; the engine calls UseHost(h) once per group before the first Decide, delivering a scoped sdk.Host the plugin may cache and use in Decide. The host enforces all resource limits: metric names are forced under the banyandb_trace_pipeline_plugin_ prefix; {group, plugin_name} labels are always present; per-plugin cardinality is capped at 100 series (overflow counted under _overflow); log throughput is capped at 50 lines/s burst 100, messages at 1 KiB, and 16 structured fields per call. Per-group delivery means a HostAware plugin receives a distinct Host per group so attribution is correct by construction. Series are reclaimed on pipeline teardown via Delete. No ABI bump — delivery is via type assertion (sdk.ABIVersion remains 1). Reference plugins _telemetrysampler (well-behaved) and _faultysampler (adversarial bounds verification) are in test/plugins/; build with make build-trace-pipeline-telemetry-plugins. Add the first-party trace-retention sampler plugins sw-trace-sampler.so and zipkin-trace-sampler.so (plugins/skywalking/), implementing the Scenario 6.1/6.2 keep logic on SkyWalking\u0026rsquo;s two trace schemas. Both compile the shared plugins/skywalking/internal/tracesampler engine and differ only in a Schema value naming where each schema keeps the rule inputs: the flattened searchable-tag array (tags for segments, query for Zipkin), the error signal (a real is_error column vs. Zipkin\u0026rsquo;s conventional error key inside query), and the duration columns (start_time/latency in ms vs. timestamp_millis/duration in µs, normalized via Schema.DurationTagNanosPerUnit). Config keys are durationThresholdMs, keepErrors (with an errorTag override), keepTagRules, and healthySampleRate: a trace is kept when any sure-keep rule matches, otherwise a deterministic FNV-1a hash of the trace ID admits healthySampleRate of the remainder — stable across the merge and finalize passes, so a trace is never half-kept. durationThresholdMs tests the end-to-end envelope max(start + duration) − min(start) over the trace\u0026rsquo;s rows, catching traces slow only through sequential spans, rather than any single span\u0026rsquo;s duration or the intrinsic MaxTS − MinTS (which is the spread of per-row start timestamps, and 0 for a single-row trace). Because OAP flattens every searchable tag into one string-array column, every keepTagRules entry resolves there; a rule naming a first-class column could never match and is rejected at admission, with the array-column workaround named in the error. keepTagRules also accepts a compact key=value,key=~regex,key string so a rule set fits in an environment variable. Unknown and empty configs are rejected rather than ignored — every option is a keep rule, so a key that silently missed would drop the whole group — and any predicate that cannot be evaluated keeps the trace. Built by make build-plugins and shipped in the -plugins-carrier image; see plugins/README.md for the config reference. Remove the latencystatussampler first-party plugin. It projected duration and status: status exists in neither shipped trace schema and duration only in Zipkin (in µs, so a thresholdMs there was off by 1000x), so it could never match live data and — failing open — kept every trace it was given. It shipped in the -plugins-carrier image beside the two real samplers, leaving three first-party plugins of which one silently retained everything. The near-identical copy under test/plugins/_latencystatussampler is unaffected and still backs the trace-pipeline integration suites, the plugin-sidecar test and sdktest\u0026rsquo;s LoadSO coverage; the worked-example references in docs/operation/plugins*.md now point at sw-trace-sampler. Add a Claude/Codex plugin packaging the BanyanDB MCP server with a parse-only validate_bydbql tool (backed by the mcp/tools/bydbql-parse Go validator) and a BydbQL skill for read-only natural-language-to-BydbQL generation over STREAM/MEASURE/TRACE/PROPERTY resources. Introduce positional parameter binding (? placeholders) into BydbQL to eliminate QL injection. Add reusable BydbQL binding and prepared-statement caching on the gRPC query path, with bounded cache and top-K retention, cache and slow-query observability, and redacted bound parameters in the latest slow-query log. Add an FODC schema consistency check comparing registry, cache, and runtime fingerprints across all nodes via lifecycle inspection. Bug Fixes Prevent long-lived client write streams from failing whole liaison batches when the write timeout expires; start the liaison admission timeout only after the batch is sealed and raise its default to two minutes. (apache/skywalking#13982) Stabilize measure, stream, and trace snapshot tests by waiting for persisted manifests and all flushed memory parts rather than directory creation or the latest snapshot creator, eliminating asynchronous-flush and fallback races. Prevent merge-time trace sampling from dropping fragments when the same trace ID may remain in unselected parts. Provisional drops are now checked against time bounds and trace-ID Bloom filters in candidate outside parts whose time bounds intersect the trace\u0026rsquo;s max-fragment-gap-expanded range, revalidated before publication, and retained on uncertainty; the default merge grace is two hours. FODC: namespace dynamic node labels as node_*, continuously resolve node roles and labels, and prevent oversized metric uploads and ghost ROLE_UNSPECIFIED series. Close BanyanDB merge write-path durability gap that allowed torn parts to be created by a crash between data write and metadata commit. Metadata files (metadata.json for trace/measure/stream, manifest.json for sidx, plus traceID.filter and tag.type) now go through a new WriteAtomic (write-tmp + fsync + rename + fsync-dir) sequence; data writers (seqWriter.Close, localFileSystem.Write) now propagate fdatasync errors instead of silently dropping them. mustOpenFilePart / mustOpenPart in each engine cleans up safe post-rename .tmp leftovers on open. (#13862, root cause for #13861) Fix bydbctl command tests using global stdout capture, which caused race-enabled runs to corrupt captured command output. Fix flaky trace query filtering caused by non-deterministic sidx tag ordering and add consistency checks for integration query cases. Fix index-mode measure queries returning documents outside the requested time range when a widened segment overlaps the query window. MCP: Add validation for properties and harden the mcp server. Make property-schema client connections recover after data-node restarts and add periodic health checks. Fix flaky on-disk integration tests caused by Ginkgo v2 random container shuffling closing gRPC connections prematurely. Fix snapshot error when there is no data in a segment. ui: fix query editor refresh/reset behavior and BydbQL keyword highlighting. Disable the rotation task on warm and cold nodes to prevent incorrect segment boundaries during lifecycle migration. Prevent epoch-dated segment directories (seg-19700101) from being created by zero timestamps in distributed sync paths. Fix SIDX streaming sync sending SegmentID as MinTimestamp instead of the actual timestamp, causing sync failures on the receiving node. Fix handoff controller TOCTOU race allowing disk size limit bypass, and populate sidx MinTimestamp/MaxTimestamp during replay to prevent corrupt segment creation on recovered nodes. Delete orphaned parts when no snapshot references them during tsTable initialization. Extract shared LocateAll on NodeRegistry to ensure resolveAssignments and syncer GetNodes always produce identical node lists, preventing liaison from enqueuing parts to online/healthy data nodes. Add validation for MATCH and IN conditions in inverted index query builder, and handle nil OR branch when all entities are specific. Fix wrong backup path of schema property. Fix lifecycle migration failure when the target stage has close: true. Fix stale sync request blocking watch session channel, causing repeated \u0026ldquo;channel full, skipping session\u0026rdquo; errors when a watch stream is in backoff. Fix nil pointer panic in disk monitor when group schema is not yet initialized during early startup, and ensure monitor loop survives recovered panics. Fix FileSystemError not satisfying errors.Is(err, io/fs.ErrNotExist), which prevented the segment controller from cleaning up half-born segment directories and left groups in a permanent zombie state after a crash or partial sync. Fix lifecycle migration panic when a stream shard\u0026rsquo;s snapshot has no element index (idx/) directory. Avoid FODC lifecycle inspection failing on busy data nodes by raising the per-broadcast CollectDataInfo / CollectLiaisonInfo deadline from 5s to 30s and parallelizing per-group inspection in the cluster-internal InspectAll. Fix deadlock when fodc-agent reconnects to fodc-proxy after a pod rotation. Fix flaky TestCollectWithPartialClosedSegments by raising SegmentIdleTimeout so wall-clock variance on slow CI does not mark still-open segments as idle. Fix FODC lifecycle cache poisoning where transient InspectAll failures were cached for 10 minutes and masked liaison recovery; raise FODC agent and proxy timeouts from 10s to 40s. Fix FODC /cluster/lifecycle dropping zero-valued group fields (e.g. replicas=0, close=false) under encoding/json + omitempty; switch to protojson so all fields are emitted (nil nested messages serialize as null). Fix trace block_writer panic on out-of-order timestamps within the same traceID, which dropped one trace-write batch per panic in multi-agent SkyWalking deployments. Spans of a single trace originate from independently-clocked services, and trace storage is organized by traceID rather than timestamp, so per-traceID timestamp monotonicity is not a writer invariant. Add GroupLifecycleInfo.errors to surface per-group collection failures from FODC InspectAll instead of silently dropping the affected node entry. Fix CollectDataInfo and CollectLiaisonInfo not handling CATALOG_PROPERTY groups. Fix lifecycle migration where the receiving node could create segments shorter than the configured SegmentInterval. Fail fast on incompatible storage version at boot. Previously the server would start in a degraded SERVING state with affected groups un-loaded because the property schema-registry retry loop swallowed the version-incompatibility panic. Compatible versions are listed in banyand/internal/storage/versions.yml. Release bluge index writers on segment rotation so analysisWorker pools sized from GOMAXPROCS don\u0026rsquo;t accumulate across rotations. Two layered defects kept the existing idle-segment reclaim path from running: segmentIdleTimeout defaulted to 0 (which disabled the 10-minute reclaim ticker), and incRef refreshed lastAccessed on every rotation tick so closeIdleSegments never observed an idle segment. Defaults to time.Hour, moves the lastAccessed bump to real read/write call sites, and rewrites closeIdleSegments to take its own CAS-bumped snapshot so a concurrent reopen cannot have its only ref dropped under the reclaimer (apache/skywalking#13874). Fix incorrect counts and missing trace fields in the lifecycle migration report. Fix lifecycle migration placing data in the wrong target segment when the source segment interval is not a multiple of the target stage\u0026rsquo;s interval, by row-level replaying parts that straddle a target-segment boundary instead of chunk-copying them into a single segment. Fix trace query identity-tag projection: when trace_id/span_id are explicitly projected, reconstruct them from span identity at response build time instead of requesting them as stored tags, and preserve tag order with null-filled per-span value alignment in the distributed trace result iterator. Fix measure, stream, and trace queries returning data from segments already expired by the TTL. Retention removes a segment only on its next scheduled run, so a fully expired segment can linger on disk and keep serving TTL-expired data; queries now skip segments whose whole time range is past the retention deadline, matching retention\u0026rsquo;s own removal condition. Trace storage metrics now expose the storage sub-scope, matching the stream_storage_* naming. The StorageMetricsFactory for trace switched from the root trace scope to trace.storage, so per-segment inverted-index metrics (inverted_index_total_updates, inverted_index_total_doc_count, inverted_index_total_term_searchers_started) are now emitted as banyandb_trace_storage_* instead of banyandb_trace_*, aligning the dashboard query names. Other trace metrics (trace_tst_*, trace_scheduler_*) are unchanged. Fix FODC proxy corrupting Prometheus metric types. The agent dropped the # TYPE line while parsing banyandb /metrics, the StreamMetrics proto carried no type field, and the proxy guessed the type from a name-suffix heuristic — downgrading counters to gauge, mislabeling _count-suffixed counters as histograms, and splitting summaries into two conflicting # TYPE lines. Capture the type with the Prometheus expfmt parser, store it in the flight recorder, thread it through a new Metric.type enum over gRPC, and emit the real type from the proxy; pre-upgrade (untyped) samples fold into the matching typed family so a mixed-version rollout never emits two conflicting # TYPE lines for one metric. Fix lifecycle row-replay OOM on large measure parts by streaming the dump reader, pooling size-classed marshal buffers, and bounding in-flight batch bytes (default 32 MiB); peak heap drops ~1.5 GB→~296 MB. Consolidate lifecycle migration report errors into a single flat list of structured, stage-aware entries. Fix backup container OOM from overlapping scheduled runs; serialize runs and upload small snapshot files concurrently. Fix block metadata reset before unmarshal for stream and trace. Lifecycle only handles stream/measure/trace snapshots, skipping other catalogs. Align code-mode editor with the handoff toolbar + footer (move \u0026ldquo;← Builder\u0026rdquo; back button into the toolbar, switch the footer to qb-foot/qb-foot-row, restore the ⌘/Ctrl+↵ kbd hint on Run, switch markup to the handoff class names, and drop the syntax-highlight \u0026lt;pre\u0026gt; that double-rendered side-by-side with the textarea). Fix liaison goroutine leak from uncanceled gRPC SyncPart stream context in write-queue part sync. Fix group update overwriting a warm/cold node\u0026rsquo;s stage-resolved segment interval, TTL and shard count with the group default. Re-index bound resources when an index rule or binding is deleted, clearing stale index config. Deleting one TopN aggregation no longer tears down sibling aggregations on the same source measure. Purge a deleted group\u0026rsquo;s resource, index-rule and binding cache entries to avoid dangling references. Clear a trace subject\u0026rsquo;s index when its last index rule or binding is removed. Allow * as a non-initial character in BydbQL identifiers, so resources named with * become queryable, not just writable. Enable periodic health checks on the queue client (--\u0026lt;prefix\u0026gt;-client-health-check-interval, default 10s), evicting dead data nodes proactively. Build the plugin .so with -tags slim to match the host binary\u0026rsquo;s ABI. Without this, pkg/pool has build-tag-dependent code (tracker.go vs tracker_stub.go) and the plugin-sidecar E2E failed with \u0026ldquo;plugin was built with a different version of package pkg/pool\u0026rdquo;. Make the trace merge loop survive a persistently failing merge instead of destroying the node. A production cluster wedged when a few unreadable parts made every merge selection fail: each failed attempt leaked its partially-written output directory (the block writer creates it before the merge runs), which exhausted the volume\u0026rsquo;s inodes at 8% byte usage; the resulting disk-full panic then killed the merge lane workers, which are recovered but never respawned, so the dispatcher blocked on the lane channel forever and the parts stayed pinned in flight. Four layered fixes: (1) mergeParts removes the output directory on any error return or panic, in trace, stream, measure, and sidx; (2) merge-read failures are wrapped in a typed error carrying the failing part\u0026rsquo;s identity, and a part accumulating three consecutive attributable failures is excluded from both the hot merge selector and the finalize-round selector — it stays in the snapshot (queryable, TTL-deleted as usual) and the registry is swept when parts leave the snapshot; (3) merge dispatch backs off exponentially from 1s to a 60s cap after consecutive failures, reset on any success, with wave mode (operator-driven controlled merges) bypassing it; (4) merge-execution panics are converted to ordinary merge errors at the lane worker and the dispatch cycle, so the semaphore release, in-flight unpin, and failure accounting always run and the goroutine survives. New metrics: total_merge_part_quarantined, merge_quarantined_parts, total_merge_backoff_seconds, and total_merge_panic_recovered. Report panics recovered by the gRPC servers through panicdiag, so they reach banyandb_panic_total{component}, the crash reporters and the artifact writer. Each server (queue/sub, liaison/grpc, property/gossip, metadata/schema/schemaserver) hand-rolled a recovery handler that only logged and returned codes.Internal, bypassing every panicdiag signal: during a production incident one data node logged 737 recovered request-path panics while banyandb_panic_total stayed at 4 (the only goroutine panics, which route through run.Go). The panics most worth alerting on were the ones nothing counted. The four copies are replaced by panicdiag.GRPCRecoveryHandler(logger, component), which reports via the new exported panicdiag.RecoverExternal \u0026ndash; the entry point for a panic the caller recovered itself \u0026ndash; and is labelled per server (grpc.queue-sub, grpc.liaison, grpc.property-gossip, grpc.schema-server). The stack is captured inside the interceptor\u0026rsquo;s recovering defer, so artifacts still show the panic site rather than the recovery machinery. Fix the measure schema-change merge test, which was time-of-day dependent rather than flaky. It wrote its two batches at now-2h and now-1h and waited for the part count to drop, but sw_metric uses a 1-day segment aligned to local midnight and a merge only ever combines parts within one segment, so any run starting within two hours of midnight split the batches across yesterday\u0026rsquo;s and today\u0026rsquo;s segments and the count could never drop. The budget had been raised twice (10x, then 20x) blaming merger starvation, which no budget could have cured; both offsets are now compressed into the elapsed part of the current segment. Fix the flaky distributed schema clamp test: it queried once immediately after AwaitRevision/AwaitApplied, which confirm the schema cache but not that the data node has loaded the group\u0026rsquo;s query topology, so the query could still fail with \u0026ldquo;group not found\u0026rdquo;. It now retries until the query resolves, the same treatment the multi-group spec in the same file already had. Stop the TopN test overriding the configured eventually timeout. It passed flags.EventuallyTimeout and then chained WithTimeout(10s), which takes precedence, so the spec always ran on a 10s budget even though CI builds the integration suites with -X ...flags.eventuallyTimeout=30s to give slow runners room. It was the only case in test/cases opting out. Await SchemaBarrier applied state after standalone schema preload so measure/stream/trace writes no longer race local caches. Fix wrong log level in trace lifecycle migration. Document Add a code-accurate, API-first \u0026ldquo;Storage \u0026amp; File Format\u0026rdquo; doc and correct stale storage/format descriptions: fix the on-disk hierarchy to group → segment → shard → part (in tsdb.md, data-model.md, clustering.md, disk-management.md, including the dump CLI path examples), correct the measure field-values file name (fv.bin, not fields.bin), clarify that the GORILLA/ZSTD enums are schema hints (the engine uses delta/dictionary + size-thresholded zstd), document the measure index_mode two-engine split and the trace span-store/sidx layout, and fix the property repair Merkle-tree SHA/snapshot-state descriptions. Replace the file-structure diagrams with inline mermaid. Refresh observability for FODC-proxy scraping: add two dashboards built for the deployment where Prometheus scrapes the FODC proxy /metrics as the single target, split by aggregation dimension — grafana-fodc-nodes.json for node/pod-level health and resources aggregated by pod_name (fleet overview, per-node health table, a \u0026ldquo;Topology: Pod-to-Pod Flows\u0026rdquo; table joining the publisher\u0026rsquo;s and subscriber\u0026rsquo;s views of each directed source → target flow into one row, resources, disk-by-path, Go runtime) and grafana-fodc-workload.json for business/data-level throughput and latency aggregated by group (cluster workload summary, liaison ingestion/query/publish + write-queue backlog, data storage/inverted-index, and the internal queue with per-operation throughput \u0026amp; p99 by group) — and remove the stale direct-scrape grafana-cluster.json. Audit and validate the observability prose/PromQL against a live cluster, replace the out-of-date self-observability-write image with an inline mermaid write-flow diagram, add a \u0026ldquo;Key Signals to Watch\u0026rdquo; section, and split the oversized observability.md into an observability/ folder (overview, logging, metrics, providers, profiling, tracing). Reconcile the FODC overview\u0026rsquo;s HTTP API surface (/metrics, /metrics-windows, /cluster/topology, /cluster/lifecycle, /diagnostics) with the proxy code and apis.md. Add a \u0026ldquo;Cluster Topology Rendering\u0026rdquo; doc (docs/operation/fodc/topology.md) that joins the FODC /cluster/topology node inventory with cluster signals to render a directed topology with two edge layers — the request pipeline (liaison→data, weighted from banyandb_queue_pub_* metrics) and the lifecycle tier migration (hot→warm→cold; path from the lifecycle service\u0026rsquo;s entries in /cluster/topology calls, weight from the banyandb_lifecycle_migration_* family). A data node and its lifecycle sidecar render inside one pod boundary with the containers labeled data / lifecycle. Includes the per-edge PromQL, three captured sample inputs (topology, queue metrics, and the migration metrics taken while the daily run was in flight), and a stdlib-only join script (render_topology.py) that emits Graphviz DOT and Mermaid; cross-linked from the FODC overview and the metrics queue reference. Chores Upgrade Go, npm, OpenTelemetry, AWS SDK, and Google Cloud dependencies. Regenerate expired TLS test certificates with 100-year validity. Set Ginkgo --repeat to 0 in the flaky-test workflow so the hourly run completes within the 50-minute timeout. Refactor the dump tool into a reusable banyand/dump parser library. Strip macOS AppleDouble (._*) and __MACOSX/ metadata from every release tarball (src, banyand, bydbctl, fodc-agent, fodc-proxy) so downstream users running make generate from a downloaded source tarball no longer hit \u0026ldquo;invalid control character\u0026rdquo; errors when buf generate walks the resource-fork files; export COPYFILE_DISABLE=1 and filter ._* files at the source. Bump Go and ui/mcp dependencies to clear Dependabot advisories: golang.org/x/net v0.52.0→v0.56.0 (CVE-2026-25680), opencontainers/runc v1.3.3→v1.3.6 (CVE-2026-41579). Refresh mcp/ui lockfiles and license attribution. Optimize the SkyWalking and Zipkin trace samplers and shared var-array decoder through deferred decoding, zero-copy string and tag handling, early tag rejection, direct scalar reads, cached rule prefixes, and escape-free decoding paths. ","excerpt":"\u003cp\u003eSkyWalking BanyanDB 0.11.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"features\"\u003eFeatures …\u003c/h3\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-0-11-0/","title":"Release Apache SkyWalking BanyanDB 0.11.0"},{"body":"SkyWalking Eyes 0.9.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed chore(deps): bump up go version by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/251 Bump golang.org/x/crypto from 0.39.0 to 0.45.0 by @dependabot[bot] in https://github.com/apache/skywalking-eyes/pull/253 Bump github.com/sirupsen/logrus from 1.9.0 to 1.9.1 by @dependabot[bot] in https://github.com/apache/skywalking-eyes/pull/254 🐛 fix(deps): Ruby: local path \u0026amp; recursive resolution of deps by @pboling in https://github.com/apache/skywalking-eyes/pull/255 Fix: Compatibility issue with Node.js 24 (related to apache/skywalking#13517) by @XL-Zhao-23 in https://github.com/apache/skywalking-eyes/pull/252 fix: improve error message formatting in license template lookup by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/256 feat: check headers concurrently by @ZacBlanco in https://github.com/apache/skywalking-eyes/pull/257 Bump github.com/go-git/go-git/v5 from 5.13.0 to 5.16.5 by @dependabot[bot] in https://github.com/apache/skywalking-eyes/pull/258 feat: add Pkl language support by @jhult in https://github.com/apache/skywalking-eyes/pull/261 feat: add Inko language support by @jhult in https://github.com/apache/skywalking-eyes/pull/262 fix: Fish completion script corrupted by INFO log message by @jhult in https://github.com/apache/skywalking-eyes/pull/259 fix: add helpful error message for git repository issues by @jhult in https://github.com/apache/skywalking-eyes/pull/260 Bump github.com/cloudflare/circl from 1.6.1 to 1.6.3 by @dependabot[bot] in https://github.com/apache/skywalking-eyes/pull/264 Bump setup-go to v6 by @adoroszlai in https://github.com/apache/skywalking-eyes/pull/265 feat: extend GoModResolver to support any valid Go *.mod file by @ivankatliarchuk in https://github.com/apache/skywalking-eyes/pull/268 ci: add golangci-lint GitHub Actions workflow by @ivankatliarchuk in https://github.com/apache/skywalking-eyes/pull/269 fix: respect specified paths in git repository by @jhult in https://github.com/apache/skywalking-eyes/pull/263 ci(action): harden header GitHub action by @domodwyer in https://github.com/apache/skywalking-eyes/pull/270 build(deps): bump github.com/go-git/go-git/v5 from 5.16.5 to 5.18.0 by @dependabot[bot] in https://github.com/apache/skywalking-eyes/pull/271 fix: corrrect RoundBracketAsterisk comment style by @bthuilot in https://github.com/apache/skywalking-eyes/pull/272 build(deps): bump github.com/go-git/go-git/v5 from 5.18.0 to 5.19.1 by @dependabot[bot] in https://github.com/apache/skywalking-eyes/pull/275 feat: add header diff command to show header differences by @wu-sheng in https://github.com/apache/skywalking-eyes/pull/276 Add support for Ruby, ERB and slim by @afriqs in https://github.com/apache/skywalking-eyes/pull/266 build(deps): bump golang.org/x/net from 0.53.0 to 0.55.0 by @dependabot[bot] in https://github.com/apache/skywalking-eyes/pull/277 build(deps): bump golang.org/x/crypto from 0.51.0 to 0.52.0 by @dependabot[bot] in https://github.com/apache/skywalking-eyes/pull/278 fix: don\u0026rsquo;t treat mid-file #! strings as Python shebangs by @cmaluend in https://github.com/apache/skywalking-eyes/pull/279 fix: pin docker/login-action to an ASF-approved commit SHA by @wu-sheng in https://github.com/apache/skywalking-eyes/pull/280 build(deps): bump github.com/go-git/go-git/v5 from 5.19.1 to 5.19.2 by @dependabot[bot] in https://github.com/apache/skywalking-eyes/pull/281 feat: resolve dependencies of pnpm-managed Node.js projects by @wu-sheng in https://github.com/apache/skywalking-eyes/pull/282 docs: fix typo seperated -\u0026gt; separated by @vaibhav8a in https://github.com/apache/skywalking-eyes/pull/283 New Contributors @XL-Zhao-23 made their first contribution in https://github.com/apache/skywalking-eyes/pull/252 @ZacBlanco made their first contribution in https://github.com/apache/skywalking-eyes/pull/257 @jhult made their first contribution in https://github.com/apache/skywalking-eyes/pull/261 @adoroszlai made their first contribution in https://github.com/apache/skywalking-eyes/pull/265 @ivankatliarchuk made their first contribution in https://github.com/apache/skywalking-eyes/pull/268 @domodwyer made their first contribution in https://github.com/apache/skywalking-eyes/pull/270 @bthuilot made their first contribution in https://github.com/apache/skywalking-eyes/pull/272 @afriqs made their first contribution in https://github.com/apache/skywalking-eyes/pull/266 @cmaluend made their first contribution in https://github.com/apache/skywalking-eyes/pull/279 @vaibhav8a made their first contribution in https://github.com/apache/skywalking-eyes/pull/283 Full Changelog: https://github.com/apache/skywalking-eyes/compare/v0.8.0...v0.9.0\n","excerpt":"\u003cp\u003eSkyWalking Eyes 0.9.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-eyes-0-9-0/","title":"Release Apache SkyWalking Eyes 0.9.0"},{"body":"SkyWalking Horizon UI 1.0.0 is released. Go to downloads page to find release tars.\nHorizon reaches 1.0 — a dark, dense, information-first console over the same OAP query protocol and MQE the previous UI used, with layer-driven dashboards you configure rather than code. This release adds three things that were not there before: an AI assistant that reads your live data and answers with real dashboard widgets, an MCP endpoint so the agent you already use can read the same data, and single sign-on with a durable login audit behind it.\nAI assistant Ask about your system in plain language and get an answer built from real widgets, not just text. A launcher on the right edge opens a chat; the assistant reads live data and streams back an ordered narrative with inline charts, top-N lists and tables drawn by the same components the dashboards use. Open it as a side drawer, expand it to a full page, or give it its own tab. Read-only, and it inherits your permissions. It lists services, reads active alarms, browses each layer\u0026rsquo;s metric catalog, drills a service down to its instances and endpoints and charts any of it — never seeing more than you can, and never changing configuration, rules or dashboards. It embeds the real product views, scoped to what you asked about. Topology, traces, logs, browser errors, deployment, API dependencies, instance map or a cross-layer hierarchy mount inside the chat with their interactions intact — click a trace and its span waterfall opens. Native SkyWalking and Zipkin tracing are both covered. Everything it shows is a snapshot, and says so. Each block carries a replay badge and its capture time, and re-renders identically when you reopen the conversation rather than quietly re-querying and showing today\u0026rsquo;s data under yesterday\u0026rsquo;s question. It can propose profiling, and only you start it. When metrics and traces cannot localise a cause it presents a decision card explaining what it found and what profiling would reveal. Nothing runs until you approve it, and only if you hold the permission; the result — flame graph, profiled trace waterfall, or network conversation graph — renders inline once collected. Guided root-cause analysis follows built-in investigation playbooks — a master method plus latency, error-rate, saturation, middleware, Kubernetes-workload and service-mesh specialisations — including following a service down into the infrastructure layer behind it. It answers in each layer\u0026rsquo;s own vocabulary, calling a Kubernetes instance a Pod and a mesh instance a Sidecar, and reads your configured warning thresholds rather than guessing what \u0026ldquo;healthy\u0026rdquo; means. Bring your own model. Off by default, vendor-neutral, configured by an administrator: any OpenAI-compatible endpoint — hosted, local or a gateway — or Amazon Bedrock. The assistant\u0026rsquo;s instructions and the starter prompts both ship with defaults you can replace entirely. Conversations are kept per user in the browser, unencrypted and labelled as such, with a usage meter, a save toggle and a clear-all. Agent access over MCP The agent you already use can read Horizon. Point Claude Code, Codex, Claude Desktop or any Model Context Protocol client at Horizon and it gets the same tools the AI assistant uses — metric catalog, figures, topology, traces, logs, Kubernetes, profiling proposals and the root-cause playbooks. The model stays on the caller\u0026rsquo;s side, so no provider and no API key are configured here. It is not a new exposure. The endpoint needs the same login as every other route, a permission gates the connection, and each tool re-checks the permission its own screen needs. Nothing served over MCP writes anything, and every tool declares itself read-only. An agent can log in through your browser instead of being handed a token. It opens Horizon\u0026rsquo;s own login page, you approve once on a consent screen, and it keeps its token from there. The screen shows the permissions the grant would really carry, filtered by what you actually hold, so it never promises access you cannot delegate. Off by default. A client that can draw gets the real widgets rather than a picture of them; a terminal client reads the data and presents it its own way. A Horizon can name itself, and that name reaches the agent, so production and staging are never told apart by guesswork. Sign-in and access control Sign in with your identity provider — Google, Okta, Entra, Keycloak or anything else speaking OpenID Connect, with each configured provider becoming a button on the login page. Providers that issue only an access token are supported too. It is additive by design: password login keeps working alongside it, so a misconfigured provider never locks you out during an incident. An identity provider says who you are; it does not decide what you may do here. New sign-ins are viewers unless you say otherwise, you decide which domains may sign in at all, and per-address and per-domain overrides raise individuals. Horizon shows the display name your directory holds, with the verified address one hover away. An account page, reached by clicking your own name — who you are, how you proved it, and which roles you hold and what they grant, so \u0026ldquo;why can I not see this page\u0026rdquo; has an answer that does not need an administrator. The roles board lists every navigation entry, marks permissions that gate nothing as reserved, and never offers a role an entry it cannot open. API tokens for callers with no browser. Scripts, CI jobs and agents authenticate under exactly the permissions each route requires; a token names a user, can never carry more than that user currently holds, and is revoked by removing the user. What one session read never reaches the next person on that browser. Signing out, signing in, or a session ending mid-use discards everything cached. Shortening the session timeout now shortens the sessions you already have, not just new ones. Fixes: sign-in accepts up to 64 characters for username and password and the form stops at the same limit; the sign-in card no longer runs off the edge below roughly 410px; source-map upload and removal are disabled with the reason on hover for operators lacking source-map:write, and removal now asks first. Login audit A durable record of who signed in, when, and from where — optional, off by default, backed by a shared database rather than a file. An hourly summary stacked by sign-in method comes first, then filters, then the list. It records only what a valid credential produced — successful sign-ins plus the two refusals that happen after authentication already succeeded. A wrong password or unknown user stays in the application log, because those are what an anonymous caller can produce at will. Nothing that could resume a session is ever recorded. Signing in never waits for the database and cannot be blocked by it. Records are written in the background; an unreachable database is invisible to the person signing in, and the page says it cannot be reached rather than showing an empty table. Token traffic is counted on its own tab, at its own grain — one row per token per hour, because stacking machine traffic beside human sign-ins let a busy script outweigh every person next to it. Reading the log needs its own permission that a wildcard does not grant, since it holds verified email and client addresses; there is no write and no delete. Fixes: a failing statistics write no longer reports the store as healthy — each write schedule is tracked separately, so the store reads unhealthy for as long as anything is failing. A failed audit write is dropped rather than retried or held, so memory does not grow for the length of an outage. Dashboards Layer dashboards are configuration, not code. Every layer\u0026rsquo;s screens are defined by a template you edit in the console — widgets, scopes, service-list columns, thresholds and labels — and published to your backend. Forty-six bundled dashboards ship ready to use. A layer\u0026rsquo;s Service, Instance and Endpoint views can each carry more than one page, each with its own URL and widgets, so a layer\u0026rsquo;s metrics need not share one screen. A page can name the entity it lists — \u0026ldquo;Brokers\u0026rdquo; rather than \u0026ldquo;Instances\u0026rdquo; — and narrow which services or instances it covers. A new tab widget packs related views into one grid slot, each tab its own small dashboard. Only the active tab is queried, so an unopened tab costs nothing. The dashboard editor works beside the canvas — picking a widget kind is a menu with descriptions, the editor pins next to the board and opens complete, and adding a widget scrolls it into view. Rows under a layer can be dragged into your own order in a live preview of the real menu. Publishing refuses a template that would break the layer, naming the field at fault and writing nothing, rather than storing it and emptying that layer\u0026rsquo;s screen for everyone. Work in progress still publishes, since a half-filled section is a normal state of an unfinished draft. Click a latency or error point on a chart to open the matching traces, pre-filtered to that service and centred on the bucket you clicked, opening slowest-first or error-only depending on the metric. Cards can render values as coloured status chips rather than bare numbers, and overview dashboards roll up a whole layer with per-widget control over the aggregation and ranking. Traces, logs and events A trace explorer with a duration-distribution scatter, a time-positioned waterfall and a span detail modal — and Zipkin traces render with the same experience as native ones, including plain-language hints for Zipkin\u0026rsquo;s annotation codes. One shareable link opens either kind. Logs and browser errors query on demand. Conditions stage until you press Run query, so a fresh tab prompts you rather than firing a broad query. Stored logs can be searched by content where the backend supports it, with the field appearing only on a backend that can actually answer it. Cross-layer inspection for raw logs, browser errors and Kubernetes pod logs. Browser errors carry source-map upload and de-obfuscation back to the original frames with a source snippet; pod logs tail a container on demand and are never stored. A per-service events popout on every layer\u0026rsquo;s service banner — agent restarts, Kubernetes events and other lifecycle records — one row per instance on a time axis, with a search box for services running hundreds of them. Fixes: a custom time range that cannot be read is now refused with the reason under the control, instead of being swapped silently for a default window so the results answer a question nobody asked. Switching service no longer leaves the previous service\u0026rsquo;s endpoints, instances or profiling segments on screen — dependent lists clear immediately, and a slow reply for a selection you have moved off is discarded rather than overwriting the current one. Profiling Five kinds of profiling in one place — trace sampling, async-profiler for JVM services, pprof for Go services, eBPF on/off-CPU, and network profiling — each with a task list, a create dialog that says upfront what it needs, and a flame graph or conversation graph for the result. Continuous profiling has a home. Arm a policy once and the task starts itself when a process crosses a threshold, with nobody present. Each target lists the instances and processes actually being evaluated and how often each has fired — the difference between a policy that is stored and one that is working. A task a policy started is visible beside the ones you started by hand, newest first. A profiling request that cannot be honoured is refused with the reason rather than quietly repaired into something more expensive. Kubernetes services gain network profiling. Languages, look and feel The whole console speaks eight languages — English plus German, Spanish, French, Japanese, Korean, Portuguese and Simplified Chinese. Product, protocol and metric names stay in their original form, because those are what operators read across the docs, the source and every other SkyWalking surface. Dashboard text is translated in the console, per language, on a page showing what the site actually renders rather than the shipped defaults — with staged drafts, a diff before you publish, and a reset to bundled. A translation belongs to its widget rather than its position, so rearranging a dashboard leaves every other widget\u0026rsquo;s translation where it was. Escape closes any dismissible panel; searchable on-theme dropdowns replace the browser\u0026rsquo;s native controls wherever a picker lists more than a handful of entries; Kubernetes tables are denser. The live debugger reads cleanly on tall and wide captures, with the frozen first column pinned as you scroll sideways, and a step that dropped a record says why in the backend\u0026rsquo;s own words. Operating Horizon The container image runs on environment variables alone — no mounted configuration file, no repackaging — and the shipped configuration file doubles as the complete, self-documenting reference for every variable. Serving Horizon under a path prefix is a first-class option. Run against a backend whose template store you cannot write, rendering every dashboard from the bundled templates and never calling the template API. The configuration surface becomes honestly read-only while metrics, traces, logs and topology work exactly as before — the supported way to run against an OAP release with no template management endpoint. Cluster Status reports what is actually reachable, testing the real path each feature calls rather than inferring health from configuration being present, so a module that is loaded but broken reads as unreachable instead of a misleading green. Configuration hot-reloads, and a rejected reload says so out loud, naming the field at fault and continuing to serve the last valid configuration. Query fan-out is tunable per deployment — batch sizes, concurrency and protective caps — with defaults matching the built-in behaviour. A strict content policy ships by default, permitting scripts only from Horizon\u0026rsquo;s own origin, forbidding inline script and refusing to be framed. Responses are not cached by the browser, so metrics, traces, logs and configuration are not left behind on a shared workstation, while the console\u0026rsquo;s own files stay cacheable. Outbound documentation links are restricted to hosts you trust. A duplicated dashboard record is reported, never resolved behind your back. A dashboard whose definition is ambiguous is hidden rather than rendered from whichever copy happened to win, and opening it by URL explains why and points at where to fix it. Fixes: the DSL editor asks before deleting a rule that has no bundled version and warns before you navigate away with unsaved YAML; the alarms and events Custom range applies on Apply rather than as soon as you open it; the OAL file viewer no longer strands you on an expired session; and cli:hash completes when you press Enter instead of waiting for end-of-input. Full release notes are here.\n","excerpt":"\u003cp\u003eSkyWalking Horizon UI 1.0.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cp\u003eHorizon reaches …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-horizon-ui-1-0-0/","title":"Release Apache SkyWalking Horizon UI 1.0.0"},{"body":"SkyWalking Java Agent 9.7.0 is released. Go to downloads page to find release tars. Changes by Version\n9.7.0 Fix plugin.http.include_http_headers not working on Spring Boot 3.x / Jakarta EE (apache/skywalking#13938). Support plugin.http.include_http_headers in the Tomcat plugin, collecting the configured request headers as the http.headers tag on Tomcat entry spans (e.g. RESTEasy on Tomcat), consistent with the Spring MVC plugin. Honors the shared plugin.http.http_headers_length_threshold. Unify the Tomcat plugins into a single tomcat plugin (Tomcat 7 - 10) and the Jetty server plugins into a single jetty-server plugin (Jetty 9 - 11), each supporting both javax.servlet and jakarta.servlet. Breaking change: plugin names tomcat-7.x/8.x and tomcat-10.x are replaced by tomcat, and jetty-server-9.x and jetty-server-11.x by jetty-server; update plugin.exclude_plugins if you reference the old names. Add a Jetty 12 server plugin (jetty-server-12.x). Jetty 12 removed the HttpChannel handle target and moved request handling to the async Server#handle(Request, Response, Callback) core API, so it needs a separate plugin from the merged jetty-server. Add a Struts 7 plugin (struts2-7.x) for Jakarta Struts, whose DefaultActionInvocation moved to org.apache.struts2. Added support for Lettuce reactive Redis commands. Add tracing support for invokeAll and invokeAny in the JDK thread pool plugin (jdk-threadpool-plugin). Add Spring AI 1.x plugin and GenAI layer. Add tracing support for vector-store retrieval operations. Fix httpclient-5.x plugin injecting sw8 propagation headers into ClickHouse HTTP requests (port 8123), causing HTTP 400. Add PROPAGATION_EXCLUDE_PORTS config to skip tracing (including header injection) for specified ports in the classic client interceptor. Add Spring RabbitMQ 2.x - 4.x plugin. Extend MySQL plugin to support MySQL Connector/J 8.4.0 and 9.x (9.0 -\u0026gt; 9.6). Extend MariaDB plugin to support MariaDB Connector/J 2.7.x. Add MariaDB 3.x plugin (all classes renamed in 3.x). Extend MongoDB 4.x plugin to support MongoDB Java Driver 4.2 -\u0026gt; 4.10. Fix db.bind_vars extraction for driver 4.9+ where InsertOperation/DeleteOperation/UpdateOperation classes were removed. Fix MongoDB 4.x plugin for driver 4.11+ where Cluster.getDescription() was removed, use getCurrentDescription() instead. Extend Feign plugin to support OpenFeign 10.x, 11.x, 12.1. Add Feign 12.2+ PathVar support (BuildTemplateByResolvingArgs moved to RequestTemplateFactoryResolver). Extend Undertow plugin to support Undertow 2.1.x, 2.2.x, 2.3.x. Extend GraphQL plugin to support graphql-java 18 -\u0026gt; 24 (20+ requires JDK 17). Extend Spring Kafka plugin to support Spring Kafka 2.4 -\u0026gt; 2.9 and 3.0 -\u0026gt; 3.3. Extend Jedis 4.x plugin to support Jedis 5.x (fix witness method for 5.x compatibility). Add Elasticsearch Java client (co.elastic.clients:elasticsearch-java) plugin for 7.16.x-9.x. Fix an issue where JDBCPluginConfig.Plugin.JDBC.SQL_BODY_MAX_LENGTH was not honored by clickhouse-0.3.1 and clickhouse-0.3.2.x plugins. Fix agent lifecycle events: the Start event now carries the service instance name, and the Shutdown event is delivered on graceful JVM exit. ServiceManager prepares/starts higher-priority BootServices first and shuts them down last (matching BootService#priority()), and the shutdown event refreshes its gRPC deadline before sending. Fix the SenderSendInterceptor in the nutz-plugins/http-1.x-plugin to avoid NPE caused by the Response status. Only publish apm-application-toolkit modules to Maven Central. Agent and plugins are distributed via download package and Docker images. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking Java Agent 9.7.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nChanges by …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-java-agent-9-7-0/","title":"Release Apache SkyWalking Java Agent 9.7.0"},{"body":"SkyWalking Python 1.3.0 is released! Go to downloads page to find release tars.\nPyPI Wheel: https://pypi.org/project/apache-skywalking/1.3.0/\nDockerHub Image: https://hub.docker.com/r/apache/skywalking-python\nWhat\u0026rsquo;s Changed chore(ci): remove changelog checkbox in pull request template by @shenxiangzhuang in https://github.com/apache/skywalking-python/pull/372 chore: fix Makefile not work in Linux and non-interactive mode by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/373 fix(plugin): add exec_module to execute the module code by @shenxiangzhuang in https://github.com/apache/skywalking-python/pull/377 Fix: kafka image in docker-compose demo by @zth9 in https://github.com/apache/skywalking-python/pull/380 fix: pin packaging dep to 25 by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/383 infra: update markdown lint check by @kevinjqliu in https://github.com/apache/skywalking-python/pull/382 feat: support Python 3.10-3.14, drop 3.8/3.9, update plugin compatibility by @wu-sheng in https://github.com/apache/skywalking-python/pull/386 feat(plugin): add urllib3 2.x support for Python 3.12+ by @wu-sheng in https://github.com/apache/skywalking-python/pull/387 feat: re-enable aiohttp/psycopg2 and add falcon v3/sanic v2 plugins by @wu-sheng in https://github.com/apache/skywalking-python/pull/389 fix: support module-level @runnable with continue_tracing() by @wu-sheng in https://github.com/apache/skywalking-python/pull/391 fix(ci): unblock CI — approved paths-filter pin + happybase test flake by @wu-sheng in https://github.com/apache/skywalking-python/pull/405 perf(demo): set a timeout on flask consumer fork HTTP calls by @basheer-cloud in https://github.com/apache/skywalking-python/pull/388 fix(ci): pin docker/* actions to ASF-approved SHAs in publish-docker by @wu-sheng in https://github.com/apache/skywalking-python/pull/406 fix: gRPC fork safety for Gunicorn prefork (grpcio \u0026gt;= 1.83), sw_grpc aio compatibility, websockets \u0026gt;= 13 support by @wu-sheng in https://github.com/apache/skywalking-python/pull/409 chore(test): upgrade the mock collector and drop the sw_fork_support seed workaround by @wu-sheng in https://github.com/apache/skywalking-python/pull/410 chore: install linters in make env, exclude *.txt from the license check by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/412 New Contributors @zth9 made their first contribution in https://github.com/apache/skywalking-python/pull/380 @kevinjqliu made their first contribution in https://github.com/apache/skywalking-python/pull/382 @basheer-cloud made their first contribution in https://github.com/apache/skywalking-python/pull/388 Full Changelog: https://github.com/apache/skywalking-python/compare/v1.2.0...v1.3.0\n","excerpt":"\u003cp\u003eSkyWalking Python 1.3.0 is released! Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePyPI Wheel\u003c/strong\u003e: …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-python-1-3-0/","title":"Release Apache SkyWalking Python 1.3.0"},{"body":"SkyWalking Go 0.7.0 is released. Go to downloads page to find release tars.\nFeatures Support Windows plugin test. Support Kafka reporter. Add recover to goroutine to prevent unexpected panics. Add mutex to fix some data race. Replace external goapi dependency with in-repo generated protocols. Support pprof profiling. Align the agent with the supported Go releases (retire EOL Go 1.19-1.23): publish Go 1.24, 1.25, 1.26 base images, bump the module go.mod floor to Go 1.24, and run the CI build, plugin, and e2e jobs on Go 1.24-1.26. Support managing toolkit spans across goroutines through SpanRef. Plugins Support gRPC v1.81.1 with Go 1.25 and Go 1.26. Documentation Bug Fixes Fix gRPC server tracing with recent internal stream types. Fix plugin interceptors bypassed on Windows. Fix wrong tracing context switch when trace ignore plugin activated. Fix data race when sending trace data to reporter. Fix multiple data races in span lifecycle, correlation context and segment collection. Add recover protection for the metrics, profile and segment-transform goroutines. Fix the RocketMQ batch consumer span: report once with one segment reference per message (new ExtractContext API). Fix nil dereference and wrong span ownership in the RocketMQ/Pulsar async producer callbacks. Fix concurrent finish flags of the gRPC streaming client and the go-micro socket close. Fix the MongoDB command span to complete through the async API (events may fire on different goroutines). Fix the gorm span storage to be per-statement and the mux response writer wrapping a nil writer. Add recover protection for the kafka instance-check and gRPC profile-fetch goroutines. Fix unsynchronized consumer-tag map access in the AMQP plugin (fatal concurrent map read/write). Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Go 0.7.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-go-0.7.0/","title":"Release Apache SkyWalking Go 0.7.0"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/tags/cloud-native/","title":"Cloud Native"},{"body":"The Meet Horizon UI series wrapped at 17/17 — a full tour of every surface of SkyWalking\u0026rsquo;s new console: the sidebar that mirrors your estate, the adaptive dashboards, topology and the 3D map, the trace and log explorers, profiling, alarms, the operations surface, access control, and config-driven customization. This is a new chapter, and it arrives with Horizon UI 1.0: an in-app AI Assistant that lets you ask your observability data in plain language instead of clicking through it.\nIt is not a chat box bolted onto a dashboard. Ask it something — \u0026ldquo;what\u0026rsquo;s unhealthy in the system right now?\u0026rdquo;, \u0026ldquo;investigate the response time for a service\u0026rdquo; — and it reads live data from your OAP backend, through the same query path the dashboards use, then streams back an ordered narrative built from the same charts, topology and tables you see everywhere else in Horizon. It is read-only, it inherits your permissions, and it is off by default until an operator enables it and points it at a model.\nYour browser does not support embedded video. Download the clip. One question, a whole investigation: the assistant triages active alarms, then draws the response-time and error-rate figures that explain them — the same widgets the dashboards use, numbered so the prose can point at them. Answers you can see, not a wall of text The assistant\u0026rsquo;s core habit is show, don\u0026rsquo;t describe. It writes a sentence or two, then draws a real figure, then interprets what that figure actually shows — and moves to the next one. Every figure is a genuine render, not a screenshot or a made-up number: line charts and single-value cards, top-N lists, labeled tables, and record lists, each chosen by the shape of the underlying metric expression. A single running Figure N counter numbers every block it draws, so the narrative can point at a figure it actually rendered — \u0026ldquo;the response-time chart above shows the spike\u0026rdquo; — and mean it.\nIt doesn\u0026rsquo;t stop at charts. When a question is about how things connect or where something ran, the assistant embeds the real feature views inline, read-only — the same components the dedicated pages use, focused for you:\nThe dependency views — a focused one-hop topology (a service\u0026rsquo;s direct upstream callers and downstream dependencies, not the whole-layer map), its cross-layer hierarchy (the Smartscape fan projecting a service up into its mesh mirror and down into its backing infrastructure), the deployment graph, the instance map for a source→destination pair, and the API-dependency chain — each zoomable and filterable in place. The signal explorers — the real Traces list with the span waterfall on a row click (native SkyWalking and Zipkin-tracing layers), the stored Logs view, and, for a browser app, the Browser errors stream with stack traces. Figure 1: Asked how a service connects, the assistant draws its dependency graph inline — no link-out, the same component the topology page uses.\nGrounded in live data — it doesn\u0026rsquo;t invent metrics Those figures are trustworthy because the assistant isn\u0026rsquo;t free-associating about your system. It answers by combining three sources, and it\u0026rsquo;s the interplay between them that turns scattered signals into one coherent picture:\nLive data — it reads through the same OAP query protocol the dashboards use, so it sees exactly what your dashboards see, scoped to your permissions and its own time window. Your layer configuration, used as a skill — the layer and overview templates are the assistant\u0026rsquo;s catalog of what each layer measures: the curated metrics and their MQE expressions, each metric\u0026rsquo;s entity scope (Service / ServiceInstance / Endpoint), and which components a layer carries. It renders those expressions verbatim rather than inventing them — so a chat figure matches the dashboard — and a layer with no trace component simply says so. SkyWalking\u0026rsquo;s model — layers, scopes, the metric catalog, topology and hierarchy — the connective tissue that lets it tie a metric to the entity it belongs to and walk a dependency edge. There\u0026rsquo;s a nice consequence: because that catalog is your configuration, edited in the Layer dashboards admin, it\u0026rsquo;s a lever you control. Add a metric to a layer or enable its traces/logs component and the assistant uses it in the next investigation; configuring your layers well is, in effect, how you extend what it can do.\nFigure 2: Before drawing anything, the assistant orients with the same building blocks the UI uses — list the layers and services, browse the metric catalog — so every figure is a real, catalog-backed query.\nGuided root-cause, with the discipline to stop Ask \u0026ldquo;what\u0026rsquo;s the root cause?\u0026rdquo; and the assistant doesn\u0026rsquo;t wander. It loads a matching investigation playbook — a master method plus focused variants for latency, error-rate / SLA, saturation, a middleware dependency, a Kubernetes workload, or a service mesh.\nWhen a service looks unhealthy, the cause may be the service itself or something it depends on. So the assistant follows the dependency graph to find where the problem originates — the root service — instead of stopping at the first symptom, separating a service\u0026rsquo;s own fault from one it inherited from a dependency it calls. From there it drills into that service\u0026rsquo;s slowest instances and endpoints, then reaches the error stack. When the trail leaves the application tier it follows the cross-layer hierarchy down into the backing infrastructure — a database, cache or queue is a topology leaf with nothing downstream, so the investigation bottoms out there and pivots to its logs, its Kubernetes hierarchy, and the network edge, where memory / disk / connection pressure actually lives.\nFor a Kubernetes workload it can pull a pod container\u0026rsquo;s on-demand logs — the error stack — and show the fetched lines inline as a read-only result. Those logs stream straight from the cluster and are never stored; the block isn\u0026rsquo;t a live console, so to see newer lines you ask again, or open the dedicated Pod Logs tab to keep a tail running. It inherits your logs:read permission and is gated on OAP, so if on-demand logs are switched off the assistant says so instead of failing.\nAnd crucially, it knows when to stop. When the available data and tools can\u0026rsquo;t localize the cause any further, it gives you a bounded, honest answer — the conclusion or best hypothesis with the evidence behind it, and a numbered list of exactly what it could not determine and why — rather than looping forever across random pods and metrics. On Kubernetes it will even hand the investigation forward with the precise kubectl commands to run and paste back.\nFigure 3: The end of an investigation is a summary, not a dead end — what\u0026rsquo;s wrong, which services, the likely pattern, and the concrete next steps to confirm it.\nRead-only, and one action you approve All of the assistant\u0026rsquo;s investigation tools are read-only — it observes and explains, and never changes configuration, rules or dashboards. Profiling is the only action, and it\u0026rsquo;s gated twice over: when metrics and traces can\u0026rsquo;t localize a cause, the assistant proposes a profiling task as a decision card — what it found, why profiling would help, what it expects to reveal — and nothing runs until you approve it in the popout, and only if you hold the profile:enable permission. You then ask it to analyze the result in a later turn, once the profile has collected; it never triggers anything on its own.\nThat posture holds all the way down. Reaching the assistant needs the ai:read permission (granted to the viewer, maintainer, operator and admin roles by default) — but that only opens the chat. Every data tool then re-checks its own read verb before it runs: metrics:read for figures, alarms:read for alarms, topology:read for the graphs, traces:read, logs:read, browser-errors:read. A tool you lack the verb for is refused and shows as a denied chip in the transcript, so the assistant can never see more than you can.\nBring your own LLM Here is the part that decides whether you can actually run this: you don\u0026rsquo;t need a frontier model. The assistant is vendor-neutral — it talks to your model through a pluggable transport, so no specific vendor is required. The default is any OpenAI-compatible endpoint (a hosted model, a self-hosted or local model, or an AI gateway); Amazon Bedrock is supported too. You set a model id, a base URL, and an API key, and that\u0026rsquo;s the integration.\nWhy can a modest model do this well? Because the observability expertise doesn\u0026rsquo;t live in the model — it lives in the tools, the metric catalog and the built-in playbooks. The model\u0026rsquo;s job is to orchestrate those tools and narrate the results, not to reason about SkyWalking from scratch (temperature is fixed at 0 for reliable tool-calling). In practice, a cost-efficient, tool-calling-capable model is usually enough — you do not have to pay for, or wait on, the largest model on the market to get a solid investigation.\nEnabling it is a small config block. In horizon.yaml, or entirely via HORIZON_AI_* environment variables:\nai: enabled: true provider: openai-compatible # or: bedrock model: \u0026#34;your-model-id\u0026#34; baseUrl: \u0026#34;https://your-endpoint/v1\u0026#34; apiKey: \u0026#34;${HORIZON_AI_API_KEY}\u0026#34; # secret — env only, redacted from logs The API key is a secret: set it via environment only, and it is redacted from logs and excluded from the audit trail. The floating AI Assistant launcher shows for every signed-in user, so the feature is discoverable — but until it\u0026rsquo;s enabled and pointed at a model, the panel opens read-only with a short \u0026ldquo;ask your administrator to set it up\u0026rdquo; notice instead of a chat box. Both the system prompt and the starter example chips ship with sensible defaults and can be replaced entirely, and a starter can embed a \u0026lt;service\u0026gt; or \u0026lt;layer\u0026gt; placeholder that opens a free-text fill-in — type an approximate name and the model resolves it to the real entity at query time.\nSafe by construction Because the assistant reads untrusted operational data — service and pod names, alarm messages, log lines, trace text — it is designed to treat everything a tool returns as data to be analyzed, never as instructions to obey. A log line that says \u0026ldquo;ignore previous instructions\u0026rdquo; is quoted and investigated, not acted on. It is instructed to keep to the observability task and not to surface its own configuration or another user\u0026rsquo;s data. Combined with read-only-by-default, a read-verb re-check on every tool, and secret redaction in logs and the audit trail, the assistant is built to be safe to point at a real production backend.\nWhere it lives, and getting started Open the assistant as a side drawer from the launcher, expand it to a full page at /ai when you want more room, or pop it into its own browser tab. It keeps its own time window (a clock in the chat header, default the last hour), and there\u0026rsquo;s no service picker — just name the service in your question. On privacy: your model credentials live only in the server configuration, never in the browser; the conversation history is kept in your browser\u0026rsquo;s local storage (capped, synced across tabs), and you can delete any conversation from the /ai History sidebar.\nThe AI Assistant ships with Horizon UI 1.0. To try it, enable the ai: block, point it at a model you already have access to, and ask it the first question you\u0026rsquo;d normally go digging for. Full configuration details are in the AI Assistant documentation.\nIf you\u0026rsquo;re new to Horizon, start with the series opener — Meet Horizon UI · 1/17 — and the getting-started guide. Then come back and let the assistant give you the guided tour of your own estate.\n","excerpt":"\u003cp\u003eThe \u003ca href=\"/blog/2026-06-21-skywalking-horizon-ui-introduction/\"\u003e\u003cstrong\u003eMeet Horizon UI\u003c/strong\u003e\u003c/a\u003e series wrapped at 17/17 — a full tour of every surface of SkyWalking\u0026rsquo;s new …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-07-06-horizon-ui-ai-assistant/","title":"Meet Horizon UI · The AI Assistant: Ask Your Observability Data in Plain Language"},{"body":"译自英文原文：Meet Horizon UI · The AI Assistant: Ask Your Observability Data in Plain Language。\nMeet Horizon UI 系列已经以 17/17 收官：它完整讲过 SkyWalking 新控制台的各个界面，从展示系统全貌的侧边栏、adaptive dashboards、topology 和 3D map，到 trace 和 log explorers、profiling、alarms、operations surface、access control，以及 config-driven customization。这篇是新的章节，随 Horizon UI 1.0 一起到来：内置的 AI Assistant 让你可以像聊天一样查询自己的可观测性数据，而不是一路点进去查。\n它不是简单地给 dashboard 加一个聊天入口。你可以直接问它：\u0026ldquo;what\u0026rsquo;s unhealthy in the system right now?\u0026quot;、\u0026ldquo;investigate the response time for a service\u0026rdquo;。它会沿用 dashboards 的查询路径，从你的 OAP backend 读取实时数据，然后流式返回一份按步骤展开的分析结果；结果里仍然是 Horizon 到处都在使用的同一套图表、拓扑和表格。它是只读的，继承你的权限，并且默认关闭，直到 operator 启用它并配置好模型。\nYour browser does not support embedded video. Download the clip. 一个问题，一次完整排查：assistant 先 triage active alarms，再画出 response-time 和 error-rate 图来解释问题；这些图使用 dashboards 同一套 widgets，并带有编号，方便正文引用。 能看见的答案，而不是一整屏文字 Assistant 的基本方式是 show, don\u0026rsquo;t describe。它先写一两句话，再画出真实图形，解释图里实际说明了什么，然后进入下一步。每个图都是实时渲染的结果，不是截图，也不是编出来的数字：line charts、single-value cards、top-N lists、带 label 的 tables 和 record lists，都会根据底层 metric expression 的形状来选择。它会给自己画出的每个 block 编上连续的 Figure N，所以正文可以引用刚刚渲染出来的图，比如 “the response-time chart above shows the spike”，而不是泛泛而谈。\n它也不只会画图。只要问题涉及对象之间的连接关系，或者某个东西运行在哪里，assistant 就会把真实功能视图以只读方式嵌进对话里。这些视图使用各个专门页面的同一套组件，只是自动聚焦到当前问题上：\nDependency views：聚焦一跳的 one-hop topology（某个 service 的直接 upstream callers 和 downstream dependencies，而不是整张 layer map）、cross-layer hierarchy（Smartscape fan，把 service 向上映射到 mesh mirror，向下映射到 backing infrastructure）、deployment graph、source→destination pair 的 instance map，以及 API-dependency chain；这些视图都可以在对话里缩放和过滤。 Signal explorers：真实的 Traces 列表，点击一行就展开 span waterfall（native SkyWalking 和 Zipkin tracing layers 都支持）；存储日志的 Logs 视图；以及浏览器应用的 Browser errors 错误流和 stack traces。 图 1：当问题是服务如何连接时，assistant 会直接在对话里画出 dependency graph；无需跳转，使用的也是 topology 页面同一套组件。\n基于实时数据，不会编造 metrics 这些 figures 可信，是因为 assistant 不是在凭空猜你的系统。它把三类信息组合起来回答问题，而正是它们之间的配合，把分散的 signals 串成一张连贯的图：\n实时数据：它通过 dashboards 使用的同一个 OAP query protocol 读取数据，所以看到的就是 dashboards 看到的内容，并且受你的权限和它自己的 time window 约束。 你的 layer 配置：assistant 会把 layer 和 overview templates 当作了解每个 layer 的目录：有哪些 curated metrics 和它们的 MQE expressions，每个 metric 属于哪个 entity scope（Service / ServiceInstance / Endpoint），以及这个 layer 带有哪些组件。它会原样渲染这些 expressions，而不是临时发明 metric 名；因此对话里的图会和 dashboard 对齐。如果某个 layer 没有 trace component，它也会直接说明。 SkyWalking 的模型：layers、scopes、metric catalog、topology 和 hierarchy。这些结构把 metric、entity 和依赖边接起来，让 assistant 能沿依赖关系继续分析。 这带来一个好处：catalog 来自你的配置，也就是你在 Layer dashboards admin 里编辑的内容，所以它也是你可以控制的扩展点。给 layer 加一个 metric，或者启用 traces/logs component，assistant 下一次排查就能用上；换句话说，把 layer 配好，就是在扩展它的能力边界。\n图 2：在画任何东西之前，assistant 会先用和 UI 相同的基础能力建立方向感：列出 layers 和 services，浏览 metric catalog；因此每个 figure 都是由 catalog 支撑的查询。\n有引导的 root-cause，也知道何时收手 你问 \u0026ldquo;what\u0026rsquo;s the root cause?\u0026quot;，assistant 不会到处乱查。它会加载匹配的 investigation playbook：一个通用方法，再加上 latency、error-rate / SLA、saturation、middleware dependency、Kubernetes workload 或 service mesh 的专门变体。\n当一个 service 看起来不健康时，原因可能出在它自己，也可能出在它调用的依赖上。所以 assistant 会沿 dependency graph 追到问题起点，也就是 root service，而不是停在第一个 symptom 上；它会区分 service 自身故障和从 dependency 传导过来的故障。找到那里后，它继续下钻到这个 service 最慢的 instances 和 endpoints，再找到 error stack。当线索离开应用层，它会沿 cross-layer hierarchy 向下进入 backing infrastructure。database、cache 或 queue 是 topology leaf，没有更下游的服务，所以 investigation 会在那里触底，然后转向它的 logs、Kubernetes hierarchy 和 network edge，因为 memory / disk / connection pressure 这类原因往往就在那里。\n对 Kubernetes workload，它可以拉取 pod container 的 on-demand logs，也就是 error stack，并把获取到的日志行作为只读结果内联展示。这些日志直接从 cluster 取回，不会被保存；这个结果块不是实时终端，所以如果要看更新的日志，可以再问一次，或者打开专门的 Pod Logs tab 保持 tail。它继承你的 logs:read 权限，并受 OAP 能力控制；如果 on-demand logs 关闭了，assistant 会直接说明，而不是静默失败。\n更关键的是，它知道什么时候该收手。当现有数据和工具无法进一步定位原因时，它会给出有边界、诚实的答案：结论或最佳假设、背后的证据，以及一个编号列表，逐条说明它无法确定什么、为什么无法确定，而不是在 pods 和 metrics 之间来回打转。对于 Kubernetes，它甚至会把调查交接下去，给出精确的 kubectl 命令，让你执行后把结果贴回来。\n图 3：一次 investigation 的结尾是一份 summary，而不是卡在那里：哪里出了问题、影响哪些 services、可能是什么模式，以及接下来如何确认。\n只读，以及一个需要你批准的动作 Assistant 的所有 investigation tools 都是只读的：它只观察和解释，不会修改 configuration、rules 或 dashboards。Profiling 是唯一的动作入口，并且有两层 gate：当 metrics 和 traces 无法定位原因时，assistant 会把 profiling task 作为 decision card 提议出来，里面写清楚它发现了什么、为什么 profiling 有帮助、它预期看到什么。只有你在 popout 里批准，并且你持有 profile:enable 权限时，任务才会运行。Profile 收集完成后，你可以在后续对话里让它分析结果；它不会自己触发任何动作。\n这个原则贯穿到底。进入 assistant 需要 ai:read 权限（内置 viewer、maintainer、operator 和 admin roles 默认都有），但这只代表能打开对话。每个 data tool 执行前都会重新检查自己的 read verb：figures 需要 metrics:read，alarms 需要 alarms:read，graphs 需要 topology:read，此外还有 traces:read、logs:read、browser-errors:read。如果你没有某个 verb，对应 tool 会被拒绝，并在对话记录里显示为 denied chip，所以 assistant 不能越过你的权限范围读取数据。\n接入你自己的 LLM 这里决定了它能不能真的落地：不需要顶级大模型。Assistant 不绑定模型厂商，通过可插拔的模型接入层访问你的模型。默认支持任何 OpenAI-compatible endpoint（hosted model、自托管或本地模型、AI gateway 都可以）；也支持 Amazon Bedrock。你只需要设置 model id、base URL 和 API key，集成就完成了。\n为什么中等规模的模型也能做好这件事？因为可观测性经验不在模型里，而在 tools、metric catalog 和内置 playbooks 里。模型的工作是编排这些 tools，并把结果组织成可读的分析，而不是从零开始理解 SkyWalking（temperature 固定为 0，以稳定 tool-calling）。实际使用中，一个高性价比、具备 tool-calling 能力的模型通常就足够；你不必为了得到可靠 investigation，就去使用或等待市场上最大的模型。\n启用它只需要一小段 config。可以写在 horizon.yaml，也可以完全通过 HORIZON_AI_* 环境变量配置：\nai: enabled: true provider: openai-compatible # or: bedrock model: \u0026#34;your-model-id\u0026#34; baseUrl: \u0026#34;https://your-endpoint/v1\u0026#34; apiKey: \u0026#34;${HORIZON_AI_API_KEY}\u0026#34; # secret — env only, redacted from logs API key 按 secret 处理：只通过环境变量设置，会在 logs 中脱敏，并排除在 audit trail 之外。浮动的 AI Assistant launcher 会对每个已登录用户显示，方便大家发现这个功能；但在它被启用并指向模型之前，panel 只会以只读方式打开，显示一条简短的 “ask your administrator to set it up” 提示，而不是 chat box。System prompt 和 starter example chips 都带有合理默认值，也可以被完整替换；starter 还可以嵌入 \u0026lt;service\u0026gt; 或 \u0026lt;layer\u0026gt; 占位符，打开一个自由输入框。你输入近似名称，模型会在查询时把它解析为真实 entity。\nSafe by construction 因为 assistant 会读取不可信的运行数据，比如 service 和 pod names、alarm messages、log lines、trace text，所以它被设计为把 tool 返回的一切都当作待分析的数据，而不是要服从的指令。一行日志如果写着 \u0026ldquo;ignore previous instructions\u0026rdquo;，它会被引用并分析，而不是被执行。Assistant 被要求留在 observability task 内，不展示自己的配置，也不展示其他用户的数据。结合默认只读、每个 tool 的 read-verb re-check，以及 logs 和 audit trail 中的 secret redaction，这个 assistant 的设计目标就是可以更放心地接到真实生产 backend 上。\n它在哪里，以及如何开始 你可以从 launcher 把 assistant 打开为 side drawer；需要更多空间时，把它展开成 /ai full page；也可以弹出到单独的 browser tab。它有自己的 time window（chat header 里有 clock，默认过去一小时），没有 service picker：直接在问题里写 service 名即可。隐私方面：你的模型凭证只存在 server configuration 里，不会进入 browser；conversation history 保存在 browser 的 local storage 中（有上限，跨 tab 同步），你可以从 /ai 的 History sidebar 删除任意 conversation。\nAI Assistant 随 Horizon UI 1.0 发布。要试用它，启用 ai: block，指向一个你已有访问权限的模型，然后问出第一个你原本会手动排查的问题。完整配置说明请看 AI Assistant 文档。\n如果你刚开始了解 Horizon，可以先读系列开篇 Meet Horizon UI · 1/17 和 getting-started guide。然后回来，让 assistant 带你走一遍你自己的系统。\n","excerpt":"\u003cp\u003e\u003cem\u003e译自英文原文：\u003ca href=\"/blog/2026-07-06-horizon-ui-ai-assistant/\"\u003eMeet Horizon UI · The AI Assistant: Ask Your Observability Data in Plain Language\u003c/a\u003e。\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"/zh/2026-06-21-skywalking-horizon-ui-introduction/\"\u003e\u003cstrong\u003eMeet …\u003c/strong\u003e\u003c/a\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-07-06-horizon-ui-ai-assistant/","title":"认识 Horizon UI · AI Assistant：用日常语言查询你的可观测性数据"},{"body":"This is the thirteenth post in the Meet Horizon UI series, and the last stop in Act 3 — operate it. The earlier operate posts were about acting on the backend — alarms, runtime rules, the live debugger, cross-layer inspect. This one turns the lens around onto the backend itself: is it healthy, how is it configured, and how long does it keep your data? Three read-only surfaces under Platform monitoring answer exactly those questions — no knobs, just the facts you need when you\u0026rsquo;re triaging.\nCluster Status: is the backend healthy, and on which port? Horizon talks to OAP over more than one channel, and Cluster Status shows the health of each one separately. It\u0026rsquo;s a two-port view with a third, independent probe:\nQuery / GraphQL (:12800) — the port every observability page rides. It reports OAP\u0026rsquo;s version, the server\u0026rsquo;s timezone and clock (next to your browser\u0026rsquo;s), and OAP\u0026rsquo;s own health score. This works on any OAP, 10.x included. Admin host (:17128) — the port the operate features need. Each admin module is probed live: the page GETs the exact REST path the feature calls and reports whether it answers, alongside the SW_… env var that enables it and what breaks if it\u0026rsquo;s down — admin-server, receiver-runtime-rule, dsl-debugging, inspect, ui-management. This host ships with OAP 11, so on a 10.x backend the pane simply isn\u0026rsquo;t there. Zipkin / OTLP (:9412) — informational: it feeds only the Zipkin trace menu, so a red dot here leaves every other page working. All three are polled independently, so one going red never drags the others down — which is precisely what makes this the first page to open when something elsewhere in Horizon looks off.\nFigure 1: Cluster Status — a two-port health view. The Query pane (:12800, any OAP) shows version, server time, and health score; the Admin pane (:17128, OAP 11) probes each admin module live with its REST path, the SW_… env var that enables it, and what breaks if it\u0026rsquo;s unreachable; the Zipkin/OTLP pane feeds only the Zipkin menu.\nOAP Configuration: what is it actually running? When a setting doesn\u0026rsquo;t behave the way you expect, the next question is \u0026ldquo;what config did OAP actually resolve?\u0026rdquo; OAP Configuration answers it without an SSH session: it reads the connected backend\u0026rsquo;s effective runtime config from the admin port\u0026rsquo;s /debugging/config/dump, grouped by module and searchable across keys and values. Secret values — passwords, tokens, access keys — are masked to ****** by OAP itself before Horizon ever sees them. It\u0026rsquo;s strictly read-only; to change a value you still edit it on the OAP side and restart. Like the admin pane above, it rides OAP 11\u0026rsquo;s admin host.\nFigure 2: OAP Configuration — the connected backend\u0026rsquo;s resolved runtime config from the admin port, grouped by module and filterable, with secrets masked to ****** by OAP. Read-only: change config on the OAP side and restart.\nData Retention: how long does it keep your data? Data Retention shows OAP\u0026rsquo;s time-to-live for each class of data, in whole days — records (trace, log, browser error, …) and metrics (metadata, minute, hour, day). And here the storage backend matters. On a flat store like Elasticsearch, each class has one retention number, full stop. On BanyanDB, data ages through configurable lifecycle stages — hot, optionally warm, optionally cold — and OAP reports the hot and warm stages as one combined, queryable window. So Horizon draws a per-class lifecycle bar instead of a single number: the hot+warm window plus the cold tail, with widths proportional to total days, so you read the relative durations at a glance.\nThere\u0026rsquo;s one operational subtlety the page calls out: the topbar Cold pill swaps reads from hot+warm to cold — it doesn\u0026rsquo;t union them — so it\u0026rsquo;s worth enabling only when the window you\u0026rsquo;re after is older than the hot+warm boundary. Data Retention reads through the standard query port, so unlike the two pages above it works on any OAP version.\nFigure 3: Data Retention — on BanyanDB, each class ages through hot → warm → cold, so Horizon draws lifecycle bars (the queryable hot+warm window plus cold, widths proportional to total days). A flat backend like Elasticsearch would show a single TTL per class instead.\nWhere it runs All three live under Platform monitoring, each gated by its own read permission — cluster:read, config:read, ttl:read — and all three are strictly read-only: this is introspection, not control. They split on the backend the same way the rest of operate does. Data Retention and the Cluster-Status Query pane ride the standard query port, so they work on any OAP, 10.x included. The Cluster-Status Admin pane and OAP Configuration read the admin host, which arrives with OAP 11 — and rather than break when it\u0026rsquo;s absent, they say so plainly (a hidden pane, a \u0026ldquo;needs the admin host\u0026rdquo; banner). (These are about the platform\u0026rsquo;s health and config; the OAP self-observability dashboards are a separate, metrics-driven story.)\nWhere to go next For the field reference — every pane, the config-dump shape, and the BanyanDB lifecycle details — see the Cluster Status, OAP Configuration, and Data Retention docs.\nThat closes Act 3 — operate it. Next up, Act 4 — govern \u0026amp; secure it, starting with Access Control \u0026amp; Security: server-enforced RBAC, LDAP/AD, the audit log, and break-glass.\n","excerpt":"\u003cp\u003eThis is the thirteenth post in the \u003ca href=\"/blog/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e series, and the last stop in \u003cstrong\u003eAct 3 — operate it\u003c/strong\u003e. …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-06-30-horizon-ui-platform-introspection/","title":"Meet Horizon UI · 13/17: Platform \u0026 Cluster Introspection"},{"body":"This is the fourteenth post in the Meet Horizon UI series, and it opens Act 4 — govern \u0026amp; secure it. Everything so far was about what Horizon shows and does; this post is about who gets to see and do it. And one architectural point matters up front: all of this — RBAC, authentication, the audit log, break-glass, themes — is Horizon\u0026rsquo;s own governance, enforced in its BFF, and it does not touch the OAP admin host. So it works the same whether your backend is OAP 10.x or 11.x.\nIt starts at the door Before anything renders, you sign in. Horizon has two authentication backends, chosen by config: Local — users and Argon2id password hashes declared in horizon.yaml — or LDAP / Active Directory, where Horizon binds against your directory and maps directory groups to Horizon roles (memberOf or a group search, your bind account does the lookup). The login screen shows which backend is live as a pill, and lets you pick a language before you\u0026rsquo;re even authenticated. Sign-in is deliberately enumeration-resistant: a wrong user, a wrong password, or a user in no mapped group all return the same \u0026ldquo;invalid credentials.\u0026rdquo;\nFigure 1: The login page — the auth-backend status pill (here Local users), a pre-auth locale selector, and the credentials form. Authentication is the first gate; everything past it is role-gated.\nFor admins, an Auth status page mirrors the backend\u0026rsquo;s health: the provider and hash algorithm, where users are defined, whether break-glass is armed, and how many sessions are currently active. One thing to read correctly — that active-session count is the sessions on the Horizon BFF you\u0026rsquo;re talking to, not a cluster-wide total; sessions live in each BFF node\u0026rsquo;s memory.\nFigure 2: Auth status — the backend\u0026rsquo;s health at a glance: provider (here local, Argon2id), where users are defined, whether break-glass is armed, and the active-session count on this Horizon BFF (not a cluster-wide figure).\nWho can do what Past the door, every action is gated by a permission. Horizon ships four cumulative roles: viewer reads the observability data (dashboards, traces, logs, alarms); maintainer adds the platform-monitoring surfaces (cluster status, inspect, retention, config); operator adds dashboards, alarms, runtime rules, and the diagnostics tools; admin holds everything, including user and access management. Permissions themselves are dot-namespaced verbs — metrics:read, rule:write:structural, inspect:read — with wildcards (*, rule:*, *:read).\nThe Roles \u0026amp; Permissions board lays the whole model out: a menu-visibility matrix (which sidebar items each role sees, and the read verb that gates each) on top, then a per-area action matrix marking exactly what viewer / maintainer / operator / admin can each do, area by area. It\u0026rsquo;s read-only — roles are defined in horizon.yaml and hot-reload on save, no restart.\nFigure 3: The top of the Roles \u0026amp; Permissions board — the role cards and the menu-visibility matrix (each sidebar item × role, gated by the read verb in the last column). Per-area action matrices follow below. Read-only: roles live in horizon.yaml and hot-reload.\nThe enforcement is server-side. The BFF gates every protected request — 401 if you\u0026rsquo;re not authenticated, 403 if your session lacks the verb — and the route→verb table is mandatory: a route with no entry fails the build, so nothing is ever accidentally left open. The UI mirrors the same verb logic to hide menus you can\u0026rsquo;t use, but that\u0026rsquo;s defense-in-depth, not the gate; a forged UI can\u0026rsquo;t escalate, because the BFF is the authority.\nWho\u0026rsquo;s signed in The Users page lists every account Horizon knows — local, LDAP, and break-glass — with its source, assigned roles, last sign-in, and IP. It\u0026rsquo;s read-only (local users live in horizon.yaml; you add one with the YAML plus a CLI-generated hash). As on Auth status, the last-seen and active-24h figures are tracked per BFF node, in memory, so in a multi-replica deployment they reflect that one node, not the whole cluster.\nFigure 4: Users — every account with its source (local / LDAP / break-glass), roles, and last sign-in. The last-seen and active counts are per-BFF-node and in-memory, so a multi-replica deploy shows that node only.\nAudit, and the break-glass hatch Every sensitive action lands in an append-only audit log — JSON Lines on disk (horizon-audit.jsonl). It records sign-ins and failures, break-glass activations, rule edits and applies (carrying the OAP outcome), alarm- and dashboard-setup changes, and live-debugger start/stop. Reads aren\u0026rsquo;t logged — only writes and sensitive operations, to keep the volume manageable — and each line carries who acted, the action and the verb checked, when, the outcome, and the source IP. Horizon doesn\u0026rsquo;t rotate the file itself, so you pair it with a log shipper and durable storage to outlive restarts.\nBreak-glass is the emergency hatch: a local admin credential that works only when your backend is LDAP and the directory is currently unreachable. It lets you back in when the directory is down without being a standing backdoor — it doesn\u0026rsquo;t bypass RBAC (the session gets whatever roles you configured, which can be as little as viewer), every use is double-logged (the audit line plus a WARN), and it stops working the moment LDAP recovers.\nMake it yours: themes Finally, the lighter touch. Five themes ship with Horizon: Horizon, the default — dark, amber, dense, observability-first; Meridian, a cooler navy/indigo for SREs who live in tables; Obsidian, true-black and monospaced for OLED screens; Daybreak, a light, airy palette for shared screens and printouts; and Aurora, a magenta-to-cyan glass look made for demos. An admin sets the org default on the Global Defaults page; every user can override it per-device from the topbar theme chip, which marks with a dot when your choice differs from the org default.\nFigure 5: Five bundled themes — Horizon (default), Meridian, Obsidian, Daybreak, Aurora — each previewed with its palette, density, and font. An admin sets the org default here; every user can override it per-device from the topbar chip.\nWhere to go next All five surfaces here are Horizon\u0026rsquo;s own BFF-side governance — they never touch the OAP admin host, so they behave identically across OAP versions. (RBAC can also be turned off entirely for local dev, in which case the Roles board flags it in red.) For the field reference — the verb list, the LDAP group mapping, the audit schema, and arming break-glass — see the RBAC, LDAP backend, Audit log, and Break-glass docs.\nNext up, Act 5 — make it yours \u0026amp; adopt, starting with Customization: Config-Driven Layer Templates — how the whole console is shaped by templates you can edit, preview, and publish.\n","excerpt":"\u003cp\u003eThis is the fourteenth post in the \u003ca href=\"/blog/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e series, and it opens \u003cstrong\u003eAct 4 — govern \u0026amp; secure …\u003c/strong\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-06-30-horizon-ui-access-control-and-security/","title":"Meet Horizon UI · 14/17: Access Control \u0026 Security"},{"body":"This is the fifteenth post in the Meet Horizon UI series, and it opens Act 5 — make it yours \u0026amp; adopt. Everything you\u0026rsquo;ve seen across this series — the per-layer dashboards, the overviews, the topology, the 3D map — is not hard-coded. It\u0026rsquo;s all driven by templates, and Horizon ships the editor for them. This post is that editor: change any dashboard in a local draft, preview it, and publish it to OAP so the whole org sees it.\nEverything is a template Open Admin → Layer dashboards and every layer the backend reports is there to configure. The editing model is the important part, and it\u0026rsquo;s the same everywhere: Save (local) keeps your edits in this browser only — it never touches the server; the live, shared version lives on OAP; and the bundled JSON Horizon ships is just the seed and a read-only fallback. A status badge tells you where each layer stands — synced, diverged (the bundle differs from what\u0026rsquo;s live on OAP, and OAP wins at render), or local (unpublished edits). Reset to ▾ reloads Bundled or Remote into the editor, and Preview ▾ opens the real page rendered from your Local, the Bundled, or the Remote version.\nA layer\u0026rsquo;s setup is more than charts: you choose which sub-views it exposes (Service, Instances, Topology, Deployment, Traces, Logs, profiling…), its display alias, and even the menu nouns — the Istio mesh layer below renames \u0026ldquo;Instances\u0026rdquo; to Sidecars.\nFigure 1: The Layer dashboards admin — every layer is a template. The header carries the whole model: Save (local) in your browser, Reset to / Preview the Bundled or Remote version, and Check diff \u0026amp; push to publish. Below, the layer chooses which sub-views it exposes, its alias, and renamed menu nouns (here Sidecars for instances).\nEdit the widgets Inside each scope, the dashboard is a 12-column grid you edit directly — drag a header to reorder, drag a corner to resize, + Add widget to add one. Click any widget and its editor drawer opens: the MQE expression(s) that feed it, the widget type (line / top / table / card…), title, unit, and a Visible when gate that hides the widget unless an expression has a value or an entity attribute matches.\nFigure 2: The widget canvas — a 12-column grid you drag to reorder and resize. Click a widget and its drawer opens: the MQE expression(s), type, title, unit, and a Visible when gate.\nPublish, safely Nothing you do reaches other users until you publish — and publishing shows you exactly what will change first. Check diff \u0026amp; push opens a side-by-side diff (the live remote on the left, your local draft on the right); only then, on Confirm push, does the draft replace the live version for everyone. The button is enabled only when your local actually differs from remote.\nFigure 3: Check diff \u0026amp; push — before a draft goes live for everyone, a side-by-side diff (remote left, your local right) shows exactly what changes. Here a widget title is renamed; nothing publishes until you Confirm push.\nThis is also how you add a layer. A layer the backend reports but Horizon ships no template for opens on a blank default — configure its components and widgets, Save, and that first push publishes the template to OAP. No per-layer JSON has to be shipped for a layer to be fully configurable.\nOverviews, and portability The Overview templates editor is the same model on a 12-column canvas that mirrors the live grid (with mock data — the real page uses real data). + New dashboard writes a local draft; Delete is a soft-disable (OAP has no hard delete). And every template admin page — layer dashboards, overviews, the 3D-map config, translations — carries Export and Import: Export downloads the in-use version (what users actually render) as a JSON file for backup, sharing, or moving a dashboard to another OAP; Import reads a JSON file, validates it, and loads it as a local draft to preview and push. Import never writes OAP directly.\nFigure 4: Overview templates are the same model — a 12-column canvas with mock data, + New dashboard as a local draft, and Export / Import to move a dashboard between OAP backends.\nWhere it runs Editing and previewing are entirely browser-local — no OAP call happens until you publish. Publishing writes the template to OAP\u0026rsquo;s ui-template store through the admin host, which arrives with OAP 11; the bundled JSON is a seed and read-only fallback, and the OAP-published version always wins at render time. Access is role-gated: publishing layer dashboards needs dashboard:write, overviews need overview:write. For the field reference — the template shapes, the widget types, and the add-a-layer recipe — see the Layer templates, Overview templates, and Adding a new layer docs.\nNext up: Localization — how those same templates speak eight languages.\n","excerpt":"\u003cp\u003eThis is the fifteenth post in the \u003ca href=\"/blog/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e series, and it opens \u003cstrong\u003eAct 5 — make it yours \u0026amp; …\u003c/strong\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-06-30-horizon-ui-customization-and-templates/","title":"Meet Horizon UI · 15/17: Customization — Config-Driven Layer Templates"},{"body":"This is the sixteenth post in the Meet Horizon UI series, still in Act 5 — make it yours \u0026amp; adopt. The previous post showed that the whole console is driven by templates. Localization builds straight on that: Horizon speaks eight languages, and a translation is just an overlay on the same template — not a fork, and not a re-translation on every render.\nTranslations are overlays, resolved once on the server Each non-English locale is an overlay catalog layered onto the English source template. When the BFF serves a template, it reads your chosen locale (from a request header), merges the overlay onto the source once, and returns the localized template — translation is resolved a single time on the server, not per chart at mount. The merge is forgiving: it deep-merges onto the source, and any leaf the overlay doesn\u0026rsquo;t translate falls back to English, so a half-translated catalog is perfectly valid. Adding or fixing a language is therefore editing overlays, never touching the source dashboards.\nThere\u0026rsquo;s a deliberate line about what gets translated. The chrome does — widget titles, tips, labels, menu aliases. But data OAP supplies never does: service, instance, and endpoint names, tags, log lines, alarm-rule names, and span operations render exactly as the backend reports them. Product and protocol names (SkyWalking, Kubernetes, Envoy), MQE expressions, and codes like RPM / P95 / SLA stay verbatim too.\nClick to translate You don\u0026rsquo;t edit JSON to localize. Admin → Translations gives you a live preview in your target language: pick the template and the language, click any widget, and a panel opens showing each translatable field — the English source above, your translation below. A progress counter tracks how far along each language is, and the same draft model from the templates post applies: Stage local keeps it in your browser, Check diff \u0026amp; push publishes the overlay to OAP, and Export / Import moves a language\u0026rsquo;s translations as a JSON file.\nFigure 1: The Translations admin — pick a language (here Français), click any widget in the live preview, and type the translation. The title and tip translate; codes like RPM / P95 / SLA stay verbatim — and so does every value OAP supplies.\nEight languages, picked per device The set is eight first-class locales: English (the source) plus Deutsch, Español, Français, 日本語, 한국어, Português, 中文 (简体). You switch from the language chip in the topbar — on every page, including the pre-auth login — and the choice persists per device.\nFigure 2: The topbar language chip — eight first-class languages (English plus Deutsch, Español, Français, 日本語, 한국어, Português, 中文), picked per device on every page including the login.\nSwitch the language and the whole console follows. Here is the General Service dashboard in Chinese — the sidebar, tabs, and widget titles are localized, while the API paths, service names, and metric values stay exactly as OAP reports them. Notice that a few widget titles are still in English: those leaves simply aren\u0026rsquo;t translated yet, so they fall back to the source — exactly the graceful degradation the overlay model is built for.\nFigure 3: The same General Service dashboard in Chinese — sidebar, tabs, and widget titles localized, while service names, API paths, and metric values stay exactly as OAP reports them. Untranslated leaves fall back to English.\nWhere it runs Locale resolution happens BFF-side, so it works on any OAP. Publishing a translation writes a sibling overlay to OAP\u0026rsquo;s template store through the admin host (OAP 11), gated on overview:write; the eight UI-chrome message catalogs are bundled and switch synchronously with no network fetch. A CI gate (i18n:validate) keeps every source template paired with an overlay per locale so nothing drifts. For the field reference — the translatable-field rules and the add-a-language recipe — see the i18n docs.\nNext, the series closes with Getting Started \u0026amp; Migration — installing Horizon and swapping it in for an existing UI.\n","excerpt":"\u003cp\u003eThis is the sixteenth post in the \u003ca href=\"/blog/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e series, still in \u003cstrong\u003eAct 5 — make it yours \u0026amp; adopt …\u003c/strong\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-06-30-horizon-ui-localization-i18n/","title":"Meet Horizon UI · 16/17: Localization in Eight Languages"},{"body":"This is the seventeenth and final post in the Meet Horizon UI series, closing Act 5 — make it yours \u0026amp; adopt. Sixteen posts toured what Horizon shows, does, governs, and lets you customize. This one is the practical close: how to get it running, what it needs from your backend, and how to swap it in for an existing UI.\nOne image, one config Horizon ships as a single container image — the Vue UI and its Fastify BFF in one artifact — published to Docker Hub as apache/skywalking-ui, with Horizon releases tagged horizon-\u0026lt;version\u0026gt; (and latest). There\u0026rsquo;s one configuration file, horizon.yaml, and its defining trait is that every field is an environment-variable token (${HORIZON_X:default}) expanded before the YAML is parsed. So you can run the image with nothing mounted and set only the env vars you care about, or copy the file, edit it, and mount it.\n# horizon.yaml — every field is an env token: ${HORIZON_X:default}, # expanded before YAML parsing. Run image-native with env vars, or mount this file. server: host: \u0026#34;${HORIZON_SERVER_HOST:127.0.0.1}\u0026#34; # the image sets 0.0.0.0 port: ${HORIZON_SERVER_PORT:8081} oap: queryUrl: \u0026#34;${HORIZON_OAP_QUERY_URL:http://127.0.0.1:12800}\u0026#34; # GraphQL — required adminUrl: \u0026#34;${HORIZON_OAP_ADMIN_URL:http://127.0.0.1:17128}\u0026#34; # admin REST — operate features zipkinUrl: \u0026#34;${HORIZON_OAP_ZIPKIN_URL:http://127.0.0.1:9412/zipkin}\u0026#34; # optional auth: backend: \u0026#34;${HORIZON_AUTH_BACKEND:local}\u0026#34; # local | ldap local: users: ${HORIZON_AUTH_LOCAL_USERS:[]} # JSON; hash with: pnpm --filter bff cli:hash session: ttlMinutes: ${HORIZON_SESSION_TTL_MINUTES:60} cookieSecure: ${HORIZON_SESSION_COOKIE_SECURE:false} # set true behind HTTPS Pointing it at an existing OAP is three URLs. A container run is then just env vars over the image:\ndocker run -d --name horizon -p 8081:8081 \\ -e HORIZON_SERVER_HOST=0.0.0.0 \\ -e HORIZON_OAP_QUERY_URL=http://oap:12800 \\ -e HORIZON_OAP_ADMIN_URL=http://oap:17128 \\ -e HORIZON_AUTH_LOCAL_USERS=\u0026#39;[{\u0026#34;username\u0026#34;:\u0026#34;admin\u0026#34;,\u0026#34;passwordHash\u0026#34;:\u0026#34;$argon2id$...\u0026#34;,\u0026#34;roles\u0026#34;:[\u0026#34;admin\u0026#34;]}]\u0026#39; \\ apache/skywalking-ui:horizon-\u0026lt;version\u0026gt; Two things worth knowing on the first boot. There is no default admin/admin — with the local backend and no users (or ldap with no group mappings) the BFF starts but no one can sign in until you configure it; you generate password hashes with pnpm --filter bff cli:hash. And for reproducible deploys, pin to a specific horizon-\u0026lt;version\u0026gt; tag rather than latest.\nWhat works on which OAP Horizon is built natively against OAP 11.x, and it partially supports OAP 10.x. The split is clean and maps to the two halves of this series: the observe data-plane runs against OAP\u0026rsquo;s query port and works on both lines; the operate surfaces live on OAP\u0026rsquo;s admin port, which only 11.x runs.\nSurface OAP 10.x OAP 11.x Port Layer dashboards, overviews, topology ✓ ✓ query :12800 Traces (native + Zipkin), logs, alarms (read), profiling ✓ ✓ query :12800 (+ :9412 for Zipkin) Inspect, DSL Management, Live Debugger, Alarm-rule editor — ✓ admin :17128 Cluster Status → Admin pane, template \u0026amp; translation publishing — ✓ admin :17128 Crucially, Horizon never reads the OAP version number — it detects each capability by probing for the module and GraphQL fields it needs, hides the sidebar entries it can\u0026rsquo;t back, and falls back to read-only for admin pages when the admin port is dark. So if you only need triage (dashboards, alarms, traces, logs), a 10.x backend is enough; anything in the operate half needs 11.x with its admin modules (admin-server, receiver-runtime-rule, dsl-debugging, inspect) enabled.\nSwapping in for an existing UI If you run the previous-generation UI today, the migration is drop-in. Horizon speaks the same OAP GraphQL query protocol and the same MQE language, so there are no agent changes and no backend changes — you point Horizon at the OAP you already run. The clean cutover is to run both side by side, let people use Horizon against live data, and retire the old UI when you\u0026rsquo;re ready. Everything Horizon adds on top — its governance (RBAC, auth, audit, themes) and its config-driven templates — is Horizon\u0026rsquo;s own, layered in the BFF, independent of what your OAP does.\nVerify and operate Confirm the connection from the Cluster Status page: the topbar carries the OAP build-version chip, and the Query pane shows version, timezone, and a health score; the Admin and Zipkin panes light up to match what your backend exposes. A few operational notes for production:\nPersist state. Admin-edited templates land under /app/bundled_templates and the audit, setup, and alarm files under /data; mount durable volumes there or those edits are ephemeral and vanish with the container. Sessions are per-BFF. They live in each node\u0026rsquo;s memory with no shared store, so multiple replicas need sticky sessions (otherwise a failover means re-login). Probes and TLS. /api/health is public with no OAP dependency — wire it to your container probes — and set session.cookieSecure: true behind HTTPS. For the full reference — the container image, the horizon.yaml field-by-field, and the OAP compatibility matrix — see the docs.\nThat\u0026rsquo;s the series Seventeen posts, five acts: we oriented in the console, observed services across layers — metrics, traces, logs, topology, profiling — operated on the backend with runtime rules, live debugging, and inspection, governed it with access control, and finally made it ours with templates, eight languages, and a clean install. The best next step is to stop reading and start clicking: the public demo at demo.skywalking.apache.org runs Horizon against a live SkyWalking backend. Thanks for following along.\n","excerpt":"\u003cp\u003eThis is the seventeenth and final post in the \u003ca href=\"/blog/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e series, closing \u003cstrong\u003eAct 5 — make it yours …\u003c/strong\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-06-30-horizon-ui-getting-started-and-migration/","title":"Meet Horizon UI · 17/17: Getting Started \u0026 Migration"},{"body":"译自英文原文：Meet Horizon UI · 13/17: Platform \u0026amp; Cluster Introspection。\n这是 Meet Horizon UI 系列的第十三篇，也是第三幕 operate it 的最后一篇。前几篇 operate 文章讲的是怎么操作后端：告警、运行时规则、Live Debugger、跨 layer Inspect。这一篇反过来看后端本身：它现在是否健康？实际加载了哪些配置？数据会保留多久？Platform monitoring 下的三个只读页面回答这些问题。这里没有开关，只给排障时需要的事实。\nCluster Status：后端是否健康，哪些端口可用？ Horizon 和 OAP 通信不只走一个通道，Cluster Status 会把每个通道的健康状态分开显示。可以把它看成两个端口的健康视图，外加一个独立探针：\nQuery / GraphQL（:12800）：所有 observability 页面都依赖这个端口。它会显示 OAP 版本、服务端时区和时钟（并排展示浏览器时间），以及 OAP 自己的健康评分。这个部分兼容任意 OAP，包括 10.x。 Admin host（:17128）：operate 功能依赖这个端口。页面会对每个 admin module 做实时探测：直接 GET 功能实际调用的 REST path，并显示它是否可达、启用它的 SW_... 环境变量，以及它不可用时哪些功能会受影响。这里包括 admin-server、receiver-runtime-rule、dsl-debugging、inspect、ui-management。Admin host 随 OAP 11 提供，所以 10.x 后端上不会显示这个面板。 Zipkin / OTLP（:9412）：这是信息性探针，只影响 Zipkin trace 菜单。这里红了，其他页面仍然正常工作。 三个面板独立轮询；其中一个变红，不会影响另外两个的判断。正因为这样，当 Horizon 里其他地方看起来不对劲时，Cluster Status 是最适合先打开的页面。\n图 1：Cluster Status 是一个双端口健康视图。Query 面板（:12800，任意 OAP）显示版本、服务端时间和健康评分；Admin 面板（:17128，OAP 11）用实际 REST path 实时探测每个 admin module，并显示启用它的 SW_... 环境变量和不可达时受影响的功能；Zipkin/OTLP 面板只服务 Zipkin 菜单。\nOAP Configuration：实际运行的配置是什么？ 当某个配置项表现得和预期不一样，下一步通常是确认：OAP 最终到底解析出了什么配置？OAP Configuration 让你不用 SSH 到后端，也能看到这个答案。它从 admin port 的 /debugging/config/dump 读取当前连接后端的有效运行时配置，按 module 分组，并支持同时搜索 key 和 value。密码、token、access key 这类敏感值会由 OAP 自己在返回前 mask 成 ******，Horizon 看到的就是脱敏后的内容。这个页面严格只读；如果要改配置，仍然需要在 OAP 侧修改并重启。和上面的 Admin 面板一样，它依赖 OAP 11 的 admin host。\n图 2：OAP Configuration 展示当前连接后端解析后的运行时配置，数据来自 admin port；配置按 module 分组，可以过滤，敏感值由 OAP mask 成 ******。页面只读：修改配置仍然要在 OAP 侧完成并重启。\nData Retention：数据会保留多久？ Data Retention 展示 OAP 为各类数据设置的 time-to-live，单位是整天：records（trace、log、browser error 等）和 metrics（metadata、minute、hour、day）都会列出来。这里还要看存储后端。像 Elasticsearch 这样的平面存储，每类数据只有一个 retention 数字。在 BanyanDB 中，数据会按可配置的生命周期阶段流转：hot、可选的 warm、可选的 cold。OAP 会把 hot 和 warm 合并报告为一个可查询窗口，所以 Horizon 不再只显示单个数字，而是为每类数据画一条 lifecycle bar：前半段是 hot+warm 窗口，后面接 cold 尾段，宽度按总天数比例展示。这样你可以一眼看出各阶段相对持续多久。\n页面还特别提醒一个操作细节：topbar 里的 Cold pill 会把查询切到 cold 数据，它不是把 hot+warm 和 cold 合并查询。所以只有当你要看的时间窗口已经早于 hot+warm 边界时，才值得打开它。Data Retention 走标准 query port，因此不同于前两个页面，它兼容任意 OAP 版本。\n图 3：Data Retention：在 BanyanDB 上，每类数据会经历 hot → warm → cold 生命周期，所以 Horizon 用 lifecycle bar 展示可查询的 hot+warm 窗口和 cold 尾段，宽度按总天数比例计算。平面存储如 Elasticsearch 则会显示每类数据的单个 TTL。\n它在哪里运行 这三个页面都在 Platform monitoring 下，各自有独立的读权限：cluster:read、config:read、ttl:read。它们都严格只读：这里做的是自检，不是控制。后端接入方式也延续了 operate 里的分层。Data Retention 和 Cluster Status 的 Query 面板走标准 query port，因此兼容任意 OAP，包括 10.x。Cluster Status 的 Admin 面板和 OAP Configuration 则读取 OAP 11 才提供的 admin host；如果后端没有这个端口，页面会明确说明原因，比如隐藏面板，或者显示“需要 admin host”的提示，而不是直接报错。（这里关注的是平台健康和配置；OAP 自观测的 dashboard 是另一条基于 metrics 的线。）\n后续阅读 字段参考，包括每个面板、config dump 结构和 BanyanDB 生命周期细节，可以看 Cluster Status、OAP Configuration 和 Data Retention 文档。\n到这里，Act 3 — operate it 结束。下一篇：访问控制与安全，进入 Act 4 — govern \u0026amp; secure it：服务端强制执行的 RBAC、LDAP/AD、audit log，以及 break-glass。\n","excerpt":"\u003cp\u003e\u003cem\u003e译自英文原文：\u003ca href=\"/blog/2026-06-30-horizon-ui-platform-introspection/\"\u003eMeet Horizon UI · 13/17: Platform \u0026amp; Cluster Introspection\u003c/a\u003e。\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e这是 \u003ca href=\"/zh/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e 系列的第十三篇，也是第 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-06-30-horizon-ui-platform-introspection/","title":"认识 Horizon UI · 13/17：平台与集群自检"},{"body":"译自英文原文：Meet Horizon UI · 14/17: Access Control \u0026amp; Security。\n这是 Meet Horizon UI 系列的第十四篇，也开启第四幕 govern \u0026amp; secure it。前面的文章讲 Horizon 能展示什么、能操作什么；这一篇讲谁可以看、谁可以操作。先明确一个架构边界：RBAC、认证、audit log、break-glass、主题，这些都是 Horizon 自己的治理逻辑，由 Horizon BFF 在服务端强制执行，不依赖也不改动 OAP admin host。因此，无论后端是 OAP 10.x 还是 11.x，行为都一样。\n先过登录这一关 任何页面渲染之前，都要先登录。Horizon 支持两种认证后端，由配置选择：Local，也就是在 horizon.yaml 里声明用户和 Argon2id 密码哈希；或者 LDAP / Active Directory，由 Horizon 绑定到目录服务，再把目录组映射成 Horizon 角色（可以用 memberOf，也可以做 group search，查询由 bind account 完成）。登录页会用一个 pill 显示当前使用的认证后端，并允许用户在认证前选择语言。登录失败时也刻意避免暴露用户枚举线索：用户名错误、密码错误、用户不在任何映射组里，都会返回同一句 \u0026ldquo;invalid credentials\u0026rdquo;。\n图 1：登录页：认证后端状态 pill（这里是 Local users）、认证前语言选择器和登录表单。认证是第一道门，之后的所有操作再由角色控制。\n对管理员来说，Auth status 页面会展示认证后端的当前状态：provider 和哈希算法、用户定义位置、break-glass 是否启用，以及当前活跃 session 数。这里有一点要读准确：active-session count 指的是你正在访问的这个 Horizon BFF 上的 session 数，不是整个集群的总数；session 保存在每个 BFF 节点的内存里。\n图 2：Auth status：一眼看到认证后端状态，包括 provider（这里是 local、Argon2id）、用户定义位置、break-glass 是否启用，以及当前 Horizon BFF 上的 active-session count（不是集群总数）。\n谁能做什么 登录之后，每个操作都要经过权限判断。Horizon 内置四个递进角色：viewer 可以读取观测数据（dashboard、trace、log、alarm）；maintainer 增加平台监控相关页面（cluster status、inspect、retention、config）；operator 再增加 dashboard、alarm、runtime rules 和诊断工具；admin 拥有全部权限，包括用户和访问控制管理。权限本身是带命名空间的 verb，例如 metrics:read、rule:write:structural、inspect:read，并支持通配符（*、rule:*、*:read）。\nRoles \u0026amp; Permissions 页面把整套模型摊开：上半部分是 menu-visibility matrix，显示每个角色能看到哪些侧边栏菜单，以及背后对应的 read verb；下半部分是按 area 拆开的 action matrix，精确标出 viewer / maintainer / operator / admin 分别能做什么。这个页面只读；角色定义在 horizon.yaml 里，保存后会 hot-reload，不需要重启。\n图 3：Roles \u0026amp; Permissions 页面顶部：角色卡片，以及 menu-visibility matrix——每个侧边栏菜单和角色的关系，由最后一列的 read verb 把守；下方还有按 area 拆分的 action matrix。页面只读：角色定义在 horizon.yaml，并支持 hot-reload。\n权限判断在服务端执行。BFF 会拦截每个受保护请求：未认证返回 401，session 缺少对应 verb 返回 403。route→verb 映射也是强制的：如果某条 route 没有配置权限项，构建会失败，因此不会因为漏配而意外开放。UI 也会按同一套 verb 逻辑隐藏不可用菜单，但这只是纵深防御，不是权限边界；伪造 UI 请求也无法提权，因为最终裁决在 BFF。\n当前有哪些用户 Users 页面列出 Horizon 知道的所有账号：local、LDAP 和 break-glass。每行会显示来源、分配到的角色、上次登录时间和 IP。页面只读；local 用户仍然定义在 horizon.yaml，新增用户需要写 YAML，并用 CLI 生成密码哈希。和 Auth status 一样，last-seen 和 active-24h 统计都是单个 BFF 节点内存里的数据。多副本部署时，它反映的是当前这个节点，不是整个集群。\n图 4：Users：列出每个账号的来源（local / LDAP / break-glass）、角色和上次登录信息。last-seen 和 active count 都是 per-BFF-node、in-memory 统计，因此多副本部署只显示当前节点的数据。\nAudit，以及 break-glass 应急入口 所有敏感操作都会进入一个 append-only audit log，以 JSON Lines 写在磁盘上（horizon-audit.jsonl）。它记录登录成功和失败、break-glass 激活、规则编辑和应用（包含 OAP 返回结果）、alarm 和 dashboard setup 变更、Live Debugger start/stop。普通读取不会记录，只记录写操作和敏感操作，避免日志量失控。每一行都会带上操作者、动作、检查的 verb、时间、结果和来源 IP。Horizon 自己不负责轮转这个文件；生产环境里应该配合 log shipper 和持久化存储，避免重启后丢失。\nBreak-glass 是应急入口：一个本地 admin credential，但它只在认证后端是 LDAP 且目录服务当前不可达时生效。它的作用是在目录故障时让你重新进系统，而不是长期存在的后门。它不绕过 RBAC，登录后的 session 只拥有你配置给它的角色，甚至可以只有 viewer；每次使用都会双重记录：audit log 一条，加上一条 WARN；LDAP 恢复后，它会立刻失效。\n主题：让界面适合自己的环境 最后是一个轻量但实际有用的部分。Horizon 内置 五套主题：Horizon 是默认主题，深色、琥珀色调、高密度，偏 observability 场景；Meridian 是偏冷的 navy/indigo 风格，适合长期看表格的 SRE；Obsidian 使用接近纯黑和等宽字体，适合 OLED 屏；Daybreak 是轻亮配色，适合共享屏幕和打印；Aurora 是 magenta 到 cyan 的玻璃质感，更适合演示。管理员可以在 Global Defaults 页面设置组织默认主题；每个用户也可以从 topbar theme chip 按当前设备覆盖默认值。如果个人选择和组织默认不同，chip 上会用一个点标出来。\n图 5：五套内置主题：Horizon（默认）、Meridian、Obsidian、Daybreak、Aurora。每套主题都展示 palette、density 和 font。管理员在这里设置组织默认主题；每个用户可以从 topbar chip 按设备覆盖。\n后续阅读 这篇里的五个入口都属于 Horizon 自己的 BFF 侧治理，不访问 OAP admin host，因此在不同 OAP 版本上行为一致。（本地开发时也可以完全关闭 RBAC；这时 Roles board 会用红色标出来。）字段参考，包括 verb 列表、LDAP group mapping、audit schema 和 break-glass 启用方式，可以看 RBAC、LDAP backend、Audit log 和 Break-glass 文档。\n下一篇进入 Act 5 — make it yours \u0026amp; adopt，从 用模板定制控制台 开始：整个控制台如何由可编辑、可预览、可发布的模板塑造。\n","excerpt":"\u003cp\u003e\u003cem\u003e译自英文原文：\u003ca href=\"/blog/2026-06-30-horizon-ui-access-control-and-security/\"\u003eMeet Horizon UI · 14/17: Access Control \u0026amp; Security\u003c/a\u003e。\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e这是 \u003ca href=\"/zh/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e 系列的第十四篇，也开启第四幕 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-06-30-horizon-ui-access-control-and-security/","title":"认识 Horizon UI · 14/17：访问控制与安全"},{"body":"译自英文原文：Meet Horizon UI · 15/17: Customization — Config-Driven Layer Templates。\n这是 Meet Horizon UI 系列的第十五篇，也开启第五幕 make it yours \u0026amp; adopt。这个系列里你看到的 per-layer dashboard、overview、topology、3D map，都不是写死在代码里。它们由一套套 template 驱动，Horizon 也把编辑这些 template 的工具做进了控制台。这篇讲的就是这个编辑器：在本地草稿里修改任意 dashboard，先预览，再发布到 OAP，让整个组织都看到同一个版本。\n一切都是模板 打开 Admin → Layer dashboards，后端上报的每个 layer 都可以配置。这里最重要的是编辑模型，而且所有模板编辑器都遵循同一套模型：Save (local) 只把修改保存在当前浏览器里，不会碰服务端；真正被所有人看到的共享版本保存在 OAP；Horizon 随版本发布的 bundled JSON 只是种子模板和只读 fallback。状态 badge 会告诉你每个 layer 当前处在哪种状态：synced 表示本地和远端一致；diverged 表示 bundled 和 OAP 上的 live 版本不同，渲染时以 OAP 为准；local 表示当前浏览器里有未发布的编辑。Reset to ▾ 可以把 Bundled 或 Remote 重新加载到编辑器；Preview ▾ 则会用 Local、Bundled 或 Remote 版本打开真实页面预览。\n一个 layer 的配置不只是图表。你还可以选择它暴露哪些 sub-view，比如 Service、Instances、Topology、Deployment、Traces、Logs、profiling；可以设置展示用的 alias，甚至可以改菜单里的名词。下面的 Istio mesh layer 就把 “Instances” 改成了 Sidecars。\n图 1：Layer dashboards 管理页：每个 layer 都是一个 template。顶部操作区体现了完整模型：Save (local) 保存在浏览器里，Reset to / Preview Bundled 或 Remote 版本，最后用 Check diff \u0026amp; push 发布。下面可以选择这个 layer 暴露哪些 sub-view、设置 alias，并重命名菜单名词，比如把 instances 叫作 Sidecars。\n编辑 Widget 每个 scope 里的 dashboard 都是一个可以直接编辑的 12 列网格：拖动 header 可以调整顺序，拖动角落可以改变大小，点 + Add widget 可以新增 widget。点击任意 widget，会打开它的编辑抽屉：这里可以改驱动它的 MQE expression、widget type（line / top / table / card 等）、title、unit，还可以设置 Visible when 条件，让某个表达式有值或者某个实体属性匹配时才显示这个 widget。\n图 2：Widget 画布：一个可以拖拽排序、调整大小的 12 列网格。点中 widget 后会打开抽屉，编辑 MQE expression、type、title、unit 和 Visible when 条件。\n发布前先看差异 在你发布之前，本地修改不会影响其他用户；而发布之前，Horizon 会先让你看清楚到底改了什么。点击 Check diff \u0026amp; push 会打开左右对照的 diff：左侧是当前 live 的 remote 版本，右侧是你的 local 草稿。只有继续点 Confirm push，本地草稿才会替换 OAP 上的 live 版本，所有用户才会看到变化。只有 local 和 remote 真的不一样时，这个按钮才会启用。\n图 3：Check diff \u0026amp; push：草稿发布给所有人之前，先用左右对照 diff 展示变化（remote 在左，本地草稿在右）。这里改了一个 widget title；只有点 Confirm push 才会真正发布。\n新增 layer 也走这套流程。后端已经上报、但 Horizon 没有内置模板的 layer，会先打开一个空白默认模板；你配置它的 components 和 widgets，Save 后第一次 push 就会把模板发布到 OAP。一个 layer 不需要先随版本带上 JSON 文件，才能变成完整可配置的页面。\nOverview 与迁移复用 Overview templates 编辑器也使用同样的模型：一个 12 列画布，布局和线上页面一致，只是编辑器里使用 mock data，真实页面会使用真实数据。+ New dashboard 会写出一个本地草稿；Delete 是软禁用，因为 OAP 目前没有 hard delete。所有模板管理页也都有 Export 和 Import，包括 layer dashboards、overviews、3D map config 和 translations：Export 会把当前实际使用的版本下载成 JSON 文件，便于备份、共享，或者把一个 dashboard 搬到另一个 OAP；Import 读取 JSON 文件、校验，然后作为本地草稿加载进来，供你预览和发布。Import 不会直接写 OAP。\n图 4：Overview templates 使用同一套模型：12 列画布、mock data、作为本地草稿创建的 + New dashboard，以及用 Export / Import 在不同 OAP 后端之间迁移 dashboard。\n它在哪里运行 编辑和预览完全发生在浏览器本地；只有发布时才会调用 OAP。发布会通过 OAP 11 提供的 admin host，把 template 写入 OAP 的 ui-template store。Bundled JSON 是种子模板和只读 fallback；只要 OAP 上发布过版本，渲染时就以 OAP 版本为准。权限也按角色控制：发布 layer dashboards 需要 dashboard:write，发布 overviews 需要 overview:write。字段参考，包括 template 结构、widget 类型和新增 layer 的流程，可以看 Layer templates、Overview templates 和 Adding a new layer 文档。\n下一篇：八种语言的本地化：同一套 template 如何支持八种语言。\n","excerpt":"\u003cp\u003e\u003cem\u003e译自英文原文：\u003ca href=\"/blog/2026-06-30-horizon-ui-customization-and-templates/\"\u003eMeet Horizon UI · 15/17: Customization — Config-Driven Layer Templates\u003c/a\u003e。\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e这是 \u003ca href=\"/zh/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e 系列 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-06-30-horizon-ui-customization-and-templates/","title":"认识 Horizon UI · 15/17：用模板定制控制台"},{"body":"译自英文原文：Meet Horizon UI · 16/17: Localization in Eight Languages。\n这是 Meet Horizon UI 系列的第十六篇，仍然属于第五幕 make it yours \u0026amp; adopt。上一篇讲到，整个控制台都由 template 驱动。本地化也建立在这个基础上：Horizon 支持八种语言，每种翻译都是覆盖在同一套 template 上的一层 overlay。它不是 fork 出一份 dashboard，也不是每次渲染图表时重新翻译。\n翻译是 overlay，在服务端合并一次 每个非英语 locale 都是一份 overlay catalog，覆盖在英语 source template 上。BFF 返回 template 时，会根据请求头里的语言选择，把 overlay 合并到 source 上一次，再返回本地化后的 template。翻译只在服务端解析一次，不会在每个图表加载时再处理。合并逻辑也比较宽容：overlay 会 deep-merge 到 source 上；没有翻译到的 leaf 会回退到英语，所以一份只翻了一半的 catalog 也是有效的。新增或修正语言，本质上就是编辑 overlay，不需要改 source dashboard。\n这里有一条刻意划清的边界：哪些内容会翻译，哪些不会。会翻译的是界面 chrome，例如 widget title、tip、label、menu alias。但 OAP 提供的数据不会翻译：service、instance、endpoint 名称，tags，log line，alarm rule name，span operation，都会按后端返回的原样展示。SkyWalking、Kubernetes、Envoy 这类产品或协议名，MQE 表达式，以及 RPM / P95 / SLA 这样的代码和缩写，也保持原样。\n点一下就能翻译 本地化不需要手写 JSON。Admin → Translations 会给出目标语言的 live preview：选择 template 和 language，点击任意 widget，右侧面板会列出可以翻译的字段：上方是英语 source，下方是你的翻译。每种语言都有进度计数，模板那篇提到的草稿模型也同样适用：Stage local 把改动保存在浏览器里，Check diff \u0026amp; push 把 overlay 发布到 OAP，Export / Import 可以把某种语言的翻译作为 JSON 文件迁移。\n图 1：Translations 管理页：选择语言（这里是 Français），在 live preview 中点击任意 widget，就可以输入翻译。title 和 tip 会翻译；RPM / P95 / SLA 这类代码保持原样，OAP 提供的所有值也保持原样。\n八种语言，按设备选择 Horizon 正式支持八种 locale：English 是 source，另外还有 Deutsch、Español、Français、日本語、한국어、Português、中文（简体）。你可以从顶栏的 language chip 切换语言；所有页面都支持，包括登录前的 login 页面。选择会按当前设备持久化。\n图 2：顶栏 language chip：八种正式支持的语言，English 加 Deutsch、Español、Français、日本語、한국어、Português、中文。每个设备可以独立选择，登录页也能切换。\n切换语言后，整个控制台都会跟着变。下面是中文里的 General Service dashboard：sidebar、tabs 和 widget title 都已经本地化，而 API path、service name 和 metric value 仍然按 OAP 返回值展示。你也会看到少数 widget title 还是英语：那几个 leaf 还没有翻译，所以自然回退到 source。这正是 overlay 模型想要的降级方式。\n图 3：同一个 General Service dashboard 的中文版本：sidebar、tabs 和 widget title 会本地化；service name、API path 和 metric value 仍然按 OAP 返回值展示。未翻译的 leaf 会回退到英语。\n它在哪里运行 Locale 解析发生在 BFF 侧，所以它兼容任意 OAP。发布翻译时，会通过 OAP 11 的 admin host，把 sibling overlay 写入 OAP 的 template store；权限由 overview:write 控制。八种 UI chrome message catalog 是随包内置的，切换时同步生效，不需要额外网络请求。CI 里的 i18n:validate 会检查每个 source template 都有对应 locale 的 overlay，避免 source 和翻译脱节。字段参考，包括哪些字段可翻译、如何新增语言，可以看 i18n 文档。\n下一篇，也就是这个系列的收官篇：安装与迁移：安装 Horizon，并把它替换到现有 UI 的位置。\n","excerpt":"\u003cp\u003e\u003cem\u003e译自英文原文：\u003ca href=\"/blog/2026-06-30-horizon-ui-localization-i18n/\"\u003eMeet Horizon UI · 16/17: Localization in Eight Languages\u003c/a\u003e。\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e这是 \u003ca href=\"/zh/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e 系列的第十六篇，仍然属于第五幕 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-06-30-horizon-ui-localization-i18n/","title":"认识 Horizon UI · 16/17：八种语言的本地化"},{"body":"译自英文原文：Meet Horizon UI · 17/17: Getting Started \u0026amp; Migration。\n这是 Meet Horizon UI 系列的第十七篇，也是最后一篇，为第五幕 make it yours \u0026amp; adopt 收尾。前十六篇已经讲过 Horizon 能展示什么、能操作什么、如何治理，以及如何定制。这篇回到最实际的问题：怎样把它跑起来，它对后端有什么要求，以及怎样替换掉现有 UI。\n一个镜像，一份配置 Horizon 以单个容器镜像发布：Vue UI 和 Fastify BFF 打在同一个 artifact 里，镜像发布在 Docker Hub：apache/skywalking-ui，Horizon 的 release 用 horizon-\u0026lt;version\u0026gt; 标签（也提供 latest）。配置文件只有一个：horizon.yaml。它最重要的特点是，每个字段都是环境变量 token（${HORIZON_X:default}），并且会在 YAML 解析之前展开。所以你可以不挂载任何文件，只用环境变量启动镜像；也可以复制这份文件，修改后挂载进去。\n# horizon.yaml — 每个字段都是 env token: ${HORIZON_X:default} # 会在 YAML 解析前展开。可以直接用环境变量启动镜像，也可以挂载这份文件。 server: host: \u0026#34;${HORIZON_SERVER_HOST:127.0.0.1}\u0026#34; # 镜像内默认设置为 0.0.0.0 port: ${HORIZON_SERVER_PORT:8081} oap: queryUrl: \u0026#34;${HORIZON_OAP_QUERY_URL:http://127.0.0.1:12800}\u0026#34; # GraphQL，必需 adminUrl: \u0026#34;${HORIZON_OAP_ADMIN_URL:http://127.0.0.1:17128}\u0026#34; # admin REST，operate 功能使用 zipkinUrl: \u0026#34;${HORIZON_OAP_ZIPKIN_URL:http://127.0.0.1:9412/zipkin}\u0026#34; # 可选 auth: backend: \u0026#34;${HORIZON_AUTH_BACKEND:local}\u0026#34; # local | ldap local: users: ${HORIZON_AUTH_LOCAL_USERS:[]} # JSON；用 pnpm --filter bff cli:hash 生成 hash session: ttlMinutes: ${HORIZON_SESSION_TTL_MINUTES:60} cookieSecure: ${HORIZON_SESSION_COOKIE_SECURE:false} # HTTPS 后面应设为 true 把 Horizon 指向已有 OAP，只需要三个 URL。容器启动命令就是给镜像加上环境变量：\ndocker run -d --name horizon -p 8081:8081 \\ -e HORIZON_SERVER_HOST=0.0.0.0 \\ -e HORIZON_OAP_QUERY_URL=http://oap:12800 \\ -e HORIZON_OAP_ADMIN_URL=http://oap:17128 \\ -e HORIZON_AUTH_LOCAL_USERS=\u0026#39;[{\u0026#34;username\u0026#34;:\u0026#34;admin\u0026#34;,\u0026#34;passwordHash\u0026#34;:\u0026#34;$argon2id$...\u0026#34;,\u0026#34;roles\u0026#34;:[\u0026#34;admin\u0026#34;]}]\u0026#39; \\ apache/skywalking-ui:horizon-\u0026lt;version\u0026gt; 第一次启动时有两点需要注意。第一，Horizon 没有默认的 admin/admin。使用 local 后端但没有配置用户，或者使用 ldap 但没有配置 group mapping 时，BFF 会启动，但没人能登录；密码哈希可以用 pnpm --filter bff cli:hash 生成。第二，为了让部署可复现，生产环境应该固定到明确的 horizon-\u0026lt;version\u0026gt; 标签，而不是 latest。\n哪些功能需要哪个 OAP Horizon 主要面向 OAP 11.x 构建，同时部分支持 OAP 10.x。边界很清楚，也对应这个系列的两半：observe 数据面走 OAP query port，两条版本线都能用；operate 页面依赖 OAP admin port，只有 11.x 提供。\n功能 OAP 10.x OAP 11.x 端口 Layer dashboards、overviews、topology ✓ ✓ query :12800 Traces（native + Zipkin）、logs、alarms（read）、profiling ✓ ✓ query :12800（Zipkin 使用 :9412） Inspect、DSL Management、Live Debugger、Alarm-rule editor — ✓ admin :17128 Cluster Status → Admin pane、template 和 translation 发布 — ✓ admin :17128 关键点是，Horizon 不会靠读取 OAP 版本号来判断能力。它会探测所需 module 和 GraphQL field：能支撑的功能才显示侧边栏入口；admin port 不可用时，admin 页面会降级成只读或显示明确提示。所以如果你只需要 triage，也就是 dashboard、alarm、trace、log，OAP 10.x 就够了；operate 这半边需要 OAP 11.x，并启用对应 admin modules（admin-server、receiver-runtime-rule、dsl-debugging、inspect）。\n替换现有 UI 如果你现在运行的是上一代 UI，迁移到 Horizon 可以做成平滑替换。Horizon 使用同一套 OAP GraphQL query protocol 和同一套 MQE 语言，所以不需要改 agent，也不需要改后端；只要把 Horizon 指向正在运行的 OAP 即可。更稳妥的切换方式是先让两个 UI 并行运行，让大家用 Horizon 访问 live data，等准备好了再下线旧 UI。Horizon 额外提供的东西，比如 governance（RBAC、auth、audit、themes）和 config-driven templates，都是 Horizon 自己在 BFF 里叠加的能力，和 OAP 本身相互独立。\n验证与生产运行 可以从 Cluster Status 页面确认连接情况：topbar 会显示 OAP build-version chip，Query 面板会展示版本、时区和健康评分；Admin 和 Zipkin 面板则会根据后端实际暴露的能力亮起。生产环境还要注意几件事：\n持久化状态。 管理员编辑的 template 会落在 /app/bundled_templates，audit、setup 和 alarm 文件会落在 /data；这两个路径应该挂载持久化 volume，否则容器重建后这些修改会丢失。 Session 按 BFF 节点保存。 Session 保存在每个节点内存里，没有共享存储；多副本部署需要 sticky sessions，否则故障切换后用户要重新登录。 探针和 TLS。 /api/health 是公开接口，不依赖 OAP，适合作为容器探针；如果部署在 HTTPS 后面，记得把 session.cookieSecure 设为 true。 完整参考可以看文档：container image、horizon.yaml 字段说明，以及 OAP compatibility matrix。\n系列到这里结束 十七篇，五幕：我们先在控制台里建立方向感，然后跨 layer 观察服务的 metrics、traces、logs、topology 和 profiling；接着用 runtime rules、live debugging 和 inspect 操作后端；再通过 access control 治理控制台；最后用 templates、八种语言和清晰的安装路径，把它变成自己的工具。下一步最好不是继续读，而是直接点开试试：公开 demo demo.skywalking.apache.org 已经把 Horizon 接到了一套 live SkyWalking 后端上。谢谢一路读到这里。\n","excerpt":"\u003cp\u003e\u003cem\u003e译自英文原文：\u003ca href=\"/blog/2026-06-30-horizon-ui-getting-started-and-migration/\"\u003eMeet Horizon UI · 17/17: Getting Started \u0026amp; Migration\u003c/a\u003e。\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e这是 \u003ca href=\"/zh/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e 系列的第十七篇，也是最后一篇，为 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-06-30-horizon-ui-getting-started-and-migration/","title":"认识 Horizon UI · 17/17：安装与迁移"},{"body":"This is the tenth post in the Meet Horizon UI series, and the first of Act 3 — operate it. The earlier posts were about seeing your data — dashboards, topology, traces, logs, profiles. This one is about the moment something breaks, and Horizon\u0026rsquo;s answer to the only two questions that matter then: what is on fire right now, and why?\nThe alarm surface is built around two ideas: incidents, not a wall of events, and replaying the exact metric snapshot that fired the rule.\nIncidents, not a wall of events OAP emits an alarm event every time a rule trips. A flapping rule on a busy service can fire dozens of times an hour, and a raw event feed buries the one new problem under a hundred repeats. Horizon groups those events the way an on-call engineer already thinks about them — by (entity, rule). Every firing of \u0026ldquo;response time of agent::gateway is more than 20ms\u0026rdquo; is the same incident, however many times it tripped.\nSo the Alarms page (top nav) is a list of incidents, not events. The KPI strip counts the active ones — total and per layer — and each row carries the entity, the rule\u0026rsquo;s message, its layer, and a triggered N× badge when it re-fired. An incident is in one of three states:\nfiring — the latest event has not recovered; recovered — the condition cleared; these drop out of the active counts but stay visible as recent history; unstable — it fired, recovered, then fired again. That badge is how a noisy, oscillating rule outs itself. The page also owns its own time window — 20m / 2h / 4h presets, or a custom range up to four hours — independent of the global topbar. You can rewind two hours of alarm history without disturbing the dashboard you were on.\nFigure 1: Nine active incidents, not a flood of raw events — each row is one (entity, rule) pair, re-fires folded into a triggered N× badge, with the firing/recovered rhythm on the timeline above.\nReplay why it fired Click an incident and the detail panel does something most alarm consoles can\u0026rsquo;t: it replays the evidence. Alongside the entity, the firing pill, the message, and the tags, it shows the rule\u0026rsquo;s trigger expression — the MQE that defines it, here sum(service_resp_time \u0026gt; 20) \u0026gt;= 1 — and the metric snapshot OAP captured at the moment it fired.\nThat snapshot is the real metric values from the rule\u0026rsquo;s evaluation window, one per minute bucket, drawn back onto a real-time axis with the trigger moment marked and the window shaded. You see exactly what crossed the threshold — no re-opening a dashboard and guessing which spike was the culprit. The alarm carries its own proof.\nFigure 2: The replay — the rule\u0026rsquo;s MQE expression and the exact values OAP captured at fire time, the five-minute snapshot window shaded (14:09→14:13) and the trigger moment marked. The list at left even tags one rule unstable · 1 firing, 1 recovered — a flapper caught mid-oscillation.\nFrom one incident to its whole history A triggered N× row expands. Click the chevron and the incident unfolds into its firing history — #1 twenty-one minutes ago, #2 fifteen, and so on — each firing (and recovery) in order, so you can tell a rule that flapped a few times from one that has been down solidly since the first alert.\nThe timeline above the list tells the same story at a glance: per-minute columns, red for firing and green for recovered events, each with a count. Click a flag to jump to that minute\u0026rsquo;s alarm; drag to brush a region and slice the list to that window — the timeline keeps every point visible, so a second spike never hides behind the first.\nFigure 3: Expand a triggered 4× incident to see each firing in order — #1 twenty-one minutes ago through #4 three minutes ago — instead of four rows scattered down the page.\nAlarms follow you everywhere The same incident model isn\u0026rsquo;t trapped on the Alarms page; it surfaces wherever you are:\nthe topbar badge polls every minute over a rolling 20-minute window and turns red with the active-incident count, so a new problem reaches you on any screen; the Active alarms dashboard widget lists the current incidents, its title carrying the window (· last 20m) so \u0026ldquo;nothing here\u0026rdquo; is never ambiguous; and on the service topology and the 3D Infrastructure Map, firing services light up red. Every one reads the same rolling window and the same (entity, rule) merge, so the number on the badge, the rows in the widget, and the red nodes on the map always agree.\nFigure 4: The same incidents everywhere — the topbar badge, red-ringed services on the topology, and the Active alarms widget, all reading one shared 20-minute window.\nWhere it runs, and what it isn\u0026rsquo;t Reading active alarms is pure observe — it streams off OAP\u0026rsquo;s query host and works on today\u0026rsquo;s OAP with nothing to enable; viewing is gated on the alarms:read permission. You can narrow the list by keyword, and — where your OAP exposes the newer alarm query — by layer and entity.\nTwo deliberate non-features are worth calling out. Horizon alarms are a read-only mirror of OAP\u0026rsquo;s evaluation state: there is no acknowledge-and-dismiss, and an incident recovers when the condition actually clears — so the page is always the truth, not a worklist someone forgot to tidy. And you don\u0026rsquo;t edit alarm rules here; rules live in OAP\u0026rsquo;s alarm-settings.yml. The live rule context — which OAP node is evaluating each entity, and its silence and recovery-observation countdowns — belongs to the Alerting Rules surface, which rides OAP\u0026rsquo;s admin host and is the subject of the next post.\nWhere to go next For the field reference — the window cap, the snapshot internals, and the pinned-layer setup — see the Alarms docs.\nNext up: Runtime Rules \u0026amp; Live Debugging — editing OAL / MAL / LAL against live samples through OAP\u0026rsquo;s admin host, the part of \u0026ldquo;operate\u0026rdquo; the open-source backend only just made possible.\n","excerpt":"\u003cp\u003eThis is the tenth post in the \u003ca href=\"/blog/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e series, and the first of \u003cstrong\u003eAct 3 — operate it\u003c/strong\u003e. The …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-06-29-horizon-ui-alarms-and-incident-triage/","title":"Meet Horizon UI · 10/17: Alarms \u0026 Incident Triage"},{"body":"This is the eleventh post in the Meet Horizon UI series, and it stays in Act 3 — operate it. The previous post was about reading what the backend already decided; this one is about changing how it decides — and then proving the change does what you meant.\nAlmost everything OAP computes runs through a small family of DSLs: OAL turns traces into service and endpoint metrics, MAL turns meters (OpenTelemetry, Telegraf) into metrics, LAL turns logs into tags and metrics. Traditionally you edit those as YAML on the server and restart. Horizon brings both halves into the console — edit and hot-apply the rules, and debug them against live data — two capabilities new to the SkyWalking UI that ride OAP\u0026rsquo;s admin host.\nYour rules, live in the console Operate → DSL management lists every analysis rule the cluster is running, grouped by source. Four catalogs are editable — MAL · OTel, MAL · Telegraf, LAL, and LAL → MAL (log-to-metric) — plus a read-only OAL browser. Rules group by prefix (ActiveMQ, BanyanDB, Elasticsearch, Flink…), each tagged by status, and you can filter by active / inactive / bundled / modified to see at a glance what an operator has changed versus what shipped.\nFigure 1: DSL management — every OAL/MAL/LAL rule the cluster runs, grouped by source and filterable by status (active / inactive / bundled / modified). Here, the OpenTelemetry MAL catalog: 37 bundled rules.\nEdit, and hot-apply safely Open a rule and it\u0026rsquo;s a Monaco YAML editor with syntax highlighting and two diff modes — vs. server (what\u0026rsquo;s live) and vs. bundled (what shipped) — so you always see what you\u0026rsquo;re about to change. The green ▶ in the gutter beside each - name: jumps that rule straight into the Live Debugger.\nFigure 2: Edit a rule as Monaco YAML — syntax-highlighted, with diffs against the live (server) and bundled versions, and a green ▶ in the gutter that jumps the rule into the Live Debugger.\nSaving is where the care shows. A body- or filter-only edit applies instantly. But a structural change — one that moves a metric\u0026rsquo;s scope, downsampling, or storage shape — reshapes the cluster\u0026rsquo;s storage, so Horizon runs it as a fenced rollout and tracks it on screen: Compiled → Confirming across the cluster → Committing → Done, reporting success only once the change is durable. If a node lags the fence, the apply ends DEGRADED — it names the laggard nodes (they self-converge on their next scan) rather than failing; a pre-commit error is rolled back with the reason inline and your edit kept in the buffer; a compile error surfaces as an inline diagnostic. A one-click Force re-apply re-runs a stuck rollout on byte-identical content to un-stick a node (it briefly pauses that one rule\u0026rsquo;s collection). Reverting a rule to its bundled default goes through the same fenced path; you can also inactivate it, delete it, or dump the whole catalog to a tarball.\nThe Live Debugger: see what a rule actually does Editing a rule is the easy part. The hard part — the part that used to mean reading code and squinting at output — is knowing what a rule computes against your real data. Operate → Live debugger answers that directly: pick a rule, click start sampling, and Horizon installs a bounded capture on every reachable OAP node, grabs a handful of real records, and shows each one stepped through the rule.\nFigure 3: Start a capture and it installs on every reachable OAP node (here 2/2), grabs real records, and bounds itself with a record cap and a retention window — the same shell serves all three analysis languages.\nIt has one tab per analysis language, because each works on a different kind of data.\nOAL → traces. A captured source row — a real trace segment — flows clause by clause: from(Service.*) reads the segment (you see its latency, status, endpoint), build_metrics shapes it, cpm() aggregates it. You watch a trace become a metric.\nFigure 4: OAL → traces — a real segment from agent::gateway (latency 38, status 200, /rcmd) stepped clause by clause, from(Service.*) → build_metrics → cpm(), into the service-CPM metric it feeds.\nMAL → metrics. A meter sample flows input → filter → function → output. Because one metric fans out into many label-sets, the samples are grouped by metric, and a diff dims the labels every sample shares and highlights only the ones that differ.\nFigure 5: MAL → metrics — samples grouped by metric, with a diff that dims the 16 labels every sample shares and lights only the two that differ (group, pod_name), so four near-identical series read apart at a glance.\nLAL → logs. Each captured log record becomes a column and each DSL block (or statement) a row, so the whole capture reads as a matrix: you can see which records the filter aborted and what the extractor pulled out of the ones that passed — and click any cell to open the record in full and compare it against another.\nFigure 6: LAL → logs — every captured record is a column, every DSL block a row. The filter aborts the normal logs; for each abnormal one that passes, the extractor row shows the tag it added (status.code=404) as a diff over the raw record.\nFigure 7: \u0026ldquo;Show complete data\u0026rdquo; opens a record in full — the entire raw log payload (here an Envoy access log) — with a Compare with selector to diff it field-by-field against any other captured record.\nWhere it runs Both surfaces are operate features: they talk to OAP\u0026rsquo;s admin host, not the query port — DSL management through the receiver-runtime-rule module, the Live Debugger through dsl-debugging. That admin host ships with OAP 11, so on today\u0026rsquo;s backend these two pages surface a clear \u0026ldquo;needs the admin host / module\u0026rdquo; banner and stay read-only, while every observe surface — dashboards, traces, logs, alarms, profiling — keeps working untouched. Access is role-gated: browsing rules and viewing captures are read permissions, while editing, structural apply, and running a capture each need their own write or execute verb — so a read-only operator can study captured samples all day without ever being able to change a rule or start a session. This is the slice of \u0026ldquo;operate\u0026rdquo; the open-source backend only just made possible.\nWhere to go next For the field reference — every apply state, the dump format, the per-tab capture controls — see the Runtime Rules and Live Debugger docs.\nNext up: Inspect — Cross-Layer Query Power-Tools — the Operate-side surfaces for running metric, trace, and log queries straight across every layer.\n","excerpt":"\u003cp\u003eThis is the eleventh post in the \u003ca href=\"/blog/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e series, and it stays in \u003cstrong\u003eAct 3 — operate it\u003c/strong\u003e. The …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-06-29-horizon-ui-runtime-rules-and-live-debugging/","title":"Meet Horizon UI · 11/17: Runtime Rules \u0026 Live Debugging"},{"body":"This is the twelfth post in the Meet Horizon UI series, still in Act 3 — operate it. The Trace Explorer and Log Explorer both start the same way: you pick a layer, then a service, then you search. That\u0026rsquo;s the right shape when you\u0026rsquo;re already looking at a service. But sometimes you aren\u0026rsquo;t — you have a trace id and no idea which layer owns it, a failing service name from an alert, or a metric you just want to chart across everything. The Inspect family under Operate is built for exactly that: three cross-layer query power-tools that drop the layer-first step — and one of them has no per-layer equivalent at all.\nMetrics inspect: the metric catalog, and the rule behind every number SkyWalking computes a lot of metrics, and until now there was no way to simply see them all. Metrics inspect is that view. Its catalog drawer lists every metric the connected OAP computes — and groups them by the rule that defines them: the OAL files and MAL rule sets you met in the previous post. Filter by source (OAL, MAL·OTel, MAL·Telegraf, LAL→MAL), search by name, and read each metric\u0026rsquo;s value type and scope at a glance.\nFigure 1: Metrics inspect\u0026rsquo;s catalog — every metric the OAP computes, grouped by the rule that defines it (the OAL files and MAL rule sets from DSL management), filterable by source and scope. Pick metrics onto the board.\nPick metrics from the catalog and they land on a board of charts, where you choose the entities to plot — a paginated top-N from OAP, or hand-entered ones — and read the values back as a line, bar, or area chart. Each widget carries its rule source and scope so you never lose the thread from \u0026ldquo;this number\u0026rdquo; to \u0026ldquo;the rule that produces it.\u0026rdquo; It\u0026rsquo;s an MQE scratchpad: the time range is browser-local but sent to OAP in server time, the board persists in your browser, and metrics that live only in shared storage (not defined on the connected OAP) can be added as foreign metrics.\nFigure 2: The board — chart any metric across entities; each widget keeps its rule source (OAL) and scope (Service), a per-widget entity paginator, and a browser-local range sent to OAP in server time.\nTrace inspect: find a trace without picking a layer Trace inspect is the Trace Explorer with the layer taken off. The Target is optional: pick a service through the layer → service → instance → endpoint cascade, type a name (with a real / conjectured flag), or leave it blank to query every service at once. Set the usual conditions — trace id, status, order, duration bounds, tags, window — and Run query. A resolved-query line spells out the exact call sent to OAP, and the results render as the same distribution scatter, trace list, and three-lens waterfall you already know — just not bound to any one layer.\nFigure 3: Trace inspect — layer-less: leave Target blank to query every service (or pick/type one), then Run query. Here one trace crosses five services (agent::ui → frontend → app → gateway → songs); the resolved-query line shows the exact OAP call (native · queryTraces).\nLog inspect: one query, three log sources Log inspect does the same for logs — \u0026ldquo;query any service across layers, pick it, type its name, or leave it blank\u0026rdquo; — and folds three different log worlds into one place via a Source switch:\nRaw — the stored service logs, streamed across services with tag and trace-id conditions, each row opening the same payload popout as the per-layer Log Explorer; Browser — the BROWSER layer\u0026rsquo;s JS errors by category, with the same source-map de-obfuscation from the Browser Errors post; Kubernetes Pod logs — an on-demand live tail of a pod\u0026rsquo;s container logs, with Start / Pause and Include/Exclude regex filters, never persisted. Figure 4: Log inspect — \u0026ldquo;query any service across layers, or leave it blank,\u0026rdquo; across three sources (Raw stored logs, Browser JS errors, Kubernetes Pod logs). Here raw logs stream from several services at once.\nWhere it runs All three live under Operate and share one permission, inspect:read. They split on the backend, though. Trace inspect and Log inspect ride OAP\u0026rsquo;s standard query protocol — the same one the dashboards and per-layer explorers use — so they\u0026rsquo;re always on and work on any OAP, including 10.x. Metrics inspect is the exception: it reads OAP\u0026rsquo;s metric catalog and entity enumerator through the admin host\u0026rsquo;s inspect module, so it needs OAP 11; when that module is absent it shows a clear \u0026ldquo;set SW_INSPECT=default\u0026rdquo; banner instead of a broken page, while the other two keep working. Think of the trio as the cross-layer, Operate-side counterparts to the per-layer Trace and Log explorers — plus the metric catalog that finally answers \u0026ldquo;what does this backend even measure, and which rule measures it?\u0026rdquo;\nWhere to go next For the field reference — the metric catalog, entity enumeration, foreign metrics, and MQE execution — see the Inspect docs.\nNext up: Platform \u0026amp; Cluster Introspection — Cluster Status, OAP configuration, and data-retention, the last stop in Act 3 before we turn to governing and securing the console.\n","excerpt":"\u003cp\u003eThis is the twelfth post in the \u003ca href=\"/blog/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e series, still in \u003cstrong\u003eAct 3 — operate it\u003c/strong\u003e. The \u003ca href=\"/blog/2026-06-22-horizon-ui-trace-explorer/\"\u003eTrace …\u003c/a\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-06-29-horizon-ui-inspect-cross-layer-query/","title":"Meet Horizon UI · 12/17: Inspect — Cross-Layer Query Power-Tools"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/tags/metrics/","title":"Metrics"},{"body":"译自英文原文：Meet Horizon UI · 10/17: Alarms \u0026amp; Incident Triage。\n这是 Meet Horizon UI 系列的第十篇，也是第三幕 operate it 的第一篇。前几篇都在讲如何把数据看清楚：dashboard、topology、trace、log、profile。这一篇切到故障刚发生的现场。那时最要紧的问题只有两个：现在到底哪里出事了，为什么会出事？\nHorizon 的告警界面抓住两件事：把重复触发归并成 incident；以及回放触发规则时的指标快照。\n把重复触发归并成 incident OAP 每次规则触发都会产生一个告警事件。忙碌服务上的抖动规则，一小时可能触发几十次；如果界面只按原始事件顺序展示，新的问题很快就会被重复事件淹没。Horizon 的归并方式更接近值班工程师看告警的方式：按 (entity, rule) 分组。比如“agent::gateway 的响应时间超过 20ms”，无论触发多少次，都是同一个 incident。\n所以顶层导航里的 Alarms 页面列出的是 incident，而不是一条条事件。KPI 条统计当前 active 的 incident，包括总数和按 layer 拆分的数量；每一行展示 entity、规则消息、layer。如果同一个 incident 反复触发，还会带上 triggered N× badge。一个 incident 会处在三种状态之一：\nfiring：最新一次触发还没有恢复； recovered：触发条件已经消失；它不再计入 active 数量，但仍会作为最近的历史记录保留； unstable：触发、恢复、又再次触发。triggered N× badge 会把这种反复抖动的规则暴露出来。 Alarms 页面还使用自己的时间窗口：20m / 2h / 4h，或者最长四小时的自定义区间；它不跟全局 topbar 的时间同步。你可以回看过去两小时的告警历史，不会影响正在查看的 dashboard。\n图 1：九个 active incident，而不是满屏重复告警；每一行都是一个 (entity, rule) 组合，重复触发被折叠进 triggered N× badge，上方时间线展示 firing/recovered 的节奏。\n回放触发证据 点开一个 incident，详情面板会做多数告警控制台做不到的事：回放证据。除了 entity、firing 状态、消息和 tags，它还会展示规则的触发表达式，也就是定义这条规则的 MQE，例如 sum(service_resp_time \u0026gt; 20) \u0026gt;= 1；以及 OAP 在触发瞬间捕获的指标快照。\n这个快照不是事后再查一遍图表，而是规则评估窗口里真实参与判断的指标值，每分钟一个 bucket。Horizon 把这些点画回实时图表的时间轴上，标出触发时刻，并给快照窗口加上阴影。你看到的是到底哪些值越过阈值；不用再打开 dashboard 猜是哪次 spike 造成告警。告警本身就带着证据。\n图 2：回放详情：规则的 MQE 表达式、OAP 在触发时捕获的准确数值、五分钟快照窗口（14:09→14:13）的阴影，以及触发时刻标记。左侧列表里还有一个 unstable · 1 firing, 1 recovered，正好是一条正在抖动的规则。\n从一次触发看到完整历史 带 triggered N× 的行可以展开。点开箭头后，这个 incident 会展开成完整触发历史：#1 二十一分钟前、#2 十五分钟前，依次往下。每次触发和恢复都按时间排列。这样一眼就能区分：这是一条偶尔抖几下的规则，还是第一次告警后就一直没有恢复的问题。\n列表上方的 timeline 则用分钟粒度展示整体节奏：红色表示 firing，绿色表示 recovered，每列带数量。点击 flag 可以跳到对应分钟的告警；拖拽选区则只看那段时间。timeline 会保留每个点，所以第二次 spike 不会被第一次盖住。\n图 3：展开 triggered 4× 后，可以按时间看到四次触发：从二十一分钟前的 #1 到三分钟前的 #4，而不是在页面上散落成四行。\n告警不只在 Alarms 页面 Alarms 页面不是唯一入口。同一套 incident 模型还会跟着你出现在这些地方：\ntopbar badge 每分钟轮询一次最近 20 分钟的滚动窗口；有 active incident 时变红并显示数量，所以你在哪个页面都能看到新问题； dashboard 的 Active alarms 组件列出当前 incidents，标题里带窗口（· last 20m），避免你看到空列表时还要猜它查的是哪个时间段； 在 service topology 和 3D Infrastructure Map 上，处于 firing 的服务会标红。 这些入口使用的是同一个滚动窗口、同一种 (entity, rule) 归并逻辑，所以 badge 上的数字、组件里的行、地图上的红点会互相对得上。\n图 4：同一批 incident 出现在不同入口：topbar badge、topology 上带红圈的服务，以及 Active alarms 组件，都读取同一个 20 分钟滚动窗口。\n它在哪里运行，又刻意不做什么 读取 active alarms 是纯 observe 操作：数据来自 OAP 的 query host，当前 OAP 上不需要额外配置；查看受 alarms:read 权限控制。你可以按关键词收窄列表；如果 OAP 暴露了新版 alarm query，也可以按 layer 和 entity 过滤。\n有两件事是 Horizon 有意不做的。第一，Horizon alarms 只是 OAP 评估状态的只读镜像：没有 acknowledge-and-dismiss；只有条件真的清除，incident 才会 recovered。所以这个页面呈现的是事实状态，不是一个靠人手清理的待办列表。第二，这里不编辑告警规则；规则仍然在 OAP 的 alarm-settings.yml。实时规则上下文——哪个 OAP node 在评估哪个 entity，以及 silence 和 recovery-observation 的倒计时——属于 Alerting Rules 界面。那个界面走 OAP admin host，是下一篇的主题。\n后续阅读 字段参考，包括窗口上限、快照内部结构和 pinned-layer 配置，可以看 Alarms 文档。\n下一篇：运行时规则与实时调试：通过 OAP admin host，用真实数据编辑并验证 OAL / MAL / LAL。\n","excerpt":"\u003cp\u003e\u003cem\u003e译自英文原文：\u003ca href=\"/blog/2026-06-29-horizon-ui-alarms-and-incident-triage/\"\u003eMeet Horizon UI · 10/17: Alarms \u0026amp; Incident Triage\u003c/a\u003e。\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e这是 \u003ca href=\"/zh/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e 系列的第十篇，也是第三幕 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-06-29-horizon-ui-alarms-and-incident-triage/","title":"认识 Horizon UI · 10/17：告警与 Incident 排查"},{"body":"译自英文原文：Meet Horizon UI · 11/17: Runtime Rules \u0026amp; Live Debugging。\n这是 Meet Horizon UI 系列的第十一篇，仍然属于第三幕 operate it。上一篇 讲的是如何读取后端已经判断出的告警；这一篇更进一步，讲如何改变后端的判断逻辑，并验证这次修改确实按预期生效。\nOAP 里的大多数分析都经过一组小型 DSL：OAL 把 trace 转成 service 和 endpoint 指标，MAL 把 meter（OpenTelemetry、Telegraf）转成指标，LAL 把 log 转成 tag 和指标。过去，这些规则通常要在服务器上改 YAML，再重启服务。Horizon 把两件事都放进控制台：编辑规则并在线生效，以及用实时数据调试规则。这两项都是 SkyWalking UI 新增的能力，底层走 OAP 的 admin host。\n规则直接在控制台里看 Operate → DSL management 会列出集群当前正在运行的分析规则，并按来源分组。四类目录可编辑：MAL · OTel、MAL · Telegraf、LAL 和 LAL → MAL（log-to-metric）；另外还有只读的 OAL 浏览器。规则会按前缀归组，比如 ActiveMQ、BanyanDB、Elasticsearch、Flink 等。每条规则都有状态标记，也可以按 active / inactive / bundled / modified 过滤，这样可以很快看出哪些是随版本发布的规则，哪些被 operator 改过。\n图 1：DSL management：集群运行的 OAL/MAL/LAL 规则按来源分组，并可按 active / inactive / bundled / modified 过滤。这里展示的是 OpenTelemetry MAL 目录，包含 37 条 bundled 规则。\n编辑，并安全地在线生效 打开一条规则后，界面是一个 Monaco YAML 编辑器，带语法高亮和两种 diff：vs. server 看当前线上版本，vs. bundled 看随版本发布的默认版本。这样你在保存前就能确认自己改了什么。每个 - name: 旁边还有绿色 ▶，点击后会把这条规则直接带到 Live Debugger。\n图 2：用 Monaco YAML 编辑规则：语法高亮、对比线上版本和 bundled 版本，并可通过代码左侧的绿色 ▶ 直接进入 Live Debugger。\n保存时，Horizon 会区分修改的风险。只改规则主体或 filter，可以立即生效。但如果是结构性变更，比如改了指标的 scope、downsampling 或存储形态，就会影响集群存储结构。Horizon 会按集群确认流程执行，并在界面上展示进度：Compiled → Confirming across the cluster → Committing → Done；只有变更在集群里确认持久化后才算成功。\n如果某个节点没有及时通过确认，应用结果会变成 DEGRADED。界面会列出落后的节点，这些节点会在下次扫描时自行追上，而不是让整次应用直接失败。如果 commit 前出错，变更会 rolled back，原因显示在界面里，你的编辑内容仍保留在 buffer 中。编译错误则会作为 inline diagnostic 展示。对于卡住的发布，可以点一次 Force re-apply，用完全相同的内容重新跑一遍应用流程，让落后的节点恢复同步；这会短暂暂停那条规则的采集。把规则恢复到 bundled 默认版本也走同一套确认流程；此外也可以 inactivate、delete，或者把整个目录导出成 tarball。\nLive Debugger：看清规则实际算出了什么 编辑规则只是第一步。更难的是确认它跑在真实数据上到底会算出什么。过去通常要读代码、对输出、靠经验判断。Operate → Live debugger 直接把这件事放到界面里：选择一条规则，点击 start sampling，Horizon 会在每个可达的 OAP 节点上安装一个受限采集任务，抓取少量真实记录，然后逐条展示这些记录如何经过规则处理。\n图 3：启动采集后，任务会安装到每个可达的 OAP 节点上（这里是 2/2），抓取真实记录，并用 record cap 和 retention window 控制边界。三种分析语言共用这套会话框架。\nLive Debugger 按分析语言分成三个标签页，因为三种规则处理的数据不同。\nOAL → traces。 捕获到的一行 source 是真实的 trace segment。它会按 clause 展开：from(Service.*) 读取 segment（可以看到 latency、status、endpoint），build_metrics 组织指标结构，cpm() 做聚合。你可以直接看到一条 trace 如何变成指标。\n图 4：OAL → traces：来自 agent::gateway 的真实 segment（latency 38，status 200，/rcmd）逐步经过 from(Service.*) → build_metrics → cpm()，最终进入 service-CPM 指标。\nMAL → metrics。 一个 meter sample 会按 input → filter → function → output 流动。因为同一个指标往往会展开成多组 label，样本会按 metric 分组；diff 会淡化所有样本共有的 label，只高亮不同的部分。\n图 5：MAL → metrics：sample 按 metric 分组，diff 会淡化 16 个所有样本共有的 label，只高亮不同的两个 label（group、pod_name）。四条非常相似的时序因此能一眼区分。\nLAL → logs。 每条捕获到的 log record 是一列，每个 DSL block（或 statement）是一行，所以整个采集结果会变成一个矩阵。你可以看到哪些记录被 filter aborted，也能看到通过 filter 的记录被 extractor 提取出了什么；点开任意一个格子，还能查看这条记录的完整内容，并和另一条记录逐字段对比。\n图 6：LAL → logs：每条捕获记录是一列，每个 DSL block 是一行。filter 会丢弃正常日志；对通过的异常日志，extractor 行会以 diff 的形式显示它新增的标签（这里是 status.code=404）。\n图 7：点击格子上的“show complete data”，即可查看整条捕获记录的原始内容（这里是一条 Envoy access log），并通过 Compare with 选择器与其他任意记录逐字段对比。\n它在哪里运行 这两个界面都属于 operate 功能：它们访问的是 OAP 的 admin host，不是 query port。DSL management 走 receiver-runtime-rule 模块，Live Debugger 走 dsl-debugging。admin host 随 OAP 11 提供；在当前后端上，这两个页面会明确提示“需要 admin host / module”，并保持只读。与此同时，所有 observe 界面不受影响：dashboard、trace、log、alarm、profiling 都照常工作。\n权限也按角色拆开：浏览规则、查看采集结果是读权限；编辑规则、执行结构性应用、启动采集分别需要对应的写权限或执行权限。因此，只读 operator 可以一直查看采集样本，但不能改规则，也不能启动新的调试会话。这正是开源后端最近才补上的那块 operate 能力。\n后续阅读 字段参考，包括每个 apply state、dump 格式和各标签页的采集控制，可以看 Runtime Rules 和 Live Debugger 文档。\n下一篇：Inspect，跨 layer 查询工具：在 Operate 界面里跨 layer 直接运行 metric、trace 和 log 查询。\n","excerpt":"\u003cp\u003e\u003cem\u003e译自英文原文：\u003ca href=\"/blog/2026-06-29-horizon-ui-runtime-rules-and-live-debugging/\"\u003eMeet Horizon UI · 11/17: Runtime Rules \u0026amp; Live Debugging\u003c/a\u003e。\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e这是 \u003ca href=\"/zh/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e 系列的第十一篇，仍然属于第 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-06-29-horizon-ui-runtime-rules-and-live-debugging/","title":"认识 Horizon UI · 11/17：运行时规则与实时调试"},{"body":"译自英文原文：Meet Horizon UI · 12/17: Inspect — Cross-Layer Query Power-Tools。\n这是 Meet Horizon UI 系列的第十二篇，仍然属于第三幕 operate it。Trace Explorer 和 Log Explorer 的入口方式很一致：先选 layer，再选 service，然后搜索。如果你已经知道要看哪个服务，这个流程很顺。但有些时候，你并不知道入口在哪：手里只有一个 trace id，却不知道它属于哪个 layer；告警里只有一个出问题的 service name；或者你只是想把某个指标拿出来，在所有实体上画一遍。Operate 下的 Inspect 家族就是为这些场景准备的：三个跨 layer 查询入口，去掉“先选 layer”这一步；其中一个甚至没有对应的单 layer 版本。\nMetrics inspect：指标目录，以及每个指标背后的规则 SkyWalking 会计算大量指标，但过去没有一个地方能把它们完整列出来。Metrics inspect 补上了这个视图。它的 catalog drawer 会列出当前连接的 OAP 计算出的所有指标，并按定义这些指标的规则分组：也就是上一篇里讲过的 OAL 文件和 MAL 规则集。你可以按来源过滤（OAL、MAL·OTel、MAL·Telegraf、LAL→MAL），也可以按名称搜索，并直接看到每个指标的 value type 和 scope。\n图 1：Metrics inspect 的指标目录：OAP 计算的所有指标按定义规则分组，也就是 DSL management 里的 OAL 文件和 MAL 规则集；可以按 source 和 scope 过滤，再把指标选到看板上。\n从目录里选中指标后，它们会进入一个图表 board。你可以选择要画哪些实体：从 OAP 返回的分页 top-N 里选，或者手动输入实体名；图表可以用 line、bar 或 area 展示。每个 widget 都会带上规则来源和 scope，所以你始终能从“这个数”追溯到“是哪条规则算出了这个数”。它也可以当作一个 MQE 临时看板：时间范围保存在浏览器本地，但发送给 OAP 时会转成服务端时间；board 本身也保存在浏览器里；那些只存在于共享存储、但不由当前连接的 OAP 定义的指标，也可以作为 foreign metrics 加进来。\n图 2：Inspect board：任选一个指标，在多个实体上画图；每个 widget 保留规则来源（OAL）、scope（Service）、独立的 entity 分页器，以及一个浏览器本地保存、提交给 OAP 时转换成服务端时间的时间范围。\nTrace inspect：不用先选 layer，也能找 trace Trace inspect 可以理解成拿掉 layer 限制的 Trace Explorer。Target 是可选的：你可以通过 layer → service → instance → endpoint 级联选择服务，也可以直接输入一个名字（并标记它是真实存在还是推测出来的），还可以留空 Target，一次查询所有 service。之后照常设置查询条件：trace id、status、排序、duration 范围、tags 和时间窗口，然后点击 Run query。界面会显示一行解析后的查询，写清楚实际发给 OAP 的调用；结果仍然是你熟悉的分布散点图、trace 列表和三视角 waterfall，只是不再绑在某个 layer 上。\n图 3：Trace inspect 不需要先选 layer：Target 留空即可查询所有 service，也可以选择或输入某个 service。这里一条 trace 跨过五个服务（agent::ui → frontend → app → gateway → songs）；解析后的查询行展示了实际 OAP 调用（native · queryTraces）。\nLog inspect：一次入口，三类日志 Log inspect 对 log 做同样的事：可以跨 layer 查询任意 service，选择它、输入它的名字，或者直接留空。它还通过 Source 切换，把三类日志放到同一个入口里：\nRaw：存储下来的 service logs，可以跨 service 流式查询，支持 tag 和 trace id 条件；每一行都能打开和单 layer Log Explorer 相同的 payload 弹窗； Browser：来自 BROWSER layer 的 JS errors，按 category 查询，并使用 Browser Errors 那篇里讲过的 source map 反混淆； Kubernetes Pod logs：按需 live tail 某个 pod 的 container logs，支持 Start / Pause 和 Include/Exclude 正则过滤，不会持久化。 图 4：Log inspect 可以跨 layer 查询任意 service，也可以留空 target；三种 source 分别对应 Raw 存储日志、Browser JS errors 和 Kubernetes Pod logs。这里展示的是多个 service 同时输出的 raw logs。\n它在哪里运行 这三个入口都在 Operate 下，共用一个权限：inspect:read。但它们访问后端的方式不同。Trace inspect 和 Log inspect 走 OAP 标准 query protocol，也就是 dashboard 和单 layer explorer 使用的同一套接口；所以它们始终可用，也兼容 10.x OAP。Metrics inspect 是例外：它通过 admin host 的 inspect 模块读取 OAP 的指标目录和实体枚举，因此需要 OAP 11。如果模块不存在，页面会给出明确的 \u0026ldquo;set SW_INSPECT=default\u0026rdquo; 提示，而不是只显示一个不可用的页面；另外两个入口仍然可以正常使用。可以把这组三个入口看成 Trace 和 Log explorer 在 Operate 侧的跨 layer 版本，再加上一个终于能回答“这个后端到底在算哪些指标、这些指标由哪条规则定义”的指标目录。\n后续阅读 字段参考，包括指标目录、实体枚举、foreign metrics 和 MQE 执行，可以看 Inspect 文档。\n下一篇是 Platform \u0026amp; Cluster Introspection：Cluster Status、OAP configuration 和 data-retention。它是第三幕的最后一站，之后这个系列会转向控制台的治理和安全。\n","excerpt":"\u003cp\u003e\u003cem\u003e译自英文原文：\u003ca href=\"/blog/2026-06-29-horizon-ui-inspect-cross-layer-query/\"\u003eMeet Horizon UI · 12/17: Inspect — Cross-Layer Query Power-Tools\u003c/a\u003e。\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e这是 \u003ca href=\"/zh/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e 系列的第十二篇， …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-06-29-horizon-ui-inspect-cross-layer-query/","title":"认识 Horizon UI · 12/17：Inspect，跨 layer 查询工具"},{"body":"This is the ninth post in the Meet Horizon UI series. Metrics tell you what slowed down; traces tell you which hop. Profiling goes one level deeper — into the call stacks, kernel events, and process-to-process conversations of a running service — to tell you where in the code. SkyWalking has five different profilers for that, and Horizon surfaces all of them. The headline of this post: four of the five pour into one shared flame graph, and the fifth is a deliberate exception.\nOne renderer, four profilers Trace, async, eBPF, and pprof profiling all produce the same fundamental thing — a tree of stack frames with sample counts — so Horizon normalizes them into one shape and renders them through one flame-graph component (a wrapper over d3-flame-graph). The payoff is that you learn the view once and it works the same everywhere:\neach frame\u0026rsquo;s width is its share of the samples, and the hover card reads out the code signature, the dump count, the time spent (including and excluding children), and the frame\u0026rsquo;s % of root; clicking a frame zooms into it and pins a highlight on it — and that selected-frame highlight is consistent across all four profilers; a dim, per-frame color keyed off the method name keeps a thousand-frame graph legible on the dark canvas. Figure 1: One flame graph for four profilers — frames by sample share, the selected frame pinned, the hover card with % of root.\nOn the Trace and eBPF tabs you can flip the same data to a Tree view instead — an indented stack table with each method\u0026rsquo;s total vs self duration and its dump count, expandable frame by frame. (Async and pprof are flame-graph-only; the toggle shows up where both views apply.)\nFigure 2: The same result, one toggle away — the Tree view swaps the flame for an indented stack table carrying total vs self duration and dump count.\nWhat each of the four catches The four stack profilers share the renderer but answer different questions, and each has its own New Task form:\nTrace Profiling samples the call stacks of slow trace segments. Scope a task to a service (and optionally one endpoint), set a slowness threshold and a dump period, and the agent snapshots thread stacks from segments that cross the threshold. Then you pick a sampled trace, drill to a profiled span, and Analyze — with a data mode that includes or excludes child-span time. Async Profiling runs the JVM async-profiler against a live Java service with no restart. A task can target several instances and several events at once — CPU, ALLOC, LOCK, WALL, and the timer events — and an event-type selector re-draws the flame for whichever one you want to read. eBPF Profiling captures kernel-level stacks with no in-process agent, driven by SkyWalking Rover: ON_CPU (where the process burns CPU) or OFF_CPU (where it\u0026rsquo;s blocked — on locks, I/O, scheduling). A process picker lets you expand a process\u0026rsquo;s attributes and pin the ones to profile, and an aggregate toggle counts samples or sums blocked time (the latter only makes sense off-CPU). pprof profiles a live Go service through the standard runtime profiler — exactly one event per task, chosen from CPU, HEAP, BLOCK, MUTEX, GOROUTINE, ALLOCS, and THREADCREATE. The dialog adapts to the choice: a duration for the timed captures, a sampling rate for BLOCK/MUTEX, and a one-shot snapshot for the rest. Figure 3: pprof takes exactly one Go event per task — GOROUTINE, MUTEX, and CPU are separate tasks, each with its own duration and sampling rate; select one and Analyze pours it into the same flame graph.\nNetwork Profiling: the deliberate exception The fifth profiler answers a different kind of question — not \u0026ldquo;where is one process spending time\u0026rdquo; but \u0026ldquo;which processes are talking to which, and over what\u0026rdquo; — so it renders differently on purpose. Network Profiling captures the network conversations between the processes of a service instance and draws them as a honeycomb topology: each process is a hexagon, the instance\u0026rsquo;s own processes pack into the centre under a dashed pod boundary, and external peers ring the edge. The links between them are directed and animated, and colored by protocol — HTTPS, TLS, HTTP, and plain TCP each get their own hue and a small pill.\nIt also runs differently: instead of a fixed duration, a network task carries sampling rules — match by URI pattern, by 4xx/5xx responses, or by a minimum duration, and choose how much of each request/response body to keep — and keeps running until you stop it. Click an edge and a Client side | Server side panel opens with that conversation\u0026rsquo;s call rate, latency, and bytes charted over the window. It\u0026rsquo;s drawn from the same process-relation data that powers the 3D Infrastructure Map — and there\u0026rsquo;s not a flame graph in sight.\nFigure 4: The odd one out — process conversations as a honeycomb. In-pod processes pack inside the dashed pod boundary, external peers ring it, and every edge is colored by protocol; clicking one opens its client-vs-server metrics.\nOne task model, two permissions For all the differences in what they capture, every profiling tab is the same workflow: a task list on the left, a New Task control, and a result panel on the right. Create a task and the list polls for a few rounds until OAP has dispatched it and the instances report back; select a task to analyze it.\nThat create-versus-read split is also a permission boundary. Starting a task needs profile:enable (an operator-and-above default) — because an unbounded profile could peg a production instance\u0026rsquo;s CPU, so the task forms are duration- and size-capped on the server. Reading a result needs only profile:read (part of the read-only data catalog). So a viewer can sit with a flame graph all day and never be able to launch a profile.\nWhich tabs you even see depends on the service: a tab appears only when OAP reports that the service supports that kind of profiling. In practice the General agent layer carries the four stack engines (trace, eBPF, async, pprof), eBPF rides wherever Rover is deployed, and network profiling lights up on the service mesh.\nWhere to go next For the field reference — every task field, the eBPF aggregate modes, the network sampling rules — see the Profiling docs.\nNext up: Alarms \u0026amp; Incident Triage — the incident-centric alarm surface, and replaying the MQE snapshot that fired a rule.\n","excerpt":"\u003cp\u003eThis is the ninth post in the \u003ca href=\"/blog/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e series. Metrics tell you \u003cem\u003ewhat\u003c/em\u003e slowed down; \u003ca href=\"/blog/2026-06-22-horizon-ui-trace-explorer/\"\u003etraces\u003c/a\u003e tell …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-06-26-horizon-ui-profiling/","title":"Meet Horizon UI · 9/17: Five Profilers, One Flame Graph"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/tags/profiling/","title":"Profiling"},{"body":"SkyWalking NodeJS 0.9.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed Modify mongoose call xxxx() throwing \u0026ldquo;is not a function\u0026rdquo; by @shivendoodeshmukh in https://github.com/apache/skywalking-nodejs/pull/128 fix: add destroy methods to prevent memory leaks in protocol clients by @wolfsilver in https://github.com/apache/skywalking-nodejs/pull/129 Bugfix: Pg plugin can\u0026rsquo;t collect paramter values by @z2015 in https://github.com/apache/skywalking-nodejs/pull/131 fix: prevent OOM when collector unreachable; upgrade grpc-js to 1.14.4 by @wu-sheng in https://github.com/apache/skywalking-nodejs/pull/132 chore: add release automation scripts (release.sh + release-finalize.sh) by @wu-sheng in https://github.com/apache/skywalking-nodejs/pull/133 New Contributors @shivendoodeshmukh made their first contribution in https://github.com/apache/skywalking-nodejs/pull/128 @wolfsilver made their first contribution in https://github.com/apache/skywalking-nodejs/pull/129 @z2015 made their first contribution in https://github.com/apache/skywalking-nodejs/pull/131 Full Changelog: https://github.com/apache/skywalking-nodejs/compare/v0.8.0...v0.9.0\n","excerpt":"\u003cp\u003eSkyWalking NodeJS 0.9.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-nodejs-0.9.0/","title":"Release Apache SkyWalking for NodeJS 0.9.0"},{"body":"译自英文原文：Meet Horizon UI · 9/17: Five Profilers, One Flame Graph。\n这是 Meet Horizon UI 系列的第九篇。指标告诉你 什么 变慢了；Trace 告诉你慢在哪一跳；Profiling 再往下一层，进入运行中服务的调用栈、内核事件和进程间通信，告诉你问题落在 哪段代码。SkyWalking 为这件事提供了五种 profiler，Horizon 会把它们都展示出来。这篇的主线是：五种里有四种进入同一套火焰图，第五种则是刻意设计的例外。\n一个渲染器，四种 profiler Trace、async、eBPF 和 pprof 这四类 profiling 最终都会产出同一类数据：一棵带采样计数的 stack frame 树。Horizon 先把它们归一成同一种结构，再交给 同一个火焰图组件（基于 d3-flame-graph 封装）渲染。好处很直接：你只需要学一次这个视图，之后四种 profiling 都按同样方式读：\n每个 frame 的宽度代表它占全部样本的比例；hover 卡片会显示代码签名、dump 次数、耗时（包含和 不包含 子调用），以及该 frame 占根节点的 % of root； 点击一个 frame 会缩放进去，并把选中高亮固定住；这个选中态在四种 profiler 里保持一致； 每个 frame 使用由方法名决定的低饱和度颜色，让上千个 frame 的图在暗色画布上仍然能读。 图 1：四种 profiler 共用一套火焰图：frame 按样本占比展开，选中 frame 会固定高亮，hover 卡片显示 % of root。\n在 Trace 和 eBPF 标签页里，同一份分析结果还可以切到 Tree 视图：它是一张缩进的 stack 表，逐帧展示每个方法的 total 和 self duration，以及 dump count。（Async 和 pprof 只提供火焰图；只有同时支持两种视图的地方才会出现这个切换。）\n图 2：同一份结果，一次切换即可从火焰图变成 Tree：缩进 stack 表展示 total/self duration 和 dump count。\n四种 stack profiler 分别抓什么 这四种 stack profiler 共用渲染器，但回答的问题不同，每种也有自己的 New Task 表单：\nTrace Profiling 会对 慢 trace segment 的调用栈采样。创建任务时指定 service（也可以限定 endpoint）、慢调用 threshold 和 dump period。segment 超过阈值时，agent 会抓取线程栈快照。之后你选择一条采样到的 Trace，下钻到带 profiling 的 span，再点 Analyze。这里还有一个 data mode，可以选择是否把 child span 时间计入结果。 Async Profiling 在运行中的 Java 服务上启动 JVM async-profiler，不需要重启。一个任务可以同时覆盖多个实例和多个事件：CPU、ALLOC、LOCK、WALL 以及 timer 类事件。选择不同 event type 后，火焰图会按对应事件重新绘制。 eBPF Profiling 不需要进程内 agent，由 SkyWalking Rover 在内核层抓 stack：ON_CPU 看进程把 CPU 花在哪里，OFF_CPU 看它阻塞在哪里，比如锁、I/O、调度。进程选择器可以展开进程属性，固定要剖析的进程；聚合开关可以选择统计样本数，或者累加 blocked time（后者只适合 off-CPU）。 pprof 通过 Go 标准 runtime profiler 剖析运行中的 Go 服务。每个任务只能选择 一个 event，来自 CPU、HEAP、BLOCK、MUTEX、GOROUTINE、ALLOCS 和 THREADCREATE。对话框会跟随 event 调整：定时采集需要 duration，BLOCK/MUTEX 需要 sampling rate，其余则是一次性快照。 图 3：pprof 每个任务只采一个 Go event：GOROUTINE、MUTEX 和 CPU 是不同任务，各自带 duration 和 sampling rate；选中后 Analyze，同样进入火焰图。\nNetwork Profiling：刻意设计的例外 第五种 profiler 问的是另一类问题：不是“一个进程把时间花在哪里”，而是“哪些进程在通信、通过什么协议通信”。所以它刻意不用火焰图。Network Profiling 会捕获某个服务实例内进程之间的网络会话，并画成 蜂窝拓扑：每个进程是一个六边形，实例自身的进程聚在虚线 pod 边界内，外部 peers 围在边缘。它们之间的边有方向、有动画，并按协议着色：HTTPS、TLS、HTTP 和普通 TCP 都有自己的颜色和小标签。\n它的运行方式也不同：network task 不是固定时长，而是带 sampling rules。你可以按 URI pattern、4xx/5xx 响应或最小时延匹配，并配置保留多少 request/response body。任务会一直运行，直到你手动停止。点击一条边，会打开 Client side | Server side 面板，展示这段会话在当前窗口内的调用速率、时延和字节数图表。它使用的是和 3D Infrastructure Map 同源的 process-relation 数据。这里看不到火焰图，这正是设计。\n图 4：这个 profiler 是例外：进程通信画成蜂窝拓扑。pod 内进程聚在虚线边界内，外部 peers 围在外侧，每条边按协议着色；点击边会打开 client-vs-server 指标。\n同一套任务模型，两类权限 虽然五种 profiler 抓取的内容不同，每个 profiling 标签页的操作流程是一样的：左侧是 任务列表，上方有 New Task，右侧是 结果面板。创建任务后，列表会轮询几轮，等待 OAP 下发任务、实例回报结果；选中一个任务后再分析。\n创建任务和读取结果也是一条权限边界。启动任务需要 profile:enable（默认 operator 及以上拥有），因为没有边界的 profile 可能把生产实例 CPU 打满，所以任务表单的时长和数据大小都在服务端限额。读取 结果只需要 profile:read（属于只读数据权限）。所以 viewer 可以一直看火焰图，但不能发起 profiling 任务。\n你能看到哪些标签页，也取决于当前服务：只有 OAP 上报该服务支持某类 profiling 时，对应标签页才会出现。实际使用中，General agent Layer 会带上四个 stack 引擎（trace、eBPF、async、pprof）；部署了 Rover 的地方会有 eBPF；service mesh 上会出现 network profiling。\n后续阅读 字段参考，包括每个任务字段、eBPF 聚合模式和 network sampling rules，可以看 Profiling 文档。\n下一篇：告警与 Incident 排查：Horizon 如何把重复告警归并成 incident，并回放触发规则时的 MQE 指标快照。\n","excerpt":"\u003cp\u003e\u003cem\u003e译自英文原文：\u003ca href=\"/blog/2026-06-26-horizon-ui-profiling/\"\u003eMeet Horizon UI · 9/17: Five Profilers, One Flame Graph\u003c/a\u003e。\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e这是 \u003ca href=\"/zh/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e 系列的第九篇。指标告诉你 \u003cem\u003e什么\u003c/em\u003e 变 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-06-26-horizon-ui-profiling/","title":"认识 Horizon UI · 9/17：五种 Profiler，一套火焰图"},{"body":"SkyWalking Horizon UI 0.7.0 is released. Go to downloads page to find release tars.\nThis release broadens what Horizon can observe — browser JS errors with source-map de-obfuscation, an Airflow layer, and a clustered BanyanDB self-observability layer — and deepens how you navigate a deployment: lock-and-compare across entities on a layer dashboard, three new topology drill-downs (instance map, intra-service deployment, API dependency), and portable dashboard config with OAP as the single source of truth.\nBrowser errors \u0026amp; source maps A new Browser Logs tab on the BROWSER layer lists the JS error logs the browser agent reports — message, category, page, app version, time, and the minified line:col — filterable by category and time window. Expanding a row shows the raw stack alongside a de-obfuscated view. Source-map de-obfuscation (skywalking#6784): upload a .map file and resolve any error\u0026rsquo;s minified stack back to the original file, line, column, symbol name, and a source snippet by picking which map to apply. Uploaded maps live in the BFF\u0026rsquo;s memory only (no backend storage) — surfaced as temporary, LRU-evicted at the configured budget, and lost on restart. For durable provisioning, mount .map files into the server\u0026rsquo;s static source-map directory (HORIZON_SOURCEMAPS_DIR, /app/sourcemaps in the image); those reload automatically and can\u0026rsquo;t be deleted from the UI. Budgets are tuned via the new sourceMaps block in horizon.yaml (defaults 64 MiB per-file / 512 MiB total). Upload/delete require the new source-map:write permission; viewing and resolving ride on browser-errors:read. Two new layers: Airflow \u0026amp; BanyanDB self-observability Airflow (SWIP-7), under Workflow Scheduler: a service dashboard (scheduler / executor / pool KPIs and trends), a Components dashboard (per-host scheduler and triggerer metrics for Airflow 3.x native OTel), and a 3D Infra Map load ring for Tasks Executable. Pairs with OAP backend SWIP-7 (meter_airflow_*). BanyanDB (SWIP-15), under Self-Observability: models a clustered, role- and tier-aware BanyanDB deployment scraped through its FODC proxy — the cluster is one Cluster (service), each container a Container (instance, carrying its container_name role and node_type tier), and each storage Group an endpoint. It ships a Cluster dashboard (write/query/error-rate, capacity, a Containers-by-Role table), a role-adaptive Container dashboard (shared CPU/memory/Go-runtime resources, plus liaison ingestion / data storage / lifecycle migration panels gated on each container\u0026rsquo;s role), a per-data-model Group dashboard (measure / stream / trace / property), and a Deployment tab rendering the container inventory with role-pair-specific call edges. The whole deployment model is editable from Layer dashboards admin → Deployment scope. Pairs with OAP backend SWIP-15 (meter_banyandb_*). Lock \u0026amp; compare entities on a dashboard Pin several services, instances, or endpoints — including ones from different services — and compare them in place. Compare is standard on every service / instance / endpoint layer dashboard; nothing to enable. The entity you\u0026rsquo;re viewing is always part of the comparison, tagged CURRENT and shown first; pinned entities add to it, up to six, each in its own stable hue. Each widget compares inline in its own tile — line widgets overlay one series per entity, card widgets show one row each, top-N and record widgets get per-entity tabs plus a merged All tab, and table widgets gain an Entity column. Each entity loads as its own request, so tiles fill in progressively and one slow or failed entity never blanks the others. The Topology, Deployment, trace, and log pages are unaffected. New topology drill-downs Instance map — click a call between two services, then Instance map →, to open each service\u0026rsquo;s instances as client / server columns with the instance-level calls between them: pan/zoom, animated flow, per-call client/server metric sidebar, and relationship-aware pair pickers drawn from the call graph. Configurable per layer (Layer dashboards → Topology → Enable instance topology), on out of the box for General, Service Mesh, Kubernetes Service, and Cilium Service. Deployment tab — the instance-to-instance call graph within a single service (e.g. a clustered store\u0026rsquo;s nodes calling each other). Instances render as hexagons that bundle into pods (main + sibling containers); cluster them by one or more instance attributes or a name regex; a tiered, draggable layout reads upstream→downstream left-to-right. Edge metrics are keyed by (source-role → target-role) pair, with a primary metric printed inline and a Flows sub-tab tabling every edge per role-pair. Off by default; opt in per layer. API dependency tab — an endpoint\u0026rsquo;s caller → callee chain as a column graph (callers left, focus centre, callees right) with the same health-ring borders, SLA-coloured RPM, and latency as the service map. A single + handle pulls in an endpoint\u0026rsquo;s own callers and callees to walk the chain; drill-outs open in a new browser tab. Localized across all eight UI languages. Topology readability A new Filter control on the per-layer service map (and the embedded topology widget) hides the conjectured peers that clutter a dense map — faceted by layer (each row carrying the layer\u0026rsquo;s own icon and localized name), plus a standalone User toggle and an Others bucket for unresolved peers. Filtering is client-side and defaults to showing everything. Technology component icons now render on service-map nodes — the same icon set the trace waterfall uses, so a PostgreSQL node looks like PostgreSQL — falling back to the generic glyph when a component ships no icon. The topology\u0026rsquo;s service selector now groups its list by OAP Service.group; clicking a group header batch-selects or unselects every service in that group. Service group as a first-class layer axis A per-layer Split menu by service group toggle fans a layer into one sidebar entry per OAP Service.group (the \u0026lt;group\u0026gt;:: prefix), each entry scoped to its group across header, picker, topology, dashboards, and roster. The service picker surfaces each service\u0026rsquo;s group chip, and the navigation sidebar is now resizable (drag the divider, double-click to reset; width persists per browser) so long group-split names stay readable. Every layer OAP reports now appears in the sidebar, including layers with no Horizon template (they render with default capabilities). A layer is hidden only when an admin disables its template or it is listed in the new config-driven layers.excluded block in horizon.yaml (defaults FAAS and VIRTUAL_GATEWAY). Dashboard config is portable, and OAP is the source of truth Every template admin page — Overview templates, Layer dashboards, the 3D-map config, and the per-locale Translations — now has Export and Import. Export downloads the in-use version (what end users render) as JSON for backup, sharing, or moving a dashboard to another OAP; Import loads a JSON file as a local draft to preview, then publish with Check diff \u0026amp; push. Import never writes OAP directly. Runtime config is strictly what\u0026rsquo;s on OAP. Dashboards, overviews, and topology now render only the version published to OAP\u0026rsquo;s UI-template store (or the in-code minimal default) — the disk-bundled templates reach a running UI only by being synced to OAP or through the admin Preview button, never as a silent live fallback. An unreachable template store is a visible block (a banner, matching the OAP-query-unreachable strip), not a quiet bundled back-fill. The Preview button now drives every template-rendered page — overview detail, per-layer topology (incl. the instance map), API dependency, traces, and network profiling. Layer landing shows every service The layer landing now probes all services up to a configurable cap (query.landingServiceCap in horizon.yaml, default 100) and runs a cheap ranking pass to pick the true top-N by the landing\u0026rsquo;s order-by column — replacing the old cap of the first 25 by list order, which both hid services and mis-ranked them. The service picker lists the whole layer (below-cap services show low in the ranked column rather than disappearing), and the header chip reads \u0026ldquo;metrics: top N\u0026rdquo; to make the trim explicit. Selecting a low-traffic, below-cap service now works on every tab — logs, traces, and endpoint-dependency resolve the name from the full roster, not just the landing sample, so a tail service drills in everywhere. Widgets \u0026amp; formatting Layer-dashboard widgets gain a structured Visible when gate — by an MQE expression (has-value, or \u0026gt; / \u0026lt; a threshold) or by an entity attribute (e.g. language equals JAVA) — evaluated server-side, so a gated-out group\u0026rsquo;s queries are skipped entirely (a non-JVM instance no longer runs the JVM widget queries at all). New card formats: enum maps a coded metric to a readable, per-locale-translatable label (1 → OK), and duration renders a SECONDS metric as a human time-ago (5m 20s ago). Record widgets drill into the originating trace — resolved by trace id (so it works across layers) — with click-to-copy statement text. The instance-list badge is now configurable per layer (any attribute instead of the fixed agent language, hidden when empty or UNKNOWN). Large numbers on axes and tooltips use compact SI suffixes (45.1k, 1.34M) instead of scientific notation. Live debugger \u0026amp; DSL apply The live debugger groups MAL sample fan-outs by metric into one-line summaries and opens an expanded group straight into a diff view — shared labels dimmed into a \u0026ldquo;common\u0026rdquo; block, only the differing labels highlighted — so it\u0026rsquo;s immediate which label distinguishes each sample. Multiple output entities fold the same way; long fractional rate() / avg() values are trimmed for display (the exact value stays on hover). DSL management shows live apply progress. A structural rule change (scope, downsampling, or metric set) now tracks the apply across the cluster through a phase stepper (Compiled → Confirming → Committing → Done) and reports success only once OAP confirms durability. \u0026ldquo;Applied — cluster propagation unconfirmed\u0026rdquo; is surfaced as a warning (the rule is applied; lagging nodes self-converge), a failed apply is called out as rolled back with the edit kept for a retry, and a one-click Force re-apply recovers a stuck node. Access control, performance \u0026amp; fixes RBAC: the Roles \u0026amp; Permissions board now lists infra-3d:read; editing a layer-dashboard template gates on dashboard:write (publishing overview / alert / 3D-map configs stays on overview:write); the Cluster Status debug view needs only live-debug:read; and saving a local draft enforces the same per-kind permission as publishing. Performance: layer dashboards reuse the warm per-layer service catalog (one fewer OAP round-trip), the alarms list and count fire their two startup probes in parallel, the 3D Infra Map loads metrics in bounded-concurrency batches, and an oversized topology (\u0026gt;5,000 services / 15,000 calls) fails with a clear \u0026ldquo;too large to render\u0026rdquo; notice instead of an unreadable map. Partial metric-load failures now surface a banner so a backend hiccup isn\u0026rsquo;t misread as real \u0026ldquo;no traffic\u0026rdquo; data. Fixes: the API-dependency tab honors the topbar time picker; one failed metric group no longer blanks an entire dashboard; trace-list rows pick the correct root span on BanyanDB; the server-timezone offset is cached per OAP URL so repointing OAP re-probes immediately; baseline security headers (X-Content-Type-Options: nosniff, X-Frame-Options: DENY, Referrer-Policy: no-referrer) are sent on every response; and the profiling pages use more of the page height. Full release notes are here.\n","excerpt":"\u003cp\u003eSkyWalking Horizon UI 0.7.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cp\u003eThis release …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-horizon-ui-0-7-0/","title":"Release Apache SkyWalking Horizon UI 0.7.0"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/tags/logging/","title":"Logging"},{"body":"This is the seventh post in the Meet Horizon UI series. Part 6 was one request\u0026rsquo;s spans; this one is the log lines around it. Horizon surfaces logs through two distinct tabs, because there are really two different questions: \u0026ldquo;what did this service log over the last half hour?\u0026rdquo; and \u0026ldquo;what is this pod printing to stdout right now?\u0026rdquo;\nThe Logs tab queries the logs SkyWalking has already collected and stored — indexed, filterable, correlated with traces. The Pod Logs tab live-tails a Kubernetes pod\u0026rsquo;s container logs on demand — these aren\u0026rsquo;t stored logs at all: OAP reads them straight from the Kubernetes API server (the kubectl logs path), Horizon shows the window, and it\u0026rsquo;s discarded. Nothing is persisted, and SkyWalking\u0026rsquo;s log storage is never involved. Which tabs a layer shows is up to its template: the Logs tab appears on layers that enable it (General, Mesh, Nginx, the Envoy AI Gateway, the mobile and mini-program layers); the Pod Logs tab appears only on the Kubernetes-aware layers (Kubernetes Service, Mesh, Mesh data plane). Browser JavaScript errors are a different thing again — not service logs but client-side error events the browser agent reports, with their own categories and their own source-map de-obfuscation (turning a minified app.min.js:1:… frame back into your original file:line). That\u0026rsquo;s a separate tab on the Browser layer, and its own post later in the series.\nThe stored log stream Open a layer that has a Logs tab, pick a service in the header, and its stored log stream loads newest-first. Like the trace explorer, this tab owns its own time range — the global topbar picker is paused while you\u0026rsquo;re here, so auto-refresh can\u0026rsquo;t shift the window out from under an investigation. Pick a rolling preset (last 15 minutes through 24 hours, default 30) or a custom absolute window; queries run at second precision so the most recent lines are never rounded off the minute.\nA conditions bar narrows the stream, and every filter is optional and AND-joined:\nInstance — restrict to one service instance (labelled Sidecar on a sidecar layer). Endpoint — type to search the service\u0026rsquo;s endpoints, click to pin, × to clear. Trace ID — show only the lines correlated with one trace. This is also how a log lands when you arrive from a trace: the field pre-fills and the stream is already scoped. Tags — a single key=value field with autocomplete; start a key to see suggestions, type = to switch to known values, Enter to commit. Committed tags ride along as removable chips. Level — the Levels strip above the stream doubles as a filter: click error, warn, info, or debug to keep only that level, click again to clear. There\u0026rsquo;s no log query language here — no LogQL box to learn. The conditions above are the whole surface, and edits refresh the stream as you make them; Run query is just the explicit \u0026ldquo;I\u0026rsquo;m done editing, refresh now\u0026rdquo; button that resets to the first page.\nReading the stream The point of a log view isn\u0026rsquo;t to list lines, it\u0026rsquo;s to find the shape in them — so the stream comes with two pieces of orientation above it.\nA density histogram plots log count over time, each bar stacked by level in the legend\u0026rsquo;s colors; hover a bar for that bucket\u0026rsquo;s time range and per-level counts. It\u0026rsquo;s drawn from the page currently on screen, so it shows the shape of what you\u0026rsquo;re looking at. And the Levels strip carries a running count per level — sampled across the window, not just the visible page, so the error/warn/info mix reflects the whole window you\u0026rsquo;re querying.\nEach row then shows the timestamp, the level (the row is color-keyed to it), the service, an ↗ trace link when the line is trace-correlated, a JSON / YAML / TEXT format chip, and a one-line preview of the content. Horizon decides that chip by what the payload actually is: OAP labels a body JSON or plain text, and on top of that Horizon sniffs for JSON and YAML structure, so an unlabelled-but-structured line still gets the right treatment — JSON flattened to one line in the preview, YAML keeping its keys, plain text whitespace-collapsed.\nFigure 1: A service\u0026rsquo;s stored log stream — a level histogram and level counts over the window, then the rows, each tagged JSON / YAML / TEXT and linked to its trace.\nInto a single line Click a row and the full payload opens in a popout: the complete content with format-aware pretty-printing — JSON and YAML laid out properly, plain text given the whole canvas instead of a cramped strip — plus a Copy button, the service / instance / endpoint / trace context, and a table of every tag on the line. When the line is trace-correlated, an ↗ trace button opens the related trace\u0026rsquo;s waterfall in an overlay without leaving the stream — and it passes the row\u0026rsquo;s timestamp along, so the trace is found even when it has aged into a colder storage tier. Escape or a backdrop click closes it.\nFigure 2: One line in full — its payload pretty-printed by format, with its context and every tag laid out beside it.\nPod Logs: tailing what\u0026rsquo;s printing right now The Pod Logs tab answers the other question, and it\u0026rsquo;s a fundamentally different source: not SkyWalking\u0026rsquo;s stored logs, but the pod\u0026rsquo;s container output read live from the Kubernetes API server through OAP — the exact thing kubectl logs -f reads. There\u0026rsquo;s no stored history to page through; each refresh pulls the trailing window, shows it, and throws it away.\nStarting a tail is a few picks: choose a Pod (one service instance, pinned), a Container (Horizon lists the pod\u0026rsquo;s containers and selects the first), a look-back Window (last 30s, 1m, 5m, 15m, or 30m — how far back each poll reaches), and a poll Interval (2s, 5s, 10s, or 30s — how often it re-fetches). Press Start and the window streams into a read-only viewer that keeps the newest line in view and re-polls until you Pause; a header strip shows the container, the line count, a live indicator, and how long ago it last updated. Two Include / Exclude filter rows narrow what you see — each chip is a full-line regular expression (.*error.*) evaluated by OAP, so they match the whole line rather than a substring, and they stack.\nOne thing to know going in: on-demand pod logs are disabled by default on OAP, because container output can carry secrets. When the feature is off — or when the pod you picked has been rolled or scaled away — OAP returns a reason, and Horizon shows it in a banner rather than an empty pane, so you can tell \u0026ldquo;turn this on\u0026rdquo; apart from \u0026ldquo;that pod is gone.\u0026rdquo;\nFigure 3: A live tail of one pod\u0026rsquo;s container — windowed, interval-polled, regex-filterable, never persisted.\nWhere to go next Both tabs — the stored queries, the tag and container autocomplete, and the live tail — are gated by a single logs:read permission, so granting \u0026ldquo;can read logs\u0026rdquo; is one switch. For the field reference — every condition, the histogram, the Pod Logs windows and filters — see the Logs docs.\nNext up: Browser \u0026amp; RUM monitoring — the browser agent\u0026rsquo;s own error stream, and de-obfuscating a minified stack with source maps.\n","excerpt":"\u003cp\u003eThis is the seventh post in the \u003ca href=\"/blog/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e series. \u003ca href=\"/blog/2026-06-22-horizon-ui-trace-explorer/\"\u003ePart 6\u003c/a\u003e was one request\u0026rsquo;s spans; this …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-06-23-horizon-ui-log-explorer/","title":"Meet Horizon UI · 7/17: The Log Explorer"},{"body":"This is the eighth post in the Meet Horizon UI series. Part 7 was your services\u0026rsquo; logs; this one is your users\u0026rsquo; errors — the JavaScript exceptions the browser agent reports — and the one capability that turns them from noise into something you can act on.\nA production JavaScript stack is unreadable. Your code shipped minified and bundled, so the browser reports an error at app.min.js:1:98412 — a position into machine-generated soup that tells you nothing. The point of this feature is to walk that stack back to your source: the original file, line, column, symbol name, and a snippet of the code around it — frame by frame — by pointing the error at the right source map.\nThe browser-error feed On the BROWSER layer, the Browser Logs tab (the on-screen label — it\u0026rsquo;s specifically the JavaScript-error feed) lists what your browser agent reports. The BROWSER layer renames its slots to match its world — services become Applications, instances Versions, endpoints Pages — and the feed reads like the Log Explorer: a clickable category legend with counts and a density histogram over a stream of rows. Each row carries the time, the category, the page, the app version, and the error message — with the minified line:col shown as a chip when there is one.\nYou scope it with the same triage instincts as the trace and log tabs: it owns its own Time range (the global topbar is paused), and you narrow by Version, Page, or Category and hit Run query — there\u0026rsquo;s no background polling to shift the view under you, and no query language to learn, just structured controls. Click a row and it expands inline, right there in the stream.\nFigure 1: The browser agent\u0026rsquo;s error feed — categorized, charted, and scoped to one app\u0026rsquo;s version and pages.\nThat minified line:col is the whole problem in miniature. It\u0026rsquo;s a real position — but into your built bundle, not your source. Which is where the rest of this post comes in.\nFrom a minified stack back to your source Expand an error and the panel splits in two: on the left, the raw stack exactly as the browser reported it (the gibberish); on the right, where you resolve it. Pick a source map from the dropdown and click Resolve, and Horizon parses the stack and maps every frame through that map:\neach frame\u0026rsquo;s original file:line:column, the original symbol name (when the map carries it), and a few lines of the original source around the offending line, with the hit line highlighted (when the map embeds sourcesContent). A frame the map doesn\u0026rsquo;t cover is shown honestly as unmapped. So a stack whose top frame read app.min.js:1:45 resolves to computeCartTotal at checkout.ts:2:20, with the lines of checkout.ts around it — the cart.items.reduce(...) that actually threw — sitting right there, the whole stack top to bottom, not just the first frame.\nIt\u0026rsquo;s careful about the details that make this either trustworthy or quietly wrong: browser stacks count columns from 1 while source maps count from 0, so the resolver shifts before each lookup — and that path is tested against real bundler output, not a hand-made fixture.\nFigure 2: The hero — point a minified stack at the right map and read it back in your own source, frame by frame.\nWhich errors carry a stack to resolve Not every category has something to translate. JS, PROMISE, and VUE are real JavaScript errors whose stack points into your bundle — these resolve. AJAX and RESOURCE are network and load failures; their \u0026ldquo;stack\u0026rdquo; is an HTTP status or a failed URL, not code, so there\u0026rsquo;s simply nothing for a source map to map (Horizon doesn\u0026rsquo;t block them — there\u0026rsquo;s just no JavaScript there to walk back). Frames from code with no source map, or from eval/inline scripts, stay unmapped too. (JS is also the only category the browser reports a top-level line:col for; the others carry their position inside the stack string, which the resolver parses.)\nGetting maps in: upload, or mount A map has to be available before you can resolve against it, and there are two ways to provide one — deliberately different in durability:\nUpload a .map straight from the tab. It\u0026rsquo;s held in the server\u0026rsquo;s memory only — there\u0026rsquo;s no backend storage — and it\u0026rsquo;s temporary by design: it counts against a memory budget, is evicted least-recently-used under pressure, is lost when the server restarts, and (in a multi-instance deployment) lives only on the instance that received it. This is the fast path for ad-hoc triage: drag a map in, resolve, move on. Mount .map files into the server\u0026rsquo;s source-map directory (/app/sourcemaps in the container image, via HORIZON_SOURCEMAPS_DIR). These are validated as Source Map v3 at boot, read from disk on demand (so they never sit in the memory budget), survive restarts, reload on their own, and can\u0026rsquo;t be deleted from the UI. This is the durable, production path — bake your builds\u0026rsquo; maps into the image and they\u0026rsquo;re always there. The manager shows each map\u0026rsquo;s origin (an uploaded · temporary map vs a mounted · durable one) and the live memory usage against the budget; budgets (a per-file cap and a total resident-upload cap, 64 MiB and 512 MiB by default) live in a sourceMaps block in horizon.yaml.\nFigure 3: Two ways to provide a map — upload for a quick triage, mount for the durable, production set.\nYou pick the map — on purpose One thing Horizon deliberately does not do is guess. The browser agent reports an app version but no exact build fingerprint, so there\u0026rsquo;s no safe way to auto-match an error to a map — and applying a map from the wrong build gives you confidently wrong line numbers, which is worse than no answer. So the choice is yours: pick the map that matches the error\u0026rsquo;s build, and keep your maps labelled by version. (One caution worth stating plainly: a source map\u0026rsquo;s sourcesContent embeds your original source code, so treat the maps you upload or mount as sensitive, and provision them only on servers you trust.)\nThat manual-by-design choice also draws a clean permission line. Viewing the errors, listing the maps, and resolving a stack are all reads, gated by browser-errors:read; uploading or removing a map is a write, gated by source-map:write. So a read-only viewer can de-obfuscate stacks all day without ever being able to change what maps are loaded — reading is reading, mutating the map store is a write.\nWhere to go next For the field reference — the categories, the two provisioning paths, the budgets and the matching-maps-to-builds guidance — see the Browser Logs \u0026amp; Source Maps docs.\nNext up: Profiling — five profilers (trace, async, eBPF, Go pprof, network) rendered through one flame graph.\n","excerpt":"\u003cp\u003eThis is the eighth post in the \u003ca href=\"/blog/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e series. \u003ca href=\"/blog/2026-06-23-horizon-ui-log-explorer/\"\u003ePart 7\u003c/a\u003e was your services\u0026rsquo; logs; this …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-06-23-horizon-ui-browser-errors-and-source-maps/","title":"Meet Horizon UI · 8/17: Browser Errors \u0026 Source Maps"},{"body":"译自英文原文：Meet Horizon UI · 7/17: The Log Explorer。\n这是 Meet Horizon UI 系列的第七篇。第六篇讲的是一个请求的 spans；这一篇讲它周围的日志行。Horizon 用 两个标签页 展示日志，对应两类排查问题：“这个服务过去半小时打了什么日志？” 以及 “这个 pod 现在正在向 stdout 打什么？”\nLogs 标签页查询 SkyWalking 已经 采集并存储 的日志：已索引、可过滤、可与 Trace 关联。 Pod Logs 标签页按需 实时 tail Kubernetes pod 的容器日志。这类日志不走 SkyWalking 的日志存储：OAP 直接从 Kubernetes API server 读取它们（也就是 kubectl logs 那条路径），Horizon 只展示当前窗口，然后丢弃，不会持久化。 某个 Layer 展示哪些标签页由模板决定：启用日志的 Layer（General、Mesh、Nginx、Envoy AI Gateway、mobile 和 mini-program Layer）会显示 Logs 标签页；只有感知 Kubernetes 的 Layer（Kubernetes Service、Mesh、Mesh data plane）会显示 Pod Logs 标签页。Browser JavaScript 错误又是另一类数据：它不是服务日志，而是浏览器端 agent 上报的客户端错误事件，有自己的分类，也可以借助 source map 把压缩后的 app.min.js:1:... frame 还原到原始 file:line。这是 Browser Layer 上的独立标签页，会在这个系列的另一篇文章里讲。\n查询已存储日志 打开一个有 Logs 标签页的 Layer，先在顶部选择服务，日志流会按新到旧加载。和 Trace 探索器一样，这个标签页使用 独立的时间范围。当你在这里排查时，全局顶栏时间选择器会暂停，自动刷新不会把你正在看的窗口不断往前推。可以选择滚动预设（最近 15 分钟到 24 小时，默认 30 分钟），也可以选择自定义绝对窗口；查询按 秒级精度 执行，所以最新日志不会被分钟取整吞掉。\n条件栏用来收窄日志流，每个过滤条件都是可选的，多个条件按 AND 连接：\nInstance：限制到某个服务实例。在 sidecar Layer 上标签显示为 Sidecar。 Endpoint：输入搜索服务 endpoints，点击固定，按 × 清除。 Trace ID：只显示和某条 Trace 关联的日志行。从 Trace 跳转过来时，也会预填这个字段并直接限定日志流。 Tags：单个 key=value 字段，带 autocomplete；输入 key 可看建议，输入 = 后切换到已知 value，按 Enter 提交。提交后的 tags 以可删除标签形式保留。 Level：日志流上方的 Levels 条也可以当过滤器用。点击 error、warn、info 或 debug 只保留该级别，再点一次清除。 这里不提供单独的日志查询语言，也不需要学习 LogQL。上面的条件就是完整界面，并且 编辑时会立即刷新日志流；Run query 只是显式告诉系统“我改完了，现在刷新”，同时回到第一页。\n用直方图和级别计数定位日志 日志视图的目标不只是 列出 行，还要帮你看出分布和异常，所以日志流上方有两类辅助信息。\nDensity histogram 按时间展示日志量，每个柱按 legend 颜色 堆叠 level；hover 柱子可以看到该 bucket 的时间范围和每个 level 的计数。它基于当前页面上可见数据绘制，所以展示的是你正在看的日志分布。Levels 条则保留每个 level 在窗口内的累计计数。这个计数跨整个查询窗口采样，而不只是当前可见页，所以 error/warn/info 比例反映的是整个窗口。\n日志行会展示 timestamp、level（行颜色跟随 level）、service、存在 Trace 关联时的 ↗ trace 链接、JSON / YAML / TEXT 格式标记，以及内容的一行预览。Horizon 按日志内容本身决定这个标记：OAP 会标注日志 body 是 JSON 还是 plain text，在此基础上 Horizon 还会识别 JSON 和 YAML 结构，所以即使一行没有被标注但内容是结构化的，也会得到正确处理。JSON 在预览里压平成一行，YAML 保留 key，plain text 会折叠空白。\n图 1：某个服务的存储日志流：窗口上的 level histogram 和 level 计数，下面是每行日志，带 JSON / YAML / TEXT 标记并可跳到 Trace。\n查看单行日志详情 点击一行后，完整日志内容会在弹层里打开：内容会按格式排版，JSON 和 YAML 正常缩进，plain text 则展开成完整内容，而不是被挤在一条窄条里。面板还提供 Copy 按钮、service / instance / endpoint / trace 上下文，以及该行所有 tag 的表格。日志行与 Trace 关联时，↗ trace 按钮会在 overlay 中打开相关 Trace 瀑布图，不离开日志流；它还会把这行日志的 timestamp 传过去，所以 Trace 即使已经进入更冷的存储 tier，也仍然能找到。按 Escape 或点击背景遮罩即可关闭。\n图 2：完整查看一行日志：内容按格式排版，旁边展示上下文和所有 tags。\nPod Logs：实时查看容器输出 Pod Logs 标签页回答另一个问题，它的数据源也完全不同：不是 SkyWalking 存储的日志，而是通过 OAP 从 Kubernetes API server 实时读取 pod 的容器输出，也就是 kubectl logs -f 读取的同一类内容。这里没有可翻页的存储历史；每次刷新拉取尾部窗口，展示出来，然后丢弃。\n开始 tail 只需要几个选择：选一个 Pod（固定的服务实例）、一个 Container（Horizon 会列出 pod 的容器并默认选第一个）、一个回看 Window（last 30s、1m、5m、15m 或 30m，决定每次轮询向前取多远），以及轮询 Interval（2s、5s、10s 或 30s，决定多久重新取一次）。按 Start 后，日志会流入只读查看器，保持最新行可见，并持续轮询直到你按 Pause。顶部状态条显示 container、行数、live 状态，以及上次更新距今多久。两行 Include / Exclude 过滤器用来收窄可见内容；每个标签都是一个 整行正则表达式（.*error.*），由 OAP 执行，所以它匹配整行而不是子串，并且可以叠加。\n使用前需要知道一点：按需读取 pod logs 在 OAP 上 默认关闭，因为容器输出可能包含敏感信息。功能关闭时，或者你选择的 pod 已经滚动或缩容消失时，OAP 会返回一个 原因，Horizon 会把它显示成横幅，而不是给你一个空面板。这样你能区分“需要打开这个功能”和“那个 pod 已经不存在”。\n图 3：一个 pod 容器的实时 tail：窗口化、按间隔轮询、可用正则过滤、永不持久化。\n后续阅读 两个标签页，包括存储查询、tag/container 自动补全和 live tail，都由同一个 logs:read 权限控制。所以授予“可以读日志”就是一个开关。字段参考，包括每个条件、histogram、Pod Logs 的窗口和过滤器，可以看 Logs 文档。\n下一篇看 Browser/RUM：浏览器端错误如何上报，又如何用 source map 把压缩后的 stack 还原到源码位置。\n","excerpt":"\u003cp\u003e\u003cem\u003e译自英文原文：\u003ca href=\"/blog/2026-06-23-horizon-ui-log-explorer/\"\u003eMeet Horizon UI · 7/17: The Log Explorer\u003c/a\u003e。\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e这是 \u003ca href=\"/zh/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e 系列的第七篇。\u003ca href=\"/zh/2026-06-22-horizon-ui-trace-explorer/\"\u003e第六篇\u003c/a\u003e讲的是一个请求的 spans；这一篇讲它周围 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-06-23-horizon-ui-log-explorer/","title":"认识 Horizon UI · 7/17：日志探索器"},{"body":"译自英文原文：Meet Horizon UI · 8/17: Browser Errors \u0026amp; Source Maps。\n这是 Meet Horizon UI 系列的第八篇。第七篇讲的是服务日志；这一篇讲 用户 遇到的错误，也就是浏览器端 agent 上报的 JavaScript 异常，以及把这些错误定位到源码的关键一步。\n生产环境 JavaScript stack 基本不可读。代码经过压缩和打包后发布，浏览器只会报告错误出现在 app.min.js:1:98412，也就是一段机器生成代码里的位置，几乎不给你任何线索。Horizon 要做的是找到正确的 source map，把 stack 里的每一帧映射回源码：原始文件、行、列、符号名，以及出错位置附近的代码片段。\n浏览器端错误流 在 BROWSER Layer 上，Browser Logs 标签页（屏幕上的标签名，专指 JavaScript 错误流）会列出浏览器端 agent 上报的内容。BROWSER Layer 会把槽位重命名成自己的语义：services 变成 Applications，instances 变成 Versions，endpoints 变成 Pages。这个列表的阅读方式类似 Log Explorer：可点击的 category legend 带计数，density histogram 位于日志流之上。每一行都有时间、category、page、app version 和错误消息；如果带有压缩后的 line:col，也会显示成标记。\n排查方式和 trace/log 标签页一致：它使用独立的 Time range（全局顶栏暂停），你可以按 Version、Page 或 Category 收窄，然后点击 Run query。没有后台轮询把视图不断往前推，也没有要学习的查询语言，只有结构化控件。点击一行后，它会在日志流里原地展开。\n图 1：浏览器端 agent 上报的错误流：按 category 组织、带图表，并限定到某个 app 版本和页面。\n压缩后的 line:col 就是最典型的问题。它是真实位置，但位置在 构建后 的 bundle 里，不在你的源码里。后面的解析流程就是为了解决这个落差。\n从压缩后的 stack 定位到源码 展开一个错误后，面板分成两侧：左边是浏览器原样报告的 raw stack，也就是那段很难读的生产栈；右边是解析结果区域。选择一个 source map，点击 Resolve，Horizon 会解析 stack，并通过这份 map 映射 每一帧：\n每帧原始 file:line:column； 原始 symbol name（如果 map 携带了）； 出错行附近几行 原始源码，命中行会高亮（如果 map 内嵌 sourcesContent）。 map 覆盖不到的 frame 会明确标为 unmapped。所以一条顶层 frame 为 app.min.js:1:45 的 stack，可以还原成 checkout.ts:2:20 上的 computeCartTotal，并把 checkout.ts 附近几行显示出来。真正抛错的 cart.items.reduce(...) 就在面板里，不只是还原第一帧，而是从上到下还原整条 stack。\n这里有一些细节决定结果是否可信：浏览器 stack 的列号从 1 开始计数，source map 从 0 开始计数，所以解析器每次查找前都会做偏移；这条路径用真实 bundler 输出测试，而不是手写 fixture。\n图 2：核心流程：把压缩后的 stack 匹配到正确 map，然后逐帧还原到你自己的源码。\n哪些错误可以用 source map 解析 不是每类错误都有可还原内容。JS、PROMISE 和 VUE 是真实 JavaScript 错误，它们的 stack 指向 bundle，可以解析。AJAX 和 RESOURCE 是网络和加载失败；它们的“stack”是 HTTP status 或失败 URL，不是代码，所以 source map 没有东西可映射（Horizon 不会阻止它们，只是没有可映射的 JavaScript 位置）。没有 source map 的代码、eval 或 inline scripts 里的 frame，也会保持 unmapped。（JS 也是唯一由浏览器上报顶层 line:col 的 category；其他 category 的位置在 stack 字符串内部，由解析器提取。）\n提供 source map：上传或挂载 解析前必须让 map 可用。Horizon 提供两种方式，持久化策略有意不同：\nUpload 一个 .map，直接从标签页上传。它只保存在服务端 内存 里，没有后端存储，并且临时性是设计目标：它占用内存预算，在压力下按 least-recently-used 淘汰，服务重启后丢失；多实例部署时，它也只存在于接收上传的那一个实例。这个路径适合临时排查：拖一个 map 进来，解析，处理完离开。 Mount .map 文件到服务端 source-map 目录（容器镜像中是 /app/sourcemaps，可通过 HORIZON_SOURCEMAPS_DIR 指定）。这些文件在启动时按 Source Map v3 校验，按需从磁盘读取（所以不占内存预算），重启后仍然存在，会自动重新加载，并且 不能从 UI 删除。这是生产路径：把构建产物里的 map 文件放进镜像或挂载目录，它们就一直可用。 管理器会显示每个 map 的来源（uploaded · temporary 还是 mounted · durable），以及当前内存预算使用量。预算配置，包括单文件上限和常驻上传总量上限，默认 64 MiB 和 512 MiB，位于 horizon.yaml 的 sourceMaps 块。\n图 3：两种提供 map 的方式：upload 用于快速排查，mount 用于生产环境长期使用。\nmap 必须手动选择 Horizon 刻意 不猜测。浏览器端 agent 会上报 app version，但不会上报精确的构建指纹，所以没有安全方法自动把一个错误匹配到某份 map。用错构建的 map 会给出非常自信、但完全错误的行号，比没有答案更糟。所以这里由你选择：挑选和这次错误对应构建的 map，并按版本给 map 清晰命名。（还有一个必须直说的注意点：source map 的 sourcesContent 会包含你的原始源码，所以无论上传还是挂载，都要把 map 当成敏感内容，只放在可信服务器上。）\n手动选择 map 这件事，也划清了 权限 边界。查看错误、列出 maps、解析 stack 都是读操作，由 browser-errors:read 控制；上传或删除 map 是写操作，由 source-map:write 控制。所以只读用户可以反复解析 stack，但没有权限改变已加载的 map 集合。读就是读，修改 map 存储才是写。\n后续阅读 字段参考，包括 categories、两种提供路径、预算，以及如何按构建版本匹配 map，可以看 Browser Logs \u0026amp; Source Maps 文档。\n下一篇进入 Profiling：五种 profiler（trace、async、eBPF、Go pprof、network）如何共用一套火焰图视图，以及为什么 network profiling 是例外。\n","excerpt":"\u003cp\u003e\u003cem\u003e译自英文原文：\u003ca href=\"/blog/2026-06-23-horizon-ui-browser-errors-and-source-maps/\"\u003eMeet Horizon UI · 8/17: Browser Errors \u0026amp; Source Maps\u003c/a\u003e。\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e这是 \u003ca href=\"/zh/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e 系列的第八篇。\u003ca href=\"/zh/2026-06-23-horizon-ui-log-explorer/\"\u003e第七篇\u003c/a\u003e讲的是服务日 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-06-23-horizon-ui-browser-errors-and-source-maps/","title":"认识 Horizon UI · 8/17：浏览器错误与 Source Map"},{"body":"This is the fifth post in the Meet Horizon UI series. Part 3 drew the map between services and Part 4 drew it inside one service. This post zooms all the way out: a single WebGL view of your entire deployment at once — every SkyWalking layer\u0026rsquo;s services rendered as cubes, stacked in 3D, with live traffic, alarms, and the calls between them. It\u0026rsquo;s the \u0026ldquo;stand back and look at everything\u0026rdquo; companion to the per-layer dashboards.\nIt\u0026rsquo;s also genuinely interactive, so rather than describe it cold, here it is — the real map running on the demo\u0026rsquo;s sample data. Drag to rotate, scroll to zoom, click a cube:\nInteractive · sample data Open the 3D map One scene for the whole estate The 3D map is a standalone, full-screen view at /3d/map, opened from the 3D Infra pill in the topbar. It deliberately drops the rest of the console — no sidebar, no topbar, no global time picker — so the scene gets the whole viewport. The SkyWalking mark sits bottom-left; the × top-right returns you to Horizon. Everything in it is read live from the same OAP the rest of Horizon talks to: the service roster, each layer\u0026rsquo;s topology, the per-service traffic, and the active alarms.\nFigure 1: The whole deployment in one scene — services as cubes, layers as colored zones, roles as stacked tiers.\nTiers are the spine A tier is a horizontal plane that groups related layers by their role in the system. Tiers read top-to-bottom the way a request flows — from the apps a user touches down to the platform everything runs on. Horizon ships four bundled tiers:\nApps (top) — application surfaces and the dependencies as the app sees them: General agent services, Browser/RUM, mobile, and the Virtual* targets (database, cache, MQ, gateway, GenAI). Middleware — the data and messaging services, gateways, and self-observability: MySQL, PostgreSQL, Redis, Kafka, RocketMQ, APISIX, Nginx, the SkyWalking SO11Y components, and cloud-managed data services. Service Mesh — the mesh that fronts the apps: Istio control/data plane, Cilium, the Envoy AI Gateway. Infra (bottom) — the platform the rest runs on: Kubernetes, hosts, VMs. Every layer OAP reports lands on exactly one tier. A layer Horizon hasn\u0026rsquo;t classified yet — a brand-new OAP layer, say — falls to the failover tier (Middleware by default) with an \u0026ldquo;unclassified\u0026rdquo; mark, so it shows up and an operator can re-assign it rather than silently dropping off the map. The panel on the right mirrors the stack: click a tier row to fly the camera to it, use the eye toggle to show or hide a whole tier (or a single layer) at once, and read off how many of its services are currently visible.\nReading the map: cubes, zones, traffic Each cube is one service. Cubes are grouped into their layer\u0026rsquo;s zone on the tier — a translucent rectangle painted in the layer\u0026rsquo;s brand color and stamped with the project\u0026rsquo;s logo (Istio\u0026rsquo;s sail, the Kubernetes helm, a database cylinder, a queue) so you can pick a zone out from any camera angle. Layers that ship a topology (General, Service Mesh, Kubernetes Service, Cilium) lay their cubes out by call dependency — upstream callers on one side, downstream services on the other, the 3D analogue of the 2D service map. Layers without a topology pack their cubes into a tidy grid.\nA small traffic pill under a cube shows that service\u0026rsquo;s live headline throughput — requests per minute for app and mesh services, queries or operations per second for data services, each with its own unit. The pills appear on cubes close enough to read and fade away as you zoom out to keep the scene clean, then return as you come back in; a selected cube always shows its number.\nAlarms, and Beacon mode for incidents When a service has a currently-firing alarm (Horizon polls the last 20 minutes and counts only service-scoped, still-firing ones), its cube pulses red — a beacon you can spot from clear across the scene, while the alarm feed refreshes on its own.\nOn a busy map, one more red cube among hundreds can still be hard to find — so there\u0026rsquo;s Beacon mode. Toggle it from the toolbar and every healthy cube dims to a dark wireframe ghost, leaving only the alarming services lit and glowing. The shape of the deployment stays legible, but the services that are actually on fire are the only thing with color. It turns the bird\u0026rsquo;s-eye view into an incident triage surface in one click.\nFigure 2: Beacon mode dims everything healthy to a ghost, so the firing services are the only thing you see.\nThe lines between things The map draws the call graph, not just the nodes:\nIn-layer calls — light cyan tubes between two services in the same layer, with packets animating along them. This is each layer\u0026rsquo;s internal call graph, always on. Cross-layer calls — soft amber arrows between services in different layers on the same tier (a Browser app calling a Frontend, a Frontend calling a Virtual Database), pointing from caller to callee. Hierarchy links — and here\u0026rsquo;s the one that makes the 3D layout earn its keep. Select a cube and thicker gray tubes connect the different faces of the same logical service across tiers — the service as its agent sees it, as the mesh sees it, as a Kubernetes service. These represent identity, not traffic, so they stay hidden until you select a cube and then show just that cube\u0026rsquo;s relatives, climbing the stack from tier to tier. It\u0026rsquo;s the Smartscape idea from Part 3, drawn in the dimension it was always meant for. Figure 3: Select a cube and its identity links climb the tiers — one service, seen by its agent, the mesh, and Kubernetes.\nMoving around Drag to rotate, scroll to zoom, and arrow keys or WASD pan the view (hold Shift for a bigger step); a top-left toolbar offers the same gestures as buttons for trackpads. There\u0026rsquo;s one deliberate rule worth knowing: clicks inside the 3D scene never move the camera — they only select. Click a cube and it highlights, a detail card appears beside it (the service\u0026rsquo;s name, its tier and layer, and an Open dashboard button that jumps into that service\u0026rsquo;s layer dashboard in a new tab), and its hierarchy links light up. The camera-move surface is the side panel and the toolbar — click a layer row to glide the camera to its zone. Keeping those two jobs separate is what makes selecting a small cube feel reliable instead of having the cube slide out from under your cursor.\nHow it builds A whole deployment is too much to fetch in one request, so the map loads in stages, and a slim timeline strip along the bottom shows the progress live: Services (the roster and their layers) → Templates (which layers carry a topology) → Topologies (each topology-bearing layer\u0026rsquo;s call graph) → Hierarchy (the cross-tier identity links) → Layout (placing the cubes) → Metrics (the per-service traffic, fetched in batches so the cubes light up progressively). Click any step for a drawer with its detail, or hit refresh to re-run the whole sequence.\nTwo touches make refreshes cheap. The hierarchy step is incremental — only services that are new since the last run get probed, the rest reused from cache, so a steady deployment costs nothing there. And the scene is re-keyed on a per-layer structure hash, so an unchanged refresh keeps your camera exactly where it was; only a real roster or edge change rebuilds the layout. Under the hood it\u0026rsquo;s Three.js with a thin Vue wrapper, every geometry and material shared across cubes of the same kind — the kind of detail that keeps a few hundred services rendering smoothly in a browser tab.\nConfigured, not coded None of the above is a hard-coded \u0026ldquo;3D screen.\u0026rdquo; What the map shows is driven by a single configuration an administrator edits in a structured form editor at /admin/3d-map — tiers, layers, colors, and metrics through form controls, not raw JSON. From it you can:\nFilter layers with one global regex — anything it excludes drops off the map entirely. Arrange tiers — rename them, reorder them top-to-bottom, and pin each layer to a tier (with a nominated failover tier so nothing falls off silently). Group layers — cluster several related layers (the SkyWalking self-observability components, say) into one labelled block, each member keeping its own color. Color each layer and choose its traffic metric — the MQE expression, a label, and a unit, seeded by default from that layer\u0026rsquo;s dashboard template so most layers show a sensible number out of the box. Horizon ships a bundled default, so the map is useful immediately; your edits live as a local draft until you Check diff \u0026amp; push them to OAP — the same draft → preview → publish model behind every dashboard, and the same Export/Import for backup or moving a configuration between deployments. The map itself is a read-only observe surface that runs against your current OAP; publishing the config that shapes it is part of the config-driven customization story a later post in this series covers end to end.\nFigure 4: The map is configuration, not code — tiers, colors, and per-layer traffic metrics edited as a form, then published to OAP.\nWhere to go next The 3D map is the bird\u0026rsquo;s-eye summary; the 2D per-layer pages stay the authoritative service maps. Viewing it needs only read access (infra-3d:read, held by the built-in viewer role and up); shaping it needs the same write permission as the dashboards. For the field reference — tiers, the config shape, the loading stages — see the 3D Infrastructure Map docs.\nNext up: the Trace Explorer — from the bird\u0026rsquo;s-eye view of the whole deployment back down to a single request, drawn three different ways.\n","excerpt":"\u003cp\u003eThis is the fifth post in the \u003ca href=\"/blog/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e series. \u003ca href=\"/blog/2026-06-21-horizon-ui-topology-and-dependency/\"\u003ePart 3\u003c/a\u003e drew the map \u003cem\u003ebetween\u003c/em\u003e services and \u003ca href=\"/blog/2026-06-21-horizon-ui-deployment-and-banyandb/\"\u003ePart …\u003c/a\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-06-22-horizon-ui-3d-infrastructure-map/","title":"Meet Horizon UI · 5/17: The 3D Infrastructure Map"},{"body":"This is the sixth post in the Meet Horizon UI series. The last few were maps — topology between services, the deployment inside one, the 3D view of the whole estate. Those answer \u0026ldquo;what does my system look like.\u0026rdquo; This post is about the opposite move: from an aggregate down to one request — its spans, its timing, the exact hop that went slow. That\u0026rsquo;s the Traces tab.\nBuilt for triage, not tailing The Traces tab is a distributed-trace explorer that lives inside a layer: pick a service, set your conditions, and read a single trace\u0026rsquo;s span timeline. And it behaves differently from the rest of the console on purpose — because traces are triage data, not a live feed.\nIt owns its own time range and conditions. It does not follow the global topbar time picker, and it does not auto-refresh. You stage what you\u0026rsquo;re looking for and press Run query; nothing is fetched until you do. Before the first run the list simply says \u0026ldquo;Pick your conditions, then click Run query.\u0026rdquo; When you\u0026rsquo;re chasing one bad trace from twenty minutes ago, the last thing you want is the page sliding forward under you every few seconds — so it doesn\u0026rsquo;t.\nConditions, not a query language The whole filter surface is structured form controls — selects, number ranges, and tag chips — staged in a toolbar that only takes effect on Run query:\nInstance and Endpoint narrow within the service (the endpoint dropdown lists the service\u0026rsquo;s own endpoints). Status — ALL / SUCCESS / ERROR. Order — Newest (by start time) or Slowest (by duration). Limit — how many rows to pull (30 by default); the BFF caps the page size server-side so a client can\u0026rsquo;t ask OAP for the world. Time range — a rolling preset (last 15 minutes through 24 hours) or a custom absolute window, evaluated at second precision so a trace that just finished still falls inside it instead of being rounded off the minute. Trace ID — paste one to look it up directly. Duration range — a min–max in milliseconds. Tag — free-form span tags as key=value (e.g. http.status_code=500), added with Enter as removable chips and AND-joined; keys and values get typeahead from the backend. One thing this is not: a query language. There\u0026rsquo;s no TraceQL box anywhere in Horizon — the structured conditions above are the entire surface. (TraceQL is a separate path: SkyWalking\u0026rsquo;s backend can serve traces to Grafana over TraceQL, which is its own story. Horizon\u0026rsquo;s explorer is forms, not a DSL.)\nA distribution you can box-select When the results come back, the toolbar is joined by a Distribution chart: one dot per trace, plotted with start time on the X axis and duration as its height — slower traces sit higher. Dots are colored by status — errors in red, successes in the accent color — so a cluster of red high up is exactly the \u0026ldquo;slow and failing\u0026rdquo; corner you came to find.\nThe chart is also a filter. Click a dot to pick it, or drag a box across a band of them, and the result list narrows to just that selection — the header switches to an \u0026ldquo;N picked\u0026rdquo; count with a Reset. This is a client-side filter over what\u0026rsquo;s already loaded; it never issues a new query. Dragging a band across the slow-and-erroring corner and reading only those rows is the fastest way to go from \u0026ldquo;200 traces\u0026rdquo; to \u0026ldquo;these six.\u0026rdquo;\nFigure 1: Stage conditions, run the query, then box-select a band of the distribution to whittle the list down to the traces worth opening.\nEach result row shows the trace\u0026rsquo;s root endpoint, an OK/ERR flag, the duration, and a bar sized against the slowest trace in the set. What a row represents depends on the storage backend, and Horizon detects it for you: a banner reads either \u0026ldquo;Full traces are returned inline\u0026rdquo; (the backend returns whole traces with their spans, so a click opens immediately) or \u0026ldquo;Each row is a trace segment — click one to fetch its full trace.\u0026rdquo; You never configure this; the banner just tells you which you\u0026rsquo;re looking at.\nThree ways to read one trace Click a row and the trace opens with a three-way view toggle — Default, Tree, and Statistics — over the same spans:\nDefault is the span waterfall: one indented row per span, each carrying a service-colored bar positioned and sized by the span\u0026rsquo;s start offset and duration on a shared timeline, a span-kind glyph, the component\u0026rsquo;s icon (the same icon set the topology map uses), the endpoint or peer name, and the span\u0026rsquo;s own duration. Errored spans are highlighted, and a flag marks any span carrying attached events. Crucially, the waterfall stitches spans across segments using their parent references, so a request that crossed five services renders as one connected timeline rather than five disjoint ones. Tree draws those same spans as a zoomable, pannable node graph — root on the left, callees flowing right — for when you care about the shape of the call tree more than the exact timing. Statistics rolls the spans up by name: a sortable table of count and total / average / maximum duration per operation, so \u0026ldquo;which span am I spending all my time in, across every occurrence in this trace\u0026rdquo; is one sort away. Figure 2: The waterfall (Default) — one connected timeline across every service the request touched.\nFigure 3: The Tree view — the same spans as the call tree\u0026rsquo;s shape, zoom and pan to explore.\nInside a span Click any span and a detail panel opens beside it. Meta lays out the essentials — service, instance, endpoint, kind (entry / exit / local / producer / consumer), component, peer, layer, start time, duration, and the error flag. Below it, when they apply:\nCross-trace refs — when a span\u0026rsquo;s parent lives in a different trace (an async hop, a message consumed later), the reference is listed with the parent\u0026rsquo;s trace id, segment, and span — and the trace id is a link that swaps you straight into that other trace. Tags, Logs (timestamped per-span entries), and Attached Events (named events with start/end times and summary key/values). The detail header reports the trace\u0026rsquo;s start, total duration, span count, and how many distinct services it touched; from there you can copy the trace id or a shareable link. Opening a shared ?traceId=… URL lands you directly on that trace in an overlay — which is what makes a trace something you can paste into an incident channel and have a teammate land on the exact same view.\nNative and Zipkin, side by side Not every layer\u0026rsquo;s traces come from SkyWalking\u0026rsquo;s own agents. A layer template carries a traces.source setting — native, zipkin, or both — and Horizon routes accordingly. Agent-instrumented layers (like General Service) use the native explorer above; service-mesh and Kubernetes-flavored layers, where spans arrive as Zipkin/OpenTelemetry data, use the Zipkin explorer; and a layer set to both simply gets two sidebar tabs, since native and Zipkin spans have genuinely different shapes and conditions.\nThe Zipkin tab queries an upstream Zipkin store through OAP\u0026rsquo;s Zipkin query API (the same compatibility surface OAP exposes for any Zipkin client — not a GraphQL or TraceQL path). Because Zipkin organizes data by its own service universe (the serviceName on each span, which can drift from SkyWalking\u0026rsquo;s service list), the tab carries its own service controls instead of binding to the page\u0026rsquo;s service picker — with Zipkin-native conditions like Remote service, Span name, and an Annotations query (error or key=value). The two stores fail independently: if Zipkin is unreachable, the native traces are unaffected, and vice versa.\nFigure 4: A layer set to Zipkin gets its own tab and its own service universe, querying the upstream Zipkin store through OAP.\nOpen a Zipkin span and its detail keeps that Zipkin shape — the CLIENT / SERVER kind, the local and remote endpoints, and the raw Zipkin/OpenTelemetry tags (the istio.* set, http.status_code, the sidecar node_id) exactly as Zipkin recorded them, with no translation into SkyWalking\u0026rsquo;s span model.\nFigure 5: A Zipkin span keeps its native fields — the kind, the endpoints, and the raw istio.* / HTTP tags.\nFrom a slow row to the trace behind it There\u0026rsquo;s a second way into a trace that doesn\u0026rsquo;t go through the explorer at all. A trace overlay is mounted once for the whole app, and several things open it by trace id: a ?traceId= link, a cross-trace ref, a log line, and — as Part 2 showed — the jump-to-trace icon on a record widget\u0026rsquo;s slow-statement row. Because it resolves by id rather than by layer, it works even when the trace belongs to a different layer than the one you\u0026rsquo;re looking at: a Virtual Database, Cache, or MQ service has no traces tab of its own, yet its slow statement still lands you on the originating trace. And when the jump carries a timestamp (a log row knows when its line was written), the lookup widens around that moment — so a trace that has already aged into cold storage still resolves instead of quietly missing.\nWhere to go next The Traces tab is one request in full detail; the dashboards and maps from the earlier posts are where you notice something\u0026rsquo;s wrong in the first place. For the field reference — every condition, the native-vs-Zipkin split, the span detail panel — see the Traces docs.\nNext up: the Log Explorer — the same triage instincts, applied to log streams instead of spans.\n","excerpt":"\u003cp\u003eThis is the sixth post in the \u003ca href=\"/blog/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e series. The last few were maps — \u003ca href=\"/blog/2026-06-21-horizon-ui-topology-and-dependency/\"\u003etopology\u003c/a\u003e between …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-06-22-horizon-ui-trace-explorer/","title":"Meet Horizon UI · 6/17: The Trace Explorer"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/tags/tracing/","title":"Tracing"},{"body":"译自英文原文：Meet Horizon UI · 5/17: The 3D Infrastructure Map。\n这是 Meet Horizon UI 系列的第五篇。第三篇画的是服务 之间 的地图，第四篇画的是单个服务 内部 的地图。这一篇把视角拉到最远：用一个 WebGL 视图一次看完整个部署。每个 SkyWalking Layer 的服务都会渲染成立方体，堆叠到 3D 空间里，并显示实时流量、告警和它们之间的调用关系。它补上了按 Layer 仪表盘之外的全局视角：退后一步，看整个系统。\n而且它不是静态截图。所以这里直接放出来：下面就是运行在 demo 数据上的真实地图。拖动旋转，滚轮缩放，点击立方体：\n交互演示 · 示例数据 Open the 3D map 一张 3D 图查看完整部署 3D 地图是 /3d/map 里的独立全屏页面，从顶栏里的 3D Infra 入口打开。它刻意拿掉控制台其余部分：没有侧边栏，没有顶栏，没有全局时间选择器。整个浏览器视口都交给场景。SkyWalking 标识放在左下角，右上角的 × 返回 Horizon。里面所有数据都来自 Horizon 其他页面访问的同一个 OAP：服务清单、每个 Layer 的拓扑、每个服务的流量，以及活跃告警。\n图 1：一张 3D 图看完整部署：服务是立方体，Layer 是带颜色的 zone，角色按 tier 堆叠。\n用 tier 组织系统层次 Tier 是一层横向平面，用来把系统中职责相近的 Layer 放在一起。Tier 从上到下的阅读顺序，就是请求流动的大致方向：从用户直接访问的应用，一路到底层平台。Horizon 内置四个 tier：\nApps（顶部）：应用界面和应用视角看到的依赖，包括 General agent services、Browser/RUM、mobile，以及 Virtual* targets（database、cache、MQ、gateway、GenAI）。 Middleware：数据和消息服务、网关，以及自观测组件，包括 MySQL、PostgreSQL、Redis、Kafka、RocketMQ、APISIX、Nginx、SkyWalking SO11Y components 和云托管数据服务。 Service Mesh：承载应用流量的 mesh，包括 Istio control/data plane、Cilium、Envoy AI Gateway。 Infra（底部）：其余内容运行在其上的平台，包括 Kubernetes、hosts、VMs。 OAP 上报的每个 Layer 都会归入一个 tier。Horizon 还没分类的新 Layer，比如 OAP 新增的 Layer，会归入 failover tier（默认 Middleware）并带 \u0026ldquo;unclassified\u0026rdquo; 标记。这样它会出现，运维人员也可以重新分配，而不是静默从地图上消失。右侧面板对应整个堆栈：点击 tier 行可以把镜头移到对应位置，用眼睛开关一次显示或隐藏整个 tier（或单个 Layer），并查看当前可见服务数量。\n地图元素：立方体、zone 和流量 每个服务对应一个 立方体。立方体会在 tier 上按所属 Layer 聚成一个 zone：半透明矩形使用该 Layer 的品牌色，并盖上项目 logo（Istio 的帆、Kubernetes 舵轮、数据库圆柱、队列图标），所以从任何视角都能辨认出 zone。带拓扑的 Layer（General、Service Mesh、Kubernetes Service、Cilium）会按 调用依赖 摆放立方体：上游 caller 在一侧，下游服务在另一侧，对应 2D 服务地图的 3D 版本。没有拓扑的 Layer 则把立方体排成整齐网格。\n立方体下方的小 traffic 标签显示该服务的实时主吞吐：应用和 mesh 服务是 requests per minute，数据服务是 queries 或 operations per second，并保留各自单位。只有镜头足够近、文字可辨认时标签才出现；缩远后会淡出，保持场景干净；选中的立方体总会显示它的数字。\n用 Beacon mode 突出告警服务 当一个服务有 当前正在触发的告警 时（Horizon 轮询最近 20 分钟，并只统计 service 作用域下仍在触发的告警），它的立方体会 红色脉冲。这就是一个信标，即使隔着整个场景也能看到，而告警列表会独立刷新。\n在繁忙地图里，几百个立方体中的一个红点仍然可能难找，所以有 Beacon mode。从工具栏打开后，所有 健康 立方体都会变成深色 wireframe，只留下正在告警的服务发光。部署结构仍然清楚，但真正出问题的服务才有颜色。这个模式可以把鸟瞰图快速切成 incident triage 视图。\n图 2：Beacon mode 会把健康对象变暗成幽影，所以你只会看到正在触发的服务。\n线表示调用和层级关系 地图不只画节点，也画调用图：\nIn-layer calls：同一个 Layer 内两个服务之间的浅青色管线，并带沿线运动的数据包动画。这是每个 Layer 自己的内部调用图，始终开启。 Cross-layer calls：同一个 tier 上不同 Layer 服务之间的柔和琥珀色箭头，比如 Browser app 调用 Frontend、Frontend 调用 Virtual Database，方向从 caller 指向 callee。 Hierarchy links：这是让 3D 布局真正发挥价值的一类线。选中一个立方体后，粗灰色管线会连接 同一个逻辑服务在不同 tier 上的不同形态：agent 看到的服务、mesh 看到的服务、Kubernetes service 看到的服务。它们表示 身份，不是流量，所以默认隐藏；选中立方体后只显示与它相关的对象，并沿着 tier 一层层爬上去。这就是第三篇里的 Smartscape，只是放到了更适合表达层级关系的 3D 视角里。 图 3：选中立方体后，身份链接沿 tier 爬升：agent、mesh 和 Kubernetes 各自看到的同一个服务。\n视角移动与选择 拖动旋转，滚轮缩放，方向键 或 WASD 平移视角（按住 Shift 步长更大）；左上角工具栏也为触控板提供同样动作的按钮。有一条刻意设计的规则值得知道：3D 场景里的点击永远不会移动视角，只负责选择。点击一个立方体，它会高亮，旁边出现详情卡片（服务名称、tier、Layer，以及会在新标签页打开该服务 Layer 仪表盘的 Open dashboard 按钮），它的 hierarchy links 也会亮起。移动视角的交互入口是 侧边面板 和 工具栏；点击 Layer 行，镜头会滑到它的 zone。把“选择”和“移动视角”分开，点击小立方体才会可靠，不会刚点中它就让它从光标下滑走。\n地图如何分阶段加载 整个部署的数据太大，不适合一次请求拉完，所以地图分阶段加载，底部细长的 timeline strip 会实时展示进度：Services（服务清单和所属 Layer）→ Templates（哪些 Layer 带拓扑）→ Topologies（每个有拓扑 Layer 的调用图）→ Hierarchy（跨 tier 身份链接）→ Layout（摆放立方体）→ Metrics（按批次获取每个服务的流量，让立方体逐步亮起来）。点击任意阶段可以打开抽屉查看详情，也可以点击 Refresh 重新跑完整流程。\n两个设计降低了刷新成本。Hierarchy 阶段是增量的：只有上次之后新增的服务需要探测，其余从缓存复用，所以稳定部署在这一步没有额外代价。场景还会按每个 Layer 的结构 hash 重新生成 key；结构没变时刷新会保留你的视角位置，只有服务清单或边真的变化时才重建布局。底层使用 Three.js 加一层很薄的 Vue 封装，同类立方体共享 geometry 和 material。正是这些细节，让几百个服务也能在浏览器标签页里平滑渲染。\n地图结构来自配置 上面这些不是一张写死的 \u0026ldquo;3D 页面\u0026rdquo;。地图展示什么，由管理员在 /admin/3d-map 的 结构化表单编辑器 里编辑：tier、Layer、颜色和指标都通过表单控制，而不是直接写 JSON。你可以在里面：\n用一个全局正则过滤 Layer：匹配排除的内容会完全从地图上消失。 安排 tier：重命名、从上到下重排，并把每个 Layer 固定到某个 tier 上，同时指定 failover tier，避免内容静默丢失。 对 Layer 分组：把多个相关 Layer，比如 SkyWalking 自观测组件，聚成一个带标签的 block，每个成员仍然保留自己的颜色。 为每个 Layer 配色并选择流量指标：配置 MQE 表达式、label 和单位；默认值会从该 Layer 的仪表盘模板里初始化，所以大多数 Layer 一开始就能显示合理数字。 Horizon 自带一份默认配置，所以地图开箱就有用。你的修改会以本地 draft 保存，直到点击 Check diff \u0026amp; push 发布到 OAP。它使用和仪表盘相同的 draft → preview → publish 模型，也支持同样的 Export/Import，用于备份或在部署之间迁移配置。地图本身是只读 observe 界面，可以直接运行在当前 OAP 上；发布用于控制地图形态的配置，则属于后续文章会完整介绍的配置化定制。\n图 4：地图是配置，不是代码。tier、颜色和每个 Layer 的流量指标都以表单方式编辑，然后发布到 OAP。\n后续阅读 3D 地图是全局入口；2D 的按 Layer 页面仍然是最完整的服务地图。查看它只需要读权限（infra-3d:read，内置 viewer 角色及以上持有）；调整它需要和仪表盘相同的写权限。字段参考，包括 tier、配置结构和加载阶段，可以看 3D Infrastructure Map 文档。\n下一篇回到单个请求：Trace Explorer 会用分布图、瀑布图和调用树帮你定位慢调用。\n","excerpt":"\u003cp\u003e\u003cem\u003e译自英文原文：\u003ca href=\"/blog/2026-06-22-horizon-ui-3d-infrastructure-map/\"\u003eMeet Horizon UI · 5/17: The 3D Infrastructure Map\u003c/a\u003e。\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e这是 \u003ca href=\"/zh/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e 系列的第五篇。\u003ca href=\"/zh/2026-06-21-horizon-ui-topology-and-dependency/\"\u003e第三篇\u003c/a\u003e画的是服务 \u003cem\u003e之间\u003c/em\u003e 的地图， …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-06-22-horizon-ui-3d-infrastructure-map/","title":"认识 Horizon UI · 5/17：3D 基础设施地图"},{"body":"译自英文原文：Meet Horizon UI · 6/17: The Trace Explorer。\n这是 Meet Horizon UI 系列的第六篇。前几篇都在讲地图：服务之间的拓扑、单个服务内部的 Deployment 标签页，以及整个系统全貌的 3D 视图。它们回答“我的系统长什么样”。这一篇把视角缩到 一个请求，看它的 spans、时序，以及到底是哪一跳变慢了。这就是 Traces 标签页。\n面向排查，不做实时滚动刷新 Traces 标签页是一个位于 Layer 内部 的分布式 Trace 探索器：选择服务，设置条件，然后阅读单条 Trace 的 span 时间线。它有意和控制台其他部分不一样，因为 Trace 是排查数据，不是实时流。\n它使用 独立的时间范围和条件。它不跟随全局顶栏时间选择器，也不会自动刷新。你先配置要找的条件，再按 Run query；点击之前不会取任何数据。第一次运行前，列表只显示 \u0026ldquo;Pick your conditions, then click Run query.\u0026quot;。当你在追二十分钟前的一条坏 Trace 时，最不需要的就是窗口每几秒自动往前滚，所以它不会这么做。\n用表单配置条件，不写查询语言 过滤条件全部通过结构化表单配置：select、数字范围、tag。条件会暂存在工具栏里，只在点击 Run query 后生效：\nInstance 和 Endpoint：在服务内部收窄范围，endpoint 下拉框只列出这个服务自己的 endpoints。 Status：ALL / SUCCESS / ERROR。 Order：Newest（按开始时间）或 Slowest（按耗时）。 Limit：拉取多少行，默认 30；BFF 会在服务端限制 page size，避免客户端向 OAP 请求过量数据。 Time range：滚动预设（最近 15 分钟到 24 小时）或自定义绝对窗口，按 秒级精度 计算，所以刚结束的 Trace 不会因为分钟取整而掉出窗口。 Trace ID：粘贴一个 id 直接查。 Duration range：毫秒级 min-max。 Tag：自由输入 span tags，格式为 key=value（比如 http.status_code=500），按 Enter 添加为可删除标签，多个 tag 按 AND 连接；key 和 value 都由后端提供自动补全。 它不是查询语言。Horizon 里没有 TraceQL 输入框。上面的结构化条件就是完整界面。（TraceQL 是另一条路径：SkyWalking 后端可以通过 TraceQL 向 Grafana 提供 Trace，这件事在另一篇文章里讲。Horizon 的探索器是表单，不是 DSL。）\n可框选的时延分布图 结果返回后，工具栏下会出现一张 Distribution 图：每个点代表一条 Trace，X 轴是开始时间，高度是耗时，越慢的 Trace 越高。点按 status 着色：错误为红色，成功为强调色。所以左上角、右上角那类高处红点，就是你要找的“又慢又失败”的区域。\n这张图本身也是过滤器。点击一个点可以选中它，或者 拖一个矩形框选一片点，结果列表会收窄到这批选择；标题区切换成 \u0026ldquo;N picked\u0026rdquo; 计数，并提供 Reset。这是基于已加载结果的客户端过滤，不会发起新查询。直接框住慢且失败的角落，只看那几行，是从“200 条 Trace”缩到“这 6 条值得打开”的最快路径。\n图 1：先配置条件、执行查询，再在分布图上框选一片区域，把列表缩到真正值得打开的 Trace。\n每条结果行会显示 Trace 的 root endpoint、OK/ERR 标记、duration，以及按当前结果中最慢 Trace 归一化后的长度条。每一行 代表什么 取决于存储后端，Horizon 会自动检测：横幅会显示 \u0026ldquo;Full traces are returned inline\u0026rdquo;，表示后端返回的是带 spans 的完整 Trace，点击即可打开；或者显示 \u0026ldquo;Each row is a trace segment — click one to fetch its full trace.\u0026quot;。你不需要配置这个，横幅只是告诉你当前看到的是什么。\n三种方式阅读同一条 Trace 点击一行后，Trace 会打开，并在同一组 spans 上提供三种视图切换：Default、Tree 和 Statistics。\nDefault 是 span 瀑布图：每个 span 一行缩进展示，带一个按服务着色的条形，条形在共享时间线上按 span 起始偏移和耗时定位；同时显示 span-kind 标记、组件图标（和拓扑地图共用同一套图标）、endpoint 或 peer 名称，以及 span 自身耗时。错误 span 会高亮，带 attached events 的 span 会有标记。关键点在于，瀑布图会用 parent references 把跨 segment 的 spans 串起来，所以一个跨过五个服务的请求会渲染成一条连贯时间线，而不是五段互不相干的内容。 Tree 把同一批 spans 画成可缩放、可平移的节点图：root 在左，callees 向右流动。适合你更关心调用树形状，而不是精确时序的时候。 Statistics 按 name 聚合 spans：一张可排序表，展示每个 operation 的 count、total / average / maximum duration。所以“这条 Trace 里到底哪个 span 名称累计耗时最多”只需要点一次排序。 图 2：瀑布图（Default），一条请求触达的所有服务会串成同一条连贯时间线。\n图 3：Tree 视图，把同一批 spans 画成调用树形状，可以缩放和平移。\n查看 span 详情 点击任意 span，旁边会打开详情面板。Meta 展示核心信息：service、instance、endpoint、kind（entry / exit / local / producer / consumer）、component、peer、layer、start time、duration 和 error 标记。适用时，下面还会出现：\nCross-trace refs：当一个 span 的 parent 位于 另一条 Trace（异步跳转、稍后消费的消息）中时，这里会列出 parent 的 trace id、segment 和 span；trace id 是一个 链接，点击可以直接切换到那条 Trace。 Tags、Logs（每个 span 的带时间戳日志项）和 Attached Events（带 start/end time 和 summary key/values 的命名事件）。 详情标题区会显示 Trace 开始时间、总耗时、span 数量，以及触达了多少个不同服务；从这里可以复制 trace id 或可分享链接。打开一个带 ?traceId=... 的分享 URL，会直接打开这条 Trace 的 overlay。这样你可以把链接粘到 incident 频道里，让同事打开同一条 Trace、同一个视图。\nNative 与 Zipkin 并列支持 并不是每个 Layer 的 Trace 都来自 SkyWalking 自己的 agent。Layer 模板带一个 traces.source 设置，可以是 native、zipkin 或 both，Horizon 据此路由。由 agent 接入的 Layer（比如 General Service）使用上面讲的 native 探索器；service-mesh 和 Kubernetes 风格的 Layer 中，spans 以 Zipkin/OpenTelemetry 数据进入，则使用 Zipkin 探索器；设置为 both 的 Layer 会得到 两个标签页，因为 native 和 Zipkin spans 的形状和条件确实不同。\nZipkin 标签页通过 OAP 的 Zipkin query API 查询上游 Zipkin store。这是 OAP 向任何 Zipkin client 暴露的兼容接口，不是 GraphQL，也不是 TraceQL。Zipkin 按自己的服务集合组织数据（每个 span 上的 serviceName，可能和 SkyWalking 的服务列表不同），所以这个标签页有自己的服务选择器，不绑定页面上的 service picker；它也带 Zipkin 原生条件，比如 Remote service、Span name 和 Annotations 查询（error 或 key=value）。两个存储路径独立失败：Zipkin 不可达时，native traces 不受影响，反之亦然。\n图 4：设置为 Zipkin 的 Layer 会得到自己的标签页和自己的服务集合，并通过 OAP 查询上游 Zipkin store。\n打开一个 Zipkin span 后，详情会保留 Zipkin 的形状：CLIENT / SERVER kind、本地和远端 endpoint，以及原始 Zipkin/OpenTelemetry tags（istio.*、http.status_code、sidecar node_id）都会按 Zipkin 记录的方式展示，不翻译成 SkyWalking span 模型。\n图 5：Zipkin span 保留自己的字段：kind、endpoints，以及原始 istio.* / HTTP tags。\n从慢记录定位到对应 Trace 还有一种进入 Trace 的方式，不需要经过探索器。Horizon 在应用级只挂载一个 Trace overlay，多个入口都能按 trace id 打开它：?traceId= 链接、cross-trace ref、日志行，以及第二篇里提到的 record 组件慢语句行上的 jump-to-trace 图标。因为它按 id 解析，而不是按 Layer 解析，所以即使 Trace 属于你当前看到的 另一个 Layer 也能工作：Virtual Database、Cache 或 MQ 服务没有自己的 traces 标签页，但它的慢语句仍然能带你到发起方 Trace。跳转如果带 timestamp（日志行知道自己写入时间），查找会围绕那个时间放宽窗口，所以即便 Trace 已经进入冷存储，也不会悄悄找不到。\n后续阅读 Traces 标签页把一个请求完整展开；前面几篇的仪表盘和地图，是你最初发现异常的地方。字段参考，包括每个条件、native-vs-Zipkin 拆分和 span 详情面板，可以看 Traces 文档。\n下一篇看日志探索器：从 Trace 切到相关日志流，沿着同一条排查路径继续定位问题。\n","excerpt":"\u003cp\u003e\u003cem\u003e译自英文原文：\u003ca href=\"/blog/2026-06-22-horizon-ui-trace-explorer/\"\u003eMeet Horizon UI · 6/17: The Trace Explorer\u003c/a\u003e。\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e这是 \u003ca href=\"/zh/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e 系列的第六篇。前几篇都在讲地图：服务之间的\u003ca href=\"/zh/2026-06-21-horizon-ui-topology-and-dependency/\"\u003e拓扑\u003c/a\u003e、单个服务内部 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-06-22-horizon-ui-trace-explorer/","title":"认识 Horizon UI · 6/17：Trace 探索器"},{"body":"Apache SkyWalking Horizon UI is the next-generation web console for SkyWalking. It talks to the same OAP backend SkyWalking already runs — the same GraphQL query protocol, the same admin REST surface, the same MQE language, the same Layer concept — so you can point it at a running OAP and log in without changing a thing on the backend. What changes is everything in front of that contract.\nThis is the first post in a series. Across it we will walk through the dashboards and the metric query language, the topology views and a WebGL 3D map of your whole deployment, the trace and log explorers, profiling, the operations surface, access control, and the config-driven customization that ties it all together. This post sets the stage: what Horizon is, the one idea the whole UI is built around, and how to put it in front of your OAP today.\nHorizon is built around four verbs. You observe — topology, traces, logs, all five flavors of profiling, read-only alarms, and per-layer dashboards. Then you operate what you observe, govern who is allowed to touch it, and customize the whole console without writing UI code. Observe, operate, govern, customize: that is the arc of this series. We start where every session starts — the sidebar.\nThe sidebar is your estate Open Horizon and the left sidebar is not a hand-built menu — it is a live reflection of what your OAP is actually reporting. Horizon asks OAP which layers exist and which of them have services, and renders exactly those, refreshing on a 60-second cadence. A layer starts reporting, it shows up; it goes quiet, it falls away. The menu can\u0026rsquo;t drift from reality because the menu is reality, polled.\nFigure 1: Horizon\u0026rsquo;s home — the live estate on the left, the cross-layer Services overview on the right.\nA few things are happening in that sidebar:\nLive service counts. The Layers heading shows how many layers currently have services — 13 with services in Figure 1 — and each layer\u0026rsquo;s own count surfaces as you open it. Those counts come from a single server-side catalog the whole UI shares, refreshed once a minute, so the sidebar, the alarm layer-tagger, and the landing pages never disagree by a stale poll. Grouped, not a flat list. Layers cluster under their group — Virtual Targets, Istio, Kubernetes, MQ — beneath the Overviews and Alarms entries up top. Operate and admin areas (Cluster Status, Alerting rules, DSL Management, Users, Roles \u0026amp; permissions) appear lower down, and only for the roles allowed to open them — access control is woven into the menu itself, not bolted on after. Nothing silently hidden. Every layer OAP reports now appears — including ones with no bundled template, which fall back to a plain Service page. A hard-coded \u0026ldquo;hidden layers\u0026rdquo; list used to quietly drop layers like BanyanDB; that\u0026rsquo;s gone. Hiding a layer is now an explicit choice in horizon.yaml via layers.excluded (which ships defaulting to FAAS and VIRTUAL_GATEWAY — clear the list to surface everything). Click into any layer and Horizon opens its first available tab, following a consistent spine across every layer:\nservice → instance → endpoint → topology → trace → logs → profiling The slot names follow the layer. The General Service layer in Figure 2 labels its endpoint slot API, adds an API dependency view, and fans Profiling into the engines it actually has — Trace, eBPF, pprof (Go), and Async. Tabs a layer can\u0026rsquo;t support are simply turned off in its template, so you never land on an empty page. Pick a service and its dashboard fills the canvas to the right — a header strip of KPIs (RPM, Apdex, error rate, each with its own sparkline) over a widget grid scoped to exactly that entity.\nFigure 2: Expand a layer and it fans out into its full workflow — the tab spine on the left, the selected service\u0026rsquo;s dashboard on the right.\nNever a blank page A console that follows live data has to handle the moments when there isn\u0026rsquo;t any — a fresh install, a partially configured deployment, an OAP that just restarted. Horizon treats those as first-class states instead of dead ends.\nOpen the app at / and Horizon cascades to a real destination: the first available public overview dashboard, or failing that the first layer with services, and only if neither exists, the empty landing. When it does land on the empty page, it tells you which problem you have in plain language — \u0026ldquo;No data is flowing yet\u0026rdquo; (nothing is reporting) versus \u0026ldquo;No dashboard configured yet\u0026rdquo; (services exist, but no overview is set up) — and points you at your operations team rather than dropping you on a blank grid. As soon as a service reports or an operator publishes a dashboard, the next 60-second refresh replaces the empty page with the real one.\nFigure 3: The empty landing names the actual situation — here, services are reporting but no overview is configured — instead of a blank dashboard.\nThe same instinct shows up when OAP itself blips. If the backend goes briefly unreachable, Horizon keeps the last known sidebar shape on screen and raises an \u0026ldquo;OAP unreachable\u0026rdquo; banner, with service counts marked unknown until it recovers — so a short outage never looks like your configuration vanished.\nWhen data is flowing, that landing is the war-room you already saw in Figure 1 — a cross-layer overview with per-kind service tiles (General services, Virtual databases, caches, MQs, GenAI), the live topology, and the active-alarm rail. Per-layer landings rank the true top-N services by your chosen column (no more capping at an arbitrary first 25 before ranking) and tell you \u0026ldquo;top N of M\u0026rdquo; so the trim is never silent.\nAnd because long layer names and deep namespaces happen, the shell gets out of the way: drag the divider to resize the sidebar (double-click to reset), or fold it to a thin icon rail to give the canvas every horizontal pixel — and the width is remembered per browser.\nFigure 4: Drag the divider to widen the sidebar — long names stop truncating, and the width is remembered per browser.\nFigure 5: Fold the sidebar to a thin icon rail when you want the canvas to have every pixel.\nOne new tier makes the rest possible Until now, SkyWalking\u0026rsquo;s web UI talked straight to OAP from the browser. Horizon introduces one small piece of infrastructure in between: a Backend-for-Frontend (BFF), a Fastify service on Node.js that serves the UI and proxies every call to OAP.\nThe browser talks only to the BFF; the BFF owns auth, RBAC, audit, capability probing, and server-side i18n / widget gating, then proxies to OAP\u0026rsquo;s query host (:12800) and admin host (:17128).\nThat tier is why the later posts in this series exist at all. Authentication, role-based access control, and the audit trail are enforced on the server, where a forged request can\u0026rsquo;t get past them. The BFF probes OAP\u0026rsquo;s GraphQL schema once at startup and degrades gracefully when a capability is missing — which is exactly how Horizon supports two OAP generations from one build (more on that below). It caches the service catalog once a minute so the whole UI shares one view of the estate. We\u0026rsquo;ll come back to each of these in the posts on operations, security, and customization; for now the thing to know is that there\u0026rsquo;s a server here now, and it\u0026rsquo;s doing real work.\nA 3D look at the whole thing One surface is worth previewing up front, because it captures the \u0026ldquo;stand back and look at everything at once\u0026rdquo; idea better than any screenshot can: the 3D Infrastructure Map. Every layer\u0026rsquo;s services become cubes, stacked onto tiers that read top-to-bottom the way a request flows, with live traffic, alarms, and call relationships drawn between them. Drag to orbit it:\nInteractive · sample data Open the 3D map A dedicated post later in the series takes the 3D map apart properly — the tiers, the alarm beacons, \u0026ldquo;Beacon mode\u0026rdquo; that ghosts everything healthy so only what\u0026rsquo;s firing glows, and the structured editor that configures it. For now, it\u0026rsquo;s a fair picture of the ambition: your whole deployment, in one view, alive.\nWhat\u0026rsquo;s in this series Fifteen posts follow this one, each a standalone tour of one corner of Horizon — read them in any order; each links back here for the lay of the land. They fall into four arcs.\nSee your data\nDashboards \u0026amp; MQE — widgets that query only what\u0026rsquo;s relevant to the entity in front of you, value formatting humans can read, synced crosshairs, and multi-entity compare. Topology \u0026amp; service dependency — one topology engine that repaints for every layer, the de-noising filter, and the multi-hop API-dependency graph. The Deployment tab \u0026amp; BanyanDB self-observability — a view that looks inside a single clustered service, and SkyWalking finally watching its own database the way it watches everything else. The 3D Infrastructure Map — the full treatment of the view above. Trace explorer — lasso the slow traces on a duration scatter, then read each one three ways. Log explorer — a Loki-style stream with facets, top patterns, and structured payloads. Browser \u0026amp; RUM monitoring — front-end error logs, and de-obfuscating a minified stack back to its original source line. Five profilers, one flame graph — trace, async-profiler, eBPF, Go pprof, and network profiling, unified behind one workflow. Operate it\nAlarms \u0026amp; incident triage — incident-centric active alarms that ship the chart that fired them. Runtime rules, live debugging \u0026amp; inspect — hot-reloadable rules and a live debugger that steps OAL (traces), MAL (metrics), and LAL (logs) against live samples. Platform \u0026amp; cluster introspection — read the OAP cluster\u0026rsquo;s health, resolved config, and data retention from the UI. Govern \u0026amp; secure it\nAccess control \u0026amp; security — server-enforced RBAC, LDAP/AD, an audit trail, and break-glass — the capabilities that make the UI enterprise-deployable. Make it yours, and adopt it\nCustomization: config-driven layer templates — draft-to-publish, and adding a whole new monitored layer with zero UI code. Localization — every dashboard in eight languages, translated by clicking widgets in a live preview. Getting started \u0026amp; migration — install, the OAP version matrix, and swapping an existing UI for Horizon. Try it against your OAP today Horizon runs against the OAP you already have — and on today\u0026rsquo;s OAP 10.x nearly all of it works. Every dashboard, the topology, traces (native and Zipkin), logs, alarms, and all five profilers render off OAP\u0026rsquo;s query host (:12800), and Horizon\u0026rsquo;s access control, audit, and themes run in the BFF, independent of the OAP version. What waits for OAP 11.0 — the admin host (:17128) — is the operate layer: runtime-rule (DSL) management, the Live Debugger, Metrics Inspect, the alarm-rule editor, the Cluster Status admin pane, and publishing template edits back to OAP. Horizon detects each admin module by its presence and simply hides the pages 10.x can\u0026rsquo;t serve, so the full observability console runs on 10.x today and the operate tooling lights up the moment you move to 11.0.\nPoint Horizon at your existing cluster and bring it up — no backend changes, the same OAP your deployment already talks to:\ndocker run -d --name horizon \\ -p 8081:8081 \\ -v \u0026#34;$PWD/horizon.yaml:/app/horizon.yaml:ro\u0026#34; \\ -v horizon-state:/data \\ ghcr.io/apache/skywalking-horizon-ui:\u0026lt;version\u0026gt; A minimal horizon.yaml is just where OAP lives and one local user to log in as:\noap: queryUrl: http://\u0026lt;oap-host\u0026gt;:12800 adminUrl: http://\u0026lt;oap-host\u0026gt;:17128 auth: backend: local local: users: - username: admin passwordHash: \u0026#34;$argon2id$v=19$...\u0026#34; # generated, never plaintext roles: [admin] Open http://\u0026lt;host\u0026gt;:8081/, log in, and the first stop is Cluster Status to confirm Horizon and OAP are talking. From there, the sidebar fills in with your estate.\nFor the full setup path — binary tarball, Kubernetes, LDAP, TLS, and the production checklist — see the Horizon UI documentation, which covers setup, compatibility, access control, customization, components, and operations from its left-side menu.\nOther notable points Drop-in against your existing OAP. Horizon is a greenfield rewrite that keeps every backend contract — the same GraphQL query protocol, admin REST surface, MQE language, and Layer concept — so you point it at a running cluster with no backend change. On today\u0026rsquo;s OAP 10.x the whole observability console works (dashboards, topology, traces including Zipkin, logs, alarms, profiling) along with Horizon\u0026rsquo;s BFF-side access control, audit, and themes; only the operate tooling — runtime rules, Live Debugger, Inspect, the Cluster Status admin pane, and publishing template edits — waits on OAP\u0026rsquo;s admin host (:17128), which ships with OAP 11.0. Dark-first and dense. A 12-column grid built for incident scanning — more signal above the fold, less whitespace. Built on a modern stack. Vue 3 + TypeScript on Vite, Pinia, Apache ECharts, D3, and Monaco on the front end; Fastify on Node.js for the BFF. It\u0026rsquo;s Apache-licensed and community-built. Horizon UI lives at apache/skywalking-horizon-ui. Try it against your cluster, and tell us what\u0026rsquo;s missing — issues and pull requests are welcome. Next up in the series: the dashboards — and why a widget can decide, on the server, that it shouldn\u0026rsquo;t even run its query for the entity you\u0026rsquo;re looking at.\n","excerpt":"\u003cp\u003eApache SkyWalking Horizon UI is the next-generation web console for SkyWalking. It talks to the same …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-06-21-skywalking-horizon-ui-introduction/","title":"Meet Horizon UI · 1/17: SkyWalking's New Observability Console"},{"body":"This is the second post in the Apache SkyWalking Horizon UI series. The first one introduced the console and its layered navigation; this one is about the surface you spend most of your day on — the dashboards.\nEvery dashboard in Horizon is the same machine: a grid of widgets, each one an MQE expression the BFF resolves against OAP. What makes the surface worth a post of its own is what that machine does around the numbers — it hides the widgets that don\u0026rsquo;t apply to the entity you\u0026rsquo;re looking at (and skips their queries entirely), it renders coded values and raw byte counts as things a human can read, it ties every chart on the page to one cursor, and it drops you from a slow-SQL row straight into the trace that produced it. Let\u0026rsquo;s walk through it.\nEvery widget is one MQE expression A layer dashboard is a dense 12-column grid (120px rows, gaps backfilled so a wall of tiles has no holes, collapsing to one column under ~1100px). Each tile is one of five widget types, and the type follows the shape of its MQE expression:\ncard — the expression collapses to a single scalar (latest(...), avg(...), service_sla/100). One big number. line — a time series; one line per expression, optional dual y-axis for mixed units (throughput on the left, latency on the right). top — a ranked list from top_n(endpoint_cpm, 20, des), with a small tab switcher to flip the ranking between Traffic / Slow / Successful Rate. record — record-shaped output like slow database statements or slow cache commands: rows of text + value. table — a labeled latest(...) metric, one row per label combination (pod phase per service, node condition, replicas per deployment). You don\u0026rsquo;t pick the chart; you write the MQE and the right widget renders it. And it\u0026rsquo;s the same grid system at every altitude — the dashboards.\u0026lt;scope\u0026gt; map on a layer template carries a different widget set for the service, instance, and endpoint pages, so drilling down swaps the whole dashboard to the right scope. (All of these run on the BFF tier introduced in part 1 — the browser never talks to OAP directly.)\nA dashboard that adapts to the entity in front of you Here\u0026rsquo;s the feature that changes how a dashboard feels. A widget can carry a visibleWhen gate, and when the gate doesn\u0026rsquo;t hold, the widget doesn\u0026rsquo;t render — and, crucially, its query never runs.\nThere are two kinds of gate:\nMQE metric — show the widget only when an expression has value (op: exists), or when its value crosses a threshold (gt / lt). Point a widget at its own metric and it self-gates: the JVM widgets carry \u0026quot;visibleWhen\u0026quot;: { \u0026quot;kind\u0026quot;: \u0026quot;mqe\u0026quot;, \u0026quot;expression\u0026quot;: \u0026quot;instance_jvm_cpu\u0026quot;, \u0026quot;op\u0026quot;: \u0026quot;exists\u0026quot; }, so they appear on a Java instance and vanish on a Go one. Entity attribute — on the Instance scope, gate on the selected instance\u0026rsquo;s attributes (language eq JAVA, or an attribute simply being present). Because the gate is evaluated server-side, a non-JVM instance doesn\u0026rsquo;t just hide the JVM tiles — the BFF never sends their queries to OAP at all. Open the same Instance dashboard for a JVM service and a non-JVM one and you\u0026rsquo;re looking at one template adapting itself, not two hand-built pages:\nFigure 1: On a JVM instance the JVM widgets render — their visibleWhen gate holds.\nFigure 2: The same dashboard on a Go instance — the JVM widgets aren\u0026rsquo;t there, and their queries never ran. One template, adapting to the entity.\nNumbers humans can read Raw metrics are not always readable metrics. Horizon\u0026rsquo;s widgets format three cases that used to make operators do math in their heads:\nenum — a value→label map turns a coded gauge into words: a 1/0 success metric renders OK / Failed instead of the bare number. The labels are translatable per locale. duration — a metric in seconds renders as a human time-ago: 5m 20s ago, compact to 5m / 2h on an axis. SI suffixes — large magnitudes on chart axes and tooltips read as 45.1k, 1.34M, 2.5G rather than 4.51e4, with the axis tick and its hovered value sharing one notation. Figure 3: Dense byte and count series get compact SI suffixes, axis and tooltip in step.\nFigure 4: The enum and duration formats in action on BanyanDB\u0026rsquo;s lifecycle cards — OK instead of 1, \u0026ldquo;5m 20s ago\u0026rdquo; instead of a seconds count.\nRead the whole grid as one timeline Every line chart on a page shares one hover cursor. Point at minute 32 on the throughput chart and minute 32 lights up on the latency chart, the error-rate chart, and every sparkline tile. The contract is enforced at the chart-wrapper level — no widget can opt out — so the page reads as a single coordinated view of one moment, not a dozen independent charts. The multi-series tooltip is a fixed, aligned table that shows each series\u0026rsquo; title (never the raw MQE), with values in one right-aligned column.\nFigure 5: One cursor moves across every line chart on the page, so you read the same instant everywhere at once.\nFrom a slow row to its trace, in one click record widgets — Slow Statements, Slow Commands, Slow Database Statements — are lists of sampled records, and each row that carries a trace id gets a jump-to-trace icon at its head that opens the originating trace\u0026rsquo;s waterfall. It resolves the trace by id, not by layer, which matters: the Slow Statements on a Virtual Database / Cache / MQ service belong to the caller on another layer, and a virtual-target layer has no traces tab of its own — yet the jump still lands. The statement text itself is click-to-copy.\nFigure 6: From a slow statement to the trace that ran it — resolved by trace id, so it works even on a virtual layer with no traces tab of its own.\nPin and compare entities Sometimes one entity isn\u0026rsquo;t enough. Horizon lets you lock several services, instances, or endpoints — even ones from different services — and compare them in place. Pin entities from the picker or the instance/endpoint list; the one you\u0026rsquo;re viewing is always part of the cohort (tagged CURRENT) and still drives the header, and each pin adds its own hue. Every widget then compares inline — line widgets overlay one series per entity, cards show a row each, top and record widgets get per-entity tabs, tables gain an Entity column. A persistent comparison bar holds the cohort no matter how the underlying list paginates or which entity you\u0026rsquo;re currently viewing, and each entity loads as its own request, so one slow one never blanks the others.\nFigure 7: Lock entities — even across services — and every line widget overlays them hue-by-hue; the comparison bar holds the cohort while the CURRENT entity still drives the header.\nThe time picker moves the whole dashboard The topbar time range drives everything on the page — the header KPI strip, the widget body, and (on BanyanDB\u0026rsquo;s tiered hot/warm/cold storage) the Cold pill flow end-to-end. Earlier the landing and topology routes were pinned to the last 60 minutes, so picking \u0026ldquo;12 days ago\u0026rdquo; quietly kept showing recent numbers; now the picker is honored everywhere, and when an upstream control changes, each dependent tile visibly resets and shows a \u0026ldquo;Reading data…\u0026rdquo; hint rather than leaving a stale value under a spinner.\nWhere to go next It\u0026rsquo;s worth stressing that this is one system. The same five widgets, the same MQE, the same gating render every layer\u0026rsquo;s dashboard — the JVM panels above, BanyanDB\u0026rsquo;s lifecycle cards, the percentile latency on a mesh service, and purpose-built panels for things like an Envoy AI Gateway (token throughput, time-to-first-token) or a GenAI virtual layer (per-model estimated cost). What changes from layer to layer is the MQE, not the machinery.\nEverything above is the reading experience. Each widget\u0026rsquo;s MQE, its visibleWhen gate, its format, and the per-scope grids are all editable from the Layer dashboards admin — but that authoring story (draft → preview → publish, with the inline-and-expand MQE editor) is its own post later in the series. For the field-level reference, see the docs on dashboard widgets and charts.\nNext up: topology and service dependency — the same data Horizon charts here, drawn as a map you can walk.\n","excerpt":"\u003cp\u003eThis is the second post in the \u003ca href=\"/blog/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eApache SkyWalking Horizon UI\u003c/a\u003e series. The first one introduced the …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-06-21-horizon-ui-dashboards-and-mqe/","title":"Meet Horizon UI · 2/17: Dashboards That Adapt — MQE, Smart Widgets, and Numbers Humans Can Read"},{"body":"This is the third post in the Meet Horizon UI series. Part 2 was about reading your services as numbers on a dashboard; this one is about reading them as a map — who calls whom, how hard, and how that same logical service looks across every layer it reports through.\nThe call data behind a SkyWalking topology is one thing; the views Horizon draws from it are several. There\u0026rsquo;s the per-layer service map, a drill-down into the instances behind a single call, an endpoint-level dependency graph, and a cross-layer overlay that ties a service\u0026rsquo;s faces together. They\u0026rsquo;re all the same engine, pointed at different questions. (The Deployment tab — the map of one clustered service\u0026rsquo;s own instances — and the WebGL 3D map are big enough to get their own posts next.)\nOne topology engine, repainted per layer Open any layer\u0026rsquo;s Topology tab and you get a left-to-right hierarchical service map: User (when present) seeds the left, and each service sits in a column by its call depth — within a column, nodes keep the order the graph walk reached them, so the dominant chain reads top-down. Each service is a hexagonal node, and everything it shows is driven by the layer\u0026rsquo;s config — nothing is hardcoded:\nthe hexagon\u0026rsquo;s border carries the node\u0026rsquo;s ring metric as an SLA-style health band (green → red); the component icon sits inside the hex — the same icon set the trace waterfall uses, so a PostgreSQL node looks like PostgreSQL, a Kafka node like Kafka; the node\u0026rsquo;s headline number — its center metric — prints just above the hex, with its unit; the service name prints below it, and a secondary metric (latency, by default) sits beneath the name; each edge carries the call\u0026rsquo;s throughput as an RPM chip (the server-side metric, falling back to the client side). Here\u0026rsquo;s the part worth stressing: every one of those is just the General layer\u0026rsquo;s bundled default. The node\u0026rsquo;s center / ring / secondary metrics and the edge metrics each live in the Layer dashboards admin → Topology scope as an MQE expression with a unit and a role — so you can point any slot at a different metric, and the same engine paints a different map for a different layer, or for this one your way. (The choices travel with the layer template\u0026rsquo;s export/import, like everything else.)\nFigure 1: One template-driven topology engine — health-banded hex nodes, RPM-chipped edges, real component icons; every metric on it is configured per layer.\nCut the noise Real topologies are noisy — a dense map fills with conjectured peers OAP couldn\u0026rsquo;t fully resolve (a bare rcmd:80, an un-instrumented address). Horizon\u0026rsquo;s Filter control turns those off without hiding your real dependencies. It derives one facet automatically — by layer — and presents it exactly as the sidebar does: each row carries the layer\u0026rsquo;s own icon and localized name (Virtual Database, Java Agent, …), plus an Others bucket for nodes OAP couldn\u0026rsquo;t classify and a standalone User toggle. Uncheck Others and the uninstrumented clutter — and its dangling edges — disappears, while your databases, caches and queues (separated by their own VIRTUAL_* rows) stay on the map. Filtering is client-side and the rows re-derive on every refresh, so it never goes stale.\nFigure 2: De-noise in one click — drop the unresolved \u0026ldquo;Others\u0026rdquo; peers and keep your real dependencies.\nWhen a layer\u0026rsquo;s services fall into OAP service groups, the map\u0026rsquo;s service-focus selector groups by them too, and clicking a group header batch-selects or clears every service in that group — so you can focus a whole team\u0026rsquo;s slice of a busy map at once.\nFigure 3: Focus a whole service group at once from the map\u0026rsquo;s selector.\nDrill from a call into its instances A service-to-service edge is an aggregate — behind it are real instances talking to real instances. Click a call on the map and choose Instance map →, and Horizon draws exactly that: the client service\u0026rsquo;s instances in the left column, the server service\u0026rsquo;s in the right, with the instance-level calls between them, animated client→server. It reuses everything from the service map — the health-ring nodes, the per-call client/server metric sidebar, a node popover with Open instance dashboard — and labels the columns in the layer\u0026rsquo;s own vocabulary (Pods on Kubernetes, Sidecars on the data plane). The two service pickers are relationship-aware: the server list is the chosen client\u0026rsquo;s callees, the client list is the chosen server\u0026rsquo;s callers, each re-deriving as you change the other.\nFigure 4: Drill from an aggregate call into the instance-to-instance traffic behind it.\nWalk the request chain, endpoint by endpoint Service topology answers \u0026ldquo;which services call this one.\u0026rdquo; The API dependency tab answers the sharper question — \u0026ldquo;which endpoints call this endpoint, and which does it call.\u0026rdquo; Pick an endpoint and it lays out in columns by direction: callers on the left, the focus endpoint in the centre, callees on the right, with the same SLA-coloured node border, the RPM and latency on every edge, and the heaviest edge labeled. A selected node shows a single + handle that pulls in its own callers and callees in one click, so you walk the chain one hop at a time instead of drowning in the whole graph; drag nodes apart, and the drill-out links (Open endpoint, Service →) open in a new tab so you keep the graph you\u0026rsquo;re exploring.\nFigure 5: Walk the endpoint call chain a hop at a time, latency on every edge.\nOne service, every layer it reports through A single logical service often reports through several layers at once — a General agent, a Service Mesh sidecar, the mesh data plane, a Kubernetes pod. SkyWalking has modeled that cross-layer hierarchy since OAP 10; what Horizon adds is making it one click from anywhere on the map. Select a node and Horizon lazily probes its hierarchy — if the service has cross-layer peers, a small chevron-stack chip clips to the node.\nFigure 6: The chevron-stack chip on a selected node — lazily probed on selection, it shows only when the service has cross-layer peers.\nClick the chip and the topology dims under a Smartscape overlay: the focused node re-renders bright in place, and its peers fan out vertically by OAP\u0026rsquo;s layer order — request-near layers above, infra-near below. From there a two-step click opens any peer in its own layer, pre-selected. (Auto-refresh pauses while the overlay is open so nothing shifts under you.)\nFigure 7: One service, every layer it reports through — the cross-layer hierarchy as a one-click overlay on the map.\nWhere to go next Every metric, threshold, and edge weight on these maps lives in the layer template\u0026rsquo;s topology block — which means you tune them the same config-driven way you tune dashboards, the subject of a later post in this series. For the field reference, see the layer-template topology docs.\nNext up: the Deployment tab and BanyanDB self-observability — where the same map technique turns inward to show how one clustered service\u0026rsquo;s own instances are deployed and talk to each other.\n","excerpt":"\u003cp\u003eThis is the third post in the \u003ca href=\"/blog/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e series. \u003ca href=\"/blog/2026-06-21-horizon-ui-dashboards-and-mqe/\"\u003ePart 2\u003c/a\u003e was about reading your services as …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-06-21-horizon-ui-topology-and-dependency/","title":"Meet Horizon UI · 3/17: Topology \u0026 Service Dependency"},{"body":"This is the fourth post in the Meet Horizon UI series. Part 3 drew the map between services. This one turns the same map inward — onto the instances inside one clustered service — and uses it for something SkyWalking has never shown well before: its own storage engine, BanyanDB, modeled as the cluster it actually is.\nThe Deployment tab: a map of one service\u0026rsquo;s own instances The service map answers \u0026ldquo;who calls this service.\u0026rdquo; The new per-layer Deployment tab answers a different question: \u0026ldquo;how is this one service deployed, and how do its own instances talk to each other?\u0026rdquo; Pick a service and the tab draws its instances as nodes with the instance-to-instance calls between them — the same pan/zoom canvas, health-ring nodes, animated edge flow and per-call metric sidebar you know from the service map, but scoped to a single service\u0026rsquo;s internals.\nThree things make it more than a flat node cloud:\nInstances render as hexagons that bundle into pods. A pod\u0026rsquo;s main container is a full hex with its sibling containers attached as smaller hexes around its edge — so a main process and its sidecars read as one unit, and a cross-pod sidecar link connects the exact small hex it belongs to. Pods cluster into labelled boxes by a rule you choose — a single instance attribute (role), several attributes combined (e.g. node_role + node_type), or a name regex — so a fleet of mixed-role nodes reads as one box per role instead of a cloud. The layout is tiered. Each cluster box lays its pods out by call depth — sources on the left, what they call to the right — so an upstream→downstream chain reads left-to-right; drag any pod and its box re-flows to keep everything enclosed. Edges are keyed by the (source-role → target-role) pair, so each kind of link shows its own metrics rather than one flat set, prints its headline number inline on the edge, and lists in full in a Flows sub-tab — one aligned table per role-pair. It\u0026rsquo;s off by default and, like the service map, entirely configured from the Layer dashboards admin → Deployment scope.\nBanyanDB, watched like everything else That machinery exists for a reason. SkyWalking\u0026rsquo;s native database, BanyanDB, is a clustered, role- and tier-aware system — and until now SkyWalking couldn\u0026rsquo;t really observe it as one. The new BanyanDB layer (under Self-Observability), pairing with OAP backend SWIP-15, models the whole deployment from metrics scraped through BanyanDB\u0026rsquo;s FODC proxy:\nthe cluster is one Cluster (a service), each container is one Container (an instance, carrying its container_name role and node_type tier as attributes), and each storage Group is an endpoint. So the same Service / Instance / Endpoint spine every other layer uses now means Cluster / Container / Group for BanyanDB — and the Deployment tab on top of it draws the cluster itself.\nFigure 1: SkyWalking watching its own database — the BanyanDB cluster drawn by role and tier, with liaison→data and lifecycle→data edges between the pods.\nCluster, Container, Group Each scope is a purpose-built dashboard:\nThe Cluster dashboard is the war-room for the whole database: write / query / error-rate KPIs, CPU / memory / disk capacity, throughput and error trends, and a Containers by Role table. The Container dashboard adapts to the selected container\u0026rsquo;s role. Every container shows CPU / memory / Go-runtime resources; a liaison adds ingestion, query, gRPC errors and the tier-2 publish pipeline and write-queue depth; a data node adds storage totals, merge / compaction, the inverted index, the subscribe queue and retention; a lifecycle sidecar shows migration cycles and last-run time / status. The role-specific panels are gated on the container\u0026rsquo;s role attribute, so you only ever see what applies to the node in front of you. The Group dashboard splits per data-model — measure, stream, trace, property — and because a BanyanDB group stores exactly one catalog, only the matching model\u0026rsquo;s panels render: a measure group shows write-rate / query-latency / merge panels, a property group its index-write / term-search / series panels, and so on. Figure 2: The Cluster scope — the whole database at a glance, with a roll-call of containers by role. The role-gating is easiest to see by opening the same Container dashboard on two different roles:\nFigure 3: A data/liaison Container — ingestion, query, storage and compaction panels on top of the shared resource panels. Figure 4: The same dashboard on the lifecycle node — just the migration panels. Same template, gated by role. And the Group scope gives each storage catalog its own page:\nFigure 5: The Group scope — one storage catalog at a time, its panels gated to the group's data model. Edges that know their role pair, and a Flows table On the Deployment tab, the call edges between containers carry role-pair-specific metrics off the SWIP-15 instance-relation families: a liaison → data edge shows write / query / part-sync throughput and p99; a liaison → liaison edge shows write-forward and control; a lifecycle → data edge shows tier-migration volume / rate / p99. Each edge prints up to three of its pair\u0026rsquo;s metrics inline, the selected-edge panel keeps the full client-vs-server breakdown, and the Flows sub-tab lays every edge out as one aligned table per role-pair.\nFigure 6: Flows — the same role-pair edges as a sortable table, one block per pair.\n(Two preconditions. The edges and the role-specific panels assume a real clustered BanyanDB — a single-process standalone instance shows only the shared resource and Go-runtime panels, with the rest lighting up as the cluster\u0026rsquo;s roles report. And the container-to-container edges in particular need the OAP build to expose the SERVICE_INSTANCE_RELATION scope; until it does, the Deployment tab still draws the full inventory — just without the edges between pods.)\nConfigured, not coded None of the above is a hand-built \u0026ldquo;BanyanDB screen.\u0026rdquo; The clustering rules, the per-role node metrics, and the role-pair edge metrics are all a self-contained block on the layer template, edited from the Layer dashboards admin → Deployment scope and carried with the template\u0026rsquo;s export/import — the same config-driven model behind every other layer, which a later post covers end to end.\nFor the BanyanDB layer\u0026rsquo;s own fields — the Cluster, Container, Group, and Deployment views — see the BanyanDB dashboards doc.\nNext up: the 3D Infrastructure Map — where this same deployment, and every other layer, lifts off the page into a WebGL view of your whole estate.\n","excerpt":"\u003cp\u003eThis is the fourth post in the \u003ca href=\"/blog/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e series. \u003ca href=\"/blog/2026-06-21-horizon-ui-topology-and-dependency/\"\u003ePart 3\u003c/a\u003e drew the map \u003cem\u003ebetween\u003c/em\u003e services. This …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-06-21-horizon-ui-deployment-and-banyandb/","title":"Meet Horizon UI · 4/17: The Deployment Tab \u0026 BanyanDB Self-Observability"},{"body":"译自英文原文：Meet Horizon UI · 1/17: SkyWalking\u0026rsquo;s New Observability Console。\nApache SkyWalking Horizon UI 是 SkyWalking 的新一代 Web 控制台。它仍然连接你已经在运行的 OAP 后端：同样的 GraphQL 查询协议，同样的 admin REST 接口，同样的 MQE 语言，同样的 Layer 概念。换句话说，你可以直接把 Horizon 指向正在运行的 OAP，然后登录使用，不需要改后端。变化集中在这些后端协议之上的整套交互体验。\n这是这个系列的第一篇。后续文章会依次介绍仪表盘和指标查询语言、拓扑视图、整个部署的 WebGL 3D 地图、链路和日志探索器、性能剖析、运维界面、访问控制，以及把这些模块串起来的配置化定制。本篇先交代背景：Horizon 是什么，整个 UI 围绕哪一个核心想法构建，以及今天如何把它接到你的 OAP 前面。\nHorizon 的主线可以概括为四个动词：先 observe，看拓扑、链路、日志、五类性能剖析、只读告警，以及每个 Layer 的仪表盘；再 operate 这些被观测对象；然后 govern 谁可以操作它们；最后在不写 UI 代码的情况下 customize 整个控制台。Observe、operate、govern、customize，就是这个系列的主线。我们从每次会话都会看到的地方开始：侧边栏。\n侧边栏显示当前系统全貌 打开 Horizon，左侧边栏不是手工写死的菜单，而是 OAP 当前上报内容的实时映射。Horizon 会向 OAP 查询有哪些 Layer、哪些 Layer 里有服务，然后只渲染这些内容，并每 60 秒刷新一次。某个 Layer 开始上报，它就出现；不再上报，它就消失。菜单不会和真实状态脱节，因为它直接来自 OAP 当前状态。\n图 1：Horizon 首页，左侧是实时系统全貌，右侧是跨 Layer 的 Services 概览。\n这个侧边栏有几个关键点：\n实时服务计数。 Layers 标题显示当前有多少个 Layer 包含服务，图 1 中是 13 with services。展开每个 Layer 后，也能看到它自己的服务数量。这些计数来自同一个服务端目录，整个 UI 共用，并且每分钟刷新一次，所以侧边栏、告警里的 Layer 标记和落地页不会因为各自轮询而出现不一致。 按组组织，而不是平铺列表。 Layer 会放在自己的分组下，比如 Virtual Targets、Istio、Kubernetes、MQ。顶部是 Overviews 和 Alarms，运维和管理区域（Cluster Status、Alerting rules、DSL Management、Users、Roles \u0026amp; permissions）放在更靠下的位置，并且只对有权限的角色展示。菜单生成时就会应用访问控制，而不是先渲染出来再补一层拦截。 不会静默隐藏内容。 OAP 上报的每个 Layer 现在都会出现。即使没有内置模板，也会回退到一个普通的 Service 页面。过去有一份写死的 \u0026ldquo;hidden layers\u0026rdquo; 列表，会悄悄隐藏 BanyanDB 这样的 Layer；现在不再有这层隐藏逻辑。想隐藏某个 Layer，需要在 horizon.yaml 里通过 layers.excluded 明确配置。默认值是 FAAS 和 VIRTUAL_GATEWAY；清空列表就可以展示所有 Layer。 点击任意 Layer，Horizon 会打开它的第一个可用标签页。所有 Layer 都沿用同一条固定路径：\nservice → instance → endpoint → topology → trace → logs → profiling 槽位名称会跟随 Layer 的语义变化。图 2 中的 General Service Layer 会把 endpoint 槽位命名为 API，增加一个 API dependency 视图，并把 Profiling 展开成它实际拥有的引擎：Trace、eBPF、pprof（Go）和 Async。某个 Layer 不支持的标签页会在模板里直接关掉，所以你不会打开一个空页面。选中一个服务后，右侧画布就是这个服务的仪表盘：上方是一排 KPI（RPM、Apdex、错误率，每个都有自己的 sparkline），下方的组件网格只查询当前服务相关数据。\n图 2：展开一个 Layer 后进入完整流程，左侧是标签路径，右侧是选中服务的仪表盘。\n没有数据时，也要说明原因 一个跟随实时数据变化的控制台，必须能处理“没有数据”的时刻：全新安装、配置不完整的部署，或者刚刚重启的 OAP。Horizon 把这些情况都作为明确状态处理，而不是留给用户一个死胡同。\n打开 / 时，Horizon 会按顺序跳到一个真实可用的页面：第一个可用的公共 overview 仪表盘；如果没有，就进入第一个有服务的 Layer；只有两者都不存在时，才展示空状态页。确实没有可跳转页面时，它会明确说明问题在哪里：\u0026ldquo;No data is flowing yet\u0026rdquo; 表示还没有任何内容上报，\u0026ldquo;No dashboard configured yet\u0026rdquo; 表示已经有服务，但没有配置 overview。它会把问题指向数据接入或运维配置，而不是把你丢在一个空网格上。只要有服务开始上报，或者运维人员发布了仪表盘，下一次 60 秒刷新就会把空状态替换成真实页面。\n图 3：空状态页说明真实原因。这里是服务已经上报，但没有配置 overview，而不是给出一个空仪表盘。\nOAP 短暂不可达时，Horizon 也遵循同样思路。如果后端短时间连不上，Horizon 会保留最后一次已知的侧边栏结构，并显示 \u0026ldquo;OAP unreachable\u0026rdquo; 横幅，服务计数标记为未知，直到恢复为止。短暂故障不会看起来像配置突然消失。\n当数据正常流入时，落地页就是图 1 里看到的总览页：跨 Layer 概览、按类型拆分的服务卡片（General services、Virtual databases、caches、MQs、GenAI）、实时拓扑和活跃告警栏。每个 Layer 自己的落地页会按你选择的列真正计算 top-N 服务，不再先截取任意前 25 个再排序；页面还会告诉你 \u0026ldquo;top N of M\u0026rdquo;，所以截断不会悄悄发生。\n长 Layer 名称和深命名空间很常见，所以页面框架需要给内容让出空间：拖动分隔线可以调整侧边栏宽度，双击重置；也可以折叠成一条窄图标栏，把水平空间尽量交给画布。宽度会按浏览器记住。\n图 4：拖动分隔线加宽侧边栏，长名称不再被截断，宽度会按浏览器记住。\n图 5：需要最大画布空间时，可以把侧边栏折叠成窄图标栏。\nBFF：Horizon 的服务端入口 过去，SkyWalking Web UI 是浏览器直接访问 OAP。Horizon 在浏览器和 OAP 之间引入了一个服务端组件：Backend-for-Frontend (BFF)，一个运行在 Node.js 上的 Fastify 服务。它负责提供 UI，并代理所有到 OAP 的调用。\n浏览器只访问 BFF；BFF 负责认证、RBAC、审计、能力探测，以及服务端 i18n / 组件开关，然后代理到 OAP 的 query host（:12800）和 admin host（:17128）。\n后续文章里很多能力都依赖这一层。认证、基于角色的访问控制和审计都在服务端执行，伪造请求绕不过去。BFF 启动时会探测一次 OAP 的 GraphQL schema，某个能力不存在时就优雅降级；Horizon 能用同一个构建支持两代 OAP，靠的就是这个机制。它还会每分钟缓存一次服务目录，让整个 UI 共享同一份系统视图。运维、安全和定制相关的文章会分别展开这些内容；这里先记住一点：UI 前面现在有了一个真正做事的服务端。\n用 3D 地图查看完整部署 有一个界面值得先提前看，因为它最能表达“后退一步，一次看清全局”的想法：3D Infrastructure Map。每个 Layer 的服务都会变成立方体，堆叠到按请求流向排列的层级上，实时流量、告警和调用关系都画在它们之间。拖动即可旋转：\n交互演示 · 示例数据 Open the 3D map 后续有一篇专门拆解 3D 地图：层级、告警信标、让健康对象变暗、只突出告警对象的 \u0026ldquo;Beacon mode\u0026rdquo;，以及用来配置它的结构化编辑器。现在先把它当成 Horizon 目标的一个缩影：整个部署放在一个视图里，而且是活的。\n系列后续内容 本篇之后，后续文章会分别介绍 Horizon 的不同模块。可以按任意顺序阅读，每篇也都会链接回这里。它们分成四条主线。\n看见你的数据\nDashboards \u0026amp; MQE：只查询当前对象相关数据的组件、指标格式化、同步十字线和多对象对比。 Topology \u0026amp; service dependency：一套可为每个 Layer 重绘的拓扑引擎、降噪过滤器和多跳 API dependency 图。 The Deployment tab \u0026amp; BanyanDB self-observability：深入单个集群服务内部的视图，以及 SkyWalking 终于像观测其他对象一样观测自己的数据库。 The 3D Infrastructure Map：详细展开上面这个 3D 视图。 Trace explorer：在时延散点图上框选慢 Trace，然后用三种方式阅读同一条 Trace。 Log explorer：类似 Loki 的日志流，带 facets、top patterns 和结构化内容。 Browser \u0026amp; RUM monitoring：前端错误日志，以及把压缩后的 stack 还原到原始源码行。 Five profilers, one flame graph：trace、async-profiler、eBPF、Go pprof 和 network profiling，统一到同一套工作方式里。 运维\nAlarms \u0026amp; incident triage：以 incident 为中心的活跃告警，并带上触发它的图表。 Runtime rules, live debugging \u0026amp; inspect：可热加载规则，以及用实时样本逐步调试 OAL（traces）、MAL（metrics）和 LAL（logs）的 Live Debugger。 Platform \u0026amp; cluster introspection：在 UI 里查看 OAP 集群健康、最终解析后的配置和数据保留策略。 治理与安全\nAccess control \u0026amp; security：服务端强制执行的 RBAC、LDAP/AD、审计轨迹和 break-glass，让 UI 可以进入企业部署。 定制与接入\nCustomization: config-driven layer templates：从 draft 到 publish，不写 UI 代码也能增加一个全新的被监控 Layer。 Localization：八种语言的仪表盘，并且可以在实时预览里点击组件完成翻译。 Getting started \u0026amp; migration：安装、OAP 版本矩阵，以及如何用 Horizon 替换现有 UI。 连接现有 OAP 即可试用 Horizon 可以运行在你已有的 OAP 之上。在今天的 OAP 10.x 上，绝大多数功能已经可用：所有仪表盘、拓扑、Trace（原生和 Zipkin）、日志、告警，以及五类性能剖析，都通过 OAP 的 query host（:12800）渲染。Horizon 的访问控制、审计和主题运行在 BFF 里，不依赖 OAP 版本。需要等待 OAP 11.0 的是 operate 层：runtime-rule（DSL）管理、Live Debugger、Metrics Inspect、告警规则编辑器、Cluster Status 管理面板，以及把模板编辑发布回 OAP。这些依赖 admin host（:17128），由 OAP 11.0 提供。Horizon 会按 admin module 是否存在来探测能力，10.x 不能提供的页面会直接隐藏；完整观测控制台今天就能跑在 10.x 上，迁移到 11.0 后运维工具会自动亮起来。\n把 Horizon 指向已有集群即可启动，不需要改后端。仍然是你的部署已经在使用的那个 OAP：\ndocker run -d --name horizon \\ -p 8081:8081 \\ -v \u0026#34;$PWD/horizon.yaml:/app/horizon.yaml:ro\u0026#34; \\ -v horizon-state:/data \\ ghcr.io/apache/skywalking-horizon-ui:\u0026lt;version\u0026gt; 最小化的 horizon.yaml 只需要说明 OAP 在哪里，以及一个可登录的本地用户：\noap: queryUrl: http://\u0026lt;oap-host\u0026gt;:12800 adminUrl: http://\u0026lt;oap-host\u0026gt;:17128 auth: backend: local local: users: - username: admin passwordHash: \u0026#34;$argon2id$v=19$...\u0026#34; # generated, never plaintext roles: [admin] 打开 http://\u0026lt;host\u0026gt;:8081/，登录后第一站是 Cluster Status，确认 Horizon 和 OAP 能正常通信。之后侧边栏就会填入你的系统全貌。\n完整安装路径，包括 binary tarball、Kubernetes、LDAP、TLS 和生产检查清单，请看 Horizon UI 文档。文档左侧菜单覆盖安装、兼容性、访问控制、定制、组件和运维。\n其他要点 可以直接接入现有 OAP。 Horizon 是一次从零开始的重写，但保留了所有后端契约：同样的 GraphQL 查询协议、admin REST 接口、MQE 语言和 Layer 概念。所以你可以把它指向一个正在运行的集群，不改后端。今天的 OAP 10.x 已经能运行完整观测控制台（仪表盘、拓扑、包括 Zipkin 在内的 Trace、日志、告警、性能剖析），以及 Horizon BFF 侧的访问控制、审计和主题。只有 operate 工具链需要等待 OAP 的 admin host（:17128），也就是随 OAP 11.0 提供的 runtime rules、Live Debugger、Inspect、Cluster Status admin pane 和模板编辑发布。 暗色优先，高密度。 12 列网格面向 incident 扫描设计，首屏承载更多信号，减少不必要留白。 基于现代技术栈。 前端是 Vue 3 + TypeScript on Vite、Pinia、Apache ECharts、D3 和 Monaco；BFF 使用 Node.js 上的 Fastify。 Apache 许可证，社区共建。 Horizon UI 位于 apache/skywalking-horizon-ui。欢迎接到你的集群上试用，也欢迎告诉我们缺什么，issue 和 pull request 都可以。 下一篇讲仪表盘：MQE 如何驱动组件，BFF 又如何在服务端判断哪些组件该显示、哪些查询可以直接省掉。\n","excerpt":"\u003cp\u003e\u003cem\u003e译自英文原文：\u003ca href=\"/blog/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI · 1/17: SkyWalking\u0026rsquo;s New Observability Console\u003c/a\u003e。\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eApache SkyWalking …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-06-21-skywalking-horizon-ui-introduction/","title":"认识 Horizon UI · 1/17：SkyWalking 新一代可观测性控制台"},{"body":"译自英文原文：Meet Horizon UI · 2/17: Dashboards That Adapt — MQE, Smart Widgets, and Numbers Humans Can Read。\n这是 Apache SkyWalking Horizon UI 系列的第二篇。第一篇介绍了控制台和按 Layer 组织的导航；这一篇讲你每天最常停留的界面：仪表盘。\nHorizon 里的每个仪表盘，本质上都使用同一套机制：一个组件网格，每个组件都是一条由 BFF 解析并发往 OAP 的 MQE 表达式。它值得单独写一篇，不只是因为能展示指标值，还因为它会处理很多实际问题：不适合当前服务、实例或 endpoint 的组件会被隐藏，相关查询也完全不会发出；编码值、耗时和原始字节数会按场景格式化；页面上所有图表都绑定到同一个游标；你还可以从一条慢 SQL 记录直接跳到产生它的 Trace。下面逐项看。\n每个组件都是一条 MQE 表达式 Layer 仪表盘是一个紧凑的 12 列网格：行高 120px，空隙会自动回填，避免组件墙上出现空洞；宽度低于约 1100px 时会折叠成单列。每个格子是五类组件之一，而组件类型取决于 MQE 表达式返回数据的 形状：\ncard：表达式收敛为单个标量，比如 latest(...)、avg(...)、service_sla/100。显示一个主指标值。 line：时间序列；每个表达式一条线，混合单位时可以使用双 Y 轴，比如左侧吞吐、右侧时延。 top：来自 top_n(endpoint_cpm, 20, des) 的排行列表，带一个小标签切换器，可以在 Traffic / Slow / Successful Rate 之间切换排序。 record：记录型输出，比如慢数据库语句或慢缓存命令：文本行加数值。 table：带标签的 latest(...) 指标，每组 label 一个表格行，比如每个服务的 pod phase、node condition、deployment replicas。 你不需要手工选择图表类型；写好 MQE，合适的组件会自己渲染。而且 service、instance、endpoint 这些层级使用同一套网格系统。Layer 模板里的 dashboards.\u0026lt;scope\u0026gt; map 会为不同页面配置不同组件集合，所以向下钻取时，仪表盘会切到对应作用域。（这些都运行在第一篇介绍的 BFF 上；浏览器不直接访问 OAP。）\n按当前服务或实例取舍组件 这个设计会明显改变仪表盘的使用感受。组件可以带一个 visibleWhen 条件。如果条件不成立，组件不渲染；更关键的是，它的查询也不会执行。\n条件有两类：\nMQE metric：只有表达式 有值 时展示组件（op: exists），或者只有值超过阈值时展示（gt / lt）。组件可以拿自己的指标做自我开关：JVM 组件带 \u0026quot;visibleWhen\u0026quot;: { \u0026quot;kind\u0026quot;: \u0026quot;mqe\u0026quot;, \u0026quot;expression\u0026quot;: \u0026quot;instance_jvm_cpu\u0026quot;, \u0026quot;op\u0026quot;: \u0026quot;exists\u0026quot; }，所以它们会出现在 Java 实例上，在 Go 实例上消失。 Entity attribute：在 Instance 作用域上，根据选中实例的属性开关，比如 language eq JAVA，或者判断某个属性是否存在。 因为条件在 服务端 计算，非 JVM 实例不只是把 JVM 格子藏起来；BFF 根本不会把这些查询发给 OAP。用同一个 Instance 仪表盘打开 JVM 实例和 Go 实例时，你看到的是同一个模板按当前实例自动取舍，而不是两套手工维护的页面。\n图 1：JVM 实例上会渲染 JVM 组件，因为它们的 visibleWhen 条件成立。\n图 2：同一个仪表盘打开在 Go 实例上，JVM 组件不存在，查询也没有执行。同一个模板，会按当前实例调整内容。\n指标值的显示格式 原始指标不一定适合直接给人看。Horizon 的组件会处理三类过去常让运维人员在脑子里换算的值：\nenum：用 value→label map 把编码型 gauge 变成文字。1/0 成功指标会显示成 OK / Failed，而不是裸数字。label 可以按 locale 翻译。 duration：以秒为单位的指标会显示成人能理解的时间差，比如 5m 20s ago；在坐标轴上会压缩成 5m / 2h。 SI suffixes：图表坐标轴和 tooltip 上的大数会显示成 45.1k、1.34M、2.5G，而不是 4.51e4。坐标轴刻度和 hover 值使用同一套写法。 图 3：字节数和计数这类大数序列使用紧凑 SI 后缀，坐标轴和 tooltip 保持一致。\n图 4：enum 和 duration 格式在 BanyanDB 生命周期卡片上的效果：OK 代替 1，\u0026ldquo;5m 20s ago\u0026rdquo; 代替秒数。\n按同一条时间线阅读整页图表 页面上所有 line 图共享 同一个 hover 游标。指向吞吐图上的第 32 分钟，时延图、错误率图和每个 sparkline 格子上的第 32 分钟也会一起亮起。这个约定在图表封装层强制执行，任何组件都不能退出，所以页面读起来是一组围绕同一时刻协同的视图，而不是十几个互不相关的图。多序列 tooltip 是固定对齐的表格，展示每条序列的 title（不会显示原始 MQE），所有值在同一列右对齐。\n图 5：一个游标跨过页面上所有折线图，所以你可以在所有图上同时读同一瞬间。\n从慢记录直接打开 Trace record 组件，比如 Slow Statements、Slow Commands、Slow Database Statements，是采样记录列表。每一行如果带 trace id，行首会出现一个 jump-to-trace 图标，点击即可打开对应 Trace 的瀑布图。它按 trace id 解析，而不是按 Layer 解析，这点很重要：Virtual Database / Cache / MQ 服务上的 Slow Statements 属于另一个 Layer 上的 caller，虚拟目标 Layer 自己没有 traces 标签页，但跳转仍然能打开正确的 Trace。语句文本本身也支持 点击复制。\n图 6：从慢语句跳到执行它的 Trace。按 trace id 解析，所以即使虚拟 Layer 自己没有 traces 标签页也能工作。\n固定多个对象做对比 有时只看一个对象不够。Horizon 允许你 锁定多个服务、实例或 endpoint，甚至跨服务锁定，并在当前页面内直接比较。可以从选择器或 instance/endpoint 列表固定对象；当前正在查看的对象始终属于对比组，会标记为 CURRENT，并继续驱动顶部信息区。每个固定对象都有自己的颜色。之后每个组件都会就地对比：line 组件为每个对象叠加一条序列，card 为每个对象显示一行，top 和 record 组件增加按对象切换的标签页，table 增加 Entity 列。对比栏会一直保留这组对象，不受底层列表分页或当前查看对象变化影响；每个对象单独发起请求，所以一个对象响应慢，不会把其他对象拖成空白。\n图 7：锁定对象，甚至跨服务锁定，所有折线组件都会按颜色叠加；对比栏保持这组对象，CURRENT 对象仍然驱动顶部信息区。\n时间范围作用于整个仪表盘 顶部时间范围会驱动页面上的所有内容：顶部 KPI 区、组件主体，以及 BanyanDB 分层 hot/warm/cold 存储里和 Cold 标记相关的整段排查路径。过去落地页和拓扑路由固定在最近 60 分钟，所以选择 \u0026ldquo;12 days ago\u0026rdquo; 后仍然会悄悄展示近期数字；现在时间选择器在所有地方都生效。上游控制变化时，每个依赖它的格子会明确重置并显示 \u0026ldquo;Reading data\u0026hellip;\u0026rdquo; 提示，而不是在加载动画下面留着旧值。\n后续阅读 需要强调的是，这是一套系统。同样的五类组件、同样的 MQE、同样的条件开关，会渲染每个 Layer 的仪表盘：上面的 JVM 面板、BanyanDB 生命周期卡片、mesh 服务上的百分位时延，以及专门为 Envoy AI Gateway（token throughput、time-to-first-token）或 GenAI virtual layer（per-model estimated cost）设计的面板。不同 Layer 之间变化的是 MQE，不是机制。\n上面讲的是 查看 体验。每个组件的 MQE、visibleWhen 条件、格式，以及各作用域的网格，都可以从 Layer dashboards 管理界面编辑。但这个创作流程（draft → preview → publish，以及可内联/展开的 MQE 编辑器）会在后续文章里单独展开。字段级参考可以看文档里的 dashboard widgets 和 charts。\n下一篇讲拓扑与服务依赖：同一批观测数据，如何从图表变成服务关系图，并继续下钻到实例和 endpoint。\n","excerpt":"\u003cp\u003e\u003cem\u003e译自英文原文：\u003ca href=\"/blog/2026-06-21-horizon-ui-dashboards-and-mqe/\"\u003eMeet Horizon UI · 2/17: Dashboards That Adapt — MQE, Smart Widgets, and Numbers Humans Can …\u003c/a\u003e\u003c/em\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-06-21-horizon-ui-dashboards-and-mqe/","title":"认识 Horizon UI · 2/17：动态仪表盘、MQE 与指标格式化"},{"body":"译自英文原文：Meet Horizon UI · 3/17: Topology \u0026amp; Service Dependency。\n这是 Meet Horizon UI 系列的第三篇。第二篇讲的是如何从仪表盘数字里理解服务；这一篇讲如何把服务关系画成一张 地图：谁调用谁，调用量有多大，以及同一个逻辑服务在不同上报 Layer 中是什么样子。\nSkyWalking 拓扑背后的调用数据只有一份，但 Horizon 不只画一种图。它有每个 Layer 自己的服务地图，可以从单条调用下钻到背后的实例，可以看 endpoint 级依赖图，也可以用跨 Layer 叠加视图把一个服务在不同 Layer 中的形态连起来。它们共用同一套引擎，只是回答的问题不同。（Deployment 标签页，也就是单个集群服务 内部 实例的地图，以及 WebGL 3D 地图，都足够大，会在接下来的文章里单独讲。）\n一套拓扑引擎，适配不同 Layer 打开任意 Layer 的 Topology 标签页，你会看到一张从左到右排列的层次化服务地图：User 存在时位于最左侧，每个服务按调用深度落在不同列里；同一列内部保留图遍历时的顺序，所以主调用链可以自上而下读。每个服务都是一个 六边形节点，节点上展示的所有东西都来自这个 Layer 的配置，没有写死逻辑：\n六边形 边框 承载节点的 ring 指标，用类似 SLA 的健康色带展示（绿色 → 红色）； 组件 图标 放在六边形内部，和 Trace 瀑布图使用同一套图标，所以 PostgreSQL 节点像 PostgreSQL，Kafka 节点像 Kafka； 节点的主数字，也就是 center 指标，带单位显示在六边形 上方； 服务 名称 显示在节点 下方，secondary 指标（默认是时延）显示在名称下面； 每条 边 都带调用吞吐的 RPM 标记，使用服务端指标，缺失时回退到客户端指标。 这里需要强调一点：上面这些都只是 General Layer 的内置默认配置。节点的 center / ring / secondary 指标，以及边指标，都位于 Layer dashboards admin → Topology 作用域，本质上是带单位和角色的 MQE 表达式。所以你可以把任意槽位指向另一条指标，同一套引擎就会为另一个 Layer 画出不同地图，或者按你的方式重画当前 Layer。（这些选择会像其他模板内容一样，跟着 Layer template 的 export/import 走。）\n图 1：一套由模板驱动的拓扑引擎：带健康色带的六边形节点、带 RPM 标记的边、真实组件图标；图上的每个指标都按 Layer 配置。\n过滤拓扑噪声 真实拓扑通常很嘈杂。一张密集地图里会混入 OAP 没法完整识别的推测节点，比如裸露的 rcmd:80、未接入探针的地址。Horizon 的 Filter 控件可以关掉这些噪声，同时保留真实依赖。它会自动生成一个 按 Layer 分组 的过滤分面，展示方式和侧边栏一致。每一行都有 Layer 自己的图标和本地化名称，比如 Virtual Database、Java Agent，还会有一个 Others 分组，用来收纳 OAP 无法分类的节点，以及一个独立的 User 开关。取消勾选 Others 后，未接入探针的杂点和悬空边会消失，而数据库、缓存和队列（各自的 VIRTUAL_* 行）仍然留在图上。过滤在客户端执行，每次刷新都会重新推导分组选项，所以不会陈旧。\n图 2：快速降噪，去掉无法解析的 \u0026ldquo;Others\u0026rdquo; 节点，同时保留真实依赖。\n当某个 Layer 的服务属于 OAP service group，地图上的服务聚焦选择器也会按这些 group 分组。点击 group header 可以 批量选中或清空该组里的所有服务，所以你能一次聚焦一张繁忙地图中某个团队负责的那一片。\n图 3：支持 service group 的服务选择器，可以一次聚焦整个分组。\n从服务调用下钻到实例关系 服务到服务的边是一条聚合调用；它背后是真实实例之间的通信。点击地图上的一条调用并选择 Instance map →，Horizon 会画出这层关系：客户端服务的实例在左列，服务端服务的实例在右列，中间是实例级调用，并带有客户端→服务端方向动画。它复用服务地图上的所有能力：健康 ring 节点、每条调用的 client/server 指标侧栏、带 Open instance dashboard 的节点 popover，并且按 Layer 自己的词汇标注列名，比如 Kubernetes 上叫 Pods，data plane 上叫 Sidecars。两个服务选择器会根据关系联动：server 列表来自当前 client 的 callees，client 列表来自当前 server 的 callers；每次改动其中一个，另一个都会重新推导。\n图 4：从一条聚合调用下钻到背后的实例到实例流量。\n按 endpoint 追完整请求链 服务拓扑回答“哪些服务调用了这个服务”。API dependency 标签页回答更具体的问题：“哪些 endpoint 调用了这个 endpoint，它又调用了哪些 endpoint”。选择一个 endpoint 后，图会按方向分列：callers 在左，焦点 endpoint 在中间，callees 在右。它同样用 SLA 色边框、每条边上的 RPM 和时延，以及最重边标签。选中节点后会出现一个 + handle，点击后拉入 它自己 的 callers 和 callees。这样你可以一跳一跳追链路，不需要一次展开整张图。把节点拖开后，跳转链接（Open endpoint、Service →）会在新标签页打开，当前正在探索的图仍然保留。\n图 5：按 endpoint 一跳一跳走请求链，每条边都有时延。\n同一个服务在不同 Layer 中的形态 一个逻辑服务经常会同时通过多个 Layer 上报：General agent、Service Mesh sidecar、mesh data plane、Kubernetes pod。SkyWalking 从 OAP 10 开始就建模了这种跨 Layer 层级关系；Horizon 把这层关系放进了地图交互。选中一个节点后，它会按需探测 hierarchy。如果这个服务有跨 Layer 对应对象，节点上会贴一个小的 chevron-stack 标记。\n图 6：选中节点上的 chevron-stack 标记。它在选择时按需探测，只在服务存在跨 Layer 对应对象时显示。\n点击这个标记，拓扑会在 Smartscape 叠加视图下变暗：焦点节点在原位高亮重绘，它的对应对象按 OAP 的 Layer 顺序纵向展开，靠近请求入口的 Layer 在上，靠近基础设施的 Layer 在下。之后两步点击就能在对应 Layer 中打开任意对象，并完成预选。（叠加视图打开时自动刷新会暂停，避免内容在你眼前移动。）\n图 7：一个服务在所有上报 Layer 中的样子，用叠加视图展示跨 Layer 层级关系。\n后续阅读 这些地图上的每个指标、阈值和边权重，都位于 Layer template 的 topology 块里。换句话说，你会用和仪表盘一样的配置驱动方式调整它们，这也是后续文章的主题。字段参考可以看 layer-template 里的 topology 文档。\n下一篇转向服务内部：Deployment 标签页如何展示集群服务的实例关系，以及 BanyanDB 如何接入 SkyWalking 自观测。\n","excerpt":"\u003cp\u003e\u003cem\u003e译自英文原文：\u003ca href=\"/blog/2026-06-21-horizon-ui-topology-and-dependency/\"\u003eMeet Horizon UI · 3/17: Topology \u0026amp; Service Dependency\u003c/a\u003e。\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e这是 \u003ca href=\"/zh/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon UI\u003c/a\u003e 系列的第三篇。\u003ca href=\"/zh/2026-06-21-horizon-ui-dashboards-and-mqe/\"\u003e第二篇\u003c/a\u003e讲的是如何 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-06-21-horizon-ui-topology-and-dependency/","title":"认识 Horizon UI · 3/17：拓扑与服务依赖"},{"body":"译自英文原文：Meet Horizon UI · 4/17: The Deployment Tab \u0026amp; BanyanDB Self-Observability。\n这是 Meet Horizon UI 系列的第四篇。第三篇画的是服务 之间 的地图。这一篇把视角收回来，画一个集群服务 内部 的实例关系，并用它解决 SkyWalking 过去一直展示得不够清楚的对象：自己的存储引擎 BanyanDB。\nDeployment 标签页：查看服务内部实例关系 服务地图回答“谁调用这个服务”。新的按 Layer 配置的 Deployment 标签页回答另一个问题：“这个 服务是怎么部署的，它自己的实例之间又是怎么通信的？”选择一个服务后，这个标签页会把它的实例画成节点，并画出实例之间的调用关系。你已经在服务地图里见过的平移/缩放画布、健康 ring 节点、边流动动画和每条调用的指标侧栏都会复用，只是作用域收缩到单个服务内部。\n它不是一张普通节点图，关键在三点：\n实例渲染成六边形，并组合成 pod。 一个 pod 的 main container 是完整六边形，sibling containers 会作为更小的六边形贴在它边上。所以主进程和 sidecar 会作为一个单元呈现，跨 pod 的 sidecar 链接也会连到它真正所属的小六边形上。 Pod 按你选择的规则聚到带标签的分组框里。 规则可以是单个实例属性（role）、多个属性组合（比如 node_role + node_type），也可以是名称正则。这样一组混合角色节点会按角色分框展示，而不是堆成一团。 布局是分层的。 每个 cluster box 会按调用深度摆放 pod：source 在左，被调用对象在右，所以 upstream→downstream 链条从左到右就能看清；拖动任意 pod 后，它所属的 box 会重新流式布局，保证内容仍在分组框内。 边按 (source-role → target-role) pair 区分，所以每类链接展示自己的指标，而不是所有边共用一组指标。主指标会直接印在边上，完整指标则进入 Flows 子标签页，每种 role-pair 对应一张对齐表。它默认关闭，和服务地图一样，完全从 Layer dashboards admin → Deployment 作用域配置。\n把 BanyanDB 纳入 SkyWalking 自观测 这套机制不是为了多画一种图，它首先服务于一个具体场景。SkyWalking 的原生数据库 BanyanDB 是一个集群化、按角色和 tier 组织的系统，而过去 SkyWalking 很难把它作为一个整体观测清楚。新的 BanyanDB Layer 位于 Self-Observability 下，并配合 OAP 后端 SWIP-15，把通过 BanyanDB FODC proxy 抓到的指标建模为整个部署：\n整个 cluster 是一个 Cluster（service）； 每个 container 是一个 Container（instance），并携带 container_name role 和 node_type tier 属性； 每个存储 Group 是一个 endpoint。 所以其他 Layer 中通用的 Service / Instance / Endpoint 导航结构，在 BanyanDB 这里就变成 Cluster / Container / Group。基于这套模型，Deployment 标签页可以直接画出数据库集群自身。\n图 1：SkyWalking 观测自己的数据库：BanyanDB 集群按角色和 tier 绘制，并展示 pod 之间的 liaison→data 与 lifecycle→data 边。\nCluster、Container、Group 每个作用域都有专门设计的仪表盘：\nCluster 仪表盘是整个数据库的总览视图：write / query / error-rate KPI、CPU / memory / disk capacity、吞吐和错误趋势，以及 Containers by Role 表。 Container 仪表盘会根据选中容器的 role 调整内容。每个容器都有 CPU / memory / Go runtime 资源面板；liaison 会增加 ingestion、query、gRPC errors、tier-2 publish pipeline 和 write-queue depth；data 节点会增加 storage totals、merge / compaction、inverted index、subscribe queue 和 retention；lifecycle sidecar 会显示 migration cycles 和 last-run time / status。角色专属面板按容器 role 属性开关，所以你只会看到当前节点适用的内容。 Group 仪表盘按 data model 拆分：measure、stream、trace、property。因为一个 BanyanDB group 只存储一个 catalog，只有匹配该模型的面板会渲染：measure group 显示 write-rate / query-latency / merge 面板，property group 显示 index-write / term-search / series 面板，以此类推。 图 2：Cluster 作用域，一眼看完整个数据库，并按角色列出所有容器。 把 同一个 Container 仪表盘分别打开到两个不同 role 上，最容易看出 role-gating 的效果：\n图 3：data/liaison Container，在共享资源面板之上展示 ingestion、query、storage 和 compaction 面板。 图 4：同一个仪表盘打开在 lifecycle 节点上，只显示 migration 面板。同一模板，按 role 开关。 Group 作用域则让每个 storage catalog 都有独立页面：\n图 5：Group 作用域，一次查看一个 storage catalog，面板按 group 的 data model 开关。 Role pair 边与 Flows 表 Deployment 标签页中，容器之间的调用边会从 SWIP-15 instance-relation families 中拿到 role-pair-specific 指标：liaison → data 边展示 write / query / part-sync 吞吐和 p99；liaison → liaison 边展示 write-forward 和 control；lifecycle → data 边展示 tier-migration volume / rate / p99。每条边最多内联显示三项属于该 pair 的指标，选中边面板保留完整 client-vs-server 拆分，Flows 子标签页则把每条边按 role-pair 展开成一组对齐表。\n图 6：Flows，把这些 role-pair 边显示成可排序表格，每个 pair 一个区块。\n这里有两个前提。边和角色专属面板假设你运行的是一个真实的 集群化 BanyanDB；单进程 standalone 实例只会显示共享资源和 Go runtime 面板，其余面板会在集群角色开始上报后亮起来。尤其是 container-to-container 边，还需要 OAP 构建暴露 SERVICE_INSTANCE_RELATION 作用域；在那之前，Deployment 标签页仍然会画出完整清单，只是 pod 之间没有边。\n由配置生成，而不是写死在页面里 上面这些不是一张手工写死的 \u0026ldquo;BanyanDB 页面\u0026rdquo;。聚类规则、每个 role 的节点指标、role-pair 边指标，都在 Layer template 中作为一个自包含块存在，可以从 Layer dashboards admin → Deployment 作用域编辑，并随模板 export/import 一起携带。它和其他 Layer 背后的配置驱动模型一样，后续文章会完整展开。\nBanyanDB Layer 自身的字段——Cluster、Container、Group 和 Deployment 视图——可以参考 BanyanDB dashboards 文档。\n下一篇看 3D Infrastructure Map：把所有 Layer 的服务放进一张 WebGL 地图，从全局看部署状态。\n","excerpt":"\u003cp\u003e\u003cem\u003e译自英文原文：\u003ca href=\"/blog/2026-06-21-horizon-ui-deployment-and-banyandb/\"\u003eMeet Horizon UI · 4/17: The Deployment Tab \u0026amp; BanyanDB Self-Observability\u003c/a\u003e。\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e这是 \u003ca href=\"/zh/2026-06-21-skywalking-horizon-ui-introduction/\"\u003eMeet Horizon …\u003c/a\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-06-21-horizon-ui-deployment-and-banyandb/","title":"认识 Horizon UI · 4/17：Deployment 标签页与 BanyanDB 自观测"},{"body":"SkyWalking BanyanDB 0.10.3 is released. Go to downloads page to find release tars.\nBug Fixes Persist segment end time in per-segment metadata so boundaries don\u0026rsquo;t shift across restarts or config changes. Fix flaky on-disk integration tests caused by Ginkgo v2 random container shuffling closing gRPC connections prematurely. ui: fix query editor refresh/reset behavior and BydbQL keyword highlighting. Fix flaky file_snapshot subtest in measure/stream/trace by waiting until every introduced mem part has been flushed to disk, instead of only checking the latest snapshot creator. Fix flaky TestCollectWithPartialClosedSegments by raising SegmentIdleTimeout so wall-clock variance on slow CI does not mark still-open segments as idle. Fix FODC lifecycle cache poisoning where transient InspectAll failures were cached for 10 minutes and masked liaison recovery; raise FODC agent and proxy timeouts from 10s to 40s. Fix FODC /cluster/lifecycle dropping zero-valued group fields (e.g. replicas=0, close=false) under encoding/json + omitempty; switch to protojson so all fields are emitted (nil nested messages serialize as null). Fix trace block_writer panic on out-of-order timestamps within the same traceID, which dropped one trace-write batch per panic in multi-agent SkyWalking deployments. Spans of a single trace originate from independently-clocked services, and trace storage is organized by traceID rather than timestamp, so per-traceID timestamp monotonicity is not a writer invariant. Fix nil-pointer panic on cold-tier data nodes when FODC InspectAll raced with idle-segment cleanup. Add GroupLifecycleInfo.errors to surface per-group collection failures from FODC InspectAll instead of silently dropping the affected node entry. Fix CollectDataInfo and CollectLiaisonInfo not handling CATALOG_PROPERTY groups. Close BanyanDB merge write-path durability gap that allowed torn parts to be created by a crash between data write and metadata commit. Metadata files (metadata.json for trace/measure/stream, manifest.json for sidx, plus traceID.filter and tag.type) now go through a new WriteAtomic (write-tmp + fsync + rename + fsync-dir) sequence; data writers (seqWriter.Close, localFileSystem.Write) now propagate fdatasync errors instead of silently dropping them. mustOpenFilePart / mustOpenPart in each engine cleans up safe post-rename .tmp leftovers on open. (#13862, root cause for #13861) Fix lifecycle migration where the receiving node could create segments shorter than the configured SegmentInterval. Fail fast on incompatible storage version at boot. Previously the server would start in a degraded SERVING state with affected groups un-loaded because the property schema-registry retry loop swallowed the version-incompatibility panic. Compatible versions are listed in banyand/internal/storage/versions.yml. Release bluge index writers on segment rotation so analysisWorker pools sized from GOMAXPROCS don\u0026rsquo;t accumulate across rotations. Two layered defects kept the existing idle-segment reclaim path from running: segmentIdleTimeout defaulted to 0 (which disabled the 10-minute reclaim ticker), and incRef refreshed lastAccessed on every rotation tick so closeIdleSegments never observed an idle segment. Defaults to time.Hour, moves the lastAccessed bump to real read/write call sites, and rewrites closeIdleSegments to take its own CAS-bumped snapshot so a concurrent reopen cannot have its only ref dropped under the reclaimer (apache/skywalking#13874). Fix incorrect counts and missing trace fields in the lifecycle migration report. Fix trace query identity-tag projection: when trace_id/span_id are explicitly projected, reconstruct them from span identity at response build time instead of requesting them as stored tags, and preserve tag order with null-filled per-span value alignment in the distributed trace result iterator. Fix FODC proxy corrupting Prometheus metric types. The agent dropped the # TYPE line while parsing banyandb /metrics, the StreamMetrics proto carried no type field, and the proxy guessed the type from a name-suffix heuristic — downgrading counters to gauge, mislabeling _count-suffixed counters as histograms, and splitting summaries into two conflicting # TYPE lines. Capture the type with the Prometheus expfmt parser, store it in the flight recorder, thread it through a new Metric.type enum over gRPC, and emit the real type from the proxy; pre-upgrade (untyped) samples fold into the matching typed family so a mixed-version rollout never emits two conflicting # TYPE lines for one metric. Trace storage metrics now expose the storage sub-scope, matching the stream_storage_* naming. The StorageMetricsFactory for trace switched from the root trace scope to trace.storage, so per-segment inverted-index metrics (inverted_index_total_updates, inverted_index_total_doc_count, inverted_index_total_term_searchers_started) are now emitted as banyandb_trace_storage_* instead of banyandb_trace_*, aligning the dashboard query names. Other trace metrics (trace_tst_*, trace_scheduler_*) are unchanged. Fix FODC agent labeling metrics with node_role=\u0026quot;ROLE_UNSPECIFIED\u0026quot;. The agent resolved the node role exactly once at startup via a single GetCurrentNode poll whose endpoint retries spanned only ~1s; when the sibling lifecycle/banyandb gRPC server was not yet listening (connect: cannot assign requested address) the role fell back to ROLE_UNSPECIFIED permanently, so most nodes never reported their real ROLE_DATA/ROLE_LIAISON. Retry the initial node-role resolution with exponential backoff until a non-unspecified role is obtained or a 25s budget elapses. Chores Regenerate expired TLS test certificate with 100-year validity. Set Ginkgo --repeat to 0 in the flaky-test workflow so the hourly run completes within the 50-minute timeout. ","excerpt":"\u003cp\u003eSkyWalking BanyanDB 0.10.3 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"bug-fixes\"\u003eBug Fixes …\u003c/h3\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-0-10-3/","title":"Release Apache SkyWalking BanyanDB 0.10.3"},{"body":"SkyWalking Horizon UI 0.6.0 is released. Go to downloads page to find release tars.\nThis release is the production-readiness pass for Horizon UI: every page now renders correctly across the eight supported languages on non-UTC OAP deployments, with deliberate caps and validation on the load surfaces operators reach.\nEight-locale internationalization Eight first-class UI languages — English (source) plus zh-CN, ja, ko, es, pt, de, fr — selectable from the top-bar locale chip on every page (including the pre-auth login), persisted per device. Every routed page and shared sub-component renders through vue-i18n; missing leaves fall back to English so partial catalogs degrade invisibly. All 42 layer dashboards and both overview dashboards carry per-locale overlay catalogs (~2,300 translatable leaves per non-English locale). The BFF picks the locale from the request\u0026rsquo;s X-Horizon-Locale header, merges the overlay onto the source, and serves the localised template — translation resolves once on the BFF, never on every chart mount. A new admin surface (Dashboard setup → Translations) edits the per-locale overlays through the live preview: pick a language, click any widget, type the translation. Per-locale status chips show which dashboards have drafts, are synced, diverge from disk, or are empty. Product, project and protocol names (SkyWalking, Kubernetes, OAP, MQE, eBPF, Zipkin, OpenTelemetry, Istio, GraphQL…), OAP scope enums, layer keys, MQE function names, env vars, HTTP status codes and runtimes stay verbatim in every locale; OAP-supplied data (service names, alarm rule names, span operations, log messages) is never translated. The i18n:validate gate is stricter: every source template must have a sibling overlay per advertised locale, and empty {} overlays are now a finding. 3D Infrastructure Map A standalone bird\u0026rsquo;s-eye view of the deployment at /3d/map: services render as cubes on stacked tier-planes (apps · service mesh · middleware · infra), each tier subdivided into per-layer zones. Drag to rotate, scroll to zoom, arrow keys / WASD to pan; click a cube for its detail card and a link into that layer\u0026rsquo;s dashboard. Live data windows: the map auto-refreshes every minute — per-cube traffic rolls up the last 2h of metrics and alarmed services light up from the last 20m of alarms. The deployment structure is read live from OAP (not a bundled snapshot), so the map is correct on any deployment. Beacon mode dims every healthy cube to a wireframe ghost so firing services jump out during an incident. Logic groups cluster related layers into one labelled block (a Self-Observability group ships by default). Call relationships animate directional particles so traffic direction reads at a glance. Tier order, per-layer plane mapping, cube colors, the traffic MQE per layer and the logic groups are all driven by a structured admin page at /admin/3d-map, published to OAP and shared across the deployment the same way as dashboards (local draft → Check diff \u0026amp; push). The per-layer service map gains a View in 3D link focused on just that layer. Smartscape service hierarchy OAP 10\u0026rsquo;s cross-layer service hierarchy is now reachable from any layer\u0026rsquo;s service map — a logical service projected across observation layers (GENERAL agent ↔ MESH sidecar ↔ MESH_DP data-plane ↔ K8S_SERVICE pod) is one click away on every selected hex. Picking a node lazily probes getServiceHierarchy; services with cross-layer peers get a chevron-stack chip. Clicking it opens a focus-and-context overlay with peers fanned by layer level; a two-step peer open arms then opens the destination layer in a new tab, pre-selecting the peer service. Every per-layer page validates the URL-hydrated ?service=\u0026lt;id\u0026gt; against the layer\u0026rsquo;s real service roster (served from the BFF\u0026rsquo;s 60s catalog cache), so deep links to low-traffic services resolve instead of silently swapping services or hanging on \u0026ldquo;Resolving service…\u0026rdquo;. On-demand pod logs \u0026amp; BanyanDB cold-stage A new per-layer Pod Logs tab live-tails a Kubernetes pod\u0026rsquo;s container logs, pulled on demand through OAP and never persisted: pick a pod and container, choose a trailing window (30s–30m) and refresh interval (2s–30s), with Include / Exclude keyword filtering forwarded to OAP. Enabled on the Kubernetes-deployed layers (K8S_SERVICE, MESH, MESH_DP). A topbar Cold pill (BanyanDB only) switches every page to read from the cold lifecycle stage — it replaces the read, not unions it. A cold-trap banner warns when the pill is on but the time range sits inside the hot+warm window. Trace lookup from a log row now carries the row\u0026rsquo;s timestamp so a trace living in cold resolves from a cold-era log row. Dashboards, templates \u0026amp; authoring The Layer dashboards and Overview templates admin pages share one editing model: your work-in-progress lives in your browser (\u0026ldquo;Save (local)\u0026rdquo; never touches the server), and the live shared version is whatever OAP serves. A Reset to ▾ control loads Bundled or Remote into the editor; a Preview ▾ control opens the real page rendering Local / Bundled / Remote; Check diff \u0026amp; push publishes with a side-by-side diff, enabled only when the local draft actually differs from remote. \u0026ldquo;+ New dashboard\u0026rdquo; writes a local draft; Delete = soft-disable (OAP has no hard delete). The Overview templates editor is rebuilt as a layer-style 12-column drag-to-reorder canvas that mirrors the live grid; the Layer dashboards picker is a single filterable dropdown with alias + key + sync status and a live menu preview. A new table widget for label-dimensioned metrics; sync-status banners now count source rows only (per-locale translation rows no longer inflate the remote-only / diverged counts). Time picker drives every page The global time picker now drives layer dashboards, overview widgets (Services Dashboard), landing and topology — previously hardcoded to the last 60 minutes. Custom range seeds from the last applied range on reopen; the Active alarms widget title shows the actual window. Locale-bleed fixes on the alarms custom-range stamp and the log date column (5月08日 → uniform MM-DD). Wire-correctness on non-UTC OAP Every BFF query route now spells Duration.start / end in the OAP server\u0026rsquo;s timezone (probed once per minute, cached). Previously only the alarms route did this; dashboards / landing / topology / endpoint / instance / eBPF / traces / logs all emitted UTC, silently shifting every query on non-UTC installs by the server\u0026rsquo;s offset. Traces and logs now query at SECOND precision (records, not metric buckets) so a just-finished trace falls inside the window instead of being rounded off the MINUTE boundary. Performance hardening Per-layer landing batches no longer 5xx on wide layers — requests are chunked at 6 services per round-trip and fired in parallel instead of one GraphQL with up to 250 aliased fragments tripping OAP\u0026rsquo;s complexity ceiling. Trace waterfalls open fast on huge traces (rows render lazily via content-visibility; a 5000-span trace no longer freezes the main thread). Backgrounded tabs stop polling and resume with one immediate tick on return. RBAC \u0026amp; input-validation hardening /api/health no longer leaks the active session count to unauthenticated callers — the public liveness probe returns only status + version. pageSize is capped server-side on every trace (200) / log (100) route, defending the OAP storage LIMIT at the BFF boundary. Profiling task bodies (async-profiler, pprof, eBPF fixed-task, network) are sanitized and bounded — duration caps, target/event caps, payload clamps — closing a DoS vector for users with profile:enable. Alerting rules \u0026amp; live debugger The Operate › Alerting rules Currently watching list now aggregates entities across the whole cluster and tags each row with the OAP node evaluating it. Clicking a watched entity opens a running-context popup: current state (FIRING / SILENCED_FIRING / RECOVERY_OBSERVATION), window size, silence/recovery countdowns, last-alarm time, and the per-metric snapshot rendered as a sparkline so you can see exactly why a rule is (or isn\u0026rsquo;t) firing. Live debugger fixes: ?historyId=… deep-links no longer render blank (TDZ ReferenceError in loadHistorical); captured records are no longer wiped on stop; tab buttons route to the correct /operate/live-debug/\u0026lt;tab\u0026gt; path; node cards keep a stable order; empty captures show an explicit placeholder. Smaller touches \u0026amp; reliability DSL / OAL catalog headers spell out the language name (Metrics Analysis Language - OpenTelemetry Rules, Log Analysis Language, Observability Analysis Language); the Layer dashboards / Overview editors now open REMOTE on diverged rows so you edit what users actually see. The four admin \u0026ldquo;Check diff \u0026amp; push\u0026rdquo; diff modals no longer log a Monaco resetSchema console error. Inter + JetBrains Mono are now self-hosted (no Google Fonts CDN), so air-gapped deployments render the intended typography; one six-step typescale now spans every admin page. The main sidebar folds to a narrow rail; overview dashboards appear in the sidebar only when their layers are reporting services; root / cascades through a sensible chain so the user never sees a blank page. When the BFF is unreachable the UI shows a clear \u0026ldquo;Cannot reach the server\u0026rdquo; message instead of \u0026ldquo;body stream already read\u0026rdquo;. A server-global service-by-layer catalog singleton (60s TTL + single-flight) caps OAP to one fan-out per minute regardless of how many routes poll. Aligned with upstream skywalking#13884 — Horizon now sends id = \u0026lt;envelope name\u0026gt; on POST /ui-management/templates, working against both current and legacy OAP. Full release notes are here.\n","excerpt":"\u003cp\u003eSkyWalking Horizon UI 0.6.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cp\u003eThis release is …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-horizon-ui-0-6-0/","title":"Release Apache SkyWalking Horizon UI 0.6.0"},{"body":"SkyWalking Cloud on Kubernetes 0.10.0 is released. Go to downloads page to find release tars.\n0.10.0 Features Add the CRD and Controller for the SkyWalking Event Exporter. Support Horizon UI in the UI CRD via the spec.kind discriminator. Documentation Add documentation for the Event Exporter. ","excerpt":"\u003cp\u003eSkyWalking Cloud on Kubernetes 0.10.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"0100\"\u003e0.10.0 …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cloud-on-kubernetes-0-10-0/","title":"Release Apache SkyWalking Cloud on Kubernetes 0.10.0"},{"body":"SkyWalking Horizon UI 0.5.0 is released. Go to downloads page to find release tars.\nFirst Apache-style release cut from this repo: source + binary tarballs, GPG-signed and SHA-512 checksummed, with a self-contained binary that boots via node server.js and no pnpm install step. Binary distribution ships a regenerated LICENSE + NOTICE that enumerate every bundled third-party package — produced by scripts/collect-dist-licenses.mjs during packaging and validated against a deny-list before signing.\nProfiling pprof (Go) profiling is fully wired: pick one event per task (CPU / HEAP / BLOCK / MUTEX / GOROUTINE / ALLOCS / THREADCREATE), with duration shown for CPU/BLOCK/MUTEX and a sampling-rate field for BLOCK/MUTEX. Create and analyze both match OAP\u0026rsquo;s single-event pprof schema. eBPF profiling gets a reworked process picker — click a row to expand its full attributes, selection lives on the checkbox, anchored pop-out — a refresh button on every task list, Intl-formatted times, and a hover-info frame on the flame graph. Flame-graph thrash on re-analyze is gone. The shared flame graph fixes \u0026ldquo;% of root\u0026rdquo; (it read a never-aggregated count), highlights the selected frame across all four profilers, and shows a single hover card (the library\u0026rsquo;s duplicate native tooltip is suppressed). After creating any profiling task (trace / async / eBPF / network / pprof) the list now polls up to 4× at 10s until the new task shows up, instead of leaving a stale pre-create list. Network profiling \u0026amp; process topology A booster-style honeycomb process topology: pods as hexagons, peers hugging the boundary, animated protocol-coloured edges (HTTP/TCP/TLS), a node pop-over, and a wide client | server edge-metric dashboard. Network task creation and the task-list query now use OAP\u0026rsquo;s schema field names. Platform monitoring (operate) Two new read-only operate pages: Data retention (TTL — getRecordsTTL / getMetricsTTL) and OAP configuration (the admin-port config dump, with OAP-masked secrets). Gated on new ttl:read / config:read verbs granted to maintainer and above. Data retention now loads on non-BanyanDB backends too (the metadata TTL field is optional). The operate sidebar now leads with a single Platform monitoring group (cluster status, data retention, OAP configuration) above the per-layer self-observability dashboards. Dashboards \u0026amp; templates The global time picker now drives dashboards. Layer dashboards query OAP at the picker\u0026rsquo;s window and precision (MINUTE / HOUR / DAY) instead of a fixed last-hour minute window, and line charts label the x-axis with real times per step (e.g. MM-DD for a 30-day view) rather than -Nm. New table widget for label-dimensioned metrics — pod phase per service, node condition, deployment replicas, etc. — rendered as one column per label (e.g. Condition | Node) instead of a scalar card or a misleading flat line. The K8S dashboards (and kong / mongodb / elasticsearch) now use it where upstream booster-ui does; widgets that were charting a single latest(…) value as a line are now cards. The K8S Cluster view is realigned to the upstream layout (totals cards · resource lines · status tables). Edit locally, publish on your terms. Saving a dashboard/overview template now writes the local bundled copy (so the edit renders immediately for preview) and marks it diverged — nothing reaches OAP until you press Sync all to OAP, which pushes only the templates that differ, behind a confirmation listing exactly what will be written. A post-save tip spells out that the change is local-only until published. Local-vs-remote, made explicit. When local edits diverge from OAP, a per-session prompt (by menu name, not file name) asks which to render — keep my local edits (preview) or use live (which overwrites the local copy with the remote version, confirmed). The layer-templates admin page carries the same Local/Remote display toggle next to Sync all, and a Diverged only filter; each diverged layer shows a yellow warning icon in the sidebar. Traces The native trace view auto-selects OAP\u0026rsquo;s trace-query API — queryTraces (whole trace inline, BanyanDB) vs queryBasicTraces (segment list + a per-trace fetch on click, every other backend) — and a banner states which is in use; in segment-list mode the list reads \u0026ldquo;Segments\u0026rdquo; and a click loads the full trace. Span kind (Entry / Exit / Local) renders as a colored word, not a filled pill. Auth, RBAC \u0026amp; resilience Every OAP call — GraphQL, admin REST, and Zipkin — now carries the configured basic-auth credentials, so a secured OAP no longer 401s pages. The sidebar is RBAC-gated by read verb, the Roles page shows a per-role menu-visibility matrix, and the Users page labels per-node \u0026ldquo;Active (24h)\u0026rdquo; / \u0026ldquo;Last seen\u0026rdquo; honestly (these are tracked per BFF replica, not cluster-wide). Routes are verb-gated, not just menus. A user without the required read verb is bounced from a restricted page (e.g. a viewer can no longer reach Cluster Status via the topbar OAP chip or a direct URL); the chip only links there for cluster:read. This sits on top of the existing per-route BFF verb enforcement. LDAP resolves group membership with the service account, not the logging-in user — directories that hide the group subtree from ordinary users no longer collapse every login to the fallback role. When OAP is unreachable the menu and admin loaders fall back to bundled templates, and non-JSON OAP responses surface a clear diagnostic. Smaller touches Top-N widgets get hover tooltips for long names and a title-bar pop-out to the full ranked list; redundant single-service name prefixes are dropped. The admin template-diff modal is a wide side-by-side view with labelled bundled-vs-OAP columns and an explanation of what the template drives; the layer-dashboards admin rail gains an in-page search. Per-layer alarm filtering uses the singular queryAlarms layer condition. The general layer drops networkProfiling, which is instance-scoped to k8s / mesh. Full release notes are here.\n","excerpt":"\u003cp\u003eSkyWalking Horizon UI 0.5.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cp\u003eFirst …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-horizon-ui-0-5-0/","title":"Release Apache SkyWalking Horizon UI 0.5.0"},{"body":"SkyWalking BanyanDB 0.10.2 is released. Go to downloads page to find release tars.\nBug Fixes Fix reuse of byte arrays in min/max implementation causing data corruption. Fix index-mode measure queries returning documents outside requested time range. Fix nil pointer panic in segment collectMetrics during shutdown. Fix property schema client connection instability after data node restart. Fix take snapshot error when no data in the segment. Fix(storage): disable rotation task on warm and cold lifecycle nodes. Fix(storage): prevent epoch segment creation from zero timestamps. Fix(sidx): use MinTimestamp/MaxTimestamp instead of SegmentID in streaming sync. Fix(handoff): prevent size limit bypass and sidx timestamp corruption in handoff replay. Fix(handoff): prevent enqueuing parts for online nodes via shared LocateAll. Fix wrong backup path of schema property. Fix OOM issue during migration when a group contains a large amount of data. Fix lifecycle migration failure when the target stage has close: true. Fix stale sync request blocking watch session channel. Fix nil pointer panic in disk monitor during early initialization. Fix FileSystemError not matching io/fs.ErrNotExist sentinel. Fix(topn): deduplicate entities in TopN aggregation query. Fix(stream): skip element-index visit when idx/ is absent. Fix(metadata): widen FODC inspection broadcast deadline and parallelize InspectAll. Fix(measure,stream,trace): eliminate flusher/introducer data race on shutdown. Fix: use topic instead of session_id as the Prometheus label. Fix(fodc): heal reconnect deadlock and add error message when no agents for lifecycle request. Fix(MCP): add explicit validation for properties and tools, and harden the server. Chores Upgrade Go and npm dependencies for CVE fixes. Bump ui and mcp npm dependencies for CVE fixes. ","excerpt":"\u003cp\u003eSkyWalking BanyanDB 0.10.2 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"bug-fixes\"\u003eBug Fixes\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eFix …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-0-10-2/","title":"Release Apache SkyWalking BanyanDB 0.10.2"},{"body":"SkyWalking Kubernetes Helm Chart 4.9.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed Update skywalking-banyandb-helm version to 0.6.0-rc0 by @hanahmily in https://github.com/apache/skywalking-helm/pull/172 Update banyandb version to 0.6.0-rc1 by @hanahmily in https://github.com/apache/skywalking-helm/pull/173 Bump up banyanDB helm version to 0.6.0-rc2 by @mrproliu in https://github.com/apache/skywalking-helm/pull/174 Update banyandb version to 0.6.0-rc3 by @hanahmily in https://github.com/apache/skywalking-helm/pull/175 Upgrade Elasticsearch dependency to ECK 8.18.8 by @wu-sheng in https://github.com/apache/skywalking-helm/pull/176 Fix SWCK oapserverconfig e2e version mismatch by @wu-sheng in https://github.com/apache/skywalking-helm/pull/177 Bump up banyanDB helm version by @mrproliu in https://github.com/apache/skywalking-helm/pull/178 Bump up banyanDB helm version to 0.6.0-rc5 by @mrproliu in https://github.com/apache/skywalking-helm/pull/179 Remove eck-operator hard dependency for non-ES storage backends by @wu-sheng in https://github.com/apache/skywalking-helm/pull/181 Bump up banyanDB helm version by @mrproliu in https://github.com/apache/skywalking-helm/pull/182 Bump up banyandb helm and remove etcd components in E2E by @mrproliu in https://github.com/apache/skywalking-helm/pull/184 Ready to release 4.9.0 by @mrproliu in https://github.com/apache/skywalking-helm/pull/185 Full Changelog: https://github.com/apache/skywalking-helm/compare/v4.8.0...v4.9.0\n","excerpt":"\u003cp\u003eSkyWalking Kubernetes Helm Chart 4.9.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars. …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-kubernetes-helm-chart-4.9.0/","title":"Release Apache SkyWalking Kubernetes Helm Chart 4.9.0"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/tags/agents/","title":"Agents"},{"body":"SkyAPM/mini-program-monitor has joined the SkyWalking ecosystem.\nThe project is a monitoring agent for WeChat (微信) and Alipay (支付宝) Mini Programs that reports telemetry to Apache SkyWalking via OTLP and SkyWalking native protocols. It extends SkyWalking\u0026rsquo;s end-user experience monitoring to the mini program platforms, which are a major part of the mobile experience in China.\nSignals Error tracking — JS errors, unhandled promise rejections, and page-not-found events, reported as OTLP logs with OpenTelemetry semantic conventions (exception.type, exception.stacktrace). Performance metrics — app launch, first render, first paint, route navigation, script execution, and sub-package load, reported as OTLP gauge metrics. Request metrics — wx.request / my.request, plus downloadFile / uploadFile, reported as an OTLP delta histogram bucketed per flush interval. Failed requests (4xx/5xx/timeout) also emit error logs. Distributed tracing (opt-in) — sw8 header propagation across outgoing requests, reported as SkyWalking SegmentObject to /v3/segments. Platform-aware backend Every signal carries miniprogram.platform: wechat | alipay (resource attribute on OTLP, span tag on segments), and each platform has its own SkyWalking component ID (WeChat = 10002, Alipay = 10003). Operators with one WeChat and one Alipay app against the same backend can slice by platform without forcing distinct service.names.\nCompatibility WeChat base library ≥ 2.11 Alipay base library ≥ 2.0 Apache SkyWalking OAP ≥ 10.x with the OTLP HTTP receiver Any other OTLP-compatible backend (OpenTelemetry Collector, Grafana, etc.) OTLP wire format is protobuf by default, with JSON available for debugging. Unsent events are persisted to storage on app hide and restored on next launch. The package has no runtime dependencies.\nSee the project README for installation, configuration, and a make preview target that boots OAP, UI, and both platform simulators for a hands-on demo.\n","excerpt":"\u003cp\u003e\u003ca href=\"https://github.com/SkyAPM/mini-program-monitor\"\u003eSkyAPM/mini-program-monitor\u003c/a\u003e has joined the SkyWalking ecosystem.\u003c/p\u003e\n\u003cp\u003eThe project is a monitoring agent …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/mini-program-monitor-joins-skywalking-ecosystem/","title":"Mini Program Monitor joins SkyWalking ecosystem"},{"body":"Mini programs are a major part of the mobile experience in China, but the open-source observability ecosystem has long focused on web browsers and native apps. SkyWalking already covers browser (client-js), iOS, and the server side; mini programs and Android were the remaining gaps. With SkyAPM/mini-program-monitor joining the SkyWalking ecosystem, the mini-program half of that gap is closed — one SDK supports both WeChat and Alipay, and the matching OAP-side component IDs, MAL rules, and UI templates are merged on main and will ship with 10.5.0.\nThis post is for teams that already run a SkyWalking backend and want to bring their mini programs into the same observability stack. The interesting parts aren\u0026rsquo;t that the project exists — they are how the data flows from a mini program to a SkyWalking dashboard, how the two platforms coexist, and what design trade-offs you should know about before rolling this out.\nData path The SDK uses two protocols:\nOTLP HTTP (error logs, performance metrics, request metrics) → OAP /v1/logs, /v1/metrics SkyWalking native (distributed tracing segments, optional) → OAP /v3/segments Why not a single protocol? OTLP already covers logs and metrics, so there\u0026rsquo;s no point reinventing native endpoints for those. But for tracing, OAP\u0026rsquo;s native SegmentObject maps more cleanly onto SkyWalking\u0026rsquo;s trace model, and sw8 header propagation to the backend works without any conversion. So traces go native, everything else goes OTLP, and neither side has to translate.\nOTLP defaults to protobuf; JSON is available for debugging. The SDK has zero runtime dependencies.\nTwo platforms, two independent Layers and dashboards Many teams maintain a WeChat mini program and an Alipay mini program against a shared backend. Rather than collapsing them into a single tagged service, the design promotes each platform to its own Layer — WECHAT_MINI_PROGRAM and ALIPAY_MINI_PROGRAM — with its own dashboard set. The SDK tags every signal with a resource attribute miniprogram.platform = wechat | alipay and assigns each platform its own component ID (WeChat = 10002, Alipay = 10003).\nOn the OAP side, the MAL rule\u0026rsquo;s filter routes data into the right Layer at ingest:\nmetricPrefix: meter_wechat_mp filter: \u0026#34;{ tags -\u0026gt; tags.miniprogram_platform == \u0026#39;wechat\u0026#39; }\u0026#34; The Alipay rule mirrors this with 'alipay'. The two rules are mutually exclusive — no double counting — and produce distinct metric prefixes (meter_wechat_mp_* vs meter_alipay_mp_*) that feed each Layer\u0026rsquo;s dashboards. Even when both platforms use the same service.name (e.g. mini-program-demo), the UI exposes two completely separate entry points.\nAsymmetric metric semantics This is the design choice I want to highlight. WeChat\u0026rsquo;s base library exposes PerformanceObserver, which gives you renderer-authoritative timings: app launch, first render, route navigation, script execution, sub-package load — all real measurements. Alipay\u0026rsquo;s base library doesn\u0026rsquo;t offer an equivalent, so the SDK falls back to lifecycle hooks: the App.onLaunch → App.onShow delta is used as an approximation of launch time, and renderer-level timings simply aren\u0026rsquo;t available.\nSo the two MAL rule sets are deliberately not the same:\nWeChat: app_launch_duration, first_render_duration, route_duration, script_duration, package_load_duration, request_duration_percentile, request_cpm Alipay: app_launch_duration, first_render_duration, request_duration_percentile, request_cpm The Alipay app_launch_duration is a lifecycle approximation and is not directly comparable to WeChat\u0026rsquo;s renderer timing — the dashboard tooltip says so explicitly. Putting the two numbers side by side is comparing two different measurement definitions.\nWhat the SDK does Four signals:\nErrors — JS exceptions, unhandled promise rejections, and pageNotFound go out as OTLP logs, following the OTel exception.* semantic conventions (exception.type, exception.stacktrace). Anything downstream that speaks OTLP — SkyWalking, OTel Collector, Grafana — recognizes them. Performance — the metrics listed above. OTLP gauge. Requests — wx.request / my.request / downloadFile / uploadFile are reported as OTLP delta histograms, one batch per flushInterval (default 5s). The le bucket labels are already in milliseconds, and the MAL rule explicitly declares MILLISECONDS to disable the default SECONDS→MS rescale. Failed requests (4xx / 5xx / timeout) additionally emit an error log so you can pivot from a dashboard to a concrete failure. Tracing (opt-in) — when enabled, outbound requests get sw8 header injection, and the resulting segments stitch together with backend traces into one end-to-end view. Trace data goes out as SkyWalking SegmentObject, not OTLP traces. Two reliability and cardinality details worth calling out:\nPersisting events on app hide. Mini programs get killed by the framework after some time in background, and weak networks make in-flight events easy to lose. The SDK writes unsent events to wx.setStorage / my.setStorage on onAppHide and restores them on the next launch.\nAvoiding cardinality explosions. Set serviceInstance to the app version (e.g. 1.4.2), not a device ID — at a million DAU the device-ID dimension blows up the OAP instance index. For request paths, the SDK exposes urlGroupRules regex patterns to fold parameterized URLs like /api/user/12345 into /api/user/{id} so the endpoint dimension doesn\u0026rsquo;t blow up either.\nWhat OAP needs If you\u0026rsquo;re on main or a release ≥ 10.5.0, the following are already shipped:\nconfig/component-libraries.yml registers WeChat-MiniProgram: 10002 and AliPay-MiniProgram: 10003 config/otel-rules/miniprogram/ holds four MAL rules — service-scoped and instance-scoped for each platform config/ui-initialized-templates/wechat_mini_program/ and alipay_mini_program/ carry root / service / instance / endpoint dashboards config/ui-initialized-templates/menu.yaml registers both layers under the Mobile menu group The only thing left is enabling the OTel receiver and giving the SDK an OTLP HTTP port it can reach. SkyWalking OAP binds its OTLP HTTP handler onto the receiver-sharing-server port, and that port defaults to 0 — meaning it\u0026rsquo;s folded into the core REST port (12800). If you want the SDK to use the standard OTLP HTTP port 4318, set the sharing port to 4318:\ndocker run -d --name sw-oap \\ -p 11800:11800 -p 12800:12800 -p 4318:4318 \\ -e SW_STORAGE=banyandb \\ -e SW_STORAGE_BANYANDB_TARGETS=banyandb:17912 \\ -e SW_OTEL_RECEIVER=default \\ -e SW_RECEIVER_SHARING_REST_PORT=4318 \\ apache/skywalking-oap-server:latest All receivers (OTLP, native segment, browser perf, log report) move to 4318 together, while GraphQL stays on 12800 for the UI.\nMinimal SDK config:\nimport MiniProgramMonitor from \u0026#39;mini-program-monitor\u0026#39;; MiniProgramMonitor.init({ service: \u0026#39;mini-program-demo\u0026#39;, serviceInstance: \u0026#39;1.4.2\u0026#39;, // Recommended: app version collector: \u0026#39;http://your-oap:4318\u0026#39;, enable: { error: true, perf: true, request: true, tracing: false, // Off by default; enable as needed }, }); WeChat and Alipay use the same config — the SDK detects the platform at runtime and tags the data accordingly.\nCompatibility WeChat base library ≥ 2.11 Alipay base library ≥ 2.0 Apache SkyWalking OAP main or ≥ 10.5.0, with the OTLP HTTP receiver enabled Any other OTLP-compatible backend (OpenTelemetry Collector, Grafana, etc.) also works, but you won\u0026rsquo;t get the SkyWalking-specific cross-platform dashboards What\u0026rsquo;s next To get involved, head over to SkyAPM/mini-program-monitor and open an issue or PR. The repo also ships a make preview target that boots OAP, the UI, and both platform simulators locally — handy if you want to play with it end-to-end.\nAndroid end-user experience monitoring is still a gap in the SkyWalking ecosystem; contributors interested in closing that one are very welcome.\n","excerpt":"\u003cp\u003eMini programs are a major part of the mobile experience in China, but the open-source observability …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-04-30-mini-program-monitoring-with-skywalking/","title":"Monitoring WeChat and Alipay Mini Programs with SkyWalking"},{"body":"小程序是国内移动端体验里绕不过去的一块，但开源监控生态长期偏向 Web 浏览器和原生 App。SkyWalking 自身已经覆盖了浏览器（client-js）、iOS、服务端，缺口主要在小程序和 Android。SkyAPM/mini-program-monitor 加入 SkyWalking 生态后，把这块缺口的小程序部分补上了 —— 一份 SDK 同时支持微信和支付宝，OAP 端的 component、MAL 规则、UI 模板已经合进 main 分支，会随 10.5.0 一起发布。\n这篇博客面向已经跑着 SkyWalking 后端、希望把小程序也接进来的团队。重点不是\u0026quot;项目存在\u0026quot;这件事，而是数据从小程序到 SkyWalking dashboard 走的是哪条路、双平台是怎么共存的、以及上线之前需要知道哪些设计取舍。\n数据通路 SDK 走两条腿：\nOTLP HTTP（错误日志、性能指标、请求指标）→ OAP 的 /v1/logs、/v1/metrics SkyWalking 原生协议（链路追踪 segment，可选）→ OAP 的 /v3/segments 为什么不是单协议？OTLP 已经覆盖了 logs 和 metrics 两类信号，没必要再造一份原生 endpoint；但分布式追踪上 OAP 的原生 SegmentObject 比 OTLP traces 表达力更贴 SkyWalking 自己的 trace 模型，且与服务端通过 sw8 header 透传时无需任何转换。所以追踪走原生，其它走 OTLP，两边都不绕路。\nOTLP 默认用 protobuf，调试时可切成 JSON。SDK 没有任何运行时依赖。\n双平台对应两个独立的 Layer 与监控面板 很多团队会同时维护一个微信小程序和一个支付宝小程序，业务逻辑共享一个后端。这套设计没有把它们塞进同一个 service 用 tag 区分，而是直接做成两个独立的 Layer：WECHAT_MINI_PROGRAM 和 ALIPAY_MINI_PROGRAM，对应两套独立的监控面板。SDK 在每个信号上打 resource 属性 miniprogram.platform = wechat | alipay，并给两端各分配独立的 component ID（微信 10002、支付宝 10003）。\nOAP 这一头是用 MAL 规则的 filter 把数据在 ingest 阶段就分流到对应 Layer 的：\nmetricPrefix: meter_wechat_mp filter: \u0026#34;{ tags -\u0026gt; tags.miniprogram_platform == \u0026#39;wechat\u0026#39; }\u0026#34; 支付宝那份规则同理过滤 alipay。两份规则互斥，不会重复计数；输出的 metric 前缀也不一样（meter_wechat_mp_* vs meter_alipay_mp_*），各自落在对应 Layer 的 dashboard 上。即使两端用同一个 service.name（比如都叫 mini-program-demo），UI 里也是两套完全独立的入口。\n不对等的指标语义 这是这套设计里我特别想强调的一处诚实选择。微信的基础库提供 PerformanceObserver，能拿到来自渲染层的权威时序：app launch、first render、route navigation、script execution、sub-package load 都是真实指标。支付宝的基础库不提供等价 API，SDK 只能用生命周期回退做近似：App.onLaunch → App.onShow 的 delta 当作启动时间，渲染相关的拿不到。\n所以两份 OAP 规则里的 metric 集合不对等：\n微信：app_launch_duration、first_render_duration、route_duration、script_duration、package_load_duration、request_duration_percentile、request_cpm 支付宝：app_launch_duration、first_render_duration、request_duration_percentile、request_cpm 支付宝侧的 app_launch_duration 是生命周期近似值，与微信的渲染层数值不可直接对比，这一点在 dashboard 的字段提示里也写明了。把两个数字放一起做横评等于在比较两种不同测量定义。\nSDK 端做了什么 四类信号：\n错误：JS 异常 / unhandled promise rejection / pageNotFound 走 OTLP logs，按 OTel exception.* 语义约定（exception.type、exception.stacktrace），下游不光 SkyWalking，OTel Collector / Grafana 也都认。 性能：上面那张表里那些。OTLP gauge。 请求：wx.request / my.request / downloadFile / uploadFile 都走 OTLP delta histogram，每个 flush 间隔（默认 5s）发一次增量。le 桶标签直接用 ms，OAP MAL 里显式声明 MILLISECONDS 阻止默认的 SECONDS→MS 缩放。失败请求（4xx/5xx/超时）额外发一条错误日志，方便从 dashboard 跳到具体错误。 追踪（可选）：开启后给出站请求注入 sw8 头，落到 OAP 后能与服务端 trace 拼成一条完整链路。trace 段以 SkyWalking SegmentObject 形式发出，不走 OTLP traces。 可靠性和基数控制的两个细节值得一提：\nApp hide 时落本地存储。小程序后台一段时间会被框架杀掉，弱网时也容易丢包。SDK 在 onAppHide 时把未发送的事件写到 wx.setStorage / my.setStorage，下次启动恢复并继续上报。\n反基数膨胀。强烈建议把 serviceInstance 设成应用版本号（如 1.4.2），不要用设备 ID —— 小程序日活百万级时设备 ID 维度直接把 OAP 的 instance 索引打爆。请求路径方面 SDK 提供 urlGroupRules 正则把 /api/user/12345 这类参数化路径归并到 /api/user/{id}，避免 endpoint 维度也膨胀。\nOAP 端要做什么 如果你用的是 main 分支或者 10.5.0 之后的发布版，下面这些已经内置：\nconfig/component-libraries.yml：注册了 WeChat-MiniProgram: 10002 和 AliPay-MiniProgram: 10003 config/otel-rules/miniprogram/：四份 MAL 规则，按 service / instance 维度分别定义 config/ui-initialized-templates/wechat_mini_program/ 和 alipay_mini_program/：root / service / instance / endpoint 四张 dashboard config/ui-initialized-templates/menu.yaml：把两个 layer 注册到 Mobile 菜单组下 唯一需要做的就是确认 OTel receiver 启用、给 OTLP HTTP 一个 SDK 能直连的端口。SkyWalking OAP 的 OTLP HTTP handler 默认绑在 receiver-sharing-server 的端口上，而该端口默认值是 0（即复用 core REST 端口 12800）。如果想让 SDK 用标准 OTLP HTTP 端口 4318，把 sharing 端口设到 4318：\ndocker run -d --name sw-oap \\ -p 11800:11800 -p 12800:12800 -p 4318:4318 \\ -e SW_STORAGE=banyandb \\ -e SW_STORAGE_BANYANDB_TARGETS=banyandb:17912 \\ -e SW_OTEL_RECEIVER=default \\ -e SW_RECEIVER_SHARING_REST_PORT=4318 \\ apache/skywalking-oap-server:latest 这样所有 receiver（OTLP + native segment + browser perf + log report）一起搬到 4318，GraphQL 仍在 12800 给 UI 用。\nSDK 端配置最小集：\nimport MiniProgramMonitor from \u0026#39;mini-program-monitor\u0026#39;; MiniProgramMonitor.init({ service: \u0026#39;mini-program-demo\u0026#39;, serviceInstance: \u0026#39;1.4.2\u0026#39;, // 推荐：应用版本号 collector: \u0026#39;http://your-oap:4318\u0026#39;, enable: { error: true, perf: true, request: true, tracing: false, // 默认关，按需开 }, }); 微信和支付宝两端配置一模一样，平台标签由 SDK 在运行时自动判定。\n兼容性 微信基础库 ≥ 2.11 支付宝基础库 ≥ 2.0 Apache SkyWalking OAP main 分支或 ≥ 10.5.0；OTLP HTTP receiver 启用即可 也可对接任意 OTLP 后端（OpenTelemetry Collector、Grafana 等），但那条路上拿不到 SkyWalking 专属的双平台 dashboard 后续 参与方式直接去 SkyAPM/mini-program-monitor 提 issue / PR。仓库里有一个 make preview 一键拉起 OAP、UI、两端模拟器的本地 demo 环境，想看效果可以直接跑。\nAndroid 端的端用户体验监控目前还是 SkyWalking 生态的空白，欢迎对这块感兴趣的同学一起补齐。\n","excerpt":"\u003cp\u003e小程序是国内移动端体验里绕不过去的一块，但开源监控生态长期偏向 Web 浏览器和原生 App。SkyWalking 自身已经覆盖了浏览器（client-js）、iOS、服务端，缺口主要在小程序和 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-04-30-mini-program-monitoring-with-skywalking/","title":"用 SkyWalking 监控微信和支付宝小程序"},{"body":"SkyWalking BanyanDB Helm 0.6.0 is released. Go to downloads page to find release tars.\nFeatures Support configure node discovery in cluster mode. Add the FODC Agent and Proxy components to the Helm chart. ","excerpt":"\u003cp\u003eSkyWalking BanyanDB Helm 0.6.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures …\u003c/h4\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-helm-0-6-0/","title":"Release Apache SkyWalking BanyanDB Helm 0.6.0"},{"body":"SkyWalking MCP 0.2.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed TLS certificate verification is now enforced for OAP connections. Added --sw-insecure flag to opt out for development and self-signed certs. Sensitive fields (authorization, password, token, secret) are redacted in --log-command output. Environment variable references in --sw-username and --sw-password now warn when the variable is not set, preventing silent unauthenticated requests. URL scheme validation now rejects non-HTTP and non-HTTPS OAP URLs. Regex patterns supplied to list_mqe_metrics are validated for complexity before compilation. Added --allowed-origins flag to sse and streamable transports for CORS origin enforcement. Increased reliability of core CLI commands through expanded automated test coverage. Removed an unused CLI tool and its associated parameter to simplify the interface and avoid confusion. Added validation for tool configuration properties, returning clear errors when required values are missing or invalid. Release Artifacts Source release: apache-skywalking-mcp-0.2.0-src.tgz Binary release: apache-skywalking-mcp-0.2.0-bin.tgz Docker image: apache/skywalking-mcp More Details Docs and tag: v0.2.0 ","excerpt":"\u003cp\u003eSkyWalking MCP 0.2.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-mcp-0-2-0/","title":"Release Apache SkyWalking MCP 0.2.0"},{"body":"SkyWalking GraalVM Distro 0.3.0 is released. Go to downloads page to find release tars.\nUpstream Sync Sync SkyWalking submodule to upstream v10.4.0 release tag. Add gen-ai-analyzer module: GenAI provider/model metrics from virtual-gen-ai.oal. Add Envoy AI Gateway MAL/LAL rules and config. Add TraceQL config properties: lookback, zipkinTracesListResultTags, skywalkingTracesListResultTags. GraalVM Native Image Compatibility Add library-server-for-graalvm: replace DynamicSslContext to use SslProvider.JDK instead of SslProvider.OPENSSL, enabling gRPC TLS in native images without netty_tcnative. Documentation Document TLS/SSL limitation: native image lacks netty_tcnative, recommend service mesh for mTLS. E2E Tests Add SSL e2e test case (gRPC TLS with JDK SSL provider in native image). Add mTLS e2e test case (mutual TLS with client certificates). Add RabbitMQ, RocketMQ, ActiveMQ, Pulsar, Kafka, Redis, MongoDB, Flink monitoring e2e test cases (OTEL metrics collection). Add AWS DynamoDB, S3, EKS, API Gateway e2e test cases (mock sender metrics). Add Auth e2e test case (token-based agent-to-OAP authentication). Add OTLP Traces e2e test case (OpenTelemetry trace ingestion via Zipkin API). Add Virtual MQ e2e test case (Kafka-instrumented virtual MQ layer metrics). Add Kafka Exporter e2e test case (trace and log export to Kafka). Add Virtual GenAI e2e test case (GenAI provider/model metrics via Spring AI + Java agent). Add Envoy AI Gateway e2e test case (ENVOY_AI_GATEWAY layer metrics/logs via OTLP). Add TraceQL SkyWalking e2e test case (Tempo API with SkyWalking native trace datasource). Add Envoy AI Gateway MAL comparison tests (34 tests for gateway-service and gateway-instance rules). Add Self-Observability e2e test case (OAP Prometheus telemetry via OTEL collector). Add MQE e2e test case (Metrics Query Engine expression evaluation with baseline). ","excerpt":"\u003cp\u003eSkyWalking GraalVM Distro 0.3.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads/#SkyWalkingGraalVMDistro\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"upstream-sync\"\u003eUpstream …\u003c/h3\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-graalvm-distro-0.3.0/","title":"Release Apache SkyWalking GraalVM Distro version 0.3.0"},{"body":"Query SkyWalking and Zipkin Traces with TraceQL and Visualize in Grafana Apache SkyWalking introduced TraceQL support in version 10.4.0, implementing Grafana Tempo\u0026rsquo;s HTTP query APIs so that Grafana can query and visualize traces stored in SkyWalking without any additional plugins. This means you can now use the familiar Grafana Tempo data source to search, filter, and drill into both SkyWalking native traces and Zipkin-compatible traces — all served by your existing SkyWalking OAP server.\nArchitecture Overview ┌────────────────────┐ Tempo HTTP API ┌─────────────────────────────┐ │ │ ──── /skywalking/api/search ──► │ SkyWalking Native Backend │ │ Grafana │ │ (Query Traces V2 API) │ │ (Tempo Data Src) │ ├─────────────────────────────┤ │ │ ──── /zipkin/api/search ──────► │ Zipkin-Compatible Backend │ └────────────────────┘ └──────────┬──────────────────┘ │ ┌──────────▼──────────────────┐ │ SkyWalking OAP Server │ │ ┌───────────────────────┐ │ │ │ TraceQL Service │ │ │ │ (port 3200) │ │ │ └───────────────────────┘ │ │ ┌───────────────────────┐ │ │ │ Storage (BanyanDB / │ │ │ │ Elasticsearch / …) │ │ │ └───────────────────────┘ │ └─────────────────────────────┘ The TraceQL Service sits inside the OAP server and exposes the Tempo-compatible HTTP API on port 3200 (default). It converts traces from their native format into Tempo\u0026rsquo;s format, where the trace detail part (Trace message) reuses OTLP Trace definitions.\nLimitations and Supported TraceQL Features TraceQL is a rich query language, but SkyWalking currently implements a practical subset. The following features are supported:\nFeature Examples Spanset filter {resource.service.name=\u0026quot;frontend\u0026quot;} Resource attributes resource.service.name Span attributes span.http.method, span.http.status_code Intrinsic fields duration, name, status Comparison operators =, \u0026gt;, \u0026gt;=, \u0026lt;, \u0026lt;= Compound conditions {resource.service.name=\u0026quot;frontend\u0026quot; \u0026amp;\u0026amp; duration\u0026gt;100ms} Duration units us/µs, ms, s, m, h The following features are not yet supported:\nSpanset logical operations ({...} AND {...}, {...} OR {...}) Pipeline operations (| operator) Aggregate functions (count(), avg(), max(), min(), sum()) Regular expression matching (=~, !~) event and link scopes kind intrinsic field Streaming mode (must be disabled in the Grafana Tempo data source settings) Important: SkyWalking native trace support in TraceQL is based on the Query Traces V2 API. Currently, only BanyanDB storage implements this API. Other storage backends (e.g., Elasticsearch, MySQL, PostgreSQL) do not support SkyWalking native trace queries via TraceQL. Zipkin-compatible traces are not subject to this restriction.\nTrace Format Conversion Since the trace detail part of Tempo\u0026rsquo;s format reuses OTLP Trace definitions, the conversion descriptions below refer to OTLP field names (e.g., span kind, status code).\nSkyWalking Native Trace Trace ID Encoding SkyWalking native trace IDs are arbitrary strings (e.g., 2a2e04e8d1114b14925c04a6321ca26c.38.17739924187687539), while Grafana Tempo requires pure hex-encoded trace IDs. The TraceQL Service encodes each UTF-8 byte of the original trace ID as two lowercase hex characters:\nOriginal: 2a2e04e8d1114b14925c04a6321ca26c.38.17739924187687539 Encoded: 32613265303465386431313134623134393235633034613633323163613236632e33382e3137373339393234313837363837353339 This encoded hex trace ID is what appears in all API responses and in Grafana. When you click a trace ID in Grafana, the TraceQL Service automatically decodes it back to the original SkyWalking trace ID for the internal query.\nSpan Kind Mapping SkyWalking Span Type OTLP Span Kind Entry SPAN_KIND_SERVER Exit SPAN_KIND_CLIENT Local SPAN_KIND_INTERNAL Status Mapping SkyWalking isError OTLP Status Code true STATUS_CODE_ERROR false STATUS_CODE_OK SpanAttachedEvents SkyWalking SpanAttachedEvents are converted to OTLP span events, with tags mapped as string attributes and summary mapped as numeric attributes (serialized as strings).\nZipkin Trace Span Kind Mapping Zipkin Span Kind OTLP Span Kind CLIENT SPAN_KIND_CLIENT SERVER SPAN_KIND_SERVER PRODUCER SPAN_KIND_PRODUCER CONSUMER SPAN_KIND_CONSUMER Status Mapping If the otel.status_code tag is present, it is used directly. Otherwise, if the error tag equals true, the status is STATUS_CODE_ERROR. If neither tag is present, the status defaults to STATUS_CODE_UNSET. Endpoint and Annotation Mapping Zipkin endpoint fields are mapped to OTLP attributes (e.g., localEndpoint.ipv4 → net.host.ip), and Zipkin annotations are converted to OTLP span events.\nFor the full conversion details, see the TraceQL Service documentation.\nHow to Enable TraceQL Step 1: Enable the TraceQL Module By default, the TraceQL module is disabled (selector: ${SW_TRACEQL:-}). To enable it, set the selector to default:\n# In application.yml traceQL: selector: ${SW_TRACEQL:default} default: enableDatasourceSkywalking: ${SW_TRACEQL_ENABLE_DATASOURCE_SKYWALKING:true} enableDatasourceZipkin: ${SW_TRACEQL_ENABLE_DATASOURCE_ZIPKIN:true} Or via environment variables:\nexport SW_TRACEQL=default export SW_TRACEQL_ENABLE_DATASOURCE_SKYWALKING=true export SW_TRACEQL_ENABLE_DATASOURCE_ZIPKIN=true Step 2: Enable the Zipkin Receiver (for Zipkin traces only) If you want to query Zipkin traces, you also need to enable the Zipkin receiver so that SkyWalking can ingest Zipkin trace data:\n# In application.yml receiver-zipkin: selector: ${SW_RECEIVER_ZIPKIN:default} default: searchableTracesTags: ${SW_ZIPKIN_SEARCHABLE_TAG_KEYS:http.method} sampleRate: ${SW_ZIPKIN_SAMPLE_RATE:10000} restHost: ${SW_RECEIVER_ZIPKIN_REST_HOST:0.0.0.0} restPort: ${SW_RECEIVER_ZIPKIN_REST_PORT:9411} Or via environment variable:\nexport SW_RECEIVER_ZIPKIN=default Full Configuration Reference For the complete list of all configuration options and their default values, see the Configuration section of the TraceQL Service documentation.\nConfiguring Grafana Tempo Data Source Prerequisite: Grafana 12 or later is required.\nEach trace backend (SkyWalking native / Zipkin) needs its own Tempo data source in Grafana, because each is served under a different context path.\nContext Paths The two backends are served under separate context paths on the same port:\nBackend Default Context Path Env Variable Full Default URL SkyWalking native /skywalking SW_TRACEQL_REST_CONTEXT_PATH_SKYWALKING http://\u0026lt;oap-host\u0026gt;:3200/skywalking Zipkin /zipkin SW_TRACEQL_REST_CONTEXT_PATH_ZIPKIN http://\u0026lt;oap-host\u0026gt;:3200/zipkin Setting Up the SkyWalking Data Source In Grafana, go to Configuration → Data Sources → Add data source. Choose Tempo. Set the URL to http://\u0026lt;oap-host\u0026gt;:3200/skywalking. Disable the Streaming option (SkyWalking does not support streaming mode). Save and test the data source. Setting Up the Zipkin Data Source Same as above, but set the URL to http://\u0026lt;oap-host\u0026gt;:3200/zipkin.\nConfiguring Trace List Result Tags When you search for traces in Grafana, the trace list panel shows a summary of each trace. The tracesListResultTags configuration controls which span tags are included in the search result and displayed as columns in the trace list.\nEnv Variable Default Value Purpose SW_TRACEQL_ZIPKIN_TRACES_LIST_RESULT_TAGS http.method,error Tags shown for Zipkin traces SW_TRACEQL_SKYWALKING_TRACES_LIST_RESULT_TAGS http.method,http.status_code,rpc.status_code,db.type,db.instance,mq.queue,mq.topic,mq.broker Tags shown for SkyWalking traces Note that service.name and span.kind are always included regardless of this setting.\nThese tags appear as attribute columns in the Grafana Tempo trace search results, making it easier to identify and group traces at a glance:\nSkyWalking native trace list:\nZipkin trace list:\nYou can customize these tags based on your application\u0026rsquo;s instrumentation. For example, if your services heavily use messaging, you might add mq.destination or messaging.system to the list.\nBuilding a Trace Dashboard in Grafana SkyWalking Native Trace Dashboard Step 1: Explore and Save Go to the Explore page in Grafana. Select the Tempo data source you configured for SkyWalking (e.g., SkyWalkingTraceQL). Run a test query, then click Add to dashboard and save it as SkyWalking Trace. Step 2: Configure Variables Add dashboard variables so users can filter traces dynamically (e.g., by service name):\nStep 3: Add a Trace Panel Choose a Table chart (or edit the panel you saved). Set Query type to Search. Set the Service Name query condition to the variable $Service. Add other query conditions as needed (e.g., duration, span name, tags). Test and save. Step 4: View Trace Details Click any trace ID in the trace panel to jump to the Explore page showing the full trace waterfall view with all spans, tags, and events:\nZipkin Trace Dashboard The setup for Zipkin traces is identical to SkyWalking native traces — just use the Zipkin Tempo data source you configured (e.g., ZipkinTraceQL).\nZipkin trace detail view:\nSummary With TraceQL support in SkyWalking 10.4.0, you can now leverage Grafana\u0026rsquo;s powerful Tempo data source to query and visualize both SkyWalking native traces and Zipkin-compatible traces. The key points to remember:\nEnable the TraceQL module by setting SW_TRACEQL=default and enabling the desired backends. Configure separate Tempo data sources in Grafana for each backend (/skywalking and /zipkin). Disable the Streaming option in the Grafana Tempo data source settings. Customize result tags via SW_TRACEQL_SKYWALKING_TRACES_LIST_RESULT_TAGS and SW_TRACEQL_ZIPKIN_TRACES_LIST_RESULT_TAGS to control what\u0026rsquo;s shown in search results. SkyWalking native trace queries require BanyanDB storage (Zipkin traces work with all storage backends). For the complete API reference and conversion details, see the TraceQL Service documentation. For Grafana integration details, see Use Grafana As The UI.\n","excerpt":"\u003ch1 id=\"query-skywalking-and-zipkin-traces-with-traceql-and-visualize-in-grafana\"\u003eQuery SkyWalking and Zipkin Traces with TraceQL and Visualize in Grafana\u003c/h1\u003e\n\u003cp\u003eApache SkyWalking …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-04-08-traceql/","title":"Query SkyWalking and Zipkin Traces with TraceQL and Visualize in Grafana"},{"body":"使用 TraceQL 查询 SkyWalking 和 Zipkin 链路追踪数据并在 Grafana 中可视化 Apache SkyWalking 在 10.4.0 版本中引入了 TraceQL 支持，实现了 Grafana Tempo 的 HTTP 查询 API，使 Grafana 无需任何额外插件即可查询和可视化 SkyWalking 中存储的链路追踪数据。 这意味着你现在可以使用熟悉的 Grafana Tempo 数据源来搜索、过滤和深入分析 SkyWalking 原生链路追踪和 Zipkin 兼容链路追踪 —— 所有数据都由现有的 SkyWalking OAP 服务器提供。\n架构概览 ┌────────────────────┐ Tempo HTTP API ┌─────────────────────────────┐ │ │ ──── /skywalking/api/search ──► │ SkyWalking Native Backend │ │ Grafana │ │ (Query Traces V2 API) │ │ (Tempo Data Src) │ ├─────────────────────────────┤ │ │ ──── /zipkin/api/search ──────► │ Zipkin-Compatible Backend │ └────────────────────┘ └──────────┬──────────────────┘ │ ┌──────────▼──────────────────┐ │ SkyWalking OAP Server │ │ ┌───────────────────────┐ │ │ │ TraceQL Service │ │ │ │ (port 3200) │ │ │ └───────────────────────┘ │ │ ┌───────────────────────┐ │ │ │ Storage (BanyanDB / │ │ │ │ Elasticsearch / …) │ │ │ └───────────────────────┘ │ └─────────────────────────────┘ TraceQL Service 位于 OAP 服务器内部，在端口 3200（默认）上暴露 Tempo 兼容的 HTTP API。 它将链路追踪数据从原生格式转换为 Tempo 的格式， 其中链路追踪详情部分（Trace 消息）复用了 OTLP Trace 定义。\n支持的 TraceQL 特性与限制 TraceQL 是一种功能丰富的查询语言，但 SkyWalking 目前实现了一个实用的子集。 以下特性已支持：\nFeature Examples Spanset filter {resource.service.name=\u0026quot;frontend\u0026quot;} Resource attributes resource.service.name Span attributes span.http.method, span.http.status_code Intrinsic fields duration, name, status Comparison operators =, \u0026gt;, \u0026gt;=, \u0026lt;, \u0026lt;= Compound conditions {resource.service.name=\u0026quot;frontend\u0026quot; \u0026amp;\u0026amp; duration\u0026gt;100ms} Duration units us/µs, ms, s, m, h 以下特性暂不支持：\nSpanset logical operations ({...} AND {...}, {...} OR {...}) Pipeline operations (| operator) Aggregate functions (count(), avg(), max(), min(), sum()) Regular expression matching (=~, !~) event and link scopes kind intrinsic field Streaming mode (must be disabled in the Grafana Tempo data source settings) 重要提示：TraceQL 中的 SkyWalking 原生链路追踪支持基于 Query Traces V2 API。 目前只有 BanyanDB 存储实现了该 API。其他存储后端 （如 Elasticsearch、MySQL、PostgreSQL）不支持通过 TraceQL 查询 SkyWalking 原生链路追踪数据。 Zipkin 兼容链路追踪不受此限制。\nTrace 格式转换 由于 Tempo 格式的链路追踪详情部分复用了 OTLP Trace 定义， 以下转换描述使用 OTLP 字段名称（如 span kind、status code）。\nSkyWalking 原生链路追踪 Trace ID 编码 SkyWalking 原生 trace ID 是任意字符串（例如 2a2e04e8d1114b14925c04a6321ca26c.38.17739924187687539），而 Grafana Tempo 要求 纯十六进制编码的 trace ID。TraceQL Service 将原始 trace ID 的每个 UTF-8 字节编码为两个小写十六进制字符：\n原始值: 2a2e04e8d1114b14925c04a6321ca26c.38.17739924187687539 编码后: 32613265303465386431313134623134393235633034613633323163613236632e33382e3137373339393234313837363837353339 编码后的十六进制 trace ID 会出现在所有 API 响应和 Grafana 中。当你在 Grafana 中点击 trace ID 时，TraceQL Service 会自动将其解码回原始的 SkyWalking trace ID 进行内部查询。\nSpan Kind 映射 SkyWalking Span Type OTLP Span Kind Entry SPAN_KIND_SERVER Exit SPAN_KIND_CLIENT Local SPAN_KIND_INTERNAL 状态映射 SkyWalking isError OTLP Status Code true STATUS_CODE_ERROR false STATUS_CODE_OK SpanAttachedEvents SkyWalking SpanAttachedEvents 被转换为 OTLP span events， 其中 tags 映射为字符串属性，summary 映射为数值属性（序列化为字符串）。\nZipkin 链路追踪 Span Kind 映射 Zipkin Span Kind OTLP Span Kind CLIENT SPAN_KIND_CLIENT SERVER SPAN_KIND_SERVER PRODUCER SPAN_KIND_PRODUCER CONSUMER SPAN_KIND_CONSUMER 状态映射 如果存在 otel.status_code 标签，则直接使用。 否则，如果 error 标签等于 true，则状态为 STATUS_CODE_ERROR。 如果以上标签都不存在，则状态默认为 STATUS_CODE_UNSET。 Endpoint 与 Annotation 映射 Zipkin endpoint 字段被映射为 OTLP 属性（例如 localEndpoint.ipv4 → net.host.ip）， Zipkin annotations 被转换为 OTLP span events。\n完整的转换详情请参阅 TraceQL Service 文档。\n如何启用 TraceQL 步骤 1：启用 TraceQL 模块 默认情况下，TraceQL 模块是禁用的（selector: ${SW_TRACEQL:-}）。要启用它，将 selector 设置为 default：\n# 在 application.yml 中 traceQL: selector: ${SW_TRACEQL:default} default: enableDatasourceSkywalking: ${SW_TRACEQL_ENABLE_DATASOURCE_SKYWALKING:true} enableDatasourceZipkin: ${SW_TRACEQL_ENABLE_DATASOURCE_ZIPKIN:true} 或通过环境变量设置：\nexport SW_TRACEQL=default export SW_TRACEQL_ENABLE_DATASOURCE_SKYWALKING=true export SW_TRACEQL_ENABLE_DATASOURCE_ZIPKIN=true 步骤 2：启用 Zipkin 接收器（仅用于 Zipkin 链路追踪） 如果你需要查询 Zipkin 链路追踪数据，还需要启用 Zipkin 接收器，以便 SkyWalking 能够接收 Zipkin 链路追踪数据：\n# 在 application.yml 中 receiver-zipkin: selector: ${SW_RECEIVER_ZIPKIN:default} default: searchableTracesTags: ${SW_ZIPKIN_SEARCHABLE_TAG_KEYS:http.method} sampleRate: ${SW_ZIPKIN_SAMPLE_RATE:10000} restHost: ${SW_RECEIVER_ZIPKIN_REST_HOST:0.0.0.0} restPort: ${SW_RECEIVER_ZIPKIN_REST_PORT:9411} 或通过环境变量设置：\nexport SW_RECEIVER_ZIPKIN=default 完整配置参考 所有配置选项及其默认值的完整列表，请参阅 TraceQL Service 文档的配置章节。\n配置 Grafana Tempo 数据源 前提条件：需要 Grafana 12 或更高版本。\n每个链路追踪后端（SkyWalking 原生 / Zipkin）需要在 Grafana 中配置各自独立的 Tempo 数据源， 因为它们分别在不同的上下文路径下提供服务。\n上下文路径 两个后端在同一端口上使用不同的上下文路径提供服务：\nBackend Default Context Path Env Variable Full Default URL SkyWalking native /skywalking SW_TRACEQL_REST_CONTEXT_PATH_SKYWALKING http://\u0026lt;oap-host\u0026gt;:3200/skywalking Zipkin /zipkin SW_TRACEQL_REST_CONTEXT_PATH_ZIPKIN http://\u0026lt;oap-host\u0026gt;:3200/zipkin 配置 SkyWalking 数据源 在 Grafana 中，前往 Configuration → Data Sources → Add data source。 选择 Tempo。 将 URL 设置为 http://\u0026lt;oap-host\u0026gt;:3200/skywalking。 禁用 Streaming 选项（SkyWalking 不支持流式模式）。 保存并测试数据源。 配置 Zipkin 数据源 与上述步骤相同，但将 URL 设置为 http://\u0026lt;oap-host\u0026gt;:3200/zipkin。\n配置链路追踪列表结果标签 在 Grafana 中搜索链路追踪时，链路追踪列表面板会显示每条追踪的摘要信息。 tracesListResultTags 配置控制哪些 span 标签会包含在搜索结果中并作为列显示在追踪列表中。\nEnv Variable Default Value Purpose SW_TRACEQL_ZIPKIN_TRACES_LIST_RESULT_TAGS http.method,error Tags shown for Zipkin traces SW_TRACEQL_SKYWALKING_TRACES_LIST_RESULT_TAGS http.method,http.status_code,rpc.status_code,db.type,db.instance,mq.queue,mq.topic,mq.broker Tags shown for SkyWalking traces 注意，无论此设置如何，service.name 和 span.kind 始终包含在结果中。\n这些标签在 Grafana Tempo 链路追踪搜索结果中显示为属性列，方便快速识别和分组追踪数据：\nSkyWalking 原生追踪列表：\nZipkin 追踪列表：\n你可以根据应用程序的埋点情况自定义这些标签。例如，如果你的服务大量使用消息队列， 可以在列表中添加 mq.destination 或 messaging.system。\n在 Grafana 中构建链路追踪仪表板 SkyWalking 原生追踪仪表板 步骤 1：探索并保存 前往 Grafana 的 Explore 页面。 选择你为 SkyWalking 配置的 Tempo 数据源（例如 SkyWalkingTraceQL）。 运行一个测试查询，然后点击 Add to dashboard 并保存为 SkyWalking Trace。 步骤 2：配置变量 添加仪表板变量，以便用户可以动态过滤追踪数据（例如按服务名称过滤）：\n步骤 3：添加追踪面板 选择 Table 图表（或编辑你保存的面板）。 将 Query type 设置为 Search。 将 Service Name 查询条件设置为变量 $Service。 根据需要添加其他查询条件（如 duration、span name、tags）。 测试并保存。 步骤 4：查看追踪详情 点击追踪面板中的任意 trace ID，即可跳转到 Explore 页面，查看完整的追踪瀑布图， 包含所有 span、标签和事件：\nZipkin 追踪仪表板 Zipkin 追踪的设置与 SkyWalking 原生追踪完全相同 —— 只需使用你配置的 Zipkin Tempo 数据源 （例如 ZipkinTraceQL）。\nZipkin 追踪详情视图：\n总结 通过 SkyWalking 10.4.0 中的 TraceQL 支持，你现在可以利用 Grafana 强大的 Tempo 数据源 来查询和可视化 SkyWalking 原生链路追踪和 Zipkin 兼容链路追踪数据。 需要记住的要点：\n启用 TraceQL 模块：设置 SW_TRACEQL=default 并启用所需的后端。 在 Grafana 中配置独立的 Tempo 数据源：为每个后端分别配置（/skywalking 和 /zipkin）。 禁用 Streaming 选项：在 Grafana Tempo 数据源设置中关闭流式模式。 自定义结果标签：通过 SW_TRACEQL_SKYWALKING_TRACES_LIST_RESULT_TAGS 和 SW_TRACEQL_ZIPKIN_TRACES_LIST_RESULT_TAGS 控制搜索结果中显示的内容。 SkyWalking 原生追踪查询需要 BanyanDB 存储（Zipkin 追踪支持所有存储后端）。 完整的 API 参考和转换详情，请参阅 TraceQL Service 文档。 Grafana 集成详情，请参阅 使用 Grafana 作为 UI。\n","excerpt":"\u003ch1 id=\"使用-traceql-查询-skywalking-和-zipkin-链路追踪数据并在-grafana-中可视化\"\u003e使用 TraceQL 查询 SkyWalking 和 Zipkin 链路追踪数据并在 Grafana 中可视化\u003c/h1\u003e\n\u003cp\u003eApache SkyWalking 在 \u003cstrong\u003e10.4.0\u003c/strong\u003e 版本中引入了 \u003cstrong\u003eTraceQL\u003c/strong\u003e 支持 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-04-08-traceql/","title":"使用 TraceQL 查询 SkyWalking 和 Zipkin 链路追踪数据并在 Grafana 中可视化"},{"body":"SkyWalking BanyanDB 0.10.1 is released. Go to downloads page to find release tars.\nBug Fixes Fix reuse of byte arrays in min/max implementation causing data corruption. Fix flaky trace query filtering due to non-deterministic sidx tag ordering. Fix index-mode measure queries returning documents outside requested time range. Fix nil pointer panic in segment collectMetrics during shutdown. Fix entity tag handling in trace filter preventing TagIdx index mismatch. Fix unstable tag filter matching order in sidx. Fix property schema client connection instability after data node restart. Fix duplicate TopN query execution in distributed measure queries. Fix bydbctl validation for malformed YAML input. Fix FODC agent test instability in Basic Metrics Buffering test. ","excerpt":"\u003cp\u003eSkyWalking BanyanDB 0.10.1 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"bug-fixes\"\u003eBug Fixes\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eFix …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-0-10-1/","title":"Release Apache SkyWalking BanyanDB 0.10.1"},{"body":"The Problem: As Applications \u0026ldquo;Consume\u0026rdquo; LLMs, Monitoring Leaves a Blind Spot With the deep penetration of Generative AI (GenAI) into enterprise workflows, developers face a challenging paradox: while powerful LLM capabilities are easily integrated via Spring AI or OpenAI SDKs, the actual performance and reliability of these calls remain largely invisible.\n1. The \u0026ldquo;Black Box\u0026rdquo; of Cost and Performance: Is the Expensive Model Worth It? Facing high LLM bills, organizations often only see a total sum paid to a provider, but cannot calculate the \u0026ldquo;ROI\u0026rdquo; within the application.\nBlind Upgrades: You might switch to a premium flagship model for a better experience. But in your specific business scenario, does paying several times more per token actually yield lower latency or a faster TTFT (Time to First Token)? Lack of Real-World Benchmarks: Official benchmarks mean little without your real-world business requests. You need to know which model achieves the perfect balance between \u0026ldquo;Token/Cost Consumption\u0026rdquo; and \u0026ldquo;Response Speed\u0026rdquo; under your actual prompt lengths and concurrency levels. 2. The Vanishing \u0026ldquo;Golden Timeout\u0026rdquo; Many teams set timeouts for LLM calls arbitrarily (e.g., 30s or 60s).\nToo Short: During peak periods or long-text generation, requests are frequently interrupted, causing business failure rates to soar. Too Long: If a provider hangs, requests pile up in memory, blocking execution threads and potentially leading to the collapse of the entire Java application or microservice cluster. Only by mastering the P99/P95 Latency can you set rational timeout policies based on data rather than intuition. 3. The Overlooked Experience Killer: TTFT In GenAI scenarios, a user\u0026rsquo;s perception of speed depends less on the total duration of the conversation and more on \u0026ldquo;when the first word appears.\u0026rdquo; * A streaming response with a 10s total duration but a 500ms TTFT feels instantaneous.\nA non-streaming response with a 5s total duration but a 4s TTFT feels \u0026ldquo;frozen.\u0026rdquo; If your observability system only tracks total latency, you miss the core UX metric that explains why users complain about \u0026ldquo;AI slowness.\u0026rdquo; SkyWalking 10.4: A \u0026ldquo;Digital Dashboard\u0026rdquo;\nFrom the Application Perspective The Virtual GenAI capability introduced in Apache SkyWalking 10.4 fills this \u0026ldquo;observability vacuum.\u0026rdquo; It avoids reliance on external gateways by using application-side probes (like the Java Agent) to collect the most authentic data from the client\u0026rsquo;s perspective.\nPrecise Latency Distribution: Multi-dimensional metrics (P50, P90, P99) help visualize LLM fluctuations to inform dynamic timeout strategies. Core UX Metric — TTFT Monitoring: Native support for first-token latency in streaming calls. Multi-dimensional Model Profiling: Aligns token usage, estimated cost, and performance across Providers and Models, helping you choose the most cost-effective solution for your specific needs. Virtual GenAI Observability Virtual GenAI represents Generative AI service nodes detected by probe plugins. All performance metrics are based on the GenAI Client Perspective.\nFor instance, the Spring AI plugin in the Java Agent detects the response latency of a Chat Completion request. SkyWalking then visualizes these in the dashboard:\nTraffic \u0026amp; Success Rate (CPM \u0026amp; SLA) Latency \u0026amp; TTFT Token Usage (Input/Output) Estimated Cost Screenshots: How It Works When the SkyWalking Java Agent or OTLP probes intercept calls to mainstream AI frameworks (e.g., Spring AI, OpenAI SDK), they report Trace data to the SkyWalking OAP. The OAP aggregates and computes this data to generate performance metrics for both Providers and Models, which are then rendered in the built-in Virtual-GenAI dashboards.\nInstallation \u0026amp; Configuration Requirements SkyWalking Java Agent: \u0026gt;= 9.7 SkyWalking OAP: \u0026gt;= 10.4 Semantic Conventions \u0026amp; Compatibility SkyWalking Virtual GenAI follows OpenTelemetry GenAI Semantic Conventions. OAP identifies GenAI-related Spans based on:\nSkyWalking Java Agent Spans must be of type Exit, have the SpanLayer attribute set to GENAI, and contain the gen_ai.response.model tag. OTLP / Zipkin Probes Spans must contain the gen_ai.response.model tag. For details, refer to the E2E configurations:\nSkyWalking Java Agent Reporting Probe Reporting OTLP Data Probe Reporting Zipkin Data GenAI Estimated Cost Configuration Overview SkyWalking provides a built-in GenAI Billing Configuration File.\nThis file defines how SkyWalking maps model names from Trace data to their corresponding providers and estimates the token cost for each LLM call. The estimated cost is displayed in the SkyWalking UI alongside trace and metric data, helping users intuitively understand the financial impact of their GenAI usage.\nImportant: The pricing in this file is intended for cost estimation only and must not be treated as actual billing or invoice amounts. Users are advised to regularly verify the latest rates on the providers\u0026rsquo; official pricing pages.\nConfiguration Structure Top-level Fields Field Type Description last-updated date The last update date of the pricing data. All prices are based on public billing standards announced by providers prior to this date. providers list List of GenAI provider definitions. Each entry contains matching rules and specific model pricing information. Provider Definition Each entry under providers defines a GenAI provider:\nproviders: - provider: \u0026lt;provider-name\u0026gt; prefix-match: - \u0026lt;prefix-1\u0026gt; - \u0026lt;prefix-2\u0026gt; models: - name: \u0026lt;model-name\u0026gt; aliases: [\u0026lt;alias-1\u0026gt;, \u0026lt;alias-2\u0026gt;] input-estimated-cost-per-m: \u0026lt;cost\u0026gt; output-estimated-cost-per-m: \u0026lt;cost\u0026gt; Field Type Required Description provider string Yes The provider identifier (e.g., openai, anthropic, gemini). It is displayed as the Virtual GenAI service name in SkyWalking. prefix-match list[string] Yes A list of prefixes used to match model names to this provider. If a model name in the Trace data starts with any of these prefixes, it will be mapped to this provider. models list[model] No A list of model definitions containing pricing information. If omitted, the system can still identify the provider but will not perform cost estimation. Model Definition Each entry under models defines the pricing for a specific model:\nField Type Required Description name string Yes The standard model name used for matching. aliases list[string] No Alternative names that should resolve to the same billing entry. This is useful when providers use different naming conventions (see the \u0026ldquo;Model Aliases\u0026rdquo; section). input-estimated-cost-per-m float No Estimated cost per 1,000,000 (one million) input (Prompt) tokens. The default unit is USD. output-estimated-cost-per-m float No Estimated cost per 1,000,000 (one million) output (Completion) tokens. The default unit is USD. Model Matching Mechanism Provider-Level Prefix Matching When SkyWalking receives a Trace containing a GenAI call, it determines the Provider based on the following priority order:\ngen_ai.provider.name tag: This tag is retrieved first. It follows the latest OpenTelemetry GenAI semantic conventions. gen_ai.system tag: If the above tag is missing, the system falls back to this legacy tag. Note: This tag is only parsed when processing OTLP or Zipkin format data, primarily for compatibility with older versions of libraries like the Python auto-instrumentation. Prefix Matching: If neither of the above tags exists, SkyWalking reads the prefix-match rules defined in gen-ai-config.yml and attempts to identify the provider by matching the Model Name. - provider: openai prefix-match: - gpt Any model name starting with gpt (such as gpt-4o, gpt-4.1-mini, or gpt-5-nano) will be mapped to the openai provider. A single provider can have multiple prefixes:\n- provider: tencent prefix-match: - hunyuan - Tencent Model-level Longest-Prefix Matching Once the provider is determined, SkyWalking uses a Trie-based longest-prefix matching algorithm to find the best billing entry. This is crucial because model names returned in provider API responses often include version numbers or timestamps, differing from the base model name in the config. Example OpenAI config:\nmodels: - name: gpt-4o input-estimated-cost-per-m: 2.5 output-estimated-cost-per-m: 10.0 - name: gpt-4o-mini input-estimated-cost-per-m: 0.15 output-estimated-cost-per-m: 0.6 Matching behavior:\nModel Name in Trace Matched Configuration Entry Reason gpt-4o gpt-4o Exact match gpt-4o-2024-08-06 gpt-4o Longest prefix is gpt-4o gpt-4o-mini gpt-4o-mini Exact match (Longer prefix gpt-4o-mini takes priority over gpt-4o) gpt-4o-mini-2024-07-18 gpt-4o-mini Longest prefix is gpt-4o-mini This mechanism ensures versioned API model names map to the correct pricing tier without requiring exact full names in the configuration file.\nModel Aliases Some providers use different naming conventions across API responses and documentation. For example, Anthropic\u0026rsquo;s model might appear as claude-4-sonnet or claude-sonnet-4. The aliases field supports both formats under a single billing entry:\n- name: claude-4-sonnet aliases: [claude-sonnet-4] input-estimated-cost-per-m: 3.0 output-estimated-cost-per-m: 15.0 Under this configuration, claude-4-sonnet and claude-sonnet-4 (as well as any versioned variants, such as claude-sonnet-4-20250514) will resolve to the same billing entry.\nNote: Aliases also participate in longest prefix matching. Therefore, claude-sonnet-4-20250514 will match the alias claude-sonnet-4, which in turn resolves to the pricing information for claude-4-sonnet.\nCustom Configuration Adding a New Provider To add a provider that is not included in the default configuration:\nproviders: # ... Existing providers ... - provider: ollama prefix-match: - mymodel models: - name: mymodel-large input-estimated-cost-per-m: 1.0 output-estimated-cost-per-m: 5.0 - name: mymodel-small input-estimated-cost-per-m: 0.1 output-estimated-cost-per-m: 0.5 For OTLP/Zipkin data, a dedicated estimated tag has been added. You can now view the cost of each GenAI call directly on the UI. Main Metrics 1.Provider Level Metric ID Description Meaning gen_ai_provider_cpm Calls Per Minute Requests per minute (Throughput) gen_ai_provider_sla Success Rate Request success rate gen_ai_provider_resp_time Avg Response Time Average response time gen_ai_provider_latency_percentile Latency Percentiles Response time percentiles (P50, P75, P90, P95, P99) gen_ai_provider_input_tokens_sum/avg Input Token Usage Total and average input token usage gen_ai_provider_output_tokens_sum/avg Output Token Usage Total and average output token usage gen_ai_provider_total_estimated_cost/avg Estimated Cost Total estimated cost and average cost per call 2. Model Level Metric ID Description Meaning gen_ai_model_call_cpm Calls Per Minute Requests per minute for this specific model gen_ai_model_sla Success Rate Model-specific request success rate gen_ai_model_latency_avg/percentile Latency Average and percentiles of model response duration gen_ai_model_ttft_avg/percentile TTFT Time to First Token (Streaming only) gen_ai_model_input_tokens_sum/avg Input Token Usage Detailed input token consumption for the model gen_ai_model_output_tokens_sum/avg Output Token Usage Detailed output token consumption for the model gen_ai_model_total_estimated_cost/avg Estimated Cost Estimated total cost and average cost for the model Recommended Usage Scenarios Performance Evaluation: Use Latency and Time to First Token (TTFT) metrics to analyze model inference efficiency and the end-user interaction experience. Token Monitoring: Real-time monitoring of Input and Output token consumption to analyze resource utilization across different business scenarios. Cost Alerting: Set alert thresholds based on Estimated Cost or token consumption to promptly detect abnormal calls and prevent budget overruns. ","excerpt":"\u003ch1 id=\"the-problem-as-applications-consume-llms-monitoring-leaves-a-blind-spot\"\u003eThe Problem: As Applications \u0026ldquo;Consume\u0026rdquo; LLMs, Monitoring Leaves a Blind Spot\u003c/h1\u003e\n\u003cp\u003eWith the …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-04-05-virtual-genai-monitoring/","title":"Monitoring LLM Applications with SkyWalking 10.4: Insights into Performance and Cost"},{"body":"问题：当应用开始“吞噬”大模型，监控却留下了盲区 随着生成式 AI（GenAI）在企业业务中的深度渗透，开发者正面临一个尴尬的局面：我们在应用中通过Spring AI或OpenAI SDK快速集成了强大的大模型能力，但对于这些调用的实际表现却几乎一无所知。\n成本与性能的“黑盒”：昂贵的模型真的更具性价比吗？\n面对高昂的大模型账单，我们往往只知道把钱交给了某个Provider，却算不清这笔账在应用内部的“投入产出比”。 盲目的选型升级：为了追求更好的体验，你可能将业务默认切换到了成本更高的旗舰模型。但在具体的业务场景下，花费数倍的 Token 成本，它真的能在真实请求中带来更低的延迟和更快的 TTFT(Time to First Token) 吗？ 缺乏真实的评估基准：脱离了真实的业务请求，单纯看官网的 Benchmark 意义不大，你需要知道在实际的 Prompt 长度和并发压力下，同一Provider下的哪个模型能在“Token/Cost 消耗”与“响应速度”之间达到完美的平衡。如果没有应用侧的数据支撑，你根本无从判断哪款模型才是当前业务的最优解。\n消失的“黄金超时时间”\n很多团队在代码里给 LLM 调用设置超时（Timeout）时，往往是拍脑袋决定（比如 30s 或 60s）。\n设太短：长文本生成或模型高峰期时，请求会被频繁强行中断，导致业务失败率飙升。\n设太长：如果下游供应商出现故障（卡死），大量的请求会堆积在应用内存中，阻塞执行线程，最终导致整个 Java 应用甚至微服务集群的瘫痪。 只有真正掌握了预估的整体调用延迟（P99/P95 Latency），你才能基于数据而非直觉，为不同模型设置最合理的超时策略。\n被忽视的体验杀手：TTFT\n在 GenAI 场景下，用户对“快”的感知并不完全取决于整个对话结束的总耗时，而取决于**“第一行字什么时候跳出来”**。 一个总耗时 10 秒但 TTFT 仅 500ms 的流式响应，给用户的观感是“秒回”。 一个总耗时 5 秒但 TTFT 需要 4s 的非流式响应，给用户的观感却是“卡死”。 如果你的观测系统只能看到总耗时，你就会漏掉最核心的 UX 指标，无法解释为什么用户反馈“AI 很慢”即便总耗时看起来还行。\nSkyWalking 10.4：应用视角的“数字仪表盘”\nApache SkyWalking 自 10.4 版本引入的 Virtual GenAI 能力，正是为了解决应用层侧的这种“观测真空”。它不依赖任何外部网关，直接通过应用侧探针（如 Java Agent）在客户端视角采集最真实的数据。\n精准的延迟分布（Latency Percentiles）：通过 P50、P90、P99 等多维指标，帮你勾勒出 LLM 调用的真实波动曲线，为设置“动态超时时间”提供科学依据。 核心 UX 指标——TTFT 监控：原生支持流式（Streaming）调用的首字延迟统计。通过对比不同 Provider 或不同模型的 TTFT，你可以优化提示词（Prompt）策略或切换更快的模型，确保用户体验始终在线。 多维度的模型“画像”分析：在 Provider 和 Model 两个维度上，将 Token 消耗、预估成本与性能指标深度对齐。这让你不再看供应商全网的“理想平均数”，而是看清你的应用在调用特定模型时的真实表现，从而在复杂的模型生态中选出最具性价比的选型方案。 虚拟 GenAI 观测 虚拟 GenAI 代表了由探针插件检测到的生成式 AI 服务节点。GenAI 操作的性能指标均基于 GenAI 客户端视角。\n例如，Java 探针中的 Spring AI 插件可以检测一次对话补全（Chat Completion）请求的响应延迟。随后，SkyWalking 将在仪表盘中展示：\n流量与成功率 (CPM \u0026amp; SLA) 响应延迟 (Latency \u0026amp; TTFT) Token 消耗 (Input/Output) 预估成本 (Estimated Cost) 如图： 原理 当 SkyWalking Java Agent 或 OTLP 探针拦截到主流 AI 框架（如 Spring AI、OpenAI SDK 等）的调用时，将Trace 数据上报至 SkyWalking OAP。 OAP会基于这些 Trace 自动完成数据的聚合与计算。最终会生成 Provider（服务商）与 Model（模型）两个维度的各类性能指标，并直接渲染填充至内置的 Virtual-GenAI 仪表盘中。\n安装配置 要求 版本要求 ● SkyWalking Java Agent: \u0026gt;= 9.7 ● SkyWalking Oap: \u0026gt;= 10.4\n语义规范与兼容性 SkyWalking 虚拟 GenAI 遵循 OpenTelemetry GenAI 语义规范。OAP 将根据以下标准识别 GenAI 相关 Span：\nSkyWalking Java Agent 上报的 Span 必须为 Exit 类型，其 SpanLayer 属性需设定为 GENAI,包含gen_ai.response.model 标签。 输出OTLP / Zipkin格式数据的探针 上报的 Span 中包含 gen_ai.response.model 标签。 具体可以参考e2e配置\nSkyWalking Java Agent上报数据\n探针上报OTLP格式数据\n探针上报Zipkin格式数据\nGenAI 预估成本配置 概览 SkyWalking 提供了一个内置的GenAI计费配置文件\n该配置定义了SkyWalking 如何将 Trace 数据中的模型名称映射到对应的供应商，并估算每次 LLM 调用的 Token 成本。估算成本将与 Trace 和指标数据一起显示在 SkyWalking UI 中，帮助用户直观了解 GenAI 使用带来的 预估费用影响。 重要提示: 此文件中的定价仅用于成本估算，不得视为实际账单或发票金额。建议用户定期从供应商官方定价页面核实最新费率。\n配置结构 Top 字段 字段 类型 描述 last-updated date 定价数据的最后更新日期。所有价格均基于该日期前各厂商官网公布的公开计费标准。 providers list GenAI 厂商定义列表。每个厂商条目下包含匹配规则（matching rules）以及具体的模型计费信息（model pricing）。 provider 定义 providers 下的每个条目定义一个 GenAI 供应商：\nproviders: - provider: \u0026lt;provider-name\u0026gt; prefix-match: - \u0026lt;prefix-1\u0026gt; - \u0026lt;prefix-2\u0026gt; models: - name: \u0026lt;model-name\u0026gt; aliases: [\u0026lt;alias-1\u0026gt;, \u0026lt;alias-2\u0026gt;] input-estimated-cost-per-m: \u0026lt;cost\u0026gt; output-estimated-cost-per-m: \u0026lt;cost\u0026gt; 字段 (Field) 类型 (Type) 必填 (Required) 描述 (Description) provider string 是 供应商标识（如 openai, anthropic, gemini）。在 SkyWalking 中作为虚拟 GenAI 服务名显示。 prefix-match list[string] 是 用于将模型名称匹配到该供应商的前缀列表。如果 Trace 数据中的模型名以其中任一前缀开头，则会被映射到该供应商。 models list[model] 否 包含定价信息的模型定义列表。如果省略，系统仍能识别供应商，但不会进行成本估算。 model 定义 models 下的每个条目定义特定模型的定价：\n字段 (Field) 类型 (Type) 必填 (Required) 描述 (Description) name string 是 用于匹配的标准模型名称。 aliases list[string] 否 应解析为同一计费条目的备选名称。当供应商使用不同的命名习惯时非常有用（参见“模型别名”部分）。 input-estimated-cost-per-m float 否 每 1,000,000（一百万）输入（Prompt）Token 的预估成本。默认单位为 USD。 output-estimated-cost-per-m float 否 每 1,000,000（一百万）输出（Completion）Token 的预估成本。默认单位为 USD。 模型匹配机制 供应商级前缀匹配 当 SkyWalking 接收到包含 GenAI 调用的 Trace 时，会按照以下优先级顺序来确定供应商（Provider）：\ngen_ai.provider.name 标签：首先检索此标签。它是OpenTelemetry最新的语义规范。 gen_ai.system 标签：如果缺少上述标签，系统将回退到此旧版（Legacy）标签。注意：此标签仅在处理 OTLP 或 Zipkin 协议的数据时会被解析，主要用于兼容旧版的 Python 自动仪表化等库。 前缀匹配 (Prefix Matching)：若上述两个标签均不存在，SkyWalking 会读取 gen-ai-config.yml 中定义的 prefix-match 规则，通过匹配 模型名称 (Model Name) 来尝试识别供应商。 - provider: openai prefix-match: - gpt 任何以 gpt 开头的模型名称（如 gpt-4o, gpt-4.1-mini, gpt-5-nano）都会被映射到 openai 供应商。 一个供应商可以拥有多个前缀：\n- provider: tencent prefix-match: - hunyuan - Tencent 模型级最长前缀匹配 (Model-Level Longest-Prefix Matching) 一旦确定了供应商，SkyWalking 会使用基于前缀树 (Trie) 的最长前缀匹配算法来查找最佳的模型计费条目。这至关重要，因为 LLM 供应商在 API 响应中返回的模型名称通常包含版本号或时间戳，与配置中的基础模型名称有所不同。 示例： 假设 OpenAI 的配置条目如下：\nmodels: - name: gpt-4o input-estimated-cost-per-m: 2.5 output-estimated-cost-per-m: 10.0 - name: gpt-4o-mini input-estimated-cost-per-m: 0.15 output-estimated-cost-per-m: 0.6 其匹配行为如下表所示：\nTrace 中的模型名称 匹配的配置条目 原因 gpt-4o gpt-4o 完全匹配 gpt-4o-2024-08-06 gpt-4o 最长前缀为 gpt-4o gpt-4o-mini gpt-4o-mini 完全匹配（比 gpt-4o 更长的前缀优先） gpt-4o-mini-2024-07-18 gpt-4o-mini 最长前缀为 gpt-4o-mini 这种机制确保了 API 返回的带有版本的模型名称能够被正确映射到相应的价格档位，而无需在配置文件中维护精确的全名。\n模型别名 (Model Aliases) 部分供应商在 API 响应和官方文档中会使用不同的命名规范。例如，Anthropic 的模型在 Trace 中可能显示为 claude-4-sonnet 或 claude-sonnet-4。通过 aliases 字段，可以让单个计费条目同时支持这两种配置：\n- name: claude-4-sonnet aliases: [claude-sonnet-4] input-estimated-cost-per-m: 3.0 output-estimated-cost-per-m: 15.0 在这种配置下，claude-4-sonnet 和 claude-sonnet-4（以及任何带有版本的变体，如 claude-sonnet-4-20250514）都会解析为同一个计费条目。\n注意： 别名同样参与最长前缀匹配。因此，claude-sonnet-4-20250514 会匹配到别名 claude-sonnet-4，进而解析到 claude-4-sonnet 的定价信息。\n自定义配置 添加新供应商 (Adding a New Provider) 要添加默认配置中未包含的供应商：\nproviders: # ... 现有供应商 ... - provider: ollama prefix-match: - mymodel models: - name: mymodel-large input-estimated-cost-per-m: 1.0 output-estimated-cost-per-m: 5.0 - name: mymodel-small input-estimated-cost-per-m: 0.1 output-estimated-cost-per-m: 0.5 针对OTLP/zipkin的数据，新增了单独的estimated tag, 可以在UI上看到这次GenAI调用消耗的cost。\n主要指标 1. Provider Level (服务商维度) 指标 ID 描述 含义 gen_ai_provider_cpm Calls Per Minute 每分钟请求数 (吞吐量) gen_ai_provider_sla Success Rate 请求成功率 gen_ai_provider_resp_time Avg Response Time 平均响应耗时 gen_ai_provider_latency_percentile Latency Percentiles 响应耗时百分位数 (P50, P75, P90, P95, P99) gen_ai_provider_input_tokens_sum/avg Input Token Usage 输入 Token 的总和及平均值 gen_ai_provider_output_tokens_sum/avg Output Token Usage 输出 Token 的总和及平均值 gen_ai_provider_total_estimated_cost/avg Estimated Cost 预估总成本及次均成本 2. Model Level (模型维度) 指标 ID 描述 含义 gen_ai_model_call_cpm Calls Per Minute 该特定模型的每分钟请求数 gen_ai_model_sla Success Rate 模型请求成功率 gen_ai_model_latency_avg/percentile Latency 模型响应耗时的平均值及百分位数 gen_ai_model_ttft_avg/percentile TTFT 首个token响应时间 (仅限流式传输 Streaming) gen_ai_model_input_tokens_sum/avg Input Token Usage 该模型的输入 Token 消耗详情 gen_ai_model_output_tokens_sum/avg Output Token Usage 该模型的输出 Token 消耗详情 gen_ai_model_total_estimated_cost/avg Estimated Cost 该模型的预估总成本及次均成本 建议使用场景 性能评估：利用 响应延迟（Latency） 和 首字响应时间（TTFT） 指标，分析模型推理效率及终端用户交互体验。 Token 监控：实时监控 输入（Input）与输出（Output）Token 的消耗，用于分析不同业务场景下的资源占用情况。 成本预警：支持基于 预估成本（Cost） 或 Token 消耗量 配置告警阈值，及时发现异常调用，防止成本超支。 ","excerpt":"\u003ch1 id=\"问题当应用开始吞噬大模型监控却留下了盲区\"\u003e问题：当应用开始“吞噬”大模型，监控却留下了盲区\u003c/h1\u003e\n\u003cp\u003e随着生成式 AI（GenAI）在企业业务中的深度渗透，开发者正面临一个尴尬的局面：我们在应用中通过\u003ccode\u003eSpring AI\u003c/code\u003e或\u003ccode\u003eOpenAI SDK\u003c/code\u003e快速集成了强 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-04-05-virtual-genai-monitoring/","title":"基于 SkyWalking 10.4 的大模型应用监控：洞察 LLM 的性能与成本"},{"body":"The Problem: Flying Blind with LLM Traffic LLM traffic is becoming a first-class citizen in production infrastructure. Teams are calling OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, Google Gemini — often multiple providers at once. But most organizations have no unified visibility into this traffic:\nToken costs spiral without knowing which teams, models, or providers drive the spend. A single misconfigured prompt template can burn through thousands of dollars before anyone notices. Provider outages cause cascading failures. When OpenAI has a bad hour, your application goes down with it — and you have no failover visibility to understand what happened or switch providers automatically. No unified metrics across heterogeneous LLM calls. Latency, Time to First Token (TTFT), Time Per Output Token (TPOT), token usage, error rates — each provider reports these differently, if at all. There is no single dashboard to compare them. This is the same observability gap that microservices faced a decade ago. The solution then was service meshes and API gateways with built-in telemetry. For AI workloads, the answer is an AI gateway.\nWhy an AI Gateway Envoy AI Gateway is an open-source AI gateway built on top of Envoy Proxy and Envoy Gateway. It is not a standalone SaaS product or a Python proxy — it is infrastructure-grade software built on the same Envoy that already handles traffic for a large portion of cloud-native deployments.\nKey capabilities:\nMulti-provider routing — supports 16+ AI providers (OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, Google Gemini, Mistral, Cohere, DeepSeek, and more) behind a unified API. Token-based rate limiting — rate limit by token consumption, not just request count. Provider fallback — automatic failover when a provider is down or slow. Model virtualization — abstract model names so applications are decoupled from specific providers. Two-tier architecture — a reference architecture with a centralized entry gateway (Tier 1) for auth and global routing, and per-cluster gateways (Tier 2) for inference optimization. CNCF ecosystem native — runs on Kubernetes, composes with existing Envoy filters, WASM plugins, and standard Kubernetes Gateway API resources. Because Envoy AI Gateway natively emits GenAI metrics and access logs via OTLP following OpenTelemetry GenAI Semantic Conventions, it plugs directly into any OpenTelemetry-compatible backend.\nStarting from SkyWalking 10.4.0, the OAP server natively receives and analyzes Envoy AI Gateway\u0026rsquo;s OTLP metrics and access logs — no OpenTelemetry Collector needed in between.\nData Flow The AI Gateway pushes telemetry directly to SkyWalking via OTLP gRPC:\nApplication sends LLM API requests through the Envoy AI Gateway. Envoy AI Gateway routes requests to AI providers (or local models like Ollama) and records GenAI metrics (token usage, latency, TTFT, TPOT) and access logs. The gateway pushes metrics and logs via OTLP gRPC directly to SkyWalking OAP on port 11800. SkyWalking OAP parses metrics with MAL rules and access logs with LAL rules, then stores everything in BanyanDB. No OpenTelemetry Collector is needed. SkyWalking OAP\u0026rsquo;s built-in OTLP receiver handles everything.\nTry It Locally This demo uses Ollama as a local LLM backend so you can try everything without an API key. The Envoy AI Gateway CLI (aigw) provides a standalone mode that runs outside Kubernetes — perfect for local testing.\nPrerequisites Docker and Docker Compose Ollama installed on your host Step 1: Start Ollama Start Ollama on all interfaces so Docker containers can reach it:\nOLLAMA_HOST=0.0.0.0 ollama serve Pull a small model for testing:\nollama pull llama3.2:1b Step 2: Start the Stack Create a docker-compose.yaml:\nservices: banyandb: image: apache/skywalking-banyandb:0.10.0 container_name: banyandb ports: - \u0026#34;17912:17912\u0026#34; command: standalone --stream-root-path /tmp/stream-data --measure-root-path /tmp/measure-data healthcheck: test: [\u0026#34;CMD-SHELL\u0026#34;, \u0026#34;wget -qO- http://localhost:17913/api/healthz || exit 1\u0026#34;] interval: 5s timeout: 3s retries: 10 oap: image: apache/skywalking-oap-server:10.4.0 container_name: oap depends_on: banyandb: condition: service_healthy ports: - \u0026#34;11800:11800\u0026#34; - \u0026#34;12800:12800\u0026#34; environment: SW_STORAGE: banyandb SW_STORAGE_BANYANDB_TARGETS: banyandb:17912 healthcheck: test: [\u0026#34;CMD-SHELL\u0026#34;, \u0026#34;bash -c \u0026#39;echo \u0026gt; /dev/tcp/localhost/12800\u0026#39; || exit 1\u0026#34;] interval: 10s timeout: 5s retries: 30 start_period: 60s ui: image: apache/skywalking-ui:10.4.0 container_name: ui depends_on: oap: condition: service_healthy ports: - \u0026#34;8080:8080\u0026#34; environment: SW_OAP_ADDRESS: http://oap:12800 aigw: image: envoyproxy/ai-gateway-cli:latest container_name: aigw depends_on: oap: condition: service_healthy environment: - OPENAI_BASE_URL=http://host.docker.internal:11434/v1 - OPENAI_API_KEY=unused - OTEL_SERVICE_NAME=my-ai-gateway - OTEL_EXPORTER_OTLP_ENDPOINT=http://oap:11800 - OTEL_EXPORTER_OTLP_PROTOCOL=grpc - OTEL_METRICS_EXPORTER=otlp - OTEL_LOGS_EXPORTER=otlp - OTEL_METRIC_EXPORT_INTERVAL=5000 - OTEL_RESOURCE_ATTRIBUTES=job_name=envoy-ai-gateway,service.instance.id=aigw-1,service.layer=ENVOY_AI_GATEWAY ports: - \u0026#34;1975:1975\u0026#34; extra_hosts: - \u0026#34;host.docker.internal:host-gateway\u0026#34; command: [\u0026#34;run\u0026#34;] Start everything:\ndocker compose up -d Wait for all services to become healthy (BanyanDB starts first, then OAP, then UI and AI Gateway):\ndocker compose ps The key OTLP configuration on the aigw service:\nEnv Var Value Purpose OTEL_SERVICE_NAME my-ai-gateway Service name in SkyWalking OTEL_EXPORTER_OTLP_ENDPOINT http://oap:11800 SkyWalking OAP gRPC endpoint OTEL_EXPORTER_OTLP_PROTOCOL grpc OTLP transport OTEL_METRICS_EXPORTER otlp Enable metrics push OTEL_LOGS_EXPORTER otlp Enable access log push The OTEL_RESOURCE_ATTRIBUTES must include:\njob_name=envoy-ai-gateway — routing tag for MAL/LAL rules service.instance.id=\u0026lt;id\u0026gt; — instance identity service.layer=ENVOY_AI_GATEWAY — routes logs to AI Gateway LAL rules The MAL and LAL rules are enabled by default in SkyWalking OAP. No OAP-side configuration is needed.\nStep 3: Run the Demo App Create a simple Python application that sends requests through the AI Gateway (app.py). It mixes normal requests, streaming requests (for TTFT/TPOT metrics), and error requests (non-existent model → HTTP 404, always captured by the LAL sampling policy):\nimport time, random, requests GATEWAY = \u0026#34;http://localhost:1975\u0026#34; HEADERS = {\u0026#34;Authorization\u0026#34;: \u0026#34;Bearer unused\u0026#34;, \u0026#34;Content-Type\u0026#34;: \u0026#34;application/json\u0026#34;} questions = [ \u0026#34;What is Apache SkyWalking? Answer in one sentence.\u0026#34;, \u0026#34;What is Envoy Proxy used for? Answer in one sentence.\u0026#34;, \u0026#34;What are the benefits of an AI gateway? Answer in two sentences.\u0026#34;, \u0026#34;Explain observability in three sentences.\u0026#34;, ] def chat(model, question, stream=False): resp = requests.post( f\u0026#34;{GATEWAY}/v1/chat/completions\u0026#34;, json={\u0026#34;model\u0026#34;: model, \u0026#34;messages\u0026#34;: [{\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: question}], \u0026#34;stream\u0026#34;: stream}, headers=HEADERS, timeout=60, stream=stream, ) if stream: chunks = [] for line in resp.iter_lines(): if line: chunks.append(line.decode()) return resp.status_code, f\u0026#34;[streamed {len(chunks)} chunks]\u0026#34; return resp.status_code, resp.json() while True: r = random.random() if r \u0026lt; 0.2: # Error request: non-existent model triggers 404 status, body = chat(\u0026#34;non-existent-model\u0026#34;, \u0026#34;hello\u0026#34;) print(f\u0026#34;[error] model=non-existent-model status={status}\u0026#34;) elif r \u0026lt; 0.5: # Streaming request — generates TTFT and TPOT metrics q = random.choice(questions) status, info = chat(\u0026#34;llama3.2:1b\u0026#34;, q, stream=True) print(f\u0026#34;[stream] status={status} {info}\u0026#34;) else: # Normal non-streaming request q = random.choice(questions) status, body = chat(\u0026#34;llama3.2:1b\u0026#34;, q) answer = body.get(\u0026#34;choices\u0026#34;, [{}])[0].get(\u0026#34;message\u0026#34;, {}).get(\u0026#34;content\u0026#34;, \u0026#34;\u0026#34;)[:80] tokens = body.get(\u0026#34;usage\u0026#34;, {}) print(f\u0026#34;[ok] status={status} tokens={tokens} answer={answer}...\u0026#34;) time.sleep(random.randint(20, 30)) Run it:\npip install requests python app.py The application talks to the AI Gateway on port 1975, which routes to Ollama. Each request generates GenAI metrics (token usage, latency, TTFT, TPOT) and access logs that the gateway pushes to SkyWalking via OTLP.\nThe error requests (non-existent model → HTTP 404) are always captured by the access log sampling policy, so you will see them in the SkyWalking log view.\nStep 4: View in SkyWalking UI Open http://localhost:8080 and select the GenAI \u0026gt; Envoy AI Gateway menu.\nThe service list shows my-ai-gateway with CPM, latency, and token rates at a glance:\nClick into the service to see the full dashboard — Request CPM, Latency (average + percentiles), Input/Output Token Rates, TTFT, and TPOT:\nThe Providers tab breaks down metrics by AI provider:\nThe Models tab shows per-model metrics including TTFT and TPOT (streaming only). Note the unknown model entries — these are the error requests with non-existent models:\nThe Log tab shows access logs. The sampling policy drops normal successful responses but always captures errors (HTTP 404) and high-token requests:\nCleanup docker compose down Deploying on Kubernetes For production deployments, Envoy AI Gateway runs as a full Kubernetes controller with Envoy Gateway as the control plane. See the Envoy AI Gateway getting started guide for Kubernetes installation.\nThe OTLP configuration is the same — set the OTEL_* environment variables on the AI Gateway\u0026rsquo;s external processor to point at SkyWalking OAP\u0026rsquo;s gRPC port (11800). See the SkyWalking Envoy AI Gateway Monitoring documentation for details.\nGenAI Observability Without an AI Gateway Not every deployment uses an AI gateway. If your applications call LLM providers directly, SkyWalking 10.4.0 also provides GenAI observability through the Virtual GenAI layer.\nThis works with any SkyWalking-instrumented, OpenTelemetry-instrumented, or Zipkin-instrumented application. When traces carry gen_ai.* tags (following OpenTelemetry GenAI Semantic Conventions), SkyWalking derives per-provider and per-model metrics from the client side: latency, token usage, success rate, and estimated cost.\nFor Java applications, the SkyWalking Java Agent (9.7+) includes a Spring AI plugin that automatically instruments calls to 13+ providers (OpenAI, Anthropic, AWS Bedrock, Google GenAI, DeepSeek, Mistral, etc.) with the correct gen_ai.* span tags — no code changes needed.\nThis is a different use case from the Envoy AI Gateway monitoring covered above:\nEnvoy AI Gateway layer: infrastructure-level observability — what the gateway sees across all traffic. Best for platform teams managing centralized AI routing. Virtual GenAI layer: application-level observability — what each instrumented app sees for its own LLM calls. Best for teams without a centralized gateway, or for per-application cost tracking. References Envoy AI Gateway — project site and documentation Envoy AI Gateway CLI — standalone mode for local development SkyWalking Envoy AI Gateway Monitoring — OAP setup doc SkyWalking Virtual GenAI — client-side GenAI observability OpenTelemetry GenAI Semantic Conventions — the metric/attribute standard both projects follow ","excerpt":"\u003ch2 id=\"the-problem-flying-blind-with-llm-traffic\"\u003eThe Problem: Flying Blind with LLM Traffic\u003c/h2\u003e\n\u003cp\u003eLLM traffic is becoming a first-class citizen in …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-04-02-envoy-ai-gateway-monitoring/","title":"Monitoring Envoy AI Gateway with Apache SkyWalking"},{"body":"问题：LLM 流量缺乏统一观测 LLM 流量正在成为生产基础设施中不可忽视的一部分。团队同时在调用 OpenAI、Anthropic、AWS Bedrock、Azure OpenAI、Google Gemini——往往还不止一个提供商。但大多数组织对这些流量缺乏统一的可见性：\nToken 费用失控，却不知道哪个团队、哪个模型、哪个提供商在烧钱。一个配置不当的 prompt 模板就可能在无人察觉的情况下烧掉几千美元。 提供商故障引发连锁反应。 OpenAI 出问题的那一小时，你的应用也跟着挂——而你既没有故障切换的可见性，也无法自动切换提供商。 缺乏统一指标。 延迟、首 Token 耗时（TTFT）、每 Token 输出耗时（TPOT）、Token 用量、错误率——每个提供商的报告方式都不一样，有些甚至不提供。没有一个统一的面板能做对比。 这和十年前微服务面临的可观测性困境如出一辙。当时的解法是服务网格和内置遥测的 API 网关。对 AI 工作负载来说，答案就是 AI 网关。\n为什么选择 AI 网关 Envoy AI Gateway 是一个开源 AI 网关，构建在 Envoy Proxy 和 Envoy Gateway 之上。底层就是云原生世界里已经广泛部署的 Envoy，天然具备基础设施级的稳定性和性能。\n核心能力：\n多提供商路由 —— 支持 16+ AI 提供商（OpenAI、Anthropic、AWS Bedrock、Azure OpenAI、Google Gemini、Mistral、Cohere、DeepSeek 等），统一 API 接入。 基于 Token 的限流 —— 按 Token 消耗限流，而不只是按请求数。 提供商故障切换 —— 某个提供商宕机或响应慢时自动切换。 模型虚拟化 —— 抽象模型名称，让应用与具体提供商解耦。 两层架构 —— 参考架构包含一个集中入口网关（Tier 1）负责认证和全局路由，以及每集群网关（Tier 2）负责推理优化。 CNCF 生态原生 —— 运行在 Kubernetes 上，兼容现有的 Envoy Filter、WASM 插件和标准 Kubernetes Gateway API 资源。 Envoy AI Gateway 原生支持通过 OTLP 发送 GenAI 指标和访问日志，遵循 OpenTelemetry GenAI 语义约定，可以直接接入任何兼容 OpenTelemetry 的后端。\n从 SkyWalking 10.4.0 开始，OAP 原生接收和分析 Envoy AI Gateway 的 OTLP 指标和访问日志——中间不需要部署 OpenTelemetry Collector。\n数据流 AI Gateway 通过 OTLP gRPC 直接将遥测数据推送到 SkyWalking：\n应用 通过 Envoy AI Gateway 发送 LLM API 请求。 Envoy AI Gateway 将请求路由到 AI 提供商（或 Ollama 这样的本地模型），同时记录 GenAI 指标（Token 用量、延迟、TTFT、TPOT）和访问日志。 网关通过 OTLP gRPC 直接将指标和日志推送到 SkyWalking OAP 的 11800 端口。 SkyWalking OAP 用 MAL 规则解析指标、用 LAL 规则解析访问日志，然后统一存储到 BanyanDB。 不需要 OpenTelemetry Collector。SkyWalking OAP 内置的 OTLP 接收器可以直接处理所有数据。\n本地体验 这个 Demo 使用 Ollama 作为本地 LLM 后端，不需要任何 API Key 就能跑起来。Envoy AI Gateway CLI（aigw）提供独立运行模式，不依赖 Kubernetes，非常适合本地测试。\n前置条件 Docker 和 Docker Compose 主机上已安装 Ollama 第一步：启动 Ollama 让 Ollama 监听所有网络接口，以便 Docker 容器能访问到：\nOLLAMA_HOST=0.0.0.0 ollama serve 拉取一个小模型用于测试：\nollama pull llama3.2:1b 第二步：启动服务栈 创建 docker-compose.yaml：\nservices: banyandb: image: apache/skywalking-banyandb:0.10.0 container_name: banyandb ports: - \u0026#34;17912:17912\u0026#34; command: standalone --stream-root-path /tmp/stream-data --measure-root-path /tmp/measure-data healthcheck: test: [\u0026#34;CMD-SHELL\u0026#34;, \u0026#34;wget -qO- http://localhost:17913/api/healthz || exit 1\u0026#34;] interval: 5s timeout: 3s retries: 10 oap: image: apache/skywalking-oap-server:10.4.0 container_name: oap depends_on: banyandb: condition: service_healthy ports: - \u0026#34;11800:11800\u0026#34; - \u0026#34;12800:12800\u0026#34; environment: SW_STORAGE: banyandb SW_STORAGE_BANYANDB_TARGETS: banyandb:17912 healthcheck: test: [\u0026#34;CMD-SHELL\u0026#34;, \u0026#34;bash -c \u0026#39;echo \u0026gt; /dev/tcp/localhost/12800\u0026#39; || exit 1\u0026#34;] interval: 10s timeout: 5s retries: 30 start_period: 60s ui: image: apache/skywalking-ui:10.4.0 container_name: ui depends_on: oap: condition: service_healthy ports: - \u0026#34;8080:8080\u0026#34; environment: SW_OAP_ADDRESS: http://oap:12800 aigw: image: envoyproxy/ai-gateway-cli:latest container_name: aigw depends_on: oap: condition: service_healthy environment: - OPENAI_BASE_URL=http://host.docker.internal:11434/v1 - OPENAI_API_KEY=unused - OTEL_SERVICE_NAME=my-ai-gateway - OTEL_EXPORTER_OTLP_ENDPOINT=http://oap:11800 - OTEL_EXPORTER_OTLP_PROTOCOL=grpc - OTEL_METRICS_EXPORTER=otlp - OTEL_LOGS_EXPORTER=otlp - OTEL_METRIC_EXPORT_INTERVAL=5000 - OTEL_RESOURCE_ATTRIBUTES=job_name=envoy-ai-gateway,service.instance.id=aigw-1,service.layer=ENVOY_AI_GATEWAY ports: - \u0026#34;1975:1975\u0026#34; extra_hosts: - \u0026#34;host.docker.internal:host-gateway\u0026#34; command: [\u0026#34;run\u0026#34;] 启动所有服务：\ndocker compose up -d 等待所有服务变为健康状态（BanyanDB 先启动，然后是 OAP，最后是 UI 和 AI Gateway）：\ndocker compose ps aigw 服务的关键 OTLP 配置：\n环境变量 值 用途 OTEL_SERVICE_NAME my-ai-gateway SkyWalking 中的服务名 OTEL_EXPORTER_OTLP_ENDPOINT http://oap:11800 SkyWalking OAP gRPC 端点 OTEL_EXPORTER_OTLP_PROTOCOL grpc OTLP 传输协议 OTEL_METRICS_EXPORTER otlp 启用指标推送 OTEL_LOGS_EXPORTER otlp 启用访问日志推送 OTEL_RESOURCE_ATTRIBUTES 必须包含：\njob_name=envoy-ai-gateway —— MAL/LAL 规则的路由标签 service.instance.id=\u0026lt;id\u0026gt; —— 实例标识 service.layer=ENVOY_AI_GATEWAY —— 将日志路由到 AI Gateway LAL 规则 MAL 和 LAL 规则在 SkyWalking OAP 中默认启用，不需要额外配置。\n第三步：运行 Demo 应用 创建一个简单的 Python 应用，通过 AI Gateway 发送请求（app.py）。 它混合了普通请求、流式请求（用于产生 TTFT/TPOT 指标）和错误请求（不存在的模型 → HTTP 404，始终会被 LAL 采样策略捕获）：\nimport time, random, requests GATEWAY = \u0026#34;http://localhost:1975\u0026#34; HEADERS = {\u0026#34;Authorization\u0026#34;: \u0026#34;Bearer unused\u0026#34;, \u0026#34;Content-Type\u0026#34;: \u0026#34;application/json\u0026#34;} questions = [ \u0026#34;What is Apache SkyWalking? Answer in one sentence.\u0026#34;, \u0026#34;What is Envoy Proxy used for? Answer in one sentence.\u0026#34;, \u0026#34;What are the benefits of an AI gateway? Answer in two sentences.\u0026#34;, \u0026#34;Explain observability in three sentences.\u0026#34;, ] def chat(model, question, stream=False): resp = requests.post( f\u0026#34;{GATEWAY}/v1/chat/completions\u0026#34;, json={\u0026#34;model\u0026#34;: model, \u0026#34;messages\u0026#34;: [{\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: question}], \u0026#34;stream\u0026#34;: stream}, headers=HEADERS, timeout=60, stream=stream, ) if stream: chunks = [] for line in resp.iter_lines(): if line: chunks.append(line.decode()) return resp.status_code, f\u0026#34;[streamed {len(chunks)} chunks]\u0026#34; return resp.status_code, resp.json() while True: r = random.random() if r \u0026lt; 0.2: # Error request: non-existent model triggers 404 status, body = chat(\u0026#34;non-existent-model\u0026#34;, \u0026#34;hello\u0026#34;) print(f\u0026#34;[error] model=non-existent-model status={status}\u0026#34;) elif r \u0026lt; 0.5: # Streaming request — generates TTFT and TPOT metrics q = random.choice(questions) status, info = chat(\u0026#34;llama3.2:1b\u0026#34;, q, stream=True) print(f\u0026#34;[stream] status={status} {info}\u0026#34;) else: # Normal non-streaming request q = random.choice(questions) status, body = chat(\u0026#34;llama3.2:1b\u0026#34;, q) answer = body.get(\u0026#34;choices\u0026#34;, [{}])[0].get(\u0026#34;message\u0026#34;, {}).get(\u0026#34;content\u0026#34;, \u0026#34;\u0026#34;)[:80] tokens = body.get(\u0026#34;usage\u0026#34;, {}) print(f\u0026#34;[ok] status={status} tokens={tokens} answer={answer}...\u0026#34;) time.sleep(random.randint(20, 30)) 运行：\npip install requests python app.py 应用通过 1975 端口与 AI Gateway 通信，AI Gateway 再路由到 Ollama。每次请求都会产生 GenAI 指标（Token 用量、延迟、TTFT、TPOT）和访问日志，由网关通过 OTLP 推送到 SkyWalking。\n错误请求（不存在的模型 → HTTP 404）始终会被访问日志采样策略捕获，所以在 SkyWalking 的日志视图中一定能看到。\n第四步：在 SkyWalking UI 中查看 打开 http://localhost:8080，选择 GenAI \u0026gt; Envoy AI Gateway 菜单。\n服务列表显示 my-ai-gateway，可以一览 CPM、延迟和 Token 速率：\n点击进入服务详情，查看完整仪表盘——请求 CPM、延迟（平均值 + 百分位数）、输入/输出 Token 速率、TTFT 和 TPOT：\nProviders 标签页按 AI 提供商维度展示指标：\nModels 标签页展示每个模型的指标，包括 TTFT 和 TPOT（仅流式请求）。注意 unknown 模型条目——这些就是使用不存在模型的错误请求：\nLog 标签页展示访问日志。采样策略会丢弃正常的成功响应，但始终保留错误（HTTP 404）和高 Token 消耗的请求：\n清理 docker compose down Kubernetes 生产部署 生产环境中，Envoy AI Gateway 作为完整的 Kubernetes 控制器运行，以 Envoy Gateway 作为控制面。详见 Envoy AI Gateway 入门指南。\nOTLP 配置方式相同——在 AI Gateway 的 External Processor 上设置 OTEL_* 环境变量，指向 SkyWalking OAP 的 gRPC 端口（11800）。详见 SkyWalking Envoy AI Gateway 监控文档。\n不用 AI 网关也能做 GenAI 可观测 并非所有场景都需要 AI 网关。如果你的应用直接调用 LLM 提供商，SkyWalking 10.4.0 也提供了基于 Virtual GenAI 层的 GenAI 可观测方案。\n任何接入了 SkyWalking、OpenTelemetry 或 Zipkin 探针的应用都能使用这个功能。只要 Trace 中携带 gen_ai.* 标签（遵循 OpenTelemetry GenAI 语义约定），SkyWalking 就能从客户端视角推导出每提供商、每模型的指标：延迟、Token 用量、成功率和预估费用。\n对于 Java 应用，SkyWalking Java Agent（9.7+）内置了 Spring AI 插件，自动为 13+ 提供商（OpenAI、Anthropic、AWS Bedrock、Google GenAI、DeepSeek、Mistral 等）的调用注入正确的 gen_ai.* Span 标签——不需要改代码。\n这与上面介绍的 Envoy AI Gateway 监控是不同的使用场景：\nEnvoy AI Gateway 层：基础设施级可观测——网关视角，覆盖所有流量。适合负责集中 AI 路由的平台团队。 Virtual GenAI 层：应用级可观测——每个应用自己看到的 LLM 调用情况。适合没有集中网关的团队，或者需要按应用维度跟踪费用的场景。 参考资料 Envoy AI Gateway —— 项目官网和文档 Envoy AI Gateway CLI —— 本地开发用的独立运行模式 SkyWalking Envoy AI Gateway 监控 —— OAP 配置文档 SkyWalking Virtual GenAI —— 客户端侧 GenAI 可观测 OpenTelemetry GenAI 语义约定 —— 两个项目共同遵循的指标/属性标准 ","excerpt":"\u003ch2 id=\"问题llm-流量缺乏统一观测\"\u003e问题：LLM 流量缺乏统一观测\u003c/h2\u003e\n\u003cp\u003eLLM 流量正在成为生产基础设施中不可忽视的一部分。团队同时在调用 OpenAI、Anthropic、AWS Bedrock、Azure OpenAI、Google …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-04-02-envoy-ai-gateway-monitoring/","title":"用 Apache SkyWalking 监控 Envoy AI Gateway"},{"body":"SkyWalking APM 10.4.0 is released. Go to downloads page to find release tars.\nProject Introduce OAL V2 engine with immutable AST models, type-safe enums, precise error location reporting, and clean separation between parsing and code generation. Introduce MAL/LAL/Hierarchy V2 engine — replace Groovy-based DSL runtime with ANTLR4 parser + Javassist bytecode generation. Fail-fast compilation at startup — syntax and type errors are caught immediately instead of at first execution. Thread-safe generated classes with no ThreadLocal or shared mutable state. JMH benchmarks confirm v2 runtime speedups: MAL execute ~6.8x, LAL compile ~39x / execute ~2.8x, Hierarchy execute ~2.6x faster than Groovy v1. Breaking Change — LAL: remove slowSql {} and sampledTrace {} sub-DSLs from the grammar. Replaced by the configurable outputType mechanism. An explicit sink {} block is now required for data to be persisted. Add def local variable support in LAL extractor with toJson() and toJsonArray() built-in functions, null-safe navigation, method chaining with compile-time type inference, and explicit type cast via as. Breaking Change — LALOutputBuilder.init() signature changed from init(LogData, NamingControl) to init(LogData, Optional\u0026lt;Object\u0026gt; extraLog, NamingControl). Support building, testing, and publishing with Java 25. Add library-batch-queue module — a partitioned, self-draining queue with type-based dispatch, adaptive partitioning, idle backoff, and throughput-weighted drain rebalancing. Replace DataCarrier with BatchQueue for L1 metrics aggregation, L2 metrics persistence, TopN persistence, all three exporters, and gRPC remote client. Total OAP threads reduced from 150+ to ~72 (~50% reduction). Remove library-datacarrier-queue module. Add virtual thread support (JDK 25+) for gRPC and Armeria HTTP server handler threads. On JDK 25+, all 11 thread pools share ~9 carrier threads instead of up to 1,400+ platform threads. Change default Docker base image to JDK 25 (eclipse-temurin:25-jre). JDK 11 kept as -java11 variant. Fix /debugging/config/dump may leak sensitive information if there are second level properties in the configuration. OAP Server KubernetesCoordinator: make self instance return real pod IP address instead of 127.0.0.1. Fix KubernetesCoordinator self-endpoint race condition. Enhance the alarm kernel with recovered status notification capability. Fix BrowserWebVitalsPerfData clsTime to cls and make it double type. Fix range matrix and scalar binary operation in PromQL. Add LatestLabeledFunction for meter. MAL Labeled metrics support additional attributes. Add support for OpenSearch/ElasticSearch client certificate authentication. Fix BanyanDB logs paging query. Replace BanyanDB Java client with native implementation. Fix trace profiling query time range condition. BrowserErrorLog, OAP Server generated UUID to replace the original client side ID. MQE: fix multiple labeled metric query and ensure no results are returned if no label value combinations match. Fix BrowserErrorLog BanyanDB storage query order. BanyanDB Client: Property query support Order By. MQE: trim the label values condition for the labeled metrics query. PromQL service: fix time parse issue when using RFC3339 time format for querying. Envoy metrics service receiver: support adapter listener metrics. Envoy metrics service receiver: support config MAL rules files. Fix HttpAlarmCallback creating a new HttpClient on every alarm post() call, leaking NIO selector threads. Add SharedKubernetesClient singleton to replace 9 separate KubernetesClientBuilder().build() calls. Reduce Armeria HTTP server event loop threads. All 7 HTTP servers now share one event loop group. Add the spring-ai components and the GenAI layer. Support TraceQL and Tempo API for Zipkin and SkyWalking native trace query. Remove initExp from MAL configuration. Activate otlp-traces handler in receiver-otel by default. Support Virtual-GenAI monitoring. Fix on-demand pod log parsing failure by replacing invalid DateTimeFormatter pattern with ISO_OFFSET_DATE_TIME. Fix Zipkin receiver compatibility with application/x-protobuf Content-Type. Support Envoy AI Gateway observability (SWIP-10): new ENVOY_AI_GATEWAY layer with MAL/LAL rules for GenAI metrics and access log sampling via OTLP. OTel metric receiver: convert data point attribute dots to underscores. OTel log handler: prefer service.instance.id over service.instance with fallback. Support virtual GenAI analysis for otlp and zipkin traces. Fix BanyanDB time range overflow in profile thread snapshot query. UI Fix the missing icon in new native trace view. Enhance the alert page to show the recovery time of resolved alerts. Implement a common pagination component. Add the coldStage to the Duration for queries. Add the GenAI icon to Topology. Add the gen-ai menu. Fix: set the step to SECOND in the duration for Log/Trace/Alarm/Tag. Documentation Restructure docs/README.md for better navigation with high-level documentation overview. Move Marketplace as a top-level menu section. Restructure agent compatibility page with OAP 10.x focus. Remove outdated FAQ docs and \u0026ldquo;since 7/8/9.x\u0026rdquo; version statements. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking APM 10.4.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch5 id=\"project\"\u003eProject\u003c/h5\u003e\n\u003cul\u003e\n\u003cli\u003eIntroduce OAL …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-10.4.0/","title":"Release Apache SkyWalking APM 10.4.0"},{"body":"SkyWalking BanyanDB 0.10.0 is released. Go to downloads page to find release tars.\nFeatures Remove Bloom filter for dictionary-encoded tags. Implement BanyanDB MCP. Support deleting non-entity tags when updating the schema. Remove check requiring tags in criteria to be present in projection. Add sorted query support for the Property. Update bydbQL to add sorted query support for the Property. Remove the windows arch for binary and docker image. Support writing data with specifications. Persist series metadata in liaison queue for measure, stream and trace models. Update the dump tool to support analyzing the parts with smeta files. Add replication integration test for measure. Activate the property repair mechanism by default. Add snapshot time retention policy to ensure the snapshot only can be deleted after the configured minimum age(time). Breaking Change: Change the data storage path structure for property model: From: \u0026lt;data-dir\u0026gt;/property/data/shard-\u0026lt;id\u0026gt;/... To: \u0026lt;data-dir\u0026gt;/property/data/\u0026lt;group\u0026gt;/shard-\u0026lt;id\u0026gt;/... Add a generic snapshot coordination package for atomic snapshot transitions across trace and sidx. Support map-reduce aggregation for measure queries: map phase (partial aggregation on data nodes) and reduce phase (final aggregation on liaison). Add eBPF-based KTM I/O monitor for FODC agent. Support relative paths in configuration. Support \u0026rsquo;none\u0026rsquo; node discovery and make it the default. Support server-side element ID generation for stream writes when clients omit element_id. Implement entire group deletion. Bug Fixes Fix the wrong retention setting of each measure/stream/trace. Fix server got panic when create/update property with high dist usage. Fix incorrect key range update in sidx part metadata. Fix panic in measure block merger when merging blocks with overlapping timestamps. Fix unsupported empty string tag bug. Fix duplicate elements in stream query results by implementing element ID-based deduplication across scan, merge, and result building stages. Fix data written to the wrong shard and related stream queries. Fix the lifecycle panic when the trace has no sidx. Fix panic in sidx merge and flush operations when part counts don\u0026rsquo;t match expectations. Fix trace queries with range conditions on the same tag (e.g., duration) combined with ORDER BY by deduplicating tag names when merging logical expression branches. Fix sidx tag filter range check returning inverted skip decision and use correct int64 encoding for block min/max. Ignore take snapshot when no data. Fix measure standalone write handler resetting accumulated groups on error, which dropped all successfully processed events in the batch. Fix memory part reference leak in mustAddMemPart when tsTable loop closes. Fix memory part leak in syncPartContext Close and prevent double-release in FinishSync. Fix segment reference leaks in measure/stream/trace queries and ensure chunked sync sessions close part contexts correctly. Fix duplicate query execution in distributed measure Agg+TopN queries by enabling push-down aggregation, removing the wasteful double-query pattern. Fix nil pointer panic in segment collectMetrics during shutdown. Fix entity tag handling in trace filter to prevent TagIdx index mismatch when filtering with both entity and non-entity tags. Document Add read write benchmark document for 0.9.0 release. Add design of KTM. Add FODC overview doc. Remove Java client doc, and recreate client APIs docs. Add common issue documentation. Chores Upgrade Node.js support from 20.12 to 24.6.0, and align CI, license checks, and documentation Add Claude Code skill for vendor dependency updates. Upgrade Go vendor dependencies and sync BPF2GO_VERSION with cilium/ebpf library. ","excerpt":"\u003cp\u003eSkyWalking BanyanDB 0.10.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"features\"\u003eFeatures\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eRemove …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-0-10-0/","title":"Release Apache SkyWalking BanyanDB 0.10.0"},{"body":"SkyWalking Client JS 1.1.0 is released. Go to downloads page to find release tars.\nOptimze E2E. Bump up dependencies. Add the NPM_TOKEN for publishing. Optimize the XHR interceptor to preserve the prototype chain by setting the prototype of custom constructor. Fix metric name and value. Release Artifacts Source release: skywalking-client-js-1.1.0-src.tgz npm package: skywalking-client-js More Details Docs and tag: v1.1.0 ","excerpt":"\u003cp\u003eSkyWalking Client JS 1.1.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eOptimze E2E. …\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-client-js-1-1-0/","title":"Release Apache SkyWalking Client JS 1.1.0"},{"body":"SkyWalking MCP 0.1.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed Initial release of the swmcp binary (SkyWalking MCP server). Support for three MCP transport modes: stdio, sse, and streamable. Integration with Apache SkyWalking OAP via GraphQL, including: Traces, logs, metrics, topology, alarms, and events query tools. MQE (Metrics Query Extension) tools using the OAP /graphql endpoint. Prompt support for trace and log analysis and utility workflows. Embedded documentation and dynamic metrics resources for MQE. Makefile targets for build, lint, license checks, and Docker image creation. Docker image is available at apache/skywalking-mcp. ","excerpt":"\u003cp\u003eSkyWalking MCP 0.1.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-mcp-0-1-0/","title":"Release Apache SkyWalking MCP 0.1.0"},{"body":"SkyWalking GraalVM Distro 0.2.1 is released. Go to downloads page to find release tars.\nThis is the first official Apache release of the GraalVM Distro.\nChanges Apache SkyWalking GraalVM Distro is a GraalVM native image distribution of the Apache SkyWalking OAP server. It compiles the full-featured OAP server into a single native binary (~200MB), delivering instant startup and reduced memory footprint compared to the standard JVM distribution.\nBuild-time OAL engine: pre-compile ~1285 metrics/builder/dispatcher classes via Javassist at Maven compile time. Build-time MAL compiler: pre-compile ~1250 MAL expressions from 71 YAML rule files into MalExpression classes. Build-time LAL compiler: pre-compile ~10 LAL scripts from 8 YAML files into LalExpression classes. Build-time Hierarchy compiler: pre-compile ~4 hierarchy matching rules into BiFunction classes. Build-time MeterSystem: pre-generate ~1188 meter function subclasses via Javassist. Auto-generate reflect-config.json by scanning HTTP handlers, GraphQL resolvers/types, config POJOs, and DSL manifests. Replace Groovy runtime with pure Java: MAL DSL, LAL DSL, and Hierarchy rules all use ANTLR4 + Javassist v2 engines. Replace Guava ClassPath.from() classpath scanning with build-time manifests for annotations, dispatchers, and source receivers. Replace Field.setAccessible() reflection in config loading with Lombok @Setter-based property copying. Replace ServiceLoader SPI discovery with direct provider wiring in ModuleDefine. Add TraceQL module (Tempo-compatible trace query API) with Zipkin and SkyWalking datasource support. JVM distribution: repackaged OAP server with all replacement classes via maven-shade-plugin. Native distribution: single binary (~200MB) with config files, LICENSE, NOTICE, and third-party licenses. Docker image available on both GHCR and Docker Hub, with multi-arch support for linux/amd64 and linux/arm64. macOS native binary: build locally via make native-image on macOS. Sync SkyWalking submodule to upstream commit 64a1795d8a. ","excerpt":"\u003cp\u003eSkyWalking GraalVM Distro 0.2.1 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cp\u003eThis is the …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-graalvm-distro-0-2-1/","title":"Release Apache SkyWalking GraalVM Distro 0.2.1"},{"body":"SkyWalking PHP 1.1.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed chore(ci): Replace archived actions-rs actions by @assignUser in #133 Update dependencies and refactor Kafka reporter by @jmjoy in #134 Update phper dependencies to support PHP 8.5 by @jmjoy in #137 Enable zend observer by default for PHP 8+ by @jmjoy in #138 Release SkyWalking PHP 1.1.0 by @jmjoy in #140 New Contributors @assignUser made their first contribution in #133 Full Changelog: v1.0.0\u0026hellip;v1.1.0\nPECL skywalking_agent 1.1.0\n","excerpt":"\u003cp\u003eSkyWalking PHP 1.1.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-php-1-1-0/","title":"Release Apache SkyWalking PHP 1.1.0"},{"body":"以 SkyWalking GraalVM Distro 为例，看 AI Coding 如何把一批探索性 PoC 打磨成一条可重复的迁移流水线。\n这个项目给我最大的启发，不是 AI 能写多少代码，而是 AI Coding 改变了架构设计的试错成本。当一个想法可以很快做成 PoC、跑起来验证、不行就推翻重来时，架构师就更有机会逼近自己真正想要的设计，而不是过早停在“团队现在做得出来”的折中方案上。\n这种变化在成熟开源系统里尤其重要。Apache SkyWalking OAP 长期以来一直是一个功能强大且经过生产验证的可观测性后端，但大型 Java 平台该有的问题它一个不少：运行时字节码生成、重反射初始化、classpath 扫描、基于 SPI 的模块装配，以及动态 DSL 执行——这些机制方便扩展，但做 GraalVM Native Image 时全是障碍。\nSkyWalking GraalVM Distro 的出现，源于我们把这个挑战当成一个架构设计问题来处理，而不是一次性的移植工程。目标不仅是让 OAP 能以原生二进制运行，更是把 GraalVM 迁移本身做成一条可重复执行、能够持续跟上上游演进的自动化流水线。\n如果你想看完整的技术设计、基准数据和上手方式，请阅读配套文章：SkyWalking GraalVM Distro：设计与基准测试。\n从停滞的想法到可运行的系统 这件事其实很多年前就开始了。在这个仓库创建不久之后，yswdqz 曾花了数个月探索迁移方案。真正做下来才发现，这个项目远比 GraalVM 文档里列出的那些单点限制复杂得多，这项工作最终也因此搁置了很多年。\n这段停滞很重要。缺少的并不是想法。成熟维护者通常从来不缺想法，真正稀缺的，是把这些想法真正做出来的时间、人力和精力。即使架构师已经看到了几条很有前景的路线，有限的开发资源也会迫使大家更早做出权衡：优先选择实现成本最低的方案，而不是那个更干净、更可复用、更经得起未来变化的方案。\n这种情况非常普遍，并不特殊。在开源社区里，很多工作依赖志愿者或有限的企业赞助；在商业产品里，约束的形式不同，但本质仍然一样：路线图承诺、团队规模和交付压力都会让工程资源始终紧张。在这两种环境里，很多好想法被放弃，并不是因为它们错了，而是因为要把它们真正验证清楚、实现完整，成本太高。\n还有一个同样重要的约束：架构师通常同时也是非常资深的工程师，而不是一个可以全职扑在实现细节上的人。问题在于个人编码精力有限、时间高度碎片化，同时还要在代码尚未出现之前，不断向其他资深工程师解释自己的设计意图。传统上，这种解释主要通过图、文档和沟通完成。它很慢、信息损失大，而且充满不确定性。我们都体验过“传话游戏”：哪怕是很简单的意思，也很容易被误解，而等误解真正暴露出来时，时间已经过去很多了。\n到了 2025 年末，AI Coding 让”同时尝试多条路线”这件事终于变得现实。我们不必再因为实现能力稀缺而过早接受折中，而是可以在多个设计之间来回切换，用代码验证，快速淘汰弱方案，持续迭代，直到架构本身变得足够稳固、足够实用、足够高效。\n这种设计自由度至关重要。GraalVM 文档对单个限制讲得很清楚，但成熟 OSS 平台遇到的是一整套彼此牵连的系统性问题。只修补一个动态机制远远不够。要让 native image 真正落地，我们必须把整类运行时行为改造成构建期产物和自动生成的元数据。\n在这条路的早期历史中，还有一座非常具体的大山。那时上游 SkyWalking 仍然大量依赖 Groovy 来处理 LAL、MAL 和 Hierarchy 脚本。理论上，这只不过是另一个“不支持运行时动态行为”的例子；但在实践中，Groovy 是整条路径上最大的障碍。它不仅意味着脚本执行，还意味着一整套在 JVM 里极其便利、在 native image 里却极其不友好的动态模型。\n为了跨过这道坎，我们围绕 AOT-first 模式重新设计了 OAP 的核心引擎。早期实验必须直接面对 Groovy 时代的运行时行为，并尝试不同的脚本编译方案来绕过去。最终方案走得更远：对齐上游编译器流水线，把动态生成前移到构建期，并引入自动化机制，让这条迁移路径在上游持续演进时依然保持可控。具体来说，就是把 OAL、MAL、LAL 和 Hierarchy 的生成过程变成构建期预编译器的输出，而不是继续保留为启动期的动态行为。\nAI Coding 如何改写架构迭代 这次转变的关键，并不只是“写代码更快了”。AI 真正改变的，是想法、原型、验证和重设计之间来回迭代的速度。围绕同一个问题，我们可以很快做出几个可运行的 PoC，迅速淘汰不成立的方向，再把值得保留的抽象慢慢沉淀成一套连贯的迁移系统。\n这并不会削弱人的架构价值，反而会放大它。哪些行为应该前移到构建期，哪些地方应该保留可配置性，哪里应该引入 same-FQCN 替换，如何让上游同步保持可控，以及哪些抽象值得不惜代价保留下来，这些判断仍然只能由人来做。不同的是，AI 的速度让我们终于有机会把这些更好的设计真正做出来，而不是过早退回到更简单、也更差的折中方案。\n这才是软件架构师工作方式真正发生变化的地方。过去，架构师往往已经知道更干净的方向在哪里，但有限的工程产能会逼着那个愿景退回到一个更便宜的妥协方案。现在，架构师在某种意义上又重新变回了“能快速动手的人”：可以直接用代码把思路搭出来，把高层抽象落成接口，再用真实运行的实现去证明设计。\n这不仅改变了实现，也改变了沟通方式。在开源里，我们常说：talk is cheap, show me the code。在 AI Coding 时代，“把代码拿出来”这件事变得容易多了。设计不再那么依赖一个缓慢的、自上而下的翻译过程：从想法到文档，再到解释，再到实现。代码可以更早出现，也可以更早跑起来。\n这也让其他资深工程师受益。他们不必只靠图、会议或长篇解释来还原整个设计，而是可以直接审查抽象、阅读真实代码、运行它、质疑它，并在具体实现上一起打磨。这让架构协作更快、更清晰，也少了很多沟通误差。\n也正因为如此，我总觉得今天很多 AI 讨论有点跑偏。很多项目确实很有趣、也很好玩，拿来体验当然没问题，但高级工程工作并不会因为“给代码库接了个 agent”就自然变好。真正重要的，不是哪个 demo 看起来最炫，而是哪些工程能力真的被放大了，同时软件开发本身的纪律有没有被保留下来。\n对于架构师和资深工程师来说，这里真正重要的能力包括：\n快速做对比式原型验证：不是只用 slides 和文档去论证某个想法，而是直接把多个方案做成可运行代码来比较。 大规模代码理解能力：能在大量模块之间快速阅读，同时保持对整个系统的全局认识。 系统性的重构能力：把基于反射、依赖运行时动态行为的路径，系统性地改造成适配 AOT 约束的设计。 搭建自动化的能力：当一个迁移步骤在每次上游同步时都必须重做一次，靠手工处理本身就很费时费力，而且越往后只会越累。AI 让我们真正有条件去投资生成器、清单、一致性检查和漂移检测，把重复的人力劳动变成可重复的自动化流程。 大范围审查能力：在很大的代码面上检查边界条件、兼容性约束，以及方案是否经得起反复执行。 这些能力也都体现在最终的设计结果里。same-FQCN 替换为 GraalVM 特定行为建立了清晰、受控的边界；反射元数据不再依赖手工维护的猜测清单，而是直接从构建产物中生成；各种清单机制和漂移检测，则把原本模糊的“上游同步风险”变成了显式的工程工作流。\n对于初级工程师，我觉得这里的启发同样重要。AI 不会让架构设计、系统约束、接口设计、测试和可维护性这些基本功变得不重要。恰恰相反，这些能力只会变得更重要，因为它们决定了“被加速的实现”最终产出的是一个可持续演进的系统，还是只是更快地制造出更多代码。真正的杠杆来自工程判断力，而不是新鲜感。\nClaude Code 和 Gemini AI 在整个过程中都扮演了工程加速器的角色。在 GraalVM Distro 这个项目里，它们具体帮我们做了几件事：\n把迁移思路直接做成可运行代码：不是争论哪个方向可能行得通，而是把多个真实原型做出来、跑起来、比较掉，把不成立的方向淘汰掉。 重构重反射、重动态的代码路径：把不适合运行时的模式系统性替换成 AOT 友好的实现方式。 让上游同步真正可持续：每次 distro 从上游 SkyWalking 拉取变更后，元数据扫描、配置再生成和重新编译都必须再来一次。AI 帮助我们把这些过程做成流水线，使每次同步都变成一个可控、且大部分自动化的过程，而不是一次比一次更长的手工重复劳动。 在大范围内审查逻辑和边界情况：特别是在功能对等性比纯实现速度更重要的地方。 最终产出的，不只是一次大重写，而是一套可重复的系统：预编译器、manifest 驱动的加载、反射配置生成、替换边界，以及让上游迁移可审查、可自动化的漂移检测机制。\n如果你想看这种开发方法背后的更广泛背景，可以读这篇文章：在成熟开源大型项目中实践 Agentic Vibe Coding：软件工程与工程控制论还在延续。这篇文章则是这个故事的下一步：不仅是在一个成熟代码库里增强功能，而是重新激活一项曾经停滞的工作，并把它真正做成可运行系统。\n真正改变的到底是什么 这个项目最重要的结果，并不是一张 benchmark 表。基准数据当然属于 distro 本身，而且它们很重要，因为它们证明这套系统是真实可运行的。但对这篇文章来说，更深层的变化发生在方法论层面：AI Coding 改变了我们探索、验证和打磨架构方案的方式。\n过去，架构往往更像一项以文档为主、后面拖着漫长而昂贵实现过程的活动。现在，我们可以更快地在想法、原型、比较和重设计之间切换。这让我们真正有机会去追求更高抽象层次的方案，保留更干净的边界，并建设那些让迁移过程可持续维护的自动化机制。\n这项工作的技术证据，就是 SkyWalking GraalVM Distro 本身：它不仅是一个可运行的系统，更是一条由预编译器、自动生成的反射元数据、受控替换边界和漂移检查组成的迁移流水线。基准数据之所以重要，是因为它们证明这套系统在实践里是成立的；但从架构角度看，真正的结果是：这次迁移不再是一场一次性的移植，而是变成了一套可重复执行的系统工程。关于完整测试方法、原始数据和技术设计，请阅读配套文章：SkyWalking GraalVM Distro：设计与基准测试。\n项目仓库位于 apache/skywalking-graalvm-distro。我们欢迎社区成员测试这个新发行版、提交 issue，并帮助它逐步走向生产可用。\n对我来说，更深层的启发并不止于这个发行版。AI Coding 不会让架构变得不重要，反而会让架构更值得被认真追求。当实现速度提升到一定程度时，我们终于有机会在真实代码里验证更多想法，保留那些真正好的抽象，并把那些过去常常因为投入太大而半途妥协的系统真正做出来。\n对于资深工程师来说，瓶颈正在从单纯的代码实现速度，转向品味、系统判断力，以及定义稳定边界的能力。对于初级工程师来说，真正该走的路不是追逐每一种看上去都很刺激的 AI 工作流，而是把基础能力练得更扎实，让加速真正产生复利：理解需求、阅读陌生系统、质疑假设，并识别出在系统快速变化时仍然必须保持正确的那些部分。AI Coding 降低了验证好设计的代价，但并没有降低工程判断本身的门槛。\n","excerpt":"\u003cp\u003e\u003cem\u003e以 SkyWalking GraalVM Distro 为例，看 AI Coding 如何把一批探索性 PoC 打磨成一条可重复的迁移流水线。\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"./graph.jpg\" alt=\"graph.jpg\"\u003e\u003c/p\u003e\n\u003cp\u003e这个项目给我最大的启发，不是 AI 能写多少代码，而是 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-03-13-how-ai-changed-the-economics-of-architecture/","title":"AI Coding 如何重塑软件架构师的工作方式"},{"body":"这篇文章会完整介绍我们如何把 Apache SkyWalking OAP 迁移到 GraalVM Native Image。目标不是做一次性移植，而是把这件事做成一套能持续跟上上游演进的流程。\n如果你想看这项工作的更大背景，以及 AI Coding 如何让这个项目真正做得出来，请阅读：AI Coding 如何重塑软件架构师的工作方式。\n为什么 GraalVM 在这里是刚需 GraalVM Native Image 可以把 Java 应用做 Ahead-of-Time（AOT）编译，生成独立可执行文件。对于像 SkyWalking OAP 这样的可观测性后端来说，这不是“锦上添花”的性能优化，而是明确的工程刚需。\n可观测性平台必须是基础设施中最可靠的部分。它必须在自己要观测的那些故障发生时依然存活。在云原生环境里，工作负载会不断扩缩容、迁移和重启，负责观测一切的后端本身不能还是那个启动慢、空闲占用大、恢复缓慢的重型进程。\n我们的基准测试结果让这个结论变得非常具体：\n**启动时间：**约 5 ms 对比约 635 ms。在 Kubernetes 集群里，当 OAP Pod 被驱逐或重新调度时，635 ms 的差距意味着这段时间里的遥测数据可能会丢失。5 ms 的情况下，新 Pod 往往在大部分客户端还没感知到中断之前就已经重新开始接收数据了。 **空闲内存：**约 41 MiB 对比约 1.2 GiB。可观测性后端是 24/7 常驻运行的。在多租户或边缘部署场景里，基础 RSS 降了 97%，可以放进更小的节点，而不再必须占用一台专用机器。 **负载下内存：**在 20 RPS 下约 629 MiB 对比约 2.0 GiB。生产级负载下内存降了 70%，直接对应更少的节点、更低的云账单，以及在后端本身成为扩容瓶颈之前更多的余量。 **没有预热惩罚：**峰值吞吐可以更早发挥出来。JVM 的 JIT 编译器往往需要数分钟流量才能完成热点优化，在这段时间里，尾延迟更差，数据处理也会滞后。原生二进制没有同样的阶段。 **更小的攻击面：**不再需要完整 JDK 运行时，需要跟踪和修补的 CVE 也就少了很多。对于一个会接收整个集群所有服务数据的组件来说，这一点很重要。 这些都不是“小修小补”。它们直接改变了哪些部署形态开始变得可行：无服务器形态的可观测性后端、边车式采集模型、内存预算极其紧张的边缘节点。只有当后端足够轻、足够快时，这些方案才真正有落地空间。\n挑战：一个成熟、动态特性很多的 Java 平台 SkyWalking OAP 身上有大型 Java 平台的所有典型问题：运行时字节码生成、重反射初始化、classpath 扫描、基于 SPI 的模块装配，以及动态 DSL 执行。这些机制方便扩展，但做 GraalVM native image 时全是障碍。\nGraalVM 文档中列出的限制，只是问题的开始。在一个成熟的 OSS 平台里，这些限制会深深缠绕在多年积累下来的运行时设计决策中。常规的 GraalVM native image 很难处理运行时类生成、反射、动态发现和脚本执行，而这些在 SkyWalking OAP 中都不是零散存在的，它们本来就是系统设计的一部分。\n在这个发行版的早期历史里，还有一座非常具体的大山。那时上游 SkyWalking 仍然高度依赖 Groovy 来处理 LAL、MAL 和 Hierarchy 脚本。理论上，它只是另一个“不支持运行时动态”的组件；但在实践里，Groovy 是整条路径上最大的障碍。它不仅仅是脚本执行问题，而是代表着一整套在 JVM 世界里极其便利、在 native image 世界里极其不友好的动态模型。\n设计目标：让迁移这件事可以重复做 设计目标不是”把 native-image 跑通一次就完”，而是做出一套能反复用、能长期维护的迁移系统：\n把运行时生成的产物前移到构建期。 OAL、MAL、LAL、Hierarchy 规则，以及 meter 相关的生成类，都在构建期完成编译并打包，而不是等到启动时才动态生成。 用确定性的加载机制替代动态发现。 classpath 扫描和运行时注册路径被转换为基于 manifest 的加载方式。 减少运行时反射，并在构建期生成 native 元数据。 反射配置不再依赖人工维护的猜测清单，而是根据真实 manifest 和扫描结果生成。 让上游同步边界保持清晰。 same-FQCN replacements 会被显式打包、列清单，并通过陈旧性检查守住边界。 让变化第一时间暴露出来。 一旦上游 provider、规则文件或被替换的源文件发生变化，测试就会失败，迫使我们做显式审查。 这才是最关键的架构转变。好的抽象和前瞻性，在 AI 时代并没有变得不重要，反而变得更重要了，因为它们决定了 AI 带来的速度，最终产出的是一个可维护的系统，还是一堆膨胀得更快的代码。\n把运行时动态行为变成构建期产物 SkyWalking OAP 里有多个在 JVM 世界里很自然、但在 native image 里很棘手的动态子系统：\nOAL 会在运行时生成类。 LAL、MAL 和 Hierarchy 在历史上与大量基于 Groovy 的运行时行为绑定在一起，这也是早期 distro 工作中最难处理的阻碍之一。 MAL、LAL 和 Hierarchy 规则依赖运行时编译行为。 基于 Guava 的 classpath 扫描会发现注解、dispatcher、decorator 和 meter function。 基于 SPI 的模块和 provider 发现依赖更动态的运行时环境。 YAML/config 初始化和框架集成依赖反射访问。 在 SkyWalking GraalVM Distro 里，这些问题不是靠零散补丁一个个修掉的，而是被统一收敛到一条构建期流水线里。\n预编译器会在构建过程中运行 DSL 引擎、导出生成类、写入 manifest、序列化配置数据，并生成 native-image 元数据。这样一来，启动时只需要做类加载和注册，不再需要运行时代码生成。运行期之所以能变得更简单，是因为原本的复杂性被前移到了构建期。\n这也是为什么这个项目不只是一次性能优化。我们的设计目标，是把复杂性前移到一个更容易验证、更容易自动化、也更便于反复执行的位置。\nsame-FQCN 替换：一条可控的边界 这个发行版里最实用的设计选择之一，就是使用 same-FQCN 替换类。我们没有依赖模糊的启动技巧，也没有依赖未文档化的加载顺序假设。相反，我们会重新打包 GraalVM 特定 jar，排除原本的上游类，再让替换类占据完全相同的 fully-qualified class name。\n这对可维护性非常关键，因为它建立了一条非常清晰的边界：\n上游类仍然定义行为契约； GraalVM 侧的替换类提供兼容的实现策略； 打包过程则让这次替换变得显式可见。 例如，OAL 的加载过程从运行时编译变成了基于 manifest 的预编译类加载。类似的替换也处理了 MAL 和 LAL DSL 加载、模块装配、配置初始化，以及多个对反射敏感的路径。目标不是把一切都 fork 出去，而是只替换那些运行时模型从根本上不适合 native image 的部分。\n随后，这条边界还会通过测试来守护：测试会对照与 replacement 对应的上游源文件做哈希。当上游改动了这些文件中的任何一个，构建就会失败，并明确告诉我们哪个 replacement 需要重新审查。这样一来，“如何跟上上游”就不再是一个充满焦虑的抽象问题，而变成一项明确、可落地的工程工作。\n反射配置不是猜出来的，而是生成出来的 在很多 GraalVM 迁移项目里，reflect-config.json 最终会变成一个靠经验不断累积的工件。它会越来越大，越来越陈旧，最后没有人真正清楚它是不是完整，也不清楚每一项配置为什么存在。这种模式在一个持续演化的大型 OSS 平台里是无法扩展的。\n在这个发行版里，反射元数据直接从构建产物和扫描结果中生成，包括：\nOAL、MAL、LAL、Hierarchy 以及 meter 生成类的 manifest； 注解扫描得到的类； Armeria HTTP handler； GraphQL resolver 和 schema 映射类型； 被接受的 ModuleConfig 类。 这是一种健康得多的模式。我们不再依赖人去记住所有可能触发反射访问的路径，而是让系统根据真实迁移流水线推导出反射元数据。构建过程本身，成为了事实来源。\n让上游同步变得现实可行 如果这个发行版只是一次性的工程冲刺，那它的意义会小很多。真正困难的事情，是在上游 SkyWalking 继续演进的同时，让它还能持续维护下去。\n这也是为什么仓库里会有一整套显式的清单和漂移检测机制：\nprovider 清单，用来强制新上游 provider 被分类； 规则文件清单，用来强制新 DSL 输入被显式确认； 预编译 YAML 输入的 SHA watcher； 带 GraalVM 特定 replacement 的上游源文件 SHA watcher。 好的抽象不仅仅是代码结构优雅，更在于你是否选择了一种能在未来变化面前继续成立的迁移设计。\n基准测试结果 我们在一台 Apple M3 Max（macOS、Docker Desktop、10 CPUs / 62.7 GB）上，对标准 JVM OAP 和 GraalVM Distro 做了对比测试，两者都连接到 BanyanDB。\n启动测试（Docker Compose，无流量，3 次取中位数） 指标 JVM OAP GraalVM OAP 差异 冷启动时间 635 ms 5 ms 约快 127 倍 热启动时间 630 ms 5 ms 约快 126 倍 空闲 RSS 约 1.2 GiB 约 41 MiB 约降低 97% 启动时间的测量方式，是从 OAP 第一条应用日志时间戳开始，到出现 listening on 11800 日志（即 gRPC 服务 ready）为止。\n持续负载下（Kind + Istio 1.25.2 + Bookinfo，约 20 RPS，2 个 OAP 副本） 在 60 秒预热之后，每 10 秒采样一次，共 30 个样本。\n指标 JVM OAP GraalVM OAP 差异 CPU 中位数（millicores） 101 68 -33% CPU 平均值（millicores） 107 67 -37% 内存中位数（MiB） 2068 629 -70% 内存平均值（MiB） 2082 624 -70% 两个版本报告的 entry-service CPM 一致，说明在这个测试负载下，两者的流量处理能力相同。\n我们每 30 秒通过 swctl 对所有已发现服务收集这些指标： service_cpm、service_resp_time、service_sla、service_apdex、service_percentile。\n完整的基准测试脚本和原始数据位于发行版仓库中的 benchmark/ 目录。\n当前状态 这个项目已经是一个可运行的实验性发行版，托管在独立仓库中：apache/skywalking-graalvm-distro。\n当前发行版有意聚焦在一种现代、高性能的运行模式上：\n存储： BanyanDB 集群模式： Standalone 和 Kubernetes 配置方式： 无配置或 Kubernetes ConfigMap 运行模型： 固定模块集合、预编译产物和 AOT 友好的装配方式 这种聚焦是刻意的。要把迁移做成一套可重复的系统，第一步必须先把边界收清楚，做出一个真正能跑起来的版本，然后再在不失控的前提下逐步扩展。\n快速开始 由于 SkyWalking GraalVM Distro 的设计目标就是追求极致性能，它目前最适合与 BanyanDB 存储后端搭配使用。当前发布的镜像已经可以在 Docker Hub 获取，你可以直接用下面这个 docker-compose.yml 启动整套系统。\nversion: \u0026#39;3.8\u0026#39; services: banyandb: image: ghcr.io/apache/skywalking-banyandb:e1ba421bd624727760c7a69c84c6fe55878fb526 container_name: banyandb restart: always ports: - \u0026#34;17912:17912\u0026#34; - \u0026#34;17913:17913\u0026#34; command: standalone --stream-root-path /tmp/stream-data --measure-root-path /tmp/measure-data --measure-metadata-cache-wait-duration 1m --stream-metadata-cache-wait-duration 1m healthcheck: test: [\u0026#34;CMD\u0026#34;, \u0026#34;sh\u0026#34;, \u0026#34;-c\u0026#34;, \u0026#34;nc -nz 127.0.0.1 17912\u0026#34;] interval: 5s timeout: 10s retries: 120 oap: image: apache/skywalking-graalvm-distro:0.1.1 container_name: oap depends_on: banyandb: condition: service_healthy restart: always ports: - \u0026#34;11800:11800\u0026#34; - \u0026#34;12800:12800\u0026#34; environment: SW_STORAGE: banyandb SW_STORAGE_BANYANDB_TARGETS: banyandb:17912 SW_HEALTH_CHECKER: default healthcheck: test: [\u0026#34;CMD-SHELL\u0026#34;, \u0026#34;nc -nz 127.0.0.1 11800 || exit 1\u0026#34;] interval: 5s timeout: 10s retries: 120 ui: image: ghcr.io/apache/skywalking/ui:10.3.0 container_name: ui depends_on: oap: condition: service_healthy restart: always ports: - \u0026#34;8080:8080\u0026#34; environment: SW_OAP_ADDRESS: http://oap:12800 只需要执行：\ndocker compose up -d 欢迎社区来测试这个新发行版、提交 issue，并帮助我们推动它走向生产可用。\n特别感谢 GraalVM 团队提供的技术基础。\n","excerpt":"\u003cp\u003e\u003cem\u003e这篇文章会完整介绍我们如何把 Apache SkyWalking OAP 迁移到 GraalVM Native Image。目标不是做一次性移植，而是把这件事做成一套能持续跟上上游演进的流程。\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"./graph.jpg\" alt=\"graph.jpg\"\u003e\u003c/p\u003e\n\u003cp\u003e如果 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-03-13-skywalking-graalvm-distro-design-and-benchmarks/","title":"SkyWalking GraalVM Distro：设计与基准测试"},{"body":"SkyWalking GraalVM Distro: A case study in turning runnable PoCs into a repeatable migration pipeline.\nThe most important lesson from this project is not that AI can generate a large amount of code. It is that AI changes the economics of architecture. When runnable PoCs become cheap to build, compare, discard, and rebuild, architects can push further toward the design they actually want instead of stopping early at a compromise they can afford to implement.\nThat shift matters a lot in mature open source systems. Apache SkyWalking OAP has long been a powerful and production-proven observability backend, but it also carries all the realities of a large Java platform: runtime bytecode generation, reflection-heavy initialization, classpath scanning, SPI-based module wiring, and dynamic DSL execution that are friendly to extensibility but hostile to GraalVM native image.\nSkyWalking GraalVM Distro is the result of treating that challenge as a design-system problem instead of a one-off porting exercise. The goal was not only to make OAP run as a native binary, but to turn GraalVM migration itself into a repeatable automation pipeline that can stay aligned with upstream evolution.\nFor the full technical design, benchmark data, and getting-started guide, see the companion post: SkyWalking GraalVM Distro: Design and Benchmarks.\nFrom Paused Idea to Runnable System This journey actually began years ago. Shortly after this repository was created, yswdqz spent several months exploring the transition. The project proved much harder in practice than the individual GraalVM limitations sounded on paper, and the work eventually paused for years.\nThat pause is important. The missing ingredient was not ideas. Mature maintainers usually have more ideas than time. The real constraint was implementation economics. Even when the architect can see several promising directions, limited developer resources force an earlier trade-off: choose the path that is cheapest to implement, not necessarily the path that is cleanest, most reusable, or most future-proof.\nThis is a very common reality, not an exceptional one. In open source communities, much of the work depends on volunteers or limited company sponsorship. In commercial products, the pressure is different but the constraint is still real: roadmap commitments, staffing limits, and delivery deadlines keep engineering resources tight. In both worlds, good ideas are often abandoned not because they are wrong, but because they are too expensive to validate and implement thoroughly.\nThere is another constraint that matters just as much: the architect is usually also a very senior engineer, not a full-time implementation machine. That means limited personal coding energy, fragmented time, and a constant need to explain ideas to other senior engineers before the code exists. Traditionally, that explanation happens through diagrams, documents, and conversations. It is slow, lossy, and unpredictable. We all know some version of the Telephone Game: even simple words are easy to misunderstand, and by the time the misunderstanding becomes visible, a lot of time has already passed.\nWhat changed in late 2025 was that AI engineering made multiple runnable ideas affordable. Instead of picking an early compromise because implementation capacity was scarce, we could switch repeatedly between designs, validate them with code, discard weak directions quickly, and keep iterating until the architecture became solid, practical, and efficient enough to hold.\nThat design freedom was critical. GraalVM documentation gives clear guidance on isolated limitations, but a mature OSS platform hits them as a connected system. Fixing only one dynamic mechanism is not enough. To make native image practical, we had to turn whole categories of runtime behavior into build-time artifacts and automated metadata generation.\nThere was also a very concrete mountain in front of us in the early history of this distro. In the first several commits of the repository, upstream SkyWalking still relied heavily on Groovy for LAL, MAL, and Hierarchy scripts. In theory, that was just one more unsupported runtime-heavy component. In practice, Groovy was the biggest obstacle in the whole path. It represented not only script execution, but a whole dynamic model that was deeply convenient on the JVM side and deeply unfriendly to native image.\nTo bridge the gap, we re-architected the core engines of OAP around an AOT-first model. Earlier experiments had to confront Groovy-era runtime behavior directly and explore alternative script-compilation approaches to get around it. The finalized direction went further: align with the upstream compiler pipeline, move dynamic generation to build time, and add automation so the migration stays controllable as upstream keeps moving. Concretely, that meant turning OAL, MAL, LAL, and Hierarchy generation into build-time precompiler outputs instead of leaving them as startup-time dynamic behavior.\nAI Speed Changed the Design Loop The scale of this transformation was not only about coding faster. AI changed the loop between idea, prototype, validation, and redesign. We could build runnable PoCs for different approaches, throw away weak ones quickly, and preserve the promising abstractions until they formed a coherent migration system.\nThat does not reduce the role of human architecture. It raises the value of it. Human judgment was still required to decide what should become build-time, what should stay configurable, where to introduce same-FQCN replacements, how to keep upstream sync controllable, and which abstractions were worth preserving. But AI speed made it realistic to pursue those better designs instead of settling for a simpler compromise too early.\nThis is the real change in the economics of architecture. In the past, an architect might already know the cleaner direction, but limited engineering capacity often forced that vision back toward a cheaper compromise. Now the architect can return much closer to being a fast developer again: building code, shaping high-abstraction interfaces, and using design patterns to prove the vision directly in the real world.\nThat changes communication as much as implementation. In open source, we often say, talk is cheap, show me the code. With AI engineering, showing the code becomes much more straightforward. The design no longer depends so heavily on a slow top-down translation from idea to documents to interpretation to implementation. The code can appear earlier, and it can run earlier.\nOther senior engineers benefit from this too. They do not need to reconstruct the whole design only from diagrams, meetings, or long explanations. They can review the actual abstraction, see the behavior in code, run it, challenge it, and refine it from something concrete. That makes architectural collaboration faster, clearer, and less lossy.\nThis is also where I think the current AI discussion is often noisy. Many projects are fun, surprising, and worth exploring, but advanced engineering work is not improved merely by attaching an agent to a codebase. The important question is not which demo looks most magical. The important question is which engineering capabilities are actually being accelerated without losing the discipline of software development itself.\nFor architects and senior engineers, the capabilities that mattered most here were:\nFast comparative prototyping: Building several runnable approaches in code instead of defending one idea with slides and documents. Large-scale code comprehension: Reading across many modules quickly enough to keep the whole system in view. Systematic refactoring: Converting reflection-heavy or runtime-dynamic paths into designs that fit AOT constraints. Automation construction: When a migration step must be repeated every upstream sync, doing it manually once is already expensive. Doing it manually again next time is even more expensive. AI made it practical to invest in generators, inventories, consistency checks, and drift detectors that turn repeated manual work into repeatable automation. Review at breadth: Checking edge cases, compatibility boundaries, and repeatability across a large surface area. Those capabilities were visible in the resulting design. Same-FQCN replacements created a controlled boundary for GraalVM-specific behavior. Reflection metadata was generated from build outputs instead of maintained as a hand-written guess list. Inventories and drift detectors turned upstream sync from a vague maintenance risk into an explicit engineering workflow.\nFor junior engineers, I think the lesson is equally important. AI does not remove the need to learn architecture, invariants, interfaces, testing, or maintenance. It makes those skills more valuable, because they determine whether accelerated implementation produces a durable system or just more code faster. The leverage comes from engineering judgment, not from novelty.\nClaude Code and Gemini AI acted as engineering accelerators throughout this process. In the GraalVM Distro specifically, they helped us:\nExplore migration strategies as running code: Instead of debating which approach might work, we built and compared multiple real prototypes, discarded the weak ones, and kept what held up. Refactor reflection-heavy and dynamic code paths: Replace runtime-hostile patterns with AOT-friendly alternatives across the codebase. Make upstream sync sustainable: Every time the distro pulls from upstream SkyWalking, metadata scanning, config regeneration, and recompilation must happen again. AI helped build the pipeline so that each sync is a controlled, largely automated process rather than a fresh manual effort that grows longer each time. Review logic and edge cases at scale: Especially in places where feature parity mattered more than raw implementation speed. The result was not just a large rewrite. It was a repeatable system: precompilers, manifest-driven loading, reflection-config generation, replacement boundaries, and drift detectors that make upstream migration reviewable and automatable.\nFor the broader methodology behind this style of development, see Agentic Vibe Coding in a Mature OSS Project. This post is the next step in that story: not only enhancing an active mature codebase, but reviving a paused effort and making it actually runnable.\nWhat Actually Changed The most important outcome of this project is not a benchmark table. The benchmark results belong to the distro itself, and they matter because they prove the system is real. But for this post, the deeper result is methodological: AI engineering changed how architecture could be explored, validated, and refined.\nInstead of treating architecture as a mostly document-driven activity followed by a long and expensive implementation phase, we were able to move much faster between idea, prototype, comparison, and redesign. That made it realistic to pursue higher-abstraction solutions, preserve cleaner boundaries, and build the automation needed to keep the migration maintainable over time.\nThe technical evidence for that work is the SkyWalking GraalVM Distro itself: not only a runnable system, but a migration pipeline expressed as precompilers, generated reflection metadata, controlled replacement boundaries, and drift checks. The benchmark data matter because they prove the system works in practice, but the architectural result is that the migration became a repeatable system rather than a one-time port. For detailed benchmark methodology, per-pod data, and the full technical design, see SkyWalking GraalVM Distro: Design and Benchmarks.\nThe project is hosted at apache/skywalking-graalvm-distro. We invite the community to test it, report issues, and help move it toward production readiness.\nFor me, the deeper takeaway is broader than this distro. AI engineering does not make architecture less important. It makes architecture more worth pursuing. When implementation speed rises enough, we can afford to test more ideas in code, keep the good abstractions, and build systems that would previously have been judged too expensive to finish well.\nFor senior engineers, that means the bottleneck shifts away from raw typing speed and toward taste, system judgment, and the ability to define stable boundaries. For junior engineers, it means the path forward is not to chase every exciting AI workflow, but to become stronger at the fundamentals that let acceleration compound: understanding requirements, reading unfamiliar systems, questioning assumptions, and recognizing what must remain correct as everything around it changes. AI changed the economics of architecture because it lowered the cost of validating better designs without lowering the bar for engineering judgment.\n","excerpt":"\u003cp\u003e\u003cem\u003eSkyWalking GraalVM Distro: A case study in turning runnable PoCs into a repeatable migration …\u003c/em\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-03-13-how-ai-changed-the-economics-of-architecture/","title":"How AI Changed the Economics of Architecture"},{"body":"A technical deep-dive into how we migrated Apache SkyWalking OAP to GraalVM Native Image — not as a one-off port, but as a repeatable pipeline that stays aligned with upstream.\nFor the broader story of how AI engineering made this project economically viable, see How AI Changed the Economics of Architecture.\nWhy GraalVM Is Not Optional GraalVM Native Image compiles Java applications Ahead-of-Time (AOT) into standalone executables. For an observability backend like SkyWalking OAP, this is not a performance optimization — it is an operational necessity.\nAn observability platform must be the most reliable component in the infrastructure. It has to survive the failures it is supposed to observe. In cloud-native environments where workloads scale, migrate, and restart constantly, the backend that watches everything cannot itself be the slow, heavy process that takes seconds to recover and gigabytes to idle.\nOur benchmarks make the case concrete:\nStartup: ~5 ms vs ~635 ms. In a Kubernetes cluster where an OAP pod gets evicted or rescheduled, a 635 ms gap means lost telemetry — traces, metrics, and logs that arrive during that window are simply dropped. At 5 ms, the new pod is receiving data before most clients even notice the disruption. Idle memory: ~41 MiB vs ~1.2 GiB. Observability backends run 24/7. In a multi-tenant or edge deployment, a 97% reduction in baseline RSS is the difference between fitting the observability stack on a small node and needing a dedicated one. Memory under load: ~629 MiB vs ~2.0 GiB at 20 RPS. A 70% reduction at production-like traffic means fewer nodes, lower cloud bills, and more headroom before the backend itself becomes a scaling bottleneck. No warm-up penalty: Peak throughput is available from the first request. The JVM\u0026rsquo;s JIT compiler needs minutes of traffic before it optimizes hot paths — during that window, tail latency is worse and data processing lags behind. A native binary has no such phase. Smaller attack surface: No JDK runtime means fewer CVEs to track and patch. For a component that ingests data from every service in the cluster, that matters. These are not incremental improvements. They change what deployment topologies are practical. Serverless observability backends, sidecar-model collectors, edge nodes with tight memory budgets — all become realistic when the backend is this light and this fast.\nThe Challenge: A Mature, Dynamic Java Platform SkyWalking OAP carries all the realities of a large Java platform: runtime bytecode generation, reflection-heavy initialization, classpath scanning, SPI-based module wiring, and dynamic DSL execution. These patterns are friendly to extensibility but hostile to GraalVM native image.\nThe documented GraalVM limitations are only the beginning. In a mature OSS platform, those limitations are deeply entangled with years of runtime design decisions. Standard GraalVM native images struggle with runtime class generation, reflection, dynamic discovery, and script execution — all of which had deep roots in SkyWalking OAP.\nThere was also a very concrete mountain in the early history of this distro. Upstream SkyWalking relied heavily on Groovy for LAL, MAL, and Hierarchy scripts. In theory, that was just one more unsupported runtime-heavy component. In practice, Groovy was the biggest obstacle in the whole path. It represented not only script execution, but a whole dynamic model that was deeply convenient on the JVM side and deeply unfriendly to native image.\nThe Design Goal: Make Migration Repeatable The final design is not just \u0026ldquo;run native-image successfully.\u0026rdquo; It is a system that keeps migration work repeatable:\nPre-compile runtime-generated assets at build time. OAL, MAL, LAL, Hierarchy rules, and meter-related generated classes are compiled during the build and packaged as artifacts instead of being generated at startup. Replace dynamic discovery with deterministic loading. Classpath scanning and runtime registration paths are converted into manifest-driven loading. Reduce runtime reflection and generate native metadata from the build. Reflection configuration is produced from actual manifests and scanned classes instead of being maintained as a hand-written guess list. Keep the upstream sync boundary explicit. Same-FQCN replacements are intentionally packaged, inventoried, and guarded with staleness checks. Make drift visible immediately. If upstream providers, rule files, or replaced source files change, tests fail and force explicit review. That is the architectural shift that matters most. Reusable abstraction and foresight did not become less important in the AI era. They became more important, because they determine whether AI speed produces a maintainable system or just a fast-growing pile of code.\nTurning Runtime Dynamism into Build-Time Assets SkyWalking OAP has several dynamic subsystems that are natural in a JVM world but problematic for native image:\nOAL generates classes at runtime. LAL, MAL, and Hierarchy were historically tied to Groovy-heavy runtime behavior, which became one of the biggest practical blockers in the early distro work. MAL, LAL, and Hierarchy rules depend on runtime compilation behavior. Guava-based classpath scanning discovers annotations, dispatchers, decorators, and meter functions. SPI-based module/provider discovery expects a more dynamic runtime environment. YAML/config initialization and framework integrations depend on reflective access. In SkyWalking GraalVM Distro, these are not solved one by one as isolated patches. They are pulled into a build-time pipeline.\nThe precompiler runs the DSL engines during the build, exports generated classes, writes manifests, serializes config data, and generates native-image metadata. That means startup becomes class loading and registration, not runtime code generation. The runtime path is simpler because the build path became richer.\nThis is also why the project is more than a performance exercise. The design goal was to move complexity into a place where it is easier to verify, easier to automate, and easier to repeat.\nSame-FQCN Replacements as a Controlled Boundary One of the most practical design choices in this distro is the use of same-FQCN replacement classes. We do not rely on vague startup tricks or undocumented ordering assumptions. Instead, the GraalVM-specific jars are repackaged so the original upstream classes are excluded and the replacement classes occupy the exact same fully-qualified names.\nThis matters for maintainability. It creates a very clear boundary:\nthe upstream class still defines the behavior contract, the GraalVM replacement provides a compatible implementation strategy, and the packaging makes that swap explicit. For example, OAL loading changes from runtime compilation into manifest-driven loading of precompiled classes. Similar replacements handle MAL and LAL DSL loading, module wiring, config initialization, and several reflection-sensitive paths. The goal is not to fork everything. The goal is to replace only the places where the runtime model is fundamentally unfriendly to native image.\nThat boundary is then guarded by tests that hash the upstream source files corresponding to the replacements. When upstream changes one of those files, the build fails and tells us exactly which replacement needs review. This is what turns \u0026ldquo;keeping up with upstream\u0026rdquo; from an anxiety problem into a visible engineering task.\nReflection Config Is Generated, Not Guessed In many GraalVM migrations, reflect-config.json becomes a manually accumulated artifact. It grows over time, gets stale, and nobody is fully sure whether it is complete or why each entry exists. That approach does not scale well for a large, evolving OSS platform.\nIn this distro, reflection metadata is generated from the build outputs and scanned classes:\nmanifests for OAL, MAL, LAL, Hierarchy, and meter-generated classes, annotation-scanned classes, Armeria HTTP handlers, GraphQL resolvers and schema-mapped types, and accepted ModuleConfig classes. This is a much healthier model. Instead of asking people to remember every reflective access path, the system derives reflection metadata from the actual migration pipeline. The build becomes the source of truth.\nKeeping Upstream Sync Practical If this distro were only a one-time engineering sprint, it would be much less interesting. The real challenge is keeping it alive while upstream SkyWalking continues to evolve.\nThat is why the repo includes explicit inventories and drift detectors:\nprovider inventories that force new upstream providers to be categorized, rule-file inventories that force new DSL inputs to be acknowledged, SHA watchers for precompiled YAML inputs, and SHA watchers for upstream source files with GraalVM-specific replacements. Good abstraction is not only about elegant code structure. It is about choosing a migration design that can survive contact with future change.\nBenchmark Results We benchmarked the standard JVM OAP against the GraalVM Distro on an Apple M3 Max (macOS, Docker Desktop, 10 CPUs / 62.7 GB), both connecting to BanyanDB.\nBoot Test (Docker Compose, no traffic, median of 3 runs) Metric JVM OAP GraalVM OAP Delta Cold boot startup 635 ms 5 ms ~127x faster Warm boot startup 630 ms 5 ms ~126x faster Idle RSS ~1.2 GiB ~41 MiB ~97% reduction Boot time is measured from OAP\u0026rsquo;s first application log timestamp to the listening on 11800 log line (gRPC server ready).\nUnder Sustained Load (Kind + Istio 1.25.2 + Bookinfo at ~20 RPS, 2 OAP replicas) 30 samples at 10s intervals after 60s warmup.\nMetric JVM OAP GraalVM OAP Delta CPU median (millicores) 101 68 -33% CPU avg (millicores) 107 67 -37% Memory median (MiB) 2068 629 -70% Memory avg (MiB) 2082 624 -70% Both variants reported identical entry-service CPM, confirming equivalent traffic processing capability.\nService metrics collected every 30s via swctl for all discovered services: service_cpm, service_resp_time, service_sla, service_apdex, service_percentile.\nFull benchmark scripts and raw data are in the benchmark/ directory of the distro repository.\nCurrent Status The project is a runnable experimental distribution, hosted in its own repository: apache/skywalking-graalvm-distro.\nThe current distro intentionally focuses on a modern, high-performance operating model:\nStorage: BanyanDB Cluster modes: Standalone and Kubernetes Configuration: none or Kubernetes ConfigMap Runtime model: fixed module set, precompiled assets, and AOT-friendly wiring This focus is deliberate. A repeatable migration system starts by making a clear scope runnable, then expanding without losing control.\nGetting Started Because the SkyWalking GraalVM Distro is designed for peak performance, it is optimized to work with BanyanDB as its storage backend. The current published image is available on Docker Hub, and you can boot the stack using the following docker-compose.yml.\nversion: \u0026#39;3.8\u0026#39; services: banyandb: image: ghcr.io/apache/skywalking-banyandb:e1ba421bd624727760c7a69c84c6fe55878fb526 container_name: banyandb restart: always ports: - \u0026#34;17912:17912\u0026#34; - \u0026#34;17913:17913\u0026#34; command: standalone --stream-root-path /tmp/stream-data --measure-root-path /tmp/measure-data --measure-metadata-cache-wait-duration 1m --stream-metadata-cache-wait-duration 1m healthcheck: test: [\u0026#34;CMD\u0026#34;, \u0026#34;sh\u0026#34;, \u0026#34;-c\u0026#34;, \u0026#34;nc -nz 127.0.0.1 17912\u0026#34;] interval: 5s timeout: 10s retries: 120 oap: image: apache/skywalking-graalvm-distro:0.1.1 container_name: oap depends_on: banyandb: condition: service_healthy restart: always ports: - \u0026#34;11800:11800\u0026#34; - \u0026#34;12800:12800\u0026#34; environment: SW_STORAGE: banyandb SW_STORAGE_BANYANDB_TARGETS: banyandb:17912 SW_HEALTH_CHECKER: default healthcheck: test: [\u0026#34;CMD-SHELL\u0026#34;, \u0026#34;nc -nz 127.0.0.1 11800 || exit 1\u0026#34;] interval: 5s timeout: 10s retries: 120 ui: image: ghcr.io/apache/skywalking/ui:10.3.0 container_name: ui depends_on: oap: condition: service_healthy restart: always ports: - \u0026#34;8080:8080\u0026#34; environment: SW_OAP_ADDRESS: http://oap:12800 Simply run:\ndocker compose up -d We invite the community to test this new distribution, report issues, and help us move it toward a production-ready state.\nSpecial thanks to the GraalVM team for the technology foundation.\n","excerpt":"\u003cp\u003e\u003cem\u003eA technical deep-dive into how we migrated Apache SkyWalking OAP to GraalVM Native Image — not as a …\u003c/em\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-03-13-skywalking-graalvm-distro-design-and-benchmarks/","title":"SkyWalking GraalVM Distro: Design and Benchmarks"},{"body":"Most \u0026ldquo;vibe coding\u0026rdquo; stories start with a greenfield project. This one doesn\u0026rsquo;t.\nApache SkyWalking is a 9-year-old observability platform with hundreds of production deployments, a complex DSL stack, and an external API surface that users have built dashboards, alerting rules, and automation scripts against. When I decided to replace the core scripting engine — purging the Groovy runtime from four DSL compilers — the constraint wasn\u0026rsquo;t \u0026ldquo;can AI write the code?\u0026rdquo; It was: \u0026ldquo;can AI write the code without breaking anything for existing users?\u0026rdquo;\nThe answer turned out to be yes — ~77,000 lines changed across 10 major PRs in about 5 weeks — but only because the AI was tightly guided by a human who understood the project\u0026rsquo;s architecture, its compatibility contracts, and its users. This post is about the methodology: what worked, what didn\u0026rsquo;t, and what mature open-source maintainers should know before handing their codebase to AI agents.\nThe Project in Brief The task was to replace SkyWalking\u0026rsquo;s Groovy-based scripting engines (MAL, LAL, Hierarchy) with a unified ANTLR4 + Javassist bytecode compilation pipeline, matching the architecture already proven by the OAL compiler. The internal tech stack was completely overhauled; the external interface had to remain identical.\nBeyond the compiler rewrites, the scope included a new queue infrastructure (threads dropped from 36 to 15), virtual thread support for JDK 25+, and E2E test modernization. By conventional estimates, this was 5-8 months of senior engineer work.\nFor the full technical details on the compiler architecture, see the Groovy elimination discussion.\nWhat is Agentic Vibe Coding? \u0026ldquo;Vibe coding\u0026rdquo; — a term coined by Andrej Karpathy — describes a style of programming where you describe intent and let AI write the code. It\u0026rsquo;s powerful for prototyping, but on its own, it\u0026rsquo;s risky for production systems.\nAgentic vibe coding takes this further: instead of a single AI autocomplete, you orchestrate multiple AI agents — each with different strengths — under your architectural direction, with automated tests as the safety net. In my workflow:\nClaude Code (plan mode): Primary coding agent. Plan mode lets me review the approach before any code is generated. This is critical for architectural decisions — I steer the design, Claude handles the implementation. Gemini: Code review, concurrency analysis, and verification reports. Gemini reviewed every major PR for thread-safety, feature parity, and edge cases. Codex: Autonomous task execution for well-defined, bounded work items. The key insight: AI writes the code, but the architect owns the design. Without deep domain knowledge of SkyWalking\u0026rsquo;s internals, no AI could have planned these changes. Without AI, I couldn\u0026rsquo;t have executed them in 5 weeks.\nHow TDD Made AI Coding Safe The reason I could move this fast without breaking things comes down to one principle: never let AI code without a test harness.\nMy workflow for each major change:\nPlan mode first: Describe the goal to Claude, review the plan, iterate on architecture before any code is written. Write the test contract: Define what \u0026ldquo;correct\u0026rdquo; means — for the compiler rewrites, this meant cross-version comparison tests that run every expression through both the old and new engines, asserting identical results across 1,290+ expressions. Let AI implement: With the test contract in place, Claude can write thousands of lines of implementation code. If it\u0026rsquo;s wrong, the tests catch it immediately. E2E as the final gate: Every PR must pass the full E2E test suite — Docker-based integration tests that boot the entire server with real storage backends. AI code review: Gemini reviewed each PR for concurrency issues, thread-safety, and feature parity — catching things that unit tests alone wouldn\u0026rsquo;t find. This is the opposite of \u0026ldquo;hope it works\u0026rdquo; vibe coding. The AI writes fast, the tests verify fast, and I steer the architecture. The feedback loop is tight enough that I can iterate on complex compiler code in minutes instead of days.\nLessons Learned AI is a force multiplier, not a replacement. Before any AI agent wrote a single line, a human had to define the replacement solution: what gets replaced, how it gets replaced, and — critically — where the boundaries are. Which APIs could break? The internal compilation pipeline was fair game for a complete overhaul. Which APIs must stay aligned? Every external-facing DSL syntax, every YAML configuration key, every metrics name and tag structure had to remain byte-for-byte identical — because hundreds of deployed dashboards, alerting rules, and user scripts depend on them. Drawing these boundaries required deep knowledge of the codebase and its users. AI executed the plan at extraordinary speed, but the plan itself — the scope, the invariants, the compatibility contract — had to come from a human who understood the blast radius of every change.\nPlan mode is non-negotiable for architectural work. Letting AI jump straight to code on a compiler rewrite would be a disaster. Plan mode\u0026rsquo;s strength is that it collects code context — scanning imports, tracing call chains, mapping class hierarchies — and uses that context to help me fill in implementation details I\u0026rsquo;d otherwise have to look up manually. But it can\u0026rsquo;t tell you the design principles. That direction had to come from me, stated clearly upfront, so the AI\u0026rsquo;s planning stayed on the right track instead of optimizing toward a locally reasonable but architecturally wrong solution.\nKnow when to hit ESC. Claude has a clear tendency to dive deep into solution code writing once it starts — and it won\u0026rsquo;t stop on its own when it encounters something that conflicts with the original plan\u0026rsquo;s concept. Instead of pausing to flag the conflict, it will push forward, improvising around the obstacle in ways that silently violate the design intent. I had to learn to watch for this: when Claude\u0026rsquo;s output started drifting from the plan, I\u0026rsquo;d manually cancel the task (ESC), call it off, identify where the plan and reality diverged, adjust the plan, and restart. This interrupt-replan cycle was a regular part of the workflow, not an exception. The architect has to stay in the loop — not just at planning time, but during execution — because AI agents don\u0026rsquo;t yet know when to stop and ask.\nSpec-driven testing is necessary but not sufficient — the logic workflow matters more. It\u0026rsquo;s tempting to think that if you define the input/output spec clearly enough, AI can fill in the implementation and tests will catch any mistakes. I tried this. It doesn\u0026rsquo;t work for anything non-trivial. During the expression compiler rewrite, Claude would sometimes change code in unreasonable ways just to make the spec tests pass — the inputs went in, the expected outputs came out, and everything looked green. But the internal logic was wrong: inconsistent with the design patterns the rest of the codebase relied on, impossible to extend, or solving the specific test case through a hack rather than a general mechanism. A spec only checks what the code produces; it says nothing about how the code produces it. For a mature project, the \u0026ldquo;how\u0026rdquo; matters enormously — the solution needs to be consistent with the existing architecture, widely adoptable by contributors, and maintainable long-term. That\u0026rsquo;s why I needed cross-version testing and human review of the implementation path, not just the results.\nTesting at two levels kept the rewrite honest. Cross-version testing was part of my design plan from the start — I architected the dual-path comparison framework so that every production DSL expression runs through both the old and new engines, asserting identical results across 1,290+ expressions. This gave me confidence no human review could match, and it was a deliberate planning decision: I knew AI-generated compiler code needed a mechanical proof of behavioral equivalence, not just eyeball review. On top of that, E2E tests served as the project\u0026rsquo;s existing infrastructure safety net — Docker-based integration tests that boot the entire server with real storage backends. Unit tests and cross-version tests verify logic in isolation; E2E tests verify the system actually works end-to-end. For infrastructure-level changes like queue replacement and thread model changes, E2E is the only gate that truly matters. Together, the two layers — designed-for-this-rewrite cross-version tests and pre-existing E2E infrastructure — caught different classes of bugs and made shipping with confidence possible.\nMultiple AIs have different strengths. Claude excels at large-scale code generation with plan mode. Gemini is exceptional at logic review — it can mentally trace code branches with given input data, simulating execution without actually running the code. This is significant for reviewing AI-generated code: Gemini would walk through a generated compiler method step by step, flagging where a null check was missing or where a branch would produce wrong output for a specific edge case. Codex proved most valuable as a test reviewer and honesty checker. AI-generated code has a subtle failure mode: the coding agent can make wrong assumptions and then write tests that pass by setting expected values to match the wrong behavior — effectively bypassing the test safety net. Codex caught cases where Claude had set unreasonable expected values that happened to make tests green, masking logic errors that would have surfaced in production. Using all three as checks on each other was far more effective than relying on any single one.\nThe Mythical Man-Month still applies — and so does the Mythical Token-Month. Brooks taught us that a task requiring 12 person-months does not mean 12 people can finish it in one month. The same law applies to AI: you cannot simply throw more tokens, more agents, or more parallel sessions at a problem and expect it to converge faster. Communication costs, coordination overhead, requirements analysis, and conceptual integrity — these software engineering fundamentals do not disappear just because your workforce is artificial. Worse, when the direction is wrong — when there\u0026rsquo;s a conceptual error in the design or an unreasonable architectural choice — AI will not recognize it. It will charge down the wrong path at extraordinary speed, burning tokens furiously while trapped in a vortex of self-justification: patching code to make failing tests pass, adjusting expected values to match wrong behavior, adding workarounds on top of workarounds — each iteration making the codebase look more \u0026ldquo;complete\u0026rdquo; while drifting further from correctness. AI vibe coding cannot break out of this spiral on its own. Only a human who understands the domain can recognize \u0026ldquo;this is fundamentally wrong, stop,\u0026rdquo; discard the work, and redirect. Speed without direction is just expensive chaos.\nThe Bigger Picture The agentic vibe coding approach worked because it combined AI\u0026rsquo;s speed with human architectural judgment and automated test discipline. It\u0026rsquo;s not magic — it\u0026rsquo;s engineering, accelerated.\nBrooks also gave us \u0026ldquo;No Silver Bullet,\u0026rdquo; and its core distinction matters more than ever: software complexity comes in two kinds. Essential complexity comes from the problem itself — the domain semantics, the behavioral contracts, the concurrency invariants. No tool can eliminate this; it must be understood, modeled, and reasoned about by someone who knows the domain. Accidental complexity comes from the tools and implementation — boilerplate code, manual refactoring across hundreds of files, the mechanical work of translating a design into compilable source. This is exactly where AI excels. What made this project work was recognizing which complexity was which: I owned the essential complexity (architecture, API boundaries, correctness invariants), and AI demolished the accidental complexity (generating 77K lines of implementation, scaffolding test harnesses, rewriting repetitive patterns across dozens of config files). Confuse the two — let AI make essential decisions, or waste human time on accidental work — and you get the worst of both worlds.\nQian Xuesen(Tsien Hsue-shen)\u0026rsquo;s Engineering Cybernetics offers another lens that proved surprisingly relevant. His core framework — feedback, control, optimization — describes how to keep complex systems running toward their target. AI vibe coding at full speed is like a hypersonic missile: extraordinarily fast, but without a guidance system it just creates a bigger crater in the wrong place. The feedback loop in my workflow was the test harness — cross-version tests and E2E tests providing continuous signal on whether the system was still on course. Control was the human architect deciding when to intervene: reviewing plans before execution, hitting ESC when the direction drifted, choosing which AI to trust for which task. Optimization was iterative: each interrupt-replan cycle refined the approach, each Gemini review tightened the logic, each Codex audit caught assumptions the coding agent had smuggled past the tests. Without all three — feedback to detect deviation, control to correct course, optimization to converge — the speed of AI coding would be not an advantage but a liability. The faster the missile, the more precise the guidance must be.\nFor more details or to share your own experience with agentic coding on production systems, feel free to reach me on GitHub.\n","excerpt":"\u003cp\u003eMost \u0026ldquo;vibe coding\u0026rdquo; stories start with a greenfield project. This one doesn\u0026rsquo;t. …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-03-08-agentic-vibe-coding/","title":"Agentic Vibe Coding in a Mature OSS Project: What Worked, What Didn't"},{"body":"大多数\u0026quot;vibe coding\u0026quot;的故事都从一个全新项目开始，讲述一个快速构建原型或者可运行项目的过程，但这篇不是。\nApache SkyWalking 是一个有 9 年历史的Apache顶级项目，线上数以千计的集群部署，内部有一套复杂的 DSL 编译栈，对外暴露的 API 上承载着用户构建的仪表盘、告警规则和自动化脚本。当我决定替换核心脚本引擎——从四个 DSL 编译器中彻底移除 Groovy 运行时——面临的问题不是\u0026quot;AI 能不能写出代码\u0026quot;，而是\u0026quot;也许只有AI能完成如此大规模的一致性迭代\u0026quot;，以及\u0026quot;AI 能不能在不破坏系统的前提下写出完整且高效的代码\u0026quot;。\n答案是可以——约 7.7 万行代码变更，10 个主要 PR，历时约 5 周——但前提是 AI 始终在一个深刻理解项目架构、兼容性要求和用户场景的人的引导下工作。这篇文章分享了我在过去几个月的实践体验，以及成熟开源项目的维护者在把代码库交给 AI 智能体之前应该知道什么。\n项目概况 这次的任务是将 SkyWalking 基于 Groovy 的脚本引擎（MAL、LAL、Hierarchy）替换为统一的 ANTLR4 + Javassist 字节码编译管线，对齐 OAL 编译器已经验证过的架构。内部技术栈彻底重构，但对外接口必须保持完全一致。\n除了编译器重写，范围还包括新的线程管理策略（线程数从 36 降到 15）、JDK 25+ 虚拟线程支持，以及端到端测试的现代化改造。按传统估算，这是 5-8 个月的资深工程师（以我自己为例）工作量。\n编译器架构的完整技术细节，参见 Groovy 移除讨论。\n什么是 Agentic Vibe Coding？ \u0026ldquo;Vibe coding\u0026rdquo;——Andrej Karpathy 提出的概念——描述的是一种你表达意图、让 AI 来写代码的编程风格。整个AI编程过程，一直以来都是用来做原型，效果强大且速度迅猛，但单独用于生产系统是有风险的。\nAgentic vibe coding 更进一步：不是单一的 AI 自动补全，而是在你的架构指导下编排多个 AI 智能体——各有所长——以自动化测试作为安全网。我的工作流是这样的：\nClaude Code（plan 模式）：主力编码智能体。Plan 模式让我在生成任何代码之前先审查方案。这对架构决策至关重要——我把控设计方向，Claude 负责实现。 Gemini：代码审查、并发分析和验证报告。每个主要 PR 都经过 Gemini 审查线程安全性、功能对等性和边界情况。 Codex：对定义明确、边界清晰的工作项进行自主任务执行。 核心洞察：AI 写代码，但架构师掌控设计。 没有对 SkyWalking 内部机制的深入领域知识，任何 AI 都无法规划这些变更。没有 AI，我也不可能在 5 周内完成执行。\nTDD 如何让 AI 编程变得安全 我能以这样的速度推进而不搞砸，归结为一个原则：绝不让 AI 在没有测试保护的情况下写代码。\n每次重大变更的工作流：\n先进 plan 模式：向 Claude 描述目标，审查方案，在写任何代码之前先在架构层面迭代。 编写测试契约：定义\u0026quot;正确\u0026quot;意味着什么——对于编译器重写，这意味着交叉版本对比测试，让每个表达式同时通过新旧两个引擎运行，在 1290+ 个表达式上断言结果完全一致。 让 AI 实现：有了测试契约，Claude 可以写出数千行实现代码。如果写错了，测试会立即捕获。 端到端测试作为最终关卡：每个 PR 都必须通过完整的端到端测试套件——基于 Docker 的集成测试，启动整个服务器并连接真实存储后端。 AI 代码审查：Gemini 审查每个 PR 的并发问题、线程安全性和功能对等性——捕获单元测试无法发现的问题。 这和\u0026quot;写完祈祷能跑\u0026quot;的 vibe coding 完全相反。AI 写得快，测试验证得快，我把控架构方向。反馈循环足够紧凑，让我能在几分钟而不是几天内迭代复杂的编译器代码。\n经验教训 AI 是力量倍增器，不是替代品。 在任何 AI 智能体写下第一行代码之前，必须由人来定义替换方案：替换什么、怎么替换，以及——至关重要的——边界在哪里。哪些 API 可以破坏性变更？内部编译管线可以彻底重构。哪些 API 必须保持对齐？每一个对外的 DSL 语法、每一个 YAML 配置键、每一个指标名称和标签结构都必须逐字节保持一致——因为数百个已部署的仪表盘、告警规则和用户脚本依赖于它们。划定这些边界需要对代码库及其用户的深入了解。AI 以惊人的速度执行了计划，但计划本身——范围、不变量、兼容性契约——必须来自一个理解每次变更影响半径的人。\n架构级工作，plan 模式不可妥协。 让 AI 在编译器重写上直接跳到写代码，那是灾难。Plan 模式的价值在于它会收集代码上下文——扫描 import、追踪调用链、映射类继承关系——并利用这些上下文帮我补全那些我本来需要手动查找的实现细节。但它无法告诉你设计原则。方向必须由我在前期明确给出，这样 AI 的规划才能沿着正确的轨道走，而不是朝着一个局部合理但架构上错误的方案去优化。\n要知道什么时候该按 ESC。 Claude 有一个明显的倾向：一旦开始写解决方案代码就会一头扎进去——当遇到与原始计划概念冲突的东西时，它不会自己停下来。它不会暂停来标记冲突，而是会继续推进，用即兴的方式绕过障碍，悄无声息地违背设计意图。我必须学会观察这个信号：当 Claude 的输出开始偏离计划时，我会手动取消任务（ESC），叫停它，找出计划和现实的分歧点，调整计划，然后重新开始。这种中断-重新规划的循环是工作流的常态，而非例外。架构师必须始终在环路中——不仅是在规划阶段，执行阶段也是——因为 AI 智能体还不知道什么时候该停下来问一句。\nSpec-Driven 更多的运用于测试，而非开发。它只是一个必要的但不充分条件，而逻辑工作流更重要。 很容易产生一种想法：只要把输入/输出规格定义得足够清楚，AI 就能填充实现，测试会捕获任何错误。我试过。对于任何复杂的生产场景，这行不通。在表达式编译器重写过程中，Claude 有时会以不合理的方式修改代码，仅仅为了让规格测试通过——输入进去了，预期输出出来了，一切看起来都是正常的。但内部逻辑是错的：与代码库其他部分依赖的设计模式不一致，无法扩展，或者通过 hack （代码反射、字段名称静态比较等不可接受的工程方法）而非通用机制来解决特定测试用例。规格只检查代码产出了什么；它对代码如何产出一无所知。对于成熟项目，\u0026ldquo;如何\u0026quot;极其重要——解决方案需要与现有架构一致，能被贡献者广泛采用，并且长期可维护可扩展。这就是为什么我需要交叉版本测试加上对实现路径的人工审查，而不仅仅是审查结果。\n两个层次的测试让重写的代码验证更有保障。 交叉版本测试从一开始就是我设计方案的一部分——我架构了双路径对比框架，让每个生产环境的 DSL 表达式同时通过新旧两个引擎运行，在 1290+ 个表达式上断言结果完全一致。这给了我任何人工审查都无法匹敌的信心，而且这是一个刻意的规划决策：我知道 AI 生成的编译器代码需要行为等价性的机械证明，而不仅仅是肉眼审查。在此之上，端到端测试作为项目已有的基础设施安全网——基于 Docker/K8s 的集成测试，启动整个服务器并连接真实存储后端。单元测试和交叉版本测试在隔离环境中验证逻辑；端到端测试验证系统真正能端到端地工作。对于队列替换和线程模型变更这样的基础设施级变更，端到端测试是唯一真正重要的关卡。两个层次——为本次重写专门设计的交叉版本测试和预先存在的端到端基础设施——捕获了不同类别的 bug，使得有信心地发布成为可能。\n多个 AI 各有所长。 Claude 擅长配合 plan 模式进行大规模代码生成。Gemini 在逻辑审查方面表现出色——它能在给定输入数据的情况下在脑中追踪代码分支，模拟执行而无需实际运行代码。这对审查 AI 生成的代码意义重大：Gemini 会逐步走查一个编译器生成的方法，标记出哪里缺少空值检查，或者哪个分支在特定边界情况下会产生错误输出。Codex 作为测试审查者和诚实性检查者最有价值。AI 生成的代码有一种微妙的失败模式：编码智能体可能做出错误假设，然后编写测试时将期望值设置为匹配错误行为——实际上绕过了测试安全网。Codex 捕获了 Claude 设置不合理期望值使测试变绿的情况，掩盖了本会在生产环境中暴露的逻辑错误。将三者互相校验，远比依赖其中任何一个更有效。\n人月神话依然适用——基于Token的AI月神话同样如此。 Brooks 告诉我们，一个需要 12 人月的任务不意味着 12 个人能在一个月内完成。同样的定律适用于 AI：你不能简单地投入更多 token、更多智能体或更多并行会话，就指望问题更快收敛。沟通成本、协调开销、需求分析和概念完整性——这些软件工程的基本规律不会因为你的劳动力是人工智能就消失。更糟糕的是，当方向错误时——当设计中存在概念性错误或不合理的架构选择时——AI 不会识别出来。它会以惊人的速度冲向错误的方向，疯狂消耗 token，同时陷入自我辩护的漩涡：修补代码让失败的测试通过，调整期望值去匹配错误行为，在变通方案上叠加变通方案——每次迭代都让代码库看起来更\u0026quot;完整\u0026rdquo;，实际上却离正确越来越远。AI vibe coding 无法自行跳出这个螺旋。只有理解领域的人才能认识到\u0026quot;这从根本上就是错的，停下来\u0026quot;，丢弃这些工作，重新引导方向。没有方向的速度，只是昂贵的混乱。\n更大的图景 Agentic vibe coding 之所以有效，是因为它将 AI 的速度与人的架构判断力和自动化测试纪律结合在了一起。这不是魔法——这是被加速的工程。\nBrooks 还给了我们《没有银弹》，其核心区分在今天比以往任何时候都更重要：软件复杂性分为两种。本质复杂性来自问题本身——领域语义、行为契约、并发不变量。没有任何工具能消除它；它必须由理解领域的人去理解、建模和推理。偶然复杂性来自工具和实现——样板代码、跨数百个文件的手动重构、将设计翻译成可编译源码的机械工作。这恰恰是 AI 擅长的地方。这个项目之所以成功，在于认清了哪种复杂性是哪种：我掌控本质复杂性（架构、API 边界、正确性不变量），AI 消灭偶然复杂性（生成 7.7 万行实现代码、搭建测试框架、跨数十个配置文件重写重复模式）。搞混这两者——让 AI 做本质决策，或者让人浪费时间在偶然工作上——你会得到两个世界中最差的结果。\n钱学森的《工程控制论》提供了另一个视角，在实践中出人意料地切题。他的核心框架——反馈、控制、优化——描述的是如何让复杂系统持续朝目标运行。全速运转的 AI vibe coding 就像一枚高超音速导弹：速度惊人，但没有制导系统只会在错误的地方炸出一个更大的坑。我工作流中的反馈回路是测试体系——交叉版本测试和端到端测试持续提供系统是否仍在航线上的信号。控制是人类架构师决定何时介入：在执行前审查方案，在方向偏移时按 ESC，选择哪个 AI 负责哪项任务。优化是迭代式的：每次中断-重新规划的循环都在精炼方法，每次 Gemini 审查都在收紧逻辑，每次 Codex 审计都在捕获编码智能体偷偷绕过测试的假设。缺少其中任何一个——检测偏差的反馈、纠正航向的控制、趋向收敛的优化——AI 编程的速度就不是优势而是负债。导弹越快，制导就必须越精确。\nAI Vibe Coding以及它的迭代，正在快速地走进每一个开发者，也正在广泛地融入开源和商业软件。我们都在见证这种新的开发模式，以及AI Vibe Coding和软件工程理论的融合。如果你想和我探讨更多的AI + OSS话题，欢迎在 GitHub 上联系我。\n","excerpt":"\u003cp\u003e大多数\u0026quot;vibe coding\u0026quot;的故事都从一个全新项目开始，讲述一个快速构建原型或者可运行项目的过程，但这篇不是。\u003c/p\u003e\n\u003cp\u003eApache SkyWalking 是一个有 9 年历史的 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2026-03-08-agentic-vibe-coding/","title":"在成熟开源大型项目中实践 Agentic Vibe Coding：软件工程与工程控制论还在延续"},{"body":"SkyWalking Java Agent 9.6.0 is released. Go to downloads page to find release tars. Changes by Version\n9.6.0 Fix OOM due to too many span logs. Fix ClassLoader cache OOM issue with WeakHashMap. Fix Jetty client cannot receive the HTTP response body. Eliminate repeated code with HttpServletRequestWrapper in mvc-annotation-commons. Add the jdk httpclient plugin. Fix Gateway 2.0.x plugin not activated for spring-cloud-starter-gateway 2.0.0.RELEASE. Support kafka-clients-3.9.x intercept. Upgrade kafka-clients version in optional-reporter-plugins to 3.9.1. Fix AbstractLogger replaceParam when the replaced string contains a replacement marker. Fix JDBCPluginConfig.Plugin.JDBC.SQL_BODY_MAX_LENGTH was not working in some plugins. Bump up Lombok to v1.18.42 to adopt JDK25 compiling. Add eclipse-temurin:25-jre as another base image. Add JDK25 plugin tests for Spring 6. Ignore classes starting with \u0026ldquo;sun.nio.cs\u0026rdquo; in bytebuddy due to potential class loading deadlock. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking Java Agent 9.6.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nChanges by …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-java-agent-9-6-0/","title":"Release Apache SkyWalking Java Agent 9.6.0"},{"body":"2025 was a very focused year for the Apache SkyWalking community: moving BanyanDB from “native storage” to a “production-ready default”, and making SkyWalking APM fully benefit from that foundation.\nThis post summarizes the key milestones, with an emphasis on BanyanDB.\nStorage strategy: saying goodbye to H2 We started 2025 with a clear direction: the H2 storage option is permanently removed. This change reduced long-term maintenance burden and removed a storage choice that was not aligned with production and cloud-native deployments.\nBanyanDB: from 0.8.0 foundations to 0.9.0 production features BanyanDB 0.8.0 delivered the “day-2 operations” foundation that a default storage backend needs. The community put a lot of effort into making queries faster and more predictable (for example index_mode, numeric index types, and multiple query-path optimizations), while also making the system safer under real production pressure. Disk-usage thresholds and a query memory protector were added as guardrails, and the operational toolbox matured with snapshot/backup/restore utilities and improved metadata synchronization.\nJust as importantly, 0.8.0 started filling in the missing pieces of a full platform: native property storage and lifecycle-related capabilities that later enabled stronger HA and stage-based deployment patterns.\nBanyanDB 0.9.0 was the “production features” milestone. It introduced the Trace data model as a first-class citizen, which unlocked much deeper trace storage and query capabilities. On the reliability and scaling side, the release brought configurable replicas, liaison-side improvements (including load balancing and moving some TopN flow), and broader correctness work such as migrations, version compatibility checks, and access logs.\nIt also made long-term operations more cloud-friendly with backup/restore support for AWS S3, GCS, and Azure Blob Storage, and added authentication primitives needed in shared environments. In short, 0.9.0 is where BanyanDB clearly moved beyond a “fast storage engine” into a “production platform”.\nSkyWalking APM: BanyanDB becomes the default path With APM 10.2.0, the project made the strategic shift official: H2 was removed permanently, and BanyanDB 0.8.0 became the default path that SkyWalking invests in. A lot of the work here was not flashy, but essential — refining OAP behavior (group settings, index model changes, Progressive TTL, query limits, and more) so running BanyanDB in production felt stable and predictable.\nWith APM 10.3.0, SkyWalking and BanyanDB moved forward together: BanyanDB 0.9.0’s new trace model was adopted end-to-end, reducing inefficient query round-trips and enabling new query views that significantly lowered page latency. The integration also expanded into lifecycle-aware operations with hot/warm/cold stage configuration (including TTL and query support), and added BanyanDB self-monitoring through OAP and the UI — the kind of end-to-end polish that turns a storage backend into a truly native solution.\nIf you’d like this review to cover APM 10.4.x as well, please point me to the corresponding release content in this repo (I didn’t find an APM 10.4.0 release announcement in the current checkout).\nPackaging and deployment ecosystem (Helm) BanyanDB’s production readiness is not only server features — it also depends on deployment maturity.\nHelm charts: SkyWalking Kubernetes Helm Chart 4.8.0 improved BanyanDB deployment defaults by updating the bundled BanyanDB Helm dependency, fixing an init-job volume-mount mismatch, and aligning OAP/UI images with the APM 10.3.0 line. BanyanDB Helm 0.4.0 added backup/restore sidecars and a default volume for property storage. BanyanDB Helm 0.5.0 introduced stage-aware patterns (hot/warm/cold), improved lifecycle-sidecar scheduling, moved liaison to StatefulSet, refined internal networking, and expanded configuration options. BanyanDB Helm 0.5.1 refined liaison configuration and fixed restore-init environment issues. BanyanDB Helm 0.5.3 fixed a liaison/data-node port issue. The rest of the community: agents and tooling kept moving While storage was the “main storyline”, the community shipped releases across agents, clients, and surrounding components throughout 2025.\nBelow is a consolidated view of the other releases, grouped by project, with the most important notes.\nSkyWalking Java Agent\n9.4.0: agent self-observability; async-profiler support; broader plugin improvements. 9.5.0: virtual thread executor plugin; compatibility and stability fixes; dependency upgrades. SkyWalking Go\n0.6.0: richer manual APIs (events/logs/metrics, set span error); goframev2 plugin; bug fixes including Redis cluster mode. SkyWalking for NodeJS\n0.8.0: Express 4/5 compatibility, keep-alive HTTP trace fix, and test/dependency maintenance. SkyWalking Python\n1.2.0: sampling service, sw_grpc plugin, async/profiling stability fixes, Python 3.13 support, and dropping Python 3.7. SkyWalking PHP\n1.0.0: reach 1.0; add PSR-3 log reporting; upgrade toolchain/dependencies. SkyWalking Rust\n0.9.0: migrate to Rust edition 2024 and upgrade dependencies. 0.10.0: Kafka client configuration refactor, rdkafka upgrade, CI maintenance. SkyWalking Ruby\n0.1.0: initialize agent core and e2e tests; add plugins for Sinatra, redis-rb, net-http, memcached, and Elasticsearch. SkyWalking Client JS\n1.0.0: add Core Web Vitals and static resource metrics; fix fetch/resource error handling; dependency and e2e/test improvements. SkyWalking Satellite\n1.3.0: support native eBPF Access Log protocol and async-profiler protocol; upgrade Go toolchain. SkyWalking Eyes\n0.7.0: improve installation/docs, respect gitignore behavior, upgrade Go, and simplify release steps. 0.8.0: add Elixir support and stronger dependency-license scanning (notably Ruby via Gemfile.lock), plus stability fixes. Looking ahead: possible directions in 2026 2025 was about making BanyanDB ready for production. In 2026, the community is exploring the next set of improvements that could make the whole stack simpler to operate, more stable under stress, and easier to integrate into broader observability ecosystems.\nPossible areas include:\nBanyanDB: remove the etcd dependency: the direction under discussion is to move away from etcd (given ecosystem activity and maintenance concerns) and rely more on DNS-based discovery plus BanyanDB’s native property capabilities. BanyanDB: stronger stability testing: more systematic testing, including chaos testing, to validate behavior under failures and noisy conditions. BanyanDB: better observability export: introducing First Occurrence Data Collection (FODC) as a sidecar and proxy server to provide a unified stream of observability data to third-party systems. SkyWalking APM: broader runtime and query capabilities: cold-stage data query support, a newer Java runtime (Java 25), and consideration of TraceQL protocol (Temper) support. Closing Thanks to everyone who contributed to SkyWalking in 2025. Every contribution is high-value — code, documentation, reviews, testing, issue triage, and operational experience — and each of them helped move the project forward.\nWe also want to say a special thank you to the countless end users across global companies. Many of the most valuable improvements don’t start from a pull request: they start from real-world use cases, performance investigations, production feedback, bug reports, and the patience to help us reproduce and validate fixes.\nAs another milestone, SkyWalking reached 968 GitHub contributors globally, and we expect the 1000th contributor milestone to arrive soon in 2026. But the community is much larger than the number suggests, and SkyWalking’s progress has always been driven by collaboration between contributors, adopters, and maintainers.\nApache SkyWalking was originally created by Sheng Wu as a personal project in May 2015. It would never have grown into what it is today without the whole community — and it will keep moving forward because of the community.\n","excerpt":"\u003cp\u003e2025 was a very focused year for the Apache SkyWalking community: \u003cstrong\u003emoving BanyanDB from “native …\u003c/strong\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2026-01-01-skywalking-2025-year-in-review/","title":"Apache SkyWalking 2025 in Review: Making BanyanDB Ready for Production"},{"body":"SkyWalking Kubernetes Helm Chart 4.8.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed Bump up banyandb-helm by @hanahmily in https://github.com/apache/skywalking-helm/pull/162 Inconsistency in Volume Mounts Between oap-deployment.yaml and oap-job.yaml Causes Init Job to Fail by @NoodlesWang2024 in https://github.com/apache/skywalking-helm/pull/163 Update Chart.yaml by @hanahmily in https://github.com/apache/skywalking-helm/pull/164 fix: update skywalking-banyandb-helm version to 0.5.0-rc3 by @hanahmily in https://github.com/apache/skywalking-helm/pull/165 feat: update Banyandb configuration and authentication settings by @hanahmily in https://github.com/apache/skywalking-helm/pull/166 Update skywalking-banyandb-helm version to 0.5.0 by @hanahmily in https://github.com/apache/skywalking-helm/pull/167 chore: bump BanyanDB Helm version to 0.5.2 and update OAP/UI image tags to 10.3.0 by @hanahmily in https://github.com/apache/skywalking-helm/pull/168 chore: draft 4.8.0 by @kezhenxu94 in https://github.com/apache/skywalking-helm/pull/169 Update skywalking-banyandb-helm version to 0.5.3 by @hanahmily in https://github.com/apache/skywalking-helm/pull/170 chore: draft release for 4.8.0 by @kezhenxu94 in https://github.com/apache/skywalking-helm/pull/171 New Contributors @NoodlesWang2024 made their first contribution in https://github.com/apache/skywalking-helm/pull/163 Full Changelog: https://github.com/apache/skywalking-helm/compare/v4.7.0...v4.8.0\n","excerpt":"\u003cp\u003eSkyWalking Kubernetes Helm Chart 4.8.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars. …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-kubernetes-helm-chart-4.8.0/","title":"Release Apache SkyWalking Kubernetes Helm Chart 4.8.0"},{"body":"SkyWalking BanyanDB Helm 0.5.3 is released. Go to downloads page to find release tars.\nBugs Fix missing port in data node list for liaison ","excerpt":"\u003cp\u003eSkyWalking BanyanDB Helm 0.5.3 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"bugs\"\u003eBugs\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-helm-0-5-3/","title":"Release Apache SkyWalking BanyanDB Helm 0.5.3"},{"body":"SkyWalking BanyanDB 0.9.0 is released. Go to downloads page to find release tars.\nFeatures Add sharding_key for TopNAggregation source measure. API: Update the data matching rule from the node selector to the stage name. Add dynamical TLS load for the gRPC and HTTP server. Implement multiple groups query in one request. Replica: Replace Any with []byte Between Liaison and Data Nodes. Replica: Support configurable replica count on Group. Replica: Move the TopN pre-calculation flow from the Data Node to the Liaison Node. Add a wait and retry to write handlers to avoid the local metadata cache being loaded. Implement primary block cache for measure. Implement versioning properties and replace physical deletion with the tombstone mechanism for the property database. Implement skipping index for stream. Add Load Balancer Feature to Liaison. Implement fadvise for large files to prevent page cache pollution. Data Model: Introduce the Trace data model to store the trace/span data. Support dictionary encoding for low cardinality columns. Push down aggregation for topN query. Push down min/max aggregation to data nodes. Introduce write queue mechanism in liaison nodes to efficiently synchronize stream and measure partition folders, improving write throughput and consistency. Add trace module metadata management. Add chunked data sync to improve memory efficiency and performance during data transfer operations, supporting configurable chunk sizes, retry mechanisms, and out-of-order handling for both measure and stream services. Implement comprehensive migration system for both measure and stream data with file-based approach and enhanced progress tracking. Backup/Restore: Add support for AWS S3, Google Cloud Storage (GCS), and Azure Blob Storage as remote targets for backup and restore operations. Improve TopN processing by adding \u0026ldquo;source\u0026rdquo; tag to track node-specific data, enhancing data handling across distributed nodes. Implement Login with Username/Password authentication in BanyanDB. Enhance flusher and introducer loops to support merging operations, improving efficiency by eliminating the need for a separate merge loop and optimizing data handling process during flushing and merging. Enhance stream synchronization with configurable sync interval - Allows customization of synchronization timing for better performance tuning. Refactor flusher and introducer loops to support conditional merging - Optimizes data processing by adding conditional logic to merge operations. New storage engine for trace: Data ingestion and retrieval, Flush memory data to disk, Merge memory data and disk data. Enhance access log functionality with sampling option. Implement a resilient publisher with circuit breaker and retry logic with exponential backoff. Optimize gRPC message size limits: increase server max receive message size to 16MB and client max receive message size to 32MB for better handling of large time-series data blocks. Add query access log support for stream, measure, trace, and property services to capture and log all query requests for monitoring and debugging purposes. Implement comprehensive version compatibility checking for both regular data transmission and chunked sync operations, ensuring proper API version and file format version validation with detailed error reporting and graceful handling of version mismatches. Breaking Change: Rename disk usage configuration flags and implement forced retention cleanup. Implement cluster mode for trace. Implement Trace views. Use Fetch request to instead of axios request and remove axios. Implement Trace Tree for debug mode. Implement bydbQL. UI: Implement the Query Page for BydbQL. Refactor router for better usability. Implement the handoff queue for Trace. Add dump command-line tool to parse and display trace part data with support for CSV export and human-readable timestamp formatting. Implement backoff retry mechanism for sending queue failures. Implement memory load shedding and dynamic gRPC buffer sizing for liaison server to prevent OOM errors under high-throughput write traffic. Add stream dump command to parse and display stream shard data with support for CSV export, filtering, and projection. Bug Fixes Fix the deadlock issue when loading a closed segment. Fix the issue that the etcd watcher gets the historical node registration events. Fix the crash when collecting the metrics from a closed segment. Fix topN parsing panic when the criteria is set. Remove the indexed_only field in TagSpec. Fix returning empty result when using IN operator on the array type tags. Fix memory leaks and OOM issues in streaming processing by implementing deduplication logic in priority queues and improving sliding window memory management. Fix etcd prefix matching any key that starts with this prefix. Fix the sorting timestamps issue of the measure model when there are more than one segment. Fix comparison issues in TopN test cases. Document Introduce AI_CODING_GUIDELINES.md to provide guidelines for using AI assistants (like Claude, Cursor, GitHub Copilot) in development, ensuring generated code follows project standards around variable shadowing, imports, error handling, code style and documentation. ","excerpt":"\u003cp\u003eSkyWalking BanyanDB 0.9.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"features\"\u003eFeatures\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eAdd …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-0-9-0/","title":"Release Apache SkyWalking BanyanDB 0.9.0"},{"body":"SkyWalking BanyanDB Helm 0.5.1 is released. Go to downloads page to find release tars.\nFeatures Support data node list in the liaison Bugs Fix missing env in the restore init container ","excerpt":"\u003cp\u003eSkyWalking BanyanDB Helm 0.5.1 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures …\u003c/h4\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-helm-0-5-1/","title":"Release Apache SkyWalking BanyanDB Helm 0.5.1"},{"body":"SkyWalking 10.3.0 is released. Go to downloads page to find release tars.\nNew Trace Model in BanyanDB Optimized the Trace model implementation for BanyanDB 0.9.0, significantly reducing query frequency between OAP and BanyanDB. Introduced new query views based on the latest query features, greatly reducing page latency.\nProject Bump up BanyanDB dependency version(server and java-client) to 0.9.0. Fix CVE-2025-54057, restrict and validate url for widgets. Fix MetricsPersistentWorker, remove DataCarrier queue from Hour/Day dimensions metrics persistent process. This is important to reduce memory cost and Hour/Day dimensions metrics persistent latency. [Break Change] BanyanDB: support new Trace model. OAP Server Implement self-monitoring for BanyanDB via OAP Server. BanyanDB: Support hot/warm/cold stages configuration. Fix query continues profiling policies error when the policy is already in the cache. Support hot/warm/cold stages TTL query in the status API and graphQL API. PromQL Service: traffic query support limit and regex match. Fix an edge case of HashCodeSelector(Integer#MIN_VALUE causes ArrayIndexOutOfBoundsException). Support Flink monitoring. BanyanDB: Support @ShardingKey for Measure tags. BanyanDB: Support cold stage data query for metrics/traces/logs. Increase the idle check interval of the message queue to 200ms to reduce CPU usage under low load conditions. Limit max attempts of DNS resolution of Istio ServiceEntry to 3, and do not wait for first resolution result in case the DNS is not resolvable at all. Support analysis waypoint metrics in Envoy ALS receiver. Add Ztunnel component in the topology. [Break Change] Change componentId to componentIds in the K8SServiceRelation Scope. Adapt the mesh metrics if detect the ambient mesh in the eBPF access log receiver. Add JSON format support for the /debugging/config/dump status API. Enhance status APIs to support multiple accept header values, e.g. Accept: application/json; charset=utf-8. Storage: separate SpanAttachedEventRecord for SkyWalking trace and Zipkin trace. [Break Change]BanyanDB: Setup new Group policy. Bump up commons-beanutils to 1.11.0. Refactor: simplify the Accept http header process. [Break Change]Storage: Move event from metrics to records. Remove string limitation in Jackson deserializer for ElasticSearch client. Fix disable.oal does not work. Enhance the stability of e2e PHP tests and update the PHP agent version. Add component ID for the dameng JDBC driver. BanyanDB: Support custom TopN pre-aggregation rules configuration in file bydb-topn.yml. refactor: implement OTEL handler with SPI for extensibility. chore: add toString implementation for StorageID. chore: add a warning log when connecting to ES takes too long. Fix the query time range in the metadata API. OAP gRPC-Client support Health Check. [Break Change] health_check_xx metrics make response 1 represents healthy, 0 represents unhealthy. Bump up grpc to 1.70.0. BanyanDB: support new Index rule type SKIPPING/TREE, and update the record log\u0026rsquo;s trace_id indexType to SKIPPING BanyanDB: remove index-only from tag setting. Fix analysis tracing profiling span failure in ES storage. Add UI dashboard for Ruby runtime metrics. Tracing Query Execution HTTP APIs: make the argument service layer optional. GraphQL API: metadata, topology, log and trace support query by name. [Break Change] MQE function sort_values sorts according to the aggregation result and labels rather than the simple time series values. Self Observability: add metrics_aggregation_queue_used_percentage and metrics_persistent_collection_cached_size metrics for the OAP server. Optimize metrics aggregate/persistent worker: separate OAL and MAL workers and consume pools. The dataflow signal drives the new MAL consumer, the following table shows the pool size, driven mode and queue size for each worker. Worker poolSize isSignalDrivenMode queueChannelSize queueBufferSize MetricsAggregateOALWorker Math.ceil(availableProcessors * 2 * 1.5) false 2 10000 MetricsAggregateMALWorker availableProcessors * 2 / 8, at least 1 true 1 1000 MetricsPersistentMinOALWorker availableProcessors * 2 / 8, at least 1 false 1 2000 MetricsPersistentMinMALWorker availableProcessors * 2 / 16, at least 1 true 1 1000 Bump up netty to 4.2.4.Final. Bump up commons-lang to 3.18.0. BanyanDB: support group replicas and user/password for basic authentication. BanyanDB: fix Zipkin query missing tag QUERY. Fix IllegalArgumentException: Incorrect number of labels, tags in the LogReportServiceHTTPHandler and LogReportServiceGrpcHandler inconsistent with LogHandler. BanyanDB: fix Zipkin query by annotationQuery HTTP Server: Use the default shared thread pool rather than creating a new event loop thread pool for each server. Remove the MAX_THREADS from each server config. Optimize all Armeria HTTP Server(s) to share the CommonPools for the whole JVM. In the CommonPools, the max threads for EventLoopGroup is processor * 2, and for BlockingTaskExecutor is 200 and can be recycled if over the keepAliveTimeMillis (60000L by default). Here is a summary of the thread dump without UI query in a simple Kind env deployed by SkyWalking showcase: Thread Type Count Main State Description JVM System Threads 12 RUNNABLE/WAITING Includes Reference Handler, Finalizer, Signal Dispatcher, Service Thread, C2/C1 CompilerThreads, Sweeper thread, Common-Cleaner, etc. Netty I/O Worker Threads 32 RUNNABLE Threads named \u0026ldquo;armeria-common-worker-epoll-*\u0026rdquo;, handling network I/O operations. gRPC Worker Threads 16 RUNNABLE Threads named \u0026ldquo;grpc-default-worker-*\u0026rdquo;. HTTP Client Threads 4 RUNNABLE Threads named \u0026ldquo;HttpClient-*-SelectorManager\u0026rdquo;. Data Consumer Threads 47 TIMED_WAITING (sleeping) Threads named \u0026ldquo;DataCarrier.*\u0026rdquo;, used for metrics data consumption. Scheduled Task Threads 10 TIMED_WAITING (parking) Threads named \u0026ldquo;pool--thread-\u0026rdquo;. ForkJoinPool Worker Threads 2 WAITING (parking) Threads named \u0026ldquo;ForkJoinPool-*\u0026rdquo;. BanyanDB Processor Threads 2 TIMED_WAITING (parking) Threads named \u0026ldquo;BanyanDB BulkProcessor\u0026rdquo;. gRPC Executor Threads 3 TIMED_WAITING (parking) Threads named \u0026ldquo;grpc-default-executor-*\u0026rdquo;. JVM GC Threads 13 RUNNABLE Threads named \u0026ldquo;GC Thread#*\u0026rdquo; for garbage collection. Other JVM Internal Threads 3 RUNNABLE Includes VM Thread, G1 Main Marker, VM Periodic Task Thread. Attach Listener 1 RUNNABLE JVM attach listener thread. Total 158 - - BanyanDB: make BanyanDBMetricsDAO output scan all blocks info log only when the model is not indexModel. BanyanDB: fix the BanyanDBMetricsDAO.multiGet not work properly in IndexMode. BanyanDB: remove @StoreIDAsTag, and automatically create a virtual String tag id for the SeriesID in IndexMode. Remove method appendMutant from StorageID. Fix otlp log handler response error and otlp span convert error. Fix service_relation source layer in mq entry span analyse. Fix metrics comparison in promql with bool modifier. Add rate limiter for Zipkin trace receiver to limit maximum spans per second. Open health-checker module by default due to latest UI changes. Change the default check period to 30s. Refactor Kubernetes coordinator to be more accurate about node readiness. Bump up netty to 4.2.5.Final. BanyanDB: fix log query missing order by condition, and fix missing service id condition when query by instance id or endpoint id. Fix potential NPE in the AlarmStatusQueryHandler. Aggregate TopN Slow SQL by service dimension. BanyanDB: support add group prefix (namespace) for BanyanDB groups. BanyanDB: fix when setting @BanyanDB.TimestampColumn, the column should not be indexed. OAP Self Observability: make Trace analysis metrics separate by label protocol, add Zipkin span dropped metrics. BanyanDB: Move data write logic from BanyanDB Java Client to OAP and support observe metrics for write operations. Self Observability: add write latency metrics for BanyanDB and ElasticSearch. Fix the malfunctioning alarm feature of MAL metrics due to unknown metadata in L2 aggregate worker. Make MAL percentile align with OAL percentile calculation. Update Grafana dashboards for OAP observability. BanyanDB: fix query getInstance by instance ID. Support the go agent(0.7.0 release) bundled pprof profiling feature. Service and TCPService source support analyze TLS mode. Library-pprof-parser: feat: add PprofSegmentParser. Storage: feat: add languageType column to ProfileThreadSnapshotRecord. Feat: add go profile analyzer Get Alarm Runtime Status: support query the running status for the whole cluster. UI Implement self-monitoring for BanyanDB via UI. Enhance the trace List/Tree/Table graph to support displaying multiple refs of spans and distinguishing different parents. Fix: correct the same labels for metrics. Refactor: use the Fetch API to instead of Axios. Support cold stage data for metrics, trace and log. Add route to status API /debugging/config/dump in the UI. Implement the Status API on Settings page. Bump vite from 6.2.6 to 6.3.6. Enhance async profiling by adding shorter and custom duration options. Fix select wrong span to analysis in trace profiling. Correct the service list for legends in trace graphs. Correct endpoint topology data to avoid undefined. Fix the snapshot charts unable to display. Bump vue-i18n from 9.14.3 to 9.14.5. Fix split queries for topology to avoid page crash. Self Observability ui-template: Add new panels for monitor metrics aggregation queue used percentage and metrics persistent collection cached size. test: introduce and set up unit tests in the UI. test: implement comprehensive unit tests for components. refactor: optimize data types for widgets and dashboards. fix: optimize appearing the wrong prompt by pop-up for the HTTP environments in copy function. refactor the configuration view and implement the optional config for displaying timestamp in Log widget. test: implement unit tests for hooks and refactor some types. fix: share OAP proxy services for different endpoints and use health checked endpoints group. Optimize buttons in time picker component. Optimize the router system and implement unit tests for router. Bump element-plus from 2.9.4 to 2.11.0. Adapt new trace protocol and implement new trace view. Implement Trace page. Support collapsing and expanding for the event widget. UI-template: add BanyanDB and Elasticsearch write latency dashboards for OAP self observability. Documentation BanyanDB: Add Data Lifecycle Stages(Hot/Warm/Cold) documentation. Add SWIP-9 Support flink monitoring. Fix Metrics Attributes menu link. Implement the Status API on Settings page. Fix: Add the prefix for http url. Enhance the async-profiling duration options. Enhance the TTL Tab on Setting page. Fix the snapshot charts in alarm page. Fix Fluent Bit dead links. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking 10.3.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"new-trace-model-in-banyandb\"\u003eNew Trace Model in …\u003c/h3\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-10.3.0/","title":"Release Apache SkyWalking APM 10.3.0"},{"body":"SkyWalking BanyanDB Helm 0.5.0 is released. Go to downloads page to find release tars.\nFeatures Support Lifecycle Sidecar for automated data management across hot/warm/cold node roles with configurable schedules Introduce the data node template system to support different node roles (hot, warm, cold) with role-specific configurations Convert liaison component from Deployment to StatefulSet for improved state management and stable network identities Implement component-based storage configuration with separate data, liaison, and standalone sections. Enable external data and liaison storage by default with persistent volume claims Add headless services for StatefulSet pod discovery and stable network identities, enabling reliable pod-to-pod communication Add internal-grpc port 18912 for liaison pod-to-pod communication, enhancing cluster internal networking Enable etcd defragmentation by default with daily scheduling (0 0 * * *) to maintain optimal etcd performance Enhance pod hostname configuration using headless services for improved service discovery and networking Implement volume permissions init containers for proper file ownership and permissions on mounted volumes Add the mount target for the trace mode Add auth to configure the basic credential file. Set etcd\u0026rsquo;s reposiotry to \u0026ldquo;bitnamilegacy\u0026rdquo;. Bitnami removed non-hardened, Debian-based software images in its free tier under https://news.broadcom.com/app-dev/broadcom-introduces-bitnami-secure-images-for-production-ready-containerized-applications Bugs Fix missing \u0026ldquo;trace\u0026rdquo; mount target in liaison and standalone storage configurations Fix typo \u0026ldquo;observability\u0026rdquo; Chores Bump up e2e test cases. ","excerpt":"\u003cp\u003eSkyWalking BanyanDB Helm 0.5.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures …\u003c/h4\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-helm-0-5-0/","title":"Release Apache SkyWalking BanyanDB Helm 0.5.0"},{"body":"SkyWalking Eyes 0.8.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed Add support for Elixir by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/193 Fix twitter/x badge by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/194 Record the use of SkyWalking Eyes Docker Image under PowerShell 7 by @linghengqian in https://github.com/apache/skywalking-eyes/pull/196 Bump up some GitHub Actions by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/197 Bump golang.org/x/net from 0.33.0 to 0.36.0 by @dependabot[bot] in https://github.com/apache/skywalking-eyes/pull/195 Add few missing compatible licenses for dependency check by @kevinw66 in https://github.com/apache/skywalking-eyes/pull/198 Bump golang.org/x/net from 0.36.0 to 0.38.0 by @dependabot[bot] in https://github.com/apache/skywalking-eyes/pull/199 Bump github.com/cloudflare/circl from 1.3.7 to 1.6.1 by @dependabot[bot] in https://github.com/apache/skywalking-eyes/pull/200 Update golang.org/x/tools to v0.34.0 for Go 1.25 compatibility by @stefanb in https://github.com/apache/skywalking-eyes/pull/201 Fix null pointer panic in listFiles for empty repos and git worktrees by @fgksgf in https://github.com/apache/skywalking-eyes/pull/202 Fix version output for go install installations by @fgksgf in https://github.com/apache/skywalking-eyes/pull/203 Bump golang.org/x/oauth2 from 0.5.0 to 0.27.0 by @dependabot[bot] in https://github.com/apache/skywalking-eyes/pull/204 Ruby dependency license scanning support via Gemfile.lock. by @pboling in https://github.com/apache/skywalking-eyes/pull/205 fix: return error in license check, add MIT header by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/206 Fixed nil dependency panic in Ruby Gemfile.lock resolver by @pboling in https://github.com/apache/skywalking-eyes/pull/207 GemfileLockResolver adjusted to exclude all specs when runtime dependencies are empty by @pboling in https://github.com/apache/skywalking-eyes/pull/208 feat: Compatibility matrix: MIT \u0026amp; Ruby by @pboling in https://github.com/apache/skywalking-eyes/pull/209 Feat/add compat licenses by @pboling in https://github.com/apache/skywalking-eyes/pull/247 Fix Gemfile / Gemspec parser to ignore commented dependencies by @pboling in https://github.com/apache/skywalking-eyes/pull/249 Update example workflow in the README for the Ruby scenario by @pboling in https://github.com/apache/skywalking-eyes/pull/248 Add fsf-free and osi-approved options by @pboling in https://github.com/apache/skywalking-eyes/pull/250 New Contributors @linghengqian made their first contribution in https://github.com/apache/skywalking-eyes/pull/196 @kevinw66 made their first contribution in https://github.com/apache/skywalking-eyes/pull/198 @stefanb made their first contribution in https://github.com/apache/skywalking-eyes/pull/201 @pboling made their first contribution in https://github.com/apache/skywalking-eyes/pull/205 Full Changelog: https://github.com/apache/skywalking-eyes/compare/v0.7.0...v0.8.0\n","excerpt":"\u003cp\u003eSkyWalking Eyes 0.8.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-eyes-0-8-0/","title":"Release Apache SkyWalking Eyes 0.8.0"},{"body":"SkyWalking BanyanDB 0.9.1 is released. Go to downloads page to find release tars.\nFeatures Bump up the API to support sharding_key. Bump up the API to support version 0.9. Support stage query on TopN. Add replicas configuration to the API: introduce replicas in LifecycleStage and ResourceOpts to support high availability. Simplify TLS options: remove unsupported mTLS client certificate settings from Options and DefaultChannelFactory; trust CA is still supported. Support auth with username and password. Update gRPC to 1.75.0. Add histogram metrics to write/insert/update operations of the measure, stream and property. Bump up parent Apache pom to v35. Bump up maven to 3.6.3. Add IDEA setup doc to support super large generated file(by protoc). Add Trace model. ","excerpt":"\u003cp\u003eSkyWalking BanyanDB 0.9.1 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"features\"\u003eFeatures\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eBump up …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-java-client-0-9-1/","title":"Release Apache SkyWalking BanyanDB Java Client 0.9.1"},{"body":"SkyWalking Java Agent 9.5.0 is released. Go to downloads page to find release tars. Changes by Version\n9.5.0 Add the virtual thread executor plugin Fix Conflicts apm-jdk-threadpool-plugin conflicts with apm-jdk-forkjoinpool-plugin Fix NPE in hikaricp-plugin if JDBC URL is not set Agent kernel services could be not-booted-yet as ServiceManager#INSTANCE#boot executed after agent transfer initialization. Delay so11y metrics#build when the services are not ready to avoid MeterService status is not initialized. Fix retransform failure when enhancing both parent and child classes. Add support for dameng(DM) JDBC url format in URLParser. Fix RabbitMQ Consumer could not receive handleCancelOk callback. Support for tracking in lettuce versions 6.5.x and above. Upgrade byte-buddy version to 1.17.6. Support gRPC 1.59.x and 1.70.x server interceptor trace Fix the CreateAopProxyInterceptor in the Spring core-patch changes the AOP proxy type when a class is enhanced by both SkyWalking and Spring AOP. Build: Centralized plugin version management in the root POM and remove redundant declarations. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking Java Agent 9.5.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nChanges by …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-java-agent-9-5-0/","title":"Release Apache SkyWalking Java Agent 9.5.0"},{"body":"SkyWalking Rust 0.10.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed chore(ci): Replace archived actions-rs actions by @assignUser in https://github.com/apache/skywalking-rust/pull/68 Update rdkafka version by @jmjoy in https://github.com/apache/skywalking-rust/pull/69 Refactor Kafka client configuration to use a new ClientConfig struct by @jmjoy in https://github.com/apache/skywalking-rust/pull/70 Update documentation attributes for management and kafka modules by @jmjoy in https://github.com/apache/skywalking-rust/pull/71 Bump skywalking version to 0.10.0 and update LICENSE references by @jmjoy in https://github.com/apache/skywalking-rust/pull/72 New Contributors @assignUser made their first contribution in https://github.com/apache/skywalking-rust/pull/68 Full Changelog: https://github.com/apache/skywalking-rust/compare/v0.9.0...v0.10.0\n","excerpt":"\u003cp\u003eSkyWalking Rust 0.10.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-rust-0-10-0/","title":"Release Apache SkyWalking Rust 0.10.0"},{"body":"SkyWalking NodeJS 0.8.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed Bump up test tool and testcontainers to fix tests by @kezhenxu94 in https://github.com/apache/skywalking-nodejs/pull/122 Fix http trace not sent if connection is keep-alive by @liu-zhizhu in https://github.com/apache/skywalking-nodejs/pull/121 fix tag name by @heyanlong in https://github.com/apache/skywalking-nodejs/pull/123 Fix: Express plugin compatibility for Express 4.x and 5.x by @thiagomatar in https://github.com/apache/skywalking-nodejs/pull/126 chore: bump up to 0.8.0 and remove changelog file by @kezhenxu94 in https://github.com/apache/skywalking-nodejs/pull/127 New Contributors @liu-zhizhu made their first contribution in https://github.com/apache/skywalking-nodejs/pull/121 @heyanlong made their first contribution in https://github.com/apache/skywalking-nodejs/pull/123 @thiagomatar made their first contribution in https://github.com/apache/skywalking-nodejs/pull/126 Full Changelog: https://github.com/apache/skywalking-nodejs/compare/v0.7.0...v0.8.0\n","excerpt":"\u003cp\u003eSkyWalking NodeJS 0.8.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-nodejs-0.8.0/","title":"Release Apache SkyWalking for NodeJS 0.8.0"},{"body":"SkyWalking Python 1.2.0 is released! Go to downloads page to find release tars.\nPyPI Wheel: https://pypi.org/project/apache-skywalking/1.2.0/\nDockerHub Image: https://hub.docker.com/r/apache/skywalking-python\nWhat\u0026rsquo;s Changed Fix: user/password replacement is not allowed for relative urls by @tsonglew in https://github.com/apache/skywalking-python/pull/349 Fix outdated make dev-fix rule in CodeStyle.md by @tsonglew in https://github.com/apache/skywalking-python/pull/350 Fix pulsar client not support init arguments other than service_url by @tsonglew in https://github.com/apache/skywalking-python/pull/351 Drop support for 3.7 and fix tests by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/356 Fix TestClient for fastapi cause the req.client None error by @CharlieSeastar in https://github.com/apache/skywalking-python/pull/355 Feat sampling service by @tsonglew in https://github.com/apache/skywalking-python/pull/357 Fix agent start failed in async mode when profiling is enabled by @tsonglew in https://github.com/apache/skywalking-python/pull/360 Perf ignore uuid and timestamp generation for NoopContext by @tsonglew in https://github.com/apache/skywalking-python/pull/361 Feat add sw_grpc plugin by @tsonglew in https://github.com/apache/skywalking-python/pull/362 feature: add support to python 3.13 by @henriquemeca in https://github.com/apache/skywalking-python/pull/366 Add isSizeLimited in SegmentObject by @CodePrometheus in https://github.com/apache/skywalking-python/pull/367 Bump gunicorn from 20.1.0 to 23.0.0 by @dependabot in https://github.com/apache/skywalking-python/pull/368 Bump up to 1.2.0 and remove change log file by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/369 chore: fix linting error by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/370 New Contributors @CharlieSeastar made their first contribution in https://github.com/apache/skywalking-python/pull/355 @henriquemeca made their first contribution in https://github.com/apache/skywalking-python/pull/366 @dependabot made their first contribution in https://github.com/apache/skywalking-python/pull/368 Full Changelog: https://github.com/apache/skywalking-python/compare/v1.1.0...v1.2.0\n","excerpt":"\u003cp\u003eSkyWalking Python 1.2.0 is released! Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePyPI Wheel\u003c/strong\u003e: …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-python-1.2.0/","title":"Release Apache SkyWalking Python 1.2.0"},{"body":"SkyWalking Go 0.6.0 is released. Go to downloads page to find release tars.\nFeatures support attaching events to span in the toolkit. support record log in the toolkit. support manually report metrics in the toolkit. support manually set span error in the toolkit. Plugins Support goframev2 goframev2. Documentation Add docs for AddEvent in Tracing APIs Add Logging APIs document into Manual APIs. Add Metric APIs document into Manual APIs. Bug Fixes Fix wrong docker image name and -version command. Fix redis plugin cannot work in cluster mode. Fix cannot find file when exec build in test/plugins. Fix not set span error when http status code \u0026gt;= 400 Fix http plugin cannot provide peer name when optional Host is empty. Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Go 0.6.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003esupport …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-go-0.6.0/","title":"Release Apache SkyWalking Go 0.6.0"},{"body":"Background Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale.\nApache SkyWalking is an application performance monitor tool for distributed systems, especially designed for microservices, cloud native and container-based (Kubernetes) architectures.\nOpenTelemetry is a collection of APIs, SDKs, and tools. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior.\nSince SkyWalking 10.3, a new out-of-the-box feature has been introduced that enables Flink monitoring data to be visualized on the SkyWalking UI via the OpenTelemetry Collector, which gathers metrics from Flink endpoints.\nDevelopment Preparation SkyWalking OAP,v10.3 + Flink v2.0-preview1 + OpenTelemetry-collector v0.87+ Process Set up SkyWalking oap and UI. Set up the Flink cluster By configuring jobmanager and taskmanager to expose prometheus http endpoints. Set up OpenTelemetry-collector. Run your job. Data flow Configuration docker-compose version: \u0026#34;3\u0026#34; services: oap: extends: file: ../../script/docker-compose/base-compose.yml service: oap ports: - \u0026#34;12800:12800\u0026#34; networks: - e2e banyandb: extends: file: ../../script/docker-compose/base-compose.yml service: banyandb ports: - 17912 jobmanager: image: flink:2.0-preview1 environment: - | FLINK_PROPERTIES= jobmanager.rpc.address: jobmanager metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory metrics.reporter.prom.port: 9260 ports: - \u0026#34;8081:8081\u0026#34; - \u0026#34;9260:9260\u0026#34; command: jobmanager healthcheck: test: [\u0026#34;CMD\u0026#34;, \u0026#34;curl\u0026#34;, \u0026#34;-f\u0026#34;, \u0026#34;http://localhost:8081\u0026#34;] interval: 30s timeout: 10s retries: 3 networks: - e2e taskmanager: image: flink:2.0-preview1 environment: - | FLINK_PROPERTIES= jobmanager.rpc.address: jobmanager metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory metrics.reporter.prom.port: 9261 depends_on: jobmanager: condition: service_healthy ports: - \u0026#34;9261:9261\u0026#34; command: taskmanager healthcheck: test: [\u0026#34;CMD\u0026#34;, \u0026#34;curl\u0026#34;, \u0026#34;-f\u0026#34;, \u0026#34;http://localhost:9261/metrics\u0026#34;] interval: 30s timeout: 10s retries: 3 networks: - e2e executeJob: image: flink:2.0-preview1 depends_on: taskmanager: condition: service_healthy command: \u0026gt; bash -c \u0026#34; ./bin/flink run -m jobmanager:8081 examples/streaming/WindowJoin.jar\u0026#34; networks: - e2e otel-collector: image: otel/opentelemetry-collector:${OTEL_COLLECTOR_VERSION} networks: - e2e command: [ \u0026#34;--config=/etc/otel-collector-config.yaml\u0026#34; ] volumes: - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml expose: - 55678 depends_on: oap: condition: service_healthy networks: e2e: If you plan to expose metrics data using the pushGateway pattern, please refer to the documentation.\nOpenTelemetry-collector receivers: prometheus: config: scrape_configs: - job_name: \u0026#34;flink-jobManager-monitoring\u0026#34; scrape_interval: 30s static_configs: - targets: [\u0026#39;jobmanager:9260\u0026#39;] labels: cluster: flink-cluster relabel_configs: - source_labels: [ __address__ ] target_label: jobManager_node replacement: $$1 metric_relabel_configs: - source_labels: [ job_name ] action: replace target_label: flink_job_name replacement: $$1 - source_labels: [ ] target_label: job_name replacement: flink-jobManager-monitoring - job_name: \u0026#34;flink-taskManager-monitoring\u0026#34; scrape_interval: 30s static_configs: - targets: [ \u0026#34;taskmanager:9261\u0026#34; ] labels: cluster: flink-cluster relabel_configs: - source_labels: [ __address__ ] regex: (.+) target_label: taskManager_node replacement: $$1 metric_relabel_configs: - source_labels: [ job_name ] action: replace target_label: flink_job_name replacement: $$1 - source_labels: [ ] target_label: job_name replacement: flink-taskManager-monitoring exporters: otlp: endpoint: oap:11800 tls: insecure: true processors: batch: service: pipelines: metrics: receivers: - prometheus processors: - batch exporters: - otlp Warning:\nPlease do not edit the value of the job_name configuration, otherwise SkyWalking will not handle these data.\noap means the address of your SkyWalking oap address,please replace it accordingly.\nSince the original Flink metrics contain the job_name labels, and SkyWalking relies on the job_name label to handle OpenTelemetry data, to avoid conflicts, we use metric_relabel_configs to rename the original job_name label to flink_job_name.\nMetrics Definition Monitoring metrics involve in Cluster Metrics, TaskManager Metrics, and Job Metrics.\nCluster Metrics Cluster Metrics mainly focuses on statistics from the perspective of the entire cluster, as well as displaying JVM-related metrics of the JobManager, such as:\nRunning Jobs：The number of currently running jobs. TaskManagers：The number of TaskManagers. Task Managers Slots Total：The total number of TaskManager slots. Task Managers Slots Available：The number of available TaskManager slots. JVM CPU Load：The CPU load of the JobManager\u0026rsquo;s JVM. TaskManager Metrics TaskManager Metrics mainly focuses on statistics from the perspective of individual TaskManager nodes, such as:\nJVM Memory Heap Used：The amount of JVM heap memory used on the TaskManager node. JVM Memory Heap Available：The amount of JVM heap memory available on the TaskManager node. NumRecordsIn：The number of records received per minute by the TaskManager. NumBytesInPerSecond：The number of bytes received per second by the TaskManager. IsBackPressured：Indicates whether the TaskManager node is under backpressure. IdleTimeMsPerSecond：The idle time per second of the TaskManager node. Job Metrics Job Metricsmainly focuses on statistics from the perspective of running jobs, such as:\nJob RunningTime：The duration for which the job has been running. Job Restart Number：The number of times the job has been restarted. Checkpoints Failed：The number of failed checkpoints. NumBytesInPerSecond：The number of bytes received per second by the job. You can find explanations for each metric in the tip of the corresponding chart.\nReferences Flink Prometheus SkyWalking Flink Monitoring ","excerpt":"\u003ch1 id=\"background\"\u003eBackground\u003c/h1\u003e\n\u003cp\u003e\u003ca href=\"https://flink.apache.org/\"\u003eApache Flink\u003c/a\u003e is a framework and distributed processing engine for stateful computations …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2024-04-19-flink-monitoring-by-skywalking/","title":"Monitoring Flink with SkyWalking"},{"body":"背景介绍 Apache Flink 是一个框架和分布式处理引擎，用于在无边界和有边界数据流上进行有状态的计算。Flink 能在所有常见集群环境中运行，并能以内存速度和任意规模进行计算。 从SkyWalking OAP 10.3 版本开始，新增了对来自Flink的指标数据监控面板，本文将展示并介绍如何使用 SkyWalking来监控Flink。\n部署 准备 SkyWalking oap服务,v10.3 + Flink v2.0-preview1 + OpenTelemetry-collector v0.87+ 启动流程 启动 jobmanager 和 taskmanager 启动 skywalking oap 和 ui 启动 opentelmetry-collector 启动job DataFlow: 配置 docker-compose version: \u0026#34;3\u0026#34; services: oap: extends: file: ../../script/docker-compose/base-compose.yml service: oap ports: - \u0026#34;12800:12800\u0026#34; networks: - e2e banyandb: extends: file: ../../script/docker-compose/base-compose.yml service: banyandb ports: - 17912 jobmanager: image: flink:2.0-preview1 environment: - | FLINK_PROPERTIES= jobmanager.rpc.address: jobmanager metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory metrics.reporter.prom.port: 9260 ports: - \u0026#34;8081:8081\u0026#34; - \u0026#34;9260:9260\u0026#34; command: jobmanager healthcheck: test: [\u0026#34;CMD\u0026#34;, \u0026#34;curl\u0026#34;, \u0026#34;-f\u0026#34;, \u0026#34;http://localhost:8081\u0026#34;] interval: 30s timeout: 10s retries: 3 networks: - e2e taskmanager: image: flink:2.0-preview1 environment: - | FLINK_PROPERTIES= jobmanager.rpc.address: jobmanager metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory metrics.reporter.prom.port: 9261 depends_on: jobmanager: condition: service_healthy ports: - \u0026#34;9261:9261\u0026#34; command: taskmanager healthcheck: test: [\u0026#34;CMD\u0026#34;, \u0026#34;curl\u0026#34;, \u0026#34;-f\u0026#34;, \u0026#34;http://localhost:9261/metrics\u0026#34;] interval: 30s timeout: 10s retries: 3 networks: - e2e executeJob: image: flink:2.0-preview1 depends_on: taskmanager: condition: service_healthy command: \u0026gt; bash -c \u0026#34; ./bin/flink run -m jobmanager:8081 examples/streaming/WindowJoin.jar\u0026#34; networks: - e2e otel-collector: image: otel/opentelemetry-collector:${OTEL_COLLECTOR_VERSION} networks: - e2e command: [ \u0026#34;--config=/etc/otel-collector-config.yaml\u0026#34; ] volumes: - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml expose: - 55678 depends_on: oap: condition: service_healthy networks: e2e: 如果是使用pushGateWay模式来暴露metrics数据请参考。\nOpenTelemetry-collector receivers: prometheus: config: scrape_configs: - job_name: \u0026#34;flink-jobManager-monitoring\u0026#34; scrape_interval: 30s static_configs: - targets: [\u0026#39;jobmanager:9260\u0026#39;] labels: cluster: flink-cluster relabel_configs: - source_labels: [ __address__ ] target_label: jobManager_node replacement: $$1 metric_relabel_configs: - source_labels: [ job_name ] action: replace target_label: flink_job_name replacement: $$1 - source_labels: [ ] target_label: job_name replacement: flink-jobManager-monitoring - job_name: \u0026#34;flink-taskManager-monitoring\u0026#34; scrape_interval: 30s static_configs: - targets: [ \u0026#34;taskmanager:9261\u0026#34; ] labels: cluster: flink-cluster relabel_configs: - source_labels: [ __address__ ] regex: (.+) target_label: taskManager_node replacement: $$1 metric_relabel_configs: - source_labels: [ job_name ] action: replace target_label: flink_job_name replacement: $$1 - source_labels: [ ] target_label: job_name replacement: flink-taskManager-monitoring exporters: otlp: endpoint: oap:11800 tls: insecure: true processors: batch: service: pipelines: metrics: receivers: - prometheus processors: - batch exporters: - otlp 注意:\njob_name的值请不要修改,否则 skyWalking 不会处理这部分数据。\noap 为 skywalking oap 地址,请自行替换。\n因为原始flink数据中含有job_name标签，而skyWalking又根据job_name标签来处理对应OTEL任务的数据， 为了避免冲突，使用metric_relabel_configs替换原始数据中job_name的标签为flink_job_name。\n监控指标 指标分为三个维度,cluster,taskManager,job\nCluster Metrics Cluster Metrics主要是站在集群的角度统计以及jobManager的jvm相关指标展示,比如\nRunning Jobs：正在运行的任务数量 TaskManagers：taskManager数量 Task Managers Slots Total：taskManager slot数量 Task Managers Slots Available：taskManager可用slot数量 JVM CPU Load：jobManager的jvm占用cpu的负载 TaskManager Metrics TaskManager Metrics主要是站在taskManager节点的角度来统计展示,比如\nJVM Memory Heap Used：taskManager节点JVM已用内存大小。 JVM Memory Heap Available：taskManager节点JVM可用内存大小。 NumRecordsIn：taskManager每分钟接受的数据数量。 NumBytesInPerSecond：taskManager每秒接受的Bytes数量。 IsBackPressured：该taskManager节点是否处在背压。 IdleTimeMsPerSecond：该taskManager节点每秒的闲置时长。 Job Metrics Job Metrics主要是站在运行任务的角度来统计展示,比如\nJob RunningTime：该任务运行的时长。 Job Restart Number：该任务重启次数。 Checkpoints Failed：失败的checkpoints数量。 NumBytesInPerSecond：该任务每秒接受的Bytes数量。 各个指标的含义可以在图标的 tip 上找到解释\n参考文档 Flink Prometheus SkyWalking Flink Monitoring ","excerpt":"\u003ch1 id=\"背景介绍\"\u003e背景介绍\u003c/h1\u003e\n\u003cp\u003eApache Flink 是一个框架和分布式处理引擎，用于在无边界和有边界数据流上进行有状态的计算。Flink 能在所有常见集群环境中运行，并能以内存速度和任意规模进行计算。\n从 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2024-04-19-flink-monitoring-by-skywalking/","title":"使用 SkyWalking 监控 Flink"},{"body":"SkyWalking PHP 1.0.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed Fix tracing time by @CodePrometheus in https://github.com/apache/skywalking-php/pull/124 Start 1.0.0 development and Update CI by @jmjoy in https://github.com/apache/skywalking-php/pull/126 Support log reporting based on PSR-3 by @jmjoy in https://github.com/apache/skywalking-php/pull/127 Upgrade MSRV and dependencies by @jmjoy in https://github.com/apache/skywalking-php/pull/128 Updated skywalking dependency by @jmjoy in https://github.com/apache/skywalking-php/pull/131 Bump version to 1.0.0 for release by @jmjoy in https://github.com/apache/skywalking-php/pull/132 New Contributors @CodePrometheus made their first contribution in https://github.com/apache/skywalking-php/pull/124 Full Changelog: https://github.com/apache/skywalking-php/compare/v0.8.0...v1.0.0\nPECL https://pecl.php.net/package/skywalking_agent/1.0.0\n","excerpt":"\u003cp\u003eSkyWalking PHP 1.0.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-php-1-0-0/","title":"Release Apache SkyWalking PHP 1.0.0"},{"body":"SkyWalking Rust 0.9.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed Update NOTICE by @jmjoy in https://github.com/apache/skywalking-rust/pull/64 Migrate to edition 2024 and upgrade dependencies by @jmjoy in https://github.com/apache/skywalking-rust/pull/65 Release SkyWalking Rust 0.9.0 by @jmjoy in https://github.com/apache/skywalking-rust/pull/66 Full Changelog: https://github.com/apache/skywalking-rust/compare/v0.8.0...v0.9.0\n","excerpt":"\u003cp\u003eSkyWalking Rust 0.9.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-rust-0-9-0/","title":"Release Apache SkyWalking Rust 0.9.0"},{"body":"SkyWalking BanyanDB 0.8.0 is released. Go to downloads page to find release tars.\nFeatures Add the bydbctl analyze series command to analyze the series data. Index: Remove sortable field from the stored field. If a field is sortable only, it won\u0026rsquo;t be stored. Index: Support InsertIfAbsent functionality which ensures documents are only inserted if their docIDs are not already present in the current index. There is a exception for the documents with extra index fields more than the entity\u0026rsquo;s index fields. Measure: Introduce \u0026ldquo;index_mode\u0026rdquo; to save data exclusively in the series index, ideal for non-timeseries measures. Index: Use numeric index type to support Int and Float TopN: Group top n pre-calculation result by the group key in the new introduced _top_n_result measure, which is used to store the pre-calculation result. Index Mode: Index measure_nam and tags in entity to improve the query performance. Encoding: Improve the performance of encoding and decoding the variable-length int64. Index: Add a cache to improve the performance of the series index write. Read cpu quota and limit from the cgroup file system to set gomaxprocs. Property: Add native storage layer for property. Add the max disk usage threshold for the Measure, Stream, and Property to control the disk usage. Add the \u0026ldquo;api version\u0026rdquo; service to gRPC and HTTP server. Metadata: Wait for the existing registration to be removed before registering the node. Stream: Introduce the batch scan to improve the performance of the query and limit the memory usage. Add memory protector to protect the memory usage of the system. It will limit the memory usage of the querying. Metadata: Introduce the periodic sync to sync the metadata from the etcd to the local cache in case of the loss of the events. Test: Add the e2e test for zipkin. Test: Limit the CPU and memory usage of the e2e test. Add taking the snapshot of data files. Add backup command line tool to backup the data files. Add restore command line tool to restore the data files. Add concurrent barrier to partition merge to improve the performance of the partition merge. Improve the write performance. Add node labels to classify the nodes. Add lifecycle management for the node. Property: Introduce the schema style to the property. Add time range parameters to stream index filter. UI: Add the stages to groups. Add time range return value from stream local index filter. Deduplicate the documents on building the series index. Bug Fixes Fix the bug that TopN processing item leak. The item can not be updated but as a new item. Resolve data race in Stats methods of the inverted index. Fix the bug when adding new tags or fields to the measure, the querying crashes or returns wrong results. Fix the bug that adding new tags to the stream, the querying crashes or returns wrong results. UI: Polish Index Rule Binding Page and Index Page. Fix: View configuration on Property page. UI: Add indexMode to display on the measure page. UI: Refactor Groups Tree to optimize style and fix bugs. UI: Add NoSort Field to IndexRule page. Metadata: Fix the bug that the cache load nil value that is the unknown index rule on the index rule binding. Queue: Fix the bug that the client remove a registered node in the eviction list. The node is controlled by the recovery loop, doesn\u0026rsquo;t need to be removed in the failover process. UI: Add prettier to enforce a consistent style by parsing code. Parse string and int array in the query result table. Fix the bug that fails to update Group Schema\u0026rsquo;s ResourceOpts. UI: Implement TopNAggregation data query page. UI: Update BanyanDB UI to Integrate New Property Query API. UI: Fix the Stream List. Fix the oom issue when loading too many unnecessary parts into memory. bydbctl: Fix the bug that the bydbctl can\u0026rsquo;t parse the absolute time flag. Documentation Improve the description of the memory in observability doc. Update kubernetes install document to align the banyandb helm v0.3.0. Add restrictions on updating schema. Add docs for the new property storage. Update quick start guide to use showcase instead of the old example. Chores Fix metrics system typo. Bump up OAP in CI to 6d262cce62e156bd197177abb3640ea65bb2d38e. Update cespare/xxhash to v2 version. Bump up Go to 1.24. CVEs GO-2024-3321: Misuse of ServerConfig.PublicKeyCallback may cause authorization bypass in golang.org/x/crypto GO-2024-3333: Non-linear parsing of case-insensitive content in golang.org/x/net/html ","excerpt":"\u003cp\u003eSkyWalking BanyanDB 0.8.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"features\"\u003eFeatures\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eAdd the …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-0-8-0/","title":"Release Apache SkyWalking BanyanDB 0.8.0"},{"body":"SkyWalking Client JS 1.0.0 is released. Go to downloads page to find release tars.\nMonitor Core Web Vitals. Monitor static resource metrics. Bump up infra-e2e. Bump dependencies to fix vulnerabilities. Adjust readme for 1.0 release. Fix can\u0026rsquo;t catch the resource error. Fix append http method to tags error. Bump up test ui. Fix the caught fetch request does not work when it receives a URL. ","excerpt":"\u003cp\u003eSkyWalking Client JS 1.0.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eMonitor Core Web …\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-client-js-1-0-0/","title":"Release Apache SkyWalking Client JS 1.0.0"},{"body":"SkyWalking BanyanDB Helm 0.4.0 is released. Go to downloads page to find release tars.\nFeatures Support Backup Sidecar and Restore Init Container Leave the image tag empty to force the users to specify the image tag Add a default volume for \u0026ldquo;property\u0026rdquo; ","excerpt":"\u003cp\u003eSkyWalking BanyanDB Helm 0.4.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures …\u003c/h4\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-helm-0-4-0/","title":"Release Apache SkyWalking BanyanDB Helm 0.4.0"},{"body":"SkyWalking BanyanDB 0.8.0 is released. Go to downloads page to find release tars.\nFeatures Bump up the API to support the index mode of Measure. Bump up the API to support the new property. Bump up the API to adopt the status field which is changed to the string type due to the compatibility issue. Bump up the API to support getting the API version. Bump up the API to support the lifecycle management. ","excerpt":"\u003cp\u003eSkyWalking BanyanDB 0.8.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"features\"\u003eFeatures\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eBump up …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-java-client-0-8-0/","title":"Release Apache SkyWalking BanyanDB Java Client 0.8.0"},{"body":"SkyWalking 10.2.0 is released. Go to downloads page to find release tars.\nNo H2, More BanyanDB Add BanyanDB 0.8.0 support. H2 storage option is removed permanently. Project Add doc_values for fields that need to be sorted or aggregated in Elasticsearch, and disable all others. This change would not impact the existing deployment and its feature for our official release users. Warning If there are custom query plugins for our Elasticsearch indices, this change could break them as sort queries and aggregation queries which used the unexpected fields are being blocked. [Breaking Change] Rename debugging-query module to status-query module. Relative exposed APIs are UNCHANGED. [Breaking Change] All jars of the skywalking-oap-server are no longer published through maven central. We will only publish the source tar and binary tar to the website download page, and docker images to docker hub. Warning If you are using the skywalking-oap-server as a dependency in your project, you need to download the source tar from the website and publish them to your private maven repository. [Breaking Change] Remove H2 as storage option permanently. BanyanDB 0.8(OAP 10.2 required) is easy, stable and production-ready. Don\u0026rsquo;t need H2 as default storage anymore. [Breaking Change] Bump up BanyanDB server version to 0.8.0. This version is not compatible with the previous versions. Please upgrade the BanyanDB server to 0.8.0 before upgrading OAP to 10.2.0. Bump up nodejs to v22.14.0 for the latest UI(booster-ui) compiling. Migrate tj-actions/changed-files to dorny/paths-filter. OAP Server Skip processing OTLP metrics data points with flag FLAG_NO_RECORDED_VALUE, which causes exceptional result. Add self observability metrics for GraphQL query, graphql_query_latency. Reduce the count of process index and adding time range when query process index. Bump up Apache commons-io to 2.17.0. Polish eBPF so11y metrics and add error count for query metrics. Support query endpoint list with duration parameter(optional). Change the endpoint_traffic to updatable for the additional column last_ping. Add Component ID(5023) for the GoZero framework. Support Kong monitoring. Support adding additional attr[0-5] for service/endpoint level metrics. Support async-profiler feature for performance analysis. Add metrics value owner for metrics topN query result. Add naming control for EndpointDependencyBuilder. The index type BanyanDB.IndexRule.IndexType#TREE is removed. All indices are using IndexType#INVERTED now. Add max query size settings to BanyanDB. Fix \u0026ldquo;BanyanDBTraceQueryDAO.queryBasicTraces\u0026rdquo; doesn\u0026rsquo;t support querying by \u0026ldquo;trace_id\u0026rdquo;. Polish mesh data dispatcher: don\u0026rsquo;t generate Instance/Endpoint metrics if they are empty. Adapt the new metadata standardization in Istio 1.24. Bump up netty to 4.1.115, grpc to 1.68.1, boringssl to 2.0.69. BanyanDB: Support update the Group settings when OAP starting. BanyanDB: Introduce index mode and refactor banyandb group settings. BanyanDB: Introduce the new Progressive TTL feature. BanyanDB: Support update the Schema when OAP starting. BanyanDB: Speed up OAP booting while initializing BanyanDB. BanyanDB: Support @EnableSort on the column to enable sorting for IndexRule and set the default to false. Support Get Effective TTL Configurations API. Fix ServerStatusService.statusWatchers concurrent modification. Add protection for dynamic config change propagate chain. Add Ruby component IDs(HttpClient=2, Redis=7, Memcached=20, Elasticsearch=47, Ruby=12000, Sinatra=12001). Add component ID(160) for Caffeine. Alarm: Support store and query the metrics snapshot when the alarm is triggered. Alarm: Remove unused Alarm Trend query. Fix missing remote endpoint IP address in span query of zipkin query module. Fix hierarchy-definition.yml config file packaged into start.jar wrongly. Add bydb.dependencies.properties config file to define server dependency versions. Fix AvgHistogramPercentileFunction doesn\u0026rsquo;t have proper field definition for ranks. BanyanDB: Support the new Property data module. MQE: Support top_n_of function for merging multiple metrics topn query. Support labelAvg function in the OAL engine. Added maxLabelCount parameter in the labelCount function of OAL to limit the number of labels can be counted. Adapt the new Browser API(/browser/perfData/webVitals, /browser/perfData/webInteractions, /browser/perfData/resources) protocol. Add Circuit Breaking mechanism. BanyanDB: Add support for compatibility checks based on the BanyanDB server\u0026rsquo;s API version. MQE: Support \u0026amp;\u0026amp;(and), ||(or) bool operators. OAP self observability: Add JVM heap and direct memory used metrics. OAP self observability: Add watermark circuit break/recover metrics. AI Pipeline: Support query baseline metrics names and predict metrics value. Add Get Node List in the Cluster API. Add type descriptor when converting Envoy logs to JSON for persistence, to avoid conversion error. Bseline: Support query baseline with MQE and use in the Alarm Rule. Bump up netty to 4.11.118 to fix CVE-2025-24970. Add Get Alarm Runtime Status API. Add lock when query the Alarm metrics window values. Add a fail-safe mechanism to prevent traffic metrics inconsistent between in-memory and database server. Add more clear logs when oap-cluster-internal data(metrics/traffic) format is inconsistent. Optimize metrics cache loading when trace latency greater than cache timeout. Allow calling lang.groovy.GString in DSL. BanyanDB: fix alarm query result without sort. Add a component ID for Virtual thread executor. Add more model installation log info for OAP storage initialization. BanyanDB: Separate the storage configuration to an independent file: bydb.yaml. Bump Armeria to 1.32.0 and some transitive dependencies. Skip persisting metrics/record data that have been expired. Fix the issue of missing Last Ping data. Add HTTP headers configuration for the alarm webhook. Bump up BanyanDB java client to 0.8.0. UI Add support for case-insensitive search in the dashboard list. Add content decorations to Table and Card widgets. Support the endpoint list widget query with duration parameter. Support ranges for Value Mappings. Add service global topN widget on General-Root, Mesh-Root, K8S-Root dashboard. Fix initialization dashboards. Update the Kubernetes metrics for reduce multiple metrics calculate in MQE. Support view data value related dashboards in TopList widgets. Add endpoint global topN widget on General-Root, Mesh-Root. Implement owner option for TopList widgets in related trace options. Hide entrances to unrelated dashboards in topn list. Split topology metric query to avoid exceeding the maximum query complexity. Fix view metrics related trace and metrics query. Add support collapse span. Refactor copy util with Web API. Releases an existing object URL. Optimize Trace Profiling widget. Implement Async Profiling widget. Fix inaccurate data query issue on endpoint topology page. Update browser dashboard for the new metrics. Visualize Snapshot on Alerting page. OAP self observability dashboard: Add JVM heap and direct memory used metrics. OAP self observability dashboard: Add watermark circuit break/recover metrics. Implement the legend selector in metrics charts. Fix repetitive names in router. Bump up dependencies. Fixes tooltips cannot completely display metrics information. Fix time range when generate link. Add skywalking go agent self observability menu. add topN selector for endpoint list. Documentation Update release document to adopt newly added revision-based process. Improve BanyanDB documentation. Improve component-libraries documentation. Improve configuration-vocabulary documentation. Add Get Effective TTL Configurations API documentation. Add Status APIs docs. Simplified the release process with removing maven central publish relative processes. Add Circuit Breaking mechanism doc. Add Get Node List in the Cluster API doc. Remove meter.md doc, because mal.md has covered all the content. Merge browser-http-api-protocol.md doc into browser-protocol.md. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking 10.2.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"no-h2-more-banyandb\"\u003eNo H2, More BanyanDB\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eAdd …\u003c/strong\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-10.2.0/","title":"Release Apache SkyWalking APM 10.2.0"},{"body":"Background Ruby is a dynamic, object-oriented programming language with concise and elegant syntax, supporting multiple programming paradigms, including object-oriented, functional, and metaprogramming. Leveraging its powerful metaprogramming capabilities, Ruby allows modifying the behavior of classes and objects at runtime. SkyWalking provides a Ruby gem to facilitate integration with Ruby projects, and this gem supports many out-of-the-box frameworks and gems.\nThis article is based on skywalking-ruby-v0.1. We will guide you on how to quickly integrate the skywalking-ruby project into Ruby projects and briefly introduce the implementation principle of SkyWalking Ruby\u0026rsquo;s auto-instrumentation plugins using redis-rb as an example.\nThe demonstration includes the following steps:\nDeploy SkyWalking: This involves setting up the SkyWalking backend and UI programs to enable you to see the final results. Integrate SkyWalking into Different Ruby Projects: This section explains how to integrate SkyWalking into different Ruby projects. Application Deployment: You will export environment variables and deploy the application to facilitate communication between your service and the SkyWalking backend. Visualization on SkyWalking UI: Finally, you will send requests and observe the results in the SkyWalking UI. Deploy SkyWalking Please download the SkyWalking APM program from the official SkyWalking website, and then you can start all the required services using the quick start script.\nNext, you can access the address http://localhost:8080/. At this point, since no applications have been deployed, you will not see any data.\nIntegrate SkyWalking into Different Ruby Projects It is recommended to use Bundler to install and manage SkyWalking dependencies. Simply declare it in the Gemfile and run bundle install to complete the installation.\n# Gemfile source \u0026#34;https://rubygems.org\u0026#34; gem \u0026#34;skywalking\u0026#34; Integration in Rails Projects For Rails projects, it is recommended to use the following command to automatically generate the configuration file:\nbundle exec rails generate skywalking:start This command will automatically generate a skywalking.rb file in the config/initializers directory, where you can configure the startup parameters.\nIntegration in Sinatra Projects For Sinatra projects, you need to manually call Skywalking.start when the application starts. For example:\nrequire \u0026#39;sinatra\u0026#39; require \u0026#39;skywalking\u0026#39; Skywalking.start get \u0026#39;/sw\u0026#39; do \u0026#34;Hello SkyWalking!\u0026#34; end In the Gemfile, place skywalking after sinatra and use Bundler.require during initialization, or call require 'skywalking' after the sinatra gem is loaded. Note that the skywalking gem needs to be placed after other gems (such as redis, elasticsearch).\nApplication Deployment Before starting the application deployment, you can change the service name of the current application in SkyWalking through environment variables. You can also modify its configuration, such as the server-side address. For more details, please refer to the documentation.\nHere, we will change the current service name to sw-ruby.\nNext, you can start the application. Here is an example using sinatra:\nexport SW_AGENT_SERVICE_NAME=sw-ruby ruby sinatra.rb Visualization on SkyWalking UI Now, send requests to the application and observe the results in the SkyWalking UI.\nAfter a few seconds, revisit the SkyWalking UI at http://localhost:8080. You will be able to see the deployed demo service on the homepage.\nAdditionally, on the tracing page, you can see the request you just sent.\nPlugin Implementation Mechanism To understand the implementation mechanism of Ruby Agent\u0026rsquo;s auto-instrumentation plugins, it is essential to understand the concept of the ancestor chain in Ruby. The ancestor chain is an ordered list, and in Ruby, each class or module has an ancestor chain that includes all its parent classes and mixin modules (modules mixed in via include, prepend, or extend). When Ruby looks up a method, it searches in the order of the ancestor chain until it finds the target method or throws a NoMethodError.\nclass User end We have defined a User class, and its ancestor chain is as shown in the following figure:\nNext, mix in a module using the prepend method:\nmodule Dapper def brave \u0026#34;Hello from brave\u0026#34; end end class User prepend Dapper end p User.new.brave # =\u0026gt; \u0026#34;Hello from brave\u0026#34; prepend will insert at position 1 in the above figure. Ruby first looks for the brave method in the Dapper module, finds it, and calls it. If the brave method is not found in Dapper, Ruby continues to search in the User class. If it is not found in the User class, Ruby continues to search in Object, and so on.\nBased on this mechanism, let\u0026rsquo;s briefly introduce how we instrument the redis-rb method. The following code is the target method to be instrumented:\n# lib/redis/client.rb class Redis class Client \u0026lt; ::RedisClient def call_v(command, \u0026amp;block) super(command, \u0026amp;block) rescue ::RedisClient::Error =\u0026gt; error Client.translate_error!(error) end end end Below is the core code for instrumentation:\nmodule Skywalking module Plugins class Redis5 \u0026lt; PluginsManager::SWPlugin module Redis5Intercept def call_v(args, \u0026amp;block) operation = args[0] rescue \u0026#34;UNKNOWN\u0026#34; return super if operation == :auth Tracing::ContextManager.new_exit_span( operation: \u0026#34;Redis/#{operation.upcase}\u0026#34; ) do |span| # Omitted handling of span super(args, \u0026amp;block) # Call the original method end end end def install ::Redis::Client.prepend Redis5Intercept end end end end Here, we define a Redis5Intercept module and prepend it to ::Redis::Client. According to Ruby\u0026rsquo;s method lookup mechanism, when the call_v method of Redis::Client is called, Ruby will first execute the call_v method in Redis5Intercept. The order of the ancestor chain is as follows:\nRedis5Intercept -\u0026gt; Redis::Client -\u0026gt; ... (other parent classes and modules) At the same time, in the call_v method of Redis5Intercept, super(args, \u0026amp;block) will find the next method with the same name along the ancestor chain, which in this case is the original call_v method in Redis::Client, while passing the original arguments and block.\nConclusion This article explained the integration methods of SkyWalking Ruby in Ruby projects and briefly introduced the implementation mechanism of SkyWalking Ruby\u0026rsquo;s auto-instrumentation plugins.\nCurrently, the Ruby auto-instrumentation is in the early stages of development. In the future, we will continue to expand the functionality of SkyWalking Ruby and add support for more plugins. So, stay tuned!\n","excerpt":"\u003ch2 id=\"background\"\u003eBackground\u003c/h2\u003e\n\u003cp\u003eRuby is a dynamic, object-oriented programming language with concise and elegant syntax, …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2025-03-06-introduction-to-skywalking-ruby/","title":"SkyWalking Ruby Quick Start and Principle Introduction"},{"body":"背景 Ruby 是一种动态、面向对象的编程语言，它的语法简洁优雅，支持多种编程范式，包括面向对象、函数式和元编程。其中依靠强大的元编程能力，Ruby 允许在运行时修改类和对象的行为。 SkyWalking 提供了 Ruby gem，方便 Ruby 项目集成, 该 gem 支持许多开箱即用的框架 和 gem。\n本文基于 skywalking-ruby-v0.1，我们将指导你如何快速将 skywalking-ruby 项目集成到 Ruby 项目中，并以 redis-rb 为例，简要地介绍 SkyWalking Ruby 对插件自动探针的实现原理。\n演示部分包括以下步骤：\n部署 SkyWalking：这涉及设置 SkyWalking 后端和 UI 程序，使你能够看到最终效果。 为不同 Ruby 项目集成 skywalking：这里介绍了不同的 Ruby 项目如何集成 skywalking。 应用部署：你将导出环境变量并部署应用程序，以促进你的服务与 SkyWalking 后端之间的通信。 在 SkyWalking UI 上可视化：最后，你将发送请求并在 SkyWalking UI 中观察效果。 部署 SkyWalking 请从官方 SkyWalking 网站下载 SkyWalking APM 程序，然后 可以根据快速启动脚本启动所有所需服务。\n接下来，你可以访问地址 http://localhost:8080/ 。此时，由于尚未部署任何应用程序，因此你将看不到任何数据。\n为不同 Ruby 项目集成 SkyWalking 推荐使用 Bundler 来安装和管理 skywalking 的依赖。只需在 Gemfile 中声明，然后运行 bundle install 即可完成安装。\n# Gemfile source \u0026#34;https://rubygems.org\u0026#34; gem \u0026#34;skywalking\u0026#34; 在 Rails 项目中集成 对于 Rails 项目，推荐使用以下命令自动生成配置文件：\nbundle exec rails generate skywalking:start 该命令会在 config/initializers 目录下自动生成 skywalking.rb 文件，你可以在其中配置启动参数。\n在 Sinatra 项目中集成 对于 Sinatra 项目，你需要手动在应用启动时调用 Skywalking.start。例如：\nrequire \u0026#39;sinatra\u0026#39; require \u0026#39;skywalking\u0026#39; Skywalking.start get \u0026#39;/sw\u0026#39; do \u0026#34;Hello SkyWalking!\u0026#34; end 在 Gemfile 中，将 skywalking 放在 sinatra 之后，并在初始化时使用 Bundler.require，或者在 sinatra gem 加载后 调用 require \u0026lsquo;skywalking\u0026rsquo;。注意，skywalking gem 需要位于其他 gem（如 redis、elasticsearch）之后。\n应用部署 在开始部署应用程序之前，你可以通过环境变量更改 SkyWalking 中当前应用程序的服务名称。你还可以更改其配置，例如服务器端的地址。有关详细信息，请参阅文档。\n在这里，我们将当前服务的名称更改为 sw-ruby。\n接下来，你可以启动应用程序，这里以 sinatra 作为示例：\nexport SW_AGENT_SERVICE_NAME=sw-ruby ruby sinatra.rb 在 SkyWalking UI 上可视化 现在，向应用程序发送请求并在 SkyWalking UI 中观察结果。\n几秒钟后，重新访问 http://localhost:8080 的 SkyWalking UI。能够在主页上看到部署的 demo 服务。\n此外，在追踪页面上，可以看到刚刚发送的请求。\n插件实现机制 要了解 Ruby Agent 对插件自动探针的实现机制，首先要了解 Ruby 中祖先链的概念。祖先链是一个有序的列表，在 Ruby 中，每个类或模块都有一个祖先链， 它包含了一个类或模块的所有父类以及 mixin 模块（通过 include、prepend 或 extend 混入的模块）。 Ruby 在查找方法时，会按照祖先链的顺序依次查找，直到找到目标方法或抛出 NoMethodError。\nclass User end 我们定义了一个 User 类，那么它的祖先链是如下图：\n接下来用 prepend 方法混入一个模块：\nmodule Dapper def brave \u0026#34;Hello from brave\u0026#34; end end class User prepend Dapper end p User.new.brave # =\u0026gt; \u0026#34;Hello from brave\u0026#34; prepend 会在上图 1 处进行插入，Ruby 首先在 Dapper 模块中查找 brave 方法，找到并调用，如果 Dapper 中没有 brave 方法， Ruby 会继续查找 User 类。 如果 User 类中也没有，Ruby 会继续查找 Object，依此类推。\n根据这样的机制，简单介绍下我们如何对 redis-rb 进行方法插桩，下面代码是要进行插桩的目标方法：\n# lib/redis/client.rb class Redis class Client \u0026lt; ::RedisClient def call_v(command, \u0026amp;block) super(command, \u0026amp;block) rescue ::RedisClient::Error =\u0026gt; error Client.translate_error!(error) end end end 下面是进行插桩的核心代码：\nmodule Skywalking module Plugins class Redis5 \u0026lt; PluginsManager::SWPlugin module Redis5Intercept def call_v(args, \u0026amp;block) operation = args[0] rescue \u0026#34;UNKNOWN\u0026#34; return super if operation == :auth Tracing::ContextManager.new_exit_span( operation: \u0026#34;Redis/#{operation.upcase}\u0026#34; ) do |span| # 省略对 span 的处理 super(args, \u0026amp;block) # 调用原方法 end end end def install ::Redis::Client.prepend Redis5Intercept end end end end 这里我们定义了一个 Redis5Intercept 模块，并将其作为 ::Redis::Client 的前置模块，根据 Ruby 方法查找机制， 当 Redis::Client 的 call_v 方法被调用时，Ruby 会首先会执行 Redis5Intercept 中的 call_v 方法，这里祖先链的顺序如下：\nRedis5Intercept -\u0026gt; Redis::Client -\u0026gt; ...（其他父类和模块） 同时在 Redis5Intercept 中的 call_v 方法中，super(args, \u0026amp;block) 会沿着祖先链找到下一个同名方法， 在这里也就是 Redis::Client 中的原始 call_v 方法，同时传递原始的参数和代码块。\n总结 本文讲述了 Skywalking Ruby 在 Ruby 项目中的集成方法，并简要介绍了 SkyWalking Ruby 对插件自动探针的实现机制。\n目前 Ruby 探针处于早期的开发阶段，未来我们将继续扩展 SkyWalking Ruby 的功能，添加更多插件支持。所以，请继续关注！\n","excerpt":"\u003ch2 id=\"背景\"\u003e背景\u003c/h2\u003e\n\u003cp\u003eRuby 是一种动态、面向对象的编程语言，它的语法简洁优雅，支持多种编程范式，包括面向对象、函数式和元编程。其中依靠强大的元编程能力，Ruby\n允许在运行时修改类和对象的行为。 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2025-03-06-introduction-to-skywalking-ruby/","title":"SkyWalking Ruby 快速开始与原理介绍"},{"body":"Background Apache SkyWalking is an open-source application performance monitoring (APM) system that collects various data from business applications, including metrics, logs, and distributed tracing information, and visualizes them through its UI. It also allows users to configure alerting rules by setting threshold values for specific metrics in the configuration file. When a metric associated with a particular service exceeds the predefined threshold within a given period, an alert is triggered.\nHowever, in real-world scenarios, traffic patterns and invocation behaviors vary across different time periods. For example, in a shopping system, the number of purchases is significantly lower during late-night hours compared to daytime. As a result, system metrics fluctuate within different ranges depending on the time of day. This makes it challenging to rely solely on static threshold values for accurate alerting.\nTherefore, dynamically generating thresholds for each time period based on historical data becomes crucial.\nIntroduce SkyAPM SkyPredictor Based on the above scenario, we developed the SkyAPM SkyPredictor project to fix this issue. SkyAPM SkyPredictor periodically collects data from SkyWalking and generates dynamic baselines. Meanwhile, SkyWalking queries from SkyPredictor to obtain predicted metric values for the recent period, enabling more precise and adaptive alerting.\nNOTE: SkyWalking does not have a hard dependency on the SkyPredictor service. If SkyPredictor is not configured, no predicted values would be retrieved, and not cause any failures in SkyWalking. Additionally, you can use your own AI engine to build a custom prediction system. Simply implement the required protocol as outlined in the official documentation: https://skywalking.apache.org/docs/main/next/en/setup/ai-pipeline/metrics-baseline-integration/\nArchitecture diagram As shown in the diagram, the process consists of two steps:\nData Collection \u0026amp; Prediction: The Predictor queries history metrics from SkyWalking\u0026rsquo;s OAP via its HTTP service. Then processes this data to generate dynamic predicted values for a future time period. Baseline Query \u0026amp; Alerting: The OAP periodically sends queries to the Predictor to fetch the predicted dynamic baseline. Then evaluates the current metric values with prediction result using MQE. If the deviation exceeds a certain threshold, an alert is triggered. Data Collection The Predictor utilizes the following three APIs to query data:\nStatus API: Retrieves the TTL (Time-to-Live) of history data stored in OAP, helping to determine the available time range for exporting all history metrics. Metadata API: Fetches the list of services within a specified Layer from OAP, providing insights into which services are generating data. MQE API: Iterates through the required metrics and the list of services to fetch all history metrics values for each metric associated with each service. These APIs collectively enable the Predictor to gather history metrics data, which is then used to compute dynamic baselines for future alerting.\nPrediction Once the Prediction service collects data from OAP, it proceeds with forecasting using the open-source Prophet library. The prediction process consists of the following steps:\nData Preparation: The collected metric data is split into multiple DataFrames, each corresponding to a unique combination of service + metric name. Data Sufficiency Check: If a DataFrame contains less than two days (configurable) of data, the prediction is skipped. This is to ensure accuracy, as an insufficient data volume may lead to unreliable forecasts. Forecasting: Using Prophet, the Predictor estimates the metric values for each hour over the next day (configurable). Result Storage: The generated predictions are stored in local files, enabling querying from external services. Predicted Value The Prediction service supports calculating the following two types of values:\nPredicted Value: Computes the expected metric value for the next hour based on history metrics data. Prediction Range: Determines the possible upper and lower bounds for the metric in the next hour, representing its expected fluctuation range. These values help establish a dynamic baseline, allowing the alerting system to account for natural variations while accurately detecting anomalies.\nBaseline MQE with Alarm In OAP, predicted values can be queried directly using an MQE within MQE operation. This operation enables retrieving forecasted values for a future time period.\nSince SkyWalking\u0026rsquo;s alerting system already supports query through MQE expressions, users can configure alerts directly in the alerting configuration file using MQE.\nFor more details, please refer to the official documentation.\nImpact of Data Collection on Prediction Accuracy The Predictor service supports two different data collection and prediction granularity, each with its own trade-offs in accuracy and resource consumption.\nMinute Level: Collects minutes level metrics data. More effective for metrics with high fluctuations, as it captures finer details. Consumes more resources (OAP, DB CPU and System Load resources, Predictor CPU and Memory resources). Alerts are configured based on current value comparisons. Hour Level: Collects hourly metrics data. Less resource-intensive compared to minute-level collection. Less data volume, resource, and processing cost. Alerts are configured based on predicted range values. Granularity Data Fluctuation Data Volume Current Value Prediction Accuracy Range Prediction Accuracy Best Use Case Minute Higher fluctuation Large Less accurate More accurate Ideal for highly fluctuating metrics, using range-based alerting rules Hour Lower fluctuation Small More accurate Relatively accurate Suitable for stable metrics, using current value-based predictions Choosing the appropriate granularity depends on the nature of the metric and the desired alerting method. For metrics with high volatility, minute-level collection provides better accuracy when using range-based alerts. For stable metrics, hourly aggregation is sufficient and allows for efficient predictions using current-value comparisons.\nPredict use Hourly level by default.\nOAP and Predictor Scheduling \u0026amp; Caching Both SkyWalking OAP and SkyAPM Predictor implement caching strategies to prevent excessive execution and optimize resource usage.\nBy default, Predictor runs at 00:10, 08:10, and 16:10 every day. It forecasts the next 24 hours and stores the results locally. Updating predictions every 8 hours balances resource efficiency and real-time accuracy. The 10-minute delay (instead of running at exactly 00:00, 08:00, etc.) ensures historical data is fully written to the database before querying.\nOAP queries Predictor for all required predicted metrics of a single service. The query covers a ±24-hour time range from the current moment. Results are cached for one hour to reduce redundant queries and improve efficiency.\nThese mechanisms ensure that predictions remain up-to-date, while minimizing unnecessary processing and system load.\nDemo In this section, I will demonstrate how to preview the predicted values of a metric by deploying a SkyWalking cluster along with the Predictor service in a Kubernetes cluster. This hands-on example will help you understand how to use these components effectively.\nDeploy SkyWalking Showcase SkyWalking Showcase contains a complete set of example services and can be monitored using SkyWalking. For more information, please check the official documentation.\nIn this demo, we only deploy the predictor service, SkyWalking OAP, and UI.\nexport FEATURE_FLAGS=single-node,banyandb,baseline make deploy.kubernetes Import History Data Since a newly deployed cluster does not contain history data, I have created a Python script to simulate data. This allows the Predictor service to import data and generate baseline predictions for a future period.\nBefore importing data, you must expose the 11800 port of the OAP service in your Kubernetes cluster. You can achieve this using kubectl by running the following command:\nkubectl port-forward -n skywalking-showcase service/demo-oap 11800:11800 Then, you can download and run the demo script using the following command:\n# clone and get into the demo repository git clone https://github.com/mrproliu/SkyPredictorDemo \u0026amp;\u0026amp; cd SkyPredictorDemo # installing dependencies make install # import data(7 days) python3 -m client.generate localhost:11800 7 Finally, you can see the output in the console: Metrics send success!.\nPrediction metrics Since the Predictor service runs based on a cron schedule, it does not automatically execute immediately after data import. To force it to collect data and perform a prediction, you can manually delete the Predictor pod, prompting Kubernetes to restart it:\nkubectl delete pod -n skywalking-showcase $(kubectl get pods -n skywalking-showcase --no-headers -o custom-columns=\u0026#34;:metadata.name\u0026#34; | grep \u0026#34;skywalking-predictor\u0026#34;) Once the Predictor pod restarts, you can check its logs to confirm that the prediction process has been completed.\nPredicted for e2e-test-dest-service of service_xxx to xxxx-xx-xx xx:xx:xx. View in SkyWalking UI Once the prediction process is complete, you can visualize the predicted values in the SkyWalking UI by configuring the appropriate metric widgets.\nFirst, Run the following command to forward the UI service port to your local machine:\nkubectl port-forward svc/demo-ui 8080:80 --namespace skywalking-showcase Then, you can access this page to view the service traffic that was generated using the Python script earlier: http://localhost:8080/dashboard/MESH/Service/ZTJlLXRlc3QtZGVzdC1zZXJ2aWNl.1/Mesh-Service\nTo display predicted values, edit the Service Avg Resp Time Widget and add the following MQE:\n# The maximum predicted response time. baseline(service_resp_time, upper) # The predicted response time. baseline(service_resp_time, value) # The minimum predicted response time. baseline(service_resp_time, lower) Finally, you can see the predicted values displayed in the widget.\nSince the default data collection is hourly and the metric has significant fluctuations, the predicted values are derived from hourly averages rather than minute-level granularity. This approach smooths out fluctuations and provides a more stable baseline for monitoring.\nNow, you should see the predicted response times visualized alongside actual values, helping you analyze trends and configure dynamic alerting thresholds effectively.\nConclusion SkyAPM SkyPredictor enhances alert accuracy by using dynamic baselines instead of static thresholds. It collects history metrics data, forecasts future values with Prophet, and supports minute or hour-level collection for better precision. By integrating predictions into SkyWalking UI, users can optimize alerting and improve system observability.\nBy integrating dynamic thresholds, SkyWalking can adapt to traffic patterns and detect anomalies more effectively, reducing false positives and improving system observability.\n","excerpt":"\u003ch2 id=\"background\"\u003eBackground\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"https://skywalking.apache.org/\"\u003eApache SkyWalking\u003c/a\u003e is an open-source application performance monitoring (APM) system\nthat …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2025-02-24-improving-alert-accuracy-with-dynamic-baselines/","title":"Improving Alert Accuracy with Dynamic Baselines"},{"body":"SkyWalking Satellite 1.3.0 is released. Go to downloads page to find release tars.\nFeatures Support native eBPF Access Log protocol. Update go library to 1.23. Support async profiler protocol. Bug Fixes Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Satellite 1.3.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-satellite-1-3-0/","title":"Release Apache SkyWalking Satellite 1.3.0"},{"body":"Background Apache SkyWalking 是一个开源的应用性能监控（APM）系统，可从业务应用程序中收集各种数据，包括 指标、日志和分布式追踪信息，并通过UI进行可视化展示。\n此外，SkyWalking允许用户在配置文件中为特定指标设置阈值规则，以实现告警。当某个服务的指标值在指定时间内超过预设阈值时，系统会触发告警。\n然而，在实际场景中，不同时间段的访问流量和调用模式存在较大差异。例如，在电商系统中，凌晨的购买人数远少于白天，导致系统的各项指标在不同时段内波动范围不同。因此，单纯依赖静态阈值进行告警，很难保证准确性。\n因此，基于历史数据动态生成每个时段的阈值变得至关重要。\nSkyAPM SkyPredictor 基于上述场景，我们开发了SkyAPM SkyPredictor项目来解决这一问题。 SkyPredictor 会定期从SkyWalking采集数据并生成动态基线，而SkyWalking则可查询SkyPredictor，获取最近一段时间的预测指标值，从而实现更精准、更自适应的告警机制。\n注意：SkyWalking 并不强依赖 SkyPredictor 服务。 如果未配置 SkyPredictor，SkyWalking不会查询预测值，也不会导致系统故障。 另外，你也可以使用自定义AI引擎来构建自己的预测系统，只需按照官方文档实现相应协议即可： https://skywalking.apache.org/docs/main/next/en/setup/ai-pipeline/metrics-baseline-integration/\n系统架构 如上图所示，该系统包含两个主要部分：\n数据采集与预测： SkyPredictor通过访问SkyWalking OAP中的HTTP服务来查询历史指标数据。 然后，对这些数据进行处理，生成未来一段时间的 动态预测值。 基线查询与告警： OAP会定期向SkyPredictor查询预测的动态基线。 然后使用MQE计算当前指标值与预测结果的偏差。 如果偏差超过设定的阈值，则触发告警。 数据采集 SkyPredictor通过以下三个API进行数据查询：\nStatus API: 获取OAP存储的历史数据TTL（存活时间），用于确定可用的历史数据时间范围，以便导出所有历史指标。 Metadata API: 从OAP中查询指定Layer下的所有服务列表，获取哪些服务正在生成数据。 MQE API: 遍历所需计算的指标和所有服务，获取每个服务在这些指标上的历史数据值。 这些API共同作用，使SkyPredictor能够收集历史指标数据，并计算未来告警的动态基线。\n预测 当SkyPredictor服务从OAP采集到数据后，它会使用开源Prophet库进行预测。预测流程包括以下步骤：\n数据准备: 将采集到的指标数据按服务+指标名称拆分为多个DataFrame。 数据时长检查: 如果 DataFrame 的数据量少于2天（可配置），则跳过预测，以避免数据不足导致预测不准确。 预测计算: 使用 Prophet 预测未来 24 小时（可配置）中每个小时的指标值。 存储结果: 将预测结果存储在本地文件，以供外部查询使用。 预测值与范围 SkyPredictor 服务支持计算以下两种类型的值：\n预测值: 根据历史数据计算下一个小时的预期指标值。 预测范围: 计算下一个小时可能的最大值和最小值，表示该指标的预期波动范围。 这些预测值可用于建立动态基线，使告警系统能够识别自然波动并准确检测异常。\n基线 MQE 与告警结合 在 OAP 中，可以直接使用 MQE 进行预测值查询，以获取未来一段时间的预测数据。\n由于 SkyWalking 告警系统 已支持通过 MQE 表达式进行告警验证，因此用户可以直接在告警配置文件中使用 MQE 表达式配置动态阈值告警。\n详细信息请参考官方文档。\n数据采集对预测精度的影响 Predictor 服务支持两种不同的数据采集与预测粒度，每种方式在准确性和资源消耗方面各有取舍。\n分钟级别采集: 采集分钟级别的指标数据。 适用于波动较大的指标，可捕捉更精细的细节。 资源消耗较高（OAP、数据库 CPU 负载、Predictor CPU 与内存消耗较大）。 适用于基于当前值进行告警配置。 小时级别采集: 采集小时级别的指标数据。 相较于分钟级别，消耗更少的资源。 数据量较小，计算与存储成本更低。 适用于基于预测范围进行告警配置。 采集粒度 数据波动 数据量 当前值预测准确性 预测范围准确性 适用场景 分钟级别 波动较大 大 不太准确 更准确 适用于 高波动指标，建议使用 范围告警规则 小时级别 波动较小 小 更准确 相对准确 适用于 稳定指标，建议使用 当前值告警规则 如何选择合适的采集粒度？ 高波动指标（如瞬时流量、短周期变化的指标）：建议使用分钟级别采集，并配置范围告警以提高准确性。 稳定指标（如长期稳定的延迟、均衡负载）：建议使用小时级别采集，并基于当前值进行告警，提高计算效率。\n默认情况下，Predictor 采用小时级别采集。\nOAP 与 Predictor 的调度与缓存 SkyWalking OAP 和 SkyAPM Predictor 都实现了缓存策略，以防止过度执行，优化资源使用。\nPredictor 默认每天在 00:10、08:10 和 16:10 运行一次，预测未来 24 小时的指标数据，并将预测结果存储在本地。 每 8 小时更新一次预测数据，以在资源消耗和实时准确性之间取得平衡。执行时间选择在整点 10 分钟后，而不是 00:00、08:00 等整点，目的是确保 历史数据已经完整写入数据库，避免因数据未及时同步而导致查询异常。\nOAP 会在需要时向 Predictor 发送查询请求，获取单个服务所有需要预测的指标，查询范围覆盖当前时间的前后 24 小时。为了提高查询效率，OAP 会将查询结果缓存 1 小时，防止短时间内重复查询 SkyPredictor，减少系统负载。\n这些机制保证了预测数据的实时性和准确性，同时降低了不必要的计算和资源消耗，确保 SkyWalking 在高效运行的同时能够提供准确的动态基线预测。\nDemo 在本节中，我将演示如何在 Kubernetes 集群中部署 SkyWalking 集群以及 SkyPredictor 服务，以预览某个指标的预测值。通过这个实践示例，你可以更直观地了解如何有效使用这些组件。\n部署 SkyWalking Showcase SkyWalking Showcase 提供了一整套示例服务，并可以通过 SkyWalking 进行监控。详细信息请参考官方文档。\n在本次演示中，我们仅部署 SkyPredictor 服务、SkyWalking OAP 和 UI，以便展示预测功能的应用。\nexport FEATURE_FLAGS=single-node,banyandb,baseline make deploy.kubernetes 导入历史数据 由于新部署的集群不包含历史数据，我编写了一个 Python 脚本来模拟数据。这使得 SkyPredictor 服务能够导入数据，从而可以生成未来一段时间的基线预测值。\n在导入数据之前，需要先在 Kubernetes 集群中开放 OAP 服务的 11800 端口。可以使用 kubectl 执行以下命令来完成端口转发：\nkubectl port-forward -n skywalking-showcase service/demo-oap 11800:11800 接下来，你可以使用以下命令下载并运行示例脚本：\n# clone and get into the demo repository git clone https://github.com/mrproliu/SkyPredictorDemo \u0026amp;\u0026amp; cd SkyPredictorDemo # installing dependencies make install # import data(7 days) python3 -m client.generate localhost:11800 7 最终, 你可以在命令行中看到: Metrics send success!.\n执行预测任务 由于 SkyPredictor 服务是基于 cron 调度运行的，因此在导入数据后，它不会立即执行预测。为了强制触发数据采集和预测，你可以手动删除 SkyPredictor Pod，让 Kubernetes 自动重启它，从而立即执行预测任务：\nkubectl delete pod -n skywalking-showcase $(kubectl get pods -n skywalking-showcase --no-headers -o custom-columns=\u0026#34;:metadata.name\u0026#34; | grep \u0026#34;skywalking-predictor\u0026#34;) 当 Predictor Pod 重新启动后，你可以检查其日志，以确认预测过程是否已经完成。\nPredicted for e2e-test-dest-service of service_xxx to xxxx-xx-xx xx:xx:xx. View in SkyWalking UI 当预测过程完成后，你可以在 SkyWalking UI 中配置指标组件来可视化预测值。\n首先，运行以下命令，将 UI 服务端口转发到本地：\nkubectl port-forward svc/demo-ui 8080:80 --namespace skywalking-showcase 然后，你可以访问该页面，查看之前使用 Python 脚本生成的服务流量数据： http://localhost:8080/dashboard/MESH/Service/ZTJlLXRlc3QtZGVzdC1zZXJ2aWNl.1/Mesh-Service\n要显示预测值，请编辑 Service Avg Resp Time Widget，并添加以下 MQE 表达式：\n# The maximum predicted response time. baseline(service_resp_time, upper) # The predicted response time. baseline(service_resp_time, value) # The minimum predicted response time. baseline(service_resp_time, lower) 最后，你可以在组件中看到预测值的可视化展示。\n由于默认的数据采集粒度为小时级，并且该指标存在较大的波动，因此预测值基于小时级平均数据而非分钟级数据进行计算。此方法能够平滑波动，提供更稳定的监控基线。\n现在，你应该可以在 UI 中看到预测的响应时间与实际值并排展示，这将帮助你更直观地分析趋势，并配置动态告警阈值，提升告警的准确性。\n结论 SkyAPM SkyPredictor 通过动态基线取代静态阈值，提高了告警的准确性。它采集历史指标数据，利用 Prophet 预测未来值，并支持分钟级或小时级数据采集，以提供更精确的预测。通过将预测结果集成到 SkyWalking UI，用户可以优化告警配置，并提升系统可观测性。\n借助动态阈值，SkyWalking 能够自适应流量模式，更有效地检测异常，减少误报，并提升系统的可观测能力。\n","excerpt":"\u003ch2 id=\"background\"\u003eBackground\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"https://skywalking.apache.org/\"\u003eApache SkyWalking\u003c/a\u003e 是一个开源的应用性能监控（APM）系统，可从业务应用程序中收集各种数据，包括 指标、日志和分布式追踪信息，并通过UI进行可视化展示。\u003c/p\u003e\n\u003cp\u003e此外 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2025-02-24-improving-alert-accuracy-with-dynamic-baselines/","title":"使用动态基线提高告警准确性"},{"body":"SkyWalking Java Agent 9.4.0 is released. Go to downloads page to find release tars. Changes by Version\n9.4.0 Upgrade nats plugin to support 2.16.5 Add agent self-observability. Fix intermittent ClassCircularityError by preloading ThreadLocalRandom since ByteBuddy 1.12.11 Add witness class/method for resteasy-server plugin(v3/v4/v6) Add async-profiler feature for performance analysis. This requires OAP server 10.2.0 Support db.instance tag,db.collection tag and AggregateOperation span for mongodb plugin(3.x/4.x) Improve CustomizeConfiguration by avoiding repeatedly resolve file config Add empty judgment for constructorInterceptPoint Bump up gRPC to 1.68.1 Bump up netty to 4.1.115.Final Fix the CreateAopProxyInterceptor in the Spring core-patch to prevent it from changing the implementation of the Spring AOP proxy Support Tracing for GlobalFilter and GatewayFilter in Spring Gateway [doc] Enhance Custom Trace Ignoring Plugin document about conflicts with the plugin of sampler plugin with CPU policy [doc] Add Spring Gateway Plugin document [doc] Add 4 menu items guiding users to find important notices for Spring Annotation Plugin, Custom Trace Ignoring Plugin, Kotlin Coroutine Plugin, and Spring Gateway Plugin Change context and parent entry span propagation mechanism from gRPC ThreadLocal context to SkyWalking native dynamic field as new propagation mechanism, to better support async scenarios. Add Caffeine plugin as optional. Add Undertow 2.1.7.final+ worker thread pool metrics. Support for tracking in spring gateway versions 4.1.2 and above. Fix ConsumeDriver running status concurrency issues. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking Java Agent 9.4.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nChanges by …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-java-agent-9-4-0/","title":"Release Apache SkyWalking Java Agent 9.4.0"},{"body":"SkyWalking Eyes 0.7.0 is released. Go to downloads page to find release tars.\nUpdate Apache-2.0.yaml to add BSD-2-Clause-Views by @spacewander in https://github.com/apache/skywalking-eyes/pull/183 Add instructions for installing license-eye using Homebrew on macOS by @CodePrometheus in https://github.com/apache/skywalking-eyes/pull/186 docs: update readme. by @yuluo-yx in https://github.com/apache/skywalking-eyes/pull/187 Bump up to go 1.23 and clean up Docker images by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/188 Bump github.com/go-git/go-git/v5 from 5.8.0 to 5.13.0 by @dependabot in https://github.com/apache/skywalking-eyes/pull/189 Fix global gitignore is not respected, move licenses to release phase by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/191 Remove change logs and link to releases page instead, simplify release steps by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/192 Full Changelog: https://github.com/apache/skywalking-eyes/compare/v0.6.0...v0.7.0\n","excerpt":"\u003cp\u003eSkyWalking Eyes 0.7.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eUpdate …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-eyes-0-7-0/","title":"Release Apache SkyWalking Eyes 0.7.0"},{"body":"SkyWalking Ruby 0.1.0 is released. Go to downloads page to find release tars.\nFeatures Initialize the ruby agent core. Implement e2e tests. Plugins Support Sinatra Support redis-rb Support net-http Support memcached Support elasticsearch Documentation Initialize the documentation. Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Ruby 0.1.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eInitialize …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-ruby-0.1.0/","title":"Release Apache SkyWalking Ruby 0.1.0"},{"body":"As we step into 2025, we are thrilled to share some major updates regarding Apache SkyWalking\u0026rsquo;s storage architecture. Starting from this year, H2 will no longer be supported as a storage option. This decision marks a significant milestone in our journey to enhance SkyWalking’s scalability, reliability, and adaptability for both local and cloud-native environments.\nBack in 2015, we introduced H2 as the default storage option for SkyWalking to simplify the first-time installation experience. Its in-memory mode provided an easy way for users to get started locally without additional setup. While it served its purpose for an initial learning curve, over the years, we’ve observed several challenges that have made H2 unsuitable for production and cloud-based environments.\nWith the progress of the BanyanDB sub-project, we are confident in making BanyanDB the default storage choice moving forward. Let\u0026rsquo;s dive into the reasoning behind this transition and what it means for you.\nWhy Are We Moving Away from H2? While H2 was a good choice for local installations in its early days, its limitations became increasingly evident as SkyWalking evolved:\nPerformance Limitations:\nH2\u0026rsquo;s in-memory mode struggles with high workloads, often leading to data loss after about 20 minutes. This behavior occurs without any warnings, making it unreliable for long-term use. Cloud-Native Deployment Challenges:\nH2’s design does not align with cloud-native architectures. When users migrated from local setups to Kubernetes (K8s) or similar environments, H2 caused endless rebooting of the OAP server due to state loss, creating confusion and frustration. Inconsistencies in Feature Implementations:\nH2 relies on the OAP’s JDBC implementation, which differs significantly from the implementations for Elasticsearch and BanyanDB. This inconsistency led to discrepancies in behavior, especially when users switched to production-ready storage backends. BanyanDB: The Future of SkyWalking Storage With the progress of the BanyanDB sub-project, we are excited to announce that BanyanDB 0.8 (scheduled for release in early 2025) is fully production-ready. BanyanDB is designed to address the limitations of H2 while providing a seamless experience for both local and cloud-native deployments.\nWhy BanyanDB? Performance and Reliability:\nBanyanDB can handle larger workloads without the risk of data loss, making it far more suitable for production environments. Cloud-Native First:\nBuilt with cloud-native architectures in mind, BanyanDB works seamlessly in Kubernetes and other containerized environments, ensuring stability and scalability. User-Friendly:\nDespite its advanced capabilities, BanyanDB is easy to set up locally, lowering the barrier to entry for new users. Unified Features:\nBanyanDB ensures consistent behavior with other storage backends like Elasticsearch, providing a more predictable and reliable user experience. How to Get Started with BanyanDB Setting up BanyanDB is simple and straightforward. Follow these steps to transition from H2 to BanyanDB:\nStep 1: Identify the BanyanDB Version Locate the bundled BanyanDB version in your OAP binary by checking the following file:\n/config/bydb.dependencies.properties Step 2: Set Up BanyanDB Using Docker Run the following commands to pull and run BanyanDB as a standalone container:\nexport BYDB_VERSION=xxx docker pull apache/skywalking-banyandb:${BYDB_VERSION} docker run -d \\ -p 17912:17912 \\ -p 17913:17913 \\ --name banyandb \\ apache/skywalking-banyandb:${BYDB_VERSION} \\ standalone Ports: 17912: gRPC port for communication. 17913: HTTP port for administration. Step 3: Start SkyWalking OAP and UI Once BanyanDB is running, you can start the OAP server and UI with their default settings using:\nbin/startup.sh OAP Server: gRPC APIs: 0.0.0.0:11800 HTTP APIs: 0.0.0.0:12800 UI: Port: 8080 Queries OAP APIs via 127.0.0.1:12800. What This Means for You For New Users:\nBanyanDB will be the default storage option, providing a smooth experience for both local and cloud-native setups. For Existing Users:\nIf you are still using H2, we strongly recommend migrating to a production-ready storage backend such as BanyanDB or Elasticsearch. BanyanDB is especially well-suited for modern cloud environments. Looking Ahead The removal of H2 is a step forward in making SkyWalking a truly robust, cloud-native observability platform. By focusing on BanyanDB, we aim to provide a storage solution that grows with your needs, whether you\u0026rsquo;re running SkyWalking locally or scaling it across distributed environments.\nWe’re excited about this transition and look forward to hearing your feedback as you adopt BanyanDB. If you have any questions or need assistance, feel free to reach out to the community.\nHere’s to a more scalable and reliable SkyWalking in 2025!\n","excerpt":"\u003cp\u003eAs we step into 2025, we are thrilled to share some major updates regarding Apache …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/remove-jdbc-as-storage/","title":"First Announcement of 2025: H2 Storage Option Permanently Removed"},{"body":"Since joining the Apache Software Foundation (ASF) incubator in 2017, the SkyWalking project has maintained a consistent release policy. Over the years, we have continuously uploaded all OAP Server and UI component libraries to Maven Central, starting from version 6.0.0 up to the latest 10.1.0 release. You can find these releases at Maven Central: org.apache.skywalking.\nInitially, releasing all component JARs to Maven Central was necessary due to the early architecture of SkyWalking, where OAP Server, UI, and Java Agent all shared the same monorepo. The Java Agent included a toolkit library that many users accessed via the central repository. However, in 2021, we split the Java Agent into its own repository skywalking-java GitHub repository, simplifying code management and release workflows.\nOver time, the OAP repository has grown significantly, incorporating many new modules to support advanced features. However, this has also made the release process heavier and more time-consuming. Currently, preparing a release takes over 1 hour, with 90% of the time spent uploading JARs to Maven Central. We believe this step is no longer necessary and adds little value.\nPlanned Changes As we step into 2025, the SkyWalking community will streamline the release process with the following changes:\nStarting in 2025, all JAR files of the skywalking-oap-server will no longer be published to Maven Central. We will instead provide the release artifacts through the following channels:\nSource tarballs and binary tarballs: These will be available on the official SkyWalking download page. Docker images: These will be published on Docker Hub, OAP server and OAP UI. Impact on Users This change will be transparent to the vast majority of users (approximately 99%). As highlighted in our Quick Start Guide, you can easily deploy OAP Server, UI, and Database with a single command using Docker Compose:\nLinux, macOS, Windows (WSL)\nbash \u0026lt;(curl -sSL https://skywalking.apache.org/quickstart-docker.sh) Windows (Powershell)\nInvoke-Expression ([System.Text.Encoding]::UTF8.GetString((Invoke-WebRequest -Uri https://skywalking.apache.org/quickstart-docker.ps1 -UseBasicParsing).Content)) You will be prompted to choose the storage type, and then the script will start the backend cluster with the selected storage.\nTo tear down the cluster, run the following command:\ndocker compose --project-name=skywalking-quickstart down To all of our re-distribution develoeprs, if you are using the skywalking-oap-server as a dependency in your project, you need to download the source tar from the website and publish them to your private maven repository. You could see change details here.\nLooking Ahead By removing the step of uploading OAP Server JARs to Maven Central, the SkyWalking release process will become more efficient. This will allow the community to focus more on delivering new features and improving the overall user experience. We are confident that this change will not impact the majority of users’ workflows.\nThank you for your continued support of the Apache SkyWalking project! If you have any questions or suggestions, feel free to reach out via our community contact channels.\nSkyWalking Community\nEnd of 2024\n","excerpt":"\u003cp\u003eSince joining the Apache Software Foundation (ASF) incubator in 2017, the SkyWalking project has …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/oap-new-release-policy-2025/","title":"Announcement - Changes in SkyWalking OAP’s Release Policy"},{"body":"Aapche SkyWalking PMC 和 committer团队参加了\u0026quot;开源之夏 2024\u0026quot;活动，作为导师，共获得了5个官方赞助名额。最终对学生开放如下任务\nBanyanDB支持自定义插入/更新触发器 在SkyWalking Go的toolkit中支持完整trace, log和meter APIs 在SkyWalking Java中集成JFR性能剖析功能 SWCK 支持注入 skywalking Python agent BanyanDB支持数据聚合 经过3个月的开发，上游评审，PMC成员评议，PMC Chair复议，OSPP官方委员会评审多个步骤，现公布项目参与人员与最终结果\n通过评审项目（共3个） 在SkyWalking Java中集成JFR性能剖析功能 学生：郑子熠 学校：南京邮电大学 本科 官方文档Profiling - Async Profiler详细介绍了此功能。 此功能将作为SkyWalking 10.2的主要新功能之一发布，预计发布时间 2025年2月（以官方Release为准）。 官网发布了blog - 使用 SkyWalking中的 async-profiler 对 Java 应用进行性能分析 介绍此功能 2024年12月9日，郑子熠因此项目在结项优秀学生评比中，获得突出贡献奖。\n在SkyWalking Go的toolkit中支持完整trace, log和meter APIs 学生：李天源 学校：广东东软学院 本科 相关Pull Requests api: add log,metric,span feat to api feat: toolkit span add event impl feat: add toolkit logging impl feat: support manual reporting of metrics in toolkit docs: add toolkit docs 相关官方文档包括 Tracing APIs Logging APIs Metric APIs 此功能将包含在SkyWalking Go Agent 0.6 release中发布，以及发布时间2025年年初（以官方Release为准）。 BanyanDB 支持自定义插入/更新触发器 学生：谢李奥 学校：中科院软件所 硕士 相关官方文档包括 [ospp] Adds MeasureAggregateFunctionService.Support API [ospp] Supports measure aggregate function avg and min, and test cases. [ospp] Implements MeasureAggregateFunctionService.Support API Dev measure aggregate function 2024年11月，在社区关于BanyanDB未来路线图的讨论决议中，服务端相关聚合功能暂时不作为BanyanDB规划功能，[Feature] Hand over downsampling(hour/day) metric processes to BanyanDB 已被关闭。此项目合并代码，不会发布，将在0.8 release中被移除。注，此代码移除与学生代码质量和完成度无关，系项目目标变更导致。 We have an agreement, this feature benefit is too limited. # Summary of downsides for server-side aggregation 1. Removing L2 would impact the alerting. Because minute dimension eventual(aggregated) data will be lost from OAP. 2. Keeping minute dimension aggregation makes the delta data lost, and causes another extra round delta data in minute dimension flushing. 3. Clearly, this server-side aggregation increases the server-side payload(CPU cost), but wouldn\u0026#39;t tradeoff for IOPS. 4. The process pipeline would be more complex. ____ The only positive part is the cache of hour/day dimension metrics could be removed. 未通过评审/未启动项目（2个） 下列项目因为质量无法达到社区要求，无学生报名等原因，将被标定为失败。\nSWCK 支持注入 skywalking Python agent。 无学生申报 BanyanDB支持数据聚合。未通过 结语 2024年，由于开源之夏提供的支持名额降低到5个，SkyWalking选题的难度都有显著提升。我们很高兴的看到，依然有3位学生较好的完成了相关课题。 感觉开源之夏和各位同学们对Apache SkyWalking的支持和热情参与。\n","excerpt":"\u003cp\u003eAapche SkyWalking PMC 和 committer团队参加了\u0026quot;开源之夏 2024\u0026quot;活动，作为导师，共获得了5个官方赞助名额。最终对学生开放如下任务 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2024-12-10-ospp-summary/","title":"开源之夏 2024 SkyWalking 社区项目情况公示"},{"body":"Background Apache SkyWalking is an open-source Application Performance Management system that helps users gather logs, traces, metrics, and events from various platforms and display them on the UI. In version 10.1.0, Apache SkyWalking can perform CPU analysis through eBPF, which supports multiple languages, but not Java. This article discusses how Apache SkyWalking 10.2.0 uses async-profiler to collect CPU, memory allocation, and locks for analysis, solving this limitation, and also provides memory allocation and occupancy analysis.\nWhy use async-profiler The async-profiler is a low overhead sampling profiler for Java that does not suffer from the Safepoint bias problem. It features HotSpot-specific API to collect stack traces and to track memory allocations. The profiler works with OpenJDK and other Java runtimes based on the HotSpot JVM. The async-profiler also officially supports the instruction set architectures commonly used on Linux and Mac platforms, and the sampling data can be stored in the JFR format. Compared with the JFR tool officially provided by JDK, it supports lower JDK versions (JDK 6).\nArchitecture diagram The processes of running a profiling task A user submits a async-profiler task in the UI The Java agent retrieves the task from the OAP Server Java agent excuses the task to collect profiling data sampling through async-profiler After the profiling is completed, the agent uploads the JFR file to the OAP server. The server parses the JFR file to generate profiling results and marks the task as completed status. The user could check the performance analysis result from the UI side. Demo You can setup SkyWalking showcase locally to preview this feature. In this demo, we only deploy service, the latest released SkyWalking OAP, and UI.\nexport FEATURE_FLAGS=java-agent-injector,single-node,elasticsearch make deploy.kubernetes After deployment is complete, please run the following script to open SkyWalking UI: http://localhost:8080/.\nkubectl port-forward svc/ui 8080:8080 --namespace default Run the Async Profiling Task Step by Step After the deployment is complete, users can navigate to the service page where the Java agent is configured. Upon entering the service page, users will be able to see the Async Profiling component. By clicking on this component, users will gain access to the relevant functionality page, where they can perform some operations.\nCreate a New Task Clicking New Task on the Async Profiling page will direct you to the following configuration page. The usage of each parameter is explained as follows:\nInstance: This parameter allows you to select the instance of the service that will execute the profiling. It supports selecting multiple instances simultaneously for performance analysis. Duration: Specifies the duration for the task. The default duration is conservatively set to a maximum of 20 minutes, but this can be adjusted through the Java agent configuration. Async Profiling Events: The profiling events are categorized into three types of sampling, which will be explained below: CPU Sampling: CPU, WALL, CTIMER, ITIMER. See the differences between these four CPU sampling types. Memory Allocation Sampling: ALLOC. Lock Occupancy Sampling: LOCK. ExecArgs: Extended parameters for async-profiler. Detailed usage instructions are available. Check the Progresses Of the Task By clicking the task details icon, users can view the task status logs, relevant parameters, as well as instances where data collection has either failed or been successfully completed. Instances that have successfully completed data collection will be available for subsequent performance analysis.\nIt is important to note that, in containerized deployments where users have not configured volume mounts, there may be cases where JFR files cannot be received. To address this, the OAP Server by default uses memory to receive and parse JFR files. The maximum acceptable size for JFR files is conservatively set to 30MB by default.\nUsers can customize the default JFR file size in the OAP configuration and opt to store the files on the filesystem before parsing them, enabling the platform to handle larger JFR files and ensuring smoother memory allocation.\nCurrently, the JFR parser requires approximately 1GB of memory to process a 200MB JFR file. (Note that this refers only to memory allocation, not the actual memory required for parsing.) Users can use this as a reference when configuring their OAP Server\nPerformance Analysis Users can select a task and choose the instances they wish to analyze for performance (multiple instances can be selected for aggregated flame graph analysis). After selecting the desired JFR event type for analysis, users can click the Analyze button to display the corresponding flame graph.\nSome Details Differences in CPU sampling during task creation The CPU sampling mechanism supports several modes, each representing a different sampling engine implemented by async-profiler. These modes include CPU, WALL, CTIMER, and ITIMER, and differ primarily in how they collect and generate sampling signals. The following provides a detailed description of each sampling:\nCPU: cpu mode relies on perf_events. The idea is the same - to generate a signal every N nanoseconds of CPU time, which in this case is achieved by configuring PMU to generate an interrupt every K CPU cycles. WALL: Same as CPU sampling, but also samples threads in non-runnable state, such as threads in sleep ITIMER: itimer mode is based on setitimer(ITIMER_PROF) syscall, which ideally generates a signal every given interval of the CPU time consumed by the process. CTIMER: ctimer aims to address these limitations of perf_events and itimer. ctimer relies on timer_create. It combines benefits of cpu and itimer, except that it does not allow collecting kernel stacks. For details, please refer to async-profiler\nExecArgs in task creation By default, task parameters are separated by commas. When creating a task, users should refer to the following example format for input: lock=10us,interval=10ms.\nCurrently, the following parameters are supported by default:\nOption Description chunksize=N approximate size of JFR chunk in bytes (default: 100 MB) chunktime=N duration of JFR chunk in seconds (default: 1 hour) lock[=DURATION] profile contended locks overflowing the DURATION ns bucket jstackdepth=N maximum Java stack depth (default: 2048) interval=N sampling interval in ns (default: 10'000'000, i.e. 10 ms) alloc[=BYTES] profile allocations with BYTES interval For other parameters, please refer to async-profiler and need to be tested by yourself\nComparison table between sampling types and JFR events in task analysis Task sample type JFR event type Description Unit CPU\nWALL\nITIMER\nCTIMER EXECUTION_SAMPLE Multiple AsyncProfilerEventType types correspond to the EXECUTION_SAMPLE event. This is primarily due to the fact that different sampling types employ distinct underlying mechanisms and have varying sampling scopes. Sample times. The execution time can be calculated based on the sampling interval. For instance, if the number of samples is 10 and the interval is set to 10ms, the total execution time can be estimated as 100ms (the default interval is 10ms) LOCK THREAD_PARK\nJAVA_MONITOR_ENTER Empty ns ALLOC OBJECT_ALLOCATION_IN_NEW_TLAB\nOBJECT_ALLOCATION_OUTSIDE_TLAB Empty byte Add live option to extended parameters PROFILER_LIVE_OBJECT Because it is not in the event parameter of async-profiler, it is not selected separately in the task sampling type of the UI during implementation, but is used as an extended parameter byte Performance expenses There is no performance overhead when an instance is not receiving an async-profiler task. Performance impact is only introduced once the async-profiler performance analysis is initiated. The extent of this overhead depends on the specific configuration parameters. When using the default settings, the performance impact typically ranges from 0.3% to 10%. For more detailed information, please refer to the issue.\n","excerpt":"\u003ch2 id=\"background\"\u003eBackground\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"https://skywalking.apache.org/\"\u003eApache SkyWalking\u003c/a\u003e is an open-source Application Performance Management system that helps …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2024-12-09-skywalking-async-profiler/","title":"Profiling Java application with SkyWalking bundled async-profiler"},{"body":"背景 Apache SkyWalking 是一个开源的应用性能管理系统，帮助用户从各种平台收集日志、跟踪、指标和事件，并在用户界面上展示它们。在10.1.0版本中，Apache SkyWalking 可以通过 eBPF 进行 CPU 分析，eBPF 支持多种语言，但并不支持 Java。本文探讨了Apache SkyWalking 10.2.0版本如何采用 async-profiler 来收集 CPU、内存分配、锁并进行分析，解决了这一限制，同时额外提供了内存分配以及占用分析。\n为什么使用 async-profiler？ async-profiler 是一个用于 Java 的低开销采样分析器，它不会受到安全点偏差问题的影响。它基于 HotSpot 特定的 API来收集堆栈并跟踪内存分配。该分析器可与 OpenJDK 和其他基于 HotSpot JVM 的 Java 运行时一起使用。async-profiler 同时支持官方支持 Linux、mac 平台常用的指令集架构，并且采样数据支持使用 JFR 格式存储，相比于 JDK 官方提供提供的 JFR 工具支持更低的 JDK 版本（JDK 6）。\n一次任务的流程 用户在 UI 中下发 async-profiler 任务 Java agent 从 OAP Server 获取任务 Java agent 执行任务，通过 async-profiler 进行数据采样，将采样的数据写入 JFR 文件中 采样指定时间后，Java agent 上传 JFR 文件至 OAP Server OAP Server 对 JFR 文件进行解析，并且记录相关实例已经完成 用户通过UI选择完成任务的实例进行性能分析 演示 您可以在本地部署 SkyWalking Showcase 来预览此功能。在此演示中，我们仅部署服务、最新发布的 SkyWalking OAP 和 UI。\nexport FEATURE_FLAGS=java-agent-injector,single-node,elasticsearch make deploy.kubernetes 部署完成后，请运行以下脚本以打开 SkyWalking UI：http://localhost:8080/ 。\nkubectl port-forward svc/ui 8080:8080 --namespace default 使用流程 部署完成后，用户可以点击进入配置了 Java agent 的 Service 页面。进入该服务页面后，用户将能够看到 Async Profiling 组件，点击该组件即可访问相关功能页面并进行操作。\n任务下发 在 Async Profiling 页面选择新建任务将会显示如下页面，下面是参数的使用说明：\n实例：可执行性能剖析的实例，支持选择多个实例同时进行分析。 持续时间：任务的执行时长（默认设置为最多 20 分钟，参数较为保守，可通过 Java agent 中的 agent.config 进行配置调整）。 分析事件：分析事件可以大致分为三种类型采样： CPU采样：包含 CPU、WALL、CTIMER、ITIMER。有关四种 CPU 采样类型的区别可以参考下文 内存分配采样：ALLOC 锁占用采样：LOCK 任务扩展参数: async-profiler 的扩展参数，具体使用说明请参考下文 任务进度展示 点击任务详情图标后，用户可以查看任务的状态日志、相关参数以及已失败、成功完成数据采集的实例。成功完成采集的实例将可用于后续的性能分析。\n值得注意的是，考虑到在容器部署中用户并未设置卷挂载时，可能会存在无法接收 JFR 文件的情况，因此 OAP 默认使用内存接收 JFR 并且解析，并且设置的可接受 JFR 文件大小比较保守（默认为30MB）。\n用户可以自行在 OAP 中设置 JFR 默认大小以及先存储到文件系统再解析，以接收更大的 JFR 文件和更平滑的内存分配。\n目前的 JFR 解析器在解析200MB的JFR文件大概会带来1GB左右的内存分配（注意只是内存分配，而不是需要1GB内存才能解析），用户可以根据这个作为参考。\n性能分析 用户可以点击任务，选择需要进行性能分析的实例（支持选择多实例，汇总生成火焰图分析结果）。然后选择分析的 JFR 事件类型，点击 分析 按钮即可生成并显示相应的火焰图\n一些细节 任务创建中不同CPU采样的区别 CPU采样有以下几种: CPU、WALL、CTIMER、ITIMER，本质为 async-profiler 实现的采样引擎不同，下面详细介绍不同采样的差别：\nCPU: 基于 perf_events。每 N 纳秒的 CPU 时间生成一个信号，在这种情况下，通过配置 PMU 每 K CPU 周期生成一个中断来实现 WALL: 与 CPU采样 相同，但同时会采集非 runnable 状态的线程，例如会采集正在 sleep 的线程 ITIMER: 基于 setitimer 系统调用，理想情况下会在进程消耗的 CPU 时间的每个给定间隔生成一个信号。 CTIMER: 基于 timer_create 系统调用. 它结合了 CPU和 ITIMER 的优点，但它不允许收集内核堆栈 详情可以参考 async-profiler 官方文档\n任务创建中的扩展参数 默认情况下，任务参数使用逗号分隔。在创建任务时，用户可以参考以下示例格式进行填写：lock=10us,interval=10ms。\n目前官方默认支持以下参数：\n选项 含义 chunksize=N JFR分chunk的大小(默认: 100 MB) chunktime=N JFR分chunk的时间(默认: 1 hour) lock[=DURATION] 在锁分析模式下，当总锁持续时间溢出阈值时，对争用锁进行采样 (默认: 10us) jstackdepth=N 采样时采集java最大栈深度(默认: 2048) interval=N CPU采样间隔 单位ns (默认: 10'000'000, 即10 ms) alloc[=BYTES] 内存分配采样间隔，以字节单位 其余参数可以参考 async-profiler 自行实验测试\n任务分析中采样类型与 JFR 事件对照表 任务采样类型 JFR事件 备注 单位 CPU、WALL、CTIMER、ITIMER EXECUTION_SAMPLE 多种 AsyncProfilerEventType 类型都对应于 EXECUTION_SAMPLE 事件，主要原因在于不同类型的采样类型采用了不同的原理，并且采样的范围有所不同。 采样次数\n执行时间可以通过interval计算，例如采样次数为10次，interval为10ms，则可以认为执行了100ms（默认interval为10ms） LOCK THREAD_PARK、JAVA_MONITOR_ENTER 无 ns ALLOC OBJECT_ALLOCATION_IN_NEW_TLAB、OBJECT_ALLOCATION_OUTSIDE_TLAB 无 byte 扩展参数中添加live选项 PROFILER_LIVE_OBJECT 因为不在 async-profiler 的 event 参数里面，所以实现时没有单独拿出来在 UI 的任务采样类型中选择，而是作为扩展参数使用 byte 性能开销 在实例未接收到 async-profiler 任务时，不会产生性能开销；仅在启动 async-profiler 性能分析后，才会引入相应的性能损耗。 性能损耗的具体程度会根据配置的参数有所不同。使用默认参数时，性能损耗大约在 0.3% 到 10% 之间。更多详细信息可参考 issue。\n","excerpt":"\u003ch2 id=\"背景\"\u003e背景\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"https://skywalking.apache.org/\"\u003eApache SkyWalking\u003c/a\u003e 是一个开源的应用性能管理系统，帮助用户从各种平台收集日志、跟踪、指标和事件，并在用户界面上展示它们。在10.1.0版本中，Apache SkyWalking …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2024-12-09-skywalking-async-profiler/","title":"使用 SkyWalking中的 async-profiler 对 Java 应用进行性能剖析"},{"body":"SkyWalking PHP 0.8.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed Fix swoole server on request param. by @jmjoy in https://github.com/apache/skywalking-php/pull/100 Update NOTICE by @jmjoy in https://github.com/apache/skywalking-php/pull/103 Bump phpseclib/phpseclib from 3.0.19 to 3.0.35 in /tests/php by @dependabot in https://github.com/apache/skywalking-php/pull/104 Inject skywalking context. by @jmjoy in https://github.com/apache/skywalking-php/pull/107 Defined instance_name by @Almot77 in https://github.com/apache/skywalking-php/pull/111 Add TLS by @heyanlong in https://github.com/apache/skywalking-php/pull/112 Add feature sasl for rdkafka by @jmjoy in https://github.com/apache/skywalking-php/pull/116 Refactor worker to standalone crate by @jmjoy in https://github.com/apache/skywalking-php/pull/118 Add standalone reporter type and standalone skywalking worker by @jmjoy in https://github.com/apache/skywalking-php/pull/119 Adapt to Swoole\\Coroutine\\Http\\Server by @jmjoy in https://github.com/apache/skywalking-php/pull/120 Adapt to Swoole\\Http\\Server by @jmjoy in https://github.com/apache/skywalking-php/pull/121 Update document by @jmjoy in https://github.com/apache/skywalking-php/pull/122 Release SkyWalking PHP 0.8.0 by @jmjoy in https://github.com/apache/skywalking-php/pull/123 New Contributors @Almot77 made their first contribution in https://github.com/apache/skywalking-php/pull/111 Full Changelog: https://github.com/apache/skywalking-php/compare/v0.7.0...v0.8.0\nPECL https://pecl.php.net/package/skywalking_agent/0.8.0\n","excerpt":"\u003cp\u003eSkyWalking PHP 0.8.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-php-0-8-0/","title":"Release Apache SkyWalking PHP 0.8.0"},{"body":"SkyWalking BanyanDB 0.7.1 is released. Go to downloads page to find release tars.\nFeatures Add the bydbctl analyze series command to analyze the series data. Index: Remove sortable field from the stored field. If a field is sortable only, it won\u0026rsquo;t be stored. Index: Support InsertIfAbsent functionality which ensures documents are only inserted if their docIDs are not already present in the current index. There is a exception for the documents with extra index fields more than the entity\u0026rsquo;s index fields. Bug Fixes Fix the bug that TopN processing item leak. The item can not be updated but as a new item. Resolve data race in Stats methods of the inverted index. Documentation Improve the description of the memory in observability doc. Update kubernetes install document to align the banyandb helm v0.3.0. Chores Fix metrics system typo. ","excerpt":"\u003cp\u003eSkyWalking BanyanDB 0.7.1 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"features\"\u003eFeatures\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eAdd the …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-0-7-1/","title":"Release Apache SkyWalking BanyanDB 0.7.1"},{"body":"SkyWalking CLI 0.14.0 is released. Go to downloads page to find release tars.\nFeatures Add the sub-command dependency global for adapt the global dependency query API by @mrproliu in https://github.com/apache/skywalking-cli/pull/198 Upgrade crypto lib to fix cve by @mrproliu in https://github.com/apache/skywalking-cli/pull/199 Add the hierarchy related commands hierarchy service, hierarchy instance and hierarchy layer-levels by @mrproliu in https://github.com/apache/skywalking-cli/pull/200 Add the layers field to nodes in the dependency service command by @mrproliu in https://github.com/apache/skywalking-cli/pull/200 Add the duration related flags in the endpoint list command by @mrproliu in https://github.com/apache/skywalking-cli/pull/201 ","excerpt":"\u003cp\u003eSkyWalking CLI 0.14.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"features\"\u003eFeatures\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eAdd the …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cli-0-14-0/","title":"Release Apache SkyWalking CLI 0.14.0"},{"body":"SkyWalking Kubernetes Helm Chart 4.7.0 is released. Go to downloads page to find release tars.\nImprove the guide on setting up SkyWalking with BanyanDB Fix incorrect link syntax by Adding secret mount in the OAP Add auth password for user postgres Fix rbac issues Bump up SkyWalking Infra E2E version and set kind version to v1.21.14 Bump up BanyanDB Helm version to 0.3.0 Bump up OAP to 10.1.0 ","excerpt":"\u003cp\u003eSkyWalking Kubernetes Helm Chart 4.7.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars. …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-kubernetes-helm-chart-4.7.0/","title":"Release Apache SkyWalking Kubernetes Helm Chart 4.7.0"},{"body":"SkyWalking Rover 0.7.0 is released. Go to downloads page to find release tars.\nFeatures Upgrade LLVM to 18. Support propagation the excluding namespaces in the access log to the backend. Add pprof module for observe self. Add detect process from CRI-O container in Kubernetes. Introduce MonitorFilter into access log module. Support monitoring ztunnel to adapt istio ambient mode. Enhance get connection address strategy in access log module. Reduce file mount needs when deploy in the Kubernetes, split env name ROVER_HOST_MAPPING to ROVER_HOST_PROC_MAPPING and ROVER_HOST_ETC_MAPPING. Bug Fixes Fixed the issue where conntrack could not find the Reply IP in the access log module. Fix errors when compiling C source files into eBPF bytecode on a system with Linux headers version 6.2 or higher. Fixed ip_list_rcv probe is not exist in older linux kernel. Fix concurrent map operation in the access log module. Fix the profiling cannot found process issue. Fix cannot translate peer address in some UDP scenarios. Fix the protocol logs may be missing if the process is short-lived. Fix some connections not called close syscall, causing unnecessary memory usage. Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Rover 0.7.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eUpgrade …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-rover-0-7-0/","title":"Release Apache SkyWalking Rover 0.7.0"},{"body":"引言 Apache SkyWalking Go是一款针对 Golang 应用程序提供可观测的工具, 旨在为单体服务、微服务、云原生架构和容器化应用设计。 它是 Apache SkyWalking 探针项目的 Go 语言实现，提供了全面的服务追踪、性能指标分析、应用拓扑分析等功能。\nSkyWalking Go利用Go语言的并发特性，实现了高效的数据采集和分析。它通过编译期间使用AST在代码中插入少量的探针代码，可以捕获到服务的请求和响应数据，以及系统的运行状态信息。 SkyWalking Go通过上报这些收集的数据，能够生成详细的服务调用链路图，帮助开发人员了解服务之间的依赖关系，以及每个服务的性能状况。\nSkyWalking Go 当前提供了以下三种能力让用户手动上报相关信息\nTrace Metrics Log 本文旨在指导用户如何使用 toolkit 手动上报 Log日志 以及 Metrics指标。有关如何操作 toolkit Trace 上报链路信息可看 SkyWalking Go Toolkit Trace 详解。 在深入了解之前，您可以参考 SkyWalking Go Agent快速开始指南 来学习如何使用SkyWalking Go Agent。\n下面将会介绍如何在特定场景中使用这些接口。\n导入 Trace Toolkit 首先在项目的根目录中执行以下命令：\ngo get github.com/apache/skywalking-go/toolkit 手动上报 Log 日志 在链路追踪中，日志扮演着至关重要的角色。它们记录系统中每个请求的详细信息，包括时间戳、处理节点、错误信息等，从而帮助开发人员和运维团队快速定位性能瓶颈和故障根源。 通过对比不同请求的日志，团队可以分析请求的流转过程，优化系统架构，提升服务响应速度和稳定性。\n在 SkyWalking Go toolkit 中，手动上报的日志将会附加在当前上下文中的 Span 上，这使得我们可以针对特定的 Span 关联特定的日志信息。\n首先我们需要导入 toolkit log 包:\n\u0026#34;github.com/apache/skywalking-go/toolkit/logging\u0026#34; 我们可以构建一个简单的 Web 服务: 根据请求参数中的用户名来判断是否合法。 当 userName 参数是非法时，我们通过 logging.Error API 记录一条错误日志。该日志将会附加到当前上下文活跃的 Span 上。\n在记录日志时，我们还可以通过可变参数追加 keyValues 到日志信息上, 让日志信息更具有描述力。\n详细的使用文档可看 SkyWalking Go toolkit-logging。\npackage main import ( \u0026#34;log\u0026#34; \u0026#34;net/http\u0026#34; _ \u0026#34;github.com/apache/skywalking-go\u0026#34; \u0026#34;github.com/apache/skywalking-go/toolkit/logging\u0026#34; ) func main() { http.HandleFunc(\u0026#34;/user\u0026#34;, func(w http.ResponseWriter, r *http.Request) { userName := r.URL.Query().Get(\u0026#34;userName\u0026#34;) if len(userName) == 0 || userName != \u0026#34;root\u0026#34; { // 记录一条日志信息, 这条日志信息将会附加到当前上下文活跃的 Span 上 // 我们可以通过可变参数追加日志Tag logging.Error(\u0026#34;拒绝非法用户登陆\u0026#34;, \u0026#34;userName\u0026#34;, userName) w.WriteHeader(http.StatusUnauthorized) return } w.WriteHeader(http.StatusAccepted) }) if err := http.ListenAndServe(\u0026#34;:8080\u0026#34;, nil); err != nil { log.Fatalln(\u0026#34;server running by err:\u0026#34;, err) } } 然后我们使用 SkyWalking Go Agent 对其进行增强:\ngo build -toolexec=\u0026#34;/path/go-agent\u0026#34; -a -o demo . 手动上报 Metrics 指标信息 Metrics在链路追踪中极为重要，它们提供了系统性能的定量分析。 通过监控请求的延迟、吞吐量和错误率等指标，团队能够识别性能瓶颈和潜在问题，从而优化系统架构和资源分配。 结合链路追踪，metrics能够揭示请求在各个服务间的流转情况，帮助团队深入了解系统的健康状态和使用模式，确保服务的高可用性和用户体验，最终实现业务目标的有效支持。\n当前 toolkit metrics 支持以下指标类型\nCounter Gauge Histogram 首先我们需要导入 toolkit metric 包:\n\u0026#34;github.com/apache/skywalking-go/toolkit/metric\u0026#34; 我们可以构建一个简单的 Echo 服务, 创建一个 Counter 类型的指标来记录请求次数。 同时使用 metric.WithLabels 为指标添加额外的标签。\npackage main import ( \u0026#34;log\u0026#34; \u0026#34;net/http\u0026#34; _ \u0026#34;github.com/apache/skywalking-go\u0026#34; \u0026#34;github.com/apache/skywalking-go/toolkit/metric\u0026#34; ) func main() { // 构建一个 Counter 类型 metric // 同时我们为该 Counter 设置 labels counter := metric.NewCounter( \u0026#34;http_request_count\u0026#34;, metric.WithLabels(\u0026#34;path\u0026#34;, \u0026#34;/ping\u0026#34;), ) http.HandleFunc(\u0026#34;/ping\u0026#34;, func(w http.ResponseWriter, r *http.Request) { // 每次请求来都计数加一\tcounter.Inc(1) w.WriteHeader(http.StatusOK) }) if err := http.ListenAndServe(\u0026#34;:8080\u0026#34;, nil); err != nil { log.Fatalln(\u0026#34;server running by err:\u0026#34;, err) } } 然后我们使用 SkyWalking Go Agent 对其进行增强:\ngo build -toolexec=\u0026#34;/path/go-agent\u0026#34; -a -o demo . 我们可以在 SkyWalking 自定义仪表盘 中展示指标信息。\n总结 本文讲述了 Skywalking Go 的 Log APIs 和 Metrics APIs 的简单使用。 它为用户提供了自定义上报日志信息和指标信息的功能。\nSkyWalking Go toolkit 设计之初就秉承简单易用的思想，旨在缩短用户和产品之间的距离。\n更多的信息可看 SkyWalking Go。\n","excerpt":"\u003ch2 id=\"引言\"\u003e引言\u003c/h2\u003e\n\u003cp\u003eApache SkyWalking Go是一款针对 Golang 应用程序提供可观测的工具, 旨在为单体服务、微服务、云原生架构和容器化应用设计。\n它是 Apache SkyWalking 探针 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2024-10-20-skywalking-go-toolkit-metrics-log/","title":"SkyWalking Go Toolkit Log 与 Metrics 的使用"},{"body":"Intro Apache SkyWalking Go is an observability tool specifically designed for Golang applications, aimed at monolithic services, microservices, cloud-native architectures, and containerized applications. It is the Go language implementation of the Apache SkyWalking probe project, providing comprehensive capabilities for service tracing, performance metrics analysis, and application topology analysis.\nSkyWalking Go leverages Go\u0026rsquo;s concurrency features to achieve efficient data collection and analysis. By inserting a minimal amount of probe code into the application during compilation using AST (Abstract Syntax Tree), it can capture service request and response data, as well as system runtime status information. By reporting this collected data, SkyWalking Go can generate detailed service call chain diagrams, helping developers understand the dependencies between services and the performance status of each service.\nSkyWalking Go currently provides the following three capabilities for users to manually report relevant information:\nTrace Metrics Log This document aims to guide users on how to manually report log entries and metrics using the toolkit. For information on how to report trace linkage information with the toolkit, please refer to the SkyWalking Go Toolkit Trace Detailed Explanation.\nBefore diving deeper, you may want to check the SkyWalking Go Agent Quick Start Guide to learn how to use the SkyWalking Go Agent.\nThe following sections will introduce how to use these interfaces in specific scenarios.\nImport Trace Toolkit First, execute the following command in the root directory of the project:\ngo get github.com/apache/skywalking-go/toolkit Manually report logs In traceability, logs play a crucial role. They record detailed information about each request in the system, including timestamps, processing nodes, error messages, etc., which helps developers and operations teams quickly identify performance bottlenecks and the root causes of failures. By comparing logs from different requests, teams can analyze the flow of requests, optimize system architecture, and improve service response speed and stability.\nIn the SkyWalking Go toolkit, manually reported logs will be attached to the current context Span, allowing us to associate specific log information with particular spans.\nFirst, we need to import the toolkit log package:\n\u0026#34;github.com/apache/skywalking-go/toolkit/logging\u0026#34; We can build a simple web service that determines the validity of a username based on the request parameters. When the userName parameter is invalid, we log an error using the logging.Error API. This log will be attached to the currently active Span in the context.\nWhen recording logs, we can also append keyValues to the log information using variadic parameters, making the log entries more descriptive.\nFor detailed usage documentation, please refer to SkyWalking Go toolkit-logging.\npackage main import ( \u0026#34;log\u0026#34; \u0026#34;net/http\u0026#34; _ \u0026#34;github.com/apache/skywalking-go\u0026#34; \u0026#34;github.com/apache/skywalking-go/toolkit/logging\u0026#34; ) func main() { http.HandleFunc(\u0026#34;/user\u0026#34;, func(w http.ResponseWriter, r *http.Request) { userName := r.URL.Query().Get(\u0026#34;userName\u0026#34;) if len(userName) == 0 || userName != \u0026#34;root\u0026#34; { // Log an entry, which will be attached to the currently active Span in the context. // We can append log tags using variadic parameters. logging.Error(\u0026#34;deny user login\u0026#34;, \u0026#34;userName\u0026#34;, userName) w.WriteHeader(http.StatusUnauthorized) return } w.WriteHeader(http.StatusAccepted) }) if err := http.ListenAndServe(\u0026#34;:8080\u0026#34;, nil); err != nil { log.Fatalln(\u0026#34;server running by err:\u0026#34;, err) } } Then we enhance it using the SkyWalking Go Agent:\ngo build -toolexec=\u0026#34;/path/go-agent\u0026#34; -a -o demo . Manually reporting metrics information Metrics are extremely important in traceability, as they provide quantitative analysis of system performance. By monitoring metrics such as request latency, throughput, and error rates, teams can identify performance bottlenecks and potential issues, allowing them to optimize system architecture and resource allocation. Combined with traceability, metrics can reveal the flow of requests between services, helping teams gain deeper insights into the health status and usage patterns of the system, ensuring high availability of services and a positive user experience, ultimately supporting business objectives effectively.\nThe current toolkit metrics support the following types of metrics:\nCounter Gauge Histogram First, execute the following command in the root directory of the project:\n\u0026#34;github.com/apache/skywalking-go/toolkit/metric\u0026#34; We can build a simple Echo service that creates a Counter type metric to record the number of requests. At the same time, we can use metric.WithLabels to add additional labels to the metric.\npackage main import ( \u0026#34;log\u0026#34; \u0026#34;net/http\u0026#34; _ \u0026#34;github.com/apache/skywalking-go\u0026#34; \u0026#34;github.com/apache/skywalking-go/toolkit/metric\u0026#34; ) func main() { // Create a Counter type metric // We also set labels for this Counter counter := metric.NewCounter( \u0026#34;http_request_count\u0026#34;, metric.WithLabels(\u0026#34;path\u0026#34;, \u0026#34;/ping\u0026#34;), ) http.HandleFunc(\u0026#34;/ping\u0026#34;, func(w http.ResponseWriter, r *http.Request) { // Increment the count by one for each incoming request counter.Inc(1) w.WriteHeader(http.StatusOK) }) if err := http.ListenAndServe(\u0026#34;:8080\u0026#34;, nil); err != nil { log.Fatalln(\u0026#34;server running by err:\u0026#34;, err) } } Then we enhance it using the SkyWalking Go Agent:\ngo build -toolexec=\u0026#34;/path/go-agent\u0026#34; -a -o demo . We can display the metric information in the SkyWalking Custom Dashboard.\nSummarize This document discusses the basic usage of the Log APIs and Metrics APIs in SkyWalking Go. It provides users with the functionality to customize and report log entries and metric information.\nThe SkyWalking Go toolkit was designed with simplicity in mind, aiming to shorten the distance between users and the product.\nFor more information, please refer to SkyWalking Go.\n","excerpt":"\u003ch2 id=\"intro\"\u003eIntro\u003c/h2\u003e\n\u003cp\u003eApache SkyWalking Go is an observability tool specifically designed for Golang applications, …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2024-10-20-skywalking-go-toolkit-metrics-log/","title":"Use of SkyWalking Go Toolkit Log and Metrics"},{"body":"Background In modern applications, services are typically provided through RESTFul HTTP URIs. Using RESTFul HTTP URIs (as unique resource identifiers) offers high readability, making it easier for both clients and servers to understand. However, in the observability field, this approach poses several challenges:\nA large number of endpoints (HTTP URI): Browsing through all externally provided endpoints becomes more difficult, making it hard to identify problematic endpoints. Metrics are difficult to collect: It becomes particularly challenging to categorize similar endpoints and generate observability metrics. In existing solutions, this issue can be resolved following these application-level resolutions:\nAgent Detection: In certain frameworks, rules are often declared to handle RESTFul requests. For example, in Java\u0026rsquo;s Spring Web, annotations like @GET can be used, which can then be linked to current requests using a Java Agent. OpenAPI: Predefined files can be associated with the application, allowing the observability system to be aware of the URIs that may be used. Both resolutions are tightly coupled with application settings, which can be limiting for unknown applications or applications where the agent cannot be monitored. Therefore, we need to consider whether there is a more general solution to identify URIs and merge metrics generated from similar URIs for better representation.\nR3 R3(RESTFul Pattern Recognition) is a high-performance RESTFul URI recognition tool inspired by Drain3. It can be deployed as a standalone application on the observability server and communicate with the SkyWalking OAP.\nR3 can accept a URI list via the gRPC protocol and aggregate similar URIs into a specific format. The aggregated (formatted) URI list can also be queried using the gRPC protocol.\nData Interaction Flow OAP receives and caches unformatted URI list: OAP receives observability data through different protocols and identifies all unformatted URIs. These URIs are stored in a temporary list categorized by the service they belong to. OAP sends URIs to be formatted to R3: OAP periodically batches the URIs that need formatting and sends them to the R3 service. R3 receives and parses the URI list: R3 asynchronously analyzes the similarity of the received URIs and stores (persists) the results on the local disk to allow features like recovery after a restart. OAP queries formatted URI list from R3: OAP periodically queries R3 for the detected formatted URIs and saves the results in memory. OAP formats URIs: When OAP receives new observability data, it matches the URIs against the formatted URIs retrieved from R3. If a match is found, the formatted URI is used for subsequent metric calculations. Scenarios In R3, the following scenarios are primarily addressed. For URIs identified as duplicates, R3 would replace the variable parts with {var} to standardize them.\nID Matching A common practice in RESTFul APIs is to include various IDs in the URI paths, which leads to a large number of unique URI endpoints. For example, paths like the following will be aggregated by R3 into a standardized format: /api/users/{var}.\n/api/users/cbf11b02ea464447b507e8852c32190a /api/users/5e363a4a18b7464b8cbff1a7ee4c91ca /api/users/44cf77fc351f4c6c9c4f1448f2f12800 /api/users/38d3be5f9bd44f7f98906ea049694511 /api/users/5ad14302e7924f4aa1d60e58d65b3dd2 Word Detection In RESTFul URIs, operations on an entity are usually specified using HTTP methods, but often additional types are needed. This is addressed by including specific nouns in the path. To handle this, R3 implements word parsing: when R3 detects specific words in the path, it will not format that part. For example, URIs like the following would not be considered similar and therefore will not be merged:\n/api/sale /api/product_sale /api/ProductSale Low Sample To prevent incorrect judgments due to insufficient sample sizes, R3 allows the configuration of a combine min URI count parameter in the configuration file. This parameter sets the minimum number of similar paths required before proceeding with the analysis.\nSuch as the threshold is 3, the following URI would keep the original URI, not parameterized.\n/api/fetch1 /api/fetch2 But the following URI would be parametrized to /api/{var}, since the sample count is bigger than the threshold.\n/api/fetch1 /api/fetch2 /api/fetch3 Version API In real-world scenarios, we often encounter URIs with multiple versions. R3 addresses this by ensuring that if a specified path contains a v\\\\d+ parameter (indicating version information), that part would not be parameterized. For example, the following URIs will be separately parsed into /test/v1/{var} and /test/v999/{var}.\n/test/v1/cbf11b02ea464447b507e8852c32190a /test/v1/5e363a4a18b7464b8cbff1a7ee4c91ca /test/v1/38d3be5f9bd44f7f98906ea049694511 /test/v999/1 /test/v999/2 /test/v999/3 Demo Next, let’s quickly demonstrate how to use R3 to format observed endpoints, so you can understand more specifically what it accomplishes.\nDeploy SkyWalking Showcase SkyWalking Showcase contains a complete set of example services and can be monitored using SkyWalking. For more information, please check the official documentation.\nIn this demo, we only deploy service, the latest released SkyWalking OAP, R3 service and UI.\nexport FEATURE_FLAGS=java-agent-injector,single-node,elasticsearch,r3 make deploy.kubernetes After deployment is complete, please run the following script to open SkyWalking UI: http://localhost:8080/.\nkubectl port-forward svc/ui 8080:8080 --namespace default Trigger RESTFul Requests In R3, a scheduled task is started by default to generate RESTFul traffic at regular intervals. However, you can also manually trigger this process using the following command:\nkubectl exec -n sample-services $(kubectl get pod -n sample-services --selector=app=gateway -o jsonpath=\u0026#39;{.items[0].metadata.name}\u0026#39;) -- /bin/bash -c \u0026#39;for i in $(seq 1 200); do curl http://rating/songs/$i/reviews/$((i+1)); sleep 1; done\u0026#39; In the above command, R3 would automatically locate the gateway node and send requests in RESTFul format to the rating service within that node. This allows R3 to generate and test traffic patterns that simulate real RESTFul requests to the target service.\nCheck Formatted URIs Once the RESTFul requests are triggered, you can view the aggregated endpoints in the UI.\nNote: Since the formatted endpoints are generated asynchronously, some of the earlier requests may not yet be formatted. You may need to wait for some time before the UI shows only the formatted addresses.\nConclusion In this article, we discussed in detail how SkyWalking utilizes the R3 service to format RESTFul URIs and aggregate related metrics upon receiving them. Currently, it applies to most RESTFul scenarios, and if more cases need to be supported, we can extend it further as needed.\n","excerpt":"\u003ch2 id=\"background\"\u003eBackground\u003c/h2\u003e\n\u003cp\u003eIn modern applications, services are typically provided through RESTFul HTTP URIs.\nUsing …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2024-010-15-introduce-r3-to-recognition-restful-url/","title":"Introduce R3 to recognition RESTFul URI"},{"body":"背景 在现代应用中，服务通常通过 RESTFul HTTP URI 提供。使用 RESTFul HTTP URI 作为唯一的资源标识符，不仅具备良好的可读性，还能让客户端和服务器更容易理解请求。然而，在可观测性领域，这种方式也带来了一些挑战：\n大量的端点（HTTP URI）：浏览所有对外提供的端点变得更加困难，因此很难识别出存在问题的端点。 指标收集困难：尤其难以对类似的端点进行归类并生成可观测性指标。 现有解决方案通常采用以下应用级别的方式来解决此问题：\n代理检测：在某些框架中，通常会声明规则来处理 RESTFul 请求。例如，在 Java 的 Spring Web 中可以使用 @GET 等注解，然后可以通过 Java Agent 将其与当前请求关联起来。 OpenAPI：应用可以关联预定义文件，使可观测性系统知晓可能使用的 URI。 这两种解决方案都与应用设置紧密耦合，这对于未知应用或者无法监控代理的应用来说是一个局限。因此，我们需要考虑是否有一种更通用的解决方案来识别 URI，并合并来自类似 URI 生成的指标，以便更好地展示数据。\nR3 R3（RESTFul 模式识别）是一个高性能的 RESTFul URI 识别工具，其灵感来自 Drain3。它可以作为独立应用部署在可观测性服务器上，并与 SkyWalking OAP 进行通信。\nR3 可以通过 gRPC 协议接收 URI 列表，并将类似的 URI 聚合为特定格式。聚合后的（格式化）URI 列表也可以通过 gRPC 协议进行查询。\n数据交互流程 OAP 接收并缓存未格式化的 URI 列表：OAP 通过不同协议接收可观测性数据，并识别出所有未格式化的 URI。然后将这些 URI 按照所属服务暂时存储在列表中。 OAP 将待格式化的 URI 发送给 R3：OAP 定期将需要格式化的 URI 批量发送到 R3 服务。 R3 接收并解析 URI 列表：R3 异步分析接收到的 URI 相似性，并将结果存储（持久化）在本地磁盘中，以便在重启后进行恢复。 OAP 查询 R3 中的格式化 URI 列表：OAP 定期查询 R3 中检测到的格式化 URI，并将结果保存在内存中。 OAP 格式化 URI：当 OAP 接收到新的可观测性数据时，会将 URI 与从 R3 获取的格式化 URI 进行匹配。如果匹配成功，后续的指标计算将使用格式化后的 URI。 场景 在 R3 中，主要解决以下场景。对于识别为重复的 URI，R3 将用 {var} 替换变量部分以标准化 URI。\nID 匹配 在 RESTFul API 中，常见做法是将各种 ID 包含在 URI 路径中，这导致了大量唯一的 URI 端点。例如，以下路径将被 R3 聚合为标准化格式 /api/users/{var}。\n/api/users/cbf11b02ea464447b507e8852c32190a /api/users/5e363a4a18b7464b8cbff1a7ee4c91ca /api/users/44cf77fc351f4c6c9c4f1448f2f12800 /api/users/38d3be5f9bd44f7f98906ea049694511 /api/users/5ad14302e7924f4aa1d60e58d65b3dd2 词语检测 在 RESTFul URI 中，实体的操作通常通过 HTTP 方法指定，但有时还需要在路径中添加特定名词。为此，R3 实现了词语解析：当 R3 检测到路径中的特定词语时，将不会格式化该部分。例如，以下 URI 将不会被视为相似，因此不会被合并：\n/api/sale /api/product_sale /api/ProductSale 样本不足 为防止由于样本不足而导致错误判断，R3 允许在配置文件中配置最小 URI 合并计数参数。\n例如，当阈值为 3 时，以下 URI 将保持原样，不会被参数化。\n/api/fetch1 /api/fetch2 但是以下 URI 将被参数化为 /api/{var}，因为样本数大于阈值。\n/api/fetch1 /api/fetch2 /api/fetch3 版本 API 在实际场景中，常会遇到包含多个版本的 URI。R3 通过确保指定路径中包含 v\\\\d+ 参数（表示版本信息）来解决这个问题，该部分将不会被参数化。例如，以下 URI 将分别解析为 /test/v1/{var} 和 /test/v999/{var}。\n/test/v1/cbf11b02ea464447b507e8852c32190a /test/v1/5e363a4a18b7464b8cbff1a7ee4c91ca /test/v1/38d3be5f9bd44f7f98906ea049694511 /test/v999/1 /test/v999/2 /test/v999/3 演示 接下来我们快速演示如何使用 R3 格式化观察到的端点，帮助你更具体地理解它的功能。\n部署 SkyWalking Showcase SkyWalking Showcase 包含一整套示例服务，可以通过 SkyWalking 进行监控。更多信息请查看官方文档。\n在此演示中，我们仅部署服务、最新发布的 SkyWalking OAP、R3 服务和 UI。\nexport FEATURE_FLAGS=java-agent-injector,single-node,elasticsearch,r3 make deploy.kubernetes 部署完成后，请运行以下脚本打开 SkyWalking UI：http://localhost:8080/。\nkubectl port-forward svc/ui 8080:8080 --namespace default 触发 RESTFul 请求 在 R3 中，默认启动了定时任务以定期生成 RESTFul 流量。不过，你也可以使用以下命令手动触发此过程：\nkubectl exec -n sample-services $(kubectl get pod -n sample-services --selector=app=gateway -o jsonpath=\u0026#39;{.items[0].metadata.name}\u0026#39;) -- /bin/bash -c \u0026#39;for i in $(seq 1 200); do curl http://rating/songs/$i/reviews/$((i+1)); sleep 1; done\u0026#39; 在上述命令中，R3 将自动定位网关节点，并以 RESTFul 格式向该节点内的 rating 服务发送请求。此操作允许 R3 生成并测试模拟实际 RESTFul 请求的流量模式。\n查看格式化 URI 一旦触发 RESTFul 请求，你可以在 UI 中查看聚合后的端点。\n注意：由于格式化端点是异步生成的，因此一些较早的请求可能尚未被格式化。你可能需要等待一段时间，UI 才会仅显示格式化后的地址。\n结论 在本文中，我们详细讨论了 SkyWalking 如何利用 R3 服务来格式化 RESTFul URI，并在接收后聚合相关的指标。目前，它适用于大多数 RESTFul 场景，如果需要支持更多情况，我们可以根据需求进一步扩展。\n","excerpt":"\u003ch2 id=\"背景\"\u003e背景\u003c/h2\u003e\n\u003cp\u003e在现代应用中，服务通常通过 RESTFul HTTP URI 提供。使用 RESTFul HTTP URI 作为唯一的资源标识符，不仅具备良好的可读性，还能让客户端和服务器更容易理解请求。然而， …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2024-010-15-introduce-r3-to-recognition-restful-url/","title":"引入 R3 识别 RESTFul URI"},{"body":"SkyWalking BanyanDB Helm 0.3.0 is released. Go to downloads page to find release tars.\nFeatures Support Anti-Affinity for banyandb cluster mode Align the modern Kubernetes label names Opt probe settings to http get /healthz instead of bydbctl health check Add standalone UI deployment Chores Bump banyandb image version to 0.7.0 ","excerpt":"\u003cp\u003eSkyWalking BanyanDB Helm 0.3.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures …\u003c/h4\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-helm-0-3-0/","title":"Release Apache SkyWalking BanyanDB Helm 0.3.0"},{"body":"SkyWalking BanyanDB 0.7.0 is released. Go to downloads page to find release tars.\nFile System Changes Bump up the version of the file system to 1.1.0 which is not compatible with the previous version. Move the series index into segment. Swap the segment and the shard. Move indexed values in a measure from data files to index files. Merge elementIDs.bin and timestamps.bin into a single file. Features Check unregistered nodes in background. Improve sorting performance of stream. Add the measure query trace. Assign a separate lookup table to each group in the maglev selector. Convert the async local pipeline to a sync pipeline. Add the stream query trace. Add the topN query trace. Introduce the round-robin selector to Liaison Node. Optimize query performance of series index. Add liaison, remote queue, storage(rotation), time-series tables, metadata cache and scheduler metrics. Add HTTP health check endpoint for the data node. Add slow query log for the distributed query and local query. Support applying the index rule to the tag belonging to the entity. Add search analyzer \u0026ldquo;url\u0026rdquo; which breaks test into tokens at any non-letter and non-digit character. Introduce \u0026ldquo;match_option\u0026rdquo; to the \u0026ldquo;match\u0026rdquo; query. Bugs Fix the filtering of stream in descending order by timestamp. Fix querying old data points when the data is in a newer part. A version column is introduced to each data point and stored in the timestamp file. Fix the bug that duplicated data points from different data nodes are returned. Fix the bug that the data node can\u0026rsquo;t re-register to etcd when the connection is lost. Fix memory leak in sorting the stream by the inverted index. Fix the wrong array flags parsing in command line. The array flags should be parsed by \u0026ldquo;StringSlice\u0026rdquo; instead of \u0026ldquo;StringArray\u0026rdquo;. Fix a bug that the Stream module didn\u0026rsquo;t support duplicated in index-based filtering and sorting Fix the bug that segment\u0026rsquo;s reference count is increased twice when the controller try to create an existing segment. Fix a bug where a distributed query would return an empty result if the \u0026ldquo;limit\u0026rdquo; was set much lower than the \u0026ldquo;offset\u0026rdquo;. Fix duplicated measure data in a single part. Fix several \u0026ldquo;sync.Pool\u0026rdquo; leak issues by adding a tracker to the pool. Fix panic when removing a expired segment. Fix panic when reading a disorder block of measure. This block\u0026rsquo;s versions are not sorted in descending order. Fix the bug that the etcd client doesn\u0026rsquo;t reconnect when facing the context timeout in the startup phase. Fix the bug that the long running query doesn\u0026rsquo;t stop when the context is canceled. Fix the bug that merge block with different tags or fields. Fix the bug that the pending measure block is not released when a full block is merged. Documentation Introduce new doc menu structure. Add installation on Docker and Kubernetes. Add quick-start guide. Add web-ui interacting guide. Add bydbctl interacting guide. Add cluster management guide. Add operation related documents: configuration, troubleshooting, system, upgrade, and observability. Chores Bump up the version of infra e2e framework. Separate the monolithic release package into two packages: banyand and bydbctl. Separate the monolithic Docker image into two images: banyand and bydbctl. Update CI to publish linux/amd64 and linux/arm64 Docker images. Make the build system compiles the binary based on the platform which is running on. Push \u0026ldquo;skywalking-banyandb:-testing\u0026rdquo; image for e2e and stress test. This image contains bydbctl to do a health check. Set etcd-client log level to \u0026ldquo;error\u0026rdquo; and etcd-server log level to \u0026ldquo;warn\u0026rdquo;. Push \u0026ldquo;skywalking-banyandb:-slim\u0026rdquo; image for the production environment. This image doesn\u0026rsquo;t contain bydbctl and Web UI. Bump go to 1.23. ","excerpt":"\u003cp\u003eSkyWalking BanyanDB 0.7.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"file-system-changes\"\u003eFile System …\u003c/h3\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-0-7-0/","title":"Release Apache SkyWalking BanyanDB 0.7.0"},{"body":"SkyWalking 10.1.0 is released. Go to downloads page to find release tars.\nA Version of PERFORMANCE Huge UI Performance Improvement. Metrics widgets queries are bundled by leveraging the GraphQL capabilities. Parallel Queries Support in GraphQL engine. Improve query performance. Significantly improve the performance of OTEL metrics handler. Reduce CPU and GC costs in OTEL metrics processes. With adopting BanyanDB 0.7, native database performance and stability are improved. Project E2E: bump up the version of the opentelemetry-collector to 0.102.1. Push snapshot data-generator docker image to ghcr.io. Bump up skywalking-infra-e2e to work around GHA removing docker-compose v1. Bump up CodeQL GitHub Actions. Fix wrong phase of delombok plugin to reduce build warnings. Use ci-friendly revision to set the project version. OAP Server Fix wrong indices in the eBPF Profiling related models. Support exclude the specific namespaces traffic in the eBPF Access Log receiver. Add Golang as a supported language for Elasticsearch. Remove unnecessary BanyanDB flushing logs(info). Increase SW_CORE_GRPC_MAX_MESSAGE_SIZE to 50MB. Support to query relation metrics through PromQL. Support trace MQE query for debugging. Add Component ID(158) for the Solon framework. Fix metrics tag in HTTP handler of browser receiver plugin. Increase alarm_record#message column length to 2000 from 200. Remove alarm_record#message column indexing. Add Python as a supported language for Pulsar. Make more proper histogram buckets for the persistence_timer_bulk_prepare_latency, persistence_timer_bulk_execute_latency and persistence_timer_bulk_all_latency metrics in PersistenceTimer. [Break Change] Update Nacos version to 2.3.2. Nacos 1.x server can\u0026rsquo;t serve as cluster coordinator and configuration server. Support tracing trace query(SkyWalking and Zipkin) for debugging. Fix BanyanDB metrics query: used the wrong Downsampling type to find the schema. Support fetch cilium flow to monitoring network traffic between cilium services. Support labelCount function in the OAL engine. Support BanyanDB internal measure query execution tracing. BanyanDB client config: rise the default maxBulkSize to 10000, add flushTimeout and set default to 10s. Polish BanyanDB group and schema creation logic to fix the schema creation failure issue in distributed race conditions. Support tracing topology query for debugging. Fix expression of graph Current QPS in MySQL dashboard. Support tracing logs query for debugging. BanyanDB: fix Tag autocomplete data storage and query. Support aggregation operators in PromQL query. Update the kubernetes HTTP latency related metrics source unit from ns to ms. Support BanyanDB internal stream query execution tracing. Fix Elasticsearch, MySQL, RabbitMQ dashboards typos and missing expressions. BanyanDB: Zipkin Module set service as Entity for improving the query performance. MQE: check the metrics value before do binary operation to improve robustness. Replace workaround with Armeria native supported context path. Add an http endpoint wrapper for health check. Bump up Armeria and transitive dependencies. BanyanDB: if the model column is already a @BanyanDB.TimestampColumn, set @BanyanDB.NoIndexing on it to reduce indexes. BanyanDB: stream sort-by time query, use internal time-series rather than index to improve the query performance. Bump up graphql-java to 21.5. Add Unknown Node when receive Kubernetes peer address is not aware in current cluster. Fix CounterWindow concurrent increase cause NPE by PriorityQueue Fix format the endpoint name with empty string. Support async query for the composite GraphQL query. Get endpoint list order by timestamp desc. Support sort queries on metrics generated by eBPF receiver. Fix the compatibility with Grafana 11 when using label_values query variables. Nacos as config server and cluster coordinator supports configuration contextPath. Update the endpoint name format to \u0026lt;Method\u0026gt;:\u0026lt;Path\u0026gt; in eBPF Access Log Receiver. Add self-observability metrics for OpenTelemetry receiver. Support service level metrics aggregate when missing pod context in eBPF Access Log Receiver. Fix query getGlobalTopology throw exception when didn\u0026rsquo;t find any services by the given Layer. Fix the previous analysis result missing in the ALS k8s-mesh analyzer. Fix findEndpoint query requires keyword when using BanyanDB. Support to analysis the ztunnel mapped IP address and mTLS mode in eBPF Access Log Receiver. Adapt BanyanDB Java Client 0.7.0. Add SkyWalking Java Agent self observability dashboard. Add Component ID(5022) for the GoFrame framework. Bump up protobuf java dependencies to 3.25.5. BanyanDB: support using native term searching for keyword in query findEndpoint and getAlarm. BanyanDB: support TLS connection and configuration. PromQL service: query API support RFC3399 time format. Improve the performance of OTEL metrics handler. PromQL service: fix operators result missing rangeExpression flag. BanyanDB: use TimestampRange to improve \u0026ldquo;events\u0026rdquo; query for BanyanDB. Optimize network_address_alias table to reduce the number of the index. PromQL service: support round brackets operator. Support query Alarm message Tag for auto-complete. UI Highlight search log keywords. Add Error URL in the browser log. Add a SolonMVC icon. Adding cilium icon and i18n for menu. Fix the mismatch between the unit and calculation of the \u0026ldquo;Network Bandwidth Usage\u0026rdquo; widget in Windows-Service Dashboard. Make a maximum 20 entities per query in service/instance/endpoint list widgets. Polish error nodes in trace widget. Introduce flame graph to the trace profiling. Correct services and instances when changing page numbers. Improve metric queries to make page opening brisker. Bump up dependencies to fix CVEs. Add a loading view for initialization page. Fix a bug for selectors when clicking the refresh icon. Fix health check to OAP backend. Add Service, ServiceInstance, Endpoint dashboard forwarder to Kubernetes Topologies. Fix pagination for service/instance list widgets. Add queries for alarm tags. Add skywalking java agent self observability menu. Documentation Update the version description supported by zabbix receiver. Move the Official Dashboard docs to marketplace docs. Add marketplace introduction docs under quick start menu to reduce the confusion of finding feature docs. Update Windows Metrics(Swap -\u0026gt; Virtual Memory) All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking 10.1.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"a-version-of-performance\"\u003eA Version of PERFORMANCE …\u003c/h4\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-10.1.0/","title":"Release Apache SkyWalking APM 10.1.0"},{"body":"SkyWalking BanyanDB 0.7.0 is released. Go to downloads page to find release tars.\nFeatures Bump up the API of BanyanDB Server to support the query trace. Add trace to response. Add ToString annotation to Tag. Enhance the BulkWriteProcessor. Provide a new method to order data by timestamp. Refactor metadata object to original protocol. Complemented the Schema management API. Enhance the MetadataCache. Add more IT tests. Remove analyze DNS name to get/refresh IP for create connection. Support new Match Query proto. Bugs Fix MeasureQuery.SumBy to use SUM instead of COUNT Add missing FloatFieldValue type in the Measure write operation Fix wrong result of the Duration.ofDay Remove duplicate orderBy method in measure query. ","excerpt":"\u003cp\u003eSkyWalking BanyanDB 0.7.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"features\"\u003eFeatures\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eBump up …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-java-client-0-7-0/","title":"Release Apache SkyWalking BanyanDB Java Client 0.7.0"},{"body":"SkyWalking Go 0.5.0 is released. Go to downloads page to find release tars.\nAdd go 1.23 support. Remove go 1.16, 1.17, and 1.18 support. Features Add support trace ignore. Enhance the observability of makefile execution. Update the error message if the peer address is empty when creating an exit span. Support enhancement go 1.23. Plugins Support Pulsar MQ. Support Segmentio-Kafka MQ. Support http headers collection for Gin. Support higher versions of grpc. Support go-elasticsearchv8 database client framework. Support http.Hijacker interface for mux plugin. Support collect statements and parameters in the Gorm plugin. Bug Fixes Fix panic error when root span finished. Fix when not route is found, the gin operation name is \u0026ldquo;http.Method:\u0026rdquo;, example: \u0026ldquo;GET:\u0026rdquo;. Fix got span type is wrong error when creating exit span with trace sampling. Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Go 0.5.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eAdd go \u003ccode\u003e1.23\u003c/code\u003e support\u003c/strong\u003e. …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-go-0.5.0/","title":"Release Apache SkyWalking Go 0.5.0"},{"body":"SkyWalking LUA Nginx 1.0.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed Update log4j to 2.17.1 to address CVE-2021-44832. by @jeremie1112 in https://github.com/apache/skywalking-nginx-lua/pull/91 Update NOTICE year to 2022 by @dmsolr in https://github.com/apache/skywalking-nginx-lua/pull/92 ci: run lua test cases with luajit by @tzssangglass in https://github.com/apache/skywalking-nginx-lua/pull/94 Add IgnoreSuffix feature by @alonelaval in https://github.com/apache/skywalking-nginx-lua/pull/93 feat: support update the peer before requesting outgoing by @dmsolr in https://github.com/apache/skywalking-nginx-lua/pull/95 use agent-test-tool docker image instead of building from source by @dmsolr in https://github.com/apache/skywalking-nginx-lua/pull/96 improve e2e test by @dmsolr in https://github.com/apache/skywalking-nginx-lua/pull/103 support to try to use request-id as trace-id when trace context absent by @dmsolr in https://github.com/apache/skywalking-nginx-lua/pull/104 stop reporting traces after the worker process begins to exit by @wangrzneu in https://github.com/apache/skywalking-nginx-lua/pull/105 Updated tag key from http.status to http.status_code by @wuwen5 in https://github.com/apache/skywalking-nginx-lua/pull/107 Prepare for version 1.0.0 release and fix typos in doc. by @wuwen5 in https://github.com/apache/skywalking-nginx-lua/pull/108 New Contributors @jeremie1112 made their first contribution in https://github.com/apache/skywalking-nginx-lua/pull/91 @tzssangglass made their first contribution in https://github.com/apache/skywalking-nginx-lua/pull/94 @alonelaval made their first contribution in https://github.com/apache/skywalking-nginx-lua/pull/93 @wuwen5 made their first contribution in https://github.com/apache/skywalking-nginx-lua/pull/107 ","excerpt":"\u003cp\u003eSkyWalking LUA Nginx 1.0.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-lua-nginx-1.0.0/","title":"Release Apache SkyWalking LUA Nginx 1.0.0"},{"body":"SkyWalking Java Agent 9.3.0 is released. Go to downloads page to find release tars. Changes by Version\n9.3.0 Remove idleCount tag in Alibaba Druid meter plugin. Fix NPE in handleMethodException method of apm-jdk-threadpool-plugin. Support for C3P0 connection pool tracing. Use a daemon thread to flush logs. Fix typos in URLParser. Add support for Derby/Sybase/SQLite/DB2/OceanBase jdbc url format in URLParser. Optimize spring-plugins:scheduled-annotation-plugin compatibility about Spring 6.1.x support. Add a forceIgnoring mechanism in a CROSS_THREAD scenario. Fix NPE in Redisson plugin since Redisson 3.20.0. Support for showing batch command details and ignoring PING commands in Redisson plugin. Fix peer value of Master-Slave mode in Redisson plugin. Support for tracing the callbacks of asynchronous methods in elasticsearch-6.x-plugin/elasticsearch-7.x-plugin. Fixed the invalid issue in the isInterface method in PluginFinder. Fix the opentracing toolkit SPI config Improve 4x performance of ContextManagerExtendService.createTraceContext() Add a plugin that supports the Solon framework. Fixed issues in the MySQL component where the executeBatch method could result in empty SQL statements. Support kafka-clients-3.7.x. Documentation Update docs to describe expired-plugins. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking Java Agent 9.3.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nChanges by …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-java-agent-9-3-0/","title":"Release Apache SkyWalking Java Agent 9.3.0"},{"body":"SkyWalking Python 1.1.0 is released! Go to downloads page to find release tars.\nPyPI Wheel: https://pypi.org/project/apache-skywalking/1.1.0/\nDockerHub Image: https://hub.docker.com/r/apache/skywalking-python\nWhat\u0026rsquo;s Changed Fix wrong docker tag name by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/307 Update release doc to reflect new steps by @Superskyyy in https://github.com/apache/skywalking-python/pull/306 Fix unexpected \u0026lsquo;No active span\u0026rsquo; IllegalStateError by @ZEALi in https://github.com/apache/skywalking-python/pull/311 Add Neo4j plugin. by @Jedore in https://github.com/apache/skywalking-python/pull/312 Update README.md to reflect new slack channel by @Superskyyy in https://github.com/apache/skywalking-python/pull/313 Replace Kafka CI image tags to sha by @FAWC438 in https://github.com/apache/skywalking-python/pull/319 Python agent performance enhancement with asyncio by @FAWC438 in https://github.com/apache/skywalking-python/pull/316 loose restrict of greenlet (#3) by @jaychoww in https://github.com/apache/skywalking-python/pull/326 Add support printing TID to logs by @CodePrometheus in https://github.com/apache/skywalking-python/pull/323 Fix psutil dockerfile version constraint by @Superskyyy in https://github.com/apache/skywalking-python/pull/328 Change from pkg_resources to importlib metadata by @shenxiangzhuang in https://github.com/apache/skywalking-python/pull/329 Update NOTICE to 2024 by @Superskyyy in https://github.com/apache/skywalking-python/pull/332 Disable uwsgi e2e by @Superskyyy in https://github.com/apache/skywalking-python/pull/337 Fix unexpected \u0026lsquo;decode\u0026rsquo; AttributeError when MySQLdb module is mapped by PyMySQL by @ZEALi in https://github.com/apache/skywalking-python/pull/336 Add Pulsar plugin by @CodePrometheus in https://github.com/apache/skywalking-python/pull/345 fix(agent): no attribute \u0026lsquo;_SkyWalkingAgent__log_queue\u0026rsquo; using kafka plain text by @tsonglew in https://github.com/apache/skywalking-python/pull/343 Bump up version to 1.1.0 by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/347 New Contributors @CodePrometheus made their first contribution in https://github.com/apache/skywalking-python/pull/323 @shenxiangzhuang made their first contribution in https://github.com/apache/skywalking-python/pull/329 @tsonglew made their first contribution in https://github.com/apache/skywalking-python/pull/343 Full Changelog: https://github.com/apache/skywalking-python/compare/v1.0.1...v1.1.0\n","excerpt":"\u003cp\u003eSkyWalking Python 1.1.0 is released! Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePyPI Wheel\u003c/strong\u003e: …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-python-1-1-0/","title":"Release Apache SkyWalking Python 1.1.0"},{"body":"SkyWalking Client JS 0.12.0 is released. Go to downloads page to find release tars.\nFix native fetch implementation when using Request object. Fix fetch implementation when using the Header object in http queries. Bump dependencies. ","excerpt":"\u003cp\u003eSkyWalking Client JS 0.12.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eFix native …\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-client-js-0-12-0/","title":"Release Apache SkyWalking Client JS 0.12.0"},{"body":"引言 我们很高兴地宣布 BanyanDB v0.6 的发布，这是我们数据库技术发展的一个重要里程碑。这个最新版本引入了一种开创性的基于列的文件系统，提高了处理大数据集的性能和效率。经过广泛测试，我们可以确认这个新文件系统已经准备好投入生产。BanyanDB 现已准备就绪。\n在这篇博客中，我们将深入探讨新的架构和观察到的性能改进，并提供一个关于如何安装并开始使用 BanyanDB v0.6 的逐步指南。\n理解 BanyanDB 架构 BanyanDB 设计为一个高度可扩展的多模型数据库。BanyanDB 的架构是模块化的，允许在存储和索引策略上具有灵活性，这使其成为处理复杂数据环境的理想选择。\n主要特性： 多模型支持：无缝处理各种数据类型。 可扩展性：设计为可以在多个节点上水平扩展。 高性能：优化了快速数据检索和高数据吞吐率。 数据模型 BanyanDB 是一个多模型数据库，旨在支持包括时间序列和键值数据在内的多种数据类型。这种灵活性对于需要多样化数据处理能力的现代 APM 系统至关重要。\nBanyanDB 模型\n时间序列数据： BanyanDB 管理时间序列数据，即按时间顺序索引的数据点，通常在等间隔的时间点记录。这使其理想用于离散时间数据的序列。在 BanyanDB 中，你可以通过两种结构存储时间序列数据：\nStream：这种类型的数据适合记录，如日志、追踪和事件。Stream（流）有助于管理连续生成并顺序记录的数据。 Measure：专为摄取度量和概况而设计。Measure（度量）对于时间间隔的统计表示很有用。 键值数据： BanyanDB 中的键值模型是属性模型的一个子集。每个属性由一个唯一的键标识，格式为\u0026lt;group\u0026gt;/\u0026lt;name\u0026gt;/\u0026lt;id\u0026gt;，作为检索数据的主键。这个键一旦设置就是不可改变，确保数据的一致性和完整性。\n属性由几个键值对组成，称为 Tag。你可以根据 Tag 的键动态地添加、更新或删除 Tag，提供灵活性在管理和存储数据方面。例如，SkyWalking UI 模板利用这个模型有效地存储配置数据。\nBanyanDB 的集群化 BanyanDB 的架构不仅确保了高效的数据管理和高可用性，还强调了其集群环境中的可扩展性。系统包括三种不同的节点类型，每种类型都能独立扩展以满足不同的工作负载需求。\n数据节点（Data Node） 数据节点是存储和管理所有原始时间序列数据、元数据和索引数据的核心。这些节点采用无共享架构运行，彼此之间不直接通信也不共享任何数据，增强了集群的可用性并简化了维护和扩展。这种设计优先考虑可用性，即使某些节点暂时不可用，系统仍能保持数据摄取和查询操作。\n元数据节点（Meta Node） 元数据节点使用 etcd 实现，管理整个集群的元数据并确保系统的一致性。它们维护节点状态和数据库架构的全局视图，促进集群内的协调操作。\n联络节点（Liaison Node） 联络节点作为通信桥梁，将查询和数据路由到适当的数据节点。它们处理安全功能，如认证和 TTL 执行，并管理分布式查询执行以优化性能。它们在维护服务可用性中发挥关键作用；只要至少有一个数据节点在运行，联络节点就可以继续服务查询。在某些数据节点不可用的情况下，联络节点会将流量重定向到剩余的健康节点，这可能导致这些节点的资源使用增加。\nBanyanDB 集群\n通信 元数据节点 在集群中同步元数据。 数据节点 与元数据节点互动，更新和获取元数据。 联络节点 将数据路由到数据节点并协调查询过程，利用元数据节点的元数据进行有效的分发和执行。 可扩展性和容错性 BanyanDB 集群中的每种节点类型都可以根据部署的具体需要独立扩展：\n扩展数据节点 增加数据处理能力并在更重的负载下改善性能。 扩展元数据节点 增强集群元数据操作的管理能力和弹性。这些节点的数量应该是奇数。 扩展联络节点 改善查询处理和数据路由能力的吞吐量。 这种灵活性使 BanyanDB 能够适应需求变化，而不会损害性能或可用性。如果某些组件暂时不可用，系统设计为继续运营，优先保证可用性而非严格一致性。然而，在这种事件中，如果活动节点没有足够的资源来处理当前的工作负载，用户可能会经历数据摄取和查询处理的延迟或失败。\n在 Kubernetes 上安装 要在 Kubernetes 上安装 BanyanDB，你可以使用我们的 Helm chart，这简化了部署过程。你可以在我们的官方文档中找到详细的安装指南。\n这个逐步指南假设你对 Kubernetes 和 Helm 有基本的了解，Helm 是 Kubernetes 的包管理器。如果你不熟悉 Helm，你可能需要在继续之前先熟悉 Helm 的基础知识。\n先决条件 在我们开始之前，请确保你有以下内容：\nKubernetes 集群：你可以使用 Minikube 进行本地设置，或使用支持 Kubernetes 的任何云提供商，如 AWS、GCP 或 Azure。 Helm 3：确保 Helm 3 已安装并配置在你的机器上。你可以从 Helm 的官方网站 下载。 第 1 步：配置 Helm 以使用 OCI 由于 BanyanDB Helm chart 托管为 Docker Hub 中的 OCI chart，你需要确保你的 Helm 配置为处理 OCI 工件。\nhelm registry login registry-1.docker.io 你将被提示输入你的 Docker Hub 用户名和密码。此步骤是从 Docker Hub 拉取 Helm chart 所必需的。\n第 2 步：设置环境变量 接下来，设置 SkyWalking 发行版本、名称和命名空间的环境变量。这些变量将用于后续命令。\nexport SKYWALKING_RELEASE_VERSION=4.6.0 export SKYWALKING_RELEASE_NAME=skywalking export SKYWALKING_RELEASE_NAMESPACE=default 第 3 步：使用 Helm 安装 BanyanDB+SkyWalking helm install \u0026#34;${SKYWALKING_RELEASE_NAME}\u0026#34; \\ oci://registry-1.docker.io/apache/skywalking-helm \\ --version \u0026#34;${SKYWALKING_RELEASE_VERSION}\u0026#34; \\ -n \u0026#34;${SKYWALKING_RELEASE_NAMESPACE}\u0026#34; \\ --set oap.image.tag=10.0.1 \\ --set oap.storageType=banyandb \\ --set ui.image.tag=10.0.1 \\ --set elasticsearch.enabled=false \\ --set banyandb.enabled=true \\ --set banyandb.image.tag=0.6.1 \\ --set banyandb.standalone.enabled=false \\ --set banyandb.cluster.enabled=true \\ --set banyandb.etcd.enabled=true 此命令将部署 SkyWalking OAP 集群和 BanyanDB 集群到你的 Kubernetes 环境。\n第 4 步：验证安装 检查 pod 的状态以确保它们正常运行：\nkubectl get pods -l release=skywalking 如果一切配置正确，你应该看到以下 pod 处于Running或Completed状态。\nBanyanDB 集群中的 Pod\n第 5 步：访问 SkyWalking UI 要访问 SkyWalking UI，你可以检查服务：\nkubectl get svc 你应该选择服务skywalkin-skywalking-helm-ui来访问 UI。\n性能测试 我们针对 Elasticsearch 8.13.2 对 BanyanDB v0.6.1 进行了基准测试，SkyWalking 推荐的数据库。新的 BanyanDB 在几个关键领域表现优于 Elasticsearch，特别是在内存使用和磁盘空间方面。\n数据生成工具 对于此测试，我们使用了一个自定义的数据生成工具，设计用来创建模拟典型实际场景的追踪和度量数据。\n服务、实例和端点 Total Services：20 Groups of Services：每 3 个生成器实例运行两组，贡献总服务数。 Instances per Service：每个服务有 20 个实例，所有服务共有 400 个实例。 Endpoints per Service：每个服务实例托管 100 个端点。 Total Endpoints：20 个服务的总端点数为 2000。 Trace 和 Segment 生成 Trace Gerneration Rate：每组服务每秒生成 1000 条 trace，有效模拟大规模微服务环境中的高负载场景。 Spans per Trace：每条追踪包含五个 segement，详细描述了各种服务和实例之间的模拟交互。 Total Writes per Second：2 group * 3 data-generator * 1000 trace * 5 segment = 30k segment。 我们设计的额外查询类型代表了生产环境中监控微服务架构的 SkyWalking 执行的典型读取操作。每种查询类型针对服务数据的不同方面：\n1. Service Dashboard Queries 目的：在过去 30 分钟内获取 5 项服务级别的度量。 频率：每秒 1 次查询 2. Top-N List Queries 目的：在过去 30 分钟内检索 2 个特定服务的前 5 项度量。 频率：每秒 1 次查询。 3. Segment List Queries 目的：在过去 30 分钟内获取按降序排列的服务段列表。 频率：每秒 1 次查询。 4. Trace Detail Queries 目的：从段列表检索所有追踪细节。 频率：每秒 2 次查询。 设置 下表详细列出了集群内每个组件的规格，允许轻松比较分配给每个系统的硬件资源。这提供了对 Elasticsearch 8.13.2 和我们性能测试中使用的 BanyanDB v0.6.1 部署配置的清晰而结构化的比较。\n性能测试设置\n集群配置表 组件 系统 数量 CPU 核心 内存 (GB) 存储 (GB) 角色描述 主节点 Elasticsearch 3 2 6 N/A 集群协调和管理 数据节点 Elasticsearch 3 4 8 50 (高级 RWO) 数据存储、索引和查询处理 ETCD 节点 BanyanDB 3 2 4 N/A 元数据和集群状态存储 数据节点 BanyanDB 3 8 4 50 (高级 RWO) 数据存储和处理 联络节点 BanyanDB 2 4 4 N/A 协调客户端应用和数据节点间的联系 结果 在此我们整理了 Elasticsearch 8.13.2 和 BanyanDB v0.6.1 的性能测试结果，重点比较资源使用情况。结果分为两个表格以便更清晰地展示——一个详细介绍 CPU 和内存使用情况，另一个关注磁盘相关指标。\nCPU 和内存使用情况 系统 平均 CPU 使用率 (核心) 平均内存使用量 (MB) Elasticsearch 数据 3.2 4147 BanyanDB 数据 3.6 738 BanyanDB 联络 1.9 62 观察： CPU 使用率：BanyanDB 数据节点的 CPU 使用率略高，因为它们在压缩和解压数据文件时操作较多。然而，BanyanDB 联络节点的 CPU 使用明显较少。 内存使用率：BanyanDB 显示出显著较低的内存使用率，数据和联络节点的内存使用量几乎比 Elasticsearch 数据节点少 5 倍，凸显其在内存利用效率方面的优势。 磁盘使用、IOPS 和吞吐量 磁盘使用\n磁盘 IOPS \u0026amp; 吞吐量\n系统 平均磁盘使用量 (GB) IOPS (千) 磁盘吞吐量 (GB/s) Elasticsearch 数据 29.6 115.5 12.8 BanyanDB 数据 21.6 21.4 3.3 观察： 磁盘空间使用：BanyanDB 比 Elasticsearch 使用约 30% 较少的磁盘空间，这可能导致较低的存储成本。 IOPS 和吞吐量：BanyanDB 的 IOPS 和磁盘吞吐量显著较低，表明对磁盘资源的压力较小。这对于降低运营成本和延长物理存储设备的使用寿命可能是有益的。 结论 BanyanDB v0.6 的发布标志着数据库技术的重大进步，其新的基于列的文件系统在性能和效率上，尤其是在内存使用和磁盘空间方面与 Elasticsearch 相比显示出显著的改进。BanyanDB 能够处理各种数据类型，具有可扩展的架构以及在数据检索和摄取方面的高性能，使其成为复杂数据环境的强大解决方案。灵活的集群系统的引入允许独立扩展节点类型，确保在不影响性能或可用性的情况下适应变化的需求。总体而言，BanyanDB v0.6 将自己定位为现代应用性能管理 (APM) 系统的一个经济高效且可靠的选择。\n","excerpt":"\u003ch1 id=\"引言\"\u003e引言\u003c/h1\u003e\n\u003cp\u003e我们很高兴地宣布 BanyanDB v0.6 的发布，这是我们数据库技术发展的一个重要里程碑。这个最新版本引入了一种开创性的基于列的文件系统，提高了处理大数据集的性能和效率。经过广泛测试，我们可 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2024-06-04-banyandb-0.6-release/","title":"BanyanDB 0.6 版本发布：性能和效率的提升"},{"body":"SkyWalking Go自动增强探针可以自动探测Go应用的运行状态，包括服务、实例、端点、拓扑等信息，同时还可以自动探测Go应用的性能指标，包括响应时间、错误率、吞吐量等指标。 而不需要用户手动埋点，这样可以大大减少用户的工作量，提高用户的使用体验。\n本次直播由 SkyWalking PMC 刘晗为大家介绍 SkyWalking Go 自动增强探针的原理和实现，主要包含以下几部分内容：\n如何快速通过将程序与 SkyWalking Go 集成 了解 SkyWalking Go 探针如何利用 Go ToolChain 进行自动增强 了解 SkyWalking Go 如何与 SkyWalking 后端进行交互 B站视频地址\n","excerpt":"\u003cp\u003eSkyWalking Go自动增强探针可以自动探测Go应用的运行状态，包括服务、实例、端点、拓扑等信息，同时还可以自动探测Go应用的性能指标，包括响应时间、错误率、吞吐量等指标。\n而不需要用户手动埋点 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2024-06-06-skywalking-in-practice-s01e06/","title":"SkyWalking从入门到精通 - 2024系列线上分享活动（第六讲）"},{"body":"Introduction We are excited to announce the release of BanyanDB v0.6, a significant milestone in the evolution of our database technology. This latest version introduces a groundbreaking column-based file system that enhances performance and improves efficiency in handling large datasets. After extensive testing, we can confirm that this new file system is ready for production. BanyanDB is now production-ready.\nIn this blog post, we’ll dive deep into the new architecture and the performance improvements observed and provide a step-by-step guide on installing and getting started with BanyanDB v0.6.\nUnderstanding BanyanDB Architecture BanyanDB is designed as a highly scalable, multi-model database. The architecture of BanyanDB is modular, allowing for flexibility in storage and indexing strategies, which makes it an ideal choice for complex data environments.\nKey Features: Multi-Model Support: Seamlessly handles various data types. Scalability: Designed to scale horizontally across multiple nodes. High Performance: Optimized for quick data retrieval and high data ingestion rates. Data Model BanyanDB is a multi-model database engineered to support diverse data types, including time series and key-value data. This flexibility is essential for modern APM systems that require versatile data handling capabilities.\nBanyanDB Models\nTime-Series Data: BanyanDB manages time-series data, which are data points indexed in time order, typically logged at successive, equally spaced points in time. This makes it ideal for a sequence of discrete-time data. In BanyanDB, you can store time-series data through two structures:\nStream: This type of data is suitable for logging, such as logs, traces, and events. Streams help manage data that are continuously generated and sequentially recorded. Measure: Designed for ingesting metrics and profiles. Measures are useful for statistical representations over intervals of time. Key-Value Data: The key-value model in BanyanDB is a subset of the Property model. Each property is identified by a unique key formatted as \u0026lt;group\u0026gt;/\u0026lt;name\u0026gt;/\u0026lt;id\u0026gt;, which acts as a primary key for retrieving data. This key is immutable once set, ensuring data consistency and integrity.\nProperties consist of several key-value pairs, referred to as Tags. You can dynamically add, update, or drop tags based on the tag\u0026rsquo;s key, offering flexibility in managing and storing data. For example, SkyWalking UI templates utilize this model to store configuration data efficiently.\nClustering in BanyanDB BanyanDB\u0026rsquo;s architecture not only ensures efficient data management and high availability but also emphasizes scalability across its clustered environment. The system includes three distinct node types, each capable of scaling independently to meet varying workload demands.\nData Nodes Data Nodes are central to storing and managing all raw time series data, metadata, and indexed data. Operating under a shared-nothing architecture, these nodes do not communicate directly with each other nor share any data, enhancing cluster availability and simplifying maintenance and scaling. This design prioritizes availability, allowing the system to remain operational for data ingestion and querying even if some nodes are temporarily unavailable.\nMeta Nodes Implemented using etcd, Meta Nodes manage the overarching cluster metadata and ensure consistency across the system. They maintain a global view of the node states and database schemas, facilitating coordinated operations within the cluster.\nLiaison Nodes Liaison Nodes serve as the communication bridge, routing queries and data to the appropriate Data Nodes. They handle security functions like authentication and TTL enforcement, and manage distributed query execution to optimize performance. They play a crucial role in maintaining service availability; as long as at least one Data Node is operational, Liaison Nodes can continue to serve queries. In scenarios where some Data Nodes are unavailable, Liaison Nodes reroute traffic to the remaining healthy nodes, which may lead to increased resource use on these nodes.\nBanyanDB Cluster\nCommunication Meta Nodes synchronize metadata across the cluster. Data Nodes interact with Meta Nodes to update and fetch metadata. Liaison Nodes route data to Data Nodes and coordinate query processes, leveraging metadata from Meta Nodes for efficient distribution and execution. Scalability and Fault Tolerance Each node type within the BanyanDB cluster can be scaled independently based on the specific needs of the deployment:\nScaling Data Nodes increases data handling capacity and improves performance under heavier loads. Scaling Meta Nodes enhances the management capabilities and resiliency of cluster metadata operations. The number of such nodes should be odd. Scaling Liaison Nodes improves the throughput of query processing and data routing capabilities. This flexibility allows BanyanDB to adapt to changes in demand without compromising performance or availability. If some components become temporarily unavailable, the system is designed to continue operations, prioritizing availability over strict consistency. However, during such events, if the active nodes do not have sufficient resources to handle the current workload, users may experience delays or failures in data ingestion and query processing.\nInstallation On Kubernetes To install BanyanDB on Kubernetes, you can use our Helm chart, which simplifies the deployment process. You can find detailed installation instructions in our official documentation.\nThis step-by-step guide assumes you have a basic understanding of Kubernetes and Helm, the package manager for Kubernetes. If you\u0026rsquo;re new to Helm, you might want to familiarize yourself with Helm basics before proceeding.\nPrerequisites Before we begin, ensure you have the following:\nA Kubernetes Cluster: You can use Minikube for a local setup, or any cloud provider like AWS, GCP, or Azure that supports Kubernetes. Helm 3: Ensure Helm 3 is installed and configured on your machine. You can download it from Helm\u0026rsquo;s official website. Step 1: Configure Helm to Use OCI Since the BanyanDB Helm chart is hosted as an OCI chart in Docker Hub, you need to ensure your Helm is configured to handle OCI artifacts.\nhelm registry login registry-1.docker.io You will be prompted to enter your Docker Hub username and password. This step is necessary to pull Helm charts from Docker Hub.\nStep 2: Setup Env Variables Next, set up the environment variables for the SkyWalking release version, name, and namespace. These variables will be used in subsequent commands.\nexport SKYWALKING_RELEASE_VERSION=4.6.0 export SKYWALKING_RELEASE_NAME=skywalking export SKYWALKING_RELEASE_NAMESPACE=default Step 3: Install BanyanDB+SkyWalking Using Helm helm install \u0026#34;${SKYWALKING_RELEASE_NAME}\u0026#34; \\ oci://registry-1.docker.io/apache/skywalking-helm \\ --version \u0026#34;${SKYWALKING_RELEASE_VERSION}\u0026#34; \\ -n \u0026#34;${SKYWALKING_RELEASE_NAMESPACE}\u0026#34; \\ --set oap.image.tag=10.0.1 \\ --set oap.storageType=banyandb \\ --set ui.image.tag=10.0.1 \\ --set elasticsearch.enabled=false \\ --set banyandb.enabled=true \\ --set banyandb.image.tag=0.6.1 \\ --set banyandb.standalone.enabled=false \\ --set banyandb.cluster.enabled=true \\ --set banyandb.etcd.enabled=true This command will deploy the SkyWalking OAP cluster and BanyanDB cluster to your Kubernetes environment.\nStep 4: Verify the Installation Check the status of the pods to ensure they are running properly:\nkubectl get pods -l release=skywalking You should see the following pods in a Running or Completed state if everything is configured correctly.\nPods in BanyanDB Cluster\nStep 5: Access SkyWalking UI To access the SkyWalking UI, you can check the service by :\nkubectl get svc You should select the service skywalkin-skywalking-helm-ui to access the UI.\nPerformance Test We benchmarked BanyanDB v0.6.1 against Elasticsearch 8.13.2, SkyWalking’s recommended database. The new BanyanDB outperformed Elasticsearch in several key areas, particularly in memory usage and disk space.\nData Generation Tool For this test, we used a custom data generation tool designed to create data that mimics a typical real-world scenario for trace and metrics data.\nServices, Instances, and Endpoints Total Services: 20 Groups of Services: Each of the 3 generator instances runs two groups, contributing to the total count of services. Instances per Service: Each service is represented by 20 instances, leading to 400 instances across all services. Endpoints per Service: Each service instance hosts 100 endpoints. Total Endpoints: With 20 services, the total number of endpoints is 2000. Trace and Segment Generation Trace Generation Rate: Each group of services generates 1000 traces per second, effectively simulating a high-load scenario typical in large-scale microservice environments. Spans per Trace: Each trace comprises five segments, detailing the simulated interactions between various services and instances. Total Writes per Second: 2 groups * 3 data-generators * 1000 traces * 5 segments = 30k segments. The additional query types were designed to represent typical read operations performed in a production environment monitoring microservice architectures by SkyWalking. Each query type targets a different aspect of service data:\n1. Service Dashboard Queries Purpose: Fetch 5 service-level metrics over the last 30 minutes. Frequency: 1 query per second 2. Top-N List Queries Purpose: Retrieve the top 5 metrics for 2 specific services over the last 30 minutes. Frequency: 1 query per second. 3. Segment List Queries Purpose: Fetch a list of service segments ordered by descending latency within the last 30 minutes. Frequency: 1 query per second. 4. Trace Detail Queries Purpose: Retrieve all trace details from the segment list. Frequency: 2 queries per second. Setup Below is a detailed table that outlines the specifications of each component within the clusters, allowing for an easy comparison of hardware resources allocated to each system. This provides a clear and structured comparison of the deployment configurations used for Elasticsearch 8.13.2 and BanyanDB v0.6.1 in our performance tests.\nPerformance Test Setup\nCluster Configuration Table Component System Quantity CPU Cores RAM (GB) Storage (GB) Role Description Master Nodes Elasticsearch 3 2 6 N/A Cluster coordination and management Data Nodes Elasticsearch 3 4 8 50 (Premium RWO) Data storage, indexing, and query processing ETCD Nodes BanyanDB 3 2 4 N/A Metadata and cluster state storage Data Nodes BanyanDB 3 8 4 50 (Premium RWO) Data storage and processing Liaison Nodes BanyanDB 2 4 4 N/A Coordination between client applications and data nodes Result Here we consolidate the performance test results for Elasticsearch 8.13.2 and BanyanDB v0.6.1, focusing on resource usage comparisons. The results are organized into two tables for better clarity—one detailing CPU and memory usage, and the other focusing on disk-related metrics.\nCPU and Memory Usage System Mean CPU Usage (cores) Mean Memory Usage (MB) Elasticsearch Data 3.2 4147 BanyanDB Data 3.6 738 BanyanDB Liaison 1.9 62 Observations: CPU Usage: BanyanDB data nodes have slightly higher CPU usage due to their operations on compressing and decompressing data files. However, BanyanDB liaison nodes use significantly less CPU. Memory Usage: BanyanDB shows markedly lower memory usage for both data and liaison nodes, using nearly 5x less memory than Elasticsearch data nodes, highlighting its efficiency in memory utilization. Disk Usage, IOPS, and Throughput Disk Usage\nDisk IOPS \u0026amp; Throughput\nSystem Mean Disk Usage (GB) IOPS (k) Disk Throughput (GB/s) Elasticsearch Data 29.6 115.5 12.8 BanyanDB Data 21.6 21.4 3.3 Observations: Disk Space Usage: BanyanDB utilizes about 30% less disk space than Elasticsearch, which can translate into lower storage costs. IOPS and Throughput: BanyanDB\u0026rsquo;s IOPS and disk throughput are significantly lower, indicating less strain on disk resources. This could be beneficial for reducing operational costs and extending the lifespan of physical storage devices. Conclusion The release of BanyanDB v0.6 marks a significant advancement in database technology with its new column-based file system. This version demonstrates substantial improvements in both performance and efficiency, particularly in memory usage and disk space compared to Elasticsearch. BanyanDB\u0026rsquo;s ability to handle various data types, its scalable architecture, and its high performance in data retrieval and ingestion make it a robust solution for complex data environments. The introduction of a flexible clustering system allows for independent scaling of node types, ensuring adaptability to changing demands without compromising on performance or availability. Overall, BanyanDB v0.6 positions itself as a cost-effective and reliable choice for modern application performance management (APM) systems.\n","excerpt":"\u003ch1 id=\"introduction\"\u003eIntroduction\u003c/h1\u003e\n\u003cp\u003eWe are excited to announce the release of BanyanDB v0.6, a significant milestone in the …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2024-06-04-banyandb-0.6-release/","title":"BanyanDB 0.6 Release: Enhanced Performance and Efficiency"},{"body":"Within 3 years (as of April 2021) of incubation, BanyanDB has progressed beyond the alpha stage and officially become the first-class database option for Apache SkyWalking.\nHere is the official statement from the Apache SkyWalking committee about this new option:\nThis is recommended for medium-scale deployments from version 0.6 to 1.0. BanyanDB is set to be our next-generation storage solution. It has demonstrated significant potential in performance improvement. As of version 0.6.1, it achieves 5x less memory usage, 1/5 disk IOPS, 1/4 disk throughput, and 30% less disk space, albeit with a slightly higher CPU trade-off.\nToday, on May 31st, 2024, the SkyWalking demo hosted on our website has officially switched to BanyanDB. Both the native UI and Grafana UI of the site are now reading telemetry data from BanyanDB version 0.6.1.\n","excerpt":"\u003cp\u003eWithin 3 years (as of April 2021) of incubation, BanyanDB has progressed beyond the alpha stage and …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/banyandb-on-demo-skywalking/","title":"BanyanDB is up and running on SkyWalking demo"},{"body":"SkyWalking 10.0.1 is released. Go to downloads page to find release tars. This release targets to collebarate with BanyanDB 0.6.1.\nProject Add SBOM (Software Bill of Materials) to the project. OAP Server Fix LAL test query api. Add component libraries of Derby/Sybase/SQLite/DB2/OceanBase jdbc driver. Fix setting the wrong interval to day level measure schema in BanyanDB installation process. UI Fix widget title and tips. Fix statistics span data. Fix browser log display. Fix the topology layout for there are multiple independent network relationships. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking 10.0.1 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nThis release targets to …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-10.0.1/","title":"Release Apache SkyWalking APM 10.0.1"},{"body":"SkyWalking BanyanDB 0.6.1 is released. Go to downloads page to find release tars.\nFeatures Add benchmarks for stream filtering and sorting. Limit the max pre-calculation result flush interval to 1 minute. Use both datapoint timestamp and server time to trigger the flush of topN pre-calculation result. Bugs Fix the bug that topN query doesn\u0026rsquo;t return when an error occurs. Data race in the hot series index selection. Remove SetSchema from measure cache which could change the schema in the cache. Fix duplicated items in the query aggregation top-n list. Fix non-\u0026ldquo;value\u0026rdquo; field in topN pre-calculation result measure is lack of data. Encode escaped characters to int64 bytes to fix the malformed data. ","excerpt":"\u003cp\u003eSkyWalking BanyanDB 0.6.1 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"features\"\u003eFeatures\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eAdd …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-0-6-1/","title":"Release Apache SkyWalking BanyanDB 0.6.1"},{"body":"MQE是SkyWalking的核心metrics查询服务引擎，自从SkyWalking v9.5.0开始支持MQE以来，MQE展现了其强大灵活和便利性，丰富了metrics的查询方式，随后在v9.6.0 中告警规则配置也全面使用MQE， 让告警规则配置更简单并且可以设定更复杂的条件和计算。在SkyWalking v10以后，官方UI已经全面迁移至使用MQE与OAP进行交互。\n本次直播由SkyWalking PMC 万凯为大家介绍 SkyWalking Metrics Query Expression(MQE)，主要包含以下几部分内容：\n什么是MQE，它的基本原理和在SkyWalking中的架构 MQE的基本语法结构和函数 MQE在查询中的应用 MQE在告警规则中的应用 B站视频地址\n","excerpt":"\u003cp\u003eMQE是SkyWalking的核心metrics查询服务引擎，自从SkyWalking v9.5.0开始支持MQE以来，MQE展现了其强大灵活和便利性，丰富了metrics的查询方式，随后在 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2024-05-30-skywalking-in-practice-s01e05/","title":"SkyWalking从入门到精通 - 2024系列线上分享活动（第五讲）"},{"body":"SkyWalking Kubernetes Helm Chart 4.6.0 is released. Go to downloads page to find release tars.\nIntegrate BanyanDB as storage solution. Bump up swck to v0.9.0. Bump up BanyanDB Helm version to 0.2.0. Bump up OAP and UI to 10.0.0. Make release process work with Linux. ","excerpt":"\u003cp\u003eSkyWalking Kubernetes Helm Chart 4.6.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars. …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-kubernetes-helm-chart-4.6.0/","title":"Release Apache SkyWalking Kubernetes Helm Chart 4.6.0"},{"body":"Chen Ziyan(GitHub ID, CzyerChen) did a lot of contributions since the end of 2023, and continued in 2024. Mainly focus on OAP monitoring, Java agent plugin development, UI bug fixing, blog writing, and showcase updates.\nShe is a designer and developer of data management platforms and operation management platforms as her daily work, and have chance to use a variety of open source big data analysis, operation and maintenance and monitoring components in projects, including SkyWalking.\nHere are her contribution list.\nskywalking Add component libraries of Derby/Sybase/SQLite/DB2/OceanBase jdbc driver.(https://github.com/apache/skywalking/pull/12208) Add C3P0 component libraries.(https://github.com/apache/skywalking/pull/12162) Add ActiveMQ classic monitoring.(https://github.com/apache/skywalking/pull/12109) Fix typos of the example in metrics-query-expression.md#AggregateLabels Operation.(https://github.com/apache/skywalking/pull/11997) Update opentelemetry-receiver.md for ClickHouse.(https://github.com/apache/skywalking/pull/11995) Support ClickHouse server monitoring and service hierarchy.(https://github.com/apache/skywalking/pull/11966) Fix day-based table rolling time range strategy in JDBC storage.(https://github.com/apache/skywalking/pull/11915) Fix table exists check in the JDBC Storage Plugin.(https://github.com/apache/skywalking/pull/11897) Fix log query by traceId in JDBCLogQueryDAO.(https://github.com/apache/skywalking/pull/11764) Fix typos in dynamic-config-configmap.md doc.(https://github.com/apache/skywalking/pull/11212) skywalking-java Fix typos in URLParser.(https://github.com/apache/skywalking-java/pull/686) Add support for C3P0 connection pool tracing.(https://github.com/apache/skywalking-java/pull/683) Add support for ActiveMQ-Artemis messaging tracing.(https://github.com/apache/skywalking-java/pull/670) Add support tracing for async producing, batch sync consuming, and batch async consuming in rocketMQ-client-java-5.x-plugin.(https://- github.com/apache/skywalking-java/pull/665) Add support for HttpExchange tracing along with webflux-webclient-6.x.(https://github.com/apache/skywalking-java/pull/664) Add support for webflux-6.x and gateway-4.x tracing.(https://github.com/apache/skywalking-java/pull/661) Fix method incompatible in mvc-annotation-commons.(https://github.com/apache/skywalking-java/pull/658) Fix PostgreSQL Jdbc URL parsing exception.(https://github.com/apache/skywalking-java/pull/649) Fix Impala Jdbc URL (including schema without properties) parsing exception.(https://github.com/apache/skywalking-java/pull/644) skywalking-booster-ui Optimize search reset and format in marketplace.(https://github.com/apache/skywalking-booster-ui/pull/392) Upgrade img of ActiveMQ.(https://github.com/apache/skywalking-booster-ui/pull/388) Fix the dev local port.(https://github.com/apache/skywalking-booster-ui/pull/375) skywalking-website ActiveMQ monitoring bilingual blog.(https://github.com/apache/skywalking-website/pull/700) ClickHouse monitoring bilingual blog.(https://github.com/apache/skywalking-website/pull/685) skywalking-showcase ActiveMQ monitor showcase.(https://github.com/apache/skywalking-showcase/pull/170) At May. 15th, 2020, the project management committee(PMC) passed the proposal of promoting her as a new committer. She has accepted the invitation at the same day.\nSkyWalking team is honored to have her in the committer team. Welcome!\n","excerpt":"\u003cp\u003eChen Ziyan(GitHub ID, CzyerChen) did a lot of contributions since the end of 2023, and continued in …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-claire-chen-new-committer/","title":"Welcome Claire Chen as new committer"},{"body":"本次直播是 Apache SkyWalking 社区和纵目联合举办分享活动的第四讲，由张跃骎为大家展示 SkyWalking 监控 MySQL Server，主要包含以下几部分内容：\n理清数据流向；MySQL 数据库的指标是如何一步一步导向 SkyWalking 的。 简单介绍 MAL；SkyWalking 如何利用领域特定语言对数据处理。 简单介绍 SkyWalking UI；讲解如何打造独属于自己的监控面板。 B站视频地址\n环境准备整理如下：\n环境准备 MySQL Server 如果读者没有直接可使用的 MySQL Server，可以使用 Docker 通过以下命令直接创建：\ndocker run -e MYSQL_ROOT_PASSWORD=password -p 3306:3306 -d mysql:latest 关于账号，创建命令如下(即请确保具有以下权限)：\nCREATE USER \u0026#39;exporter\u0026#39;@\u0026#39;localhost\u0026#39; IDENTIFIED BY \u0026#39;XXXXXXXX\u0026#39; WITH MAX_USER_CONNECTIONS 3; GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO \u0026#39;exporter\u0026#39;@\u0026#39;localhost\u0026#39;; MySQL Exporter 安装 指标导出工具最简单的部署方式是通过 Docker，命令如下：\ndocker run -d \\ -p 9104:9104 \\ -e \u0026#34;DATA_SOURCE_NAME=root:password@(host.docker.internal:3306)/\u0026#34; \\ prom/mysqld-exporter:v0.14.0 请注意，命令中的 DATA_SOURCE_NAME 部分应该按照实际情况更换。\nOTEL Collctor 由于笔者使用的是 macos 系统，而部分版本的 OpenTelemetry Collector 在 docker desktop 上调试会有 bug，因此演示使用本地安装方式，具体安装方式请参考：OpenTelemetry Collector安装。\n安装好后，我们可以执行命令开始运行：\n./otelcol --config=test.conf 配置文件，请参考：OTEL Collector 配置。\n","excerpt":"\u003cp\u003e本次直播是 Apache SkyWalking 社区和纵目联合举办分享活动的第四讲，由张跃骎为大家展示 SkyWalking 监控 MySQL Server，主要包含以下几部分内容：\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e理清数据流向 …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/zh/2024-05-23-skywalking-in-practice-s01e04/","title":"SkyWalking从入门到精通 - 2024系列线上分享活动（第四讲）"},{"body":"SkyWalking BanyanDB Helm 0.2.0 is released. Go to downloads page to find release tars.\nFeatures Support banyandb cluster mode Add e2e test to CI Chores Update relevant documents Bump banyandb image version to 0.6.0 Bump several dependencies in e2e test ","excerpt":"\u003cp\u003eSkyWalking BanyanDB Helm 0.2.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures …\u003c/h4\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-helm-0-2-0/","title":"Release Apache SkyWalking BanyanDB Helm 0.2.0"},{"body":"SkyWalking 10.0.0 is released. Go to downloads page to find release tars.\nService Hierarchy Service Hierarchy Hierarchy Graph Run with BanyanDB 0.6 in the Cluster Mode Project Support Java 21 runtime. Support oap-java21 image for Java 21 runtime. Upgrade OTEL collector version to 0.92.0 in all e2e tests. Switch CI macOS runner to m1. Upgrade PostgreSQL driver to 42.4.4 to fix CVE-2024-1597. Remove CLI(swctl) from the image. Remove CLI_VERSION variable from Makefile build. Add BanyanDB to docker-compose quickstart. Bump up Armeria, jackson, netty, jetcd and grpc to fix CVEs. Bump up BanyanDB Java Client to 0.6.0. OAP Server Add layer parameter to the global topology graphQL query. Add is_present function in MQE for check if the list metrics has a value or not. Remove unreasonable default configurations for gRPC thread executor. Remove gRPCThreadPoolQueueSize (SW_RECEIVER_GRPC_POOL_QUEUE_SIZE) configuration. Allow excluding ServiceEntries in some namespaces when looking up ServiceEntries as a final resolution method of service metadata. Set up the length of source and dest IDs in relation entities of service, instance, endpoint, and process to 250(was 200). Support build Service/Instance Hierarchy and query. Change the string field in Elasticsearch storage from keyword type to text type if it set more than 32766 length. [Break Change] Change the configuration field of ui_template and ui_menu in Elasticsearch storage from keyword type to text. Support Service Hierarchy auto matching, add auto matching layer relationships (upper -\u0026gt; lower) as following: MESH -\u0026gt; MESH_DP MESH -\u0026gt; K8S_SERVICE MESH_DP -\u0026gt; K8S_SERVICE GENERAL -\u0026gt; K8S_SERVICE Add namespace suffix for K8S_SERVICE_NAME_RULE/ISTIO_SERVICE_NAME_RULE and metadata-service-mapping.yaml as default. Allow using a dedicated port for ALS receiver. Fix log query by traceId in JDBCLogQueryDAO. Support handler eBPF access log protocol. Fix SumPerMinFunctionTest error function. Remove unnecessary annotations and functions from Meter Functions. Add max and min functions for MAL down sampling. Fix critical bug of uncontrolled memory cost of TopN statistics. Change topN group key from StorageId to entityId + timeBucket. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: MYSQL -\u0026gt; K8S_SERVICE POSTGRESQL -\u0026gt; K8S_SERVICE SO11Y_OAP -\u0026gt; K8S_SERVICE VIRTUAL_DATABASE -\u0026gt; MYSQL VIRTUAL_DATABASE -\u0026gt; POSTGRESQL Add Golang as a supported language for AMQP. Support available layers of service in the topology. Add count aggregation function for MAL Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: NGINX -\u0026gt; K8S_SERVICE APISIX -\u0026gt; K8S_SERVICE GENERAL -\u0026gt; APISIX Add Golang as a supported language for RocketMQ. Support Apache RocketMQ server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: ROCKETMQ -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; ROCKETMQ Fix ServiceInstance in query. Mock /api/v1/status/buildinfo for PromQL API. Fix table exists check in the JDBC Storage Plugin. Fix day-based table rolling time range strategy in JDBC storage. Add maxInboundMessageSize (SW_DCS_MAX_INBOUND_MESSAGE_SIZE) configuration to change the max inbound message size of DCS. Fix Service Layer when building Events in the EventHookCallback. Add Golang as a supported language for Pulsar. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: RABBITMQ -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; RABBITMQ Remove Column#function mechanism in the kernel. Make query readMetricValue always return the average value of the duration. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: KAFKA -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; KAFKA Support ClickHouse server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: CLICKHOUSE -\u0026gt; K8S_SERVICE VIRTUAL_DATABASE -\u0026gt; CLICKHOUSE Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: PULSAR -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; PULSAR Add Golang as a supported language for Kafka. Support displaying the port services listen to from OAP and UI during server start. Refactor data-generator to support generating metrics. Fix AvgHistogramPercentileFunction legacy name. [Break Change] Labeled Metrics support multiple labels. Storage: store all label names and values instead of only the values. MQE: Support querying by multiple labels(name and value) instead using _ as the anonymous label name. aggregate_labels function support aggregate by specific labels. relabels function require target label and rename label name and value. PromQL: Support querying by multiple labels(name and value) instead using lables as the anonymous label name. Remove general labels labels/relabels/label function. API /api/v1/labels and /api/v1/label/\u0026lt;label_name\u0026gt;/values support return matched metrics labels. OAL: Deprecate percentile function and introduce percentile2 function instead. Bump up Kafka to fix CVE. Fix NullPointerException in Istio ServiceEntry registry. Remove unnecessary componentIds as series ID in the ServiceRelationClientSideMetrics and ServiceRelationServerSideMetrics entities. Fix not throw error when part of expression not matched any expression node in the MQE and `PromQL. Remove kafka-fetcher/default/createTopicIfNotExist as the creation is automatically since #7326 (v8.7.0). Fix inaccuracy nginx service metrics. Fix/Change Windows metrics name(Swap -\u0026gt; Virtual Memory) memory_swap_free -\u0026gt; memory_virtual_memory_free memory_swap_total -\u0026gt; memory_virtual_memory_total memory_swap_percentage -\u0026gt; memory_virtual_memory_percentage Fix/Change UI init setting for Windows Swap -\u0026gt; Virtual Memory Fix Memory Swap Usage/Virtual Memory Usage display with UI init.(Linux/Windows) Fix inaccurate APISIX metrics. Fix inaccurate MongoDB Metrics. Support Apache ActiveMQ server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: ACTIVEMQ -\u0026gt; K8S_SERVICE Calculate Nginx service HTTP Latency by MQE. MQE query: make metadata not return null. MQE labeled metrics Binary Operation: return empty value if the labels not match rather than report error. Fix inaccurate Hierarchy of RabbitMQ Server monitoring metrics. Fix inaccurate MySQL/MariaDB, Redis, PostgreSQL metrics. Support DoubleValue,IntValue,BoolValue in OTEL metrics attributes. [Break Change] gGRPC metrics exporter unified the metric value type and support labeled metrics. Add component definition(ID=152) for c3p0(JDBC3 Connection and Statement Pooling). Fix MQE top_n global query. Fix inaccurate Pulsar and Bookkeeper metrics. MQE support sort_values and sort_label_values functions. UI Fix the mismatch between the unit and calculation of the \u0026ldquo;Network Bandwidth Usage\u0026rdquo; widget in Linux-Service Dashboard. Add theme change animation. Implement the Service and Instance hierarchy topology. Support Tabs in the widget visible when MQE expressions. Support search on Marketplace. Fix default route. Fix layout on the Log widget. Fix Trace associates with Log widget. Add isDefault to the dashboard configuration. Add expressions to dashboard configurations on the dashboard list page. Update Kubernetes related UI templates for adapt data from eBPF access log. Fix dashboard K8S-Service-Root metrics expression. Add dashboards for Service/Instance Hierarchy. Fix MQE in dashboards when using Card widget. Optimize tooltips style. Fix resizing window causes the trace graph to display incorrectly. Add the not found page(404). Enhance VNode logic and support multiple Trace IDs in span\u0026rsquo;s ref. Add the layers filed and associate layers dashboards for the service topology nodes. Fix Nginx-Instance metrics to instance level. Update tabs of the Kubernetes service page. Add Airflow menu i18n. Add Support for dragging in the trace panel. Add workflow icon. Metrics support multiple labels. Support the SINGLE_VALUE for table widgets. Remove the General metric mode and related logical code. Remove metrics for unreal nodes in the topology. Enhance the Trace widget for batch consuming spans. Clean the unused elements in the UI-templates. Documentation Update the release doc to remove the announcement as the tests are through e2e rather than manually. Update the release notification mail a little. Polish docs structure. Move customization docs separately from the introduction docs. Add webhook/gRPC hooks settings example for backend-alarm.md. Begin the process of SWIP - SkyWalking Improvement Proposal. Add SWIP-1 Create and detect Service Hierarchy Relationship. Add SWIP-2 Collecting and Gathering Kubernetes Monitoring Data. Update the Overview docs to add the Service Hierarchy Relationship section. Fix incorrect words for backend-bookkeeper-monitoring.md and backend-pulsar-monitoring.md Document a new way to load balance OAP. Add SWIP-3 Support RocketMQ monitoring. Add OpenTelemetry SkyWalking Exporter deprecated warning doc. Update i18n for rocketmq monitoring. Fix: remove click event after unmounted. Fix: end loading without query results. Update nanoid version to 3.3.7. Update postcss version to 8.4.33. Fix kafka topic name in exporter doc. Fix query-protocol.md, make it consistent with the GraphQL query protocol. Add SWIP-5 Support ClickHouse Monitoring. Remove OpenTelemetry Exporter support from meter doc, as this has been flagged as unmaintained on OTEL upstream. Add doc of one-line quick start script for different storage types. Add FAQ for Why is Clickhouse or Loki or xxx not supported as a storage option?. Add SWIP-8 Support ActiveMQ Monitoring. Move BanyanDB storage to the recommended storage. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking 10.0.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"service-hierarchy\"\u003eService Hierarchy …\u003c/h4\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-10.0.0/","title":"Release Apache SkyWalking APM 10.0.0"},{"body":"SkyWalking BanyanDB 0.6.0 is released. Go to downloads page to find release tars.\nFeatures Support etcd client authentication. Implement Local file system. Add health check command for bydbctl. Implement Inverted Index for SeriesDatabase. Remove Block Level from TSDB. Remove primary index. Measure column-based storage: Data ingestion and retrieval. Flush memory data to disk. Merge memory data and disk data. Stream column-based storage: Data ingestion and retrieval. Flush memory data to disk. Merge memory data and disk data. Add HTTP services to TopNAggregation operations. Add preload for the TopN query of index. Remove \u0026ldquo;TREE\u0026rdquo; index type. The \u0026ldquo;TREE\u0026rdquo; index type is merged into \u0026ldquo;INVERTED\u0026rdquo; index type. Remove \u0026ldquo;Location\u0026rdquo; field on IndexRule. Currently, the location of index is in a segment. Remove \u0026ldquo;BlockInterval\u0026rdquo; from Group. The block size is determined by the part. Support querying multiple groups in one request. Bugs Fix the bug that property merge new tags failed. Fix CPU Spike and Extended Duration in BanyanDB\u0026rsquo;s etcd Watching Registration Process. Fix panic when closing banyand. Fix NPE when no index filter in the query. Chores Bump go to 1.22. Bump node to 2.12.2. Bump several tools. Bump all dependencies of Go and Node. Combine banyand and bydbctl Dockerfile. Update readme for bydbctl ","excerpt":"\u003cp\u003eSkyWalking BanyanDB 0.6.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"features\"\u003eFeatures\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eSupport …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-0-6-0/","title":"Release Apache SkyWalking BanyanDB 0.6.0"},{"body":"The Apache SkyWalking team today announced the 10 release. SkyWalking 10 provides a host of groundbreaking features and enhancements. The introduction of Layer and Service Hierarchy streamlines monitoring by organizing services and metrics into distinct layers and providing seamless navigation across them. Leveraging eBPF technology, Kubernetes Network Monitoring delivers granular insights into network traffic, topology, and TCP/HTTP metrics. BanyanDB emerges as a high-performance native storage solution, while expanded monitoring support encompasses Apache RocketMQ, ClickHouse, and Apache ActiveMQ Classic. Support for Multiple Labels Names enhances flexibility in metrics analysis, while enhanced exporting and querying capabilities streamline data dissemination and processing.\nThis release blog briefly introduces these new features and enhancements as well as some other notable changes.\nLayer and Service Hierarchy Layer concept was introduced in SkyWalking 9.0.0, it represents an abstract framework in computer science, such as Operating System(OS_LINUX layer), Kubernetes(k8s layer). It organizes services and metrics into different layers based on their roles and responsibilities in the system. SkyWalking provides a suite of monitoring and diagnostic tools for each layer, but there is a gap between the layers, which can not easily bridge the data across different layers.\nIn SkyWalking 10, SkyWalking provides new abilities to jump/connect across different layers and provide a seamless monitoring experience for users.\nLayer Jump In the topology graph, users can click on a service node to jump to the dashboard of the service in another layer. The following figures show the jump from the GENERAL layer service topology to the VIRTUAL_DATABASE service layer dashboard by clicking the topology node. Figure 1: Layer Jump\nFigure 2: Layer jump Dashboard\nService Hierarchy SkyWalking 10 introduces a new concept called Service Hierarchy, which defines the relationships of existing logically same services in various layers. OAP will detect the services from different layers, and try to build the connections. Users can click the Hierarchy Services in any layer\u0026rsquo;s service topology node or service dashboard to get the Hierarchy Topology. In this topology graph, users can see the relationships between the services in different layers and the summary of the metrics and also can jump to the service dashboard in the layer. When a service occurs performance issue, users can easily analyze the metrics from different layers and track down the root cause:\nThe examples of the Service Hierarchy relationships:\nThe application song deployed in the Kubernetes cluster with SkyWalking agent and Service Mesh at the same time. So the application song across the GENERAL, MESH, MESH_DP and K8S_SERVICE layers which could be monitored by SkyWalking, the Service Hierarchy topology as below: Figure 3: Service Hierarchy Agent With K8s Service And Mesh With K8s Service. And can also have the Service Instance Hierarchy topology to get the single instance status across the layers as below: Figure 4: Instance Hierarchy Agent With K8s Service(Pod) The PostgreSQL database psql deployed in the Kubernetes cluster and used by the application song. So the database psql across the VIRTUAL_DATABASE, POSTGRESQL and K8S_SERVICE layers which could be monitored by SkyWalking, the Service Hierarchy topology as below: Figure 5: Service Hierarchy Agent(Virtual Database) With Real Database And K8s Service For more supported layers and how to detect the relationships between services in different layers please refer to the Service Hierarchy. how to configure the Service Hierarchy in SkyWalking, please refer to the Service Hierarchy Configuration section.\nMonitoring Kubernetes Network Traffic by using eBPF In the previous version, skyWalking provides Kubernetes (K8s) monitoring from kube-state-metrics and cAdvisor, which can monitor the Kubernetes cluster status and the metrics of the Kubernetes resources.\nIn SkyWalking 10, by leverage Apache SkyWalking Rover 0.6+, SkyWalking has the ability to monitor the Kubernetes network traffic by using eBPF, which can collect and map access logs from applications in Kubernetes environments. Through these data, SkyWalking can analyze and provide the Service Traffic, Topology, TCP/HTTP level metrics from the Kubernetes aspect.\nThe following figures show the Topology and TCP Dashboard of the Kubernetes network traffic:\nFigure 6: Kubernetes Network Traffic Topology\nFigure 7: Kubernetes Network Traffic TCP Dashboard\nMore details about how to monitor the Kubernetes network traffic by using eBPF, please refer to the Monitoring Kubernetes Network Traffic by using eBPF.\nBanyanDB - Native APM Database BanyanDB 0.6.0 and BanyanDB Java client 0.6.0 are released with SkyWalking 10, As a native storage solution for SkyWalking, BanyanDB is going to be SkyWalking\u0026rsquo;s next-generation storage solution. This is recommended to use for medium-scale deployments from 0.6 until 1.0.\nIt has shown high potential performance improvement. Less than 50% CPU usage and 50% memory usage with 40% disk volume compared to Elasticsearch in the same scale.\nApache RocketMQ Server Monitoring Apache RocketMQ is an open-source distributed messaging and streaming platform, which is widely used in various scenarios including Internet, big data, mobile Internet, IoT, and other fields. SkyWalking provides a basic monitoring dashboard for RocketMQ, which includes the following metrics:\nCluster Metrics: including messages produced/consumed today, total producer/consumer TPS, producer/consumer message size, messages produced/consumed until yesterday, max consumer latency, max commitLog disk ratio, commitLog disk ratio, pull/send threadPool queue head wait time, topic count, and broker count. Broker Metrics: including produce/consume TPS, producer/consumer message size. Topic Metrics: including max producer/consumer message size, consumer latency, producer/consumer TPS, producer/consumer offset, producer/consumer message size, consumer group count, and broker count. The following figure shows the RocketMQ Cluster Metrics dashboard: Figure 8: Apache RocketMQ Server Monitoring\nFor more metrics and details about the RocketMQ monitoring, please refer to the Apache RocketMQ Server Monitoring,\nClickHouse Server Monitoring ClickHouse is an open-source column-oriented database management system that allows generating analytical data reports in real-time, it is widely used for online analytical processing (OLAP). ClickHouse monitoring provides monitoring of the metrics 、events and asynchronous metrics of the ClickHouse server, which includes the following parts of metrics:\nServer Metrics Query Metrics Network Metrics Insert Metrics Replica Metrics MergeTree Metrics ZooKeeper Metrics Embedded ClickHouse Keeper Metrics The following figure shows the ClickHouse Cluster Metrics dashboard: Figure 9: ClickHouse Server Monitoring\nFor more metrics and details about the ClickHouse monitoring, please refer to the ClickHouse Server Monitoring, and here is a blog that can help for a quick start Monitoring ClickHouse through SkyWalking.\nApache ActiveMQ Server Monitoring Apache ActiveMQ Classic is a popular and powerful open-source messaging and integration pattern server. SkyWalking provides a basic monitoring dashboard for ActiveMQ, which includes the following metrics:\nCluster Metrics: including memory usage, rates of write/read, and average/max duration of write. Broker Metrics: including node state, number of connections, number of producers/consumers, and rate of write/read under the broker. Depending on the cluster mode, one cluster may include one or more brokers. Destination Metrics: including number of producers/consumers, messages in different states, queues, and enqueue duration in a queue/topic. The following figure shows the ActiveMQ Cluster Metrics dashboard: Figure 10: Apache ActiveMQ Server Monitoring\nFor more metrics and details about the ActiveMQ monitoring, please refer to the Apache ActiveMQ Server Monitoring, and here is a blog that can help for a quick start Monitoring ActiveMQ through SkyWalking.\nSupport Multiple Labels Names Before SkyWalking 10, SkyWalking does not store the labels names in the metrics data, which makes MQE have to use _ as the generic label name, it can\u0026rsquo;t query the metrics data with multiple labels names.\nSkyWalking 10 supports storing the labels names in the metrics data, and MQE can query or calculate the metrics data with multiple labels names. For example: The k8s_cluster_deployment_status metric has labels namespace, deployment and status. If we want to query all deployment metric values with namespace=skywalking-showcase and status=true, we can use the following expression:\nk8s_cluster_deployment_status{namespace=\u0026#39;skywalking-showcase\u0026#39;, status=\u0026#39;true\u0026#39;} related enhancement:\nSince Alarm rule configuration had migrated to the MQE in SkyWalking 9.6.0, the alarm rule also supports multiple labels names. PromeQL service supports multiple labels names query. Metrics gRPC exporter SkyWalking 10 enhanced the metrics gPRC exporter, it supports exporting all types of metrics data to the gRPC server.\nSkyWalking Native UI Metrics Query Switch to V3 APIs SkyWalking Native UI metrics query deprecate the V2 APIs, and all migrated to V3 APIs and MQE.\nOther Notable Enhancements Support Java 21 runtime and oap-java21 image for Java 21 runtime. Remove CLI(swctl) from the image. More MQE functions and operators supported. Enhance the native UI and improve the user experience. Several bugs and CVEs fixed. ","excerpt":"\u003cp\u003eThe Apache SkyWalking team today announced the 10 release. SkyWalking 10 provides a host of …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2024-05-13-skywalking-10-release/","title":"SkyWalking 10 Release: Service Hierarchy, Kubernetes Network Monitoring by eBPF, BanyanDB, and More"},{"body":"Apache SkyWalking 团队今天宣布发布 SkyWalking 10。SkyWalking 10 提供了一系列突破性的功能和增强功能。Layer 和 Service Hierarchy 的引入通过将服务和指标组织成不同的层次，并提供跨层无缝导航，从而简化了监控。利用 eBPF 技术，Kubernetes 网络监控提供了对网络流量、拓扑和 TCP/HTTP 指标的详细洞察。BanyanDB 作为高性能的原生存储解决方案出现，同时扩展的监控支持包括 Apache RocketMQ、ClickHouse 和 Apache ActiveMQ Classic。对多标签名称的支持增强了指标分析的灵活性，而增强的导出和查询功能简化了数据分发和处理。\n本文简要介绍了这些新功能和增强功能以及其他一些值得注意的变化。\nLayer 和 Service Hierarchy Layer 概念是在 SkyWalking 9.0.0 中引入的，它代表计算机科学中的一个抽象框架，例如操作系统（OS_LINUX layer）、Kubernetes（k8s layer）。它根据系统中服务和指标的角色和职责将其组织到不同的层次。SkyWalking 为每个层提供了一套监控和诊断工具，但层之间存在 gap，无法轻松跨层桥接数据。\n在 SkyWalking 10 中，SkyWalking 提供了跨层跳转/连接的新功能，为用户提供无缝的监控体验。\nLayer Jump 在拓扑图中，用户可以点击服务节点跳转到另一层服务的仪表板。下图显示了通过点击拓扑节点从 GENERAL 层服务拓扑跳转到 VIRTUAL_DATABASE 服务层仪表板的过程。\nService Hierarchy SkyWalking 10 引入了一个新概念，称为 Service Hierarchy，它定义了各层中现有逻辑相同服务的关系。OAP 将检测不同层次的服务，并尝试建立连接。用户可以点击任何层的服务拓扑节点或服务仪表板中的 Hierarchy Services 获取 Hierarchy Topology。在此拓扑图中，用户可以看到不同层次服务之间的关系和指标摘要，并且可以跳转到该层的服务仪表板。当服务发生性能问题时，用户可以轻松分析不同层次的指标并找出根本原因：\n以下是 Service Hierarchy 关系的示例：\n应用程序 song 同时在 Kubernetes 集群中部署了 SkyWalking agent 和 Service Mesh。因此，应用程序 song 跨越了 GENERAL、MESH、MESH_DP 和 K8S_SERVICE 层，SkyWalking 可以监控这些层次，Service Hierarchy 拓扑如下： 还可以有 Service Instance Hierarchy 拓扑来获取跨层的单实例状态，如下所示： 在 Kubernetes 集群中部署并由应用程序 song 使用的 PostgreSQL 数据库 psql。因此，数据库 psql 跨越 VIRTUAL_DATABASE、POSTGRESQL 和 K8S_SERVICE 层，SkyWalking 可以监控这些层次，Service Hierarchy 拓扑如下： 有关更多支持的层次以及如何检测不同层次服务之间的关系，请参阅 Service Hierarchy。有关如何在 SkyWalking 中配置 Service Hierarchy，请参阅 Service Hierarchy Configuration 部分。\n使用 eBPF 监控 Kubernetes 网络流量 在之前的版本中，SkyWalking 提供了 来自 kube-state-metrics 和 cAdvisor 的 Kubernetes (K8s) 监控，它可以监控 Kubernetes 集群状态和 Kubernetes 资源的指标。\n在 SkyWalking 10 中，通过利用 Apache SkyWalking Rover 0.6+，SkyWalking 具有使用 eBPF 监控 Kubernetes 网络流量的能力，可以收集和映射 Kubernetes 环境中应用程序的访问日志。通过这些数据，SkyWalking 可以从 Kubernetes 角度分析和提供服务流量、拓扑、TCP/HTTP 级别指标。\n下图显示了 Kubernetes 网络流量的拓扑和 TCP 仪表板：\n有关如何使用 eBPF 监控 Kubernetes 网络流量的更多详细信息，请参阅 使用 eBPF 监控 Kubernetes 网络流量。\nBanyanDB - 原生 APM 数据库 BanyanDB 0.6.0 和 BanyanDB Java 客户端 0.6.0 随 SkyWalking 10 一起发布。作为 SkyWalking 的原生存储解决方案，BanyanDB 将成为 SkyWalking 的下一代存储解决方案。推荐在 0.6 到 1.0 期间用于中等规模的部署。 它展示了高性能改进的潜力。与 Elasticsearch 在同一规模下相比，CPU 使用率降低 50%，内存使用率降低 50%，磁盘使用量减少 40%。\nApache RocketMQ 服务器监控 Apache RocketMQ 是一个开源的分布式消息和流平台，广泛应用于互联网、大数据、移动互联网、物联网等领域。SkyWalking 为 RocketMQ 提供了一个基本的监控仪表板，包括以下指标：\nCluster Metrics：包括当天产生/消费的消息数、总生产者/消费者 TPS、生产者/消费者消息大小、截至昨天产生/消费的消息数、最大消费者延迟、最大 commitLog 磁盘比、commitLog 磁盘比、拉取/发送线程池队列头等待时间、topic count 和 broker count。 Broker Metrics：包括生产/消费 TPS、生产者/消费者消息大小。 Topic Metrics：包括最大生产者/消费者消息大小、消费者延迟、生产/消费 TPS、生产/消费偏移、生产/消费消息大小、消费者组数和代理数。 下图显示了 RocketMQ Cluster Metrics 仪表板：\n有关 RocketMQ 监控的更多指标和详细信息，请参阅 Apache RocketMQ 服务器监控。\nClickHouse Server 监控 ClickHouse 是一个开源的列式数据库管理系统，可以实时生成分析数据报告，广泛用于在线分析处理 (OLAP)。ClickHouse 监控提供了 ClickHouse 服务器的指标、事件和异步指标的监控，包括以下部分的指标：\nServer Metrics Query Metrics Network Metrics Insert Metrics Replica Metrics MergeTree Metrics ZooKeeper Metrics Embedded ClickHouse Keeper Metrics 下图显示了 ClickHouse Cluster Metrics 仪表板：\n有关 ClickHouse 监控的更多指标和详细信息，请参阅 ClickHouse 服务器监控，以及一篇可以帮助快速入门的博客 通过 SkyWalking 监控 ClickHouse。\nApache ActiveMQ 服务器监控 Apache ActiveMQ Classic 是一个流行且强大的开源消息和集成模式服务器。SkyWalking 为 ActiveMQ 提供了一个基本的监控仪表板，包括以下指标：\nCluster Metrics：包括内存使用率、写入/读取速率和平均/最大写入持续时间。 Broker Metrics：包括节点状态、连接数、生产者/消费者数和代理下的写入/读取速率。根据集群模式，一个集群可以包含一个或多个代理。 Destination Metrics：包括生产者/消费者数、不同状态的消息、队列和队列/主题中的入队持续时间。 下图显示了 ActiveMQ Cluster Metrics 仪表板：\n有关 ActiveMQ 监控的更多指标和详细信息，请参阅 Apache ActiveMQ 服务器监控，以及一篇可以帮助快速入门的博客 通过 SkyWalking 监控 ActiveMQ。\n支持多标签名称 在 SkyWalking 10 之前，SkyWalking 不会在指标数据中存储标签名称，这使得 MQE 必须使用 _ 作为通用标签名称，无法使用多个标签名称查询指标数据。\nSkyWalking 10 支持在指标数据中存储标签名称，MQE 可以使用多个标签名称查询或计算指标数据。例如：k8s_cluster_deployment_status 指标具有 namespace、deployment 和 status 标签。如果我们想查询所有 namespace=skywalking-showcase 和 status=true 的部署指标值，可以使用以下表达式：\nk8s_cluster_deployment_status{namespace=\u0026#39;skywalking-showcase\u0026#39;, status=\u0026#39;true\u0026#39;} 相关增强：\n由于在 SkyWalking 9.6.0 中告警规则配置已迁移到 MQE，因此告警规则也支持多标签名称。 PromeQL 服务支持多标签名称查询。 Metrics gRPC 导出器 SkyWalking 10 增强了 metrics gPRC exporter，支持将所有类型的指标数据导出到 gRPC 服务器。\nSkyWalking 原生 UI 指标查询切换到 V3 API SkyWalking 原生 UI 指标查询弃用 V2 API，全部迁移到 V3 API 和 MQE。\n其他值得注意的增强功能 支持 Java 21 运行时和 oap-java21 镜像用于 Java 21 运行时。 从镜像中删除 CLI（swctl）。 支持更多的 MQE 函数和操作。 增强原生 UI 并改善用户体验。 修复了一些 Bug 和 CVE。 ","excerpt":"\u003cp\u003eApache SkyWalking 团队今天宣布发布 SkyWalking 10。SkyWalking 10 提供了一系列突破性的功能和增强功能。Layer 和 Service Hierarchy 的 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2024-05-13-skywalking-10-release/","title":"SkyWalking 10 发布：服务层次结构、基于 eBPF 的 Kubernetes 网络监控、BanyanDB 等"},{"body":"Youliang Huang(GitHub ID, butterbright[1]) began the code contributions since June 9th, 2023.\nUp to date, he has submitted 16 PRs in the BanyanDB repository, 6 PRs in the BanyanDB Helm repo, 4 PR in the SkyWalking Helm repository and 1 PR in the SWCK repository.\nAt Map 7th, 2024, the project management committee(PMC) passed the proposal of promoting him as a new committer. He has accepted the invitation at the same day.\nWelcome Youliang Huang join the committer team.\n[1] https://github.com/butterbright\n","excerpt":"\u003cp\u003eYouliang Huang(GitHub ID, butterbright[1]) began the code contributions since June 9th, 2023.\u003c/p\u003e\n\u003cp\u003eUp to …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-youliang-huang-as-new-committer/","title":"Welcome Youliang Huang as new committer"},{"body":"本次直播是 Apache SkyWalking 社区和纵目联合举办分享活动的第二讲，由魏翔为大家介绍 SkyWalking LAL(Log Analysis Language)，主要包含以下几部分内容：\nSkyWalking LAL(Log Analysis Language) 语法介绍 使用 LAL 监控服务日志异常实验 OAP log-analyzer 模块源码讲解 B站视频地址\n实验中涉及到的知识点比较零散，为了方便大家复现实验结果，现将实验步骤整理如下：\n1. 接入服务日志至SkyWalking 首先，我们启动 demo 服务，并通过一个定时任务模拟异常，并输出异常至日志中，下面的方法会每秒钟执行一次，因为除数为零，所以会产生 java.lang.ArithmeticException: / by zero 的异常：\n@Scheduled(fixedDelay = 1000) public void mockException() throws Exception { int i = 1 / 0; } 2024-04-22 23:03:54 SW_CTX:[gateway,3a96549cb6474607be27e3ce481c2629@198.18.0.1,N/A,N/A,-1] [scheduling-1] ERROR [org.springframework.scheduling.support.TaskUtils$LoggingErrorHandler:95] - Unexpected error occurred in scheduled task java.lang.ArithmeticException: / by zero at com.test.ConsumerApplication.mockException(ConsumerApplication.java:47) at sun.reflect.GeneratedMethodAccessor195.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.scheduling.support.ScheduledMethodRunnable.$sw$original$run$c8tpsq2(ScheduledMethodRunnable.java:84) at org.springframework.scheduling.support.ScheduledMethodRunnable.$sw$original$run$c8tpsq2$accessor$$sw$p2boiv3(ScheduledMethodRunnable.java) at org.springframework.scheduling.support.ScheduledMethodRunnable$$sw$auxiliary$k466ps2.call(Unknown Source) at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86) at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java) 接着，我们为该服务启动参数添加 skywalking agent 启动参数，并接入日志至 skywalking，由于我们demo使用的是logback，我们在 pom.xml 中添加以下依赖：\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.apache.skywalking\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;apm-toolkit-logback-1.x\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;${version}\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; 同时，在logback.xml中添加 skywalking-grpc appender:\n\u0026lt;appender name=\u0026#34;grpc-log\u0026#34; class=\u0026#34;org.apache.skywalking.apm.toolkit.log.logback.v1.x.log.GRPCLogClientAppender\u0026#34;\u0026gt; \u0026lt;encoder class=\u0026#34;ch.qos.logback.core.encoder.LayoutWrappingEncoder\u0026#34;\u0026gt; \u0026lt;layout class=\u0026#34;org.apache.skywalking.apm.toolkit.log.logback.v1.x.mdc.TraceIdMDCPatternLogbackLayout\u0026#34;\u0026gt; \u0026lt;Pattern\u0026gt;%d{yyyy-MM-dd HH:mm:ss.SSS} [%X{tid}] [%thread] %-5level %logger{36} -%msg%n\u0026lt;/Pattern\u0026gt; \u0026lt;/layout\u0026gt; \u0026lt;/encoder\u0026gt; \u0026lt;/appender\u0026gt; 启动 SkyWalking OAP 服务，一切顺利的话，你会在 SkyWalking 日志面板中看到 demo 服务上报的日志信息：\n2. 配置 LAL 解析上报的日志并提取指标 默认情况下，SkyWalking只会保存原始的日志数据，不做任何的处理分析，我们修改 config/lal/default.xml:\nrules: - name: default layer: GENERAL dsl: | filter { text { abortOnFailure false regexp $/(?\u0026lt;time\u0026gt;\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{3}) \\[.+] \\[.+] (?\u0026lt;level\u0026gt;\\w+) (?\u0026lt;msg\u0026gt;.*)/$ } extractor { tag level: parsed.level timestamp parsed.time as String, \u0026#34;yyyy-MM-dd HH:mm:ss.SSS\u0026#34; if (parsed.level == \u0026#34;ERROR\u0026#34;) { metrics { timestamp log.timestamp as Long labels service: log.service, service_instance_id: log.serviceInstance name \u0026#34;log_exception_count\u0026#34; value 1 } } } sink { } } 上面的 dsl 中，首先使用 text regex 解析器解析日志内容，分别解析出了日志的时间、日志等级等信息，大家可以根据需要自行调整 regexp 表达式（如果你的日志是json格式，你也可以尝试json 解析器 ）。\n接着 extractor 会从 regexp 解析结果中，提取出日志额外的 tag 以及 timestamp 信息，并且会检查 level，如果 level 级别为 ERROR，就会生成一个名为log_exception_count，值为 1 的指标，在打上 service 及 service_instance_id 标签后，会交给 skywalking meter system 接着处理。\n3. 定义 log-mal 进一步分析 LAL 中提取的指标 上一步中，我们定义了日志的解析规则，并成功提取到了 log_exception_count 指标，接着我们定义指标分析规则，创建 config/lal-mal/rules/default.yaml:\nmetricPrefix: instance metricsRules: - name: log_exception_count exp: log_exception_count.sum([\u0026#39;service\u0026#39;,\u0026#39;service_instance_id\u0026#39;]).downsampling(SUM).instance([\u0026#39;service\u0026#39;], [\u0026#39;service_instance_id\u0026#39;], Layer.GENERAL) 上面的 mal 中，我们指定 downsampling 函数为 SUM，这样可以帮助我们计算一分钟内的错误数和，由于是新创建的文件，别忘了在 config/application.yml 中注册该配置文件：\nlog-analyzer: selector: ${SW_LOG_ANALYZER:default} default: lalFiles: ${SW_LOG_LAL_FILES:envoy-als,mesh-dp,mysql-slowsql,pgsql-slowsql,redis-slowsql,k8s-service,nginx,default} malFiles: ${SW_LOG_MAL_FILES:\u0026#34;nginx,default\u0026#34;} 最后我们打开 skywalking-ui，在 dashboard 中添加指标 instance_log_exception_count 并验证指标结果正确性:\n4. 配置指标告警规则 有了指标数据后，我们可以在 config/alarm-settings.yml 添加对应的告警规则，该规则定义如果一分钟内日志异常数量超过 5 就会发出告警信息：\ninstance_error_log_rule: expression: sum(instance_log_exception_count \u0026gt; 5) \u0026gt;= 1 period: 1 tags: level: WARNING 配置好以上规则后，我们稍等 1 分钟，便可以在告警记录面板查看到响应的告警信息：\n附：想参与直播的小伙伴，可以关注后续的直播安排和我们的B站直播预约\n","excerpt":"\u003cp\u003e本次直播是 Apache SkyWalking 社区和纵目联合举办分享活动的第二讲，由魏翔为大家介绍 SkyWalking LAL(Log Analysis Language)，主要包含以下几部分内容 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2024-05-09-skywalking-in-practice-s01e02/","title":"SkyWalking从入门到精通 - 2024系列线上分享活动（第二讲）"},{"body":"SkyWalking BanyanDB 0.6.0 is released. Go to downloads page to find release tars.\nFeatures Support JDK21 build. Upgrade lombok version to 1.18.30. Bump up the API of BanyanDB Server. Bugs Fix the number of channel and the channel size in the BulkWriteProcessor. ","excerpt":"\u003cp\u003eSkyWalking BanyanDB 0.6.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-java-client-0-6-0/","title":"Release Apache SkyWalking BanyanDB Java Client 0.6.0"},{"body":"Apache SkyWalking从2015年开源到2024年，已经走过了9个年头，项目的规模和功能也得到了极大的丰富。 2024年4月至6月，SkyWalking社区联合纵目，举办线上的联合直播，分多个主题介绍SkyWalking的核心特性，也提供更多的答疑时间。\n2024年4月25日，SkyWalking创始人带来了第一次分享和Q\u0026amp;A\n熟悉SkyWalking项目结构 介绍项目工程划分，边界，定位 SkyWalking文档使用，以及如何使用AI助手 Q\u0026amp;A B站视频地址\n想参与直播的小伙伴，可以关注后续的直播安排和我们的B站直播预约\n","excerpt":"\u003cp\u003eApache SkyWalking从2015年开源到2024年，已经走过了9个年头，项目的规模和功能也得到了极大的丰富。\n2024年4月至6月，SkyWalking社区联合纵目，举办线上的联合直播，分 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2024-04-26-skywalking-in-practice-s01e01/","title":"SkyWalking从入门到精通 - 2024系列线上分享活动（第一讲）"},{"body":"\nIntroduction Apache ActiveMQ Classic is a popular and powerful open-source messaging and integration pattern server. Founded in 2004, it has evolved into a mature and widely used open-source messaging middleware that complies with the Java Message Service (JMS). Today, with its stability and wide range of feature support, it still has a certain number of users of small and medium-sized enterprises. It‘s high-performance version Apache Artemis is developing rapidly and is also attracting attention from users of ActiveMQ.\nActiveMQ has broad support for JMX (Java Management Extensions), allowing to be monitored through JMX MBean. After enabling JMX, you can use JAVA\u0026rsquo;s built-in jconsole or VisualVM to view the metrics. In addition, some Collector components can also be used to convert JMX-style data into Prometheus-style data, which is suitable for more tools.\nOpenTelemetry as an industry-recognized, standardized solution that provides consistent and interoperable telemetry data collection, transmission, and analysis capabilities for distributed systems, and is also used here for data collection and transmission. Although it can directly accept JMX type data, the JMX indicators for collecting ActiveMQ are not in the standard library, and some versions are incompatible, so this article adopts two steps: convert JMX data into Prometheus-style indicator data, and then use OpenTelemetry to scrape HTTP endpoint data.\nSkyWalking as a one-stop distributed system monitoring solution, it accepts metrics from ActiveMQ and provides a basic monitoring dashboard.\nDeployment Please set up the following services:\nSkyWalking OAP, v10.0+. ActiveMQ v6.0.X+. JMX Exporter v0.20.0. If using docker, refer bitnami/jmx-exporter. OpenTelemetry-Collector v0.92.0. Preparation The following describes how to deploy ActiveMQ with 2 single-node brokers and SkyWalking OAP with one single node. JMX Exporter runs in agent mode (recommended).\nConfiguration Enable JMX in ActiveMQ, the JMX remote port defaults to 1616, you can change it through ACTIVEMQ_SUNJMX_START. Set up the exporter: [Recommended] If run exporter in agent mode, need to append the startup parameter -DACTIVEMQ_OPTS=-javaagent:{activemqPath}/bin/jmx_prometheus_javaagent-0.20.0.jar=2345:{activemqPath}/conf/config.yaml in ActiveMQ env, then exporter server starts at the same time. If run exporter in single server, refer here to deploy the server alone. 2345 is open HTTP port that can be customized. JMX\u0026rsquo;s metrics can be queried through http://localhost:2345/metrics. example of docker-compose.yml with agent exporter for ActiveMQ:\nversion: \u0026#39;3.8\u0026#39; services: amq1: image: apache/activemq-classic:latest container_name: amq1 hostname: amq1 volumes: - ~/activemq1/conf/activemq.xml:/opt/apache-activemq/conf/activemq.xml - ~/activemq1/bin/jmx_prometheus_javaagent-0.20.0.jar:/opt/apache-activemq/bin/jmx_prometheus_javaagent-0.20.0.jar - ~/activemq1/conf/config.yaml:/opt/apache-activemq/conf/config.yaml ports: - \u0026#34;61616:61616\u0026#34; - \u0026#34;8161:8161\u0026#34; - \u0026#34;2345:2345\u0026#34; environment: ACTIVEMQ_OPTS: \u0026#34;-javaagent:/opt/apache-activemq/bin/jmx_prometheus_javaagent-0.20.0.jar=2345:/opt/apache-activemq/conf/config.yaml\u0026#34; ACTIVEMQ_BROKER_NAME: broker-1 networks: - amqtest amq2: image: apache/activemq-classic:latest container_name: amq2 hostname: amq2 volumes: - ~/activemq2/conf/activemq.xml:/opt/apache-activemq/conf/activemq.xml - ~/activemq2/bin/jmx_prometheus_javaagent-0.20.0.jar:/opt/apache-activemq/bin/jmx_prometheus_javaagent-0.20.0.jar - ~/activemq2/conf/config.yaml:/opt/apache-activemq/conf/config.yaml ports: - \u0026#34;61617:61616\u0026#34; - \u0026#34;8162:8161\u0026#34; - \u0026#34;2346:2346\u0026#34; environment: ACTIVEMQ_OPTS: \u0026#34;-javaagent:/opt/apache-activemq/bin/jmx_prometheus_javaagent-0.20.0.jar=2346:/opt/apache-activemq/conf/config.yaml\u0026#34; ACTIVEMQ_BROKER_NAME: broker-2 networks: - amqtest otel-collector1: image: otel/opentelemetry-collector:latest container_name: otel-collector1 command: [ \u0026#34;--config=/etc/otel-collector-config.yaml\u0026#34; ] volumes: - ./otel-collector-config1.yaml:/etc/otel-collector-config.yaml depends_on: - amq1 networks: - amqtest otel-collector2: image: otel/opentelemetry-collector:latest container_name: otel-collector2 command: [ \u0026#34;--config=/etc/otel-collector-config.yaml\u0026#34; ] volumes: - ./otel-collector-config2.yaml:/etc/otel-collector-config.yaml depends_on: - amq2 networks: - amqtest networks: amqtest: example of otel-collector-config.yaml for OpenTelemetry:\nreceivers: prometheus: config: scrape_configs: - job_name: \u0026#39;activemq-monitoring\u0026#39; scrape_interval: 30s static_configs: - targets: [\u0026#39;amq1:2345\u0026#39;] labels: cluster: activemq-broker1 processors: batch: exporters: otlp: endpoint: oap:11800 tls: insecure: true service: pipelines: metrics: receivers: - prometheus processors: - batch exporters: - otlp example of config.yaml for ActiveMQ Exporter:\n--- startDelaySeconds: 10 username: admin password: activemq ssl: false lowercaseOutputName: false lowercaseOutputLabelNames: false includeObjectNames: [\u0026#34;org.apache.activemq:*\u0026#34;,\u0026#34;java.lang:type=OperatingSystem\u0026#34;,\u0026#34;java.lang:type=GarbageCollector,*\u0026#34;,\u0026#34;java.lang:type=Threading\u0026#34;,\u0026#34;java.lang:type=Runtime\u0026#34;,\u0026#34;java.lang:type=Memory\u0026#34;,\u0026#34;java.lang:name=*\u0026#34;] excludeObjectNames: [\u0026#34;org.apache.activemq:type=ColumnFamily,*\u0026#34;] autoExcludeObjectNameAttributes: true excludeObjectNameAttributes: \u0026#34;java.lang:type=OperatingSystem\u0026#34;: - \u0026#34;ObjectName\u0026#34; \u0026#34;java.lang:type=Runtime\u0026#34;: - \u0026#34;ClassPath\u0026#34; - \u0026#34;SystemProperties\u0026#34; rules: - pattern: \u0026#34;.*\u0026#34; Steps Start ActiveMQ, and the Exporter(agent) and the service start at the same time. Start SkyWalking OAP and SkyWalking UI. Start OpenTelmetry-Collector. After completed, node metrics will be captured and pushed to SkyWalking.\nMetrics Monitoring metrics involve in Cluster Metrics, Broker Metrics, and Destination Metrics.\nCluster Metrics: including memory usage, rates of write/read, and average/max duration of write. Broker Metrics: including node state, number of connections, number of producers/consumers, and rate of write/read under the broker. Depending on the cluster mode, one cluster may include one or more brokers. Destination Metrics: including number of producers/consumers, messages in different states, queues, and enqueue duration in a queue/topic. Cluster Metrics System Load: range in [0, 100]. Thread Count: the number of threads currently used by the JVM. Heap Memory: capacity of heap memory. GC: memory of ActiveMQ is managed by Java\u0026rsquo;s garbage collection (GC) process. Enqueue/Dequeue/Dispatch/Expired Rate: growth rate of messages in different states. Average/Max Enqueue Time: time taken to join the queue. Broker Metrics Uptime: duration of the node. State: 1 = slave node, 0 = master node. Current Connentions: number of connections. Current Producer/Consumer Count: number of current producers/consumers. Increased Producer/Consumer Count: number of increased producers/consumers. Enqueue/Dequeue Count: number of enqueue and dequeue. Enqueue/Dequeue Rate: rate of enqueue and dequeue. Memory Percent Usage: amount of memory space used by undelivered messages. Store Percent Usage: space used by pending persistent messages. Temp Percent Usage: space used by non-persistent messages. Average/Max Message Size: number of messages. Queue Size: number of messages in the queue. Destination Metrics Produser/Consumer Count: number of producers/Consumers. Queue Size: unacknowledged messages of the queue. Memory usage: usage of memory. Enqueue/Dequeue/Dispatch/Expired/Inflight Count: number of messages in different states. Average/Max Message Size: number of messages. Average/Max Enqueue Time: time taken to join the queue. Reference ActiveMQ Classic clustering JMX Exporter Configuration JMX Exporter-Running the Standalone HTTP Server OpenTelemetry Collector Contrib Jmxreceiver ","excerpt":"\u003cp\u003e\u003cimg src=\"activemq_logo.png\" alt=\"icon\"\u003e\u003c/p\u003e\n\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"https://activemq.apache.org/components/classic/\"\u003eApache ActiveMQ Classic\u003c/a\u003e is a popular and powerful open-source messaging and …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2024-04-19-monitoring-activemq-through-skywalking/","title":"Monitoring ActiveMQ through SkyWalking"},{"body":"\n引言 Apache ActiveMQ Classic 是一个流行且功能强大的开源消息传递和集成模式服务器。始于2004年，逐渐发展成为了一个成熟且广泛使用的开源消息中间件，符合Java消息服务（JMS）规范。 发展至今，凭借其稳定性和广泛的特性支持，仍然拥有一定数量的中小型企业的使用者。其高性能版本 Apache Artemis 目前处于快速发展阶段，也受到了 ActiveMQ 现有使用者的关注。\nActiveMQ 对 JMX(Java Management Extensions) 有广泛的支持，允许通过 JMX MBean 监视和控制代理的行为。 开启JMX之后，就可以使用 JAVA 自带的 jconsole 工具或者 VisualVM 等工具直观查看指标。此外也可以通过一些 Collector 组件，将 JMX 风格的数据转换为 prometheus 风格的数据，适配更多查询与展示工具。\nOpenTelemetry 作为业界公认的标准化解决方案，可为分布式系统提供一致且可互操作的遥测数据收集、传输和分析能力，这里也主要借助它实现数据的采集和传输。 它虽然可以直接接受 JMX 类型的数据，但是关于采集 ActiveMQ 的 JMX 指标并不在标准库，存在部分版本不兼容，因此本文采用两步：将 JMX 数据转换为 Prometheus 风格的指标数据，再使用 OpenTelemetry 传递。\nSkyWalking 作为一站式的分布式系统监控解决方案，接纳来自 ActiveMQ 的指标数据，并提供基础的指标监控面板。\n服务部署 请准备以下服务\nSkyWalking OAP, v10.0+。 ActiveMQ v6.0.X+。 JMX Exporter v0.20.0。如果你使用docker，参考使用 bitnami/jmx-exporter。 OpenTelmetry-Collector v0.92.0。 服务准备 以下通过 SkyWalking OAP 单节点、ActiveMQ 2个单节点服务的部署方式介绍。JMX Exporter 采用推荐的 agent 方式启动。\n配置流程 在 ActiveMQ 中开启JMX，其中 JMX 远程端口默认1616，如需修改可通过 ACTIVEMQ_SUNJMX_START 参数调整。 设置 Exporter： 如果采用推荐的 Agent 方式启动，需要追加启动参数 -DACTIVEMQ_OPTS=-javaagent:{activemqPath}/bin/jmx_prometheus_javaagent-0.20.0.jar=2345:{activemqPath}/conf/config.yaml 如果采用单独服务的方式启动，可以参考这里独立部署 Exporter 服务。 其中 2345 为开放的 HTTP 端口可自定义。最终可通过访问 http://localhost:2345/metrics 查询到 JMX 的指标数据。 采用 Agent Exporter 方式的 docker-compose.yml 配置样例：\nversion: \u0026#39;3.8\u0026#39; services: amq1: image: apache/activemq-classic:latest container_name: amq1 hostname: amq1 volumes: - ~/activemq1/conf/activemq.xml:/opt/apache-activemq/conf/activemq.xml - ~/activemq1/bin/jmx_prometheus_javaagent-0.20.0.jar:/opt/apache-activemq/bin/jmx_prometheus_javaagent-0.20.0.jar - ~/activemq1/conf/config.yaml:/opt/apache-activemq/conf/config.yaml ports: - \u0026#34;61616:61616\u0026#34; - \u0026#34;8161:8161\u0026#34; - \u0026#34;2345:2345\u0026#34; environment: ACTIVEMQ_OPTS: \u0026#34;-javaagent:/opt/apache-activemq/bin/jmx_prometheus_javaagent-0.20.0.jar=2345:/opt/apache-activemq/conf/config.yaml\u0026#34; ACTIVEMQ_BROKER_NAME: broker-1 networks: - amqtest amq2: image: apache/activemq-classic:latest container_name: amq2 hostname: amq2 volumes: - ~/activemq2/conf/activemq.xml:/opt/apache-activemq/conf/activemq.xml - ~/activemq2/bin/jmx_prometheus_javaagent-0.20.0.jar:/opt/apache-activemq/bin/jmx_prometheus_javaagent-0.20.0.jar - ~/activemq2/conf/config.yaml:/opt/apache-activemq/conf/config.yaml ports: - \u0026#34;61617:61616\u0026#34; - \u0026#34;8162:8161\u0026#34; - \u0026#34;2346:2346\u0026#34; environment: ACTIVEMQ_OPTS: \u0026#34;-javaagent:/opt/apache-activemq/bin/jmx_prometheus_javaagent-0.20.0.jar=2346:/opt/apache-activemq/conf/config.yaml\u0026#34; ACTIVEMQ_BROKER_NAME: broker-2 networks: - amqtest otel-collector1: image: otel/opentelemetry-collector:latest container_name: otel-collector1 command: [ \u0026#34;--config=/etc/otel-collector-config.yaml\u0026#34; ] volumes: - ./otel-collector-config1.yaml:/etc/otel-collector-config.yaml depends_on: - amq1 networks: - amqtest otel-collector2: image: otel/opentelemetry-collector:latest container_name: otel-collector2 command: [ \u0026#34;--config=/etc/otel-collector-config.yaml\u0026#34; ] volumes: - ./otel-collector-config2.yaml:/etc/otel-collector-config.yaml depends_on: - amq2 networks: - amqtest networks: amqtest: OpenTelemetry otel-collector-config.yaml 配置样例：\nreceivers: prometheus: config: scrape_configs: - job_name: \u0026#39;activemq-monitoring\u0026#39; scrape_interval: 30s static_configs: - targets: [\u0026#39;amq1:2345\u0026#39;] labels: cluster: activemq-broker1 processors: batch: exporters: otlp: endpoint: oap:11800 tls: insecure: true service: pipelines: metrics: receivers: - prometheus processors: - batch exporters: - otlp ActiveMQ Exporter config.yaml 配置样例：\n--- startDelaySeconds: 10 username: admin password: activemq ssl: false lowercaseOutputName: false lowercaseOutputLabelNames: false includeObjectNames: [\u0026#34;org.apache.activemq:*\u0026#34;,\u0026#34;java.lang:type=OperatingSystem\u0026#34;,\u0026#34;java.lang:type=GarbageCollector,*\u0026#34;,\u0026#34;java.lang:type=Threading\u0026#34;,\u0026#34;java.lang:type=Runtime\u0026#34;,\u0026#34;java.lang:type=Memory\u0026#34;,\u0026#34;java.lang:name=*\u0026#34;] excludeObjectNames: [\u0026#34;org.apache.activemq:type=ColumnFamily,*\u0026#34;] autoExcludeObjectNameAttributes: true excludeObjectNameAttributes: \u0026#34;java.lang:type=OperatingSystem\u0026#34;: - \u0026#34;ObjectName\u0026#34; \u0026#34;java.lang:type=Runtime\u0026#34;: - \u0026#34;ClassPath\u0026#34; - \u0026#34;SystemProperties\u0026#34; rules: - pattern: \u0026#34;.*\u0026#34; 启动步骤 启动 ActiveMQ，Exporter 和服务同时启动。 启动 SkyWalking OAP 和 SkyWalking UI。 启动 OpenTelmetry-Collector。 以上步骤执行完成后，节点指标就会定时抓取后推送到 SkyWalking，经过分组聚合后前端页面可查看到 ActiveMQ 的面板数据。\n监控指标 监控指标主要分为3类：Cluster 指标、Broker 指标、Destination 指标\nCluster 指标：主要关注集群的内存使用情况、数据写入与读取速率平均情况、平均与最大的写入时长等。 Broker 指标：主要关注 Broker 下节点状态、连接数、生产者消费者数量、写入读取速率等。根据集群形式不同，一个Cluster可能包括一个或多个Broker。 Destination 指标：主要关注 Queue/Topic 下的生产者消费者数量、不同状态消息数量、队列数量、入队时长等。 Cluster 指标 System Load：[0, 100]的值来反馈系统负载。 Thread Count：JVM 当前使用的线程数。 Heap Memory：堆内存的容量一定程度反映服务的处理性能。 GC：ActiveMQ 在 JVM 中运行，其内存由 Java 的垃圾回收 （GC） 进程管理，GC能直接反映服务的状态。 Enqueue/Dequeue/Dispatch/Expired Rate：不同状态信息的增长速率能直接反映生产活动。 Average/Max Enqueue Time：入队的耗时能一定程度影响生产者。 Broker 指标 Uptime：节点存活时长。 State：是否为从节点，1=从节点，0=主节点。 Current Connentions：目前的连接数。 Current Producer/Consumer Count：目前生产者消费者数量。 Increased Producer/Consumer Count：增长的生产者消费者数量。 Enqueue/Dequeue Count： 入队出队数量。 Enqueue/Dequeue Rate： 入队出队速率。 Memory Percent Usage：未送达消息使用的内存空间。 Store Percent Usage： 挂起的持久性消息占用的空间。 Temp Percent Usage：非持久化消息占用的空间。 Average/Max Message Size：消息量。 Queue Size：队列中消息量。 Destination 指标 Producer/Consumer Count：生产者/消费者数量。 Queue Size：队列的未消费数量。 Memory Usage：内存的使用。 Enqueue/Dequeue/Dispatch/Expired/Inflight Count：不同状态消息数。 Average/Max Enqueue Time：入队的耗时。 Average/Max Message Size：消息量。 参考文档 ActiveMQ Classic clustering JMX Exporter Configuration JMX Exporter-Running the Standalone HTTP Server OpenTelemetry Collector Contrib Jmxreceiver ","excerpt":"\u003cp\u003e\u003cimg src=\"activemq_logo.png\" alt=\"icon\"\u003e\u003c/p\u003e\n\u003ch2 id=\"引言\"\u003e引言\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"https://activemq.apache.org/components/classic/\"\u003eApache ActiveMQ Classic\u003c/a\u003e 是一个流行且功能强大的开源消息传递和集成模式服务器。始于2004年，逐渐发展成为了一个成熟且广泛使用的开源消息中间件，符合Java消息服务 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2024-04-19-monitoring-activemq-through-skywalking/","title":"使用 SkyWalking 监控 ActiveMQ"},{"body":"Zixin Zhou(GitHub ID, CodePrometheus[1]) began the code contributions since Oct 28, 2023.\nUp to date, he has submitted 8 PRs in the Go agent repository, 7 PRs in the main repo, 1 PR in the UI repository and 2 PRs in the showcase repository.\nAt Apr 15th, 2024, the project management committee(PMC) passed the proposal of promoting him as a new committer. He has accepted the invitation at the same day.\nWelcome Zixin Zhou join the committer team.\n[1] https://github.com/CodePrometheus\n","excerpt":"\u003cp\u003eZixin Zhou(GitHub ID, CodePrometheus[1]) began the code contributions since Oct 28, 2023.\u003c/p\u003e\n\u003cp\u003eUp to …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-zixin-zhou-as-new-committer/","title":"Welcome Zixin Zhou as new committer"},{"body":"SkyWalking Eyes 0.6.0 is released. Go to downloads page to find release tars.\nAdd | as comment indicator by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/168 Correct the way of joining slack channels by @wu-sheng in https://github.com/apache/skywalking-eyes/pull/169 update: add weak-compatible to dependency check by @Two-Hearts in https://github.com/apache/skywalking-eyes/pull/171 feature: add support for Protocol Buffer by @spacewander in https://github.com/apache/skywalking-eyes/pull/172 feature: add support for OPA policy files by @spacewander in https://github.com/apache/skywalking-eyes/pull/174 add Eclipse Foundation specific Apache 2.0 license header by @gdams in https://github.com/apache/skywalking-eyes/pull/178 add instructions to fix header issues in markdown comment by @gdams in https://github.com/apache/skywalking-eyes/pull/179 bump action/setup-go to v5 by @gdams in https://github.com/apache/skywalking-eyes/pull/180 Draft release notes for 0.6.0 by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/181 Full Changelog: https://github.com/apache/skywalking-eyes/compare/v0.5.0...v0.6.0\n","excerpt":"\u003cp\u003eSkyWalking Eyes 0.6.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAdd \u003ccode\u003e|\u003c/code\u003e as comment …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-eyes-0-6-0/","title":"Release Apache SkyWalking Eyes 0.6.0"},{"body":"SkyWalking Java Agent 9.2.0 is released. Go to downloads page to find release tars. Changes by Version\n9.2.0 Fix NoSuchMethodError in mvc-annotation-commons and change deprecated method. Fix forkjoinpool plugin in JDK11. Support for tracing spring-cloud-gateway 4.x in gateway-4.x-plugin. Fix re-transform bug when plugin enhanced class proxy parent method. Fix error HTTP status codes not recording as SLA failures in Vert.x plugins. Support for HttpExchange request tracing. Support tracing for async producing, batch sync consuming, and batch async consuming in rocketMQ-client-java-5.x-plugin. Convert the Redisson span into an async span. Rename system env name from sw_plugin_kafka_producer_config to SW_PLUGIN_KAFKA_PRODUCER_CONFIG. Support for ActiveMQ-Artemis messaging tracing. Archive the expired plugins impala-jdbc-2.6.x-plugin. Fix a bug in Spring Cloud Gateway if HttpClientFinalizer#send does not invoke, the span created at NettyRoutingFilterInterceptor can not stop. Fix not tracing in HttpClient v5 when HttpHost(arg[0]) is null but RoutingSupport#determineHost works. Support across thread tracing for SOFA-RPC. Update Jedis 4.x plugin to support Sharding and Cluster models. Documentation Update docs to describe expired-plugins. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking Java Agent 9.2.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nChanges by …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-java-agent-9-2-0/","title":"Release Apache SkyWalking Java Agent 9.2.0"},{"body":"SkyWalking Rover 0.6.0 is released. Go to downloads page to find release tars.\nFeatures Enhance compatibility when profiling with SSL. Update LabelValue obtain pod information function to add default value parameter. Add HasOwnerName to judgement pod has owner name. Publish the latest Docker image tag. Improve the stability of Off CPU Profiling. Support collecting the access log from Kubernetes. Remove the scanner mode in the process discovery module. Upgrade Go library to 1.21, eBPF library to 0.13.2. Support using make docker.debug to building the debug docker image. Bug Fixes Documentation Update architecture diagram. Delete module design and project structure document. Adjust configuration modules during setup. Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Rover 0.6.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eEnhance …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-rover-0-6-0/","title":"Release Apache SkyWalking Rover 0.6.0"},{"body":"SkyWalking Cloud on Kubernetes 0.9.0 is released. Go to downloads page to find release tars.\n0.9.0 Features Add a getting started document about how to deploy swck on the kubernetes cluster. Bugs Fix the bug that the java agent is duplicated injected when update the pod. Chores Bump up custom-metrics-apiserver Bump up golang to v1.22 Bump up controller-gen to v0.14.0 ","excerpt":"\u003cp\u003eSkyWalking Cloud on Kubernetes 0.9.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"090\"\u003e0.9.0 …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cloud-on-kubernetes-0-9-0/","title":"Release Apache SkyWalking Cloud on Kubernetes 0.9.0"},{"body":"Background Apache SkyWalking is an open-source Application Performance Management system that helps users gather logs, traces, metrics, and events from various platforms and display them on the UI. With version 9.7.0, SkyWalking can collect access logs from probes in multiple languages and from Service Mesh, generating corresponding topologies, tracing, and other data. However, it could not initially collect and map access logs from applications in Kubernetes environments. This article explores how the 10.0.0 version of Apache SkyWalking employs eBPF technology to collect and store application access logs, addressing this limitation.\nWhy eBPF? To monitor the network traffic in Kubernetes, the following features support be support:\nCross Language: Applications deployed in Kubernetes may be written in any programming language, making support for diverse languages important. Non-Intrusiveness: It\u0026rsquo;s imperative to monitor network traffic without making any modifications to the applications, as direct intervention with applications in Kubernetes is not feasible. Kernel Metrics Monitoring: Often, diagnosing network issues by analyzing traffic performance at the user-space level is insufficient. A deeper analysis incorporating kernel-space network traffic metrics is frequently necessary. Support for Various Network Protocols: Applications may communicate using different transport protocols, necessitating support for a range of protocols. Given these requirements, eBPF emerges as a capable solution. In the next section, we will delve into detailed explanations of how Apache SkyWalking Rover resolves these aspects.\nKernel Monitoring and Protocol Analysis In previous articles, we\u0026rsquo;ve discussed how to monitor network traffic from programs written in various languages. This technique remains essential for network traffic monitoring, allowing for the collection of traffic data without language limitations. However, due to the unique aspects of our monitoring trigger mechanism and the specific features of kernel monitoring, these two areas warrant separate explanations.\nKernel Monitoring Kernel monitoring allows users to gain insights into network traffic performance based on the execution at the kernel level, specifically from Layer 2 (Data Link) to Layer 4 (Transport) of the OSI model.\nNetwork monitoring at the kernel layer is deference from the syscall (user-space) layer in terms of the metrics and identifiers used. While the syscalls layer can utilize file descriptors to correlate various operations, kernel layer network operations primarily use packets as unique identifiers. This discrepancy necessitates a mapping relationship that SkyWalking Rover can use to bind these two layers together for comprehensive monitoring.\nLet\u0026rsquo;s dive into the details of how data is monitored in both sending and receiving modes.\nObserve Sending When sending data, tracking the status and timing of each packet is crucial for understanding the state of each transmission. Within the kernel, operations progress from Layer 4 (L4) down to Layer 2 (L2), maintaining the same thread ID as during the syscalls layer, which simplifies data correlation.\nSkyWalking Rover monitors several key kernel functions to observe packet transmission dynamics, listed from L4 to L2:\nkprobe/tcp_sendmsg: Captures the time when a packet enters the L4 protocol stack for sending and the time it finishes processing. This function is essential for tracking the initial handling of packets at the transport layer. kprobe/tcp_transmit_skb: Records the total number of packet transmissions and the size of each packet sent. This function helps identify how many times a packet or a batch of packets is attempted to be sent, which is critical for understanding network throughput and congestion. tracepoint/tcp/tcp_retransmit_skb: Notes whether packet retransmission occurs, providing insights into network reliability and connection quality. Retransmissions can significantly impact application performance and user experience. tracepoint/skb/kfree_skb: Records packet loss during transmission and logs the reason for such occurrences. Understanding packet loss is crucial for diagnosing network issues and ensuring data integrity. kprobe/__ip_queue_xmit: Records the start and end times of processing by the L3 protocol. This function is vital for understanding the time taken for IP-level operations, including routing decisions. kprobe/nf_hook_slow: Records the total time and number of occurrences spent in Netfilter hooks, such as iptables rule evaluations. This monitoring point is important for assessing the impact of firewall rules and other filtering mechanisms on packet flow. kprobe/neigh_resolve_output: If resolving an unknown MAC address is necessary before sending a network request, this function records the occurrences and total time spent on this resolution. MAC address resolution times can affect the initial packet transmission delay. kprobe/__dev_queue_xmit: Records the start and end times of entering the L2 protocol stack, providing insights into the data link layer\u0026rsquo;s processing times. tracepoint/net/net_dev_start_xmit and tracepoint/net/net_dev_xmit: Records the actual time taken to transmit each packet at the network interface card (NIC). These functions are crucial for understanding the hardware-level performance and potential bottlenecks at the point of sending data to the physical network. According to the interception of the above method, Apache SkyWalking Rover can provide key execution time and metrics for each level when sending network data, from the application layer (Layer 7) to the transport layer (Layer 4), and finally to the data link layer (Layer 2).\nObserve Receiving When receiving data, the focus is often on the time it takes for packets to travel from the network interface card (NIC) to the user space. Unlike the process of sending data, data receiving in the kernel proceeds from the data link layer (Layer 2) up to the transport layer (Layer 4), until the application layer (Layer 7) retrieves the packet\u0026rsquo;s content. In SkyWalking Rover, monitors the following key system functions to observe this process, listed from L2 to L4:\ntracepoint/net/netif_receive_skb: Records the time when a packet is received by the network interface card. This tracepoint is crucial for understanding the initial point of entry for incoming data into the system. kprobe/ip_rcv: Records the start and end times of packet processing at the network layer (Layer 3). This probe provides insights into how long it takes for the IP layer to handle routing, forwarding, and delivering packets to the correct application. kprobe/nf_hook_slow: Records the total time and occurrences spent in Netfilter hooks, same with the sending traffic flow. kprobe/tcp_v4_rcv: Records the start and end times of packet processing at the transport layer (Layer 4). This probe is key to understanding the efficiency of TCP operations, including connection management, congestion control, and data flow. tracepoint/skb/skb_copy_datagram_iovec: When application layer protocols use the data, this tracepoint binds the packet to the syscall layer data at Layer 7. This connection is essential for correlating the kernel\u0026rsquo;s handling of packets with their consumption by user-space applications. Based on the above methods, network monitoring can help you understand the complete execution process and execution time from when data is received by the network card to when it is used by the program.\nMetrics By intercepting the methods mentioned above, we can gather key metrics that provide insights into network performance and behavior. These metrics include:\nPackets: The size of the packets and the frequency of their transmission or reception. These metric offers a fundamental understanding of the network load and the efficiency of data movement between the sender and receiver. Connections: The number of connections established or accepted between services and the time taken for these connections to be set up. This metric is crucial for analyzing the efficiency of communication and connection management between different services within the network. L2-L4 Events: The time spent on key events within the Layer 2 to Layer 4 protocols. This metric sheds light on the processing efficiency and potential bottlenecks within the lower layers of the network stack, which are essential for data transmission and reception. Protocol Analyzing In previous articles, we have discussed parsing HTTP/1.x protocols. However, with HTTP/2.x, the protocol\u0026rsquo;s stateful nature and the pre-established connections between services complicate network profiling. This complexity makes it challenging for Apache SkyWalking Rover to fully perceive the connection context, hindering protocol parsing operations.\nTransitioning network monitoring to Daemon mode offers a solution to this challenge. By continuously observing service operations around the clock, SkyWalking Rover can begin monitoring as soon as a service starts. This immediate initiation allows for the tracking of the complete execution context, making the observation of stateful protocols like HTTP/2.x feasible.\nProbes To detect when a process is started, monitoring a specific trace point (tracepoint/sched/sched_process_fork) is essential. This approach enables the system to be aware of process initiation events. Given the necessity to filter process traffic based on certain criteria such as the process\u0026rsquo;s namespace, Apache SkyWalking Rover follows a series of steps to ensure accurate and efficient monitoring. These steps include:\nMonitoring Activation: The process is immediately added to a monitoring whitelist upon detection. This step ensures that the process is considered for monitoring from the moment it starts, without delay. Push to Queue: The process\u0026rsquo;s PID (Process ID) is pushed into a monitoring confirmation queue. This queue holds the PIDs of newly detected processes that are pending further confirmation from a user-space program. This asynchronous approach allows for the separation of immediate detection and subsequent processing, optimizing the monitoring workflow. User-Space Program Confirmation: The user-space program retrieves process PIDs from the queue and assesses whether each process should continue to be monitored. If a process is deemed unnecessary for monitoring, it is removed from the whitelist. This process ensures that SkyWalking Rover can dynamically adapt its monitoring scope based on real-time conditions and configurations, allowing for both comprehensive coverage and efficient resource use.\nLimitations The monitoring of stateful protocols like HTTP/2.x currently faces certain limitations:\nInability to Observe Pre-existing Connections: Monitoring the complete request and response cycle requires that monitoring be initiated before any connections are established. This requirement means that connections set up before the start of monitoring cannot be observed. Challenges with TLS Requests: Observing TLS encrypted traffic is complex because it relies on asynchronously attaching uprobes (user-space attaching) for observation. If new requests are made before these uprobes are successfully attached, it becomes impossible to access the data before encryption or after decryption. Demo Next, let’s quickly demonstrate the Kubernetes monitoring feature, so you can understand more specifically what it accomplishes.\nDeploy SkyWalking Showcase SkyWalking Showcase contains a complete set of example services and can be monitored using SkyWalking. For more information, please check the official documentation.\nIn this demo, we only deploy service, the latest released SkyWalking OAP, and UI.\nexport FEATURE_FLAGS=java-agent-injector,single-node,elasticsearch,rover make deploy.kubernetes After deployment is complete, please run the following script to open SkyWalking UI: http://localhost:8080/.\nkubectl port-forward svc/ui 8080:8080 --namespace default Done Once deployed, Apache SkyWalking Rover automatically begins monitoring traffic within the system upon startup. Then, reports this traffic data to SkyWalking OAP, where it is ultimately stored in a database.\nIn the Service Dashboard within Kubernetes, you can view a list of monitored Kubernetes services. If any of these services have HTTP traffic, this information would be displayed alongside them in the dashboard.\nFigure 1: Kubernetes Service List\nAdditionally, within the Topology Tab, you can observe the topology among related services. In each service or call relationship, there would display relevant TCP and HTTP metrics.\nFigure 2: Kubernetes Service Topology\nWhen you select a specific service from the Service list, you can view service metrics at both the TCP and HTTP levels for the chosen service.\nFigure 3: Kubernetes Service TCP Metrics\nFigure 4: Kubernetes Service HTTP Metrics\nFurthermore, by using the Endpoint Tab, you can see which URIs have been accessed for the current service.\nFigure 5: Kubernetes Service Endpoint List\nConclusion In this article, I\u0026rsquo;ve detailed how to utilize eBPF technology for network monitoring of services within a Kubernetes cluster, a capability that has been implemented in Apache SkyWalking Rover. This approach leverages the power of eBPF to provide deep insights into network traffic and service interactions, enhancing visibility and observability across the cluster.\n","excerpt":"\u003ch2 id=\"background\"\u003eBackground\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"https://skywalking.apache.org/\"\u003eApache SkyWalking\u003c/a\u003e is an open-source Application Performance Management system that helps …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2024-03-18-monitor-kubernetes-network-by-ebpf/","title":"Monitoring Kubernetes network traffic by using eBPF"},{"body":"SkyWalking Client JS 0.11.0 is released. Go to downloads page to find release tars.\nFixed the bug that navigator.sendBeacon sent json to backend report \u0026ldquo;No suitable request converter found for a @RequestObject List\u0026rdquo;. Fix reading property from null. Pin selenium version and update license CI. Bump dependencies. Update README. ","excerpt":"\u003cp\u003eSkyWalking Client JS 0.11.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eFixed the bug …\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-client-js-0-11-0/","title":"Release Apache SkyWalking Client JS 0.11.0"},{"body":"背景 Apache SkyWalking 是一个开源的应用性能管理系统，帮助用户从各种平台收集日志、跟踪、指标和事件，并在用户界面上展示它们。\n在9.7.0版本中，Apache SkyWalking 可以从多语言的探针和 Service Mesh 中收集访问日志，并生成相应的拓扑图、链路和其他数据。 但是对于Kubernetes环境，暂时无法提供对应用程序的访问日志进行采集并生成拓扑图。本文探讨了Apache SkyWalking 10.0.0版本如何采用eBPF技术来收集和存储应用访问日志，解决了这一限制。\n为什么使用 eBPF？ 为了在Kubernetes中监控网络流量，以下特性需得到支持：\n跨语言: 在Kubernetes部署的应用可能使用任何编程语言编写，因此对多种语言的支持十分重要。 非侵入性: 监控网络流量时不对应用程序进行任何修改是必要的，因为直接干预Kubernetes中的应用程序是不可行的。 内核指标监控: 通常，仅通过分析用户空间级别的流量来诊断网络问题是不够的。经常需要深入分析，结合内核空间的网络流量指标。 支持多种网络协议: 应用程序可能使用不同的传输协议进行通信，这就需要支持一系列的协议。 鉴于这些要求，eBPF显现出作为一个有能力的解决方案。在下一节中，我们将深入讨论Apache SkyWalking Rover是如何解决这些方面作出更详细解释。\n内核监控与协议分析 在之前的文章中，我们讨论了如何对不同编程语言的程序进行网络流量获取。在网络流量监控中，我们仍然会使用该技术进行流量采集。 但是由于这次监控触发方式和内核监控方面的不同特性，所以这两部分会单独进行说明。\n内核监控 内核监控允许用户根据在内核层面的执行，洞察网络流量性能，特别是从OSI模型的第2层（数据链路层）到第4层（传输层）。\n内核层的网络监控与syscall（用户空间系统调用）层在关联指标不同。虽然syscall层可以利用文件描述符来关联各种操作，但内核层的网络操作主要使用数据包作为唯一标识符。 这种差异需要映射关系，Apache SkyWalking Rover可以使用它将这两层绑定在一起，进行全面监控。\n让我们深入了解数据在发送和接收模式下是如何被监控的。\n监控数据发送 在发送数据时，跟踪每个数据包的状态和时间对于理解每次传输的状态至关重要。在内核中，操作从第4层（L4）一直调用到第2层（L2），并且会保持与在syscall层相同的线程ID，这简化了数据的相关性分析。\nSkyWalking Rover监控了几个关键的内核函数，以观察数据包传输动态，顺序从L4到L2：\nkprobe/tcp_sendmsg: 记录数据包进入L4协议栈进行发送以及完成处理的时间。这个函数对于跟踪传输层对数据包的初始处理至关重要。 kprobe/tcp_transmit_skb: 记录数据包传输的总次数和每个发送的数据包的大小。这个函数有助于识别尝试发送一个数据包或一段时间内发送一批数据包的次数，这对于理解网络吞吐量和拥塞至关重要。 tracepoint/tcp/tcp_retransmit_skb: 记录是否发生数据包重传，提供网络可靠性和连接质量的见解。重传可以显著影响应用性能和用户体验。 tracepoint/skb/kfree_skb: 记录传输过程中的数据包丢失，并记录发生这种情况的原因。理解数据包丢失对于诊断网络问题和确保数据完整性至关重要。 kprobe/__ip_queue_xmit: 记录L3协议处理的开始和结束时间。这个功能对于理解IP级操作所需的时间至关重要，包括路由决策。 kprobe/nf_hook_slow: 记录在Netfilter钩子中花费的总时间和发生次数，例如 iptables 规则评估。这个函数对于评估防火墙规则和其他过滤机制对数据流的影响非常重要。 kprobe/neigh_resolve_output: 如果在发送网络请求之前需要解析未知的MAC地址，这个函数会记录发生的次数和在这个解析上花费的总时间。MAC地址解析时间可以影响初始数据包传输的延迟。 kprobe/__dev_queue_xmit: 记录进入L2协议栈的开始和结束时间，提供对数据链路层处理时间的见解。 tracepoint/net/net_dev_start_xmit and tracepoint/net/net_dev_xmit: 记录在网卡（NIC）上传输每个数据包所需的实际时间。这些功能对于理解硬件级性能和在将数据发送到物理网络时可能出现的瓶颈至关重要。 根据上述方法的拦截，Apache SkyWalking Rover可以在发送网络数据时为每个层级提供关键的执行时间和指标，从应用层（第7层）到传输层（第4层），最终到数据链路层（第2层）。\n监控数据接收 在接收数据时，通常关注的是数据包从网卡（NIC）到用户空间的传输时间。与发送数据的过程不同，在内核中接收数据是从数据链路层（第2层）开始，一直上升到传输层（第4层），直到应用层（第7层）检索到数据包的内容。\n在SkyWalking Rover中，监控以下关键系统功能以观察这一过程，顺序从L2到L4：\ntracepoint/net/netif_receive_skb: 记录网卡接收到数据包的时间。这个追踪点对于理解进入系统的传入数据的初始入口点至关重要。 kprobe/ip_rcv: 记录网络层（第3层）数据包处理的开始和结束时间。这个探针提供了IP层处理路由、转发和将数据包正确传递给应用程序所需时间的见解。 kprobe/nf_hook_slow: 记录在Netfilter钩子中花费的总时间和发生次数，与发送流量的情况相同。 kprobe/tcp_v4_rcv: 记录传输层（第4层）数据包处理的开始和结束时间。这个探针对于理解TCP操作的效率至关重要，包括连接管理、拥塞控制和数据流。 tracepoint/skb/skb_copy_datagram_iovec: 当应用层协议使用数据时，这个追踪点在第7层将数据包与syscall层的数据绑定。这种连接对于将内核对数据包的处理与用户空间应用程序的消费相关联是至关重要的。 基于上述方法，网络监控可以帮助您理解从网卡接收数据到程序使用数据的完整执行过程和执行时间。\n指标 通过拦截上述提到的方法，我们可以收集提供网络性能的关键指标。这些指标包括：\n数据包: 数据包的大小及其传输或接收的频率。这些指标提供了对网络负载和数据在发送者与接收者之间传输效率的基本理解。 连接: 服务之间建立或接收的连接数量，以及设置这些连接所需的时间。这个指标对于分析网络内不同服务之间的通信效率和连接管理至关重要。 L2-L4 事件: 在第2层到第4层协议中关键事件上所花费的时间。这个指标揭示了网络堆栈较低层的处理效率和潜在瓶颈，这对于数据传输至关重要。 协议分析 在之前的文章中，我们已经讨论了解析 HTTP/1.x 协议。然而，对于 HTTP/2.x，协议的有状态性质和服务之间预先建立的连接使得网络分析变得复杂。 这种复杂性使得Apache SkyWalking Rover很难完全感知连接上下文，阻碍了协议解析操作。\n将网络监控转移到守护进程模式提供了一种解决这一挑战的方法。通过全天候不断观察服务，Apache SkyWalking Rover可以在服务启动时立即开始监控。 这种立即启动允许跟踪完整的执行上下文，使得观察像 HTTP/2.x 这样的有状态协议变得可行。\n追踪 为了检测到一个进程何时启动，监控一个特定的追踪点 (tracepoint/sched/sched_process_fork) 是必不可少的。这追踪点使系统能够意识到进程启动事件。\n鉴于需要根据某些标准（如进程的命名空间）过滤进程流量，Apache SkyWalking Rover遵循一系列步骤来确保准确和高效的监控。这些步骤包括：\n启动监控: 一旦检测到进程，立即将其添加到监控白名单中。这一步确保从进程启动的那一刻起就考虑对其进行监控，不会有延迟。 推送队列: 进程的PID（进程ID）被推送到一个监控确认队列中。这个队列保存了新检测到的进程的PID，这些进程等待来自用户空间程序的进一步确认。这种异步方法对立即检测和后续处理进行分离，优化了监控工作流程。 用户态程序确认: 用户空间程序从队列中检索进程PID，并评估每个进程是否应该继续被监控。如果一个进程被认为不必要进行监控，它将被从白名单中移除。 这个过程确保了Apache SkyWalking Rover可以根据实时条件和配置动态调整其监控范围，允许既全面覆盖又有效的资源监控。\n限制 像 HTTP/2.x 这样的有状态协议的监控目前仍然面临一些限制：\n无法观察现有连接: 要监控完整的请求和响应周期，需要在建立任何连接之前启动监控。这个要求意味着在监控开始之前建立的连接无法被观察到。 TLS请求的挑战: 观察TLS加密流量是复杂的，因为它依赖于异步加载uprobes（用户空间加载）进行观察。如果在成功加载这些uprobes之前发出新的请求，那么在加密之前或解密之后访问数据就变得不可能。 演示 接下来，让我们快速演示Kubernetes监控功能，以便更具体地了解它的功能。\n部署 SkyWalking Showcase SkyWalking Showcase 包含完整的示例服务，并可以使用 SkyWalking 进行监视。有关详细信息，请查看官方文档。\n在此演示中，我们只部署服务、最新发布的 SkyWalking OAP，UI和Rover。\nexport FEATURE_FLAGS=java-agent-injector,single-node,elasticsearch,rover make deploy.kubernetes 部署完成后，请运行以下脚本以打开 SkyWalking UI：http://localhost:8080/ 。\nkubectl port-forward svc/ui 8080:8080 --namespace default 完成 一旦部署，Apache SkyWalking Rover在启动时会自动开始监控系统中的流量。然后，它将这些流量数据报告给SkyWalking OAP，并最终存储在数据库中。\n在Kubernetes中的服务仪表板中，您可以查看被监控的Kubernetes服务列表。如果其中任何服务具有HTTP流量，这些指标信息将在列表中显示。\n图 1: Kubernetes 服务列表\n此外，在拓扑图选项卡中，您可以观察相关服务之间的拓扑关系。在每个服务节点或服务之间调用关系中，将显示相关的TCP和HTTP指标。\n图 2: Kubernetes 服务拓扑图\n当您从服务列表中选择特定服务时，您可以查看所选服务在TCP和HTTP级别的服务指标。\n图 3: Kubernetes 服务 TCP 指标\n图 4: Kubernetes 服务 HTTP 指标\n此外，通过使用端点选项卡，您可以查看当前服务所访问的URI。\n图 5: Kubernetes 服务端点列表\n结论 在本文中，我详细介绍了如何利用eBPF技术对Kubernetes集群中的服务进行网络流量监控，这是Apache SkyWalking Rover中实现的一项功能。\n这项功能利用了eBPF的强大功能，提供了对网络流量和服务交互的深入洞察，增强了对整个集群的可观测性。\n","excerpt":"\u003ch2 id=\"背景\"\u003e背景\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"https://skywalking.apache.org/\"\u003eApache SkyWalking\u003c/a\u003e 是一个开源的应用性能管理系统，帮助用户从各种平台收集日志、跟踪、指标和事件，并在用户界面上展示它们。\u003c/p\u003e\n\u003cp\u003e在9.7.0版本中，Apache SkyWalking …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2024-03-18-monitor-kubernetes-network-by-ebpf/","title":"使用 eBPF 监控 Kubernetes 网络流量"},{"body":"Background ClickHouse is an open-source column-oriented database management system that allows generating analytical data reports in real-time, so it is widely used for online analytical processing (OLAP).\nApache SkyWalking is an open-source APM system that provides monitoring, tracing and diagnosing capabilities for distributed systems in Cloud Native architectures. Increasingly, App Service architectures incorporate Skywalking as an essential monitoring component of a service or instance.\nBoth ClickHouse and Skywalking are popular frameworks, and it would be great to monitor your ClickHouse database through Skywalking. Next, let\u0026rsquo;s share how to monitor ClickHouse database with Skywalking.\nPrerequisites and configurations Make sure you\u0026rsquo;ve met the following prerequisites before you start onboarding your monitor.\nConfig steps:\nExposing prometheus endpoint. Fetching ClickHouse metrics by OpenTelemetry. Exporting metrics to Skywalking OAP server. Prerequisites for setup The monitoring for ClickHouse relies on the embedded prometheus endpoint of ClickHouse and will not be supported in previous versions starting from v20.1.2.4.\nYou can check the version of your server:\n:) select version(); SELECT version() Query id: 2d3773ca-c320-41f6-b2ac-7ebe37eddc58 ┌─version()───┐ │ 24.2.1.2248 │ └─────────────┘ If your ClickHouse version is earlier than v20.1.2.4, you need to set up ClickHouse-exporter to access data.\nExpose prometheus Endpoint The embedded prometheus endpoint will make it easy for data collection, you just need to open the required configuration in the core configuration file config.xml of ClickHouse. In addition to your original configuration, you only need to modify the configuration of Prometheus.\n/etc/clickhouse-server/config.xml:\n\u0026lt;clickhouse\u0026gt; ...... \u0026lt;prometheus\u0026gt; \u0026lt;endpoint\u0026gt;/metrics\u0026lt;/endpoint\u0026gt; \u0026lt;port\u0026gt;9363\u0026lt;/port\u0026gt; \u0026lt;metrics\u0026gt;true\u0026lt;/metrics\u0026gt; \u0026lt;events\u0026gt;true\u0026lt;/events\u0026gt; \u0026lt;asynchronous_metrics\u0026gt;true\u0026lt;/asynchronous_metrics\u0026gt; \u0026lt;errors\u0026gt;true\u0026lt;/errors\u0026gt; \u0026lt;/prometheus\u0026gt; \u0026lt;/clickhouse\u0026gt; Settings:\nendpoint – HTTP endpoint for scraping metrics by prometheus server. Start from ‘/’. port – Port for endpoint. metrics – Expose metrics from the system.metrics table. events – Expose metrics from the system.events table. asynchronous_metrics – Expose current metrics values from the system.asynchronous_metrics table. errors - Expose the number of errors by error codes occurred since the last server restart. This information could be obtained from the system.errors as well. Save the config and restart the ClickHouse server.\nIt contains more than 1,000 metrics, covering services、networks、disk、MergeTree、errors and so on. For more details, after restarting the server, you can call curl 127.0.0.1:9363/metrics to know about the metrics.\nYou also can check the metrics by tables to make a contrast.\n:) select * from system.metrics limit 10 SELECT * FROM system.metrics LIMIT 10 Query id: af677622-960e-4589-b2ca-0b6a40c443aa ┌─metric───────────────────────────────┬─value─┬─description─────────────────────────────────────────────────────────────────────┐ │ Query │ 1 │ Number of executing queries │ │ Merge │ 0 │ Number of executing background merges │ │ Move │ 0 │ Number of currently executing moves │ │ PartMutation │ 0 │ Number of mutations (ALTER DELETE/UPDATE) │ │ ReplicatedFetch │ 0 │ Number of data parts being fetched from replica │ │ ReplicatedSend │ 0 │ Number of data parts being sent to replicas │ │ ReplicatedChecks │ 0 │ Number of data parts checking for consistency │ │ BackgroundMergesAndMutationsPoolTask │ 0 │ Number of active merges and mutations in an associated background pool │ │ BackgroundMergesAndMutationsPoolSize │ 64 │ Limit on number of active merges and mutations in an associated background pool │ │ BackgroundFetchesPoolTask │ 0 │ Number of active fetches in an associated background pool │ └──────────────────────────────────────┴───────┴─────────────────────────────────────────────────────────────────────────────────┘ :) select * from system.events limit 10; SELECT * FROM system.events LIMIT 10 Query id: 32c618d0-037a-400a-92a4-59fde832e4e2 ┌─event────────────────────────────┬──value─┬─description────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ Query │ 7 │ Number of queries to be interpreted and potentially executed. Does not include queries that failed to parse or were rejected due to AST size limits, quota limits or limits on the number of simultaneously running queries. May include internal queries initiated by ClickHouse itself. Does not count subqueries. │ │ SelectQuery │ 7 │ Same as Query, but only for SELECT queries. │ │ InitialQuery │ 7 │ Same as Query, but only counts initial queries (see is_initial_query). │ │ QueriesWithSubqueries │ 40 │ Count queries with all subqueries │ │ SelectQueriesWithSubqueries │ 40 │ Count SELECT queries with all subqueries │ │ QueryTimeMicroseconds │ 202862 │ Total time of all queries. │ │ SelectQueryTimeMicroseconds │ 202862 │ Total time of SELECT queries. │ │ FileOpen │ 40473 │ Number of files opened. │ │ Seek │ 100 │ Number of times the \u0026#39;lseek\u0026#39; function was called. │ │ ReadBufferFromFileDescriptorRead │ 67995 │ Number of reads (read/pread) from a file descriptor. Does not include sockets. │ └──────────────────────────────────┴────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ Start up Opentelemetry-Collector Configure OpenTelemetry based on your own requirements. Following the example below:\notel-collector-config.yaml:\nreceivers: prometheus: config: scrape_configs: - job_name: \u0026#39;clickhouse-monitoring\u0026#39; scrape_interval: 15s static_configs: - targets: [\u0026#39;127.0.0.1:9363\u0026#39;,\u0026#39;127.0.0.1:9364\u0026#39;,\u0026#39;127.0.0.1:9365\u0026#39;] labels: host_name: prometheus-clickhouse processors: batch: exporters: otlp: endpoint: 127.0.0.1:11800 tls: insecure: true service: pipelines: metrics: receivers: - prometheus processors: - batch exporters: - otlp Please ensure:\njob_name: 'clickhouse-monitoring' that marked the data from ClickHouse, If modified, it will be ignored. host_name defines the service name, you have to make one. endpoint point to the oap server address. the network between ClickHouse, OpenTelemetry Collector, and Skywalking OAP Server must be accessible. If goes well, refresh the Skywalking-ui home page in a few seconds and you can see ClickHouse under the database menu.\nsuccess log:\n2024-03-12T03:57:39.407Z\tinfo\tservice@v0.93.0/telemetry.go:76\tSetting up own telemetry... 2024-03-12T03:57:39.412Z\tinfo\tservice@v0.93.0/telemetry.go:146\tServing metrics\t{\u0026#34;address\u0026#34;: \u0026#34;:8888\u0026#34;, \u0026#34;level\u0026#34;: \u0026#34;Basic\u0026#34;} 2024-03-12T03:57:39.416Z\tinfo\tservice@v0.93.0/service.go:139\tStarting otelcol...\t{\u0026#34;Version\u0026#34;: \u0026#34;0.93.0\u0026#34;, \u0026#34;NumCPU\u0026#34;: 4} 2024-03-12T03:57:39.416Z\tinfo\textensions/extensions.go:34\tStarting extensions... 2024-03-12T03:57:39.423Z\tinfo\tprometheusreceiver@v0.93.0/metrics_receiver.go:240\tStarting discovery manager\t{\u0026#34;kind\u0026#34;: \u0026#34;receiver\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;prometheus\u0026#34;, \u0026#34;data_type\u0026#34;: \u0026#34;metrics\u0026#34;} 2024-03-12T03:57:59.431Z\tinfo\tprometheusreceiver@v0.93.0/metrics_receiver.go:231\tScrape job added\t{\u0026#34;kind\u0026#34;: \u0026#34;receiver\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;prometheus\u0026#34;, \u0026#34;data_type\u0026#34;: \u0026#34;metrics\u0026#34;, \u0026#34;jobName\u0026#34;: \u0026#34;clickhouse-monitoring\u0026#34;} 2024-03-12T03:57:59.431Z\tinfo\tservice@v0.93.0/service.go:165\tEverything is ready. Begin running and processing data. 2024-03-12T03:57:59.432Z\tinfo\tprometheusreceiver@v0.93.0/metrics_receiver.go:282\tStarting scrape manager\t{\u0026#34;kind\u0026#34;: \u0026#34;receiver\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;prometheus\u0026#34;, \u0026#34;data_type\u0026#34;: \u0026#34;metrics\u0026#34;} ClickHouse monitoring dashboard About the dashboard The dashboard includes the service dashboard and the instance dashboard.\nMetrics include servers, queries, networks, insertions, replicas, MergeTree, ZooKeeper and embedded ClickHouse Keeper.\nThe service dashboard displays the metrics of the entire cluster.\nThe instance dashboard displays the metrics of an instance.\nAbout the metrics Here are some meanings of ClickHouse Instance metrics, more here.\nMonitoring Panel Unit Description Data Source CpuUsage count CPU time spent seen by OS per second(according to ClickHouse.system.dashboard.CPU Usage (cores)). ClickHouse MemoryUsage percentage Total amount of memory (bytes) allocated by the server/ total amount of OS memory. ClickHouse MemoryAvailable percentage Total amount of memory (bytes) available for program / total amount of OS memory. ClickHouse Uptime sec The server uptime in seconds. It includes the time spent for server initialization before accepting connections. ClickHouse Version string Version of the server in a single integer number in base-1000. ClickHouse FileOpen count Number of files opened. ClickHouse metrics about ZooKeeper are valid when managing cluster by ZooKeeper metrics about embedded ClickHouse Keeper are valid when ClickHouse Keeper is enabled References ClickHouse prometheus endpoint ClickHouse built-in observability dashboard ClickHouse Keeper ","excerpt":"\u003ch2 id=\"background\"\u003eBackground\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"https://clickhouse.com/\"\u003eClickHouse\u003c/a\u003e is an open-source column-oriented database management system that allows …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2024-03-12-monitoring-clickhouse-through-skywalking/","title":"Monitoring Clickhouse Server through SkyWalking"},{"body":"背景介绍 ClickHouse 是一个开源的面向列的数据库管理系统，可以实时生成分析数据报告，因此被广泛用于在线分析处理（OLAP）。\nApache SkyWalking 是一个开源的 APM 系统，为云原生架构中的分布式系统提供监控、跟踪和诊断能力。应用服务体系越来越多地将 Skywalking 作为服务或实例的基本监视组件。\nClickHouse 和 Skywalking 框架都是当下流行的服务组件，通过 Skywalking 监控您的 ClickHouse 数据库将是一个不错的选择。接下来，就来分享一下如何使用 Skywalking 监控 ClickHouse 数据库。\n前提与配置 在开始接入监控之前，请先确认以下前提条件。\n配置步骤:\n暴露 Prometheus 端点。 通过 OpenTelemetry 拉取 ClickHouse 的指标数据。 将指标数据发送到 Skywalking OAP server. 使用的前提 ClickHouse 的监控依赖于 ClickHouse 的内嵌 Prometheus 端点配置，配置从 v20.1.2.4 开始支持，因此之前的老版本将无法支持。\n您可以检查 ClickHouse 服务的版本:\n:) select version(); SELECT version() Query id: 2d3773ca-c320-41f6-b2ac-7ebe37eddc58 ┌─version()───┐ │ 24.2.1.2248 │ └─────────────┘ 如果您的 ClickHouse 版本低于 v20.1.2.4，则需要依靠 ClickHouse-exporter 获取数据。\n暴露 Prometheus 端点 内嵌的 Prometheus 端点简化了数据采集流程，您只需要在 ClickHouse 的核心配置文件 config.xml 打开所需的配置即可。除了您原来的配置，您只需要参考如下修改 Prometheus 的配置。\n/etc/clickhouse-server/config.xml:\n\u0026lt;clickhouse\u0026gt; ...... \u0026lt;prometheus\u0026gt; \u0026lt;endpoint\u0026gt;/metrics\u0026lt;/endpoint\u0026gt; \u0026lt;port\u0026gt;9363\u0026lt;/port\u0026gt; \u0026lt;metrics\u0026gt;true\u0026lt;/metrics\u0026gt; \u0026lt;events\u0026gt;true\u0026lt;/events\u0026gt; \u0026lt;asynchronous_metrics\u0026gt;true\u0026lt;/asynchronous_metrics\u0026gt; \u0026lt;errors\u0026gt;true\u0026lt;/errors\u0026gt; \u0026lt;/prometheus\u0026gt; \u0026lt;/clickhouse\u0026gt; 配置说明:\nendpoint – 通过 prometheus 服务器抓取指标的 HTTP 端点。从/开始。 port – 端点的端口。 metrics – 暴露 system.metrics 表中的指标。 events – 暴露 system.events 表中的指标。 asynchronous_metrics – 暴露 system.asynchronous_metrics 表中的当前指标值。 errors - 按错误代码暴露自上次服务器重新启动以来发生的错误数。此信息也可以从 system.errors 中获得。 保存配置并重启 ClickHouse 服务。\n端点数据包含1000多个指标，涵盖服务、网络、磁盘、MergeTree、错误等。想了解更多指标细节，在重启服务后，可以调用 curl 127.0.0.1:9363/metrics 看到具体指标的内容。\n您还可以通过数据库表的数据与端点数据进行检查对比。\n:) select * from system.metrics limit 10 SELECT * FROM system.metrics LIMIT 10 Query id: af677622-960e-4589-b2ca-0b6a40c443aa ┌─metric───────────────────────────────┬─value─┬─description─────────────────────────────────────────────────────────────────────┐ │ Query │ 1 │ Number of executing queries │ │ Merge │ 0 │ Number of executing background merges │ │ Move │ 0 │ Number of currently executing moves │ │ PartMutation │ 0 │ Number of mutations (ALTER DELETE/UPDATE) │ │ ReplicatedFetch │ 0 │ Number of data parts being fetched from replica │ │ ReplicatedSend │ 0 │ Number of data parts being sent to replicas │ │ ReplicatedChecks │ 0 │ Number of data parts checking for consistency │ │ BackgroundMergesAndMutationsPoolTask │ 0 │ Number of active merges and mutations in an associated background pool │ │ BackgroundMergesAndMutationsPoolSize │ 64 │ Limit on number of active merges and mutations in an associated background pool │ │ BackgroundFetchesPoolTask │ 0 │ Number of active fetches in an associated background pool │ └──────────────────────────────────────┴───────┴─────────────────────────────────────────────────────────────────────────────────┘ :) select * from system.events limit 10; SELECT * FROM system.events LIMIT 10 Query id: 32c618d0-037a-400a-92a4-59fde832e4e2 ┌─event────────────────────────────┬──value─┬─description────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ Query │ 7 │ Number of queries to be interpreted and potentially executed. Does not include queries that failed to parse or were rejected due to AST size limits, quota limits or limits on the number of simultaneously running queries. May include internal queries initiated by ClickHouse itself. Does not count subqueries. │ │ SelectQuery │ 7 │ Same as Query, but only for SELECT queries. │ │ InitialQuery │ 7 │ Same as Query, but only counts initial queries (see is_initial_query). │ │ QueriesWithSubqueries │ 40 │ Count queries with all subqueries │ │ SelectQueriesWithSubqueries │ 40 │ Count SELECT queries with all subqueries │ │ QueryTimeMicroseconds │ 202862 │ Total time of all queries. │ │ SelectQueryTimeMicroseconds │ 202862 │ Total time of SELECT queries. │ │ FileOpen │ 40473 │ Number of files opened. │ │ Seek │ 100 │ Number of times the \u0026#39;lseek\u0026#39; function was called. │ │ ReadBufferFromFileDescriptorRead │ 67995 │ Number of reads (read/pread) from a file descriptor. Does not include sockets. │ └──────────────────────────────────┴────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ 启动 Opentelemetry-Collector 根据自身环境 配置 OpenTelemetry。 您可参照下面的例子:\notel-collector-config.yaml:\nreceivers: prometheus: config: scrape_configs: - job_name: \u0026#39;clickhouse-monitoring\u0026#39; scrape_interval: 15s static_configs: - targets: [\u0026#39;127.0.0.1:9363\u0026#39;,\u0026#39;127.0.0.1:9364\u0026#39;,\u0026#39;127.0.0.1:9365\u0026#39;] labels: host_name: prometheus-clickhouse processors: batch: exporters: otlp: endpoint: 127.0.0.1:11800 tls: insecure: true service: pipelines: metrics: receivers: - prometheus processors: - batch exporters: - otlp 请着重关注:\njob_name: 'clickhouse-monitoring' 标记着来自 ClickHouse 的数据，如果自行修改，数据会被服务忽略。 host_name 定义服务的名称。 endpoint 指向您的 OAP 服务地址. ClickHouse、OpenTelemetry Collector 和 Skywalking OAP Server 之间的网络必须可访问。 如果进展顺利，几秒钟后刷新 Skywalking-ui 网页，您可以在数据库的菜单下看到 ClickHouse。\n启动成功日志样例:\n2024-03-12T03:57:39.407Z\tinfo\tservice@v0.93.0/telemetry.go:76\tSetting up own telemetry... 2024-03-12T03:57:39.412Z\tinfo\tservice@v0.93.0/telemetry.go:146\tServing metrics\t{\u0026#34;address\u0026#34;: \u0026#34;:8888\u0026#34;, \u0026#34;level\u0026#34;: \u0026#34;Basic\u0026#34;} 2024-03-12T03:57:39.416Z\tinfo\tservice@v0.93.0/service.go:139\tStarting otelcol...\t{\u0026#34;Version\u0026#34;: \u0026#34;0.93.0\u0026#34;, \u0026#34;NumCPU\u0026#34;: 4} 2024-03-12T03:57:39.416Z\tinfo\textensions/extensions.go:34\tStarting extensions... 2024-03-12T03:57:39.423Z\tinfo\tprometheusreceiver@v0.93.0/metrics_receiver.go:240\tStarting discovery manager\t{\u0026#34;kind\u0026#34;: \u0026#34;receiver\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;prometheus\u0026#34;, \u0026#34;data_type\u0026#34;: \u0026#34;metrics\u0026#34;} 2024-03-12T03:57:59.431Z\tinfo\tprometheusreceiver@v0.93.0/metrics_receiver.go:231\tScrape job added\t{\u0026#34;kind\u0026#34;: \u0026#34;receiver\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;prometheus\u0026#34;, \u0026#34;data_type\u0026#34;: \u0026#34;metrics\u0026#34;, \u0026#34;jobName\u0026#34;: \u0026#34;clickhouse-monitoring\u0026#34;} 2024-03-12T03:57:59.431Z\tinfo\tservice@v0.93.0/service.go:165\tEverything is ready. Begin running and processing data. 2024-03-12T03:57:59.432Z\tinfo\tprometheusreceiver@v0.93.0/metrics_receiver.go:282\tStarting scrape manager\t{\u0026#34;kind\u0026#34;: \u0026#34;receiver\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;prometheus\u0026#34;, \u0026#34;data_type\u0026#34;: \u0026#34;metrics\u0026#34;} ClickHouse 监控面板 关于面板 这个仪表盘包含服务仪表盘和实例仪表盘。\n指标涵盖服务器、查询、网络、插入、副本、MergeTree、ZooKeeper 和内嵌 ClickHouse Keeper。\n服务仪表盘主要展示整个集群相关的指标。\n实例仪表盘主要展示单个实例相关的指标。\n关于指标 以下是ClickHouse实例指标的一些含义，前往了解完整的指标列表。\n面板名称 单位 指标含义 数据源 CpuUsage count 操作系统每秒花费的 CPU 时间（根据 ClickHouse.system.dashboard.CPU 使用率（核心数））。 ClickHouse MemoryUsage percentage 服务器分配的内存总量（字节）/操作系统内存总量。 ClickHouse MemoryAvailable percentage 可用于程序的内存总量（字节）/操作系统内存总量。 ClickHouse Uptime sec 服务器正常运行时间（以秒为单位）。它包括在接受连接之前进行服务器初始化所花费的时间。 ClickHouse Version string 以 base-1000 样式展示的服务器版本。 ClickHouse FileOpen count 打开的文件数。 ClickHouse ZooKeeper 的指标在 ZooKeeper 管理集群时有效。 内嵌ClickHouse Keeper的指标在开启内嵌 ClickHouse Keeper 配置时有效。 参考文档 ClickHouse prometheus endpoint ClickHouse built-in observability dashboard ClickHouse Keeper ","excerpt":"\u003ch2 id=\"背景介绍\"\u003e背景介绍\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"https://clickhouse.com/\"\u003eClickHouse\u003c/a\u003e 是一个开源的面向列的数据库管理系统，可以实时生成分析数据报告，因此被广泛用于在线分析处理（OLAP）。\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://skywalking.apache.org/\"\u003eApache SkyWalking\u003c/a\u003e 是一个开源的 APM 系统， …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2024-03-12-monitoring-clickhouse-through-skywalking/","title":"使用 SkyWalking 监控 ClickHouse Server"},{"body":"背景介绍 Apache RocketMQ 是一个开源的低延迟、高并发、高可用、高可靠的分布式消息中间件, 从SkyWalking OAP 10.0 版本开始, 新增了 对 RocketMQ Server的监控面板。本文将展示并介绍如何使用 Skywalking来监控RocketMQ\n部署 流程 通过RocketMQ官方提供的RocketMQ exporter来采集RocketMQ Server数据,再通过opentelmetry-collector来拉取RocketMQ exporter并传输到skywalking oap服务来处理\nDataFlow: 准备 Skywalking oap服务,v10.0 + RocketMQ v4.3.2 + RocketMQ exporter v0.0.2+ Opentelmetry-collector v0.87+ 启动顺序 启动 RocketMQ namesrv 和 broker 启动 skywalking oap 和 ui 启动 RocketMQ exporter 启动 opentelmetry-collector 具体如何启动和配置请参考以上链接中官方教程.\n需要注意下的是 opentelmetry-collector 的配置文件.\njob_name: \u0026quot;rocketmq-monitoring\u0026quot; 请不要修改,否则 skywalking 不会处理这部分数据.\nrocketmq-exporter 替换成RocketMQ exporter 的地址.\nreplacement: rocketmq-cluster 中的rocketmq-cluster如果想要使用下文介绍的服务分层功能,请自行定义为其他服务层级相匹配的名称.\noap 为 skywalking oap 地址,请自行替换.\nreceivers: prometheus: config: scrape_configs: - job_name: \u0026#34;rocketmq-monitoring\u0026#34; scrape_interval: 30s static_configs: - targets: [\u0026#39;rocketmq-exporter:5557\u0026#39;] relabel_configs: - source_labels: [ ] target_label: cluster replacement: rocketmq-cluster exporters: otlp: endpoint: oap:11800 tls: insecure: true processors: batch: service: pipelines: metrics: receivers: - prometheus processors: - batch exporters: - otlp 监控指标 指标分为 三个维度, cluster,broker,topic\ncluster监控 cluster 主要是站在集群的角度来统计展示,比如\nMessages Produced Today 今日集群产生的消息数\nMax CommitLog Disk Ratio 展示集群中磁盘使用率最高的broker\nTotal Producer Tps 集群生产者tps\nbroker 监控 broker 主要是站在节点的角度来统计展示,比如\nProduce Tps 节点生产者tps\nProducer Message Size(MB)节点生产消息大小\ntopic 监控 topic 主要是站在主题的角度来统计展示,比如\nConsumer Group Count 消费该主题的消费者组个数\nConsumer Latency(s) 消费者组的消费延时时间\nBacklogged Messages 消费者组消费消息堆积\n注意:topic 维度是整个 topic 来聚合,并不是在一个 broker 上的 topic 聚合,在 dashboard 上你也可以看到 broker 跟 topic 是平级的。\n各个指标的含义可以在图标的 tip 上找到解释\n更多指标可以参考文档\ndemo 已经在 skywalking showcase 上线,可以在上面看到展示效果\n服务分层 skywalking 10 新增了重要功能Service Hierarchy,接收来自不同层级的服务数据,比如 java agent 上报,k8s 监控数据或者 otel 的监控数据. 根据设置规则如果发现这些服务名称符合匹配规则,则可以将这些不同层级的服务联系起来。\n如下图所示：\nskywalking 采集部署在 k8s 的 RocketMQ 服务端的k8s 数据,并接收来自 otel 的 RocketMQ 服务端监控数据,根据匹配规则这些服务具有相同的服务名称,则可以在 ui 上观察到它们的联系\n","excerpt":"\u003ch1 id=\"背景介绍\"\u003e背景介绍\u003c/h1\u003e\n\u003cp\u003eApache RocketMQ 是一个开源的低延迟、高并发、高可用、高可靠的分布式消息中间件, 从SkyWalking OAP 10.0 版本开始, 新增了 对 RocketMQ …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2024-02-29-rocketmq-monitoring-by-skywalking/","title":"使用 SkyWalking 监控 RocketMQ Server"},{"body":"SkyWalking Go 0.4.0 is released. Go to downloads page to find release tars.\nFeatures Add support ignore suffix for span name. Adding go 1.21 and 1.22 in docker image. Plugins Support setting a discard type of reporter. Add redis.max_args_bytes parameter for redis plugin. Changing intercept point for gin, make sure interfaces could be grouped when params defined in relativePath. Support RocketMQ MQ. Support AMQP MQ. support Echov4 framework. Documentation Bug Fixes Fix users can not use async api in toolkit-trace. Fix cannot enhance the vendor management project. Fix SW_AGENT_REPORTER_GRPC_MAX_SEND_QUEUE not working on metricsSendCh \u0026amp; logSendCh chans of gRPC reporter. Fix ParseVendorModule error for special case in vendor/modules.txt. Fix enhance method error when unknown parameter type. Fix wrong tracing context when trace have been sampled. Fix enhance param error when there are multiple params. Fix lost trace when multi middleware handlerFunc in gin plugin. Fix DBQueryContext execute error in sql plugin. Fix stack overflow as endless logs triggered. Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Go 0.4.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd support …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-go-0.4.0/","title":"Release Apache SkyWalking Go 0.4.0"},{"body":"背景介绍 在 Scala 中，纯函数式中主要使用 Fiber，而不是线程，诸如 Cats-Effect、ZIO 等 Effect 框架。 您可以将 Fiber 视为轻量级线程，它是一种并发模型，由框架本身掌控控制权，从而消除了上下文切换的开销。 基于这些 Effect 框架开发的 HTTP、gRCP、GraphQL 库而开发的应用，我们一般称为 纯函数式应用程序。\n我们以 ZIO 为切入点， 演示 SkyWalking Scala 如何支持 Effect 生态。\nZIO Trace 首先，我们想要实现 Fiber 上下文传递，而不是监控 Fiber 本身。对于一个大型应用来说，可能存在成千上万个 Fiber，监控 Fiber 本身的意义不大。\n虽然 Fiber 的 Span 是在活跃时才会创建，但难免会有目前遗漏的场景，所以提供了一个配置 plugin.ziov2.ignore_fiber_regexes。 它将使用正则去匹配 Fiber location，匹配上的 Fiber 将不会创建 Span。\nFiber Span的信息如下：\n下面是我们使用本 ZIO 插件，和一些官方插件（hikaricp、jdbc、pulsar）完成的 Trace：\n分析 在 ZIO 中，Fiber可以有两种方式被调度，它们都是 zio.Executor 的子类。当然您也可以使用自己的线程池，这样也需被 ZIO 包装，其实就类似下面的 blockingExecutor。\nabstract class Executor extends ExecutorPlatformSpecific { self =\u0026gt; def submit(runnable: Runnable)(implicit unsafe: Unsafe): Boolean } 一种是系统默认线程池 defaultExecutor：\nprivate[zio] trait RuntimePlatformSpecific { final val defaultExecutor: Executor = Executor.makeDefault() } 另一种是专用于阻塞 IO 的线程池 blockingExecutor：\nprivate[zio] trait RuntimePlatformSpecific { final val defaultBlockingExecutor: Executor = Blocking.blockingExecutor } 默认线程池 defaultExecutor 对于 defaultExecutor，其本身是很复杂的，但它就是一个 ZIO 的 Fiber 调度（执行）器：\n/** * A `ZScheduler` is an `Executor` that is optimized for running ZIO * applications. Inspired by \u0026#34;Making the Tokio Scheduler 10X Faster\u0026#34; by Carl * Lerche. [[https://tokio.rs/blog/2019-10-scheduler]] */ private final class ZScheduler extends Executor 由于它们都是 zio.Executor 的子类，我们只需要对其及其子类进行增强：\nfinal val ENHANCE_CLASS = LogicalMatchOperation.or( HierarchyMatch.byHierarchyMatch(\u0026#34;zio.Executor\u0026#34;), MultiClassNameMatch.byMultiClassMatch(\u0026#34;zio.Executor\u0026#34;) ) 它们都是线程池，我们只需要在 zio.Executor 的 submit 方法上进行类似 ThreadPoolExecutor 上下文捕获的操作，可以参考 jdk-threadpool-plugin\n这里需要注意，因为 Fiber 也是一种 Runnable：\nprivate[zio] trait FiberRunnable extends Runnable { def location: Trace def run(depth: Int): Unit } zio-v2x-plugin\n阻塞线程池 blockingExecutor 对于 blockingExecutor，其实它只是对 Java 线程池进行了一个包装：\nobject Blocking { val blockingExecutor: zio.Executor = zio.Executor.fromThreadPoolExecutor { val corePoolSize = 0 val maxPoolSize = Int.MaxValue val keepAliveTime = 60000L val timeUnit = TimeUnit.MILLISECONDS val workQueue = new SynchronousQueue[Runnable]() val threadFactory = new NamedThreadFactory(\u0026#34;zio-default-blocking\u0026#34;, true) val threadPool = new ThreadPoolExecutor( corePoolSize, maxPoolSize, keepAliveTime, timeUnit, workQueue, threadFactory ) threadPool } } 由于其本身是对 ThreadPoolExecutor 的封装，所以，当我们已经实现了 zio.Executor 的增强后，只需要使用官方 jdk-threadpool-plugin 插件即可。 这里我们还想要对代码进行定制修改和复用，所以重新使用 Scala 实现了一个 executors-plugin 插件。\n串连 Fiber 上下文 最后，上面谈到过，Fiber 也是一种 Runnable，因此还需要对 zio.internal.FiberRunnable 进行增强。大致分为两点，其实与 jdk-threading-plugin 是一样的。\n每次创建 zio.internal.FiberRunnable 实例时，都需要保存 现场，即构造函数增强。 每次运行时创建一个过渡的 Span，将当前线程上下文与之前保存在构造函数中的上下文进行关联。Fiber 可能被不同线程执行，所以这是必须的。 zio-v2x-plugin\n说明 当我们完成了对 ZIO Fiber 的上下文传播处理后，任意基于 ZIO 的应用层框架都可以按照普通的 Java 插件思路去开发。 我们只需要找到一个全局切入点，这个切入点应该是每个请求都会调用的方法，然后对这个方法进行增强。\n要想激活插件，只需要在 Release Notes 下载插件，放到您的 skywalking-agent/plugins 目录，重新启动服务即可。\n如果您的项目使用 sbt assembly 打包，您可以参考这个 示例。该项目使用了下列技术栈：\nlibraryDependencies ++= Seq( \u0026#34;io.d11\u0026#34; %% \u0026#34;zhttp\u0026#34; % zioHttp2Version, \u0026#34;dev.zio\u0026#34; %% \u0026#34;zio\u0026#34; % zioVersion, \u0026#34;io.grpc\u0026#34; % \u0026#34;grpc-netty\u0026#34; % \u0026#34;1.50.1\u0026#34;, \u0026#34;com.thesamet.scalapb\u0026#34; %% \u0026#34;scalapb-runtime-grpc\u0026#34; % scalapb.compiler.Version.scalapbVersion ) ++ Seq( \u0026#34;dev.profunktor\u0026#34; %% \u0026#34;redis4cats-effects\u0026#34; % \u0026#34;1.3.0\u0026#34;, \u0026#34;dev.profunktor\u0026#34; %% \u0026#34;redis4cats-log4cats\u0026#34; % \u0026#34;1.3.0\u0026#34;, \u0026#34;dev.profunktor\u0026#34; %% \u0026#34;redis4cats-streams\u0026#34; % \u0026#34;1.3.0\u0026#34;, \u0026#34;org.typelevel\u0026#34; %% \u0026#34;log4cats-slf4j\u0026#34; % \u0026#34;2.5.0\u0026#34;, \u0026#34;dev.zio\u0026#34; %% \u0026#34;zio-interop-cats\u0026#34; % \u0026#34;23.0.03\u0026#34;, \u0026#34;ch.qos.logback\u0026#34; % \u0026#34;logback-classic\u0026#34; % \u0026#34;1.2.11\u0026#34;, \u0026#34;dev.zio\u0026#34; %% \u0026#34;zio-cache\u0026#34; % zioCacheVersion ) ","excerpt":"\u003ch2 id=\"背景介绍\"\u003e背景介绍\u003c/h2\u003e\n\u003cp\u003e在 Scala 中，纯函数式中主要使用 Fiber，而不是线程，诸如 \u003ca href=\"https://github.com/typelevel/cats-effect\"\u003eCats-Effect\u003c/a\u003e、\u003ca href=\"https://github.com/zio/zio\"\u003eZIO\u003c/a\u003e 等 Effect 框架。\n您可以将 Fiber 视为轻量级线程，它是一种并发模型，由框架 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2024-01-04-skywalking-for-scala-effect-runtime/","title":"SkyWalking 如何支持 ZIO 等 Scala Effect Runtime"},{"body":"Xiang Wei(GitHub ID, weixiang1862) made a lot of significant contributions to SkyWalking since 2023. He made dozens of pull requests to multiple SkyWalking repositories, including very important features, such as Loki LogQL support, Nginx monitoring, MongoDB monitoring, as well as bug fixes, blog posts, and showcase updates.\nHere are the complete pull request list grouped by repositories.\nskywalking Support Nginx monitoring. (https://github.com/apache/skywalking/pull/11558) Fix JDBC Log query order. (https://github.com/apache/skywalking/pull/11544) Isolate MAL CounterWindow cache by metric name.(https://github.com/apache/skywalking/pull/11526) Support extract timestamp from patterned datetime string in LAL.(https://github.com/apache/skywalking/pull/11489) Adjust AlarmRecord alarmMessage column length to 512. (https://github.com/apache/skywalking/pull/11404) Use listening mode for Apollo configuration.(https://github.com/apache/skywalking/pull/11186) Support LogQL HTTP query APIs. (https://github.com/apache/skywalking/pull/11168) Support MongoDB monitoring (https://github.com/apache/skywalking/pull/11111) Support reduce aggregate function in MQE.(https://github.com/apache/skywalking/pull/11036) Fix instance query in JDBC implementation.(https://github.com/apache/skywalking/pull/11024) Fix metric session cache saving after batch insert when using mysql-connector-java.(https://github.com/apache/skywalking/pull/11012) Add component ID for WebSphere.(https://github.com/apache/skywalking/pull/10974) Support sumLabeled in MAL (https://github.com/apache/skywalking/pull/10916) skywalking-java Optimize plugin selector logic.(https://github.com/apache/skywalking-java/pull/651) Fix config length limitation.(https://github.com/apache/skywalking-java/pull/623) Optimize spring-cloud-gateway 2.1.x, 3.x witness class.(https://github.com/apache/skywalking-java/pull/610) Add WebSphere Liberty 23.x plugin.(https://github.com/apache/skywalking-java/pull/560) skywalking-swck Remove SwAgent default env JAVA_TOOL_OPTIONS.(https://github.com/apache/skywalking-swck/pull/106) Fix panic in storage reconciler.(https://github.com/apache/skywalking-swck/pull/94) Support inject java agent bootstrap-plugins.(https://github.com/apache/skywalking-swck/pull/91) Fix number env value format error in template yaml.(https://github.com/apache/skywalking-swck/pull/90) skywalking-showcase Nginx monitoring showcase.(https://github.com/apache/skywalking-showcase/pull/153) LogQL showcase. (https://github.com/apache/skywalking-showcase/pull/146) MongoDB monitoring showcase. (https://github.com/apache/skywalking-showcase/pull/144)## skywalking-website Add blog: monitoring-nginx-by-skywalking.(https://github.com/apache/skywalking-website/pull/666) Add blog: collect and analyse nginx access log by LAL.(https://github.com/apache/skywalking-website/pull/652) Add blog: integrating-skywalking-with-arthas.(https://github.com/apache/skywalking-website/pull/641) At Dec. 28th, 2023, the project management committee (PMC) passed the proposal of promoting him as a new committer. He has accepted the invitation at the same day.\nWelcome to join the committer team, Xiang Wei! We are honored to have you in the team.\n","excerpt":"\u003cp\u003eXiang Wei(GitHub ID, weixiang1862) made a lot of significant contributions to SkyWalking since 2023. …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-xiang-wei-as-new-committer/","title":"Welcome Xiang Wei as new committer"},{"body":"Background Apache SkyWalking is an open-source application performance management system that helps users collect and aggregate logs, traces, metrics, and events, and display them on the UI.\nIn order to achieve monitoring capabilities for Nginx, we have introduced the Nginx monitoring dashboard in SkyWalking 9.7, and this article will demonstrate the use of this monitoring dashboard and introduce the meaning of related metrics.\nSetup Monitoring Dashboard Metric Define and Collection Since nginx-lua-prometheus is used to define and expose metrics, we need to install lua_nginx_module for Nginx, or use OpenResty directly.\nIn the following example, we define four metrics via nginx-lua-prometheus and expose the metrics interface via nginx ip:9145/metrics:\nhistogram: nginx_http_latency，monitoring http latency gauge: nginx_http_connections，monitoring nginx http connections counter: nginx_http_size_bytes，monitoring http size of request and response counter: nginx_http_requests_total，monitoring total http request numbers http { log_format main \u0026#39;$remote_addr - $remote_user [$time_local] \u0026#34;$request\u0026#34; \u0026#39; \u0026#39;$status $body_bytes_sent \u0026#34;$http_referer\u0026#34; \u0026#39; \u0026#39;\u0026#34;$http_user_agent\u0026#34; \u0026#34;$http_x_forwarded_for\u0026#34;\u0026#39;; access_log /var/log/nginx/access.log main; lua_shared_dict prometheus_metrics 10M; # lua_package_path \u0026#34;/path/to/nginx-lua-prometheus/?.lua;;\u0026#34;; init_worker_by_lua_block { prometheus = require(\u0026#34;prometheus\u0026#34;).init(\u0026#34;prometheus_metrics\u0026#34;) metric_bytes = prometheus:counter( \u0026#34;nginx_http_size_bytes\u0026#34;, \u0026#34;Total size of HTTP\u0026#34;, {\u0026#34;type\u0026#34;, \u0026#34;route\u0026#34;}) metric_requests = prometheus:counter( \u0026#34;nginx_http_requests_total\u0026#34;, \u0026#34;Number of HTTP requests\u0026#34;, {\u0026#34;status\u0026#34;, \u0026#34;route\u0026#34;}) metric_latency = prometheus:histogram( \u0026#34;nginx_http_latency\u0026#34;, \u0026#34;HTTP request latency\u0026#34;, {\u0026#34;route\u0026#34;}) metric_connections = prometheus:gauge( \u0026#34;nginx_http_connections\u0026#34;, \u0026#34;Number of HTTP connections\u0026#34;, {\u0026#34;state\u0026#34;}) } server { listen 8080; location /test { default_type application/json; return 200 \u0026#39;{\u0026#34;code\u0026#34;: 200, \u0026#34;message\u0026#34;: \u0026#34;success\u0026#34;}\u0026#39;; log_by_lua_block { metric_bytes:inc(tonumber(ngx.var.request_length), {\u0026#34;request\u0026#34;, \u0026#34;/test/**\u0026#34;}) metric_bytes:inc(tonumber(ngx.var.bytes_send), {\u0026#34;response\u0026#34;, \u0026#34;/test/**\u0026#34;}) metric_requests:inc(1, {ngx.var.status, \u0026#34;/test/**\u0026#34;}) metric_latency:observe(tonumber(ngx.var.request_time), {\u0026#34;/test/**\u0026#34;}) } } } server { listen 9145; location /metrics { content_by_lua_block { metric_connections:set(ngx.var.connections_reading, {\u0026#34;reading\u0026#34;}) metric_connections:set(ngx.var.connections_waiting, {\u0026#34;waiting\u0026#34;}) metric_connections:set(ngx.var.connections_writing, {\u0026#34;writing\u0026#34;}) prometheus:collect() } } } } In the above example, we exposed the route-level metrics, and you can also choose to expose the host-level metrics according to the monitoring granularity:\nhttp { log_by_lua_block { metric_bytes:inc(tonumber(ngx.var.request_length), {\u0026#34;request\u0026#34;, ngx.var.host}) metric_bytes:inc(tonumber(ngx.var.bytes_send), {\u0026#34;response\u0026#34;, ngx.var.host}) metric_requests:inc(1, {ngx.var.status, ngx.var.host}) metric_latency:observe(tonumber(ngx.var.request_time), {ngx.var.host}) } } or upstream-level metrics：\nupstream backend { server ip:port; } server { location /test_upstream { proxy_pass http://backend; log_by_lua_block { metric_bytes:inc(tonumber(ngx.var.request_length), {\u0026#34;request\u0026#34;, \u0026#34;upstream/backend\u0026#34;}) metric_bytes:inc(tonumber(ngx.var.bytes_send), {\u0026#34;response\u0026#34;, \u0026#34;upstream/backend\u0026#34;}) metric_requests:inc(1, {ngx.var.status, \u0026#34;upstream/backend\u0026#34;}) metric_latency:observe(tonumber(ngx.var.request_time), {\u0026#34;upstream/backend\u0026#34;}) } } } After defining the metrics, we start nginx and opentelemetry-collector to collect the metrics and send them to the SkyWalking backend for analysis and storage.\nPlease ensure that job_name: 'nginx-monitoring', otherwise the reported data will be ignored by SkyWalking. If you have multiple Nginx instances, you can distinguish them using the service and service_instance_id labels：\nreceivers: prometheus: config: scrape_configs: - job_name: \u0026#39;nginx-monitoring\u0026#39; scrape_interval: 5s metrics_path: \u0026#34;/metrics\u0026#34; static_configs: - targets: [\u0026#39;nginx:9145\u0026#39;] labels: service: nginx service_instance_id: nginx-instance processors: batch: exporters: otlp: endpoint: oap:11800 tls: insecure: true service: pipelines: metrics: receivers: - prometheus processors: - batch exporters: - otlp If everything goes well, you will see the metric data reported by Nginx under the gateway menu of the skywalking-ui:\nAccess \u0026amp; Error Log Collection SkyWalking Nginx monitoring provides log collection and error log analysis. We can use fluent-bit to collect and report access logs and error logs to SkyWalking for analysis and storage.\nFluent-bit configuration below defines the log collection directory as /var/log/nginx/. The access and error logs will be reported through rest port 12800 of oap after being processed by rewrite_access_log and rewrite_error_log functions:\n[SERVICE] Flush 5 Daemon Off Log_Level warn [INPUT] Name tail Tag access Path /var/log/nginx/access.log [INPUT] Name tail Tag error Path /var/log/nginx/error.log [FILTER] Name lua Match access Script fluent-bit-script.lua Call rewrite_access_log [FILTER] Name lua Match error Script fluent-bit-script.lua Call rewrite_error_log [OUTPUT] Name stdout Match * Format json [OUTPUT] Name http Match * Host oap Port 12800 URI /v3/logs Format json In the fluent-bit-script.lua, we use LOG_KIND tag to distinguish between access logs and error logs.\nTo associate with the metrics, please ensure that the values of service and serviceInstance are consistent with the metric collection definition in the previous section.\nfunction rewrite_access_log(tag, timestamp, record) local newRecord = {} newRecord[\u0026#34;layer\u0026#34;] = \u0026#34;NGINX\u0026#34; newRecord[\u0026#34;service\u0026#34;] = \u0026#34;nginx::nginx\u0026#34; newRecord[\u0026#34;serviceInstance\u0026#34;] = \u0026#34;nginx-instance\u0026#34; newRecord[\u0026#34;body\u0026#34;] = { text = { text = record.log } } newRecord[\u0026#34;tags\u0026#34;] = { data = {{ key = \u0026#34;LOG_KIND\u0026#34;, value = \u0026#34;NGINX_ACCESS_LOG\u0026#34;}}} return 1, timestamp, newRecord end function rewrite_error_log(tag, timestamp, record) local newRecord = {} newRecord[\u0026#34;layer\u0026#34;] = \u0026#34;NGINX\u0026#34; newRecord[\u0026#34;service\u0026#34;] = \u0026#34;nginx::nginx\u0026#34; newRecord[\u0026#34;serviceInstance\u0026#34;] = \u0026#34;nginx-instance\u0026#34; newRecord[\u0026#34;body\u0026#34;] = { text = { text = record.log } } newRecord[\u0026#34;tags\u0026#34;] = { data = {{ key = \u0026#34;LOG_KIND\u0026#34;, value = \u0026#34;NGINX_ERROR_LOG\u0026#34; }}} return 1, timestamp, newRecord end After starting fluent-it, we can see the collected log information in the Log tab of the monitoring panel：\nMeaning of Metrics Metric Name Unit Description Data Source HTTP Request Trend The increment rate of HTTP requests nginx-lua-prometheus HTTP Latency ms The increment rate of the latency of HTTP requests nginx-lua-prometheus HTTP Bandwidth KB The increment rate of the bandwidth of HTTP requests nginx-lua-prometheus HTTP Connections The avg number of the connections nginx-lua-prometheus HTTP Status Trend % The increment rate of the status of HTTP requests nginx-lua-prometheus HTTP Status 4xx Percent % The percentage of 4xx status of HTTP requests nginx-lua-prometheus HTTP Status 5xx Percent % The percentage of 4xx status of HTTP requests nginx-lua-prometheus Error Log Count The count of log level of nginx error.log fluent-bit References nginx-lua-prometheus fluent-bit-lua-filter skywalking-apisix-monitoring ","excerpt":"\u003ch2 id=\"background\"\u003eBackground\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"https://skywalking.apache.org/\"\u003eApache SkyWalking\u003c/a\u003e is an open-source application performance management system that helps …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2023-12-23-monitoring-nginx-by-skywalking/","title":"Monitoring Nginx with SkyWalking"},{"body":"背景介绍 在前面的 Blog 使用 LAL 收集并分析 Nginx access log 中，我们以 Nginx access log 为切入点， 演示了 SkyWalking LAL 的日志分析能力。\n为了实现对 Nginx 更全面的监控能力，我们在 SkyWalking 9.7 中引入了 Nginx 监控面板，本文将演示该监控面板的使用，并介绍相关指标的含义。\n监控面板接入 Metric 定义与采集 由于使用了 nginx-lua-prometheus 来定义及暴露指标， 我们需要为 Nginx 安装 lua_nginx_module， 或者直接使用OpenResty。\n下面的例子中，我们通过 nginx-lua-prometheus 定义了四个指标，并通过 ip:9145/metrics 暴露指标接口：\nhistogram: nginx_http_latency，监控 http 延时 gauge: nginx_http_connections，监控 http 连接数 counter: nginx_http_size_bytes，监控 http 请求和响应大小 counter: nginx_http_requests_total，监控 http 请求次数 http { log_format main \u0026#39;$remote_addr - $remote_user [$time_local] \u0026#34;$request\u0026#34; \u0026#39; \u0026#39;$status $body_bytes_sent \u0026#34;$http_referer\u0026#34; \u0026#39; \u0026#39;\u0026#34;$http_user_agent\u0026#34; \u0026#34;$http_x_forwarded_for\u0026#34;\u0026#39;; access_log /var/log/nginx/access.log main; lua_shared_dict prometheus_metrics 10M; # lua_package_path \u0026#34;/path/to/nginx-lua-prometheus/?.lua;;\u0026#34;; init_worker_by_lua_block { prometheus = require(\u0026#34;prometheus\u0026#34;).init(\u0026#34;prometheus_metrics\u0026#34;) metric_bytes = prometheus:counter( \u0026#34;nginx_http_size_bytes\u0026#34;, \u0026#34;Total size of HTTP\u0026#34;, {\u0026#34;type\u0026#34;, \u0026#34;route\u0026#34;}) metric_requests = prometheus:counter( \u0026#34;nginx_http_requests_total\u0026#34;, \u0026#34;Number of HTTP requests\u0026#34;, {\u0026#34;status\u0026#34;, \u0026#34;route\u0026#34;}) metric_latency = prometheus:histogram( \u0026#34;nginx_http_latency\u0026#34;, \u0026#34;HTTP request latency\u0026#34;, {\u0026#34;route\u0026#34;}) metric_connections = prometheus:gauge( \u0026#34;nginx_http_connections\u0026#34;, \u0026#34;Number of HTTP connections\u0026#34;, {\u0026#34;state\u0026#34;}) } server { listen 8080; location /test { default_type application/json; return 200 \u0026#39;{\u0026#34;code\u0026#34;: 200, \u0026#34;message\u0026#34;: \u0026#34;success\u0026#34;}\u0026#39;; log_by_lua_block { metric_bytes:inc(tonumber(ngx.var.request_length), {\u0026#34;request\u0026#34;, \u0026#34;/test/**\u0026#34;}) metric_bytes:inc(tonumber(ngx.var.bytes_send), {\u0026#34;response\u0026#34;, \u0026#34;/test/**\u0026#34;}) metric_requests:inc(1, {ngx.var.status, \u0026#34;/test/**\u0026#34;}) metric_latency:observe(tonumber(ngx.var.request_time), {\u0026#34;/test/**\u0026#34;}) } } } server { listen 9145; location /metrics { content_by_lua_block { metric_connections:set(ngx.var.connections_reading, {\u0026#34;reading\u0026#34;}) metric_connections:set(ngx.var.connections_waiting, {\u0026#34;waiting\u0026#34;}) metric_connections:set(ngx.var.connections_writing, {\u0026#34;writing\u0026#34;}) prometheus:collect() } } } } 上面的例子中，我们暴露了 route 级别的指标，你也可以根据监控粒度的需要，选择暴露 host 指标：\nhttp { log_by_lua_block { metric_bytes:inc(tonumber(ngx.var.request_length), {\u0026#34;request\u0026#34;, ngx.var.host}) metric_bytes:inc(tonumber(ngx.var.bytes_send), {\u0026#34;response\u0026#34;, ngx.var.host}) metric_requests:inc(1, {ngx.var.status, ngx.var.host}) metric_latency:observe(tonumber(ngx.var.request_time), {ngx.var.host}) } } 或者 upstream 指标：\nupstream backend { server ip:port; } server { location /test_upstream { proxy_pass http://backend; log_by_lua_block { metric_bytes:inc(tonumber(ngx.var.request_length), {\u0026#34;request\u0026#34;, \u0026#34;upstream/backend\u0026#34;}) metric_bytes:inc(tonumber(ngx.var.bytes_send), {\u0026#34;response\u0026#34;, \u0026#34;upstream/backend\u0026#34;}) metric_requests:inc(1, {ngx.var.status, \u0026#34;upstream/backend\u0026#34;}) metric_latency:observe(tonumber(ngx.var.request_time), {\u0026#34;upstream/backend\u0026#34;}) } } } 完成指标定义后，我们启动 nginx 和 opentelemetry-collector，将指标采集到 SkyWalking 后端进行分析和存储。\n请确保job_name: 'nginx-monitoring'，否则上报的数据将被 SkyWalking 忽略。如果你有多个 Nginx 实例，你可以通过service及service_instance_id这两个 label 进行区分：\nreceivers: prometheus: config: scrape_configs: - job_name: \u0026#39;nginx-monitoring\u0026#39; scrape_interval: 5s metrics_path: \u0026#34;/metrics\u0026#34; static_configs: - targets: [\u0026#39;nginx:9145\u0026#39;] labels: service: nginx service_instance_id: nginx-instance processors: batch: exporters: otlp: endpoint: oap:11800 tls: insecure: true service: pipelines: metrics: receivers: - prometheus processors: - batch exporters: - otlp 如果一切顺利，你将在 skywalking-ui 的网关菜单下看到 nginx 上报的指标数据：\nAccess \u0026amp; Error Log 采集 SkyWalking Nginx 监控提供了日志采集及错误日志统计功能，我们可以借助 fluent-bit 采集并上报 access log、error log 给 SkyWalking 分析存储。\n下面 fluent-bit 配置定义了日志采集目录为/var/log/nginx/，access 和 error log 经过 rewrite_access_log 和 rewrite_error_log 处理后会通过 oap 12800 端口进行上报：\n[SERVICE] Flush 5 Daemon Off Log_Level warn [INPUT] Name tail Tag access Path /var/log/nginx/access.log [INPUT] Name tail Tag error Path /var/log/nginx/error.log [FILTER] Name lua Match access Script fluent-bit-script.lua Call rewrite_access_log [FILTER] Name lua Match error Script fluent-bit-script.lua Call rewrite_error_log [OUTPUT] Name stdout Match * Format json [OUTPUT] Name http Match * Host oap Port 12800 URI /v3/logs Format json 在 fluent-bit-script.lua 中，我们通过 LOG_KIND 来区分 access log 和 error log。\n为了能够关联上文采集的 metric，请确保 service 和 serviceInstance 值与上文中指标采集定义一致。\nfunction rewrite_access_log(tag, timestamp, record) local newRecord = {} newRecord[\u0026#34;layer\u0026#34;] = \u0026#34;NGINX\u0026#34; newRecord[\u0026#34;service\u0026#34;] = \u0026#34;nginx::nginx\u0026#34; newRecord[\u0026#34;serviceInstance\u0026#34;] = \u0026#34;nginx-instance\u0026#34; newRecord[\u0026#34;body\u0026#34;] = { text = { text = record.log } } newRecord[\u0026#34;tags\u0026#34;] = { data = {{ key = \u0026#34;LOG_KIND\u0026#34;, value = \u0026#34;NGINX_ACCESS_LOG\u0026#34;}}} return 1, timestamp, newRecord end function rewrite_error_log(tag, timestamp, record) local newRecord = {} newRecord[\u0026#34;layer\u0026#34;] = \u0026#34;NGINX\u0026#34; newRecord[\u0026#34;service\u0026#34;] = \u0026#34;nginx::nginx\u0026#34; newRecord[\u0026#34;serviceInstance\u0026#34;] = \u0026#34;nginx-instance\u0026#34; newRecord[\u0026#34;body\u0026#34;] = { text = { text = record.log } } newRecord[\u0026#34;tags\u0026#34;] = { data = {{ key = \u0026#34;LOG_KIND\u0026#34;, value = \u0026#34;NGINX_ERROR_LOG\u0026#34; }}} return 1, timestamp, newRecord end 启动 fluent-it 后，我们便可以在监控面板的 Log tab 看到采集到的日志信息：\n面板指标含义 面板名称 单位 指标含义 数据源 HTTP Request Trend 每秒钟平均请求数 nginx-lua-prometheus HTTP Latency ms 平均响应延时 nginx-lua-prometheus HTTP Bandwidth KB 请求响应流量 nginx-lua-prometheus HTTP Connections nginx http 连接数 nginx-lua-prometheus HTTP Status Trend % 每分钟 http 状态码统计 nginx-lua-prometheus HTTP Status 4xx Percent % 4xx状态码比例 nginx-lua-prometheus HTTP Status 5xx Percent % 5xx状态码比例 nginx-lua-prometheus Error Log Count 每分钟错误日志数统计 fluent-bit 参考文档 nginx-lua-prometheus fluent-bit-lua-filter skywalking-apisix-monitoring ","excerpt":"\u003ch2 id=\"背景介绍\"\u003e背景介绍\u003c/h2\u003e\n\u003cp\u003e在前面的 Blog \u003ca href=\"https://skywalking.apache.org/zh/2023-10-29-collect-and-analyse-nginx-accesslog-by-lal/\"\u003e使用 LAL 收集并分析 Nginx access log\u003c/a\u003e 中，我们以 Nginx access log 为切入点，\n演示了 SkyWalking LAL 的日志分析能力 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2023-12-23-monitoring-nginx-by-skywalking/","title":"使用 SkyWalking 监控 Nginx"},{"body":"🚀 Dive into the World of Cutting-Edge Technology with Apache\u0026rsquo;s Finest! 🌐 Join me today as we embark on an exhilarating journey with two of Apache\u0026rsquo;s most brilliant minds - Sheng Wu and Trista Pan. We\u0026rsquo;re exploring the realms of Apache SkyWalking and Apache ShardingSphere, two groundbreaking initiatives that are reshaping the landscape of open-source technology. 🌟\nIn this exclusive session, we delve deep into Apache SkyWalking - an innovative observability platform that\u0026rsquo;s revolutionizing how we monitor and manage distributed systems in the cloud. Witness firsthand how SkyWalking is empowering developers and organizations to gain unparalleled insights into their applications, ensuring performance, reliability, and efficient troubleshooting. 🛰️🔍\nBut there\u0026rsquo;s more! We\u0026rsquo;re also unveiling the secrets of Apache ShardingSphere, a dynamic distributed database ecosystem. Learn how ShardingSphere is making waves in the world of big data, offering scalable, high-performance solutions for data sharding, encryption, and more. This is your gateway to understanding how these technologies are pivotal in handling massive data sets across various industries. 🌐💾\nWhether you\u0026rsquo;re a developer, tech enthusiast, or just curious about the future of open-source technology, this is a conversation you don\u0026rsquo;t want to miss! Get ready to be inspired and informed as we unlock new possibilities and applications of Apache SkyWalking and ShardingSphere. 🚀🌟\nJoin us, and let\u0026rsquo;s decode the future together!\nPlease join and follow Josh\u0026rsquo;s 龙之春 Youtube Coffee + Software with Josh Long Channel to learn more about technology and open source from telanted engineers and industry leads.\n","excerpt":"\u003cp\u003e🚀 Dive into the World of Cutting-Edge Technology with Apache\u0026rsquo;s Finest! 🌐 Join me today as we …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2023-12-04-coffee+software-with-josh-long/","title":"[Video] Coffee + Software with Josh Long - Apache SkyWalking with Sheng Wu and Apache ShardingSphere with Trista Pan"},{"body":"SkyWalking CLI 0.13.0 is released. Go to downloads page to find release tars.\nFeatures Add the sub-command menu get for get the ui menu items by @mrproliu in https://github.com/apache/skywalking-cli/pull/187 Bug Fixes Fix the record list query does not support new OAP versions (with major version number \u0026gt; 9). ","excerpt":"\u003cp\u003eSkyWalking CLI 0.13.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"features\"\u003eFeatures\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eAdd the …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cli-0-13-0/","title":"Release Apache SkyWalking CLI 0.13.0"},{"body":"SkyWalking Java Agent 9.1.0 is released. Go to downloads page to find release tars. Changes by Version\n9.1.0 Fix hbase onConstruct NPE in the file configuration scenario Fix the issue of createSpan failure caused by invalid request URL in HttpClient 4.x/5.x plugin Optimize ElasticSearch 6.x 7.x plugin compatibility Fix an issue with the httpasyncclient component where the isError state is incorrect. Support customization for the length limitation of string configurations Add max length configurations in agent.config file for service_name and instance_name Optimize spring-cloud-gateway 2.1.x, 3.x witness class. Support report MongoDB instance info in Mongodb 4.x plugin. To compatible upper and lower case Oracle TNS url parse. Support collecting ZGC memory pool metrics. Require OAP 9.7.0 to support these new metrics. Upgrade netty-codec-http2 to 4.1.100.Final Add a netty-http 4.1.x plugin to trace HTTP requests. Fix Impala Jdbc URL (including schema without properties) parsing exception. Optimize byte-buddy type description performance. Add eclipse-temurin:21-jre as another base image. Bump byte-buddy to 1.14.9 for JDK21 support. Add JDK21 plugin tests for Spring 6. Bump Lombok to 1.18.30 to adopt JDK21 compiling. Fix PostgreSQL Jdbc URL parsing exception. Bump up grpc version. Optimize plugin selector logic. Documentation Fix JDK requirement in the compiling docs. Add JDK21 support in the compiling docs. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking Java Agent 9.1.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nChanges by …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-java-agent-9-1-0/","title":"Release Apache SkyWalking Java Agent 9.1.0"},{"body":"SkyWalking 9.7.0 is released. Go to downloads page to find release tars.\nDark Mode The dafult style mode is changed to the dark mode, and light mode is still available.\nNew Design Log View A new design for the log view is currently available. Easier to locate the logs, and more space for the raw text.\nProject Bump Java agent to 9.1-dev in the e2e tests. Bump up netty to 4.1.100. Update Groovy 3 to 4.0.15. Support packaging the project in JDK21. Compiler source and target remain in JDK11. OAP Server ElasticSearchClient: Add deleteById API. Fix Custom alarm rules are overwritten by \u0026lsquo;resource/alarm-settings.yml\u0026rsquo; Support Kafka Monitoring. Support Pulsar server and BookKeeper server Monitoring. [Breaking Change] Elasticsearch storage merge all management data indices into one index management, including ui_template，ui_menu，continuous_profiling_policy. Add a release mechanism for alarm windows when it is expired in case of OOM. Fix Zipkin trace receiver response: make the HTTP status code from 200 to 202. Update BanyanDB Java Client to 0.5.0. Fix getInstances query in the BanyanDB Metadata DAO. BanyanDBStorageClient: Add keepAliveProperty API. Fix table exists check in the JDBC Storage Plugin. Enhance extensibility of HTTP Server library. Adjust AlarmRecord alarmMessage column length to 512. Fix EventHookCallback build event: build the layer from Service's Layer. Fix AlarmCore doAlarm: catch exception for each callback to avoid interruption. Optimize queryBasicTraces in TraceQueryEsDAO. Fix WebhookCallback send incorrect messages, add catch exception for each callback HTTP Post. Fix AlarmRule expression validation: add labeled metrics mock data for check. Support collect ZGC memory pool metrics. Add a component ID for Netty-http (ID=151). Add a component ID for Fiber (ID=5021). BanyanDBStorageClient: Add define(Property property, PropertyStore.Strategy strategy) API. Correct the file format and fix typos in the filenames for monitoring Kafka\u0026rsquo;s e2e tests. Support extract timestamp from patterned datetime string in LAL. Support output key parameters in the booting logs. Fix cannot query zipkin traces with annotationQuery parameter in the JDBC related storage. Fix limit doesn\u0026rsquo;t work for findEndpoint API in ES storage. Isolate MAL CounterWindow cache by metric name. Fix JDBC Log query order. Change the DataCarrier IF_POSSIBLE strategy to use ArrayBlockingQueue implementation. Change the policy of the queue(DataCarrier) in the L1 metric aggregate worker to IF_POSSIBLE mode. Add self-observability metric metrics_aggregator_abandon to count the number of abandon metrics. Support Nginx monitoring. Fix BanyanDB Metadata Query: make query single instance/process return full tags to avoid NPE. Repleace go2sky E2E to GO agent. Replace Metrics v2 protocol with MQE in UI templates and E2E Test. Fix incorrect apisix metrics otel rules. Support Scratch The OAP Config Dump. Support increase/rate function in the MQE query language. Group service endpoints into _abandoned when endpoints have high cardinality. UI Add new menu for kafka monitoring. Fix independent widget duration. Fix the display height of the link tree structure. Replace the name by shortName on service widget. Refactor: update pagination style. No visualization style change. Apply MQE on K8s layer UI-templates. Fix icons display in trace tree diagram. Fix: update tooltip style to support multiple metrics scrolling view in a metrics graph. Add a new widget to show jvm memory pool detail. Fix: avoid querying data with empty parameters. Add a title and a description for trace segments. Add Netty icon for Netty HTTP plugin. Add Pulsar menu i18n files. Refactor Logs view. Implement the Dark Theme. Change UI templates for Text widgets. Add Nginx menu i18n. Fix the height for trace widget. Polish list style. Fix Log associate with Trace. Enhance layout for broken Topology widget. Fix calls metric with call type for Topology widget. Fix changing metrics config for Topology widget. Fix routes for Tab widget. Remove OpenFunction(FAAS layer) relative UI templates and menu item. Fix: change colors to match dark theme for Network Profiling. Remove the description of OpenFunction in the UI i18n. Reduce component chunks to improve page loading resource time. Documentation Separate storage docs to different files, and add an estimated timeline for BanyanDB(end of 2023). Add topology configuration in UI-Grafana doc. Add missing metrics to the OpenTelemetry Metrics doc. Polish docs of Concepts and Designs. Fix incorrect notes of slowCacheReadThreshold. Update OAP setup and cluster coordinator docs to explain new booting parameters table in the logs, and how to setup cluster mode. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking 9.7.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"dark-mode\"\u003eDark Mode\u003c/h4\u003e\n\u003cp\u003eThe dafult style …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-9.7.0/","title":"Release Apache SkyWalking APM 9.7.0"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/conference/","title":"Conference"},{"body":"SkyWalking Summit 2023 @ Shanghai 会议时间：2023年11月4日 全天 地点：上海大华虹桥假日酒店 赞助商：纵目科技，Tetrate\n会议议程 与 PDF SkyWalking V9 In 2023 - 5 featured releases 吴晟 PDF B站视频地址\n使用 Terraform 与 Ansible 快速部署 SkyWalking 集群 柯振旭 PDF B站视频地址\n基于SkyWalking构建全域一体化观测平台 陈修能 PDF B站视频地址\n云原生可观测性数据库BanyanDB 高洪涛 PDF B站视频地址\n基于 SkyWalking Agent 的性能剖析和实时诊断 陆家靖 PDF B站视频地址\n太保科技-多云环境下Zabbix的运用实践 田川 PDF B站视频地址\nKubeSphere 在可观测性领域的探索与实践 霍秉杰 PDF B站视频地址\n大型跨国企业的微服务治理 张文杰 PDF B站视频地址\n","excerpt":"\u003ch1 id=\"skywalking-summit-2023--shanghai\"\u003eSkyWalking Summit 2023 @ Shanghai\u003c/h1\u003e\n\u003cimg src=\"banner.jpg\"\u003e\n\u003cp\u003e会议时间：2023年11月4日 全天\n地点：上海大华虹桥假日酒店\n赞助商：纵目科技，Tetrate\u003c/p\u003e\n\u003ch1 id=\"会议议程-与-pdf\"\u003e会议议程 与 PDF …\u003c/h1\u003e","ref":"https://skywalking.apache.org/zh/2023-11-04-skywalking-summit-shanghai/","title":"SkyWalking Summit 2023 @ Shanghai 会议回顾"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/video/","title":"Video"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/","title":"Zh_tags"},{"body":"SkyWalking Infra E2E 1.3.0 is released. Go to downloads page to find release tars.\nFeatures Support sha256enc and sha512enc encoding in verify case. Support hasPrefix and hasSuffix string verifier in verify case. Bump up kind to v0.14.0. Add a field kubeconfig to support running e2e test on an existing kubernetes cluster. Support non-fail-fast execution of test cases support verify cases concurrently Add .exe suffix to windows build artifact Export the kubeconfig path during executing the following steps Automatically pull images before loading into KinD Support outputting the result of \u0026lsquo;verify\u0026rsquo; in YAML format and only outputting the summary of the result of \u0026lsquo;verify\u0026rsquo; Make e2e test itself in github action Support outputting the summary of \u0026lsquo;verify\u0026rsquo; in YAML format Make e2e output summary with numeric information Add \u0026lsquo;subtractor\u0026rsquo; function Improvements Bump up GHA to avoid too many warnings Leverage the built-in cache in setup-go@v4 Add batchOutput config to reduce outputs Disable batch mode by default, add it to GHA and enable by default Improve GitHub Actions usability and speed by using composite actions\u0026rsquo; new feature Migrate deprecated GitHub Actions command to recommended ones Bump up kind to v0.14.0 Optimization of the output information of verification verifier: notEmpty should be able to handle nil Remove invalid configuration in GitHub Actions Bug Fixes Fix deprecation warnings Ignore cancel error when copying container logs Documentation Add a doc to introduce how to use e2e to test itself Issues and PR All issues are here All pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Infra E2E 1.3.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-infra-e2e-1-3-0/","title":"Release Apache SkyWalking Infra E2E 1.3.0"},{"body":"Aapche SkyWalking PMC 和 committer团队参加了\u0026quot;开源之夏 2023\u0026quot;活动，作为导师，共获得了9个官方赞助名额。最终对学生开放如下任务\nSkyWalking 支持 GraalVM Skywalking Infra E2E 自测试 监控Apache Pulsar 统一BanyanDB的查询计划和查询执行器 使用Helm部署BanyanDB 编写go agent的gRPC插件 监控Kafka 集成SkyWalking PHP到SkyWalking E2E 测试 在线黄金指标异常检测 经过3个月的开发，上游评审，PMC成员评议，PMC Chair复议，OSPP官方委员会评审多个步骤，现公布项目参与人员与最终结果\n通过评审项目（共6个） SkyWalking 支持 GraalVM 学生：张跃骎 学校：辽宁大学 本科 合并PR：11354 后续情况说明：GraalVM因为复杂的生态，替代的代码将被分离到SkyWalking GraalVM Distro, 相关讨论，请参见Issue 11518 Skywalking Infra E2E 自测试 学生：王子忱 学校：华中师范大学 本科 合并PR：115, 116, 117, 118, 119 后续情况说明：此特性已经包含在发行版skywalking-infra-e2e v1.3.0中 统一BanyanDB的查询计划和查询执行器 学生：曾家华 学校：电子科技大学 本科 合并PR：343 使用Helm部署BanyanDB 学生：黄友亮 学校：北京邮电大学 硕士研究生 合并PR：1 情况说明：因为BanyanDB Helm为新项目，学生承接了项目初始化、功能提交、自动化测试，发布准备等多项任务。所参与功能包含在skywalking-banyandb-helm v0.1.0中 编写go agent的gRPC插件 学生：胡宇腾 学校：西安邮电大学 合并PR：88, 94 后续情况说明：该学生在开源之夏相关项目外，完成了feature: add support for iris #99和Go agent APIs功能开发。并发表文章SkyWalking Go Toolkit Trace 详解以及英文译本Detailed explanation of SkyWalking Go Toolkit Trace 监控Kafka 学生：王竹 学校：美国东北大学 ( Northeastern University) 合并PR：11282, UI 318 未通过评审项目（3个） 下列项目因为质量无法达到社区要求，违规等原因，将被标定为失败。 注：在开源之夏中失败的项目，其Pull Reqeust可能因为符合社区功能要求，也被接受合并。\n监控Apache Pulsar 学生：孟祥迎 学校：重庆邮电大学 本科 合并PR：11339 失败原因：项目申请成员，作为ASF Pulsar项目的Committer，在担任Pulsar开源之夏项目导师期间，但依然申请了学生参与项目。属于违规行为。SkyWalking PMC审查了此行为并通报开源之夏组委会。开源之夏组委会依据活动规则取消其结项奖金。 集成SkyWalking PHP到SkyWalking E2E 测试 学生：罗文 学校：San Jose State University B.S. 合并PR：11330 失败原因：根据pull reqeust中的提交记录，SkyWalking PMC Chair审查了提交明细，学生参与代码数量大幅度小于导师的提交代码。并在考虑到这个项目难度以及明显低于SkyWalking 开源之夏项目的平均水平的情况下，通报给开源之夏组委会。经过组委会综合评定，项目不合格。 在线黄金指标异常检测 学生：黄颖 学校：同济大学 研究生 合并PR：无 失败原因：项目在进度延迟后实现较为简单且粗糙，并且没有提供算法评估结果和文档等。在 PR 开启后的为期一个月审核合并期间，学生并未能成功按预定计划改善实现的质量和文档。和导师以及 SkyWalking 社区缺少沟通。 结语 SkyWalking社区每年都有近10位PMC成员或Committer参与开源之夏中，帮助在校学生了解顶级开源项目、开源社区的运作方式。我们希望大家在每年经过3个月的时间，能够真正的帮助在校学生了解开源和参与开源。 因为，社区即使在考虑到学生能力的情况下，不会明显的降低pull request的接受标准。希望今后的学生，能够在早期，积极、主动和导师，社区其他成员保持高频率的沟通，对参与的项目有更深入、准确的了解。\n","excerpt":"\u003cp\u003eAapche SkyWalking PMC 和 committer团队参加了\u0026quot;开源之夏 2023\u0026quot;活动，作为导师，共获得了9个官方赞助名额。最终对学生开放如下任务 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2023-11-09-ospp-summary/","title":"开源之夏 2023 SkyWalking 社区项目情况公示"},{"body":"SkyWalking NodeJS 0.7.0 is released. Go to downloads page to find release tars.\nAdd deadline config for trace request (#118) ","excerpt":"\u003cp\u003eSkyWalking NodeJS 0.7.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAdd deadline config …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-nodejs-0-7-0/","title":"Release Apache SkyWalking for NodeJS 0.7.0"},{"body":"背景介绍 Nginx access log 中包含了丰富的信息，例如：日志时间、状态码、响应时间、body 大小等。通过收集并分析 access log，我们可以实现对 Nginx 中接口状态的监控。\n在本案例中，将由 fluent-bit 收集 access log，并通过 HTTP 将日志信息发送给 SkyWalking OAP Server 进行进一步的分析。\n环境准备 实验需要的 Nginx 及 Fluent-bit 相关配置文件都被上传到了Github，有需要的读者可以自行 git clone 并通过 docker compose 启动，本文中将介绍配置文件中几个关键点。\nNginx日志格式配置 LAL 目前支持 JSON、YAML 及 REGEX 日志解析，为了方便获取到日志中的指标字段，我们将 Nginx 的日志格式定义为 JSON.\nhttp { ... ... log_format main \u0026#39;{\u0026#34;remote_addr\u0026#34;: \u0026#34;$remote_addr\u0026#34;,\u0026#39; \u0026#39;\u0026#34;remote_user\u0026#34;: \u0026#34;$remote_user\u0026#34;,\u0026#39; \u0026#39;\u0026#34;request\u0026#34;: \u0026#34;$request\u0026#34;,\u0026#39; \u0026#39;\u0026#34;time\u0026#34;: \u0026#34;$time_iso8601\u0026#34;,\u0026#39; \u0026#39;\u0026#34;status\u0026#34;: \u0026#34;$status\u0026#34;,\u0026#39; \u0026#39;\u0026#34;request_time\u0026#34;:\u0026#34;$request_time\u0026#34;,\u0026#39; \u0026#39;\u0026#34;body_bytes_sent\u0026#34;: \u0026#34;$body_bytes_sent\u0026#34;,\u0026#39; \u0026#39;\u0026#34;http_referer\u0026#34;: \u0026#34;$http_referer\u0026#34;,\u0026#39; \u0026#39;\u0026#34;http_user_agent\u0026#34;: \u0026#34;$http_user_agent\u0026#34;,\u0026#39; \u0026#39;\u0026#34;http_x_forwarded_for\u0026#34;: \u0026#34;$http_x_forwarded_for\u0026#34;}\u0026#39;; access_log /var/log/nginx/access.log main; ... ... } Fluent bit Filter 我们通过 Fluent bit 的 lua filter 进行日志格式的改写，将其调整为 SkyWalking 所需要的格式，record的各个字段含义如下：\nbody：日志内容体 service：服务名称 serviceInstance：实例名称 function rewrite_body(tag, timestamp, record) local newRecord = {} newRecord[\u0026#34;body\u0026#34;] = { json = { json = record.log } } newRecord[\u0026#34;service\u0026#34;] = \u0026#34;nginx::nginx\u0026#34; newRecord[\u0026#34;serviceInstance\u0026#34;] = \u0026#34;localhost\u0026#34; return 1, timestamp, newRecord end OAP 日志分析 LAL定义 在 filter 中，我们通过条件判断，只处理 service=nginx::nginx 的服务，其他服务依旧走默认逻辑：\n第一步，使用 json 指令对日志进行解析，解析的结果会被存放到 parsed 字段中，通过 parsed 字段我们可以获取 json 日志中的字段信息。\n第二步，使用 timestamp 指令解析 parsed.time 并将其赋值给日志的 timestamp 字段，这里的 time 就是access log json 中的 time。\n第三步，使用 tag 指令给日志打上对应的标签，标签的值依然可以通过 parsed 字段获取。\n第四步，使用 metrics 指令从日志中提取出指标信息，我们共提取了四个指标：\nnginx_log_count：Nginx 每次请求都会生成一条 access log，该指标可以帮助我们统计 Nginx 当前的请求数。 nginx_request_time：access log 中会记录请求时间，该指标可以帮助我们统计上游接口的响应时长。 nginx_body_bytes_sent：body 大小指标可以帮助我们了解网关上的流量情况。 nginx_status_code：状态码指标可以实现对状态码的监控，如果出现异常上涨可以结合 alarm 进行告警。 rules: - name: default layer: GENERAL dsl: | filter { if (log.service == \u0026#34;nginx::nginx\u0026#34;) { json { abortOnFailure true } extractor { timestamp parsed.time as String, \u0026#34;yyyy-MM-dd\u0026#39;T\u0026#39;HH:mm:ssXXX\u0026#34; tag status: parsed.status tag remote_addr: parsed.remote_addr metrics { timestamp log.timestamp as Long labels service: log.service, instance: log.serviceInstance name \u0026#34;nginx_log_count\u0026#34; value 1 } metrics { timestamp log.timestamp as Long labels service: log.service, instance: log.serviceInstance name \u0026#34;nginx_request_time\u0026#34; value parsed.request_time as Double } metrics { timestamp log.timestamp as Long labels service: log.service, instance: log.serviceInstance name \u0026#34;nginx_body_bytes_sent\u0026#34; value parsed.body_bytes_sent as Long } metrics { timestamp log.timestamp as Long labels service: log.service, instance: log.serviceInstance, status: parsed.status name \u0026#34;nginx_status_code\u0026#34; value 1 } } } sink { } } 经过 LAL 处理后，我们已经可以在日志面板看到日志信息了，接下来我们将对 LAL 中提取的指标进行进一步分析：\nMAL定义 在 MAL 中，我们可以对上一步 LAL 中提取的指标进行进一步的分析聚合，下面的例子里：\nnginx_log_count、nginx_request_time、nginx_status_code 使用 sum 聚合函数处理，并使用 SUM 方式 downsampling，\nnginx_request_time 使用 avg 聚合函数求平均值，默认使用 AVG 方式 downsampling。\n完成聚合分析后，SkyWalking Meter System 会完成对上述指标的持久化。\nexpSuffix: service([\u0026#39;service\u0026#39;], Layer.GENERAL) metricPrefix: nginx metricsRules: - name: cpm exp: nginx_log_count.sum([\u0026#39;service\u0026#39;]).downsampling(SUM) - name: avg_request_time exp: nginx_request_time.avg([\u0026#39;service\u0026#39;]) - name: body_bytes_sent_count exp: nginx_body_bytes_sent.sum([\u0026#39;service\u0026#39;]).downsampling(SUM) - name: status_code_count exp: nginx_status_code.sum([\u0026#39;service\u0026#39;,\u0026#39;status\u0026#39;]).downsampling(SUM) 最后，我们便可以来到 SkyWalking UI 页面新建 Nginx 仪表板，使用刚刚 MAL 中定义的指标信息创建 Nginx Dashboard（也可以通过上文提到仓库中的 dashboard.json 直接导入测试）：\n参考文档 Fluent Bit lua Filter Log Analysis Language Meter Analysis Language ","excerpt":"\u003ch2 id=\"背景介绍\"\u003e背景介绍\u003c/h2\u003e\n\u003cp\u003eNginx access log 中包含了丰富的信息，例如：日志时间、状态码、响应时间、body 大小等。通过收集并分析 access log，我们可以实现对 Nginx 中接口状态的监控。 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2023-10-29-collect-and-analyse-nginx-accesslog-by-lal/","title":"使用 LAL 收集并分析 Nginx access log"},{"body":"SkyWalking BanyanDB 0.5.0 is released. Go to downloads page to find release tars.\nFeatures List all properties in a group. Implement Write-ahead Logging Document the clustering. Support multiple roles for banyand server. Support for recovery buffer using wal. Register the node role to the metadata registry. Implement the remote queue to spreading data to data nodes. Fix parse environment variables error Implement the distributed query engine. Add mod revision check to write requests. Add TTL to the property. Implement node selector (e.g. PickFirst Selector, Maglev Selector). Unified the buffers separated in blocks to a single buffer in the shard. Bugs BanyanDB ui unable to load icon. BanyanDB ui type error Fix timer not released BanyanDB ui misses fields when creating a group Fix data duplicate writing Syncing metadata change events from etcd instead of a local channel. Chores Bump several dependencies and tools. Drop redundant \u0026ldquo;discovery\u0026rdquo; module from banyand. \u0026ldquo;metadata\u0026rdquo; module is enough to play the node and shard discovery role. ","excerpt":"\u003cp\u003eSkyWalking BanyanDB 0.5.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"features\"\u003eFeatures\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eList all …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-0-5-0/","title":"Release Apache SkyWalking BanyanDB 0.5.0"},{"body":"SkyWalking Go 0.3.0 is released. Go to downloads page to find release tars.\nFeatures Support manual tracing APIs for users. Plugins Support mux HTTP server framework. Support grpc server and client framework. Support iris framework. Documentation Add Tracing APIs document into Manual APIs. Bug Fixes Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Go 0.3.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport manual …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-go-0.3.0/","title":"Release Apache SkyWalking Go 0.3.0"},{"body":"Background SkyWalking Go is an open-source, non-intrusive Golang agent used for monitoring, tracing, and data collection within distributed systems. It enables users to observe the flow and latency of requests within the system, collect performance data from various system components for performance monitoring, and troubleshoot issues by tracing the complete path of requests.\nIn version v0.3.0, Skywalking Go introduced the toolkit trace tool. Trace APIs allow users to include critical operations, functions, or services in the tracing scope in situations where plugins do not support them. This inclusion enables tracking and monitoring of these operations and can be used for fault analysis, diagnosis, and performance monitoring.\nBefore diving into this, you can learn how to use the Skywalking Go agent by referring to the SkyWalking Go Agent Quick Start Guide.\nThe following sections will explain how to use these interfaces in specific scenarios.\nIntroducing the Trace Toolkit Execute the following command in the project\u0026rsquo;s root directory:\ngo get github.com/apache/skywalking-go/toolkit To use the toolkit trace interface, you need to import the package into your project:\n\u0026#34;github.com/apache/skywalking-go/toolkit/trace\u0026#34; Manual Tracing A Span is the fundamental unit of an operation in Tracing. It represents an operation within a specific timeframe, such as a request, a function call, or a specific action. It records essential information about a particular operation, including start and end times, the operation\u0026rsquo;s name, tags (key-value pairs), and relationships between operations. Multiple Spans can form a hierarchical structure.\nIn situations where Skywalking-go doesn\u0026rsquo;t support a particular framework, users can manually create Spans to obtain tracing information.\n(Here, I have removed the supported frameworks for the sake of the example. These are only examples. You should reference this when using the APIs in private and/or unsupported frameworks)\nFor example, when you need to trace an HTTP response, you can create a span using trace.CreateEntrySpan() within the method handling the request, and end the span using trace.StopSpan() after processing. When sending an HTTP request, use trace.CreateExitSpan() to create a span, and end the span after the request returns.\nHere are two HTTP services named consumer and provider. When a user accesses the consumer service, it receives the user\u0026rsquo;s request internally and then accesses the provider to obtain resources.\n// consumer.go package main import ( \u0026#34;io\u0026#34; \u0026#34;net/http\u0026#34; _ \u0026#34;github.com/apache/skywalking-go\u0026#34; \u0026#34;github.com/apache/skywalking-go/toolkit/trace\u0026#34; ) func getProvider() (*http.Response, error) { // Create an HTTP request req, err := http.NewRequest(\u0026#34;GET\u0026#34;, \u0026#34;http://localhost:9998/provider\u0026#34;, http.NoBody) // Create an ExitSpan before sending the HTTP request. trace.CreateExitSpan(\u0026#34;GET:/provider\u0026#34;, \u0026#34;localhost:9999\u0026#34;, func(headerKey, headerValue string) error { // Injector adds specific header information to the request. req.Header.Add(headerKey, headerValue) return nil }) // Finish the ExitSpan and ensure it executes when the function returns using defer. defer trace.StopSpan() // Send the request. client := \u0026amp;http.Client{} resp, err := client.Do(req) if err != nil { return nil, err } return resp, nil } func consumerHandler(w http.ResponseWriter, r *http.Request) { // Create an EntrySpan to trace the execution of the consumerHandler method. trace.CreateEntrySpan(r.Method+\u0026#34;/consumer\u0026#34;, func(headerKey string) (string, error) { // Extractor retrieves the header information added to the request. return r.Header.Get(headerKey), nil }) // Finish the EntrySpan. defer trace.StopSpan() // Prepare to send an HTTP request. resp, err := getProvider() body, err := io.ReadAll(resp.Body) if err != nil { return } _, _ = w.Write(body) } func main() { http.HandleFunc(\u0026#34;/consumer\u0026#34;, consumerHandler) _ = http.ListenAndServe(\u0026#34;:9999\u0026#34;, nil) } // provider.go package main import ( \u0026#34;net/http\u0026#34; _ \u0026#34;github.com/apache/skywalking-go\u0026#34; \u0026#34;github.com/apache/skywalking-go/toolkit/trace\u0026#34; ) func providerHandler(w http.ResponseWriter, r *http.Request) { //Create an EntrySpan to trace the execution of the providerHandler method. trace.CreateEntrySpan(\u0026#34;GET:/provider\u0026#34;, func(headerKey string) (string, error) { return r.Header.Get(headerKey), nil }) // Finish the EntrySpan. defer trace.StopSpan() _, _ = w.Write([]byte(\u0026#34;success from provider\u0026#34;)) } func main() { http.HandleFunc(\u0026#34;/provider\u0026#34;, providerHandler) _ = http.ListenAndServe(\u0026#34;:9998\u0026#34;, nil) } Then, in the terminal, execute:\ngo build -toolexec=\u0026#34;/path/to/go-agent\u0026#34; -a -o consumer ./consumer.go ./consumer go build -toolexec=\u0026#34;/path/to/go-agent\u0026#34; -a -o provider ./provider.go ./provider curl 127.0.0.1:9999/consumer At this point, the UI will display the span information you created.\nIf you need to trace methods that are executed only locally, you can use trace.CreateLocalSpan(). If you don\u0026rsquo;t need to monitor information or states from the other end, you can change ExitSpan and EntrySpan to LocalSpan.\nThe usage examples provided are for illustration purposes, and users can decide the tracing granularity and where in the program they need tracing.\nPlease note that if a program ends too quickly, it may cause tracing data to be unable to be asynchronously sent to the SkyWalking backend.\nPopulate The Span When there\u0026rsquo;s a necessity to record additional information, including creating/updating tags, appending logs, and setting a new operation name of the current traced Span, these APIs should be considered. These actions are used to enhance trace information, providing a more detailed and precise contextual description, which aids in better understanding the events or operations being traced.\nToolkit trace APIs provide a convenient way to access and manipulate trace data, including:\nSetting Tags: SetTag() Adding Logs: AddLog() Setting Span Names: SetOperationName() Getting various IDs: GetTraceID(), GetSegmentID(), GetSpanID() For example, if you need to record the HTTP status code in a span, you can use the following interfaces while the span is not yet finished:\ntrace.CreateExitSpan(\u0026#34;GET:/provider\u0026#34;, \u0026#34;localhost:9999\u0026#34;, func(headerKey, headerValue string) error { r.Header.Add(headerKey, headerValue) return nil }) resp, err := http.Get(\u0026#34;http://localhost:9999/provider\u0026#34;) trace.SetTag(\u0026#34;status_code\u0026#34;, fmt.Sprintf(\u0026#34;%d\u0026#34;, resp.StatusCode)) spanID := trace.GetSpanID() trace.StopSpan() It\u0026rsquo;s important to note that when making these method calls, the current thread should have an active span.\nAsync APIs Async APIs work for manipulating spans across Goroutines. These scenarios might include:\nApplications involving concurrency or multiple goroutines where operating on Spans across different execution contexts is necessary. Updating or logging information for a Span during asynchronous operations. Requiring a delayed completion of a Span. To use it, follow these steps:\nObtain the return value of CreateSpan, which is SpanRef. Call spanRef.PrepareAsync() to prepare for operations in another goroutine. When the current goroutine\u0026rsquo;s work is done, call trace.StopSpan() to end the span (affecting only in the current goroutine). Pass the spanRef to another goroutine. After the work is done in any goroutine, call spanRef.AsyncFinish(). Here\u0026rsquo;s an example:\nspanRef, err := trace.CreateLocalSpan(\u0026#34;LocalSpan\u0026#34;) if err != nil { return } spanRef.PrepareAsync() go func(){ // some work spanRef.AsyncFinish() }() // some work trace.StopSpan() Correlation Context Correlation Context is used to pass parameters within a Span, and the parent Span will pass the Correlation Context to all its child Spans. It allows the transmission of information between spans across different applications. The default number of elements in the Correlation Context is 3, and the content\u0026rsquo;s length cannot exceed 128 bytes.\nCorrelation Context is commonly applied in the following scenarios:\nPassing Information Between Spans: It facilitates the transfer of critical information between different Spans, enabling upstream and downstream Spans to understand the correlation and context between each other. Passing Business Parameters: In business scenarios, it involves transmitting specific parameters or information between different Spans, such as authentication tokens, business transaction IDs, and more. Users can set the Correlation Context using trace.SetCorrelation(key, value) and then retrieve the corresponding value in downstream spans using value := trace.GetCorrelation(key).\nFor example, in the code below, we store the value in the tag of the span, making it easier to observe the result:\npackage main import ( _ \u0026#34;github.com/apache/skywalking-go\u0026#34; \u0026#34;github.com/apache/skywalking-go/toolkit/trace\u0026#34; \u0026#34;net/http\u0026#34; ) func providerHandler(w http.ResponseWriter, r *http.Request) { ctxValue := trace.GetCorrelation(\u0026#34;key\u0026#34;) trace.SetTag(\u0026#34;result\u0026#34;, ctxValue) } func consumerHandler(w http.ResponseWriter, r *http.Request) { trace.SetCorrelation(\u0026#34;key\u0026#34;, \u0026#34;value\u0026#34;) _, err := http.Get(\u0026#34;http://localhost:9999/provider\u0026#34;) if err != nil { return } } func main() { http.HandleFunc(\u0026#34;/provider\u0026#34;, providerHandler) http.HandleFunc(\u0026#34;/consumer\u0026#34;, consumerHandler) _ = http.ListenAndServe(\u0026#34;:9999\u0026#34;, nil) } Then, in the terminal, execute:\nexport SW_AGENT_NAME=server go build -toolexec=\u0026#34;/path/to/go-agent\u0026#34; -a -o server ./server.go ./server curl 127.0.0.1:9999/consumer Finally, in the providerHandler() span, you will find the information from the Correlation Context:\nConclusion This article provides an overview of Skywalking Go\u0026rsquo;s Trace APIs and their practical application. These APIs empower users with the ability to customize tracing functionality according to their specific needs.\nFor detailed information about the interfaces, please refer to the documentation: Tracing APIs.\nWelcome everyone to try out the new version.\n","excerpt":"\u003ch2 id=\"background\"\u003eBackground\u003c/h2\u003e\n\u003cp\u003eSkyWalking Go is an open-source, non-intrusive Golang agent used for monitoring, tracing, …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2023-10-18-skywalking-toolkit-trace/","title":"Detailed explanation of SkyWalking Go Toolkit Trace"},{"body":"背景介绍 SkyWalking Go是一个开源的非侵入式Golang代理程序，用于监控、追踪和在分布式系统中进行数据收集。它使用户能够观察系统内请求的流程和延迟，从各个系统组件收集性能数据以进行性能监控，并通过追踪请求的完整路径来解决问题。\n在版本v0.3.0中，Skywalking Go引入了 toolkit-trace 工具。Trace APIs 允许用户在插件不支持的情况下将关键操作、函数或服务添加到追踪范围。从而实现追踪和监控这些操作，并可用于故障分析、诊断和性能监控。\n在深入了解之前，您可以参考SkyWalking Go Agent快速开始指南来学习如何使用SkyWalking Go Agent。\n下面将会介绍如何在特定场景中使用这些接口。\n导入 Trace Toolkit 在项目的根目录中执行以下命令：\ngo get github.com/apache/skywalking-go/toolkit 使用 toolkit trace 接口前，需要将该包导入到您的项目中：\n\u0026#34;github.com/apache/skywalking-go/toolkit/trace\u0026#34; 手动追踪 Span 是 Tracing 中单个操作的基本单元。它代表在特定时间范围内的操作，比如一个请求、一个函数调用或特定动作。Span记录了特定操作的关键信息，包括开始和结束时间、操作名称、标签（键-值对）以及操作之间的关系。多个 Span 可以形成层次结构。\n在遇到 Skywalking Go 不支持的框架的情况下，用户可以手动创建 Span 以获取追踪信息。\n（为了作为示例，我删除了已支持的框架。以下仅为示例。请在使用私有或不支持的框架的 API 时参考）\n例如，当需要追踪HTTP响应时，可以在处理请求的方法内部使用 trace.CreateEntrySpan() 来创建一个 span，在处理完成后使用 trace.StopSpan() 来结束这个 span。在发送HTTP请求时，使用 trace.CreateExitSpan() 来创建一个 span，在请求返回后结束这个 span。\n这里有两个名为 consumer 和 provider 的HTTP服务。当用户访问 consumer 服务时，它在内部接收用户的请求，然后访问 provider 以获取资源。\n// consumer.go package main import ( \u0026#34;io\u0026#34; \u0026#34;net/http\u0026#34; _ \u0026#34;github.com/apache/skywalking-go\u0026#34; \u0026#34;github.com/apache/skywalking-go/toolkit/trace\u0026#34; ) func getProvider() (*http.Response, error) { // 新建 HTTP 请求 req, err := http.NewRequest(\u0026#34;GET\u0026#34;, \u0026#34;http://localhost:9998/provider\u0026#34;, http.NoBody) // 在发送 HTTP 请求之前创建 ExitSpan trace.CreateExitSpan(\u0026#34;GET:/provider\u0026#34;, \u0026#34;localhost:9999\u0026#34;, func(headerKey, headerValue string) error { // Injector 向请求中添加特定的 header 信息 req.Header.Add(headerKey, headerValue) return nil }) // 结束 ExitSpan，使用 defer 确保在函数返回时执行 defer trace.StopSpan() // 发送请求 client := \u0026amp;http.Client{} resp, err := client.Do(req) if err != nil { return nil, err } return resp, nil } func consumerHandler(w http.ResponseWriter, r *http.Request) { // 创建 EntrySpan 来追踪 consumerHandler 方法的执行 trace.CreateEntrySpan(r.Method+\u0026#34;/consumer\u0026#34;, func(headerKey string) (string, error) { // Extractor 获取请求中添加的 header 信息 return r.Header.Get(headerKey), nil }) // 结束 EntrySpan defer trace.StopSpan() // 准备发送 HTTP 请求 resp, err := getProvider() body, err := io.ReadAll(resp.Body) if err != nil { return } _, _ = w.Write(body) } func main() { http.HandleFunc(\u0026#34;/consumer\u0026#34;, consumerHandler) _ = http.ListenAndServe(\u0026#34;:9999\u0026#34;, nil) } // provider.go package main import ( \u0026#34;net/http\u0026#34; _ \u0026#34;github.com/apache/skywalking-go\u0026#34; \u0026#34;github.com/apache/skywalking-go/toolkit/trace\u0026#34; ) func providerHandler(w http.ResponseWriter, r *http.Request) { // 创建 EntrySpan 来追踪 providerHandler 方法的执行 trace.CreateEntrySpan(\u0026#34;GET:/provider\u0026#34;, func(headerKey string) (string, error) { return r.Header.Get(headerKey), nil }) // 结束 EntrySpan defer trace.StopSpan() _, _ = w.Write([]byte(\u0026#34;success from provider\u0026#34;)) } func main() { http.HandleFunc(\u0026#34;/provider\u0026#34;, providerHandler) _ = http.ListenAndServe(\u0026#34;:9998\u0026#34;, nil) } 然后中终端中执行：\ngo build -toolexec=\u0026#34;/path/to/go-agent\u0026#34; -a -o consumer ./consumer.go ./consumer go build -toolexec=\u0026#34;/path/to/go-agent\u0026#34; -a -o provider ./provider.go ./provider curl 127.0.0.1:9999/consumer 此时 UI 中将会显示你所创建的span信息\n如果需要追踪仅在本地执行的方法，可以使用 trace.CreateLocalSpan()。如果不需要监控来自另一端的信息或状态，可以将 ExitSpan 和 EntrySpan 更改为 LocalSpan。\n以上方法仅作为示例，用户可以决定追踪的粒度以及程序中需要进行追踪的位置。\n注意，如果程序结束得太快，可能会导致 Tracing 数据无法异步发送到 SkyWalking 后端。\n填充 Span 当需要记录额外信息时，包括创建/更新标签、追加日志和设置当前被追踪 Span 的新操作名称时，可以使用这些API。这些操作用于增强追踪信息，提供更详细的上下文描述，有助于更好地理解被追踪的事件或操作。\nToolkit trace APIs 提供了一种简便的方式来访问和操作 Trace 数据：\n设置标签：SetTag() 添加日志：AddLog() 设置 Span 名称：SetOperationName() 获取各种ID：GetTraceID(), GetSegmentID(), GetSpanID() 例如，如果需要在一个 Span 中记录HTTP状态码，就可以在 Span 未结束时调用以下接口：\ntrace.CreateExitSpan(\u0026#34;GET:/provider\u0026#34;, \u0026#34;localhost:9999\u0026#34;, func(headerKey, headerValue string) error { r.Header.Add(headerKey, headerValue) return nil }) resp, err := http.Get(\u0026#34;http://localhost:9999/provider\u0026#34;) trace.SetTag(\u0026#34;status_code\u0026#34;, fmt.Sprintf(\u0026#34;%d\u0026#34;, resp.StatusCode)) spanID := trace.GetSpanID() trace.StopSpan() 在调用这些方法时，当前线程需要有正在活跃的 span。\n异步 APIs 异步API 用于跨 goroutines 操作 spans。包括以下情况：\n包含多个 goroutines 的程序，需要在不同上下文中中操作 Span。 在异步操作时更新或记录 Span 的信息。 延迟结束 Span。 按照以下步骤使用：\n获取 CreateSpan 的返回值 SpanRef。 调用 spanRef.PrepareAsync() ，准备在另一个 goroutine 中执行操作。 当前 goroutine 工作结束后，调用 trace.StopSpan() 结束该 span（仅影响当前 goroutine）。 将 spanRef 传递给另一个 goroutine。 完成工作后在任意 goroutine 中调用 spanRef.AsyncFinish()。 以下为示例：\nspanRef, err := trace.CreateLocalSpan(\u0026#34;LocalSpan\u0026#34;) if err != nil { return } spanRef.PrepareAsync() go func(){ // some work spanRef.AsyncFinish() }() // some work trace.StopSpan() Correlation Context Correlation Context 用于在 Span 间传递参数，父 Span 会把 Correlation Context 递给其所有子 Spans。它允许在不同应用程序的 spans 之间传输信息。Correlation Context 的默认元素个数为3，其内容长度不能超过128字节。\nCorrelation Context 通常用于以下等情况:\n在 Spans 之间传递信息：它允许关键信息在不同 Span 之间传输，使上游和下游 Spans 能够获取彼此之间的关联和上下文。 传递业务参数：在业务场景中，涉及在不同 Span 之间传输特定参数或信息，如认证令牌、交易ID等。 用户可以使用 trace.SetCorrelation(key, value) 设置 Correlation Context ，并可以使用 value := trace.GetCorrelation(key) 在下游 spans 中获取相应的值。\n例如在下面的代码中，我们将值存储在 span 的标签中，以便观察结果：\npackage main import ( _ \u0026#34;github.com/apache/skywalking-go\u0026#34; \u0026#34;github.com/apache/skywalking-go/toolkit/trace\u0026#34; \u0026#34;net/http\u0026#34; ) func providerHandler(w http.ResponseWriter, r *http.Request) { ctxValue := trace.GetCorrelation(\u0026#34;key\u0026#34;) trace.SetTag(\u0026#34;result\u0026#34;, ctxValue) } func consumerHandler(w http.ResponseWriter, r *http.Request) { trace.SetCorrelation(\u0026#34;key\u0026#34;, \u0026#34;value\u0026#34;) _, err := http.Get(\u0026#34;http://localhost:9999/provider\u0026#34;) if err != nil { return } } func main() { http.HandleFunc(\u0026#34;/provider\u0026#34;, providerHandler) http.HandleFunc(\u0026#34;/consumer\u0026#34;, consumerHandler) _ = http.ListenAndServe(\u0026#34;:9999\u0026#34;, nil) } 然后在终端执行：\nexport SW_AGENT_NAME=server go build -toolexec=\u0026#34;/path/to/go-agent\u0026#34; -a -o server ./server.go ./server curl 127.0.0.1:9999/consumer 最后在 providerHandler() 的 Span 中找到了 Correlation Context 的信息：\n总结 本文讲述了Skywalking Go的 Trace APIs 及其应用。它为用户提供了自定义追踪的功能。\n更多关于该接口的介绍见文档：Tracing APIs。\n欢迎大家来使用新版本。\n","excerpt":"\u003ch2 id=\"背景介绍\"\u003e背景介绍\u003c/h2\u003e\n\u003cp\u003eSkyWalking Go是一个开源的非侵入式Golang代理程序，用于监控、追踪和在分布式系统中进行数据收集。它使用户能够观察系统内请求的流程和延迟，从各个系统组件收集性能数据以进行性能监 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2023-10-18-skywalking-toolkit-trace/","title":"SkyWalking Go Toolkit Trace 详解"},{"body":"CommunityOverCode (原 ApacheCon) 是 Apache 软件基金会（ASF）的官方全球系列大会。自 1998 年以来\u0026ndash;在 ASF 成立之前 \u0026ndash; ApacheCon 已经吸引了各个层次的参与者，在 300 多个 Apache 项目及其不同的社区中探索 \u0026ldquo;明天的技术\u0026rdquo;。CommunityOverCode 通过动手实作、主题演讲、实际案例研究、培训、黑客松活动等方式，展示 Apache 项目的最新发展和新兴创新。\nCommunityOverCode 展示了无处不在的 Apache 项目的最新突破和 Apache 孵化器中即将到来的创新，以及开源开发和以 Apache 之道领导社区驱动的项目。与会者可以了解到独立于商业利益、企业偏见或推销话术之外的核心开源技术。\nSkyWalking的Golang自动探针实践 刘晗 分布式追踪技术在可观测领域尤为重要，促使各个语言的追踪探针的易用性获得了更多的关注。目前在golang语言探针方面大多为手动埋点探针，接入流程过于复杂，而且局限性很强。本次讨论的重点着重于简化golang语言探针的接入方式，创新性的使用了自动埋点技术，并且突破了很多框架中对于上下文信息的依赖限制。\nB站视频地址\nBanyanDB一个高扩展性的分布式追踪数据库 高洪涛 追踪数据是一种用于分析微服务系统性能和故障的重要数据源，它记录了系统中每个请求的调用链路和相关指标。随着微服务系统的规模和复杂度的增长，追踪数据的量级也呈指数级增长，给追踪数据的存储和查询带来了巨大的挑战。传统的关系型数据库或者时序数据库往往难以满足追踪数据的高效存储和灵活查询的需求。 BanyanDB是一个专为追踪数据而设计的分布式数据库，它具有高扩展性、高性能、高可用性和高灵活性的特点。BanyanDB采用了基于时间序列的分片策略，将追踪数据按照时间范围划分为多个分片，每个分片可以独立地进行存储、复制和负载均衡。BanyanDB还支持多维索引，可以根据不同的维度对追踪数据进行快速过滤和聚合。 在本次演讲中，我们将介绍BanyanDB的设计思想、架构和实现细节，以及它在实际场景中的应用和效果。我们也将展示BanyanDB与其他数据库的对比和优势，以及它未来的发展方向和计划。\nB站视频地址\n","excerpt":"\u003cp\u003eCommunityOverCode (原 ApacheCon) 是 Apache 软件基金会（ASF）的官方全球系列大会。自 1998 年以来\u0026ndash;在 ASF 成立之前 \u0026ndash; …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2023-08-20-coc-asia-2023/","title":"CommunityOverCode Conference 2023 Asia"},{"body":"SkyWalking PHP 0.7.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed Start 0.7.0 development. by @jmjoy in https://github.com/apache/skywalking-php/pull/90 Add more info for error log. by @jmjoy in https://github.com/apache/skywalking-php/pull/91 Fix amqplib and predis argument problems. by @jmjoy in https://github.com/apache/skywalking-php/pull/92 Add Memcache plugin. by @jmjoy in https://github.com/apache/skywalking-php/pull/93 Refactor mysqli plugin, support procedural api. by @jmjoy in https://github.com/apache/skywalking-php/pull/94 Fix target address in cross process header. by @jmjoy in https://github.com/apache/skywalking-php/pull/95 Release SkyWalking PHP 0.7.0 by @jmjoy in https://github.com/apache/skywalking-php/pull/96 Full Changelog: https://github.com/apache/skywalking-php/compare/v0.7.0...v0.7.0\nPECL https://pecl.php.net/package/skywalking_agent/0.7.0\n","excerpt":"\u003cp\u003eSkyWalking PHP 0.7.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-php-0-7-0/","title":"Release Apache SkyWalking PHP 0.7.0"},{"body":"SkyWalking BanyanDB Helm 0.1.0 is released. Go to downloads page to find release tars.\nFeatures Deploy banyandb with standalone mode by Chart ","excerpt":"\u003cp\u003eSkyWalking BanyanDB Helm 0.1.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"features\"\u003eFeatures …\u003c/h3\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-helm-0-1-0/","title":"Release Apache SkyWalking BanyanDB Helm 0.1.0"},{"body":"背景介绍 Arthas 是一款常用的 Java 诊断工具，我们可以在 SkyWalking 监控到服务异常后，通过 Arthas 进一步分析和诊断以快速定位问题。\n在 Arthas 实际使用中，通常由开发人员拷贝或者下载安装包到服务对应的VM或者容器中，attach 到对应的 Java 进程进行问题排查。这一过程不可避免的会造成服务器敏感运维信息的扩散， 而且在分秒必争的问题排查过程中，这些繁琐的操作无疑会浪费大量时间。\nSkyWalking Java Agent 伴随 Java 服务一起启动，并定期上报服务、实例信息给OAP Server。我们可以借助 SkyWalking Java Agent 的插件化能力，开发一个 Arthas 控制插件， 由该插件管理 Arthas 运行生命周期，通过页面化的方式，完成Arthas的启动与停止。最终实现效果可以参考下图：\n要完成上述功能，我们需要实现以下几个关键点：\n开发 agent arthas-control-plugin，执行 arthas 的启动与停止命令 开发 oap arthas-controller-module ，下发控制命令给 arthas agent plugin 定制 skywalking-ui, 连接 arthas-tunnel-server，发送 arthas 命令并获取执行结果 以上各个模块之间的交互流程如下图所示：\nconnect disconnect 本文涉及的所有代码均已发布在 github skywalking-x-arthas 上，如有需要，大家可以自行下载代码测试。 文章后半部分将主要介绍代码逻辑及其中包含的SkyWalking扩展点。\nagent arthas-control-plugin 首先在 skywalking-java/apm-sniffer/apm-sdk-plugin 下创建一个 arthas-control-plugin， 该模块在打包后会成为 skywalking-agent/plugins 下的一个插件， 其目录结构如下：\narthas-control-plugin/ ├── pom.xml └── src └── main ├── java │ └── org │ └── apache │ └── skywalking │ └── apm │ └── plugin │ └── arthas │ ├── config │ │ └── ArthasConfig.java # 模块配置 │ ├── service │ │ └── CommandListener.java # boot service，监听 oap command │ └── util │ ├── ArthasCtl.java # 控制 arthas 的启动与停止 │ └── ProcessUtils.java ├── proto │ └── ArthasCommandService.proto # 与oap server通信的 grpc 协议定义 └── resources └── META-INF └── services # boot service spi service └── org.apache.skywalking.apm.agent.core.boot.BootService 16 directories, 7 files 在 ArthasConfig.java 中，我们定义了以下配置，这些参数将在 arthas 启动时传递。\n以下的配置可以通过 agent.config 文件、system prop、env variable指定。 关于 skywalking-agent 配置的初始化的具体流程，大家可以参考 SnifferConfigInitializer 。\npublic class ArthasConfig { public static class Plugin { @PluginConfig(root = ArthasConfig.class) public static class Arthas { // arthas 目录 public static String ARTHAS_HOME; // arthas 启动时连接的tunnel server public static String TUNNEL_SERVER; // arthas 会话超时时间 public static Long SESSION_TIMEOUT; // 禁用的 arthas command public static String DISABLED_COMMANDS; } } } 接着，我们看下 CommandListener.java 的实现，CommandListener 实现了 BootService 接口， 并通过 resources/META-INF/services 下的文件暴露给 ServiceLoader。\nBootService 的定义如下，共有prepare()、boot()、onComplete()、shutdown()几个方法，这几个方法分别对应插件生命周期的不同阶段。\npublic interface BootService { void prepare() throws Throwable; void boot() throws Throwable; void onComplete() throws Throwable; void shutdown() throws Throwable; default int priority() { return 0; } } 在 ServiceManager 类的 boot() 方法中， 定义了BootService 的 load 与启动流程，该方法 由SkyWalkingAgent 的 premain 调用，在主程序运行前完成初始化与启动：\npublic enum ServiceManager { INSTANCE; ... ... public void boot() { bootedServices = loadAllServices(); prepare(); startup(); onComplete(); } ... ... } 回到我们 CommandListener 的 boot 方法，该方法在 agent 启动之初定义了一个定时任务，这个定时任务会轮询 oap ，查询是否需要启动或者停止arthas:\npublic class CommandListener implements BootService, GRPCChannelListener { ... ... @Override public void boot() throws Throwable { getCommandFuture = Executors.newSingleThreadScheduledExecutor( new DefaultNamedThreadFactory(\u0026#34;CommandListener\u0026#34;) ).scheduleWithFixedDelay( new RunnableWithExceptionProtection( this::getCommand, t -\u0026gt; LOGGER.error(\u0026#34;get arthas command error.\u0026#34;, t) ), 0, 2, TimeUnit.SECONDS ); } ... ... } getCommand方法中定义了start、stop的处理逻辑，分别对应页面上的 connect 和 disconnect 操作。 这两个 command 有分别转给 ArthasCtl 的 startArthas 和 stopArthas 两个方法处理，用来控制 arthas 的启停。\n在 startArthas 方法中，启动arthas-core.jar 并使用 skywalking-agent 的 serviceName 和 instanceName 注册连接至配置文件中指定的arthas-tunnel-server。\nArthasCtl 逻辑参考自 Arthas 的 BootStrap.java ，由于不是本篇文章的重点，这里不再赘述，感兴趣的小伙伴可以自行查看。\nswitch (commandResponse.getCommand()) { case START: if (alreadyAttached()) { LOGGER.warn(\u0026#34;arthas already attached, no need start again\u0026#34;); return; } try { arthasTelnetPort = SocketUtils.findAvailableTcpPort(); ArthasCtl.startArthas(PidUtils.currentLongPid(), arthasTelnetPort); } catch (Exception e) { LOGGER.info(\u0026#34;error when start arthas\u0026#34;, e); } break; case STOP: if (!alreadyAttached()) { LOGGER.warn(\u0026#34;no arthas attached, no need to stop\u0026#34;); return; } try { ArthasCtl.stopArthas(arthasTelnetPort); arthasTelnetPort = null; } catch (Exception e) { LOGGER.info(\u0026#34;error when stop arthas\u0026#34;, e); } break; } 看完 arthas 的启动与停止控制逻辑，我们回到 CommandListener 的 statusChanged 方法， 由于要和 oap 通信，这里我们按照惯例监听 grpc channel 的状态，只有状态正常时才会执行上面的getCommand轮询。\npublic class CommandListener implements BootService, GRPCChannelListener { ... ... @Override public void statusChanged(final GRPCChannelStatus status) { if (GRPCChannelStatus.CONNECTED.equals(status)) { Object channel = ServiceManager.INSTANCE.findService(GRPCChannelManager.class).getChannel(); // DO NOT REMOVE Channel CAST, or it will throw `incompatible types: org.apache.skywalking.apm.dependencies.io.grpc.Channel // cannot be converted to io.grpc.Channel` exception when compile due to agent core\u0026#39;s shade of grpc dependencies. commandServiceBlockingStub = ArthasCommandServiceGrpc.newBlockingStub((Channel) channel); } else { commandServiceBlockingStub = null; } this.status = status; } ... ... } 上面的代码，细心的小伙伴可能会发现，getChannel() 的返回值被向上转型成了 Object, 而在下面的 newBlockingStub 方法中，又强制转成了 Channel。\n看似有点多此一举，其实不然，我们将这里的转型去掉,尝试编译就会收到下面的错误：\n[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.10.1:compile (default-compile) on project arthas-control-plugin: Compilation failure [ERROR] .../CommandListener.java:[59,103] 不兼容的类型: org.apache.skywalking.apm.dependencies.io.grpc.Channel无法转换为io.grpc.Channel 上面的错误提示 ServiceManager.INSTANCE.findService(GRPCChannelManager.class).getChannel() 的返回值类型是 org.apache.skywalking.apm.dependencies.io.grpc.Channel，无法被赋值给 io.grpc.Channel 引用。\n我们查看GRPCChannelManager的getChannel()方法代码会发现，方法定义的返回值明明是 io.grpc.Channel，为什么编译时会报上面的错误？\n其实这是skywalking-agent的一个小魔法，由于 agent-core 最终会被打包进 skywalking-agent.jar，启动时由系统类装载器（或者其他父级类装载器）直接装载， 为了防止所依赖的类库和被监控服务的类发生版本冲突，agent 核心代码在打包时使用了maven-shade-plugin, 该插件会在 maven package 阶段改变 grpc 依赖的包名， 我们在源代码里看到的是 io.grpc.Channel，其实在真正运行时已经被改成了 org.apache.skywalking.apm.dependencies.io.grpc.Channel，这便可解释上面编译报错的原因。\n除了grpc以外，其他一些 well-known 的 dependency 也会进行 shade 操作，详情大家可以参考 apm-agent-core pom.xml ：\n\u0026lt;plugin\u0026gt; \u0026lt;artifactId\u0026gt;maven-shade-plugin\u0026lt;/artifactId\u0026gt; \u0026lt;executions\u0026gt; \u0026lt;execution\u0026gt; \u0026lt;phase\u0026gt;package\u0026lt;/phase\u0026gt; \u0026lt;goals\u0026gt; \u0026lt;goal\u0026gt;shade\u0026lt;/goal\u0026gt; \u0026lt;/goals\u0026gt; \u0026lt;configuration\u0026gt; ... ... \u0026lt;relocations\u0026gt; \u0026lt;relocation\u0026gt; \u0026lt;pattern\u0026gt;${shade.com.google.source}\u0026lt;/pattern\u0026gt; \u0026lt;shadedPattern\u0026gt;${shade.com.google.target}\u0026lt;/shadedPattern\u0026gt; \u0026lt;/relocation\u0026gt; \u0026lt;relocation\u0026gt; \u0026lt;pattern\u0026gt;${shade.io.grpc.source}\u0026lt;/pattern\u0026gt; \u0026lt;shadedPattern\u0026gt;${shade.io.grpc.target}\u0026lt;/shadedPattern\u0026gt; \u0026lt;/relocation\u0026gt; \u0026lt;relocation\u0026gt; \u0026lt;pattern\u0026gt;${shade.io.netty.source}\u0026lt;/pattern\u0026gt; \u0026lt;shadedPattern\u0026gt;${shade.io.netty.target}\u0026lt;/shadedPattern\u0026gt; \u0026lt;/relocation\u0026gt; \u0026lt;relocation\u0026gt; \u0026lt;pattern\u0026gt;${shade.io.opencensus.source}\u0026lt;/pattern\u0026gt; \u0026lt;shadedPattern\u0026gt;${shade.io.opencensus.target}\u0026lt;/shadedPattern\u0026gt; \u0026lt;/relocation\u0026gt; \u0026lt;relocation\u0026gt; \u0026lt;pattern\u0026gt;${shade.io.perfmark.source}\u0026lt;/pattern\u0026gt; \u0026lt;shadedPattern\u0026gt;${shade.io.perfmark.target}\u0026lt;/shadedPattern\u0026gt; \u0026lt;/relocation\u0026gt; \u0026lt;relocation\u0026gt; \u0026lt;pattern\u0026gt;${shade.org.slf4j.source}\u0026lt;/pattern\u0026gt; \u0026lt;shadedPattern\u0026gt;${shade.org.slf4j.target}\u0026lt;/shadedPattern\u0026gt; \u0026lt;/relocation\u0026gt; \u0026lt;/relocations\u0026gt; ... ... \u0026lt;/configuration\u0026gt; \u0026lt;/execution\u0026gt; \u0026lt;/executions\u0026gt; \u0026lt;/plugin\u0026gt; 除了上面的注意点以外，我们来看一下另一个场景，假设我们需要在 agent plugin 的 interceptor 中使用 plugin 中定义的 BootService 会发生什么？\n我们回到 BootService 的加载逻辑，为了加载到 plugin 中定义的BootService，ServiceLoader 指定了类装载器为AgentClassLoader.getDefault()， （这行代码历史非常悠久，可以追溯到2018年：Allow use SkyWalking plugin to override service in Agent core. #1111 ）， 由此可见，plugin 中定义的 BootService 的 classloader 是 AgentClassLoader.getDefault()：\nvoid load(List\u0026lt;BootService\u0026gt; allServices) { for (final BootService bootService : ServiceLoader.load(BootService.class, AgentClassLoader.getDefault())) { allServices.add(bootService); } } 再来看下 interceptor 的加载逻辑，InterceptorInstanceLoader.java 的 load 方法规定了如果父加载器相同，plugin 中的 interceptor 将使用一个新创建的 AgentClassLoader （在绝大部分简单场景中，plugin 的 interceptor 都由同一个 AgentClassLoader 加载）：\npublic static \u0026lt;T\u0026gt; T load(String className, ClassLoader targetClassLoader) throws IllegalAccessException, InstantiationException, ClassNotFoundException, AgentPackageNotFoundException { ... ... pluginLoader = EXTEND_PLUGIN_CLASSLOADERS.get(targetClassLoader); if (pluginLoader == null) { pluginLoader = new AgentClassLoader(targetClassLoader); EXTEND_PLUGIN_CLASSLOADERS.put(targetClassLoader, pluginLoader); } ... ... } 按照类装载器的委派机制，interceptor 中如果用到了 BootService，也会由当前的类的装载器去装载。 所以 ServiceManager 中装载的 BootService 和 interceptor 装载的 BootService 并不是同一个 （一个 class 文件被不同的 classloader 装载了两次），如果在 interceptor 中 调用 BootService 方法，同样会发生 cast 异常。 由此可见，目前的实现并不支持我们在interceptor中直接调用 plugin 中 BootService 的方法，如果需要调用，只能将 BootService 放到 agent-core 中，由更高级别的类装载器优先装载。\n这其实并不是 skywalking-agent 的问题，skywalking agent plugin 专注于自己的应用场景，只需要关注 trace、meter 以及默认 BootService 的覆盖就可以了。 只是我们如果有扩展 skywalking-agent 的需求，要对其类装载机制做到心中有数，否则可能会出现一些意想不到的问题。\noap arthas-controller-module 看完 agent-plugin 的实现，我们再来看看 oap 部分的修改，oap 同样是模块化的设计，我们可以很轻松的增加一个新的模块，在 /oap-server/ 目录下新建 arthas-controller 子模块：\narthas-controller/ ├── pom.xml └── src └── main ├── java │ └── org │ └── apache │ └── skywalking │ └── oap │ └── arthas │ ├── ArthasControllerModule.java # 模块定义 │ ├── ArthasControllerProvider.java # 模块逻辑实现者 │ ├── CommandQueue.java │ └── handler │ ├── CommandGrpcHandler.java # grpc handler,供 plugin 通信使用 │ └── CommandRestHandler.java # http handler,供 skywalking-ui 通信使用 ├── proto │ └── ArthasCommandService.proto └── resources └── META-INF └── services # 模块及模块实现的 spi service ├── org.apache.skywalking.oap.server.library.module.ModuleDefine └── org.apache.skywalking.oap.server.library.module.ModuleProvider 模块的定义非常简单，只包含一个模块名，由于我们新增的模块并不需要暴露service给其他模块调用，services 我们返回一个空数组\npublic class ArthasControllerModule extends ModuleDefine { public static final String NAME = \u0026#34;arthas-controller\u0026#34;; public ArthasControllerModule() { super(NAME); } @Override public Class\u0026lt;?\u0026gt;[] services() { return new Class[0]; } } 接着是模块实现者，实现者取名为 default，module 指定该 provider 所属模块，由于没有模块的自定义配置，newConfigCreator 我们返回null即可。 start 方法分别向 CoreModule 的 grpc 服务和 http 服务注册了两个 handler，grpc 服务和 http 服务就是我们熟知的 11800 和 12800 端口：\npublic class ArthasControllerProvider extends ModuleProvider { @Override public String name() { return \u0026#34;default\u0026#34;; } @Override public Class\u0026lt;? extends ModuleDefine\u0026gt; module() { return ArthasControllerModule.class; } @Override public ConfigCreator\u0026lt;?\u0026gt; newConfigCreator() { return null; } @Override public void prepare() throws ServiceNotProvidedException { } @Override public void start() throws ServiceNotProvidedException, ModuleStartException { // grpc service for agent GRPCHandlerRegister grpcService = getManager().find(CoreModule.NAME) .provider() .getService(GRPCHandlerRegister.class); grpcService.addHandler( new CommandGrpcHandler() ); // rest service for ui HTTPHandlerRegister restService = getManager().find(CoreModule.NAME) .provider() .getService(HTTPHandlerRegister.class); restService.addHandler( new CommandRestHandler(), Collections.singletonList(HttpMethod.POST) ); } @Override public void notifyAfterCompleted() throws ServiceNotProvidedException { } @Override public String[] requiredModules() { return new String[0]; } } 最后在配置文件中注册本模块及模块实现者，下面的配置表示 arthas-controller 这个 module 由 default provider 提供实现：\narthas-controller: selector: default default: CommandGrpcHandler 和 CommandHttpHandler 的逻辑非常简单，CommandHttpHandler 定义了 connect 和 disconnect 接口， 收到请求后会放到一个 Queue 中供 CommandGrpcHandler 消费，Queue 的实现如下，这里不再赘述：\npublic class CommandQueue { private static final Map\u0026lt;String, Command\u0026gt; COMMANDS = new ConcurrentHashMap\u0026lt;\u0026gt;(); // produce by connect、disconnect public static void produceCommand(String serviceName, String instanceName, Command command) { COMMANDS.put(serviceName + instanceName, command); } // consume by agent getCommand task public static Optional\u0026lt;Command\u0026gt; consumeCommand(String serviceName, String instanceName) { return Optional.ofNullable(COMMANDS.remove(serviceName + instanceName)); } } skywalking-ui arthas console 完成了 agent 和 oap 的开发，我们再看下 ui 部分：\nconnect：调用oap server connect 接口，并连接 arthas-tunnel-server disconnect：调用oap server disconnect 接口，并与 arthas-tunnel-server 断开连接 arthas 命令交互，这部分代码主要参考 arthas，大家可以查看 web-ui console 的实现 修改完skywalking-ui的代码后，我们可以直接通过 npm run dev 测试了。\n如果需要通过主项目打包，别忘了在apm-webapp 的 ApplicationStartUp.java 类中添加一条 arthas 的路由：\nServer .builder() .port(port, SessionProtocol.HTTP) .service(\u0026#34;/arthas\u0026#34;, oap) .service(\u0026#34;/graphql\u0026#34;, oap) .service(\u0026#34;/internal/l7check\u0026#34;, HealthCheckService.of()) .service(\u0026#34;/zipkin/config.json\u0026#34;, zipkin) .serviceUnder(\u0026#34;/zipkin/api\u0026#34;, zipkin) .serviceUnder(\u0026#34;/zipkin\u0026#34;, FileService.of( ApplicationStartUp.class.getClassLoader(), \u0026#34;/zipkin-lens\u0026#34;) .orElse(zipkinIndexPage)) .serviceUnder(\u0026#34;/\u0026#34;, FileService.of( ApplicationStartUp.class.getClassLoader(), \u0026#34;/public\u0026#34;) .orElse(indexPage)) .build() .start() .join(); 总结 BootService 启动及停止流程 如何利用 BootService 实现自定义逻辑 Agent Plugin 的类装载机制 maven-shade-plugin 的使用与注意点 如何利用 ModuleDefine 与 ModuleProvider 定义新的模块 如何向 GRPC、HTTP Service 添加新的 handler 如果你还有任何的疑问，欢迎大家与我交流 。\n","excerpt":"\u003ch2 id=\"背景介绍\"\u003e背景介绍\u003c/h2\u003e\n\u003cp\u003eArthas 是一款常用的 Java 诊断工具，我们可以在 SkyWalking 监控到服务异常后，通过 Arthas 进一步分析和诊断以快速定位问题。\u003c/p\u003e\n\u003cp\u003e在 Arthas 实际使用中，通常由 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2023-09-17-integrating-skywalking-with-arthas/","title":"将 Apache SkyWalking 与 Arthas 集成"},{"body":"SkyWalking Eyes 0.5.0 is released. Go to downloads page to find release tars.\nfeat(header templates): add support for AGPL-3.0 by @elijaholmos in https://github.com/apache/skywalking-eyes/pull/125 Upgrade go version to 1.18 by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/126 Add MulanPSL-2.0 support. by @jmjoy in https://github.com/apache/skywalking-eyes/pull/127 New Header Template: GPL-3.0-or-later by @ddlees in https://github.com/apache/skywalking-eyes/pull/128 Update README.md by @rovast in https://github.com/apache/skywalking-eyes/pull/129 Add more .env.[mode] support for VueJS project by @rovast in https://github.com/apache/skywalking-eyes/pull/130 Docker Multiple Architecture Support :fixes#9089 by @mohammedtabish0 in https://github.com/apache/skywalking-eyes/pull/132 Polish maven test for convenient debug by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/134 feat: list files by git when possible by @tisonkun in https://github.com/apache/skywalking-eyes/pull/133 Switch to npm ci for reliable builds by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/135 Fix optional dependencies are not excluded by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/136 Fix exclude not work for transitive dependencies and add recursive config by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/137 Add some tests for maven resovler by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/138 feat(header-fix): add Svelte support by @elijaholmos in https://github.com/apache/skywalking-eyes/pull/139 dep: do not write license files if they already exist by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/140 fix: not ignore *.txt to make sure files like CMakeLists.txt can be checked by @acelyc111 in https://github.com/apache/skywalking-eyes/pull/141 fix license header normalizer by @xiaoyawei in https://github.com/apache/skywalking-eyes/pull/142 Substitute variables in license content for header command by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/143 Correct indent in Apache-2.0 template by @tisonkun in https://github.com/apache/skywalking-eyes/pull/144 Add copyright-year configuration by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/145 dep/maven: use output file to store the dep tree for cleaner result by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/146 dep/maven: resolve dependencies before analysis by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/147 gha: switch to composite running mode and set up cache by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/149 gha: switch to composite running mode and set up cache by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/150 Fix GitHub Actions wrong path by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/151 Normalize license for cargo. by @jmjoy in https://github.com/apache/skywalking-eyes/pull/153 Remove space characters in license for cargo. by @jmjoy in https://github.com/apache/skywalking-eyes/pull/154 Bump up dependencies to fix CVE by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/155 Bump up GHA to depress warnings by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/156 Leverage the built-in cache in setup-go@v4 by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/157 Dependencies check should report unknown licneses by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/158 Fix wrong indentation in doc by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/159 Add EPL-2.0 header template by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/160 Fix wrong indentation in doc about multi license config by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/161 dependency resolve with default template and specified output of license by @crholm in https://github.com/apache/skywalking-eyes/pull/163 Bump up go git to support .gitconfig user path by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/164 Draft release notes for 0.5.0 by @kezhenxu94 in https://github.com/apache/skywalking-eyes/pull/165 Remove \u0026ldquo;portions copyright\u0026rdquo; header normalizer by @antgamdia in https://github.com/apache/skywalking-eyes/pull/166 Full Changelog: https://github.com/apache/skywalking-eyes/compare/v0.4.0...v0.5.0\n","excerpt":"\u003cp\u003eSkyWalking Eyes 0.5.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003efeat(header …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-eyes-0-5-0/","title":"Release Apache SkyWalking Eyes 0.5.0"},{"body":" Abstract Apache SkyWalking hosts SkyWalking Summit 2023 on Nov. 4th, 2023, UTC+8, sponsored by ZMOps and Tetrate.\nWe are going to share SkyWalking\u0026rsquo;s roadmap, features, product experiences, and open-source culture.\nWelcome to join us.\nVenue Addr./地址 上海大华虹桥假日酒店\nDate 8:00 - 17:00, Nov 4th.\nRegister Register for IN-PERSON ticket\nCall For Proposals (CFP) The Call For Proposals open from now to 18:00 on Oct. 27th 2023, UTC+8. Submit your proposal at here\nWe have 1 open session and 8 sessions for the whole event.\nOpen session is reserved for SkyWalking PMC members. 6 sessions are opened for CFP process. 2 sessions are reserved for sponsors. Sponsors ZMOps Inc. Tetrate Inc. Anti-harassment policy SkyWalkingDay is dedicated to providing a harassment-free experience for everyone. We do not tolerate harassment of participants in any form. Sexual language and imagery will also not be tolerated in any event venue. Participants violating these rules may be sanctioned or expelled without a refund, at the discretion of the event organizers. Our anti-harassment policy can be found at Apache website.\nContact Us Send mail to dev@skywalking.apache.org.\n","excerpt":"\u003cimg src=\"banner.jpg\"\u003e\n\u003ch2 id=\"abstract\"\u003eAbstract\u003c/h2\u003e\n\u003cp\u003eApache SkyWalking hosts SkyWalking Summit 2023 on Nov. 4th, 2023, UTC+8, sponsored by …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/summit-23-cn/","title":"SkyWalking Summit 2023 @ Shanghai China"},{"body":"SkyWalking 9.6.0 is released. Go to downloads page to find release tars.\nNew Alerting Kernel MQE(Metrics Query Expression) and a new notification mechanism are supported. Support Loki LogQL Newly added support for Loki LogQL and Grafana Loki Dashboard for SkyWalking collected logs WARNING ElasticSearch 6 storage relative tests are removed. It worked and is not promised due to end of life officially. Project Bump up Guava to 32.0.1 to avoid the lib listed as vulnerable due to CVE-2020-8908. This API is never used. Maven artifact skywalking-log-recevier-plugin is renamed to skywalking-log-receiver-plugin. Bump up cli version 0.11 to 0.12. Bump up the version of ASF parent pom to v30. Make builds reproducible for automatic releases CI. OAP Server Add Neo4j component ID(112) language: Python. Add Istio ServiceEntry registry to resolve unknown IPs in ALS. Wrap deleteProperty API to the BanyanDBStorageClient. [Breaking change] Remove matchedCounter from HttpUriRecognitionService#feedRawData. Remove patterns from HttpUriRecognitionService#feedRawData and add max 10 candidates of raw URIs for each pattern. Add component ID for WebSphere. Fix AI Pipeline uri caching NullPointer and IllegalArgument Exceptions. Fix NPE in metrics query when the metric is not exist. Remove E2E tests for Istio \u0026lt; 1.15, ElasticSearch \u0026lt; 7.16.3, they might still work but are not supported as planed. Scroll all results in ElasticSearch storage and refactor scrolling logics, including Service, Instance, Endpoint, Process, etc. Improve Kubernetes coordinator to remove Terminating OAP Pods in cluster. Support SW_CORE_SYNC_PERIOD_HTTP_URI_RECOGNITION_PATTERN and SW_CORE_TRAINING_PERIOD_HTTP_URI_RECOGNITION_PATTERN to control the period of training and sync HTTP URI recognition patterns. And shorten the default period to 10s for sync and 60s for training. Fix ElasticSearch scroller bug. Add component ID for Aerospike(ID=149). Packages with name recevier are renamed to receiver. BanyanDBMetricsDAO handles storeIDTag in multiGet for BanyanDBModelExtension. Fix endpoint grouping-related logic and enhance the performance of PatternTree retrieval. Fix metric session cache saving after batch insert when using mysql-connector-java. Support dynamic UI menu query. Add comment for docker/.env to explain the usage. Fix wrong environment variable name SW_OTEL_RECEIVER_ENABLED_OTEL_RULES to right SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES. Fix instance query in JDBC implementation. Set the SW_QUERY_MAX_QUERY_COMPLEXITY default value to 3000(was 1000). Accept length=4000 parameter value of the event. It was 2000. Tolerate parameter value in illegal JSON format. Update BanyanDB Java Client to 0.4.0 Support aggregate Labeled Value Metrics in MQE. [Breaking change] Change the default label name in MQE from label to _. Bump up grpc version to 1.53.0. [Breaking change] Removed \u0026lsquo;\u0026amp;\u0026rsquo; symbols from shell scripts to avoid OAP server process running as a background process. Revert part of #10616 to fix the unexpected changes: if there is no data we should return an array with 0s, but in #10616, an empty array is returned. Cache all service entity in memory for query. Bump up jackson version to 2.15.2. Increase the default memory size to avoid OOM. Bump up graphql-java to 21.0. Add Echo component ID(5015) language: Golang. Fix index out of bounds exception in aggregate_labels MQE function. Support MongoDB Server/Cluster monitoring powered by OTEL. Do not print configurations values in logs to avoid sensitive info leaked. Move created the latest index before retrieval indexes by aliases to avoid the 404 exception. This just prevents some interference from manual operations. Add more Go VM metrics, as new skywalking-go agent provided since its 0.2 release. Add component ID for Lock (ID=5016). [Breaking change] Adjust the structure of hooks in the alarm-settings.yml. Support multiple configs for each hook types and specifying the hooks in the alarm rule. Bump up Armeria to 1.24.3. Fix BooleanMatch and BooleanNotEqualMatch doing Boolean comparison. Support LogQL HTTP query APIs. Add Mux Server component ID(5017) language: Golang. Remove ElasticSearch 6.3.2 from our client lib tests. Bump up ElasticSearch server 8.8.1 to 8.9.0 for latest e2e testing. 8.1.0, 7.16.3 and 7.17.10 are still tested. Add OpenSearch 2.8.0 to our client lib tests. Use listening mode for apollo implementation of dynamic configuration. Add view_as_seq function in MQE for listing metrics in the given prioritized sequence. Fix the wrong default value of k8sServiceNameRule if it\u0026rsquo;s not explicitly set. Improve PromQL to allow for multiple metric operations within a single query. Fix MQE Binary Operation between labeled metrics and other type of value result. Add component ID for Nacos (ID=150). Support Compare Operation in MQE. Fix the Kubernetes resource cache not refreshed. Fix wrong classpath that might cause OOM in startup. Enhance the serviceRelation in MAL by adding settings for the delimiter and component fields. [Breaking change] Support MQE in the Alerting. The Alarm Rules configuration(alarm-settings.yml), add expression field and remove metrics-name/count/threshold/op/only-as-condition fields and remove composite-rules configuration. Check results in ALS as per downstream/upstream instead of per log. Fix GraphQL query listInstances not using endTime query Do not start server and Kafka consumer in init mode. Add Iris component ID(5018). Add OTLP Tracing support as a Zipkin trace input. UI Fix metric name browser_app_error_rate in Browser-Root dashboard. Fix display name of endpoint_cpm for endpoint list in General-Service dashboard. Implement customize menus and marketplace page. Fix minTraceDuration and maxTraceDuration types. Fix init minTime to Infinity. Bump dependencies to fix vulnerabilities. Add scss variables. Fix the title of instance list and notices in the continue profiling. Add a link to explain the expression metric, add units in the continue profiling widget. Calculate string width to set Tabs name width. [Breaking change] Removed \u0026lsquo;\u0026amp;\u0026rsquo; symbols from shell scripts to avoid web application server process running as a background process. Reset chart label. Fix service associates instances. Remove node-sass. Fix commit error on Windows. Apply MQE on MYSQL, POSTGRESQL, REDIS, ELASTICSEARCH and DYNAMODB layer UI-templates. Apply MQE on Virtual-Cache layer UI-templates Apply MQE on APISIX, AWS_EKS, AWS_GATEWAY and AWS_S3 layer UI templates. Apply MQE on RabbitMQ Dashboards. Apply MQE on Virtual-MQ layer UI-templates Apply MQE on Infra-Linux layer UI-templates Apply MQE on Infra-Windows layer UI-templates Apply MQE on Browser layer UI-templates. Implement MQE on topology widget. Fix getEndpoints keyword blank. Implement a breadcrumb component as navigation. Documentation Add Go agent into the server agent documentation. Add data unit description in the configuration of continuous profiling policy. Remove storage extension doc, as it is expired. Remove how to add menu doc, as SkyWalking supports marketplace and new backend-based setup. Separate contribution docs to a new menu structure. Add a doc to explain how to manage i18n. Add a doc to explain OTLP Trace support. Fix typo in dynamic-config-configmap.md. Fix out-dated docs about Kafka fetcher. Remove 3rd part fetchers from the docs, as they are not maintained anymore. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking 9.6.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"new-alerting-kernel\"\u003eNew Alerting Kernel …\u003c/h4\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-9.6.0/","title":"Release Apache SkyWalking APM 9.6.0"},{"body":"SkyWalking Java Agent 9.0.0 is released. Go to downloads page to find release tars. Changes by Version\n9.0.0 Kernel Updates Support re-transform/hot-swap classes with other java agents, and remove the obsolete cache enhanced class feature. Implement new naming policies for names of auxiliary type, interceptor delegate field, renamed origin method, method access name, method cache value field. All names are under sw$ name trait. They are predictable and unchanged after re-transform. * SWAuxiliaryTypeNamingStrategy Auxiliary type name pattern: \u0026lt;origin_class_name\u0026gt;$\u0026lt;name_trait\u0026gt;$auxiliary$\u0026lt;auxiliary_type_instance_hash\u0026gt; * DelegateNamingResolver Interceptor delegate field name pattern: \u0026lt;name_trait\u0026gt;$delegate$\u0026lt;class_name_hash\u0026gt;$\u0026lt;plugin_define_hash\u0026gt;$\u0026lt;intercept_point_hash\u0026gt; * SWMethodNameTransformer Renamed origin method pattern: \u0026lt;name_trait\u0026gt;$original$\u0026lt;method_name\u0026gt;$\u0026lt;method_description_hash\u0026gt; * SWImplementationContextFactory Method cache value field pattern: cachedValue$\u0026lt;name_trait\u0026gt;$\u0026lt;origin_class_name_hash\u0026gt;$\u0026lt;field_value_hash\u0026gt; Accessor method name pattern: \u0026lt;renamed_origin_method\u0026gt;$accessor$\u0026lt;name_trait\u0026gt;$\u0026lt;origin_class_name_hash\u0026gt; Here is an example of manipulated enhanced class with new naming policies of auxiliary classes, fields, and methods\nimport sample.mybatis.controller.HotelController$sw$auxiliary$19cja42; import sample.mybatis.controller.HotelController$sw$auxiliary$p257su0; import sample.mybatis.domain.Hotel; import sample.mybatis.service.HotelService; @RequestMapping(value={\u0026#34;/hotel\u0026#34;}) @RestController public class HotelController implements EnhancedInstance { @Autowired @lazy private HotelService hotelService; private volatile Object _$EnhancedClassField_ws; // Interceptor delegate fields public static volatile /* synthetic */ InstMethodsInter sw$delegate$td03673$ain2do0$8im5jm1; public static volatile /* synthetic */ InstMethodsInter sw$delegate$td03673$ain2do0$edkmf61; public static volatile /* synthetic */ ConstructorInter sw$delegate$td03673$ain2do0$qs9unv1; public static volatile /* synthetic */ InstMethodsInter sw$delegate$td03673$fl4lnk1$m3ia3a2; public static volatile /* synthetic */ InstMethodsInter sw$delegate$td03673$fl4lnk1$sufrvp1; public static volatile /* synthetic */ ConstructorInter sw$delegate$td03673$fl4lnk1$cteu7s1; // Origin method cache value field private static final /* synthetic */ Method cachedValue$sw$td03673$g5sobj1; public HotelController() { this(null); sw$delegate$td03673$ain2do0$qs9unv1.intercept(this, new Object[0]); } private /* synthetic */ HotelController(sw.auxiliary.p257su0 p257su02) { } @GetMapping(value={\u0026#34;city/{cityId}\u0026#34;}) public Hotel selectByCityId(@PathVariable(value=\u0026#34;cityId\u0026#34;) int n) { // call interceptor with auxiliary type and parameters and origin method object return (Hotel)sw$delegate$td03673$ain2do0$8im5jm1.intercept(this, new Object[]{n}, new HotelController$sw$auxiliary$19cja42(this, n), cachedValue$sw$td03673$g5sobj1); } // Renamed origin method private /* synthetic */ Hotel sw$origin$selectByCityId$a8458p3(int cityId) { /*22*/ return this.hotelService.selectByCityId(cityId); } // Accessor of renamed origin method, calling from auxiliary type final /* synthetic */ Hotel sw$origin$selectByCityId$a8458p3$accessor$sw$td03673(int n) { // Calling renamed origin method return this.sw$origin$selectByCityId$a8458p3(n); } @OverRide public Object getSkyWalkingDynamicField() { return this._$EnhancedClassField_ws; } @OverRide public void setSkyWalkingDynamicField(Object object) { this._$EnhancedClassField_ws = object; } static { ClassLoader.getSystemClassLoader().loadClass(\u0026#34;org.apache.skywalking.apm.dependencies.net.bytebuddy.dynamic.Nexus\u0026#34;).getMethod(\u0026#34;initialize\u0026#34;, Class.class, Integer.TYPE).invoke(null, HotelController.class, -1072476370); // Method object cachedValue$sw$td03673$g5sobj1 = HotelController.class.getMethod(\u0026#34;selectByCityId\u0026#34;, Integer.TYPE); } } Auxiliary type of Constructor :\nclass HotelController$sw$auxiliary$p257su0 { } Auxiliary type of selectByCityId method:\nclass HotelController$sw$auxiliary$19cja42 implements Runnable, Callable { private HotelController argument0; private int argument1; public Object call() throws Exception { return this.argument0.sw$origin$selectByCityId$a8458p3$accessor$sw$td03673(this.argument1); } @OverRide public void run() { this.argument0.sw$origin$selectByCityId$a8458p3$accessor$sw$td03673(this.argument1); } HotelController$sw$auxiliary$19cja42(HotelController hotelController, int n) { this.argument0 = hotelController; this.argument1 = n; } } Features and Bug Fixes Support Jdk17 ZGC metric collect Support Jetty 11.x plugin Support access to the sky-walking tracer context in spring gateway filter Fix the scenario of using the HBase plugin with spring-data-hadoop. Add RocketMQ 5.x plugin Fix the conflict between the logging kernel and the JDK threadpool plugin. Fix the thread safety bug of finishing operation for the span named \u0026ldquo;SpringCloudGateway/sendRequest\u0026rdquo; Fix NPE in guava-eventbus-plugin. Add WebSphere Liberty 23.x plugin Add Plugin to support aerospike Java client Add ClickHouse parsing to the jdbc-common plugin. Support to trace redisson lock Upgrade netty-codec-http2 to 4.1.94.Final Upgrade guava to 32.0.1 Fix issue with duplicate enhancement by ThreadPoolExecutor Add plugin to support for RESTeasy 6.x. Fix the conditions for resetting UUID, avoid the same uuid causing the configuration not to be updated. Fix witness class in springmvc-annotation-5.x-plugin to avoid falling into v3 use cases. Fix Jedis-2.x plugin bug and add test for Redis cluster scene Merge two instrumentation classes to avoid duplicate enhancements in MySQL plugins. Support asynchronous invocation in jetty client 9.0 and 9.x plugin Add nacos-client 2.x plugin Staticize the tags for preventing synchronization in JDK 8 Add RocketMQ-Client-Java 5.x plugin Fix NullPointerException in lettuce-5.x-plugin. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking Java Agent 9.0.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nChanges by …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-java-agent-9-0-0/","title":"Release Apache SkyWalking Java Agent 9.0.0"},{"body":"SkyWalking PHP 0.6.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed Polish doc about Swoole by @wu-sheng in https://github.com/apache/skywalking-php/pull/73 Start 0.6.0 development. by @jmjoy in https://github.com/apache/skywalking-php/pull/74 Fix hook for Doctrine PDO class by @matikij in https://github.com/apache/skywalking-php/pull/76 Log Exception in tracing span when throw. by @jmjoy in https://github.com/apache/skywalking-php/pull/75 Upgrade dependencies and adapt. by @jmjoy in https://github.com/apache/skywalking-php/pull/77 Fix required rust version and add runing php-fpm notice in docs. by @jmjoy in https://github.com/apache/skywalking-php/pull/78 Bump openssl from 0.10.48 to 0.10.55 by @dependabot in https://github.com/apache/skywalking-php/pull/79 Fix the situation where the redis port is string. by @jmjoy in https://github.com/apache/skywalking-php/pull/80 Optionally enable zend observer api for auto instrumentation. by @jmjoy in https://github.com/apache/skywalking-php/pull/81 Fix the empty span situation in redis after hook. by @jmjoy in https://github.com/apache/skywalking-php/pull/82 Add mongodb pluhgin. by @jmjoy in https://github.com/apache/skywalking-php/pull/83 Update rust nightly toolchain in CI and format. by @jmjoy in https://github.com/apache/skywalking-php/pull/84 Add notice document for skywalking_agent.enable. by @jmjoy in https://github.com/apache/skywalking-php/pull/85 Upgrade dependencies. by @jmjoy in https://github.com/apache/skywalking-php/pull/86 Fix docs by @heyanlong in https://github.com/apache/skywalking-php/pull/87 Add kafka reporter. by @jmjoy in https://github.com/apache/skywalking-php/pull/88 Release SkyWalking PHP Agent 0.6.0 by @jmjoy in https://github.com/apache/skywalking-php/pull/89 New Contributors @matikij made their first contribution in https://github.com/apache/skywalking-php/pull/76 Full Changelog: https://github.com/apache/skywalking-php/compare/v0.5.0...v0.6.0\nPECL https://pecl.php.net/package/skywalking_agent/0.6.0\n","excerpt":"\u003cp\u003eSkyWalking PHP 0.6.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-php-0-6-0/","title":"Release Apache SkyWalking PHP 0.6.0"},{"body":"On Aug. 10th, 2023, HashiCorp announced to adopt the Business Source License (BSL) from Mozilla Public License v2.0 (MPL 2.0), here is their post. They officially annouced they have changed the license for the ALL of their open-source products from the previous MPL 2.0 to a source-available license, BSL 1.1. Meanwhile, HashiCorp APIs, SDKs, and almost all other libraries will remain MPL 2.0.\nHashiCorp Inc. is one of the most important vendors in the cloud-native landscape, as well as Golang ecosystem. This kind of changes would have potential implications for SkyWalking, which is closely integrated with cloud-native technology stacks.\nConclusion First What does that mean for SkyWalking users? SkyWalking community has evaluated our dependencies from HashiCorp products and libraries, the current conclusion is\nSkyWalking users would NOT suffer any implication. All components of SkyWalking don\u0026rsquo;t have hard-dependency on BSL license affected codes.\nSkyWalking community have found out all following dependencies of all relative repositories, all licenses are TRUELY stayed unchanged, and compatible with Apache 2.0 License.\nOAP Server @kezhenxu94 @wu-sheng consul-client Apache 2.0 Repo archived on Jul 27, 2023 BanyanDB @hanahmily @lujiajing1126 Server @hanahmily hashicorp/golang-lru MPL-2.0 hashicorp/hcl MPL-2.0 CLI @hanahmily No HashiCorp Dependency SkyWalking OAP CLI @kezhenxu94 github.com/hashicorp/hcl v1.0.0 MPL-2.0 All under swck as transitive dependencies SWCK @hanahmily hashicorp/consul/api MPL-2.0 hashicorp/consul/sdk MPL-2.0 hashicorp/errwrap MPL-2.0 hashicorp/go-cleanhttp MPL-2.0 hashicorp/go-immutable-radix MPL-2.0 hashicorp/go-msgpack MIT hashicorp/go-multierror MPL-2.0 hashicorp/go-rootcerts MPL-2.0 hashicorp/go-sockaddr MPL-2.0 hashicorp/go-syslog MIT hashicorp/go-uuid MPL-2.0 hashicorp/go.net BSD-3 hashicorp/golang-lru MPL-2.0 hashicorp/hcl MPL-2.0 hashicorp/logutils MPL-2.0 hashicorp/mdns MIT hashicorp/memberlist MPL-2.0 hashicorp/serf MPL-2.0 Go agent @mrproliu hashicorp/consul/api MPL-2.0 hashicorp/consul/sdk MPL-2.0 hashicorp/errwrap MPL-2.0 hashicorp/go-cleanhttp MPL-2.0 hashicorp/go-hclog MIT hashicorp/go-immutable-radix MPL-2.0 hashicorp/go-kms-wrapping/entropy MPL-2.0 hashicorp/go-kms-wrapping/entropy/v2 MPL-2.0 hashicorp/go-msgpack MIT hashicorp/go-multierror MPL-2.0 hashicorp/go-plugin MPL-2.0 hashicorp/go-retryablehttp MPL-2.0 hashicorp/go-rootcerts MPL-2.0 hashicorp/go-secure-stdlib/base62 MPL-2.0 hashicorp/go-secure-stdlib/mlock MPL-2.0 hashicorp/go-secure-stdlib/parseutil MPL-2.0 hashicorp/go-secure-stdlib/password MPL-2.0 hashicorp/go-secure-stdlib/tlsutil MPL-2.0 hashicorp/go-sockaddr MPL-2.0 hashicorp/go-syslog MIT hashicorp/go-uuid MPL-2.0 hashicorp/go-version MPL-2.0 hashicorp/go.net BSD-3-Clause hashicorp/golang-lru MPL-2.0 hashicorp/logutils MPL-2.0 hashicorp/mdns MIT hashicorp/memberlist MPL-2.0 hashicorp/serf MPL-2.0 hashicorp/vault/api MPL-2.0 hashicorp/vault/sdk MPL-2.0 hashicorp/yamux MPL-2.0 SkyWalking eyes @kezhenxu94 none SkyWalking Infra e2e @kezhenxu94 all under swck as transitive dependencies SkyWalking rover(ebpf agent) @mrproliu hashicorp/consul/api MPL-2.0 hashicorp/consul/sdk MPL-2.0 hashicorp/errwrap MPL-2.0 hashicorp/go-cleanhttp MPL-2.0 hashicorp/go-hclog MIT hashicorp/go-immutable-radix MPL-2.0 hashicorp/go-msgpack MIT hashicorp/go-multierror MPL-2.0 hashicorp/go-retryablehttp MPL-2.0 hashicorp/go-rootcerts MPL-2.0 hashicorp/go-sockaddr MPL-2.0 hashicorp/go-syslog MIT hashicorp/go-uuid MPL-2.0 hashicorp/golang-lru MPL-2.0 hashicorp/hcl MPL-2.0 hashicorp/logutils MPL-2.0 hashicorp/mdns MIT hashicorp/memberlist MPL-2.0 hashicorp/serf MPL-2.0 SkyWalking satellite @mrproliu hashicorp/consul/api MPL-2.0 hashicorp/consul/sdk MPL-2.0 hashicorp/errwrap MPL-2.0 hashicorp/go-cleanhttp MPL-2.0 hashicorp/go-immutable-radix MPL-2.0 hashicorp/go-msgpack MIT hashicorp/go-multierror MPL-2.0 hashicorp/go-rootcerts MPL-2.0 hashicorp/go-sockaddr MPL-2.0 hashicorp/go-syslog MIT hashicorp/go-uuid MPL-2.0 hashicorp/go.net BSD-3-Clause hashicorp/golang-lru MPL-2.0 hashicorp/hcl MPL-2.0 hashicorp/logutils MPL-2.0 hashicorp/mdns MIT hashicorp/memberlist MPL-2.0 hashicorp/serf MPL-2.0 SkyWalking Terraform (scripts) @kezhenxu94 No HashiCorp Dependency The scripts for Terraform users only. No hard requirement. The GitHub ID is listed about the PMC members did the evaluations.\nFAQ If I am using Consul to manage SkyWalking Cluster or configurations, does this license change bring an implication? YES, anyone using their server sides would be affected once you upgrade to later released versions after Aug. 10th, 2023.\nThis is HashiCorp\u0026rsquo;s statement\nEnd users can continue to copy, modify, and redistribute the code for all non-commercial and commercial use, except where providing a competitive offering to HashiCorp. Partners can continue to build integrations for our joint customers. We will continue to work closely with the cloud service providers to ensure deep support for our mutual technologies. Customers of enterprise and cloud-managed HashiCorp products will see no change as well. Vendors who provide competitive services built on our community products will no longer be able to incorporate future releases, bug fixes, or security patches contributed to our products.\nSo, notice that, the implication about whether voilating BSL 1.1 is determined by the HashiCorp Inc about the status of the identified competitive relationship. We can\u0026rsquo;t provide any suggestions. Please refer to FAQs and contacts for the official explanations.\nWill SkyWalking continoue to use HashiCorp Consul as an optional cluster coordinator and/or an optional dynamic configuration server? For short term, YES, we will keep that part of codes, as the licenses of the SDK and the APIs are still in the MPL 2.0.\nBut, during the evaluation, we noticed the consul client we are using is rickfast/consul-client which had been archived by the owner on Jul 27, 2023. So, we are facing the issues that no maintaining and no version to upgrade. If there is not a new consul Java client lib available, we may have to remove this to avoid CVEs or version incompatible with new released servers.\n","excerpt":"\u003cp\u003eOn Aug. 10th, 2023, HashiCorp announced to adopt the Business Source License (BSL) from Mozilla …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2023-08-13-hashicorp-bsl/","title":"The Statement for SkyWalking users on HashiCorp license changes"},{"body":"SkyWalking Rust 0.8.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed Add kafka reporter. by @jmjoy in https://github.com/apache/skywalking-rust/pull/61 Rename AbstractSpan to HandleSpanObject. by @jmjoy in https://github.com/apache/skywalking-rust/pull/62 Bump to 0.8.0. by @jmjoy in https://github.com/apache/skywalking-rust/pull/63 ","excerpt":"\u003cp\u003eSkyWalking Rust 0.8.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-rust-0-8-0/","title":"Release Apache SkyWalking Rust 0.8.0"},{"body":"SkyWalking Cloud on Kubernetes 0.8.0 is released. Go to downloads page to find release tars.\nFeatures [Breaking Change] Remove the way to configure the agent through Configmap. Bugs Fix errors in banyandb e2e test. Chores Bump up golang to v1.20. Bump up golangci-lint to v1.53.3. Bump up skywalking-java-agent to v8.16.0. Bump up kustomize to v4.5.6. Bump up SkyWalking OAP to 9.5.0. ","excerpt":"\u003cp\u003eSkyWalking Cloud on Kubernetes 0.8.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars. …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cloud-on-kubernetes-0-8-0/","title":"Release Apache SkyWalking Cloud on Kubernetes 0.8.0"},{"body":"Announcing Apache SkyWalking Go 0.2.0 I\u0026rsquo;m excited to announce the release of Apache SkyWalking Go 0.2.0! This version packs several awesome new features that I\u0026rsquo;ll overview below.\nLog Reporting The log reporting feature allows the Go agent to automatically collect log content from supported logging frameworks like logrus and zap. The logs are organized and sent to the SkyWalking backend for visualization. You can see how the logs appear for each service in the SkyWalking UI:\nMaking Logs Searchable You can configure certain log fields to make them searchable in SkyWalking. Set the SW_AGENT_LOG_REPORTER_LABEL_KEYS environment variable to include additional fields beyond the default log level.\nFor example, with logrus:\n# define log with fields logrus.WithField(\u0026#34;module\u0026#34;, \u0026#34;test-service\u0026#34;).Info(\u0026#34;test log\u0026#34;) Metrics Reporting The agent can now collect and report custom metrics data from runtime/metrics to the backend. Supported metrics are documented here.\nAutomatic Instrumentation In 0.1.0, you had to manually integrate the agent into your apps. Now, the new commands can automatically analyze and instrument projects at a specified path, no code changes needed! Try using the following command to import skywalking-go into your project:\n# inject to project at current path skywalking-go-agent -inject=./ -all Or you can still use the original manual approach if preferred.\nGet It Now! Check out the CHANGELOG for the full list of additions and fixes. I encourage you to try out SkyWalking Go 0.2.0 today! Let me know if you have any feedback.\n","excerpt":"\u003ch1 id=\"announcing-apache-skywalking-go-020\"\u003eAnnouncing Apache SkyWalking Go 0.2.0\u003c/h1\u003e\n\u003cp\u003eI\u0026rsquo;m excited to announce the release of Apache SkyWalking …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2023-07-31-skywalking-go-0.2.0-release/","title":"New Features of SkyWalking Go 0.2.0"},{"body":"SkyWalking Go 0.2.0 is released. Go to downloads page to find release tars.\nFeatures Enhance the plugin rewrite ability to support switch and if/else in the plugin codes. Support inject the skywalking-go into project through agent. Support add configuration for plugin. Support metrics report API for plugin. Support report Golang runtime metrics. Support log reporter. Enhance the logrus logger plugin to support adapt without any settings method invoke. Disable sending observing data if the gRPC connection is not established for reducing the connection error log. Support enhance vendor management project. Support using base docker image to building the application. Plugins Support go-redis v9 redis client framework. Support collecting Native HTTP URI parameter on server side. Support Mongo database client framework. Support Native SQL database client framework with MySQL Driver. Support Logrus log report to the backend. Support Zap log report to the backend. Documentation Combine Supported Libraries and Performance Test into Plugins section. Add Tracing, Metrics and Logging document into Plugins section. Bug Fixes Fix throw panic when log the tracing context before agent core initialized. Fix plugin version matcher tryToFindThePluginVersion to support capital letters in module paths and versions. Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Go 0.2.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eEnhance the …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-go-0.2.0/","title":"Release Apache SkyWalking Go 0.2.0"},{"body":"今年 COSCUP 2023 在国立台湾科技大学举办。 COSCUP 是由台湾开放原始码社群联合推动的年度研讨会，起源于2006年，是台湾自由软体运动 (FOSSM) 重要的推动者之一。活动包括有讲座、摊位、社团同乐会等，除了邀请国际的重量级演讲者之外，台湾本土的自由软体推动者也经常在此发表演说，会议的发起人、工作人员与演讲者都是志愿参与的志工。COSCUP 的宗旨在于提供一个连接开放原始码开发者、使用者与推广者的平台。希望借由每年一度的研讨会来推动自由及开放原始码软体 (FLOSS)。由于有许多赞助商及热心捐助者，所有议程都是免费参加。\n在Go语言中使用自动增强探针完成链路追踪以及监控 B站视频地址\n刘晗，Tetrate\n讲师介绍 刘晗，Tetrate 工程师，Apache SkyWalking PMC 成员，专注于应用性能可观测性领域。\n议题概要\n为什么需要自动增强探针 Go Agent演示 实现原理 未来展望 ","excerpt":"\u003cp\u003e今年 COSCUP 2023 在国立台湾科技大学举办。\nCOSCUP 是由台湾开放原始码社群联合推动的年度研讨会，起源于2006年，是台湾自由软体运动 (FOSSM) 重要的推动者之一。活动包括有讲座 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2023-07-30-complete-auto-instrumentation-go-agent-for-distributed-tracing-and-monitoring/","title":"[视频] 在Go语言中使用自动增强探针完成链路追踪以及监控 - COSCUP Taiwan 2023"},{"body":"SkyWalking Kubernetes Helm Chart 4.5.0 is released. Go to downloads page to find release tars.\nAdd helm chart for swck v0.7.0. Add pprof port export in satellite. Trunc the resource name in swck\u0026rsquo;s helm chart to no more than 63 characters. Adding the configmap into cluster role for oap init mode. Add config to set Pod securityContext. Keep the job name prefix the same as OAP Deployment name. Use startup probe option for first initialization of application Allow setting env for UI deployment. Add Istio ServiceEntry permissions. ","excerpt":"\u003cp\u003eSkyWalking Kubernetes Helm Chart 4.5.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAdd …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-kubernetes-helm-chart-4.5.0/","title":"Release Apache SkyWalking Kubernetes Helm Chart 4.5.0"},{"body":"SkyWalking BanyanDB 0.4.0 is released. Go to downloads page to find release tars.\nFeatures Add TSDB concept document. [UI] Add YAML editor for inputting query criteria. Refactor TopN to support NULL group while keeping seriesID from the source measure. Add a sharded buffer to TSDB to replace Badger\u0026rsquo;s memtable. Badger KV only provides SST. Add a meter system to control the internal metrics. Add multiple metrics for measuring the storage subsystem. Refactor callback of TopNAggregation schema event to avoid deadlock and reload issue. Fix max ModRevision computation with inclusion of TopNAggregation Enhance meter performance Reduce logger creation frequency Add units to memory flags Introduce TSTable to customize the block\u0026rsquo;s structure Add /system endpoint to the monitoring server that displays a list of nodes\u0026rsquo; system information. Enhance the liaison module by implementing access logging. Add the Istio scenario stress test based on the data generated by the integration access log. Generalize the index\u0026rsquo;s docID to uint64. Remove redundant ID tag type. Improve granularity of index in measure by leveling up from data point to series. [UI] Add measure CRUD operations. [UI] Add indexRule CRUD operations. [UI] Add indexRuleBinding CRUD operations. Bugs Fix iterator leaks and ensure proper closure and introduce a closer to guarantee all iterators are closed Fix resource corrupts caused by update indexRule operation Set the maximum integer as the limit for aggregation or grouping operations when performing aggregation or grouping operations in a query plan. Chores Bump go to 1.20. Set KV\u0026rsquo;s minimum memtable size to 8MB [docs] Fix docs crud examples error Modified TestGoVersion to check for CPU architecture and Go Version Bump node to 18.16 ","excerpt":"\u003cp\u003eSkyWalking BanyanDB 0.4.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"features\"\u003eFeatures\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eAdd TSDB …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-0-4-0/","title":"Release Apache SkyWalking BanyanDB 0.4.0"},{"body":"Background In previous articles, We have discussed how to use SkyWalking and eBPF for performance problem detection within processes and networks. They are good methods to locate issues, but still there are some challenges:\nThe timing of the task initiation: It\u0026rsquo;s always challenging to address the processes that require performance monitoring when problems occur. Typically, manual engagement is required to identify processes and the types of performance analysis necessary, which cause extra time during the crash recovery. The root cause locating and the time of crash recovery conflict with each other from time to time. In the real case, rebooting would be the first choice of recovery, meanwhile, it destroys the site of crashing. Resource consumption of tasks: The difficulties to determine the profiling scope. Wider profiling causes more resources than it should. We need a method to manage resource consumption and understand which processes necessitate performance analysis. Engineer capabilities: On-call is usually covered by the whole team, which have junior and senior engineers, even senior engineers have their understanding limitation of the complex distributed system, it is nearly impossible to understand the whole system by a single one person. The Continuous Profiling is a new created mechanism to resolve the above issues.\nAutomate Profiling As profiling is resource costing and high experience required, how about introducing a method to narrow the scope and automate the profiling driven by polices creates by senior SRE engineer? So, in 9.5.0, SkyWalking first introduced preset policy rules for specific services to be monitored by the eBPF Agent in a low-energy manner, and run profiling when necessary automatically.\nPolicy Policy rules specify how to monitor target processes and determine the type of profiling task to initiate when certain threshold conditions are met.\nThese policy rules primarily consist of the following configuration information:\nMonitoring type: This specifies what kind of monitoring should be implemented on the target process. Threshold determination: This defines how to determine whether the target process requires the initiation of a profiling task. Trigger task: This specifies what kind of performance analysis task should be initiated. Monitoring type The type of monitoring is determined by observing the data values of a specified process to generate corresponding metrics. These metric values can then facilitate subsequent threshold judgment operations. In eBPF observation, we believe the following metrics can most directly reflect the current performance of the program:\nMonitor Type Unit Description System Load Load System load average over a specified period. Process CPU Percentage The CPU usage of the process as a percentage. Process Thread Count Count The number of threads in the process. HTTP Error Rate Percentage The percentage of HTTP requests that result in error responses (e.g., 4xx or 5xx status codes). HTTP Avg Response Time Millisecond The average response time for HTTP requests. Network related monitoring Monitoring network type metrics is not as simple as obtaining basic process information. It requires the initiation of eBPF programs and attaching them to the target process for observation. This is similar to the principles of network profiling task we introduced in the previous article, except that we no longer collect the full content of the data packets. Instead, we only collect the content of messages that match specified HTTP prefixes.\nBy using this method, we can significantly reduce the number of times the kernel sends data to the user space, and the user-space program can parse the data content with less system resource usage. This ultimately helps in conserving system resources.\nMetrics collector The eBPF agent would report metrics of processes periodically as follows to indicate the process performance in time.\nName Unit Description process_cpu (0-100)% The CPU usage percent process_thread_count count The thread count of process system_load count The average system load for the last minute, each process have same value http_error_rate (0-100)% The network request error rate percentage http_avg_response_time ms The network average response duration Threshold determination For the threshold determination, the judgement is made by the eBPF Agent based on the target monitoring process in its own memory, rather than relying on calculations performed by the SkyWalking backend. The advantage of this approach is that it doesn\u0026rsquo;t have to wait for the results of complex backend computations, and it reduces potential issues brought about by complicated interactions.\nBy using this method, the eBPF Agent can swiftly initiate tasks immediately after conditions are met, without any delay.\nIt includes the following configuration items:\nThreshold: Check if the monitoring value meets the specified expectations. Period: The time period(seconds) for monitoring data, which can also be understood as the most recent duration. Count: The number of times(seconds) the threshold is triggered within the detection period, which can also be understood as the total number of times the specified threshold rule is triggered in the most recent duration(seconds). Once the count check is met, the specified Profiling task will be started. Trigger task When the eBPF Agent detects that the threshold determination in the specified policy meets the rules, it can initiate the corresponding task according to pre-configured rules. For each different target performance task, their task initiation parameters are different:\nOn/Off CPU Profiling: It automatically performs performance analysis on processes that meet the conditions, defaulting to 10 minutes of monitoring. Network Profiling: It performs network performance analysis on all processes in the same Service Instance on the current machine, to prevent the cause of the issue from being unrealizable due to too few process being collected, defaulting to 10 minutes of monitoring. Once the task is initiated, no new profiling tasks would be started for the current process for a certain period. The main reason for this is to prevent frequent task creation due to low threshold settings, which could affect program execution. The default time period is 20 minutes.\nData Flow The figure 1 illustrates the data flow of the continuous profiling feature:\nFigure 1: Data Flow of Continuous Profiling\neBPF Agent with Process Firstly, we need to ensure that the eBPF Agent and the process to be monitored are deployed on the same host machine, so that we can collect relevant data from the process. When the eBPF Agent detects a threshold validation rule that conforms to the policy, it immediately triggers the profiling task for the target process, thereby reducing any intermediate steps and accelerating the ability to pinpoint performance issues.\nSliding window The sliding window plays a crucial role in the eBPF Agent\u0026rsquo;s threshold determination process, as illustrated in the figure 2:\nFigure 2: Sliding Window in eBPF Agent\nEach element in the array represents the data value for a specified second in time. When the sliding window needs to verify whether it is responsible for a rule, it fetches the content of each element from a certain number of recent elements (period parameter). If an element exceeds the threshold, it is marked in red and counted. If the number of red elements exceeds a certain number, it is deemed to trigger a task.\nUsing a sliding window offers the following two advantages:\nFast retrieval of recent content: With a sliding window, complex calculations are unnecessary. You can know the data by simply reading a certain number of recent array elements. Solving data spikes issues: Validation through count prevents situations where a data point suddenly spikes and then quickly returns to normal. Verification with multiple values can reveal whether exceeding the threshold is frequent or occasional. eBPF Agent with SkyWalking Backend The eBPF Agent communicates periodically with the SkyWalking backend, involving three most crucial operations:\nPolicy synchronization: Through periodic policy synchronization, the eBPF Agent can keep processes on the local machine updated with the latest policy rules as much as possible. Metrics sending: For processes that are already being monitored, the eBPF Agent periodically sends the collected data to the backend program. This facilitates real-time query of current data values by users, who can also compare this data with historical values or thresholds when problems arise. Profiling task reporting: When the eBPF detects that a certain process has triggered a policy rule, it automatically initiates a performance task, collects relevant information from the current process, and reports it to the SkyWalking backend. This allows users to know when, why, and what type of profiling task was triggered from the interface. Demo Next, let\u0026rsquo;s quickly demonstrate the continuous profiling feature, so you can understand more specifically what it accomplishes.\nDeploy SkyWalking Showcase SkyWalking Showcase contains a complete set of example services and can be monitored using SkyWalking. For more information, please check the official documentation.\nIn this demo, we only deploy service, the latest released SkyWalking OAP, and UI.\nexport SW_OAP_IMAGE=apache/skywalking-oap-server:9.5.0 export SW_UI_IMAGE=apache/skywalking-ui:9.5.0 export SW_ROVER_IMAGE=apache/skywalking-rover:0.5.0 export FEATURE_FLAGS=mesh-with-agent,single-node,elasticsearch,rover make deploy.kubernetes After deployment is complete, please run the following script to open SkyWalking UI: http://localhost:8080/.\nkubectl port-forward svc/ui 8080:8080 --namespace default Create Continuous Profiling Policy Currently, continues profiling feature is set by default in the Service Mesh panel at the Service level.\nFigure 3: Continuous Policy Tab\nBy clicking on the edit button aside from the Policy List, the polices of current service could be created or updated.\nFigure 4: Edit Continuous Profiling Policy\nMultiple polices are supported. Every policy has the following configurations.\nTarget Type: Specifies the type of profiling task to be triggered when the threshold determination is met. Items: For profiling task of the same target, one or more validation items can be specified. As long as one validation item meets the threshold determination, the corresponding performance analysis task will be launched. Monitor Type: Specifies the type of monitoring to be carried out for the target process. Threshold: Depending on the type of monitoring, you need to fill in the corresponding threshold to complete the verification work. Period: Specifies the number of recent seconds of data you want to monitor. Count: Determines the total number of seconds triggered within the recent period. URI Regex/List: This is applicable to HTTP monitoring types, allowing URL filtering. Done After clicking the save button, you can see the currently created monitoring rules, as shown in the figure 5:\nFigure 5: Continuous Profiling Monitoring Processes\nThe data can be divided into the following parts:\nPolicy list: On the left, you can see the rule list you have created. Monitoring Summary List: Once a rule is selected, you can see which pods and processes would be monitored by this rule. It also summarizes how many profiling tasks have been triggered in the last 48 hours by the current pod or process, as well as the last trigger time. This list is also sorted in descending order by the number of triggers to facilitate your quick review. When you click on a specific process, a new dashboard would show to list metrics and triggered profiling results.\nFigure 6: Continuous Profiling Triggered Tasks\nThe current figure contains the following data contents:\nTask Timeline: It lists all profiling tasks in the past 48 hours. And when the mouse hovers over a task, it would also display detailed information: Task start and end time: It indicates when the current performance analysis task was triggered. Trigger reason: It would display the reason why the current process was profiled and list out the value of the metric exceeding the threshold when the profiling was triggered. so you can quickly understand the reason. Task Detail: Similar to the CPU Profiling and Network Profiling introduced in previous articles, this would display the flame graph or process topology map of the current task, depending on the profiling type. Meanwhile, on the Metrics tab, metrics relative to profiling policies are collected to retrieve the historical trend, in order to provide a comprehensive explanation of the trigger point about the profiling.\nFigure 7: Continuous Profiling Metrics\nConclusion In this article, I have detailed how the continuous profiling feature in SkyWalking and eBPF works. In general, it involves deploying the eBPF Agent service on the same machine where the process to be monitored resides, and monitoring the target process with low resource consumption. When it meets the threshold conditions, it would initiate more complex CPU Profiling and Network Profiling tasks.\nIn the future, we will offer even more features. Stay tuned!\nTwitter, ASFSkyWalking Slack. Send Request to join SkyWalking slack mail to the mail list(dev@skywalking.apache.org), we will invite you in. Subscribe to our medium list. ","excerpt":"\u003ch1 id=\"background\"\u003eBackground\u003c/h1\u003e\n\u003cp\u003eIn previous articles, We have discussed how to use SkyWalking and eBPF for performance …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2023-06-25-intruducing-continuous-profiling-skywalking-with-ebpf/","title":"Activating Automatical Performance Analysis -- Continuous Profiling"},{"body":"SkyWalking CLI 0.12.0 is released. Go to downloads page to find release tars.\nAdd the sub-command records list for adapt the new record query API by @mrproliu in https://github.com/apache/skywalking-cli/pull/167 Add the attached events fields into the trace sub-command by @mrproliu in https://github.com/apache/skywalking-cli/pull/169 Add the sampling config file into the profiling ebpf create network sub-command by @mrproliu in https://github.com/apache/skywalking-cli/pull/171 Add the sub-command profiling continuous for adapt the new continuous profiling API by @mrproliu in https://github.com/apache/skywalking-cli/pull/173 Adapt the sub-command metrics for deprecate scope fron entity by @mrproliu in https://github.com/apache/skywalking-cli/pull/173 Add components in topology related sub-commands. @mrproliu in https://github.com/apache/skywalking-cli/pull/175 Add the sub-command metrics nullable for query the nullable metrics value. @mrproliu in https://github.com/apache/skywalking-cli/pull/176 Adapt the sub-command profiling trace for adapt the new trace profiling protocol. @mrproliu in https://github.com/apache/skywalking-cli/pull/177 Add isEmptyValue field in metrics related sub-commands. @mrproliu in https://github.com/apache/skywalking-cli/pull/180 Add the sub-command metrics execute for execute the metrics query. @mrproliu in https://github.com/apache/skywalking-cli/pull/182 Add the sub-command profiling continuous monitoring for query all continuous profiling monitoring instances. @mrproliu in https://github.com/apache/skywalking-cli/pull/182 Add continuousProfilingCauses.message field in the profiling ebpf list comamnds by @mrproliu in https://github.com/apache/skywalking-cli/pull/184 ","excerpt":"\u003cp\u003eSkyWalking CLI 0.12.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAdd the sub-command …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cli-0-12-0/","title":"Release Apache SkyWalking CLI 0.12.0"},{"body":"SkyWalking Rover 0.5.0 is released. Go to downloads page to find release tars.\nFeatures Enhance the protocol reader for support long socket data. Add the syscall level event to the trace. Support OpenSSL 3.0.x. Optimized the data structure in BPF. Support continuous profiling. Improve the performance when getting goid in eBPF. Support build multiple architecture docker image: x86_64, arm64. Bug Fixes Fix HTTP method name in protocol analyzer. Fixed submitting multiple network profiling tasks with the same uri causing the rover to restart. Documentation Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Rover 0.5.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eEnhance the …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-rover-0-5-0/","title":"Release Apache SkyWalking Rover 0.5.0"},{"body":"SkyWalking Satellite 1.2.0 is released. Go to downloads page to find release tars.\nFeatures Introduce pprof module. Support export multiple telemetry service. Update the base docker image. Add timeout configuration for gRPC client. Reduce log print when the enqueue data to the pipeline error. Support transmit the Continuous Profiling protocol. Bug Fixes Fix CVE-2022-41721. Use Go 19 to build the Docker image to fix CVEs. Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Satellite 1.2.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures …\u003c/h4\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-satellite-1-2-0/","title":"Release Apache SkyWalking Satellite 1.2.0"},{"body":"背景 在之前的文章中，我们讨论了如何使用 SkyWalking 和 eBPF 来检测性能问题，包括进程和网络。这些方法可以很好地定位问题，但仍然存在一些挑战：\n任务启动的时间: 当需要进行性能监控时，解决需要性能监控的进程始终是一个挑战。通常需要手动参与，以标识进程和所需的性能分析类型，这会在崩溃恢复期间耗费额外的时间。根本原因定位和崩溃恢复时间有时会发生冲突。在实际情况中，重新启动可能是恢复的第一选择，同时也会破坏崩溃的现场。 任务的资源消耗: 确定分析范围的困难。过宽的分析范围会导致需要更多的资源。我们需要一种方法来管理资源消耗并了解哪些进程需要性能分析。 工程师能力: 通常由整个团队负责呼叫，其中有初级和高级工程师，即使是高级工程师也对复杂的分布式系统有其理解限制，单个人几乎无法理解整个系统。 持续剖析（Continuous Profiling） 是解决上述问题的新机制。\n自动剖析 由于性能分析的资源消耗和高经验要求，因此引入一种方法以缩小范围并由高级 SRE 工程师创建策略自动剖析。因此，在 9.5.0 中，SkyWalking 首先引入了预设策略规则，以低功耗方式监视特定服务的 eBPF 代理，并在必要时自动运行剖析。\n策略 策略规则指定了如何监视目标进程并确定在满足某些阈值条件时应启动何种类型的分析任务。\n这些策略规则主要包括以下配置信息：\n监测类型: 这指定了应在目标进程上实施什么样的监测。 阈值确定: 这定义了如何确定目标进程是否需要启动分析任务。 触发任务: 这指定了应启动什么类型的性能分析任务。 监测类型 监测类型是通过观察指定进程的数据值来生成相应的指标来确定的。这些指标值可以促进后续的阈值判断操作。在 eBPF 观测中，我们认为以下指标最能直接反映程序的当前性能：\n监测类型 单位 描述 系统负载 负载 在指定时间段内的系统负载平均值。 进程 CPU 百分比 进程的 CPU 使用率百分比。 进程线程计数 计数 进程中的线程数。 HTTP 错误率 百分比 导致错误响应（例如，4xx 或 5xx 状态代码）的 HTTP 请求的百分比。 HTTP 平均响应时间 毫秒 HTTP 请求的平均响应时间。 相关网络监测 监测网络类型的指标不像获取基本进程信息那么简单。它需要启动 eBPF 程序并将其附加到目标进程以进行观测。这类似于我们在先前文章中介绍的网络分析任务，不同的是我们不再收集数据包的完整内容。相反，我们仅收集与指定 HTTP 前缀匹配的消息的内容。\n通过使用此方法，我们可以大大减少内核向用户空间发送数据的次数，用户空间程序可以使用更少的系统资源来解析数据内容。这最终有助于节省系统资源。\n指标收集器 eBPF 代理会定期报告以下进程度量，以指示进程性能：\n名称 单位 描述 process_cpu (0-100)% CPU 使用率百分比 process_thread_count 计数 进程中的线程数 system_load 计数 最近一分钟的平均系统负载，每个进程的值相同 http_error_rate (0-100)% 网络请求错误率百分比 http_avg_response_time 毫秒 网络平均响应持续时间 阈值确定 对于阈值的确定，eBPF 代理是基于其自身内存中的目标监测进程进行判断，而不是依赖于 SkyWalking 后端执行的计算。这种方法的优点在于，它不必等待复杂后端计算的结果，减少了复杂交互所带来的潜在问题。\n通过使用此方法，eBPF 代理可以在条件满足后立即启动任务，而无需任何延迟。\n它包括以下配置项：\n阈值: 检查监测值是否符合指定的期望值。 周期: 监控数据的时间周期（秒），也可以理解为最近的持续时间。 计数: 检测期间触发阈值的次数（秒），也可以理解为最近持续时间内指定阈值规则触发的总次数（秒）。一旦满足计数检查，指定的分析任务将被开始。 触发任务 当 eBPF Agent 检测到指定策略中的阈值决策符合规则时，根据预配置的规则可以启动相应的任务。对于每个不同的目标性能任务，它们的任务启动参数都不同：\nOn/Off CPU Profiling: 它会自动对符合条件的进程进行性能分析，缺省情况下监控时间为 10 分钟。 Network Profiling: 它会对当前机器上同一 Service Instance 中的所有进程进行网络性能分析，以防问题的原因因被收集进程太少而无法实现，缺省情况下监控时间为 10 分钟。 一旦任务启动，当前进程将在一定时间内不会启动新的剖析任务。主要原因是为了防止因低阈值设置而频繁创建任务，从而影响程序执行。缺省时间为 20 分钟。\n数据流 图 1 展示了持续剖析功能的数据流：\n图 1: 持续剖析的数据流\neBPF Agent进行进程跟踪 首先，我们需要确保 eBPF Agent 和要监测的进程部署在同一台主机上，以便我们可以从进程中收集相关数据。当 eBPF Agent 检测到符合策略的阈值验证规则时，它会立即为目标进程触发剖析任务，从而减少任何中间步骤并加速定位性能问题的能力。\n滑动窗口 滑动窗口在 eBPF Agent 的阈值决策过程中发挥着至关重要的作用，如图 2 所示：\n图 2: eBPF Agent 中的滑动窗口\n数组中的每个元素表示指定时间内的数据值。当滑动窗口需要验证是否负责某个规则时，它从最近的一定数量的元素 (period 参数) 中获取每个元素的内容。如果一个元素超过了阈值，则标记为红色并计数。如果红色元素的数量超过一定数量，则被认为触发了任务。\n使用滑动窗口具有以下两个优点：\n快速检索最近的内容：使用滑动窗口，无需进行复杂的计算。你可以通过简单地读取一定数量的最近数组元素来了解数据。 解决数据峰值问题：通过计数进行验证，可以避免数据点突然增加然后快速返回正常的情况。使用多个值进行验证可以揭示超过阈值是频繁还是偶然发生的。 eBPF Agent与OAP后端通讯 eBPF Agent 定期与 SkyWalking 后端通信，涉及三个最关键的操作：\n策略同步：通过定期的策略同步，eBPF Agent 可以尽可能地让本地机器上的进程与最新的策略规则保持同步。 指标发送：对于已经被监视的进程，eBPF Agent 定期将收集到的数据发送到后端程序。这就使用户能够实时查询当前数据值，用户也可以在出现问题时将此数据与历史值或阈值进行比较。 剖析任务报告：当 eBPF 检测到某个进程触发了策略规则时，它会自动启动性能任务，从当前进程收集相关信息，并将其报告给 SkyWalking 后端。这使用户可以从界面了解何时、为什么和触发了什么类型的剖析任务。 演示 接下来，让我们快速演示持续剖析功能，以便你更具体地了解它的功能。\n部署 SkyWalking Showcase SkyWalking Showcase 包含完整的示例服务，并可以使用 SkyWalking 进行监视。有关详细信息，请查看官方文档。\n在此演示中，我们只部署服务、最新发布的 SkyWalking OAP 和 UI。\nexport SW_OAP_IMAGE=apache/skywalking-oap-server:9.5.0 export SW_UI_IMAGE=apache/skywalking-ui:9.5.0 export SW_ROVER_IMAGE=apache/skywalking-rover:0.5.0 export FEATURE_FLAGS=mesh-with-agent,single-node,elasticsearch,rover make deploy.kubernetes 部署完成后，请运行以下脚本以打开 SkyWalking UI：http://localhost:8080/。\nkubectl port-forward svc/ui 8080:8080 --namespace default 创建持续剖析策略 目前，持续剖析功能在 Service Mesh 面板的 Service 级别中默认设置。\n图 3: 持续策略选项卡\n通过点击 Policy List 旁边的编辑按钮，可以创建或更新当前服务的策略。\n图 4: 编辑持续剖析策略\n支持多个策略。每个策略都有以下配置。\nTarget Type：指定符合阈值决策时要触发的剖析任务的类型。 Items：对于相同目标的剖析任务，可以指定一个或多个验证项目。只要一个验证项目符合阈值决策，就会启动相应的性能分析任务。 Monitor Type：指定要为目标进程执行的监视类型。 Threshold：根据监视类型的不同，需要填写相应的阈值才能完成验证工作。 Period：指定你要监测的最近几秒钟的数据数量。 Count：确定最近时间段内触发的总秒数。 URI 正则表达式/列表：这适用于 HTTP 监控类型，允许 URL 过滤。 完成 单击保存按钮后，你可以看到当前已创建的监控规则，如图 5 所示：\n图 5: 持续剖析监控进程\n数据可以分为以下几个部分：\n策略列表：在左侧，你可以看到已创建的规则列表。 监测摘要列表：选择规则后，你可以看到哪些 pod 和进程将受到该规则的监视。它还总结了当前 pod 或进程在过去 48 小时内触发的性能分析任务数量，以及最后一个触发时间。该列表还按触发次数降序排列，以便你快速查看。 当你单击特定进程时，将显示一个新的仪表板以列出指标和触发的剖析结果。\n图 6: 持续剖析触发的任务\n当前图包含以下数据内容：\n任务时间轴：它列出了过去 48 小时的所有剖析任务。当鼠标悬停在任务上时，它还会显示详细信息： 任务的开始和结束时间：它指示当前性能分析任务何时被触发。 触发原因：它会显示为什么会对当前进程进行剖析，并列出当剖析被触发时超过阈值的度量值，以便你快速了解原因。 任务详情：与前几篇文章介绍的 CPU 剖析和网络剖析类似，它会显示当前任务的火焰图或进程拓扑图，具体取决于剖析类型。 同时，在 Metrics 选项卡中，收集与剖析策略相关的指标以检索历史趋势，以便在剖析的触发点提供全面的解释。\n图 7: 持续剖析指标\n结论 在本文中，我详细介绍了 SkyWalking 和 eBPF 中持续剖析功能的工作原理。通常情况下，它涉及将 eBPF Agent 服务部署在要监视的进程所在的同一台计算机上，并以低资源消耗监测目标进程。当它符合阈值条件时，它会启动更复杂的 CPU 剖析和网络剖析任务。\n在未来，我们将提供更多功能。敬请期待！\nTwitter：ASFSkyWalking Slack：向邮件列表 (dev@skywalking.apache.org) 发送“Request to join SkyWalking Slack”，我们会邀请你加入。 订阅我们的 Medium 列表。 ","excerpt":"\u003ch1 id=\"背景\"\u003e背景\u003c/h1\u003e\n\u003cp\u003e在之前的文章中，我们讨论了如何使用 SkyWalking 和 eBPF 来检测性能问题，包括\u003ca href=\"/blog/2022-07-05-pinpoint-service-mesh-critical-performance-impact-by-using-ebpf\"\u003e进程\u003c/a\u003e和\u003ca href=\"/zh/diagnose-service-mesh-network-performance-with-ebpf\"\u003e网络\u003c/a\u003e。这些方法可以很好地定位问题，但仍然存在一些挑战：\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003e任务启动的时间\u003c/strong\u003e: 当需要进行性能监控时 …\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/zh/2023-06-25-intruducing-continuous-profiling-skywalking-with-ebpf/","title":"自动化性能分析——持续剖析"},{"body":"SkyWalking 9.5.0 is released. Go to downloads page to find release tars.\nNew Topology Layout Elasticsearch Server Monitoring Project Fix Duplicate class found due to the delombok goal. OAP Server Fix wrong layer of metric user error in DynamoDB monitoring. ElasticSearch storage does not check field types when OAP running in no-init mode. Support to bind TLS status as a part of component for service topology. Fix component ID priority bug. Fix component ID of topology overlap due to storage layer bugs. [Breaking Change] Enhance JDBC storage through merging tables and managing day-based table rolling. [Breaking Change] Sharding-MySQL implementations and tests get removed due to we have the day-based rolling mechanism by default Fix otel k8s-cluster rule add namespace dimension for MAL aggregation calculation(Deployment Status,Deployment Spec Replicas) Support continuous profiling feature. Support collect process level related metrics. Fix K8sRetag reads the wrong k8s service from the cache due to a possible namespace mismatch. [Breaking Change] Support cross-thread trace profiling. The data structure and query APIs are changed. Fix PromQL HTTP API /api/v1/labels response missing service label. Fix possible NPE when initialize IntList. Support parse PromQL expression has empty labels in the braces for metadata query. Support alarm metric OP !=. Support metrics query indicates whether value == 0 represents actually zero or no data. Fix NPE when query the not exist series indexes in ElasticSearch storage. Support collecting memory buff/cache metrics in VM monitoring. PromQL: Remove empty values from the query result, fix /api/v1/metadata param limit could cause out of bound. Support monitoring the total number metrics of k8s StatefulSet and DaemonSet. Support Amazon API Gateway monitoring. Bump up graphql-java to fix cve. Bump up Kubernetes Java client. Support Redis Monitoring. Add component ID for amqp, amqp-producer and amqp-consumer. Support no-proxy mode for aws-firehose receiver Bump up armeria to 1.23.1 Support Elasticsearch Monitoring. Fix PromQL HTTP API /api/v1/series response missing service label when matching metric. Support ServerSide TopN for BanyanDB. Add component ID for Jersey. Remove OpenCensus support, the related codes and docs as it\u0026rsquo;s sunsetting. Support dynamic configuration of searchableTracesTags Support exportErrorStatusTraceOnly for export the error status trace segments through the Kafka channel Add component ID for Grizzly. Fix potential NPE in Zipkin receiver when the Span is missing some fields. Filter out unknown_cluster metric data. Support RabbitMQ Monitoring. Support Redis slow logs collection. Fix data loss when query continuous profiling task record. Adapt the continuous profiling task query GraphQL. Support Metrics Query Expression(MQE) and allows users to do simple query-stage calculation through the expression. Deprecated metrics query v2 protocol. Deprecated record query protocol. Add component ID for go-redis. Add OpenSearch 2.8.0 to test case. Add ai-pipeline module. Support HTTP URI formatting through ai-pipeline to do pattern recognition. Add new HTTP URI grouping engine with benchmark. [Breaking Change] Use the new HTTP URI grouping engine to replace the old regex based mechanism. Support sumLabeled in MAL. Migrate from kubernetes-client/java to fabric8 client. Envoy ALS generated relation metrics considers http status codes \u0026gt;= 400 has an error at the client side. Add cause message field when query continuous profiling task. UI Revert: cpm5d function. This feature is cancelled from backend. Fix: alerting link breaks on the topology. Refactor Topology widget to make it more hierarchical. Choose User as the first node. If User node is absent, choose the busiest node(which has the most calls of all). Do a left-to-right flow process. At the same level, list nodes from top to bottom in alphabetical order. Fix filter ID when ReadRecords metric associates with trace. Add AWS API Gateway menu. Change trace profiling protocol. Add Redis menu. Optimize data types. Support isEmptyValue flag for metrics query. Add elasticsearch menu. [Clean UI templates before upgrade] Set showSymbol: true, and make the data point shows on the Line graph. Please clean ui_template index in elasticsearch storage or table in JDBC storage. [Clean UI templates before upgrade] UI templates: Simplify metric name with the label. Add MQ menu. Add Jeysey icon. Fix: set endpoint and instance selectors with url parameters correctly. Bump up dependencies versions icons-vue 1.1.4, element-plus 2.1.0, nanoid 3.3.6, postcss 8.4.23 Add OpenTelemetry log protocol support. [Breaking Change] Configuration key enabledOtelRules is renamed to enabledOtelMetricsRules and the corresponding environment variable is renamed to SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES. Add grizzly icon. Fix: the Instance List data display error. Fix: set topN type to Number. Support Metrics Query Expression(MQE) and allows users to do simple query-stage calculation through the expression. Bump up zipkin ui dependency to 2.24.1. Bump up vite to 4.0.5. Apply MQE on General and Virtual-Database layer UI-templates. Documentation Add Profiling related documentations. Add SUM_PER_MIN to MAL documentation. Make the log relative docs more clear, and easier for further more formats support. Update the cluster management and advanced deployment docs. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking 9.5.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"new-topology-layout\"\u003eNew Topology Layout …\u003c/h4\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-9.5.0/","title":"Release Apache SkyWalking APM 9.5.0"},{"body":"Celebrating 22k Stars! The Apache SkyWalking community is thrilled to reach the milestone of 22k stars on GitHub! This showcases its popularity and impact as an APM and observability tool.\nSince launching in 2016 to provide an open source APM solution, SkyWalking has evolved into a full stack observability platform with distributed tracing, metrics monitoring and alerting. It\u0026rsquo;s seeing widespread adoption globally, especially in Asia where APM needs are expanding rapidly.\nThe growing user base has enabled SkyWalking to achieve massive deployments demonstrating its ability to scale to extreme levels. There have been reported deployments collecting over 100TB of data from companies\u0026rsquo; complex distributed applications, monitoring over 8000 microservices and analyzing 100 billion distributed traces - providing end-to-end visibility, performance monitoring and issue troubleshooting for some of the largest distributed systems in the world.\nThis success and widespread adoption has attracted an active community of nearly 800 contributors, thanks in part to programs like GSoC and OSPP(Open Source Promotion Plan) that bring in university contributors. The SkyWalking team remains focused on building a reliable, performant platform to observe complex distributed systems. We\u0026rsquo;ll continue innovating with features like service mesh monitoring and metric analytics.Your ongoing support, feedback and contributions inspire us!\nThank you for helping SkyWalking reach 22k stars on GitHub! This is just the beginning - we have ambitious plans and can\u0026rsquo;t wait to have you along our journey!\n","excerpt":"\u003ch1 id=\"celebrating-22k-stars\"\u003eCelebrating 22k Stars!\u003c/h1\u003e\n\u003cp\u003e\u003cimg src=\"./skywalking-22k.png\" alt=\"Stars\"\u003e\u003c/p\u003e\n\u003cp\u003eThe Apache SkyWalking community is thrilled to reach the milestone of 22k …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2023-06-13-celebrate-22k-stars/","title":"Celebrate 22k stars"},{"body":"本文演示如何将 Dubbo-Go 应用程序与 SkyWalking Go 集成，并在 SkyWalking UI 中查看结果。\n以前，如果你想要在 SkyWalking 中监控 Golang 应用程序，需要将项目与 go2sky 项目集成，并手动编写各种带有 go2sky 插件的框架。现在，我们有一个全新的项目（ Skywalking Go ），允许你将 Golang 项目集成到 SkyWalking 中，几乎不需要编码，同时提供更大的灵活性和可扩展性。\n在本文中，我们将指导你快速将 skywalking-go 项目集成到 dubbo-go 项目中。\n演示包括以下步骤：\n部署 SkyWalking：这涉及设置 SkyWalking 后端和 UI 程序，使你能够看到最终效果。 使用 SkyWalking Go 编译程序：在这里，你将把 SkyWalking Go Agent 编译到要监控的 Golang 程序中。 应用部署：你将导出环境变量并部署应用程序，以促进你的服务与 SkyWalking 后端之间的通信。 在 SkyWalking UI 上可视化：最后，你将发送请求并在 SkyWalking UI 中观察效果。 部署 SkyWalking 请从官方 SkyWalking 网站下载 SkyWalking APM 程序 。然后执行以下两个命令来启动服务:\n# 启动 OAP 后端 \u0026gt; bin/oapService.sh # 启动 UI \u0026gt; bin/webappService.sh 接下来，你可以访问地址 http://localhost:8080/ 。此时，由于尚未部署任何应用程序，因此你将看不到任何数据。\n使用 SkyWalking GO 编译 Dubbo Go 程序 这里将演示如何将 Dubbo-go 程序与SkyWalking Go Agent集成。请依次执行如下命令来创建一个新的项目:\n# 安装dubbo-go基础环境 \u0026gt; export GOPROXY=\u0026#34;https://goproxy.cn\u0026#34; \u0026gt; go install github.com/dubbogo/dubbogo-cli@latest \u0026gt; dubbogo-cli install all # 创建demo项目 \u0026gt; mkdir demo \u0026amp;\u0026amp; cd demo \u0026gt; dubbogo-cli newDemo . # 升级dubbo-go依赖到最新版本 \u0026gt; go get -u dubbo.apache.org/dubbo-go/v3 在项目的根目录中执行以下命令。此命令将下载 skywalking-go 所需的依赖项：\ngo get github.com/apache/skywalking-go 接下来，请分别在服务端和客户端的main包中引入。包含之后，代码将会更新为：\n// go-server/cmd/server.go package main import ( \u0026#34;context\u0026#34; ) import ( \u0026#34;dubbo.apache.org/dubbo-go/v3/common/logger\u0026#34; \u0026#34;dubbo.apache.org/dubbo-go/v3/config\u0026#34; _ \u0026#34;dubbo.apache.org/dubbo-go/v3/imports\u0026#34; \u0026#34;helloworld/api\u0026#34; // 引入skywalking-go _ \u0026#34;github.com/apache/skywalking-go\u0026#34; ) type GreeterProvider struct { api.UnimplementedGreeterServer } func (s *GreeterProvider) SayHello(ctx context.Context, in *api.HelloRequest) (*api.User, error) { logger.Infof(\u0026#34;Dubbo3 GreeterProvider get user name = %s\\n\u0026#34;, in.Name) return \u0026amp;api.User{Name: \u0026#34;Hello \u0026#34; + in.Name, Id: \u0026#34;12345\u0026#34;, Age: 21}, nil } // export DUBBO_GO_CONFIG_PATH= PATH_TO_SAMPLES/helloworld/go-server/conf/dubbogo.yaml func main() { config.SetProviderService(\u0026amp;GreeterProvider{}) if err := config.Load(); err != nil { panic(err) } select {} } 在客户端代码中除了需要引入skywalking-go之外，还需要在main方法中的最后一行增加主携程等待语句，以防止因为客户端快速关闭而无法将Tracing数据异步发送到SkyWalking后端：\npackage main import ( \u0026#34;context\u0026#34; ) import ( \u0026#34;dubbo.apache.org/dubbo-go/v3/common/logger\u0026#34; \u0026#34;dubbo.apache.org/dubbo-go/v3/config\u0026#34; _ \u0026#34;dubbo.apache.org/dubbo-go/v3/imports\u0026#34; \u0026#34;helloworld/api\u0026#34; // 引入skywalking-go _ \u0026#34;github.com/apache/skywalking-go\u0026#34; ) var grpcGreeterImpl = new(api.GreeterClientImpl) // export DUBBO_GO_CONFIG_PATH= PATH_TO_SAMPLES/helloworld/go-client/conf/dubbogo.yaml func main() { config.SetConsumerService(grpcGreeterImpl) if err := config.Load(); err != nil { panic(err) } logger.Info(\u0026#34;start to test dubbo\u0026#34;) req := \u0026amp;api.HelloRequest{ Name: \u0026#34;laurence\u0026#34;, } reply, err := grpcGreeterImpl.SayHello(context.Background(), req) if err != nil { logger.Error(err) } logger.Infof(\u0026#34;client response result: %v\\n\u0026#34;, reply) // 增加主携程等待语句 select {} } 接下来，请从官方 SkyWalking 网站下载 Go Agent 程序 。当你使用 go build 命令进行编译时，请在 bin 目录中找到与当前操作系统匹配的代理程序，并添加 -toolexec=\u0026quot;/path/to/go-agent -a 参数。例如，请使用以下命令：\n# 进入项目主目录 \u0026gt; cd demo # 分别编译服务端和客户端 # -toolexec 参数定义为go-agent的路径 # -a 参数用于强制重新编译所有依赖项 \u0026gt; cd go-server \u0026amp;\u0026amp; go build -toolexec=\u0026#34;/path/to/go-agent\u0026#34; -a -o go-server cmd/server.go \u0026amp;\u0026amp; cd .. \u0026gt; cd go-client \u0026amp;\u0026amp; go build -toolexec=\u0026#34;/path/to/go-agent\u0026#34; -a -o go-client cmd/client.go \u0026amp;\u0026amp; cd .. 应用部署 在开始部署应用程序之前，你可以通过环境变量更改 SkyWalking 中当前应用程序的服务名称。你还可以更改其配置，例如服务器端的地址。有关详细信息，请参阅文档 。\n在这里，我们分别启动两个终端窗口来分别启动服务端和客户端。\n在服务端，将服务的名称更改为dubbo-server：\n# 导出dubbo-go服务端配置文件路径 export DUBBO_GO_CONFIG_PATH=/path/to/demo/go-server/conf/dubbogo.yaml # 导出skywalking-go的服务名称 export SW_AGENT_NAME=dubbo-server ./go-server/go-server 在客户端，将服务的名称更改为dubbo-client：\n# 导出dubbo-go客户端配置文件路径 export DUBBO_GO_CONFIG_PATH=/path/to/demo/go-client/conf/dubbogo.yaml # 导出skywalking-go的服务名称 export SW_AGENT_NAME=dubbo-client ./go-client/go-client 在 SkyWalking UI 上可视化 现在，由于客户端会自动像服务器端发送请求，现在就可以在 SkyWalking UI 中观察结果。\n几秒钟后，重新访问 http://localhost:8080 的 SkyWalking UI。能够在主页上看到部署的 dubbo-server 和 dubbo-client 服务。\n此外，在追踪页面上，可以看到刚刚发送的请求。\n并可以在拓扑图页面中看到服务之间的关系。\n总结 在本文中，我们指导你快速开发dubbo-go服务，并将其与 SkyWalking Go Agent 集成。这个过程也适用于你自己的任意 Golang 服务。最终，可以在 SkyWalking 服务中查看显示效果。如果你有兴趣了解 SkyWalking Go 代理当前支持的框架，请参阅此文档 。\n将来，我们将继续扩展 SkyWalking Go 的功能，添加更多插件支持。所以，请继续关注！\n","excerpt":"\u003cp\u003e本文演示如何将 \u003ca href=\"https://github.com/apache/dubbo-go\"\u003eDubbo-Go\u003c/a\u003e 应用程序与 SkyWalking Go 集成，并在 SkyWalking UI 中查看结果。\u003c/p\u003e\n\u003cp\u003e以前，如果你想要在 SkyWalking 中监控 Golang 应用程序 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2023-06-05-quick-start-using-skywalking-go-monitoring-dubbo-go/","title":"使用SkyWalking go agent快速实现Dubbo Go监控"},{"body":"SkyWalking Go 0.1.0 is released. Go to downloads page to find release tars.\nFeatures Initialize the agent core and user import library. Support gRPC reporter for management, tracing protocols. Automatic detect the log frameworks and inject the log context. Plugins Support Gin framework. Support Native HTTP server and client framework. Support Go Restful v3 framework. Support Dubbo server and client framework. Support Kratos v2 server and client framework. Support Go-Micro v4 server and client framework. Support GORM v2 database client framework. Support MySQL Driver detection. Documentation Initialize the documentation. Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Go 0.1.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eInitialize the …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-go-0.1.0/","title":"Release Apache SkyWalking Go 0.1.0"},{"body":"SkyWalking Java Agent 8.16.0 is released. Go to downloads page to find release tars. Changes by Version\n8.16.0 Exclude synthetic methods for the WitnessMethod mechanism Support ForkJoinPool trace Support clickhouse-jdbc-plugin trace sql parameters Support monitor jetty server work thread pool metric Support Jersey REST framework Fix ClassCastException when SQLServer inserts data [Chore] Exclude org.checkerframework:checker-qual and com.google.j2objc:j2objc-annotations [Chore] Exclude proto files in the generated jar Fix Jedis-2.x plugin can not get host info in jedis 3.3.x+ Change the classloader to locate the agent path in AgentPackagePath, from SystemClassLoader to AgentPackagePath\u0026rsquo;s loader. Support Grizzly Trace Fix possible IllegalStateException when using Micrometer. Support Grizzly Work ThreadPool Metric Monitor Fix the gson dependency in the kafka-reporter-plugin. Fix deserialization of kafka producer json config in the kafka-reporter-plugin. Support to config custom decode methods for kafka configurations All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking Java Agent 8.16.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nChanges by …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-java-agent-8-16-0/","title":"Release Apache SkyWalking Java Agent 8.16.0"},{"body":"Background Previously, if you wanted to monitor a Golang application in SkyWalking, you would integrate your project with the go2sky project and manually write various frameworks with go2sky plugins. Now, we have a brand-new project (Skywalking Go) that allows you to integrate your Golang projects into SkyWalking with almost zero coding, while offering greater flexibility and scalability.\nIn this article, we will guide you quickly integrating the skywalking-go project into your Golang project.\nQuick start This demonstration will consist of the following steps:\nDeploy SkyWalking: This involves setting up the SkyWalking backend and UI programs, enabling you to see the final effect. Compile Golang with SkyWalking Go: Here, you\u0026rsquo;ll compile the SkyWalking Go Agent into the Golang program you wish to monitor. Application Deployment: You\u0026rsquo;ll export environment variables and deploy the application to facilitate communication between your service and the SkyWalking backend. Visualization on SkyWalking UI: Finally, you\u0026rsquo;ll send requests and observe the effects within the SkyWalking UI. Deploy SkyWalking Please download the SkyWalking APM program from the official SkyWalking website. Then execute the following two commands to start the service:\n# startup the OAP backend \u0026gt; bin/oapService.sh # startup the UI \u0026gt; bin/webappService.sh Next, you can access the address at http://localhost:8080/. At this point, as no applications have been deployed yet, you will not see any data.\nCompile Golang with SkyWalking GO Here is a simple business application here that starts an HTTP service.\npackage main import \u0026#34;net/http\u0026#34; func main() { http.HandleFunc(\u0026#34;/hello\u0026#34;, func(writer http.ResponseWriter, request *http.Request) { writer.Write([]byte(\u0026#34;Hello World\u0026#34;)) }) err := http.ListenAndServe(\u0026#34;:8000\u0026#34;, nil) if err != nil { panic(err) } } Execute the following command in the project\u0026rsquo;s root directory. This command will download the dependencies required for skywalking-go:\ngo get github.com/apache/skywalking-go Also, include it in the main package of the project. After the inclusion, the code will update to:\npackage main import ( \u0026#34;net/http\u0026#34; // This is an important step. DON\u0026#39;T MISS IT. _ \u0026#34;github.com/apache/skywalking-go\u0026#34; ) func main() { http.HandleFunc(\u0026#34;/hello\u0026#34;, func(writer http.ResponseWriter, request *http.Request) { writer.Write([]byte(\u0026#34;Hello World\u0026#34;)) }) err := http.ListenAndServe(\u0026#34;:8000\u0026#34;, nil) if err != nil { panic(err) } } Next, please download the Go Agent program from the official SkyWalking website. When you compile with the go build command, find the agent program that matches your current operating system in the bin directory, and add the -toolexec=\u0026quot;/path/to/go-agent -a parameter. For example, use the following command:\n# Build application with SkyWalking go agent # -toolexec parameter define the path of go-agent # -a parameter is used to force rebuild all packages \u0026gt; go build -toolexec=\u0026#34;/path/to/go-agent\u0026#34; -a -o test . Application Deployment Before you start to deploy the application, you can change the service name of the current application in SkyWalking through environment variables. You can also change its configuration such as the address with the server-side. For specific details, please refer to the documentation.\nHere, we\u0026rsquo;re just changing the name of the current service to demo.\n# Change the service name \u0026gt; export SW_AGENT_NAME=demo Next, you can start the application:\n# Start the application \u0026gt; ./test Visualization on SkyWalking UI Now, you can send a request to the application and observe the results in the SkyWalking UI.\n# Send a request \u0026gt; curl http://localhost:8000/hello After a few seconds, you can revisit the SkyWalking UI at http://localhost:8080. You will be able to see the demo service you deployed on the homepage.\nMoreover, on the Trace page, you can see the request you just sent.\nConclusion In this article, we\u0026rsquo;ve guided you to quickly develop a demo service and integrate it with SkyWalking Go Agent. This process is also applicable to your own Golang services. Ultimately, you can view the display effect in the SkyWalking service. If you\u0026rsquo;re interested in learning which frameworks the SkyWalking Go agent currently supports, please refer to this documentation.\nIn the future, we will continue to expand the functionality of SkyWalking Go, adding more plugin support. So, stay tuned!\n","excerpt":"\u003ch1 id=\"background\"\u003eBackground\u003c/h1\u003e\n\u003cp\u003ePreviously, if you wanted to monitor a Golang application in SkyWalking, you would …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2023-06-01-quick-start-with-skywalking-go-agent/","title":"Quick start with SkyWalking Go Agent"},{"body":"本文演示如何将应用程序与 SkyWalking Go 集成，并在 SkyWalking UI 中查看结果。\n以前，如果你想要在 SkyWalking 中监控 Golang 应用程序，需要将项目与 go2sky 项目集成，并手动编写各种带有 go2sky 插件的框架。现在，我们有一个全新的项目（Skywalking Go ），允许你将 Golang 项目集成到 SkyWalking 中，几乎不需要编码，同时提供更大的灵活性和可扩展性。\n在本文中，我们将指导你快速将 skywalking-go 项目集成到 Golang 项目中。\n演示包括以下步骤：\n部署 SkyWalking：这涉及设置 SkyWalking 后端和 UI 程序，使你能够看到最终效果。 使用 SkyWalking Go 编译 Golang：在这里，你将把 SkyWalking Go Agent 编译到要监控的 Golang 程序中。 应用部署：你将导出环境变量并部署应用程序，以促进你的服务与 SkyWalking 后端之间的通信。 在 SkyWalking UI 上可视化：最后，你将发送请求并在 SkyWalking UI 中观察效果。 部署 SkyWalking 请从官方 SkyWalking 网站下载 SkyWalking APM 程序 。然后执行以下两个命令来启动服务:\n# 启动 OAP 后端 \u0026gt; bin/oapService.sh # 启动 UI \u0026gt; bin/webappService.sh 接下来，你可以访问地址 http://localhost:8080/ 。此时，由于尚未部署任何应用程序，因此你将看不到任何数据。\n使用 SkyWalking GO 编译 Golang 这里有一个简单的业务应用程序，启动了一个 HTTP 服务。\npackage main import \u0026#34;net/http\u0026#34; func main() { http.HandleFunc(\u0026#34;/hello\u0026#34;, func(writer http.ResponseWriter, request *http.Request) { writer.Write([]byte(\u0026#34;Hello World\u0026#34;)) }) err := http.ListenAndServe(\u0026#34;:8000\u0026#34;, nil) if err != nil { panic(err) } } 在项目的根目录中执行以下命令。此命令将下载 skywalking-go 所需的依赖项：\ngo get github.com/apache/skywalking-go 接下来，请将其包含在项目的 main 包中。包含之后，代码将会更新为：\npackage main import ( \u0026#34;net/http\u0026#34; _ \u0026#34;github.com/apache/skywalking-go\u0026#34; ) func main() { http.HandleFunc(\u0026#34;/hello\u0026#34;, func(writer http.ResponseWriter, request *http.Request) { writer.Write([]byte(\u0026#34;Hello World\u0026#34;)) }) err := http.ListenAndServe(\u0026#34;:8000\u0026#34;, nil) if err != nil { panic(err) } } 接下来，请从官方 SkyWalking 网站下载 Go Agent 程序 。当你使用 go build 命令进行编译时，请在 bin 目录中找到与当前操作系统匹配的代理程序，并添加 -toolexec=\u0026quot;/path/to/go-agent\u0026quot; -a 参数。例如，请使用以下命令：\ngo build -toolexec=\u0026#34;/path/to/go-agent\u0026#34; -a -o test . 应用部署 在开始部署应用程序之前，你可以通过环境变量更改 SkyWalking 中当前应用程序的服务名称。你还可以更改其配置，例如服务器端的地址。有关详细信息，请参阅文档 。\n在这里，我们只是将当前服务的名称更改为 demo。\n接下来，你可以启动应用程序：\nexport SW_AGENT_NAME=demo ./test 在 SkyWalking UI 上可视化 现在，向应用程序发送请求并在 SkyWalking UI 中观察结果。\n几秒钟后，重新访问 http://localhost:8080 的 SkyWalking UI。能够在主页上看到部署的 demo 服务。\n此外，在追踪页面上，可以看到刚刚发送的请求。\n总结 在本文中，我们指导你快速开发 demo 服务，并将其与 SkyWalking Go Agent 集成。这个过程也适用于你自己的 Golang 服务。最终，可以在 SkyWalking 服务中查看显示效果。如果你有兴趣了解 SkyWalking Go 代理当前支持的框架，请参阅此文档 。\n将来，我们将继续扩展 SkyWalking Go 的功能，添加更多插件支持。所以，请继续关注！\n","excerpt":"\u003cp\u003e本文演示如何将应用程序与 SkyWalking Go 集成，并在 SkyWalking UI 中查看结果。\u003c/p\u003e\n\u003cp\u003e以前，如果你想要在 SkyWalking 中监控 Golang 应用程序，需要将项目与 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2023-06-01-quick-start-with-skywalking-go-agent/","title":"SkyWalking Go Agent 快速开始指南"},{"body":"SkyWalking Rust 0.7.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed Obtain Span object without intermediary. by @jmjoy in https://github.com/apache/skywalking-rust/pull/57 Rename module skywalking_proto to proto. by @jmjoy in https://github.com/apache/skywalking-rust/pull/59 Add Span::prepare_for_async method and AbstractSpan trait. by @jmjoy in https://github.com/apache/skywalking-rust/pull/58 Bump to 0.7.0. by @jmjoy in https://github.com/apache/skywalking-rust/pull/60 ","excerpt":"\u003cp\u003eSkyWalking Rust 0.7.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-rust-0-7-0/","title":"Release Apache SkyWalking Rust 0.7.0"},{"body":"SkyWalking PHP 0.5.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed Bump openssl from 0.10.45 to 0.10.48 by @dependabot in https://github.com/apache/skywalking-php/pull/60 Make the SKYWALKING_AGENT_ENABLE work in the request hook as well. by @jmjoy in https://github.com/apache/skywalking-php/pull/61 Support tracing curl_multi_* api. by @jmjoy in https://github.com/apache/skywalking-php/pull/62 Fix parent endpoint and peer in segment ref and tag url in entry span. by @jmjoy in https://github.com/apache/skywalking-php/pull/63 Bump h2 from 0.3.15 to 0.3.17 by @dependabot in https://github.com/apache/skywalking-php/pull/65 Add amqplib plugin for producer. by @jmjoy in https://github.com/apache/skywalking-php/pull/64 Upgrade and adapt phper. by @jmjoy in https://github.com/apache/skywalking-php/pull/66 Refactor script create_package_xml. by @jmjoy in https://github.com/apache/skywalking-php/pull/67 Refactor predis plugin to hook Client. by @jmjoy in https://github.com/apache/skywalking-php/pull/68 Canonicalize unknown. by @jmjoy in https://github.com/apache/skywalking-php/pull/69 Bump guzzlehttp/psr7 from 2.4.0 to 2.5.0 in /tests/php by @dependabot in https://github.com/apache/skywalking-php/pull/70 Enhance support for Swoole. by @jmjoy in https://github.com/apache/skywalking-php/pull/71 Bump to 0.5.0. by @jmjoy in https://github.com/apache/skywalking-php/pull/72 Full Changelog: https://github.com/apache/skywalking-php/compare/v0.4.0...v0.5.0\nPECL https://pecl.php.net/package/skywalking_agent/0.5.0\n","excerpt":"\u003cp\u003eSkyWalking PHP 0.5.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-php-0-5-0/","title":"Release Apache SkyWalking PHP 0.5.0"},{"body":"SkyWalking Python 1.0.1 is released! Go to downloads page to find release tars.\nPyPI Wheel: https://pypi.org/project/apache-skywalking/1.0.1/\nDockerHub Image: https://hub.docker.com/r/apache/skywalking-python\nUpgrading from v1.0.0 to v1.0.1 is strongly encouraged\nThis is a critical performance-oriented patch to address a CPU surge reported in https://github.com/apache/skywalking/issues/10672 Feature:\nAdd a new workflow to push docker images for arm64 and amd64 (#297) Plugins:\nOptimize loguru reporter plugin.(#302) Fixes:\nFix sw8 loss when use aiohttp (#299, issue#10669) Critical: Fix a bug that leads to high cpu usage (#300, issue#10672) Others:\nUse Kraft mode in E2E Kafka reporter tests (#303) New Contributors @Forstwith made their first contribution in https://github.com/apache/skywalking-python/pull/299 @FAWC438 made their first contribution in https://github.com/apache/skywalking-python/pull/300 Full Changelog: https://github.com/apache/skywalking-python/compare/v1.0.0...v1.0.1\n","excerpt":"\u003cp\u003eSkyWalking Python 1.0.1 is released! Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePyPI Wheel\u003c/strong\u003e: …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-python-1-0-1/","title":"Release Apache SkyWalking Python 1.0.1"},{"body":"本次活动于 2023 年 4 月 22 日在北京奥加美术馆酒店举行。该会议旨在探讨和分享有关可观测性的最佳实践， 包括在云原生应用程序和基础架构中实现可观测性的最新技术和工具。与会者将有机会了解行业领袖的最新见解，并与同行们分享经验和知识。 我们期待这次会议能够给云原生社区带来更多的启发和动力，推动我们在可观测性方面的进一步发展。\n圆桌讨论：云原生应用可观测性现状及趋势 B站视频地址\n嘉宾\n罗广明，主持人 吴晟，Tetrate 创始工程师 向阳，云杉科技研发 VP 乔新亮，原苏宁科技副总裁，现彩食鲜 CTO 董江，中国移动云能力中心高级系统架构专家 为 Apache SkyWalking 构建 Grafana dashboards \u0026ndash; 基于对原生 PromQL 的支持 B站视频地址\n万凯，Tetrate\n讲师介绍 万凯，Tetrate 工程师，Apache SkyWalking PMC 成员，专注于应用性能可观测性领域。\n议题概要 本次分享将介绍 Apache SkyWalking 的新特性 PromQL Service，它将为 SkyWalking 带来更广泛的生态集成能力: 什么是 PromQL SkyWalking 的 PromQL Service 是什么，能够做什么 SkyWalking 中的基本概念和 metrics 的特性 如何使用 PromQL Service 使用 PromQL Service 构建 Grafana dashboards 的实践\n","excerpt":"\u003cp\u003e本次活动于 2023 年 4 月 22 日在北京奥加美术馆酒店举行。该会议旨在探讨和分享有关可观测性的最佳实践，\n包括在云原生应用程序和基础架构中实现可观测性的最新技术和工具。与会者将有机会了解行业领 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2023-04-23-obs-summit-china/","title":"[视频] 可观测性峰会2023 - Observability Summit"},{"body":"SkyWalking Client JS 0.10.0 is released. Go to downloads page to find release tars.\nFix the ability of Fetch constructure. Update README. Bump up dependencies. ","excerpt":"\u003cp\u003eSkyWalking Client JS 0.10.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eFix the ability …\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-client-js-0-10-0/","title":"Release Apache SkyWalking Client JS 0.10.0"},{"body":"SkyWalking Java Agent 8.15.0 is released. Go to downloads page to find release tars. Changes by Version\n8.15.0 Enhance lettuce plugin to adopt uniform tags. Expose complete Tracing APIs in the tracing toolkit. Add plugin to trace Spring 6 and Resttemplate 6. Move the baseline to JDK 17 for development, the runtime baseline is still Java 8 compatible. Remove Powermock entirely from the test cases. Fix H2 instrumentation point Refactor pipeline in jedis-plugin. Add plugin to support ClickHouse JDBC driver (0.3.2.*). Refactor kotlin coroutine plugin with CoroutineContext. Fix OracleURLParser ignoring actual port when :SID is absent. Change gRPC instrumentation point to fix plugin not working for server side. Fix servicecomb plugin trace break. Adapt Armeria\u0026rsquo;s plugins to the latest version 1.22.x Fix tomcat-10x-plugin and add test case to support tomcat7.x-8.x-9.x. Fix thrift plugin generate duplicate traceid when sendBase error occurs Support keep trace profiling when cross-thread. Fix unexpected whitespace of the command catalogs in several Redis plugins. Fix a thread leak in SamplingService when updated sampling policy in the runtime. Support MySQL plugin tracing SQL parameters when useServerPrepStmts Update the endpoint name of Undertow plugin to Method:Path. Build a dummy(empty) javadoc of finagle and jdk-http plugins due to incompatibility. Documentation Update docs of Tracing APIs, reorganize the API docs into six parts. Correct missing package name in native manual API docs. Add a FAQ doc about \u0026ldquo;How to make SkyWalking agent works in OSGI environment?\u0026rdquo; All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking Java Agent 8.15.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nChanges by …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-java-agent-8-15-0/","title":"Release Apache SkyWalking Java Agent 8.15.0"},{"body":"SkyWalking PHP 0.4.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed Bump tokio from 1.24.1 to 1.24.2 by @dependabot in https://github.com/apache/skywalking-php/pull/52 Bump to 0.4.0-dev by @heyanlong in https://github.com/apache/skywalking-php/pull/53 Avoid potential panic for logger. by @jmjoy in https://github.com/apache/skywalking-php/pull/54 Fix the curl plugin hook curl_setopt by mistake. by @jmjoy in https://github.com/apache/skywalking-php/pull/55 Update documents. by @jmjoy in https://github.com/apache/skywalking-php/pull/56 Upgrade dependencies and adapt the codes. by @jmjoy in https://github.com/apache/skywalking-php/pull/57 Add sub components licenses in dist material. by @jmjoy in https://github.com/apache/skywalking-php/pull/58 Bump to 0.4.0. by @jmjoy in https://github.com/apache/skywalking-php/pull/59 New Contributors @dependabot made their first contribution in https://github.com/apache/skywalking-php/pull/52 Full Changelog: https://github.com/apache/skywalking-php/compare/v0.3.0...v0.4.0\nPECL https://pecl.php.net/package/skywalking_agent/0.4.0\n","excerpt":"\u003cp\u003eSkyWalking PHP 0.4.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-php-0-4-0/","title":"Release Apache SkyWalking PHP 0.4.0"},{"body":"Background As an application performance monitoring tool for distributed systems, Apache SkyWalking provides monitoring, tracing, diagnosing capabilities for distributed system in Cloud Native architecture. Prometheus is an open-source systems monitoring and alerting toolkit with an active ecosystem. Especially Prometheus metrics receive widespread support through exporters and integrations. PromQL as Prometheus Querying Language containing a set of expressions and expose HTTP APIs to read metrics.\nSkyWalking supports to ingest Prometheus metrics through OpenTelemetry collector and through the aggregate calculation of these metrics to provide a variety of systems monitoring, such as Linux Monitoring and Kubernetes monitoring. SkyWalking already provides native UI and GraphQL API for users. But as designed to provide wider ecological integration capabilities, since 9.4.0, it provides PromQL Service, the third-party systems or visualization platforms that already support PromQL (such as Grafana), could obtain metrics through it. SkyWalking users will benefit from it when they integrate with different systems.\nWhat is PromQL Service in SkyWalking? PromQL Service is a query engine on the top of SkyWalking native GraphQL query, with additional query stage calculation capabilities powered by Prometheus expressions. It can accept PromQL HTTP API requests, parse Prometheus expressions, and transform between Prometheus metrics and SkyWalking metrics.\nThe PromQL Service follows all PromQL\u0026rsquo;s protocols and grammar and users can use it as they would with PromQL. As SkyWalking is fundamentally different from Prometheus in terms of metric classification, format, storage, etc. PromQL Service doesn\u0026rsquo;t have to implement the full PromQL feature. Refer to the documentation for the detail.\nSkyWalking Basic Concepts Here are some basic concepts and differences from Prometheus that users need to understand in order to use the PromQL service: Prometheus metrics specify the naming format and structure, the actual metric names and labels are determined by the client provider, and the details are stored. The user aggregates and calculates the metrics using the expression in PromQL. Unlike Prometheus, SkyWalking\u0026rsquo;s metric mechanism is built around the following core concepts with a hierarchical structure:\nLayer: represents an abstract framework in computer science, such as Operating System(OS_LINUX layer), Kubernetes(k8s layer). This layer would be the owner of different services detected from different technologies. All Layers definitions can be found here. Service: Represents a set/group of workloads which provides the same behaviors for incoming requests. Service Instance: An individual workload in the Service group. Endpoint: A path in a service for incoming requests. Process: An operating system process. In some scenarios, a service instance is not a process, such as a pod Kubernetes could contain multiple processes. The metric name and properties (labels) are configured by the SkyWalking OAP server based on the data source as well as OAL and MAL. SkyWalking provides the ability to down-sampling time series metrics, and generate different time bucket data (minute, hour, day).\nThe SkyWalking metric stream is as follows:\nTraffic The metadata of the Service/ServiceRelation/Instance/ServiceInstanceRelation/Endpoint/EndpointRelation/Process/ProcessRelation. Include names, layers, properties, relations between them, etc. Metric Name: metric name, configuration from OAL and MAL. Entity: represents the metrics\u0026rsquo; belonging and used for the query. An Entity will contain the following information depending on the Scope： Scope represents the metrics level and in query stage represents the Scope catalog, Scope catalog provides high-dimension classifications for all scopes as a hierarchy structure. Scope Entity Info Service Service(include layer info) ServiceInstance Service, ServiceInstance Endpoint Service, Endpoint ServiceRelation Service, DestService ServiceInstanceRelation ServiceInstance, DestServiceInstance EndpointRelation Endpoint, DestEndpoint Process Service, ServiceInstance, Process ProcessRelation Process, ServiceInstance, DestProcess Value: single value: long. labeled value: text, label1,value1|label2,value2|..., such as L2 aggregation,5000 | L1 aggregation,8000. TimeBucket: the time is accurate to minute, hour, day. How to use PromQL Service Setup PromQL Service is enabled by default after v9.4.0, so no additional configuration is required. The default ports, for example, can be configured by using OAP environment variables:\nrestHost: ${SW_PROMQL_REST_HOST:0.0.0.0} restPort: ${SW_PROMQL_REST_PORT:9090} restContextPath: ${SW_PROMQL_REST_CONTEXT_PATH:/} restMaxThreads: ${SW_PROMQL_REST_MAX_THREADS:200} restIdleTimeOut: ${SW_PROMQL_REST_IDLE_TIMEOUT:30000} restAcceptQueueSize: ${SW_PROMQL_REST_QUEUE_SIZE:0} Use Prometheus expression PromQL matches metric through the Prometheus expression. Here is a typical Prometheus metric.\nTo match the metric, the Prometheus expression is as follows:\nIn the PromQL Service, these reserved labels would be parsed as the metric name and entity info fields with other labels for the query. The mappings are as follows.\nSkyWalking Concepts Prometheus expression Metric name Metric name Layer Label Service Label ServiceInstance Label\u0026lt;service_instance\u0026gt; Endpoint Label \u0026hellip; \u0026hellip; For example, the following expressions are used to match query metrics: service_cpm, service_instance_cpm, endpoint_cpm\nservice_cpm{service=\u0026#39;agent::songs\u0026#39;, layer=\u0026#39;GENERAL\u0026#39;} service_instance_cpm{service=\u0026#39;agent::songs\u0026#39;, service_instance=\u0026#39;agent::songs_instance_1\u0026#39;, layer=\u0026#39;GENERAL\u0026#39;} endpoint_cpm{service=\u0026#39;agent::songs\u0026#39;, endpoint=\u0026#39;GET:/songs\u0026#39;, layer=\u0026#39;GENERAL\u0026#39;} Typical Query Example At here, we take the SkyWalking Showcase deployment as the playground to demonstrate how to use PromQL for SkyWalking metrics.\nThe following examples can be used to query the metadata and metrics of services through PromQL Service.\nGet metrics names Query:\nhttp://localhost:9099/api/v1/label/__name__/values Result:\n{ \u0026#34;status\u0026#34;: \u0026#34;success\u0026#34;, \u0026#34;data\u0026#34;: [ \u0026#34;meter_mysql_instance_qps\u0026#34;, \u0026#34;service_cpm\u0026#34;, \u0026#34;envoy_cluster_up_rq_active\u0026#34;, \u0026#34;instance_jvm_class_loaded_class_count\u0026#34;, \u0026#34;k8s_cluster_memory_requests\u0026#34;, \u0026#34;meter_vm_memory_used\u0026#34;, \u0026#34;meter_apisix_sv_bandwidth_unmatched\u0026#34;, \u0026#34;meter_vm_memory_total\u0026#34;, ... ] } Select a metric and get the labels Query:\nhttp://localhost:9099/api/v1/labels?match[]=service_cpm Result:\n{ \u0026#34;status\u0026#34;: \u0026#34;success\u0026#34;, \u0026#34;data\u0026#34;: [ \u0026#34;layer\u0026#34;, \u0026#34;service\u0026#34;, \u0026#34;top_n\u0026#34;, \u0026#34;order\u0026#34; ] } Get services from a specific layer Query:\nhttp://127.0.0.1:9099/api/v1/series?match[]=service_traffic{layer=\u0026#39;GENERAL\u0026#39;}\u0026amp;start=1677479336\u0026amp;end=1677479636 Result:\n{ \u0026#34;status\u0026#34;: \u0026#34;success\u0026#34;, \u0026#34;data\u0026#34;: [ { \u0026#34;__name__\u0026#34;: \u0026#34;service_traffic\u0026#34;, \u0026#34;service\u0026#34;: \u0026#34;agent::songs\u0026#34;, \u0026#34;scope\u0026#34;: \u0026#34;Service\u0026#34;, \u0026#34;layer\u0026#34;: \u0026#34;GENERAL\u0026#34; }, { \u0026#34;__name__\u0026#34;: \u0026#34;service_traffic\u0026#34;, \u0026#34;service\u0026#34;: \u0026#34;agent::recommendation\u0026#34;, \u0026#34;scope\u0026#34;: \u0026#34;Service\u0026#34;, \u0026#34;layer\u0026#34;: \u0026#34;GENERAL\u0026#34; }, { \u0026#34;__name__\u0026#34;: \u0026#34;service_traffic\u0026#34;, \u0026#34;service\u0026#34;: \u0026#34;agent::app\u0026#34;, \u0026#34;scope\u0026#34;: \u0026#34;Service\u0026#34;, \u0026#34;layer\u0026#34;: \u0026#34;GENERAL\u0026#34; }, { \u0026#34;__name__\u0026#34;: \u0026#34;service_traffic\u0026#34;, \u0026#34;service\u0026#34;: \u0026#34;agent::gateway\u0026#34;, \u0026#34;scope\u0026#34;: \u0026#34;Service\u0026#34;, \u0026#34;layer\u0026#34;: \u0026#34;GENERAL\u0026#34; }, { \u0026#34;__name__\u0026#34;: \u0026#34;service_traffic\u0026#34;, \u0026#34;service\u0026#34;: \u0026#34;agent::frontend\u0026#34;, \u0026#34;scope\u0026#34;: \u0026#34;Service\u0026#34;, \u0026#34;layer\u0026#34;: \u0026#34;GENERAL\u0026#34; } ] } Query specific metric for a service Query:\nhttp://127.0.0.1:9099/api/v1/query?query=service_cpm{service=\u0026#39;agent::songs\u0026#39;, layer=\u0026#39;GENERAL\u0026#39;} Result:\n{ \u0026#34;status\u0026#34;: \u0026#34;success\u0026#34;, \u0026#34;data\u0026#34;: { \u0026#34;resultType\u0026#34;: \u0026#34;vector\u0026#34;, \u0026#34;result\u0026#34;: [ { \u0026#34;metric\u0026#34;: { \u0026#34;__name__\u0026#34;: \u0026#34;service_cpm\u0026#34;, \u0026#34;layer\u0026#34;: \u0026#34;GENERAL\u0026#34;, \u0026#34;scope\u0026#34;: \u0026#34;Service\u0026#34;, \u0026#34;service\u0026#34;: \u0026#34;agent::songs\u0026#34; }, \u0026#34;value\u0026#34;: [ 1679559960, \u0026#34;6\u0026#34; ] } ] } } About the range query and different metrics type for query can refer to the document here.\nBuild Grafana Dashboard From the above, we know the mechanism and how to query from PromQL Service, now we can build the Grafana Dashboard for the above service example. Note: All the following configurations are based on Grafana version 9.1.0.\nSkyWalking Showcase provides dashboards files such as services of General and Service Mesh layers， we can quickly create a dashboard for the General layer service by importing the dashboard JSON file.\nAfter the Grafana application is deployed, follow the steps below:\nConfigure Data Source First, we need to create a data source: In the data source config panel, chose Prometheus and set the URL to the OAP server address, the default port is 9090. Here set the data source name SkyWalking in case there are multiple Prometheus data sources.\nImport Dashboard File Create a dashboard folder named SkyWalking.\nImport the dashboard file into Grafana, there are two ways to get the file:\nFrom SkyWalking Showcase. Go to SkyWaking Demo: Preview metrics on Grafana, and export it from the General Service dashboard. Done! Now we can see the dashboard is working, the services are in the drop-down list and the metrics are displayed on the panels.\nThis is an easy way to build, but we need to know how it works if we want to customize it.\nHow the dashboard works Dashboard Settings Open the Settings-Variables we can see the following variables:\nLet\u0026rsquo;s look at what each variable does:\n$DS_SkyWalking\nThis is a data source ty variable that specifies the Prometheus data source which was defined earlier as SkyWalking.\n$layer\nThis is a constant type because in the \u0026lsquo;General Service\u0026rsquo; dashboard, all services belong to the \u0026lsquo;GENERAL\u0026rsquo; layer, so they can be used directly in each query Note When you customize other layers, this value must be defined in the Layer mentioned above.\n$service\nQuery type variable, to get all service names under this layer for the drop-down list.\nQuery expression:\nlabel_values(service_traffic{layer=\u0026#39;$layer\u0026#39;}, service) The query expression will query HTTP API /api/v1/series for service metadata in $layer and fetch the service name according to the label(service).\n$service_instance\nSame as the $service is a query variable that is used to select all instances of the service in the drop-down list.\nQuery expression:\nlabel_values(instance_traffic{layer=\u0026#39;$layer\u0026#39;, service=\u0026#39;$service\u0026#39;}, service_instance) The query expression here not only specifies the $layer but also contains the variable $service, which is used to correlate with the services for the drop-down list.\n$endpoint\nSame as the $service is a query variable that is used to select all endpoints of the service in the drop-down list.\nQuery expression:\nlabel_values(endpoint_traffic{layer=\u0026#39;$layer\u0026#39;, service=\u0026#39;$service\u0026#39;, keyword=\u0026#39;$endpoint_keyword\u0026#39;, limit=\u0026#39;$endpoint_limit\u0026#39;}, endpoint) The query expression here specifies the $layer and $service which are used to correlate with the services for the drop-down list. And also accept variables $endpoint_keyword and $endpoint_limit as filtering condition.\n$endpoint_keyword\nA text type variable that the user can input to filter the return value of $endpoint.\n$endpoint_limit\nCustom type, which the user can select to limit the maximum number of returned endpoints.\nPanel Configurations There are several typical metrics panels on this dashboard, let\u0026rsquo;s see how it\u0026rsquo;s configured.\nCommon Value Metrics Select Time series chart panel Service Apdex and click edit. Query expression service_apdex{service=\u0026#39;$service\u0026#39;, layer=\u0026#39;$layer\u0026#39;} / 10000 The metric scope is Service, add labels service and layer for the match, and the label value used the variables configured above. The calculation Divided by 10000 is used for matching the result units. The document for the query can refer to here. Set Query options --\u0026gt; Min interval = 1m, because the metrics min time bucket in SkyWalking is 1m. Set Connect null values --\u0026gt; Always and Show points --\u0026gt; Always because when the query interval \u0026gt; 1 hour or 1 day SkyWalking returns the hour/day step metrics values. Labeled Value Metrics Select Time series chart panel Service Response Time Percentile and click edit. Query expression service_percentile{service=\u0026#39;$service\u0026#39;, layer=\u0026#39;$layer\u0026#39;, labels=\u0026#39;0,1,2,3,4\u0026#39;, relabels=\u0026#39;P50,P75,P90,P95,P99\u0026#39;} The metric scope is Service, add labels service and layer for the match, and the label value used the variables configured above. Add labels='0,1,2,3,4' filter the result label, and addrelabels='P50,P75,P90,P95,P99' rename the result label. The document for the query can refer to here. Set Query options --\u0026gt; Min interval = 1m, because the metrics min time bucket in SkyWalking is 1m. Set Connect null values --\u0026gt; Always and Show points --\u0026gt; Always because when the query interval \u0026gt; 1 hour or 1 day SkyWalking returns the hour/day step metrics values. Set Legend to {{label}} for show up. Sort Metrics Select Time series chart panel Service Response Time Percentile and click edit. Query expression service_instance_cpm{parent_service=\u0026#39;$service\u0026#39;, layer=\u0026#39;$layer\u0026#39;, top_n=\u0026#39;10\u0026#39;, order=\u0026#39;DES\u0026#39;} The expression is used for query the sore metrics under service, so add labels parent_service and layer for the match. Add top_n='10' and order='DES' filter the result. The document for the query can refer to here. Set Query options --\u0026gt; Min interval = 1m, because the metrics min time bucket in SkyWalking is 1m. Set the Calculation --\u0026gt; Latest*. Set Legend to {{service_instance}} for show up. Conclusion In this article, we introduced what is the PromQL Service in SkyWalking and its background. Detailed how to use PromQL Service and the basic concepts related to SkyWalking, and show how to use PromQL Service to build Grafana dashboards for SkyWalking.\nIn the future, there will be more integrations by leveraging this protocol, such as CI/CD, HPA (scaling), etc.\n","excerpt":"\u003ch2 id=\"background\"\u003eBackground\u003c/h2\u003e\n\u003cp\u003eAs an application performance monitoring tool for distributed systems, Apache SkyWalking …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2023-03-17-build-grafana-dashboards-for-apache-skywalking-native-promql-support/","title":"Build Grafana dashboards for Apache SkyWalking -- Native PromQL Support"},{"body":"背景 Apache SkyWalking 作为分布式系统的应用性能监控工具，提供了对云原生架构下的分布式系统的监控、跟踪、诊断能力。Prometheus 是一个开源系统监控和警报工具包，具有活跃的生态系统。特别是 Prometheus 指标通过 导出器和集成 得到广泛支持。 PromQL 作为 Prometheus 查询语言，包含一组表达式并公开 HTTP API 以读取指标。\nSkyWalking 支持通过 OpenTelemetry 收集器 摄取 Prometheus 指标，并通过这些指标的聚合计算提供多种系统监控，例如 Linux 监控和 Kubernetes 监控。SkyWalking 已经为用户提供了 原生 UI 和 GraphQL API。但为了提供更广泛的生态整合能力，从 9.4.0 开始，它提供了 PromQL 服务，已经支持 PromQL 的第三方系统或可视化平台（如 Grafana），可以通过它获取指标。SkyWalking 用户在与不同系统集成时将从中受益。\nSkyWalking 中的 PromQL 服务是什么？ PromQL 服务是 SkyWalking 原生 GraphQL 查询之上的查询引擎，具有由 Prometheus 表达式提供支持的附加查询阶段计算能力。它可以接受 PromQL HTTP API 请求，解析 Prometheus 表达式，并在 Prometheus 指标和 SkyWalking 指标之间进行转换。\nPromQL 服务遵循 PromQL 的所有协议和语法，用户可以像使用 PromQL 一样使用它。由于 SkyWalking 在度量分类、格式、存储等方面与 Prometheus 有根本不同，因此 PromQL 服务不必实现完整的 PromQL 功能。有关详细信息，请参阅文档。\nSkyWalking 基本概念 以下是用户使用 PromQL 服务需要了解的一些基本概念和与 Prometheus 的区别： Prometheus 指标指定命名格式和结构，实际指标名称和标签由客户端提供商确定，并存储详细信息。用户使用 PromQL 中的表达式聚合和计算指标。与 Prometheus 不同，SkyWalking 的度量机制是围绕以下具有层次结构的核心概念构建的：\n层（Layer）：表示计算机科学中的一个抽象框架，如 Operating System（OS_LINUX 层）、Kubernetes（k8s 层）。该层将是从不同技术检测到的不同服务的所有者。可以在此处\n找到所有层定义。\n服务：表示一组 / 一组工作负载，它为传入请求提供相同的行为。\n服务实例：服务组中的单个工作负载。\n端点：传入请求的服务路径。\n进程：操作系统进程。在某些场景下，service instance 不是一个进程，比如一个 Kubernetes Pod 可能包含多个进程。\nMetric 名称和属性（标签）由 SkyWalking OAP 服务器根据数据源以及 OAL 和 MAL 配置。SkyWalking 提供了对时间序列指标进行下采样（down-sampling），并生成不同时间段数据（分钟、小时、天）的能力。\nSkyWalking 指标流如下：\n流量 Service/ServiceRelation/Instance/ServiceInstanceRelation/Endpoint/EndpointRelation/Process/ProcessRelation 的元数据。包括名称、层、属性、它们之间的关系等。 指标 名称（Name）：指标名称，来自 OAL 和 MAL 的配置。 实体（Entity）：表示指标的归属，用于查询。一个 Entity 根据 Scope 不同会包含如下信息： Scope 代表指标级别，在查询阶段代表 Scope catalog，Scope catalog 为所有的 scope 提供了高维的分类，层次结构。 Scope 实体信息 Service 服务（包括图层信息） ServiceInstance 服务、服务实例 Endpoint 服务、端点 ServiceRelation 服务，目标服务 ServiceInstanceRelation 服务实例、目标服务实例 EndpointRelation 端点、目标端点 Process 服务、服务实例、流程 ProcessRelation 进程、服务实例、DestProcess 值： 单值：long 标签值：文本，label1,value1|label2,value2|... ，例如 L2 aggregation,5000 | L1 aggregation,8000 TimeBucket：时间精确到分钟、小时、天 如何使用 PromQL 服务 设置 PromQL 服务在 v9.4.0 之后默认开启，不需要额外配置。例如，可以使用 OAP 环境变量配置默认端口：\nrestHost: ${SW_PROMQL_REST_HOST:0.0.0.0} restPort: ${SW_PROMQL_REST_PORT:9090} restContextPath: ${SW_PROMQL_REST_CONTEXT_PATH:/} restMaxThreads: ${SW_PROMQL_REST_MAX_THREADS:200} restIdleTimeOut: ${SW_PROMQL_REST_IDLE_TIMEOUT:30000} restAcceptQueueSize: ${SW_PROMQL_REST_QUEUE_SIZE:0} 使用 Prometheus 表达式 PromQL 通过 Prometheus 表达式匹配指标。这是一个典型的 Prometheus 指标。\n为了匹配指标，Prometheus 表达式如下：\n在 PromQL 服务中，这些保留的标签将被解析为度量名称和实体信息字段以及用于查询的其他标签。映射如下。\nSkyWalking 概念 Prometheus 表达 指标名称 指标名称 层 标签 服务 标签 服务实例 标签 \u0026lt;服务实例\u0026gt; 端点 标签 …… …… 例如，以下表达式用于匹配查询指标：service_cpm、service_instance_cpm、endpoint_cpm\nservice_cpm {service=\u0026#39;agent::songs\u0026#39;, layer=\u0026#39;GENERAL\u0026#39;} service_instance_cpm {service=\u0026#39;agent::songs\u0026#39;, service_instance=\u0026#39;agent::songs_instance_1\u0026#39;, layer=\u0026#39;GENERAL\u0026#39;} endpoint_cpm {service=\u0026#39;agent::songs\u0026#39;, endpoint=\u0026#39;GET:/songs\u0026#39;, layer=\u0026#39;GENERAL\u0026#39;} 典型查询示例 在这里，我们将 SkyWalking Showcase 部署作为 Playground 来演示如何使用 PromQL 获取 SkyWalking 指标。\n以下示例可用于通过 PromQL 服务查询服务的元数据和指标。\n获取指标名称 查询：\nhttp://localhost:9099/api/v1/label/__name__/values 结果：\n{ \u0026#34;status\u0026#34;: \u0026#34;success\u0026#34;, \u0026#34;data\u0026#34;: [ \u0026#34;meter_mysql_instance_qps\u0026#34;, \u0026#34;service_cpm\u0026#34;, \u0026#34;envoy_cluster_up_rq_active\u0026#34;, \u0026#34;instance_jvm_class_loaded_class_count\u0026#34;, \u0026#34;k8s_cluster_memory_requests\u0026#34;, \u0026#34;meter_vm_memory_used\u0026#34;, \u0026#34;meter_apisix_sv_bandwidth_unmatched\u0026#34;, \u0026#34;meter_vm_memory_total\u0026#34;, ... ] } 选择一个指标并获取标签 查询：\nhttp://localhost:9099/api/v1/labels?match []=service_cpm 结果：\n{ \u0026#34;status\u0026#34;: \u0026#34;success\u0026#34;, \u0026#34;data\u0026#34;: [ \u0026#34;layer\u0026#34;, \u0026#34;service\u0026#34;, \u0026#34;top_n\u0026#34;, \u0026#34;order\u0026#34; ] } 从特定层获取服务 查询：\nhttp://127.0.0.1:9099/api/v1/series?match []=service_traffic {layer=\u0026#39;GENERAL\u0026#39;}\u0026amp;start=1677479336\u0026amp;end=1677479636 结果：\n{ \u0026#34;status\u0026#34;: \u0026#34;success\u0026#34;, \u0026#34;data\u0026#34;: [ {\u0026#34;__name__\u0026#34;: \u0026#34;service_traffic\u0026#34;, \u0026#34;service\u0026#34;: \u0026#34;agent::songs\u0026#34;, \u0026#34;scope\u0026#34;: \u0026#34;Service\u0026#34;, \u0026#34;layer\u0026#34;: \u0026#34;GENERAL\u0026#34; }, {\u0026#34;__name__\u0026#34;: \u0026#34;service_traffic\u0026#34;, \u0026#34;service\u0026#34;: \u0026#34;agent::recommendation\u0026#34;, \u0026#34;scope\u0026#34;: \u0026#34;Service\u0026#34;, \u0026#34;layer\u0026#34;: \u0026#34;GENERAL\u0026#34; }, {\u0026#34;__name__\u0026#34;: \u0026#34;service_traffic\u0026#34;, \u0026#34;service\u0026#34;: \u0026#34;agent::app\u0026#34;, \u0026#34;scope\u0026#34;: \u0026#34;Service\u0026#34;, \u0026#34;layer\u0026#34;: \u0026#34;GENERAL\u0026#34; }, {\u0026#34;__name__\u0026#34;: \u0026#34;service_traffic\u0026#34;, \u0026#34;service\u0026#34;: \u0026#34;agent::gateway\u0026#34;, \u0026#34;scope\u0026#34;: \u0026#34;Service\u0026#34;, \u0026#34;layer\u0026#34;: \u0026#34;GENERAL\u0026#34; }, {\u0026#34;__name__\u0026#34;: \u0026#34;service_traffic\u0026#34;, \u0026#34;service\u0026#34;: \u0026#34;agent::frontend\u0026#34;, \u0026#34;scope\u0026#34;: \u0026#34;Service\u0026#34;, \u0026#34;layer\u0026#34;: \u0026#34;GENERAL\u0026#34; } ] } 查询服务的特定指标 查询：\nhttp://127.0.0.1:9099/api/v1/query?query=service_cpm {service=\u0026#39;agent::songs\u0026#39;, layer=\u0026#39;GENERAL\u0026#39;} 结果：\n{ \u0026#34;status\u0026#34;: \u0026#34;success\u0026#34;, \u0026#34;data\u0026#34;: { \u0026#34;resultType\u0026#34;: \u0026#34;vector\u0026#34;, \u0026#34;result\u0026#34;: [ {\u0026#34;metric\u0026#34;: { \u0026#34;__name__\u0026#34;: \u0026#34;service_cpm\u0026#34;, \u0026#34;layer\u0026#34;: \u0026#34;GENERAL\u0026#34;, \u0026#34;scope\u0026#34;: \u0026#34;Service\u0026#34;, \u0026#34;service\u0026#34;: \u0026#34;agent::songs\u0026#34; },\u0026#34;value\u0026#34;: [ 1679559960, \u0026#34;6\u0026#34; ] } ] } } 关于range query和不同的metrics type for query 可以参考 这里的 文档。\n构建 Grafana Dashboard 从上面我们知道了 PromQL 服务的机制和查询方式，现在我们可以为上面的服务示例构建 Grafana Dashboard。注：以下所有配置均基于 Grafana 9.1.0 版本。\nSkyWalking Showcase 提供了 General Service 和 Service Mesh 层等 Dashboard 文件，我们可以通过导入 Dashboard JSON 文件快速为层服务创建 Dashboard。\n部署 Grafana 应用程序后，请按照以下步骤操作：\n配置数据源 首先，我们需要创建一个数据源： 在数据源配置面板中，选择 Prometheus 并设置 URL 为 OAP 服务器地址，默认端口为 9090。 SkyWalking 如果有多个 Prometheus 数据源，请在此处设置数据源名称。\n导入 Dashboard 文件 创建一个名为 SkyWalking 的 Dashboard 文件夹。\n将 Dashboard 文件导入到 Grafana 中，有两种获取文件的方式：\n来自 SkyWalking Showcase 转到 SkyWaking Demo：在 Grafana 上预览指标，并将其从 General Service Dashboard 导出。 完毕！现在我们可以看到 Dashboard 正在运行，服务位于下拉列表中，指标显示在面板上。\n这是一种简单的构建方式，但是如果我们想要自定义它，我们需要知道它是如何工作的。\nDashboard 的工作原理 Dashboard 设置 打开 Settings-Variables 我们可以看到如下变量：\n让我们看看每个变量的作用：\n$DS_SkyWalking\n这是一个数据源 ty 变量，它指定了之前定义为 SkyWalking 的 Prometheus 数据源。\n$layer\n这是一个常量类型，因为在 \u0026lsquo;General Service\u0026rsquo; Dashboard 中，所有服务都属于 \u0026lsquo;GENERAL\u0026rsquo; 层，因此可以在每个查询中直接使用它们。注意，当您自定义其他层时，必须在 Layer 上面定义该值。\n$service\n查询类型变量，为下拉列表获取该层下的所有服务名称。\n查询表达式：\nlabel_values (service_traffic {layer=\u0026#39;$layer\u0026#39;}, service) 查询表达式将查询 HTTP API /api/v1/series，以获取 $layer 中服务元数据，并根据标签（服务）提取服务名称。\n$service_instance\n与 $service 一样，是一个查询变量，用于在下拉列表中选择服务的所有实例。\n查询表达式：\nlabel_values (instance_traffic {layer=\u0026#39;$layer\u0026#39;, service=\u0026#39;$service\u0026#39;}, service_instance) 这里的查询表达式不仅指定了 $layer 还包含 $service 变量，用于关联下拉列表的服务。\n$endpoint\n与 $service 一样，是一个查询变量，用于在下拉列表中选择服务的所有端点。\n查询表达式：\nlabel_values (endpoint_traffic {layer=\u0026#39;$layer\u0026#39;, service=\u0026#39;$service\u0026#39;, keyword=\u0026#39;$endpoint_keyword\u0026#39;, limit=\u0026#39;$endpoint_limit\u0026#39;}, endpoint) 此处的查询表达式指定 $layer 和 $service 用于与下拉列表的服务相关联的。并且还接受 $endpoint_keyword 和 $endpoint_limit 变量作为过滤条件。\n$endpoint_keyword\n一个文本类型的变量，用户可以输入它来过滤 $endpoint 的返回值。\n$endpoint_limit\n自定义类型，用户可以选择它以限制返回端点的最大数量。\nDashboard 配置 这个 Dashboard 上有几个典型的指标面板，让我们看看它是如何配置的。\n普通值指标 选择 Time series chart 面板 Service Apdex 并单击 edit。\n查询表达式\nservice_apdex {service=\u0026#39;$service\u0026#39;, layer=\u0026#39;$layer\u0026#39;} / 10000 指标范围为 Service，添加 service 和 layer 标签用于匹配，label 值使用上面配置的变量。该计算 Divided by 10000 用于匹配结果单位。查询文档可以参考 这里。\n设置 Query options --\u0026gt; Min interval = 1m，因为 SkyWalking 中的指标最小时间段是 1m。\n设置 Connect null values --\u0026gt; AlwaysShow points --\u0026gt; Always，因为当查询间隔大于 1 小时或 1 天时，SkyWalking 返回小时 / 天步长指标值。\n标签值指标 选择 Time series chart 面板 Service Response Time Percentile 并单击 edit。\n查询表达式\nservice_percentile {service=\u0026#39;$service\u0026#39;, layer=\u0026#39;$layer\u0026#39;, labels=\u0026#39;0,1,2,3,4\u0026#39;, relabels=\u0026#39;P50,P75,P90,P95,P99\u0026#39;} 指标范围为 Service，添加 service 和 layer 标签用于匹配，label 值使用上面配置的变量。添加 labels='0,1,2,3,4' 过滤结果标签，并添加 relabels='P50,P75,P90,P95,P99' 重命名结果标签。查询文档可以参考 这里。\n设置 Query options --\u0026gt; Min interval = 1m，因为 SkyWalking 中的指标最小时间段是 1m。\n设置 Connect null values --\u0026gt; AlwaysShow points --\u0026gt; Always，因为当查询间隔 \u0026gt; 1 小时或 1 天时，SkyWalking 返回小时 / 天步长指标值。\n设置 Legend 为 {{label}} 来展示。\n排序指标 选择 Time series chart 面板 Service Response Time Percentile 并单击 edit。\n查询表达式\nservice_instance_cpm {parent_service=\u0026#39;$service\u0026#39;, layer=\u0026#39;$layer\u0026#39;, top_n=\u0026#39;10\u0026#39;, order=\u0026#39;DES\u0026#39;} 该表达式用于查询服务下的排序指标，因此添加标签 parent_service 和 layer 进行匹配。添加 top_n='10' 和 order='DES' 过滤结果。查询文档可以参考 这里。\n设置 Query options --\u0026gt; Min interval = 1m，因为 SkyWalking 中的指标最小时间段是 1m。\n设置 Calculation --\u0026gt; Latest*。\n设置 Legend 为 {{service_instance}} 来展示。\n结论 在这篇文章中，我们介绍了 SkyWalking 中的 PromQL 服务是什么以及它的背景。详细介绍了 PromQL 服务的使用方法和 SkyWalking 相关的基本概念，展示了如何使用 PromQL 服务为 SkyWalking 构建 Grafana Dashboard。\n未来，将会有更多的集成利用这个协议，比如 CI/CD、HPA（缩放）等。\n","excerpt":"\u003ch2 id=\"背景\"\u003e背景\u003c/h2\u003e\n\u003cp\u003eApache SkyWalking 作为分布式系统的应用性能监控工具，提供了对云原生架构下的分布式系统的监控、跟踪、诊断能力。\u003ca href=\"https://prometheus.io/docs/introduction/overview/#what-is-prometheus\"\u003ePrometheus\u003c/a\u003e 是一个开源系统监控和警报工具包，具有活跃的生态 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2023-03-17-build-grafana-dashboards-for-apache-skywalking-native-promql-support/","title":"为 Apache SkyWalking 构建 Grafana Dashboard —— 原生 PromQL 支持"},{"body":"\nBackground Apache SkyWalking is an open-source application performance management system that helps users collect and aggregate logs, traces, metrics, and events, and display them on the UI. Starting from OAP 9.4.0, SkyWalking has added AWS Firehose receiver, which is used to receive and calculate the data of CloudWatch metrics. In this article, we will take DynamoDB as an example to show how to use SkyWalking to receive and calculate CloudWatch metrics data for monitoring Amazon Web Services.\nWhat are Amazon CloudWatch and Amazon Kinesis Data Firehose？ Amazon CloudWatch is a metrics repository, this tool can collect raw data from AWS (e.g. DynamoDB) and process it into readable metrics in near real-time. Also, we can use Metric Stream to continuously stream CloudWatch metrics to a selected target location for near real-time delivery and low latency. SkyWalking takes advantage of this feature to create metric streams and direct them to Amazon Kinesis Data Firehose transport streams for further transport processing.\nAmazon Kinesis Data Firehoseis an extract, transform, and load (ETL) service that reliably captures, transforms, and delivers streaming data to data lakes, data stores, and analytics services. SkyWalking takes advantage of this feature to eventually direct the metrics stream to the aws-firehose-receiver for OAP to calculate and ultimately display the metrics.\nThe flow chart is as follows.\nNotice Due to Kinesis Data Firehose specifications, the URL of the HTTP endpoint must use the HTTPS protocol and must use port 443. Also, this URL must be proxied by Gateway and forwarded to the real aws-firehose-receiver. The TLS certificate must be signed by a CA and the self-signed certificate will not be trusted by Kinesis Data Firehose. Setting up DynamoDB monitoring Next, let\u0026rsquo;s take DynamoDB as an example to illustrate the necessary settings in aws before using OAP to collect CloudWatch metrics:\nGo to Kinesis Console, create a data stream, and select Direct PUT for Source and HTTP Endpoint for Destination. And set HTTP Endpoint URL to Gateway URL. The rest of the configuration options can be configured as needed. Go to the CloudWatch Console, select Metrics-Stream in the left control panel, and click Create metric stream. Select AWS/DynamoDB for namespace. Also, you can add other namespaces as needed. Kinesis Data Firehose selects the data stream created in the first step. Finally, set the output format to opentelemetry0.7. The rest of the configuration options can be configured as needed. At this point, the AWS side of DynamoDB monitoring configuration is set up.\nSkyWalking OAP metrics processing analysis SkyWalking uses aws-firehose-receiver to receive and decode AWS metrics streams forwarded by Gateway, and send it to Opentelemetry-receiver for processing and transforming into SkyWalking metrics. Then, the metrics are analyzed and aggregated by Meter Analysis Language (MAL) and finally presented on the UI.\nThe MAL part and the UI part of SkyWalking support users\u0026rsquo; customization, to display the metrics data in a more diversified way. For details, please refer to MAL doc and UI doc.\nTypical metrics analysis Scope In SkyWalking, there is the concept of scope. By using scopes, we can classify and aggregate metrics more rationally. In the monitoring of DynamoDB, two of these scopes are used - Service and Endpoint.\nService represents a set of workloads that provide the same behavior for incoming requests. Commonly used as cluster-level scopes for services, user accounts are closer to the concept of clusters in AWS. So SkyWalking uses AWS account id as a key to map AWS accounts to Service types.\nSimilarly, Endpoint represents a logical concept, often used in services for the path of incoming requests, such as HTTP URI path or gRPC service class + method signature, and can also represent the table structure in the database. So SkyWalking maps DynamoDB tables to Endpoint type.\nMetrics Metric Name Meaning AccountMaxReads / AccountMaxWrites The maximum number of read/write capacity units that can be used by an account. AccountMaxTableLevelReads / AccountMaxTableLevelWrites The maximum number of read/write capacity units that can be used by a table or global secondary index of an account. AccountProvisionedReadCapacityUtilization / AccountProvisionedWriteCapacityUtilization The percentage of provisioned read/write capacity units utilized by an account. MaxProvisionedTableReadCapacityUtilization / MaxProvisionedTableWriteCapacityUtilization The percentage of provisioned read/write capacity utilized by the highest provisioned read table or global secondary index of an account. Above are some common account metrics (Serivce scope). They are various configuration information in DynamoDB, and SkyWalking can show a complete picture of the database configuration changes by monitoring these metrics.\nMetric Name Meaning ConsumedReadCapacityUnits / ConsumedWriteCapacityUnits The number of read/write capacity units consumed over the specified time period. ReturnedItemCount The number of items returned by Query, Scan or ExecuteStatement (select) operations during the specified time period. SuccessfulRequestLatency The latency of successful requests to DynamoDB or Amazon DynamoDB Streams during the specified time period. TimeToLiveDeletedItemCount The number of items deleted by Time to Live (TTL) during the specified time period. The above are some common table metrics (Endpoint scope), which will also be aggregated into account metrics. These metrics are generally used to analyze the performance of the database, and users can use them to determine the reasonable level of database configuration. For example, users can track how much of their provisioned throughput is used through ConsumedReadCapicityUnits / ConsumedReadCapicityUnits to determine the reasonableness of the preconfigured throughput of a table or account. For more information about provisioned throughput, see Provisioned Throughput Intro.\nMetric Name Meaning UserErrors Requests to DynamoDB or Amazon DynamoDB Streams that generate an HTTP 400 status code during the specified time period. SystemErrors The requests to DynamoDB or Amazon DynamoDB Streams that generate an HTTP 500 status code during the specified time period. ThrottledRequests Requests to DynamoDB that exceed the provisioned throughput limits on a resource. TransactionConflict Rejected item-level requests due to transactional conflicts between concurrent requests on the same items. The above are some common error metrics, among which UserErrors are account-level metrics and the rest are table-level metrics. Users can set alarms on these metrics, and if warnings appear, then it may indicate that there are some problems with the use of the database, and users need to check and verify by themselves.\nNotice SkyWalking\u0026rsquo;s metrics selection for DynamoDB comes directly from CloudWatch metrics, which can also be found at CloudWatch metrics doc to get metrics details.\nDemo In this section, we will demonstrate how to use terraform to create a DynamoDB table and other AWS services that can generate metrics streams, and deploy Skywalking to complete the metrics collection.\nFirst, you need a running gateway instance, such as NGINX, which is responsible for receiving metrics streams from AWS and forwarding them to the aws-firehose-receiver. Note that the gateway needs to be configured with certificates to accept HTTPS protocol requests.\nBelow is an example configuration for NGINX. The configuration does not need to be identical, as long as it can send incoming HTTPS requests to oap host:12801/aws/firehose/metrics.\nserver { listen 443 ssl; ssl_certificate /crt/test.pem; ssl_certificate_key /crt/test.key; ssl_session_timeout 5m; ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_prefer_server_ciphers on; location /aws/firehose/metrics { proxy_pass http://test.xyz:12801/aws/firehose/metrics; } } Deploying SkyWalking There are various ways to deploy SkyWalking, and you can get them directly from the release page.\nOf course, if you are more comfortable with Kubernetes, you can also find the appropriate deployment method from SkyWalking-kubernetes.\nPlease note that no matter which deployment method you use, please make sure that the OAP and UI version is 9.4.0 or higher and that port 12801 needs to be open.\nThe following is an example of a deployment using the helm command.\nexport SKYWALKING_RELEASE_VERSION=4.3.0 export SKYWALKING_RELEASE_NAME=skywalking export SKYWALKING_RELEASE_NAMESPACE=default helm install \u0026#34;${SKYWALKING_RELEASE_NAME}\u0026#34; \\ oci://registry-1.docker.io/apache/skywalking-helm \\ --version \u0026#34;${SKYWALKING_RELEASE_VERSION}\u0026#34; \\ -n \u0026#34;${SKYWALKING_RELEASE_NAMESPACE}\u0026#34; \\ --set oap.image.tag=9.4.0 \\ --set oap.storageType=elasticsearch \\ --set ui.image.tag=9.4.0 \\ --set oap.ports.firehose=12801 Start the corresponding AWS service The terraform configuration file is as follows (example modified inTerraform Registry - kinesis_firehose_delivery_stream）：\nterraform configuration file provider \u0026#34;aws\u0026#34; { region = \u0026#34;ap-northeast-1\u0026#34; access_key = \u0026#34;[need change]your access_key\u0026#34; secret_key = \u0026#34;[need change]your secret_key\u0026#34; } resource \u0026#34;aws_dynamodb_table\u0026#34; \u0026#34;basic-dynamodb-table\u0026#34; { name = \u0026#34;GameScores\u0026#34; billing_mode = \u0026#34;PROVISIONED\u0026#34; read_capacity = 20 write_capacity = 20 hash_key = \u0026#34;UserId\u0026#34; range_key = \u0026#34;GameTitle\u0026#34; attribute { name = \u0026#34;UserId\u0026#34; type = \u0026#34;S\u0026#34; } attribute { name = \u0026#34;GameTitle\u0026#34; type = \u0026#34;S\u0026#34; } attribute { name = \u0026#34;TopScore\u0026#34; type = \u0026#34;N\u0026#34; } ttl { attribute_name = \u0026#34;TimeToExist\u0026#34; enabled = true } global_secondary_index { name = \u0026#34;GameTitleIndex\u0026#34; hash_key = \u0026#34;GameTitle\u0026#34; range_key = \u0026#34;TopScore\u0026#34; write_capacity = 10 read_capacity = 10 projection_type = \u0026#34;INCLUDE\u0026#34; non_key_attributes = [\u0026#34;UserId\u0026#34;] } tags = { Name = \u0026#34;dynamodb-table-1\u0026#34; Environment = \u0026#34;production\u0026#34; } } resource \u0026#34;aws_cloudwatch_metric_stream\u0026#34; \u0026#34;main\u0026#34; { name = \u0026#34;my-metric-stream\u0026#34; role_arn = aws_iam_role.metric_stream_to_firehose.arn firehose_arn = aws_kinesis_firehose_delivery_stream.http_stream.arn output_format = \u0026#34;opentelemetry0.7\u0026#34; include_filter { namespace = \u0026#34;AWS/DynamoDB\u0026#34; } } # https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-trustpolicy.html data \u0026#34;aws_iam_policy_document\u0026#34; \u0026#34;streams_assume_role\u0026#34; { statement { effect = \u0026#34;Allow\u0026#34; principals { type = \u0026#34;Service\u0026#34; identifiers = [\u0026#34;streams.metrics.cloudwatch.amazonaws.com\u0026#34;] } actions = [\u0026#34;sts:AssumeRole\u0026#34;] } } resource \u0026#34;aws_iam_role\u0026#34; \u0026#34;metric_stream_to_firehose\u0026#34; { name = \u0026#34;metric_stream_to_firehose_role\u0026#34; assume_role_policy = data.aws_iam_policy_document.streams_assume_role.json } # https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-trustpolicy.html data \u0026#34;aws_iam_policy_document\u0026#34; \u0026#34;metric_stream_to_firehose\u0026#34; { statement { effect = \u0026#34;Allow\u0026#34; actions = [ \u0026#34;firehose:PutRecord\u0026#34;, \u0026#34;firehose:PutRecordBatch\u0026#34;, ] resources = [aws_kinesis_firehose_delivery_stream.http_stream.arn] } } resource \u0026#34;aws_iam_role_policy\u0026#34; \u0026#34;metric_stream_to_firehose\u0026#34; { name = \u0026#34;default\u0026#34; role = aws_iam_role.metric_stream_to_firehose.id policy = data.aws_iam_policy_document.metric_stream_to_firehose.json } resource \u0026#34;aws_s3_bucket\u0026#34; \u0026#34;bucket\u0026#34; { bucket = \u0026#34;metric-stream-test-bucket\u0026#34; } resource \u0026#34;aws_s3_bucket_acl\u0026#34; \u0026#34;bucket_acl\u0026#34; { bucket = aws_s3_bucket.bucket.id acl = \u0026#34;private\u0026#34; } data \u0026#34;aws_iam_policy_document\u0026#34; \u0026#34;firehose_assume_role\u0026#34; { statement { effect = \u0026#34;Allow\u0026#34; principals { type = \u0026#34;Service\u0026#34; identifiers = [\u0026#34;firehose.amazonaws.com\u0026#34;] } actions = [\u0026#34;sts:AssumeRole\u0026#34;] } } resource \u0026#34;aws_iam_role\u0026#34; \u0026#34;firehose_to_s3\u0026#34; { assume_role_policy = data.aws_iam_policy_document.firehose_assume_role.json } data \u0026#34;aws_iam_policy_document\u0026#34; \u0026#34;firehose_to_s3\u0026#34; { statement { effect = \u0026#34;Allow\u0026#34; actions = [ \u0026#34;s3:AbortMultipartUpload\u0026#34;, \u0026#34;s3:GetBucketLocation\u0026#34;, \u0026#34;s3:GetObject\u0026#34;, \u0026#34;s3:ListBucket\u0026#34;, \u0026#34;s3:ListBucketMultipartUploads\u0026#34;, \u0026#34;s3:PutObject\u0026#34;, ] resources = [ aws_s3_bucket.bucket.arn, \u0026#34;${aws_s3_bucket.bucket.arn}/*\u0026#34;, ] } } resource \u0026#34;aws_iam_role_policy\u0026#34; \u0026#34;firehose_to_s3\u0026#34; { name = \u0026#34;default\u0026#34; role = aws_iam_role.firehose_to_s3.id policy = data.aws_iam_policy_document.firehose_to_s3.json } resource \u0026#34;aws_kinesis_firehose_delivery_stream\u0026#34; \u0026#34;http_stream\u0026#34; { name = \u0026#34;metric-stream-test-stream\u0026#34; destination = \u0026#34;http_endpoint\u0026#34; http_endpoint_configuration { name = \u0026#34;test_http_endpoint\u0026#34; url = \u0026#34;[need change]Gateway url\u0026#34; role_arn = aws_iam_role.firehose_to_s3.arn } s3_configuration { role_arn = aws_iam_role.firehose_to_s3.arn bucket_arn = aws_s3_bucket.bucket.arn } } Steps to use.\nGet the access_key and secret_key of the AWS account.( For how to get them, please refer to create-access-key )\nFill in the access_key and secret_key you got in the previous step, and fill in the corresponding URL of your gateway in the corresponding location of aws_kinesis_firehose_delivery_stream configuration.\nCopy the above content and save it to the main.tf file.\nExecute the following code in the corresponding path.\nterraform init terraform apply At this point, all the required AWS services have been successfully created, and you can check your console to see if the services were successfully created.\nDone! If all the above steps were successful, please wait for about five minutes. After that, you can visit the SkyWalking UI to see the metrics.\nCurrently, the metrics collected by SkyWalking by default are displayed as follows.\naccount metrics:\ntable metrics：\nOther services Currently, SkyWalking officially supports EKS, S3, DynamoDB monitoring. Users also refer to the OpenTelemetry receiver to configure OTel rules to collect and analyze CloudWatch metrics of other AWS services and display them through a custom dashboard.\nMaterial Monitoring S3 metrics with Amazon CloudWatch Monitoring DynamoDB metrics with Amazon CloudWatch Supported metrics in AWS Firehose receiver of OAP Configuration Vocabulary | Apache SkyWalking ","excerpt":"\u003cp\u003e\u003cimg src=\"./icon.png\" alt=\"icon.png\"\u003e\u003c/p\u003e\n\u003ch2 id=\"background\"\u003eBackground\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"https://skywalking.apache.org/\"\u003eApache SkyWalking\u003c/a\u003e is an open-source application performance management system that helps …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2023-03-13-skywalking-aws-dynamodb/","title":"Monitoring DynamoDB with SkyWalking"},{"body":"\n背景 Apache SkyWalking 是一个开源应用性能管理系统，帮助用户收集和聚合日志、追踪、指标和事件，并在 UI 上显示。从 OAP 9.4.0 开始，SkyWalking 新增了 AWS Firehose receiver，用来接收，计算CloudWatch metrics的数据。本文将以DynamoDB为例，展示如何使用 SkyWalking接收并计算 CloudWatch metrics 数据，以监控Amazon Web Services。\n什么是 Amazon CloudWatch 与 Amazon Kinesis Data Firehose ？ Amazon CloudWatch 是一个指标存储库, 此工具可从 AWS中 ( 如 DynamoDB ) 收集原始数据，近实时处理为可读取的指标。同时，我们也可以使用指标流持续地将 CloudWatch 指标流式传输到所选的目标位置，实现近实时传送和低延迟。SkyWalking 利用此特性，创建指标流并将其导向 Amazon Kinesis Data Firehose 传输流，并由后者进一步传输处理。\nAmazon Kinesis Data Firehose是一项提取、转换、加载服务，可以将流式处理数据以可靠方式捕获、转换和提供到数据湖、数据存储和分析服务中。SkyWalking利用此特性，将指标流最终导向 aws-firehose-receiver，交由OAP计算并最终展示指标。\n整体过程流程图如下：\n注意 由于 Kinesis Data Firehose 规定，HTTP端点的URL必须使用HTTPS协议，且必须使用443端口。同时，此URL必须由Gateway代理并转发到真正的aws-firehose-receiver。 TLS 证书必须由CA签发的，自签证书不会被 Kinesis Data Firehose 信任。 设置DynamoDB监控 接下来以DynamoDB为例说明使用OAP 收集CloudWatch metrics 前，aws中必要的设置:\n进入 Kinesis 控制台，创建数据流， Source选择 Direct PUT, Destination 选择 HTTP Endpoint. 并且设置HTTP Endpoint URL 为 Gateway对应URL。 其余配置选项可由需要自行配置。 进入 CloudWatch 控制台，在左侧控制面板中选择Metrics-Stream，点击Create metric stream。其中，namespace 选择 AWS/DynamoDB。同时，根据需要，也可以增加其他命名空间。 Kinesis Data Firehose选择在第一步中创建好的数据流。最后，设置输出格式为opentelemetry0.7。其余配置选项可由需要自行配置。 至此，DynamoDB监控配置的AWS方面设置完成。\nSkyWalking OAP 指标处理分析 SkyWalking 利用 aws-firehose-receiver 接收并解码由Gateway转发来的 AWS 指标流，交由Opentelemetry-receiver进行处理，转化为SkyWalking metrics。并由Meter Analysis Language (MAL)进行指标的分析与聚合，最终呈现在UI上。\n其中 MAL 部分以及 UI 部分，SkyWalking支持用户自由定制，从而更多样性的展示指标数据。详情请参考MAL doc 以及 UI doc。\n典型指标分析 作用域 SkyWalking中，有作用域 ( scope ) 的概念。通过作用域, 我们可以对指标进行更合理的分类与聚合。在对DynamoDB的监控中，使用到了其中两种作用域———Service和Endpoint。\nService表示一组工作负荷，这些工作负荷为传入请求提供相同的行为。常用作服务的集群级别作用域，在AWS中，用户的账户更接近集群的概念。 所以SkyWalking将AWS account id作为key，将AWS账户映射为Service类型。\n同理，Endpoint表示一种逻辑概念，常用于服务中用于传入请求的路径，例如 HTTP URI 路径或 gRPC 服务类 + 方法签名，也可以表示数据库中的表结构。所以SkyWalking将DynamoDB表映射为Endpoint类型。\n指标 指标名称 含义 AccountMaxReads / AccountMaxWrites 账户可以使用的最大 读取/写入 容量单位数。 AccountMaxTableLevelReads / AccountMaxTableLevelWrites 账户的表或全局二级索引可以使用的最大 读取/写入 容量单位数。 AccountProvisionedReadCapacityUtilization / AccountProvisionedWriteCapacityUtilization 账户使用的预置 读取/写入 容量单位百分比。 MaxProvisionedTableReadCapacityUtilization / MaxProvisionedTableWriteCapacityUtilization 账户的最高预调配 读取/写入 表或全局二级索引使用的预调配读取容量单位百分比。 以上为一些常用的账户指标(Serivce 作用域)。它们是DynamoDB中的各种配置信息，SkyWalking通过对这些指标的监控，可以完整的展示出数据库配置的变动情况。\n指标名称 含义 ConsumedReadCapacityUnits / ConsumedWriteCapacityUnits 指定时间段内占用的 读取/写入 容量单位数 ReturnedItemCount Query、Scan 或 ExecuteStatement（可选择）操作在指定时段内返回的项目数。 SuccessfulRequestLatency 指定时间段内对于 DynamoDB 或 Amazon DynamoDB Streams 的成功请求的延迟。 TimeToLiveDeletedItemCount 指定时间段内按存活时间 (TTL) 删除的项目数。 以上为一些常用的表指标(Endpoint作用域)，它们也会被聚合到账户指标中。这些指标一般用于分析数据库的性能，用户可以通过它们判断出数据库配置的合理程度。例如，用户可以通过ConsumedReadCapicityUnits / ConsumedReadCapicityUnits，跟踪预置吞吐量的使用，从而判断表或账户的预制吞吐量的合理性。关于预置吞吐量，请参见读/写容量模式。\n指标名称 含义 UserErrors 在指定时间段内生成 HTTP 400 状态代码的对 DynamoDB 或 Amazon DynamoDB Streams 的请求。HTTP 400 通常表示客户端错误，如参数组合无效，尝试更新不存在的表或请求签名错误。 SystemErrors 在指定的时间段内生成 HTTP 500 状态代码的对 DynamoDB 或 Amazon DynamoDB Streams 的请求。HTTP 500 通常指示内部服务错误。 ThrottledRequests 超出资源（如表或索引）预置吞吐量限制的 DynamoDB 请求。 TransactionConflict 由于同一项目的并发请求之间的事务性冲突而被拒绝的项目级请求。 以上为一些常用的错误指标，其中UserErrors为用户级别指标，其余为表级别指标。用户可以在这些指标上设置告警，如果警告出现，那么可能说明数据库的使用出现了一些问题，需要用户自行查看验证。\n注意 SkyWalking对于DynamoDB的指标选取直接来源于CloudWatch metrics， 您也可以通过CloudWatch metrics doc来获取指标详细信息。\nDemo 在本节中，我们将演示如何利用terraform创建一个DynamoDB表，以及可以产生指标流的其他AWS服务，并部署Skywalking完成指标收集。\n首先，您需要一个正在运行的网关实例，例如 NGINX，它负责接收AWS传来的指标流并且转发到aws-firehose-receiver。注意, 网关需要配置证书以便接受HTTPS协议的请求。\n下面是一个NGINX的示例配置。配置不要求完全一致，只要能将收到的HTTPS请求发送到oap所在host:12801/aws/firehose/metrics即可。\nserver { listen 443 ssl; ssl_certificate /crt/test.pem; ssl_certificate_key /crt/test.key; ssl_session_timeout 5m; ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_prefer_server_ciphers on; location /aws/firehose/metrics { proxy_pass http://test.xyz:12801/aws/firehose/metrics; } } 部署SkyWalking SkyWalking的部署方式有很多种，您可以直接从release页面中直接获取。\n当然，如果您更习惯于 Kubernetes，您也可以从SkyWalking-kubernetes找到相应部署方式。\n请注意，无论使用哪种部署方式，请确保OAP和UI的版本为9.4.0以上，并且需要开放12801端口。\n下面是一个使用helm指令部署的示例：\nexport SKYWALKING_RELEASE_VERSION=4.3.0 export SKYWALKING_RELEASE_NAME=skywalking export SKYWALKING_RELEASE_NAMESPACE=default helm install \u0026#34;${SKYWALKING_RELEASE_NAME}\u0026#34; \\ oci://registry-1.docker.io/apache/skywalking-helm \\ --version \u0026#34;${SKYWALKING_RELEASE_VERSION}\u0026#34; \\ -n \u0026#34;${SKYWALKING_RELEASE_NAMESPACE}\u0026#34; \\ --set oap.image.tag=9.4.0 \\ --set oap.storageType=elasticsearch \\ --set ui.image.tag=9.4.0 \\ --set oap.ports.firehose=12801 开启对应AWS服务 terraform 配置文件如下（实例修改于Terraform Registry - kinesis_firehose_delivery_stream）：\nterraform 配置文件 provider \u0026#34;aws\u0026#34; { region = \u0026#34;ap-northeast-1\u0026#34; access_key = \u0026#34;在这里填入您的access_key\u0026#34; secret_key = \u0026#34;在这里填入您的secret_key\u0026#34; } resource \u0026#34;aws_dynamodb_table\u0026#34; \u0026#34;basic-dynamodb-table\u0026#34; { name = \u0026#34;GameScores\u0026#34; billing_mode = \u0026#34;PROVISIONED\u0026#34; read_capacity = 20 write_capacity = 20 hash_key = \u0026#34;UserId\u0026#34; range_key = \u0026#34;GameTitle\u0026#34; attribute { name = \u0026#34;UserId\u0026#34; type = \u0026#34;S\u0026#34; } attribute { name = \u0026#34;GameTitle\u0026#34; type = \u0026#34;S\u0026#34; } attribute { name = \u0026#34;TopScore\u0026#34; type = \u0026#34;N\u0026#34; } ttl { attribute_name = \u0026#34;TimeToExist\u0026#34; enabled = true } global_secondary_index { name = \u0026#34;GameTitleIndex\u0026#34; hash_key = \u0026#34;GameTitle\u0026#34; range_key = \u0026#34;TopScore\u0026#34; write_capacity = 10 read_capacity = 10 projection_type = \u0026#34;INCLUDE\u0026#34; non_key_attributes = [\u0026#34;UserId\u0026#34;] } tags = { Name = \u0026#34;dynamodb-table-1\u0026#34; Environment = \u0026#34;production\u0026#34; } } resource \u0026#34;aws_cloudwatch_metric_stream\u0026#34; \u0026#34;main\u0026#34; { name = \u0026#34;my-metric-stream\u0026#34; role_arn = aws_iam_role.metric_stream_to_firehose.arn firehose_arn = aws_kinesis_firehose_delivery_stream.http_stream.arn output_format = \u0026#34;opentelemetry0.7\u0026#34; include_filter { namespace = \u0026#34;AWS/DynamoDB\u0026#34; } } # https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-trustpolicy.html data \u0026#34;aws_iam_policy_document\u0026#34; \u0026#34;streams_assume_role\u0026#34; { statement { effect = \u0026#34;Allow\u0026#34; principals { type = \u0026#34;Service\u0026#34; identifiers = [\u0026#34;streams.metrics.cloudwatch.amazonaws.com\u0026#34;] } actions = [\u0026#34;sts:AssumeRole\u0026#34;] } } resource \u0026#34;aws_iam_role\u0026#34; \u0026#34;metric_stream_to_firehose\u0026#34; { name = \u0026#34;metric_stream_to_firehose_role\u0026#34; assume_role_policy = data.aws_iam_policy_document.streams_assume_role.json } # https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-trustpolicy.html data \u0026#34;aws_iam_policy_document\u0026#34; \u0026#34;metric_stream_to_firehose\u0026#34; { statement { effect = \u0026#34;Allow\u0026#34; actions = [ \u0026#34;firehose:PutRecord\u0026#34;, \u0026#34;firehose:PutRecordBatch\u0026#34;, ] resources = [aws_kinesis_firehose_delivery_stream.http_stream.arn] } } resource \u0026#34;aws_iam_role_policy\u0026#34; \u0026#34;metric_stream_to_firehose\u0026#34; { name = \u0026#34;default\u0026#34; role = aws_iam_role.metric_stream_to_firehose.id policy = data.aws_iam_policy_document.metric_stream_to_firehose.json } resource \u0026#34;aws_s3_bucket\u0026#34; \u0026#34;bucket\u0026#34; { bucket = \u0026#34;metric-stream-test-bucket\u0026#34; } resource \u0026#34;aws_s3_bucket_acl\u0026#34; \u0026#34;bucket_acl\u0026#34; { bucket = aws_s3_bucket.bucket.id acl = \u0026#34;private\u0026#34; } data \u0026#34;aws_iam_policy_document\u0026#34; \u0026#34;firehose_assume_role\u0026#34; { statement { effect = \u0026#34;Allow\u0026#34; principals { type = \u0026#34;Service\u0026#34; identifiers = [\u0026#34;firehose.amazonaws.com\u0026#34;] } actions = [\u0026#34;sts:AssumeRole\u0026#34;] } } resource \u0026#34;aws_iam_role\u0026#34; \u0026#34;firehose_to_s3\u0026#34; { assume_role_policy = data.aws_iam_policy_document.firehose_assume_role.json } data \u0026#34;aws_iam_policy_document\u0026#34; \u0026#34;firehose_to_s3\u0026#34; { statement { effect = \u0026#34;Allow\u0026#34; actions = [ \u0026#34;s3:AbortMultipartUpload\u0026#34;, \u0026#34;s3:GetBucketLocation\u0026#34;, \u0026#34;s3:GetObject\u0026#34;, \u0026#34;s3:ListBucket\u0026#34;, \u0026#34;s3:ListBucketMultipartUploads\u0026#34;, \u0026#34;s3:PutObject\u0026#34;, ] resources = [ aws_s3_bucket.bucket.arn, \u0026#34;${aws_s3_bucket.bucket.arn}/*\u0026#34;, ] } } resource \u0026#34;aws_iam_role_policy\u0026#34; \u0026#34;firehose_to_s3\u0026#34; { name = \u0026#34;default\u0026#34; role = aws_iam_role.firehose_to_s3.id policy = data.aws_iam_policy_document.firehose_to_s3.json } resource \u0026#34;aws_kinesis_firehose_delivery_stream\u0026#34; \u0026#34;http_stream\u0026#34; { name = \u0026#34;metric-stream-test-stream\u0026#34; destination = \u0026#34;http_endpoint\u0026#34; http_endpoint_configuration { name = \u0026#34;test_http_endpoint\u0026#34; url = \u0026#34;这里填入Gateway的url\u0026#34; role_arn = aws_iam_role.firehose_to_s3.arn } s3_configuration { role_arn = aws_iam_role.firehose_to_s3.arn bucket_arn = aws_s3_bucket.bucket.arn } } 使用步骤：\n1.获取AWS账户的access_key以及secret_key。( 关于如何获取，请参考：create-access-key )\n2.将上一步中获取的access_key与secret_key填入对应位置，并将您的网关对应 url 填入 aws_kinesis_firehose_delivery_stream 配置的对应位置中。\n3.复制以上内容并保存到main.tf文件中。\n4.在对应路径下执行以下代码。\nterraform init terraform apply 至此，需要的AWS服务已全部建立成功，您可以检查您的控制台，查看服务是否成功创建。\n完成！ 如果以上步骤全部成功，请耐心等待约五分钟。之后您可以访问SkyWalking UI，查看指标变动情况\n目前，SkyWalking 默认收集的指标展示如下：\n账户指标:\n表指标：\n现已支持的服务 目前SkyWalking官方支持EKS，S3，DynamoDB监控。 用户也参考 OpenTelemetry receiver 配置OTEL rules来收集，计算AWS其他服务的CloudWatch metrics，并且通过自定义dashboard展示。\n相关的资料 Monitoring S3 metrics with Amazon CloudWatch Monitoring DynamoDB metrics with Amazon CloudWatch Supported metrics in AWS Firehose receiver of OAP Configuration Vocabulary | Apache SkyWalking ","excerpt":"\u003cp\u003e\u003cimg src=\"./icon.png\" alt=\"icon.png\"\u003e\u003c/p\u003e\n\u003ch2 id=\"背景\"\u003e背景\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"https://skywalking.apache.org/\"\u003eApache SkyWalking\u003c/a\u003e  是一个开源应用性能管理系统，帮助用户收集和聚合日志、追踪、指标和事件，并在 UI 上显示。从 OAP 9.4.0 开始，SkyWalking 新增了 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2023-03-13-skywalking-aws-dynamodb/","title":"使用SkyWalking监控DynamoDB"},{"body":"\nSKyWalking OAP\u0026rsquo;s existing OpenTelemetry receiver can receive metrics through the OTLP protocol, and use MAL to analyze related metrics in real time. Starting from OAP 9.4.0, SkyWalking has added an AWS Firehose receiver to receive and analyze CloudWatch metrics data. This article will take EKS and S3 as examples to introduce the process of SkyWalking OAP receiving and analyzing the indicator data of AWS services.\nEKS OpenTelemetry Collector OpenTelemetry (OTel) is a series of tools, APIs, and SDKs that can generate, collect, and export telemetry data, such as metrics, logs, and traces. OTel Collector is mainly responsible for collecting, processing, and exporting. For telemetry data, Collector consists of the following main components:\nReceiver: Responsible for obtaining telemetry data, different receivers support different data sources, such as prometheus, kafka, otlp. Processor: Process data between receiver and exporter, such as adding or deleting attributes. Exporter: Responsible for sending data to different backends, such as kafka, SkyWalking OAP (via OTLP). Service: Components enabled as a unit configuration, only configured components will be enabled. OpenTelemetry Protocol Specification(OTLP) OTLP mainly describes how to receive (pull) indicator data through gRPC and HTTP protocols. The OpenTelemetry receiver of SKyWalking OAP implements the OTLP/gRPC protocol, and the indicator data can be exported to OAP through the OTLP/gRPC exporter. Usually the data flow of a Collector is as follows:\nMonitor EKS with OTel EKS monitoring is realized through OTel. You only need to deploy OpenTelemetry Collector in the EKS cluster in the way of DaemonSet \u0026ndash; use AWS Container Insights Receiver as the receiver, and set the address of otlp exporter to the address of OAP. In addition, it should be noted that OAP is used job_name : aws-cloud-eks-monitoring as the identifier of EKS metrics according to the attribute, so it is necessary to configure a processor in the collector to add this attribute.\nOTel Collector configuration demo extensions: health_check: receivers: awscontainerinsightreceiver: processors: # To enable OAP to correctly identify EKS metrics, add the job_name attribute resource/job-name: attributes: - key: job_name value: aws-cloud-eks-monitoring action: insert # Specify OAP as exporters exporters: otlp: endpoint: oap-service:11800 tls: insecure: true logging: loglevel: debug service: pipelines: metrics: receivers: [awscontainerinsightreceiver] processors: [resource/job-name] exporters: [otlp,logging] extensions: [health_check] By default, SkyWalking OAP counts the network, disk, CPU and other related indicator data in the three dimensions of Node, Pod, and Service. Only part of the content is shown here.\nPod dimensions Service dimensions EKS monitoring complete configuration Click here to view complete k8s resource configuration apiVersion: v1 kind: ServiceAccount metadata: name: aws-otel-sa namespace: aws-otel-eks --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: aoc-agent-role rules: - apiGroups: [\u0026#34;\u0026#34;] resources: [\u0026#34;pods\u0026#34;, \u0026#34;nodes\u0026#34;, \u0026#34;endpoints\u0026#34;] verbs: [\u0026#34;list\u0026#34;, \u0026#34;watch\u0026#34;] - apiGroups: [\u0026#34;apps\u0026#34;] resources: [\u0026#34;replicasets\u0026#34;] verbs: [\u0026#34;list\u0026#34;, \u0026#34;watch\u0026#34;] - apiGroups: [\u0026#34;batch\u0026#34;] resources: [\u0026#34;jobs\u0026#34;] verbs: [\u0026#34;list\u0026#34;, \u0026#34;watch\u0026#34;] - apiGroups: [\u0026#34;\u0026#34;] resources: [\u0026#34;nodes/proxy\u0026#34;] verbs: [\u0026#34;get\u0026#34;] - apiGroups: [\u0026#34;\u0026#34;] resources: [\u0026#34;nodes/stats\u0026#34;, \u0026#34;configmaps\u0026#34;, \u0026#34;events\u0026#34;] verbs: [\u0026#34;create\u0026#34;, \u0026#34;get\u0026#34;] - apiGroups: [\u0026#34;\u0026#34;] resources: [\u0026#34;configmaps\u0026#34;] resourceNames: [\u0026#34;otel-container-insight-clusterleader\u0026#34;] verbs: [\u0026#34;get\u0026#34;,\u0026#34;update\u0026#34;] - apiGroups: [\u0026#34;coordination.k8s.io\u0026#34;] resources: [\u0026#34;leases\u0026#34;] verbs: [\u0026#34;create\u0026#34;,\u0026#34;get\u0026#34;,\u0026#34;update\u0026#34;] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: aoc-agent-role-binding subjects: - kind: ServiceAccount name: aws-otel-sa namespace: aws-otel-eks roleRef: kind: ClusterRole name: aoc-agent-role apiGroup: rbac.authorization.k8s.io --- apiVersion: v1 kind: ConfigMap metadata: name: otel-agent-conf namespace: aws-otel-eks labels: app: opentelemetry component: otel-agent-conf data: otel-agent-config: | extensions: health_check: receivers: awscontainerinsightreceiver: processors: resource/job-name: attributes: - key: job_name value: aws-cloud-eks-monitoring action: insert exporters: otlp: endpoint: oap-service:11800 tls: insecure: true logging: loglevel: debug service: pipelines: metrics: receivers: [awscontainerinsightreceiver] processors: [resource/job-name] exporters: [otlp,logging] extensions: [health_check] --- apiVersion: apps/v1 kind: DaemonSet metadata: name: aws-otel-eks-ci namespace: aws-otel-eks spec: selector: matchLabels: name: aws-otel-eks-ci template: metadata: labels: name: aws-otel-eks-ci spec: containers: - name: aws-otel-collector image: amazon/aws-otel-collector:v0.23.0 env: # Specify region - name: AWS_REGION value: \u0026#34;ap-northeast-1\u0026#34; - name: K8S_NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: HOST_IP valueFrom: fieldRef: fieldPath: status.hostIP - name: HOST_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: K8S_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace imagePullPolicy: Always command: - \u0026#34;/awscollector\u0026#34; - \u0026#34;--config=/conf/otel-agent-config.yaml\u0026#34; volumeMounts: - name: rootfs mountPath: /rootfs readOnly: true - name: dockersock mountPath: /var/run/docker.sock readOnly: true - name: varlibdocker mountPath: /var/lib/docker readOnly: true - name: containerdsock mountPath: /run/containerd/containerd.sock readOnly: true - name: sys mountPath: /sys readOnly: true - name: devdisk mountPath: /dev/disk readOnly: true - name: otel-agent-config-vol mountPath: /conf - name: otel-output-vol mountPath: /otel-output resources: limits: cpu: 200m memory: 200Mi requests: cpu: 200m memory: 200Mi volumes: - configMap: name: otel-agent-conf items: - key: otel-agent-config path: otel-agent-config.yaml name: otel-agent-config-vol - name: rootfs hostPath: path: / - name: dockersock hostPath: path: /var/run/docker.sock - name: varlibdocker hostPath: path: /var/lib/docker - name: containerdsock hostPath: path: /run/containerd/containerd.sock - name: sys hostPath: path: /sys - name: devdisk hostPath: path: /dev/disk/ - name: otel-output-vol hostPath: path: /otel-output serviceAccountName: aws-otel-sa S3 Amazon CloudWatch Amazon CloudWatch is a monitoring service provided by AWS. It is responsible for collecting indicator data of AWS services and resources. CloudWatch metrics stream is responsible for converting indicator data into stream processing data, and supports output in two formats: json and OTel v0.7.0.\nAmazon Kinesis Data Firehose (Firehose) Firehose is an extract, transform, load (ETL) service that reliably captures, transforms, and serves streaming data into data lakes, data stores (such as S3), and analytics services.\nTo ensure that external services can correctly receive indicator data, AWS provides Kinesis Data Firehose HTTP Endpoint Delivery Request and Response Specifications (Firehose Specifications) . Firhose pushes Json data by POST\nJson data example { \u0026#34;requestId\u0026#34;: \u0026#34;ed4acda5-034f-9f42-bba1-f29aea6d7d8f\u0026#34;, \u0026#34;timestamp\u0026#34;: 1578090901599 \u0026#34;records\u0026#34;: [ { \u0026#34;data\u0026#34;: \u0026#34;aGVsbG8=\u0026#34; }, { \u0026#34;data\u0026#34;: \u0026#34;aGVsbG8gd29ybGQ=\u0026#34; } ] } requestId: Request id, which can achieve deduplication and debugging purposes. timestamp: Firehose generated the timestamp of the request (in milliseconds). records: Actual delivery records data: The delivered data, encoded in base64, can be in json or OTel v0.7.0 format, depending on the format of CloudWatch data (described later). Skywalking currently supports OTel v0.7.0 format. aws-firehose-receiver aws-firehose-receiver provides an HTTP Endpoint that implements Firehose Specifications: /aws/firehose/metrics. The figure below shows the data flow of monitoring DynamoDB, S3 and other services through CloudWatch, and using Firehose to send indicator data to SKywalking OAP.\nStep-by-step setup of S3 monitoring Enter the S3 console and create a filter forRequest metrics: Amazon S3 \u0026gt;\u0026gt; Buckets \u0026gt;\u0026gt; (Your Bucket) \u0026gt;\u0026gt; Metrics \u0026gt;\u0026gt; metrics \u0026gt;\u0026gt; View additional charts \u0026gt;\u0026gt; Request metrics Enter the Amazon Kinesis console, create a delivery stream, Source select Direct PUT, Destination select HTTP Endpoint. And set HTTP endpoint URL to https://your_domain/aws/firehose/metrics. Other configuration items: Buffer hints: Set the size and period of the cache Access key just matches the AccessKey in aws-firehose-receiver Retry duration: Retry period Backup settings: Backup settings, optionally backup the posted data to S3 at the same time. Enter the CloudWatch console Streams and click Create CloudWatch Stream. And Select your Kinesis Data Firehose stream configure the delivery stream created in the second step in the item. Note that it needs to be set Change output format to OpenTelemetry v0.7.0. At this point, the S3 monitoring configuration settings are complete. The S3 metrics currently collected by SkyWalking by default are shown below:\nOther service Currently SkyWalking officially supports EKS, S3, DynamoDB monitoring. Users also refer to the OpenTelemetry receiver to configure OTel rules to collect and analyze CloudWatch metrics of other AWS services, and display them through a custom dashboard.\nMaterial Monitoring S3 metrics with Amazon CloudWatch Monitoring DynamoDB metrics with Amazon CloudWatch Supported metrics in AWS Firehose receiver of OAP Configuration Vocabulary | Apache SkyWalking ","excerpt":"\u003cp\u003e\u003cimg src=\"./icon.png\" alt=\"icon.png\"\u003e\u003c/p\u003e\n\u003cp\u003eSKyWalking OAP\u0026rsquo;s existing \u003ca href=\"https://skywalking.apache.org/docs/main/next/en/setup/backend/opentelemetry-receiver/\"\u003eOpenTelemetry receiver\u003c/a\u003e can receive metrics through the \u003ca href=\"https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md\"\u003eOTLP …\u003c/a\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2023-03-12-skywalking-aws-s3-eks/","title":"Monitoring AWS EKS and S3 with SkyWalking"},{"body":"\nSKyWalking OAP 现有的 OpenTelemetry receiver 可以通过OTLP协议接收指标(metrics)，并且使用MAL实时分析相关指标。从OAP 9.4.0开始，SkyWalking 新增了AWS Firehose receiver，用来接收，分析CloudWatch metrics数据。本文将以EKS和S3为例介绍SkyWalking OAP 接收，分析 AWS 服务的指标数据的过程\nEKS OpenTelemetry Collector OpenTelemetry (OTel) 是一系列tools，API，SDK，可以生成，收集，导出遥测数据，比如 指标(metrics)，日志(logs)和链路信息(traces)，而OTel Collector主要负责收集、处理和导出遥测数据，Collector由以下主要组件组成:\nreceiver: 负责获取遥测数据，不同的receiver支持不同的数据源，比如prometheus ，kafka，otlp， processor：在receiver和exporter之间处理数据，比如增加或者删除attributes， exporter：负责发送数据到不同的后端，比如kafka，SkyWalking OAP(通过OTLP) service: 作为一个单元配置启用的组件，只有配置的组件才会被启用 OpenTelemetry Protocol Specification(OTLP) OTLP 主要描述了如何通过gRPC，HTTP协议接收(拉取)指标数据。SKyWalking OAP的 OpenTelemetry receiver 实现了OTLP/gRPC协议，通过OTLP/gRPC exporter可以将指标数据导出到OAP。通常一个Collector的数据流向如下:\n使用OTel监控EKS EKS的监控就是通过OTel实现的，只需在EKS集群中以DaemonSet 的方式部署 OpenTelemetry Collector，使用 AWS Container Insights Receiver 作为receiver，并且设置otlp exporter的地址为OAP的的地址即可。另外需要注意的是OAP根据attribute job_name : aws-cloud-eks-monitoring 作为EKS metrics的标识，所以还需要再collector中配置一个processor来增加这个属性\nOTel Collector配置demo extensions: health_check: receivers: awscontainerinsightreceiver: processors: # 为了OAP能够正确识别EKS metrics，增加job_name attribute resource/job-name: attributes: - key: job_name value: aws-cloud-eks-monitoring action: insert # 指定OAP作为 exporters exporters: otlp: endpoint: oap-service:11800 tls: insecure: true logging: loglevel: debug service: pipelines: metrics: receivers: [awscontainerinsightreceiver] processors: [resource/job-name] exporters: [otlp,logging] extensions: [health_check] SkyWalking OAP 默认统计 Node，Pod，Service 三个维度的网络、磁盘、CPU等相关的指标数据，这里仅展示了部分内容\nPod 维度 Service 维度 EKS监控完整配置 Click here to view complete k8s resource configuration apiVersion: v1 kind: ServiceAccount metadata: name: aws-otel-sa namespace: aws-otel-eks --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: aoc-agent-role rules: - apiGroups: [\u0026#34;\u0026#34;] resources: [\u0026#34;pods\u0026#34;, \u0026#34;nodes\u0026#34;, \u0026#34;endpoints\u0026#34;] verbs: [\u0026#34;list\u0026#34;, \u0026#34;watch\u0026#34;] - apiGroups: [\u0026#34;apps\u0026#34;] resources: [\u0026#34;replicasets\u0026#34;] verbs: [\u0026#34;list\u0026#34;, \u0026#34;watch\u0026#34;] - apiGroups: [\u0026#34;batch\u0026#34;] resources: [\u0026#34;jobs\u0026#34;] verbs: [\u0026#34;list\u0026#34;, \u0026#34;watch\u0026#34;] - apiGroups: [\u0026#34;\u0026#34;] resources: [\u0026#34;nodes/proxy\u0026#34;] verbs: [\u0026#34;get\u0026#34;] - apiGroups: [\u0026#34;\u0026#34;] resources: [\u0026#34;nodes/stats\u0026#34;, \u0026#34;configmaps\u0026#34;, \u0026#34;events\u0026#34;] verbs: [\u0026#34;create\u0026#34;, \u0026#34;get\u0026#34;] - apiGroups: [\u0026#34;\u0026#34;] resources: [\u0026#34;configmaps\u0026#34;] resourceNames: [\u0026#34;otel-container-insight-clusterleader\u0026#34;] verbs: [\u0026#34;get\u0026#34;,\u0026#34;update\u0026#34;] - apiGroups: [\u0026#34;coordination.k8s.io\u0026#34;] resources: [\u0026#34;leases\u0026#34;] verbs: [\u0026#34;create\u0026#34;,\u0026#34;get\u0026#34;,\u0026#34;update\u0026#34;] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: aoc-agent-role-binding subjects: - kind: ServiceAccount name: aws-otel-sa namespace: aws-otel-eks roleRef: kind: ClusterRole name: aoc-agent-role apiGroup: rbac.authorization.k8s.io --- apiVersion: v1 kind: ConfigMap metadata: name: otel-agent-conf namespace: aws-otel-eks labels: app: opentelemetry component: otel-agent-conf data: otel-agent-config: | extensions: health_check: receivers: awscontainerinsightreceiver: processors: resource/job-name: attributes: - key: job_name value: aws-cloud-eks-monitoring action: insert exporters: otlp: endpoint: oap-service:11800 tls: insecure: true logging: loglevel: debug service: pipelines: metrics: receivers: [awscontainerinsightreceiver] processors: [resource/job-name] exporters: [otlp,logging] extensions: [health_check] --- apiVersion: apps/v1 kind: DaemonSet metadata: name: aws-otel-eks-ci namespace: aws-otel-eks spec: selector: matchLabels: name: aws-otel-eks-ci template: metadata: labels: name: aws-otel-eks-ci spec: containers: - name: aws-otel-collector image: amazon/aws-otel-collector:v0.23.0 env: # Specify region - name: AWS_REGION value: \u0026#34;ap-northeast-1\u0026#34; - name: K8S_NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: HOST_IP valueFrom: fieldRef: fieldPath: status.hostIP - name: HOST_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: K8S_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace imagePullPolicy: Always command: - \u0026#34;/awscollector\u0026#34; - \u0026#34;--config=/conf/otel-agent-config.yaml\u0026#34; volumeMounts: - name: rootfs mountPath: /rootfs readOnly: true - name: dockersock mountPath: /var/run/docker.sock readOnly: true - name: varlibdocker mountPath: /var/lib/docker readOnly: true - name: containerdsock mountPath: /run/containerd/containerd.sock readOnly: true - name: sys mountPath: /sys readOnly: true - name: devdisk mountPath: /dev/disk readOnly: true - name: otel-agent-config-vol mountPath: /conf - name: otel-output-vol mountPath: /otel-output resources: limits: cpu: 200m memory: 200Mi requests: cpu: 200m memory: 200Mi volumes: - configMap: name: otel-agent-conf items: - key: otel-agent-config path: otel-agent-config.yaml name: otel-agent-config-vol - name: rootfs hostPath: path: / - name: dockersock hostPath: path: /var/run/docker.sock - name: varlibdocker hostPath: path: /var/lib/docker - name: containerdsock hostPath: path: /run/containerd/containerd.sock - name: sys hostPath: path: /sys - name: devdisk hostPath: path: /dev/disk/ - name: otel-output-vol hostPath: path: /otel-output serviceAccountName: aws-otel-sa S3 Amazon CloudWatch Amazon CloudWatch 是AWS提供的监控服务，负责收集AWS 服务，资源的指标数据，CloudWatch metrics stream 负责将指标数据转换为流式处理数据，支持输出json，OTel v0.7.0 两种格式。\nAmazon Kinesis Data Firehose (Firehose) Firehose 是一项提取、转换、加载（ETL）服务，可以将流式处理数据以可靠方式捕获、转换和提供到数据湖、数据存储(比如S3)和分析服务中。\n为了确保外部服务能够正确地接收指标数据， AWS提供了 Kinesis Data Firehose HTTP Endpoint Delivery Request and Response Specifications (Firehose Specifications)。Firhose以POST的方式推送Json数据\nJson数据示例 { \u0026#34;requestId\u0026#34;: \u0026#34;ed4acda5-034f-9f42-bba1-f29aea6d7d8f\u0026#34;, \u0026#34;timestamp\u0026#34;: 1578090901599 \u0026#34;records\u0026#34;: [ { \u0026#34;data\u0026#34;: \u0026#34;aGVsbG8=\u0026#34; }, { \u0026#34;data\u0026#34;: \u0026#34;aGVsbG8gd29ybGQ=\u0026#34; } ] } requestId: 请求id，可以实现去重，debug目的 timestamp: Firehose 产生该请求的时间戳(毫秒) records: 实际投递的记录 data: 投递的数据，以base64编码数据，可以是json或者OTel v0.7.0格式，取决于CloudWatch数据数据的格式(稍后会有描述)。Skywalking目前支持OTel v0.7.0格式 aws-firehose-receiver aws-firehose-receiver 就是提供了一个实现了Firehose Specifications的HTTP Endpoint:/aws/firehose/metrics。下图展示了通过CloudWatch监控DynamoDB，S3等服务，并利用Firehose将指标数据发送到SKywalking OAP的数据流向\n从上图可以看到 aws-firehose-receiver 将数据转换后交由 OpenTelemetry-receiver处理 ，所以 OpenTelemetry receiver 中配置的 otel-rules 同样可以适用CloudWatch metrics\n注意 因为 Kinesis Data Firehose 要求，必须在AWS Firehose receiver 前放置一个Gateway用来建立HTTPS链接。aws-firehose-receiver 将从v9.5.0开始支持HTTPS协议 TLS 证书必须是CA签发的 逐步设置S3监控 进入 S3控制台，通过 Amazon S3 \u0026gt;\u0026gt; Buckets \u0026gt;\u0026gt; (Your Bucket) \u0026gt;\u0026gt; Metrics \u0026gt;\u0026gt; metrics \u0026gt;\u0026gt; View additional charts \u0026gt;\u0026gt; Request metrics 为 Request metrics 创建filter 进入Amazon Kinesis 控制台，创建一个delivery stream， Source选择 Direct PUT, Destination 选择 HTTP Endpoint. 并且设置HTTP endpoint URL 为 https://your_domain/aws/firehose/metrics。其他配置项: Buffer hints: 设置缓存的大小和周期 Access key 与aws-firehose-receiver中的AccessKey一致即可 Retry duration: 重试周期 Backup settings: 备份设置，可选地将投递的数据同时备份到S3。 进入 CloudWatch控制台，Streams 标签创建CloudWatch Stream。并且在Select your Kinesis Data Firehose stream项中配置第二步创建的delivery stream。注意需要设置Change output format 为 OpenTelemetry v0.7.0。 至此，S3监控配置设置完成。目前SkyWalking默认收集的S3 metrics 展示如下\n其他服务 目前SkyWalking官方支持EKS，S3，DynamoDB监控。 用户也参考 OpenTelemetry receiver 配置OTel rules来收集，分析AWS其他服务的CloudWatch metrics，并且通过自定义dashboard展示\n资料 Monitoring S3 metrics with Amazon CloudWatch Monitoring DynamoDB metrics with Amazon CloudWatch Supported metrics in AWS Firehose receiver of OAP Configuration Vocabulary | Apache SkyWalking ","excerpt":"\u003cp\u003e\u003cimg src=\"./icon.png\" alt=\"icon.png\"\u003e\u003c/p\u003e\n\u003cp\u003eSKyWalking OAP 现有的 \u003ca href=\"https://skywalking.apache.org/docs/main/next/en/setup/backend/opentelemetry-receiver/\"\u003eOpenTelemetry receiver\u003c/a\u003e 可以通过\u003ca href=\"https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md\"\u003eOTLP\u003c/a\u003e协议接收指标(metrics)，并且使用\u003ca href=\"https://skywalking.apache.org/docs/main/next/en/concepts-and-designs/mal/\"\u003eMAL\u003c/a\u003e实时分析相关指标。从OAP 9.4.0开始 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2023-03-12-skywalking-aws-s3-eks/","title":"使用SkyWalking监控AWS EKS和S3"},{"body":"SkyWalking Rust 0.6.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed Refactor span object api to make it more friendly. by @jmjoy in https://github.com/apache/skywalking-rust/pull/52 Refactor management report and keep alive api. by @jmjoy in https://github.com/apache/skywalking-rust/pull/53 Use stream and completed for a bulk to collect for grpc reporter. by @jmjoy in https://github.com/apache/skywalking-rust/pull/54 Add sub components licenses in dist material. by @jmjoy in https://github.com/apache/skywalking-rust/pull/55 Bump to 0.6.0. by @jmjoy in https://github.com/apache/skywalking-rust/pull/56 ","excerpt":"\u003cp\u003eSkyWalking Rust 0.6.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-rust-0-6-0/","title":"Release Apache SkyWalking Rust 0.6.0"},{"body":"SkyWalking 9.4.0 is released. Go to downloads page to find release tars.\nPromQL and Grafana Support Zipkin Lens UI Bundled AWS S3 and DynamoDB monitoring Project Bump up Zipkin and Zipkin lens UI dependency to 2.24.0. Bump up Apache parent pom version to 29. Bump up Armeria version to 1.21.0. Clean up maven pom.xmls. Bump up Java version to 11. Bump up snakeyaml to 2.0. OAP Server Add ServerStatusService in the core module to provide a new way to expose booting status to other modules. Adds Micrometer as a new component.(ID=141) Refactor session cache in MetricsPersistentWorker. Cache enhancement - don\u0026rsquo;t read new metrics from database in minute dimensionality. // When // (1) the time bucket of the server\u0026#39;s latest stability status is provided // 1.1 the OAP has booted successfully // 1.2 the current dimensionality is in minute. // 1.3 the OAP cluster is rebalanced due to scaling // (2) the metrics are from the time after the timeOfLatestStabilitySts // (3) the metrics don\u0026#39;t exist in the cache // the kernel should NOT try to load it from the database. // // Notice, about condition (2), // for the specific minute of booted successfully, the metrics are expected to load from database when // it doesn\u0026#39;t exist in the cache. Remove the offset of metric session timeout according to worker creation sequence. Correct MetricsExtension annotations declarations in manual entities. Support component IDs\u0026rsquo; priority in process relation metrics. Remove abandon logic in MergableBufferedData, which caused unexpected no-update. Fix miss set LastUpdateTimestamp that caused the metrics session to expire. Rename MAL rule spring-sleuth.yaml to spring-micrometer.yaml. Fix memory leak in Zipkin API. Remove the dependency of refresh_interval of ElasticSearch indices from elasticsearch/flushInterval config. Now, it uses core/persistentPeriod + 5s as refresh_interval for all indices instead. Change elasticsearch/flushInterval to 5s(was 15s). Optimize flushInterval of ElasticSearch BulkProcessor to avoid extra periodical flush in the continuous bulk streams. An unexpected dot is added when exp is a pure metric name and expPrefix != null. Support monitoring MariaDB. Remove measure/stream specific interval settings in BanyanDB. Add global-specific settings used to override global configurations (e.g segmentIntervalDays, blockIntervalHours) in BanyanDB. Use TTL-driven interval settings for the measure-default group in BanyanDB. Fix wrong group of non time-relative metadata in BanyanDB. Refactor StorageData#id to the new StorageID object from a String type. Support multiple component IDs in the service topology level. Add ElasticSearch.Keyword annotation to declare the target field type as keyword. [Breaking Change] Column component_id of service_relation_client_side and service_relation_server_side have been replaced by component_ids. Support priority definition in the component-libraries.yml. Enhance service topology query. When there are multiple components detected from the server side, the component type of the node would be determined by the priority, which was random in the previous release. Remove component_id from service_instance_relation_client_side and service_instance_relation_server_side. Make the satellite E2E test more stable. Add Istio 1.16 to test matrix. Register ValueColumn as Tag for Record in BanyanDB storage plugin. Bump up Netty to 4.1.86. Remove unnecessary additional columns when storage is in logical sharding mode. The cluster coordinator support watch mechanism for notifying RemoteClientManager and ServerStatusService. Fix ServiceMeshServiceDispatcher overwrite ServiceDispatcher debug file when open SW_OAL_ENGINE_DEBUG. Use groupBy and in operators to optimize topology query for BanyanDB storage plugin. Support server status watcher for MetricsPersistentWorker to check the metrics whether required initialization. Fix the meter value are not correct when using sumPerMinLabeld or sumHistogramPercentile MAL function. Fix cannot display attached events when using Zipkin Lens UI query traces. Remove time_bucket for both Stream and Measure kinds in BanyanDB plugin. Merge TIME_BUCKET of Metrics and Record into StorageData. Support no layer in the listServices query. Fix time_bucket of ServiceTraffic not set correctly in slowSql of MAL. Correct the TopN record query DAO of BanyanDB. Tweak interval settings of BanyanDB. Support monitoring AWS Cloud EKS. Bump BanyanDB Java client to 0.3.0-rc1. Remove id tag from measures. Add Banyandb.MeasureField to mark a column as a BanyanDB Measure field. Add BanyanDB.StoreIDTag to store a process\u0026rsquo;s id for searching. [Breaking Change] The supported version of ShardingSphere-Proxy is upgraded from 5.1.2 to 5.3.1. Due to the changes of ShardingSphere\u0026rsquo;s API, versions before 5.3.1 are not compatible. Add the eBPF network profiling E2E Test in the per storage. Fix TCP service instances are lack of instance properties like pod and namespace, which causes Pod log not to work for TCP workloads. Add Python HBase happybase module component ID(94). Fix gRPC alarm cannot update settings from dynamic configuration source. Add batchOfBytes configuration to limit the size of bulk flush. Add Python Websocket module component ID(7018). [Optional] Optimize single trace query performance by customizing routing in ElasticSearch. SkyWalking trace segments and Zipkin spans are using trace ID for routing. This is OFF by default, controlled by storage/elasticsearch/enableCustomRouting. Enhance OAP HTTP server to support HTTPS Remove handler scan in otel receiver, manual initialization instead Add aws-firehose-receiver to support collecting AWS CloudWatch metric(OpenTelemetry format). Notice, no HTTPS/TLS setup support. By following AWS Firehose request, it uses proxy request (https://... instead of /aws/firehose/metrics), there must be a proxy(Nginx, Envoy, etc.). Avoid Antlr dependencies\u0026rsquo; versions might be different in compile time and runtime. Now PrometheusMetricConverter#escapedName also support converting / to _. Add missing TCP throughput metrics. Refactor @Column annotation, swap Column#name and ElasticSearch.Column#columnAlias and rename ElasticSearch.Column#columnAlias to ElasticSearch.Column#legacyName. Add Python HTTPX module component ID(7019). Migrate tests from junit 4 to junit 5. Refactor http-based alarm plugins and extract common logic to HttpAlarmCallback. Support Amazon Simple Storage Service (Amazon S3) metrics monitoring Support process Sum metrics with AGGREGATION_TEMPORALITY_DELTA case Support Amazon DynamoDB monitoring. Support prometheus HTTP API and promQL. Scope in the Entity of Metrics query v1 protocol is not required and automatical correction. The scope is determined based on the metric itself. Add explicit ReadTimeout for ConsulConfigurationWatcher to avoid IllegalArgumentException: Cache watchInterval=10sec \u0026gt;= networkClientReadTimeout=10000ms. Fix DurationUtils.getDurationPoints exceed, when startTimeBucket equals endTimeBucket. Support process OpenTelemetry ExponentialHistogram metrics Add FreeRedis component ID(3018). UI Add Zipkin Lens UI to webapp, and proxy it to context path /zipkin. Migrate the build tool from vue cli to Vite4. Fix Instance Relation and Endpoint Relation dashboards show up. Add Micrometer icon. Update MySQL UI to support MariaDB. Add AWS menu for supporting AWS monitoring. Add missing FastAPI logo. Update the log details page to support the formatted display of JSON content. Fix build config. Avoid being unable to drag process nodes for the first time. Add node folder into ignore list. Add ElPopconfirm to component types. Add an iframe widget for zipkin UI. Optimize graph tooltips to make them more friendly. Bump json5 from 1.0.1 to 1.0.2. Add websockets icon. Implement independent mode for widgets. Bump http-cache-semantics from 4.1.0 to 4.1.1. Update menus for OpenFunction. Add auto fresh to widgets independent mode. Fix: clear trace ID on the Log and Trace widgets after using association. Fix: reset duration for query conditions after time range changes. Add AWS S3 menu. Refactor: optimize side bar component to make it more friendly. Fix: remove duplicate popup message for query result. Add logo for HTTPX. Refactor: optimize the attached events visualization in the trace widget. Update BanyanDB client to 0.3.1. Add AWS DynamoDB menu. Fix: add auto period to the independent mode for widgets. Optimize menus and add Windows monitoring menu. Add a calculation for the cpm5dAvg. add a cpm5d calculation. Fix data processing error in the eBPF profiling widget. Support for double quotes in SlowSQL statements. Fix: the wrong position of the menu when clicking the topology node. Documentation Remove Spring Sleuth docs, and add Spring MicroMeter Observations Analysis with the latest Java agent side enhancement. Update monitoring MySQL document to add the MariaDB part. Reorganize the protocols docs to a more clear API docs. Add documentation about replacing Zipkin server with SkyWalking OAP. Add Lens UI relative docs in Zipkin trace section. Add Profiling APIs. Fix backend telemetry doc and so11y dashboard doc as the OAP Prometheus fetcher was removed since 9.3.0 All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking 9.4.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"promql-and-grafana-support\"\u003ePromQL and Grafana Support …\u003c/h4\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-9.4.0/","title":"Release Apache SkyWalking APM 9.4.0"},{"body":"SkyWalking BanyanDB 0.3.1 is released. Go to downloads page to find release tars.\nBugs Fix the broken of schema chain. Add a timeout to all go leaking checkers. Chores Bump golang.org/x/net from 0.2.0 to 0.7.0. ","excerpt":"\u003cp\u003eSkyWalking BanyanDB 0.3.1 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"bugs\"\u003eBugs\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003eFix the …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-0-3-1/","title":"Release Apache SkyWalking BanyanDB 0.3.1"},{"body":"SkyWalking Python 1.0.0 is released! Go to downloads page to find release tars.\nPyPI Wheel: https://pypi.org/project/apache-skywalking/1.0.0/\nDockerHub Image: https://hub.docker.com/r/apache/skywalking-python\nImportant Notes and Breaking Changes:\nThe new PVM metrics reported from Python agent requires SkyWalking OAP v9.3.0 to show out-of-the-box. BREAKING: Python 3.6 is no longer supported and may not function properly, Python 3.11 support is added and tested. BREAKING: A number of common configuration options and environment variables are renamed to follow the convention of Java agent, please check with the latest official documentation before upgrading. (#273, #282) https://skywalking.apache.org/docs/skywalking-python/v1.0.0/en/setup/configuration/ BREAKING: All agent core capabilities are now covered by test cases and enabled by default (Trace, Log, PVM runtime metrics, Profiler) BREAKING: DockerHub Python agent images since v1.0.0 will no longer include the run part in ENTRYPOINT [\u0026quot;sw-python\u0026quot;, \u0026quot;run\u0026quot;], user should prefix their command with [-d/--debug] run [-p/--prefork] \u0026lt;Command\u0026gt; for extra flexibility. Packaged wheel now provides a extra [all] option to support all three report protocols Feature:\nAdd support for Python 3.11 (#285) Add MeterReportService (gRPC, Kafka reporter) (default:enabled) (#231, #236, #241, #243) Add reporter for PVM runtime metrics (default:enabled) (#238, #247) Add Greenlet profiler (#246) Add test and support for Python Slim base images (#249) Add support for the tags of Virtual Cache for Redis (#263) Add a new configuration kafka_namespace to prefix the kafka topic names (#277) Add log reporter support for loguru (#276) Add experimental support for explicit os.fork(), restarts agent in forked process (#286) Add experimental sw-python CLI sw-python run [-p] flag (-p/\u0026ndash;prefork) to enable non-intrusive uWSGI and Gunicorn postfork support (#288) Plugins:\nAdd aioredis, aiormq, amqp, asyncpg, aio-pika, kombu RMQ plugins (#230 Missing test coverage) Add Confluent Kafka plugin (#233 Missing test coverage) Add HBase plugin Python HappyBase model (#266) Add FastAPI plugin websocket protocol support (#269) Add Websockets (client) plugin (#269) Add HTTPX plugin (#283) Fixes:\nAllow RabbitMQ BlockingChannel.basic_consume() to link with outgoing spans (#224) Fix RabbitMQ basic_get bug (#225, #226) Fix case when tornado socket name is None (#227) Fix misspelled text \u0026ldquo;PostgreSLQ\u0026rdquo; -\u0026gt; \u0026ldquo;PostgreSQL\u0026rdquo; in Postgres-related plugins (#234) Make sure span.component initialized as Unknown rather than 0 (#242) Ignore websocket connections inside fastapi temporarily (#244, issue#9724) Fix Kafka-python plugin SkyWalking self reporter ignore condition (#249) Add primary endpoint in tracing context and endpoint info to log reporter (#261) Enforce tag class type conversion (#262) Fix sw_logging (log reporter) potentially throw exception leading to traceback confusion (#267) Avoid reporting meaningless tracecontext with logs when there\u0026rsquo;s no active span, UI will now show empty traceID (#272) Fix exception handler in profile_context (#273) Add namespace suffix to service name (#275) Add periodical instance property report to prevent data loss (#279) Fix sw_logging when Logger.disabled is true (#281) Docs:\nNew documentation on how to test locally (#222) New documentation on the newly added meter reporter feature (#240) New documentation on the newly added greenlet profiler and the original threading profiler (#250) Overhaul documentation on development setup and testing (#249) Add tables to state currently supported features of Python agent. (#271) New configuration documentation generator (#273) Others:\nPin CI SkyWalking License Eye (#221) Fix dead link due to the \u0026rsquo;next\u0026rsquo; url change (#235) Pin CI SkyWalking Infra-E2E (#251) Sync OAP, SWCTL versions in E2E and fix test cases (#249) Overhaul development flow with Poetry (#249) Fix grpcio-tools generated message type (#253) Switch plugin tests to use slim Python images (#268) Add unit tests to sw_filters (#269) New Contributors @ZEALi made their first contribution in https://github.com/apache/skywalking-python/pull/242 @westarest made their first contribution in https://github.com/apache/skywalking-python/pull/246 @Jedore made their first contribution in https://github.com/apache/skywalking-python/pull/263 @alidisi made their first contribution in https://github.com/apache/skywalking-python/pull/266 @SheltonZSL made their first contribution in https://github.com/apache/skywalking-python/pull/275 @XinweiLyu made their first contribution in https://github.com/apache/skywalking-python/pull/283 Full Changelog: https://github.com/apache/skywalking-python/compare/v0.8.0...v1.0.0\n","excerpt":"\u003cp\u003eSkyWalking Python 1.0.0 is released! Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePyPI Wheel\u003c/strong\u003e: …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-python-1-0-0/","title":"Release Apache SkyWalking Python 1.0.0"},{"body":"SkyWalking BanyanDB 0.3.0 is released. Go to downloads page to find release tars.\nFeatures Support 64-bit float type. Web Application. Close components in tsdb gracefully. Add TLS for the HTTP server. Use the table builder to compress data. Bugs Open blocks concurrently. Sync index writing and shard closing. TimestampRange query throws an exception if no data in this time range. Chores Fixes issues related to leaked goroutines. Add validations to APIs. ","excerpt":"\u003cp\u003eSkyWalking BanyanDB 0.3.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"features\"\u003eFeatures\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eSupport …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-0-3-0/","title":"Release Apache SkyWalking BanyanDB 0.3.0"},{"body":"SkyWalking PHP 0.3.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed Make explicit rust version requirement by @wu-sheng in https://github.com/apache/skywalking-php/pull/35 Update dependencies version limitation. by @jmjoy in https://github.com/apache/skywalking-php/pull/36 Startup 0.3.0 by @heyanlong in https://github.com/apache/skywalking-php/pull/37 Support PHP 8.2 by @heyanlong in https://github.com/apache/skywalking-php/pull/38 Fix php-fpm freeze after large amount of request. by @jmjoy in https://github.com/apache/skywalking-php/pull/39 Lock develop rust version to 1.65, upgrade deps. by @jmjoy in https://github.com/apache/skywalking-php/pull/41 Fix worker unexpected shutdown. by @jmjoy in https://github.com/apache/skywalking-php/pull/42 Update docs about installing rust. by @jmjoy in https://github.com/apache/skywalking-php/pull/43 Retry cargo test when failed in CI. by @jmjoy in https://github.com/apache/skywalking-php/pull/44 Hack dtor for mysqli to cleanup resources. by @jmjoy in https://github.com/apache/skywalking-php/pull/45 Report instance properties and keep alive. by @jmjoy in https://github.com/apache/skywalking-php/pull/46 Add configuration option skywalking_agent.runtime_dir. by @jmjoy in https://github.com/apache/skywalking-php/pull/47 Add authentication support. by @jmjoy in https://github.com/apache/skywalking-php/pull/48 Support TLS. by @jmjoy in https://github.com/apache/skywalking-php/pull/49 Periodic reporting instance properties. by @jmjoy in https://github.com/apache/skywalking-php/pull/50 Bump to 0.3.0. by @jmjoy in https://github.com/apache/skywalking-php/pull/51 Breaking Remove http:// scheme in skywalking_agent.server_addr. New Contributors @wu-sheng made their first contribution in https://github.com/apache/skywalking-php/pull/35 Full Changelog: https://github.com/apache/skywalking-php/compare/v0.2.0...v0.3.0\nPECL https://pecl.php.net/package/skywalking_agent/0.3.0\n","excerpt":"\u003cp\u003eSkyWalking PHP 0.3.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-php-0-3-0/","title":"Release Apache SkyWalking PHP 0.3.0"},{"body":"SkyWalking Java Agent 8.14.0 is released. Go to downloads page to find release tars. Changes by Version\n8.14.0 Polish test framework to support arm64/v8 platforms Fix wrong config name plugin.toolkit.use_qualified_name_as_operation_name, and system variable name SW_PLUGIN_TOOLKIT_USE_QUALIFIED_NAME_AS_OPERATION_NAME:false. They were toolit. Rename JDBI to JDBC Support collecting dubbo thread pool metrics Bump up byte-buddy to 1.12.19 Upgrade agent test tools [Breaking Change] Compatible with 3.x and 4.x RabbitMQ Client, rename rabbitmq-5.x-plugin to rabbitmq-plugin Polish JDBC plugins to make DBType accurate Report the agent version to OAP as an instance attribute Polish jedis-4.x-plugin to change command to lowercase, which is consistent with jedis-2.x-3.x-plugin Add micronauthttpclient,micronauthttpserver,memcached,ehcache,guavacache,jedis,redisson plugin config properties to agent.config Add Micrometer Observation support Add tags mq.message.keys and mq.message.tags for RocketMQ producer span Clean the trace context which injected into Pulsar MessageImpl after the instance recycled Fix In the higher version of mysql-connector-java 8x, there is an error in the value of db.instance. Add support for KafkaClients 3.x. Support to customize the collect period of JVM relative metrics. Upgrade netty-codec-http2 to 4.1.86.Final. Put Agent-Version property reading in the premain stage to avoid deadlock when using jarsigner. Add a config agent.enable(default: true) to support disabling the agent through system property -Dskywalking.agent.disable=false or system environment variable setting SW_AGENT_ENABLE=false. Enhance redisson plugin to adopt uniform tags. Documentation Update Plugin-test.md, support string operators start with and end with Polish agent configurations doc to fix type error All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking Java Agent 8.14.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nChanges by …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-java-agent-8-14-0/","title":"Release Apache SkyWalking Java Agent 8.14.0"},{"body":"\nBackground Apache SkyWalking is an open-source Application Performance Management system that helps users collect and aggregate logs, traces, metrics, and events for display on a UI. In the previous article, we introduced how to use Apache SkyWalking Rover to analyze the network performance issue in the service mesh environment. However, in business scenarios, users often rely on mature layer 7 protocols, such as HTTP, for interactions between systems. In this article, we will discuss how to use eBPF techniques to analyze performance bottlenecks of layer 7 protocols and how to enhance the tracing system using network sampling.\nThis article will show how to use Apache SkyWalking with eBPF to enhance metrics and traces in HTTP observability.\nHTTP Protocol Analysis HTTP is one of the most common Layer 7 protocols and is usually used to provide services to external parties and for inter-system communication. In the following sections, we will show how to identify and analyze HTTP/1.x protocols.\nProtocol Identification In HTTP/1.x, the client and server communicate through a single file descriptor (FD) on each side. Figure 1 shows the process of communication involving the following steps:\nConnect/accept: The client establishes a connection with the HTTP server, or the server accepts a connection from the client. Read/write (multiple times): The client or server reads and writes HTTPS requests and responses. A single request-response pair occurs within the same connection on each side. Close: The client and server close the connection. To obtain HTTP content, it’s necessary to read it from the second step of this process. As defined in the RFC, the content is contained within the data of the Layer 4 protocol and can be obtained by parsing the data. The request and response pair can be correlated because they both occur within the same connection on each side.\nFigure 1: HTTP communication timeline.\nHTTP Pipeline HTTP pipelining is a feature of HTTP/1.1 that enables multiple HTTP requests to be sent over a single TCP connection without waiting for the corresponding responses. This feature is important because it ensures that the order of the responses on the server side matches the order of the requests.\nFigure 2 illustrates how this works. Consider the following scenario: an HTTP client sends multiple requests to a server, and the server responds by sending the HTTP responses in the same order as the requests. This means that the first request sent by the client will receive the first response from the server, the second request will receive the second response, and so on.\nWhen designing HTTP parsing, we should follow this principle by adding request data to a list and removing the first item when parsing a response. This ensures that the responses are processed in the correct order.\nFigure 2: HTTP/1.1 pipeline.\nMetrics Based on the identification of the HTTP content and process topology diagram mentioned in the previous article, we can combine these two to generate process-to-process metrics data.\nFigure 3 shows the metrics that currently support the analysis between the two processes. Based on the HTTP request and response data, we can analyze the following data:\nMetrics Name Type Unit Description Request CPM(Call Per Minute) Counter count The HTTP request count Response Status CPM(Call Per Minute) Counter count The count of per HTTP response status code Request Package Size Counter/Histogram Byte The request package size Response Package Size Counter/Histogram Byte The response package size Client Duration Counter/Histogram Millisecond The duration of single HTTP response on the client side Server Duration Counter/Histogram Millisecond The duration of single HTTP response on the server side Figure 3: Process-to-process metrics.\nHTTP and Trace During the HTTP process, if we unpack the HTTP requests and responses from raw data, we can use this data to correlate with the existing tracing system.\nTrace Context Identification In order to track the flow of requests between multiple services, the trace system usually creates a trace context when a request enters a service and passes it along to other services during the request-response process. For example, when an HTTP request is sent to another server, the trace context is included in the request header.\nFigure 4 displays the raw content of an HTTP request intercepted by Wireshark. The trace context information generated by the Zipkin Tracing system can be identified by the “X-B3” prefix in the header. By using eBPF to intercept the trace context in the HTTP header, we can connect the current request with the trace system.\nFigure 4: View of HTTP headers in Wireshark.\nTrace Event We have added the concept of an event to traces. An event can be attached to a span and consists of start and end times, tags, and summaries, allowing us to attach any desired information to the Trace.\nWhen performing eBPF network profiling, two events can be generated based on the request-response data. Figure 5 illustrates what happens when a service performs an HTTP request with profiling. The trace system generates trace context information and sends it in the request. When the service executes in the kernel, we can generate an event for the corresponding trace span by interacting with the request-response data and execution time in the kernel space.\nPreviously, we could only observe the execution status in the user space. However, by combining traces and eBPF technologies, we can now also get more information about the current trace in the kernel space, which would impact less performance for the target service if we do similar things in the tracing SDK and agent.\nFigure 5: Logical view of profiling an HTTP request and response.\nSampling To ensure efficient data storage and minimize unnecessary data sampling, we use a sampling mechanism for traces in our system. This mechanism triggers sampling only when certain conditions are met. We also provide a list of the top N traces, which allows users to quickly access the relevant request information for a specific trace.\nTo help users easily identify and analyze relevant events, we offer three different sampling rules:\nSlow Traces: Sampling is triggered when the response time for a request exceeds a specified threshold. Response Status [400, 500): Sampling is triggered when the response status code is greater than or equal to 400 and less than 500. Response Status [500, 600): Sampling is triggered when the response status code is greater than or equal to 500 and less than 600. In addition, we recognize that not all request or response raw data may be necessary for analysis. For example, users may be more interested in requesting data when trying to identify performance issues, while they may be more interested in response data when troubleshooting errors. As such, we also provide configuration options for request or response events to allow users to specify which type of data they would like to sample.\nProfiling in a Service Mesh The SkyWalking and SkyWalking Rover projects have already implemented the HTTP protocol analyze and trace associations. How do they perform when running in a service mesh environment?\nDeployment Figure 6 demonstrates the deployment of SkyWalking and SkyWalking Rover in a service mesh environment. SkyWalking Rover is deployed as a DaemonSet on each machine where a service is located and communicates with the SkyWalking backend cluster. It automatically recognizes the services on the machine and reports metadata information to the SkyWalking backend cluster. When a new network profiling task arises, SkyWalking Rover senses the task and analyzes the designated processes, collecting and aggregating network data before ultimately reporting it back to the SkyWalking backend service.\nFigure 6: SkyWalking rover deployment topology in a service mesh.\nTracing Systems Starting from version 9.3.0, the SkyWalking backend fully supports all functions in the Zipkin server. Therefore, the SkyWalking backend can collect traces from both the SkyWalking and Zipkin protocols. Similarly, SkyWalking Rover can identify and analyze trace context in both the SkyWalking and Zipkin trace systems. In the following two sections, network analysis results will be displayed in the SkyWalking and Zipkin UI respectively.\nSkyWalking When SkyWalking performs network profiling, similar to the TCP metrics in the previous article, the SkyWalking UI will first display the topology between processes. When you open the dashboard of the line representing the traffic metrics between processes, you can see the metrics of HTTP traffic from the “HTTP/1.x” tab and the sampled HTTP requests with tracing in the “HTTP Requests” tab.\nAs shown in Figure 7, there are three lists in the tab, each corresponding to a condition in the event sampling rules. Each list displays the traces that meet the pre-specified conditions. When you click on an item in the trace list, you can view the complete trace.\nFigure 7: Sampled HTTP requests within tracing context.\nWhen you click on an item in the trace list, you can quickly view the specified trace. In Figure 8, we can see that in the current service-related span, there is a tag with a number indicating how many HTTP events are related to that trace span.\nSince we are in a service mesh environment, each service involves interacting with Envoy. Therefore, the current span includes Envoy’s request and response information. Additionally, since the current service has both incoming and outgoing requests, there are events in the corresponding span.\nFigure 8: Events in the trace detail.\nWhen the span is clicked, the details of the span will be displayed. If there are events in the current span, the relevant event information will be displayed on a time axis. As shown in Figure 9, there are a total of 6 related events in the current Span. Each event represents a data sample of an HTTP request/response. One of the events spans multiple time ranges, indicating a longer system call time. It may be due to a blocked system call, depending on the implementation details of the HTTP request in different languages. This can also help us query the possible causes of errors.\nFigure 9: Events in one trace span.\nFinally, we can click on a specific event to see its complete information. As shown in Figure 10, it displays the sampling information of a request, including the SkyWalking trace context protocol contained in the request header from the HTTP raw data. The raw request data allows you to quickly re-request the request to solve any issues.\nFigure 10: The detail of the event.\nZipkin Zipkin is one of the most widely used distributed tracing systems in the world. SkyWalking can function as an alternative server to provide advanced features for Zipkin users. Here, we use this way to bring the feature into the Zipkin ecosystem out-of-box. The new events would also be treated as a kind of Zipkin’s tags and annotations.\nTo add events to a Zipkin span, we need to do the following:\nSplit the start and end times of each event into two annotations with a canonical name. Add the sampled HTTP raw data from the event to the Zipkin span tags, using the same event name for corresponding purposes. Figures 11 and 12 show annotations and tags in the same span. In these figures, we can see that the span includes at least two events with the same event name and sequence suffix (e.g., “Start/Finished HTTP Request/Response Sampling-x” in the figure). Both events have separate timestamps to represent their relative times within the span. In the tags, the data content of the corresponding event is represented by the event name and sequence number, respectively.\nFigure 11: Event timestamp in the Zipkin span annotation.\nFigure 12: Event raw data in the Zipkin span tag.\nDemo In this section, we demonstrate how to perform network profiling in a service mesh and complete metrics collection and HTTP raw data sampling. To follow along, you will need a running Kubernetes environment.\nDeploy SkyWalking Showcase SkyWalking Showcase contains a complete set of example services and can be monitored using SkyWalking. For more information, please check the official documentation.\nIn this demo, we only deploy service, the latest released SkyWalking OAP, and UI.\nexport SW_OAP_IMAGE=apache/skywalking-oap-server:9.3.0 export SW_UI_IMAGE=apache/skywalking-ui:9.3.0 export SW_ROVER_IMAGE=apache/skywalking-rover:0.4.0 export FEATURE_FLAGS=mesh-with-agent,single-node,elasticsearch,rover make deploy.kubernetes After deployment is complete, please run the following script to open SkyWalking UI: http://localhost:8080/.\nkubectl port-forward svc/ui 8080:8080 --namespace default Start Network Profiling Task Currently, we can select the specific instances that we wish to monitor by clicking the Data Plane item in the Service Mesh panel and the Service item in the Kubernetes panel.\nIn figure 13, we have selected an instance with a list of tasks in the network profiling tab.\nFigure 13: Network Profiling tab in the Data Plane.\nWhen we click the Start button, as shown in Figure 14, we need to specify the sampling rules for the profiling task. The sampling rules consist of one or more rules, each of which is distinguished by a different URI regular expression. When the HTTP request URI matches the regular expression, the rule is used. If the URI regular expression is empty, the default rule is used. Using multiple rules can help us make different sampling configurations for different requests.\nEach rule has three parameters to determine if sampling is needed:\nMinimal Request Duration (ms): requests with a response time exceeding the specified time will be sampled. Sampling response status code between 400 and 499: all status codes in the range [400-499) will be sampled. Sampling response status code between 500 and 599: all status codes in the range [500-599) will be sampled. Once the sampling configuration is complete, we can create the task.\nFigure 14: Create network profiling task page.\nDone! After a few seconds, you will see the process topology appear on the right side of the page.\nWhen you click on the line between processes, you can view the data between the two processes, which is divided into three tabs:\nTCP: displays TCP-related metrics. HTTP/1.x: displays metrics in the HTTP 1 protocol. HTTP Requests: displays the analyzed request and saves it to a list according to the sampling rule. Figure 16: TCP metrics in a network profiling task.\nFigure 17: HTTP/1.x metrics in a network profiling task.\nFigure 18: HTTP sampled requests in a network profiling task.\nConclusion In this article, we detailed the overview of how to analyze the Layer 7 HTTP/1.x protocol in network analysis, and how to associate it with existing trace systems. This allows us to extend the scope of data we can observe from just user space to also include kernel-space data.\nIn the future, we will delve further into the analysis of kernel data, such as collecting information on TCP packet size, transmission frequency, network card, and help on enhancing distributed tracing from another perspective.\nAdditional Resources SkyWalking Github Repo › SkyWalking Rover Github Repo › SkyWalking Rover Documentation › Diagnose Service Mesh Network Performance with eBPF blog post \u0026gt; SkyWalking Profiling Documentation \u0026gt; SkyWalking Trace Context Propagation \u0026gt; Zipkin Trace Context Propagation \u0026gt; RFC - Hypertext Transfer Protocol – HTTP/1.1 \u0026gt; ","excerpt":"\u003cp\u003e\u003cimg src=\"banner.jpg\" alt=\"banner\"\u003e\u003c/p\u003e\n\u003ch2 id=\"background\"\u003eBackground\u003c/h2\u003e\n\u003cp\u003eApache SkyWalking is an open-source Application Performance Management system that helps …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/ebpf-enhanced-http-observability-l7-metrics-and-tracing/","title":"eBPF enhanced HTTP observability - L7 metrics and tracing"},{"body":"\n背景 Apache SkyWalking 是一个开源应用性能管理系统，帮助用户收集和聚合日志、追踪、指标和事件，并在 UI 上显示。在上一篇文章中，我们介绍了如何使用 Apache SkyWalking Rover 分析服务网格环境中的网络性能问题。但是，在商业场景中，用户通常依靠成熟的第 7 层协议（如 HTTP）来进行系统之间的交互。在本文中，我们将讨论如何使用 eBPF 技术来分析第 7 层协议的性能瓶颈，以及如何使用网络采样来增强追踪系统。\n本文将演示如何使用 Apache SkyWalking 与 eBPF 来增强 HTTP 可观察性中的指标和追踪。\nHTTP 协议分析 HTTP 是最常用的 7 层协议之一，通常用于为外部方提供服务和进行系统间通信。在下面的章节中，我们将展示如何识别和分析 HTTP/1.x 协议。\n协议识别 在 HTTP/1.x 中，客户端和服务器通过两端的单个文件描述符（File Descriptor）进行通信。图 1 显示了涉及以下步骤的通信过程：\nConnect/Accept：客户端与 HTTP 服务器建立连接，或者服务器接受客户端的连接。 Read/Write（多次）：客户端或服务器读取和写入 HTTPS 请求和响应。单个请求 - 响应对在每边的同一连接内发生。 Close：客户端和服务器关闭连接。 为了获取 HTTP 内容，必须从此过程的第二步读取它。根据 RFC 定义，内容包含在 4 层协议的数据中，可以通过解析数据来获取。请求和响应对可以相关联，因为它们都在两端的同一连接内发生。\n图 1：HTTP 通信时间线。\nHTTP 管线化 HTTP 管线化（Pipelining）是 HTTP/1.1 的一个特性，允许在等待对应的响应的情况下在单个 TCP 连接上发送多个 HTTP 请求。这个特性很重要，因为它确保了服务器端的响应顺序必须与请求的顺序匹配。\n图 2 说明了这是如何工作的，考虑以下情况：HTTP 客户端向服务器发送多个请求，服务器通过按照请求的顺序发送 HTTP 响应来响应。这意味着客户端发送的第一个请求将收到服务器的第一个响应，第二个请求将收到第二个响应，以此类推。\n在设计 HTTP 解析时，我们应该遵循这个原则，将请求数据添加到列表中，并在解析响应时删除第一个项目。这可以确保响应按正确的顺序处理。\n图 2： HTTP/1.1 管道。\n指标 根据前文提到的 HTTP 内容和流程拓扑图的识别，我们可以将这两者结合起来生成进程间的指标数据。\n图 3 显示了目前支持两个进程间分析的指标。基于 HTTP 请求和响应数据，可以分析以下数据：\n指标名称 类型 单位 描述 请求 CPM（Call Per Minute） 计数器 计数 HTTP 请求计数 响应状态 CPM (Call Per Minute) 计数器 计数 每个 HTTP 响应状态码的计数 请求包大小 计数器 / 直方图 字节 请求包大小 响应包大小 计数器 / 直方图 字节 响应包大小 客户端持续时间 计数器 / 直方图 毫秒 客户端单个 HTTP 响应的持续时间 服务器持续时间 计数器 / 直方图 毫秒 服务器端单个 HTTP 响应的持续时间 图 3：进程到进程指标。\nHTTP 和追踪 在 HTTP 过程中，如果我们能够从原始数据中解包 HTTP 请求和响应，就可以使用这些数据与现有的追踪系统进行关联。\n追踪上下文标识 为了追踪多个服务之间的请求流，追踪系统通常在请求进入服务时创建追踪上下文，并在请求 - 响应过程中将其传递给其他服务。例如，当 HTTP 请求发送到另一个服务器时，追踪上下文包含在请求头中。\n图 4 显示了 Wireshark 拦截的 HTTP 请求的原始内容。由 Zipkin Tracing 系统生成的追踪上下文信息可以通过头中的 “X-B3” 前缀进行标识。通过使用 eBPF 拦截 HTTP 头中的追踪上下文，可以将当前请求与追踪系统连接起来。\n图 4：Wireshark 中的 HTTP Header 视图。\nTrace 事件 我们已经将事件这个概念加入了追踪中。事件可以附加到跨度上，并包含起始和结束时间、标签和摘要，允许我们将任何所需的信息附加到追踪中。\n在执行 eBPF 网络分析时，可以根据请求 - 响应数据生成两个事件。图 5 说明了在带分析的情况下执行 HTTP 请求时发生的情况。追踪系统生成追踪上下文信息并将其发送到请求中。当服务在内核中执行时，我们可以通过与内核空间中的请求 - 响应数据和执行时间交互，为相应的追踪跨度生成事件。\n以前，我们只能观察用户空间的执行状态。现在，通过结合追踪和 eBPF 技术，我们还可以在内核空间获取更多关于当前追踪的信息，如果我们在追踪 SDK 和代理中执行类似的操作，将对目标服务的性能产生较小的影响。\n图 5：分析 HTTP 请求和响应的逻辑视图。\n抽样 该机制仅在满足特定条件时触发抽样。我们还提供了前 N 条追踪的列表，允许用户快速访问特定追踪的相关请求信息。为了帮助用户轻松识别和分析相关事件，我们提供了三种不同的抽样规则：\n慢速追踪：当请求的响应时间超过指定阈值时触发抽样。 响应状态 [400,500)：当响应状态代码大于或等于 400 且小于 500 时触发抽样。 响应状态 [500,600)：当响应状态代码大于或等于 500 且小于 600 时触发抽样。 此外，我们认识到分析时可能并不需要所有请求或响应的原始数据。例如，当试图识别性能问题时，用户可能更感兴趣于请求数据，而在解决错误时，他们可能更感兴趣于响应数据。因此，我们还提供了请求或响应事件的配置选项，允许用户指定要抽样的数据类型。\n服务网格中的分析 SkyWalking Rover 项目已经实现了 HTTP 协议的分析和追踪关联。当在服务网格环境中运行时它们的表现如何？\n部署 图 6 演示了 SkyWalking 和 SkyWalking Rover 在服务网格环境中的部署方式。SkyWalking Rover 作为一个 DaemonSet 部署在每台服务所在的机器上，并与 SkyWalking 后端集群通信。它会自动识别机器上的服务并向 SkyWalking 后端集群报告元数据信息。当出现新的网络分析任务时，SkyWalking Rover 会感知该任务并对指定的进程进行分析，在最终将数据报告回 SkyWalking 后端服务之前，收集和聚合网络数据。\n图 6：服务网格中的 SkyWalking rover 部署拓扑。\n追踪系统 从版本 9.3.0 开始，SkyWalking 后端完全支持 Zipkin 服务器中的所有功能。因此，SkyWalking 后端可以收集来自 SkyWalking 和 Zipkin 协议的追踪。同样，SkyWalking Rover 可以在 SkyWalking 和 Zipkin 追踪系统中识别和分析追踪上下文。在接下来的两节中，网络分析结果将分别在 SkyWalking 和 Zipkin UI 中显示。\nSkyWalking 当 SkyWalking 执行网络分析时，与前文中的 TCP 指标类似，SkyWalking UI 会首先显示进程间的拓扑图。当打开代表进程间流量指标的线的仪表板时，您可以在 “HTTP/1.x” 选项卡中看到 HTTP 流量的指标，并在 “HTTP Requests” 选项卡中看到带追踪的抽样的 HTTP 请求。\n如图 7 所示，选项卡中有三个列表，每个列表对应事件抽样规则中的一个条件。每个列表显示符合预先规定条件的追踪。当您单击追踪列表中的一个项目时，就可以查看完整的追踪。\n图 7：Tracing 上下文中的采样 HTTP 请求。\n当您单击追踪列表中的一个项目时，就可以快速查看指定的追踪。在图 8 中，我们可以看到在当前的服务相关的跨度中，有一个带有数字的标签，表示与该追踪跨度相关的 HTTP 事件数。\n由于我们在服务网格环境中，每个服务都涉及与 Envoy 交互。因此，当前的跨度包括 Envoy 的请求和响应信息。此外，由于当前的服务有传入和传出的请求，因此相应的跨度中有事件。\n图 8：Tracing 详细信息中的事件。\n当单击跨度时，将显示跨度的详细信息。如果当前跨度中有事件，则相关事件信息将在时间轴上显示。如图 9 所示，当前跨度中一共有 6 个相关事件。每个事件代表一个 HTTP 请求 / 响应的数据样本。其中一个事件跨越多个时间范围，表示较长的系统调用时间。这可能是由于系统调用被阻塞，具体取决于不同语言中的 HTTP 请求的实现细节。这也可以帮助我们查询错误的可能原因。\n图 9：一个 Tracing 范围内的事件。\n最后，我们可以单击特定的事件查看它的完整信息。如图 10 所示，它显示了一个请求的抽样信息，包括从 HTTP 原始数据中的请求头中包含的 SkyWalking 追踪上下文协议。原始请求数据允许您快速重新请求以解决任何问题。\n图 10：事件的详细信息。\nZipkin Zipkin 是世界上广泛使用的分布式追踪系统。SkyWalking 可以作为替代服务器，提供高级功能。在这里，我们使用这种方式将功能无缝集成到 Zipkin 生态系统中。新事件也将被视为 Zipkin 的标签和注释的一种。\n为 Zipkin 跨度添加事件，需要执行以下操作：\n将每个事件的开始时间和结束时间分别拆分为两个具有规范名称的注释。 将抽样的 HTTP 原始数据从事件添加到 Zipkin 跨度标签中，使用相同的事件名称用于相应的目的。 图 11 和图 12 显示了同一跨度中的注释和标签。在这些图中，我们可以看到跨度包含至少两个具有相同事件名称和序列后缀的事件（例如，图中的 “Start/Finished HTTP Request/Response Sampling-x”）。这两个事件均具有单独的时间戳，用于表示其在跨度内的相对时间。在标签中，对应事件的数据内容分别由事件名称和序列号表示。\n图 11：Zipkin span 注释中的事件时间戳。\n图 12：Zipkin span 标签中的事件原始数据。\n演示 在本节中，我们将演示如何在服务网格中执行网络分析，并完成指标收集和 HTTP 原始数据抽样。要进行操作，您需要一个运行中的 Kubernetes 环境。\n部署 SkyWalking Showcase SkyWalking Showcase 包含一套完整的示例服务，可以使用 SkyWalking 进行监控。有关详细信息，请参阅官方文档。\n在本演示中，我们只部署了服务、最新发布的 SkyWalking OAP 和 UI。\nexport SW_OAP_IMAGE=apache/skywalking-oap-server:9.3.0 export SW_UI_IMAGE=apache/skywalking-ui:9.3.0 export SW_ROVER_IMAGE=apache/skywalking-rover:0.4.0 export FEATURE_FLAGS=mesh-with-agent,single-node,elasticsearch,rover make deploy.kubernetes 部署完成后，运行下面的脚本启动 SkyWalking UI：http://localhost:8080/。\nkubectl port-forward svc/ui 8080:8080 --namespace default 启动网络分析任务 目前，我们可以通过单击服务网格面板中的 Data Plane 项和 Kubernetes 面板中的 Service 项来选择要监视的特定实例。\n在图 13 中，我们已在网络分析选项卡中选择了一个具有任务列表的实例。\n图 13：数据平面中的网络分析选项卡。\n当我们单击 “开始” 按钮时，如图 14 所示，我们需要为分析任务指定抽样规则。抽样规则由一个或多个规则组成，每个规则都由不同的 URI 正则表达式区分。当 HTTP 请求的 URI 与正则表达式匹配时，将使用该规则。如果 URI 正则表达式为空，则使用默认规则。使用多个规则可以帮助我们为不同的请求配置不同的抽样配置。\n每个规则都有三个参数来确定是否需要抽样：\n最小请求持续时间（毫秒）：响应时间超过指定时间的请求将被抽样。 在 400 和 499 之间的抽样响应状态代码：范围 [400-499) 中的所有状态代码将被抽样。 在 500 和 599 之间的抽样响应状态代码：范围 [500-599) 中的所有状态码将被抽样。 抽样配置完成后，我们就可以创建任务了。\n图 14：创建网络分析任务页面。\n完成 几秒钟后，你会看到页面的右侧出现进程拓扑结构。\n图 15：网络分析任务中的流程拓扑。\n当您单击进程之间的线时，您可以查看两个过程之间的数据，它被分为三个选项卡：\nTCP：显示与 TCP 相关的指标。 HTTP/1.x：显示 HTTP 1 协议中的指标。 HTTP 请求：显示已分析的请求，并根据抽样规则保存到列表中。 图 16：网络分析任务中的 TCP 指标。\n图 17：网络分析任务中的 HTTP/1.x 指标。\n图 18：网络分析任务中的 HTTP 采样请求。\n总结 在本文中，我们详细介绍了如何在网络分析中分析 7 层 HTTP/1.x 协议，以及如何将其与现有追踪系统相关联。这使我们能够将我们能够观察到的数据从用户空间扩展到内核空间数据。\n在未来，我们将进一步探究内核数据的分析，例如收集 TCP 包大小、传输频率、网卡等信息，并从另一个角度提升分布式追踪。\n其他资源 SkyWalking Github Repo › SkyWalking Rover Github Repo › SkyWalking Rover Documentation › Diagnose Service Mesh Network Performance with eBPF blog post \u0026gt; SkyWalking Profiling Documentation \u0026gt; SkyWalking Trace Context Propagation \u0026gt; Zipkin Trace Context Propagation \u0026gt; RFC - Hypertext Transfer Protocol – HTTP/1.1 \u0026gt; ","excerpt":"\u003cp\u003e\u003cimg src=\"banner.jpg\" alt=\"banner\"\u003e\u003c/p\u003e\n\u003ch2 id=\"背景\"\u003e背景\u003c/h2\u003e\n\u003cp\u003eApache SkyWalking 是一个开源应用性能管理系统，帮助用户收集和聚合日志、追踪、指标和事件，并在 UI 上显示。在\u003ca href=\"/zh/diagnose-service-mesh-network-performance-with-ebpf/\"\u003e上一篇文章\u003c/a\u003e中，我们介绍了如何使用 Apache …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/ebpf-enhanced-http-observability-l7-metrics-and-tracing/","title":"使用 eBPF 提升 HTTP 可观测性 - L7 指标和追踪"},{"body":"SkyWalking Rust 0.5.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed Add management support. by @jmjoy in https://github.com/apache/skywalking-rust/pull/48 Add missing_docs lint and supply documents. by @jmjoy in https://github.com/apache/skywalking-rust/pull/49 Add authentication and custom intercept support. by @jmjoy in https://github.com/apache/skywalking-rust/pull/50 Bump to 0.5.0. by @jmjoy in https://github.com/apache/skywalking-rust/pull/51 ","excerpt":"\u003cp\u003eSkyWalking Rust 0.5.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-rust-0-5-0/","title":"Release Apache SkyWalking Rust 0.5.0"},{"body":"SkyWalking Satellite 1.1.0 is released. Go to downloads page to find release tars.\nFeatures Support transmit the OpenTelemetry Metrics protocol. Upgrade to GO 1.18. Add Docker images for arm64 architecture. Support transmit Span Attached Event protocol data. Support dotnet CLRMetric forward. Bug Fixes Fix the missing return data when receive metrics in batch mode. Fix CVE-2022-21698, CVE-2022-27664. Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Satellite 1.1.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-satellite-1-1-0/","title":"Release Apache SkyWalking Satellite 1.1.0"},{"body":"Apache SkyWalking is an open-source APM for a distributed system, Apache Software Foundation top-level project.\nOn Jan. 3rd, 2023, we received reports about Aliyun Trace Analysis Service. It provides a cloud service compatible with SkyWalking trace APIs and agents.\nOn their product page, there is a best-practice document describing about their service is not SkyWalking OAP, but can work with SkyWalking agents to support SkyWalking\u0026rsquo;s In-Process(Trace) Profiling.\nBUT, they copied the whole page of SkyWalking\u0026rsquo;s profiling UI, including page layout, words, and profiling task setup. The only difference is the color schemes.\nSkyWalking UI Aliyun Trace Analysis UI on their document page The UI visualization is a part of the copyright. Aliyun declared their backend is NOT a re-distribution of SkyWalking repeatedly on their website, and they never mentioned this page is actually copied from upstream.\nThis is a LICENSE issue, violating SkyWalking\u0026rsquo;s copyright and Apache 2.0 License. They don\u0026rsquo;t respect Apache Software Foundation and Apache SkyWalking\u0026rsquo;s IP and Branding.\n","excerpt":"\u003cp\u003e\u003ca href=\"https://skywalking.apache.org\"\u003eApache SkyWalking\u003c/a\u003e is an open-source APM for a distributed system, Apache Software Foundation …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2023-01-03-aliyun-copy-page/","title":"[License Issue] Aliyun(阿里云)'s trace analysis service copied SkyWalking's trace profiling page."},{"body":"SkyWalking Rover 0.4.0 is released. Go to downloads page to find release tars.\nFeatures Enhancing the render context for the Kubernetes process. Simplify the logic of network protocol analysis. Upgrade Go library to 1.18, eBPF library to 0.9.3. Make the Profiling module compatible with more Linux systems. Support monitor HTTP/1.x in the NETWORK profiling. Bug Fixes Documentation Adding support version of Linux documentation. Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Rover 0.4.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eEnhancing …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-rover-0-4-0/","title":"Release Apache SkyWalking Rover 0.4.0"},{"body":"\nObservability for modern distributed applications work is critical for understanding how they behave under a variety of conditions and for troubleshooting and resolving issues when they arise. Traces, metrics, and logs are regarded as fundamental parts of the observability stack. Traces are the footprints of distributed system executions, meanwhile, metrics measure system performance with numbers in the timeline. Essentially, they measure the performance from two dimensions. Being able to quickly visualize the connection between traces and corresponding metrics makes it possible to quickly diagnose which process flows are correlated to potentially pathological behavior. This powerful new capability is now available in SkyWalking 9.3.0.\nThe SkyWalking project started only with tracing, with a focus on 100% sampling-based metrics and topology analysis since 2018. When users face anomaly trends of time-series metrics, like a peak on the line chart, or histogram shows a larger gap between p95 and p95, the immediate question is, why is this happening? One of SkyWalking\u0026rsquo;s latest features, the trace-metric association, makes it much easier to answer that question and to address the root cause.\nHow Are Metrics Generated? SkyWalking provides three ways to calculate metrics:\nMetrics built from trace spans, depending on the span’s layer, kind, and tags. Metrics extracted from logs—a kind of keyword and tags-based metrics extraction. Metrics reported from mature and mainstream metrics/meter systems, such as OpenTelemetry, Prometheus, and Zabbix. Tracing tracks the processes of requests between an application\u0026rsquo;s services. Most systems that generate traffic and performance-related metrics also generate tracing data, either from server-side trace-based aggregations or through client SDKs.\nUse SkyWalking to Reduce the Traditional Cost of Trace Indexing Tracing data and visualization are critical troubleshooting tools for both developers and operators alike because of how helpful they are in locating issue boundaries. But, because it has traditionally been difficult to find associations between metrics and traces, teams have added increasingly more tags into the spans, and search through various combinations. This trend of increased instrumentation and searching has required increased infrastructure investment to support this kind of search. SkyWalking\u0026rsquo;s metrics and tracing association capabilities can help reduce the cost of indexing and searching that data.\nFind the Associated Trace When looking for association between metrics and traces, the kind of metrics we\u0026rsquo;re dealing with determines their relationships to traces. Let’s review the standard request rate, error, and duration (RED) metrics to see how it works.\nSuccess Rate Metrics The success rate is determined by the return code, RPC response code, or exceptions of the process. When the success rate decreases, looking for errors in the traces of this service or pod are the first place to look to find clues.\nFigure 1: The success rate graph from SkyWalking\u0026rsquo;s 9.3.0 dashboard with the option to view related traces at a particular time.\nDrilling down from the peak of the success rate, SkyWalking lists all traces and their error status that were collected in this particular minute (Figure 2):\nFigure 2: SkyWalking shows related traces with an error status.\nRequests to /test can be located from the trace, and the span’s tag indicates a 404 response code of the HTTP request.\nFigure 3: A detail view of a request to http://frontend/test showing that the URI doesn\u0026rsquo;t exist.\nBy looking at the trace data, it becomes immediately clear that the drop in success rate is caused by requests to a nonexistent URI.\nAverage Response Time The average response time metric provides a general overview of service performance. When average response time is unstable, this usually means that the system is facing serious performance impacts.\nFigure 4: SkyWalking\u0026rsquo;s query UI for searching for related traces showing traces for requests that exceed a particular duration threshold.\nWhen you drill down from this metric, this query condition (Figure 4) will reveal the slowest traces of the service in this specific minute. Notice, at least 168ms is added as a condition automatically, to avoid scanning a large number of rows in the Database.\nApdex Apdex—the Application Performance Index—is a measure of response time based against a set threshold. It measures the ratio of satisfactory response times to unsatisfactory response times (Figure 5). The response time is measured from an asset request to completed delivery back to the requestor.\nFigure 5: The Apdex formula\nA user defines a response time tolerating threshold T. All responses handled in T or less time satisfy the user.\nFor example, if T is 1.2 seconds and a response completes in 0.5 seconds, then the user is satisfied. All responses greater than 1.2 seconds dissatisfy the user. Responses greater than 4.8 seconds frustrate the user.\nWhen the Apdex score decreases, we need to find related traces from two perspectives: slow traces and error status traces. SkyWalking\u0026rsquo;s new related tracing features offers a quick way to view both (Figure 6) directly from the Apdex graph.\nFigure 6: Show slow trace and error status traces from the Apdex graph\nService Response Time Percentile MetricThe percentile graph (Figure 7) provides p50, p75, p90, p95, and p99 latency ranks to measure the long-tail issues of service performance.\nFigure 7: The service response time percentile graph helps to highlight long-tail issues of service performance.\nThis percentile graph shows a typical long-tail issue. P99 latency is four times slower than the P95. When we use the association, we see the traces with latency between P95 - P99 and P99 - Infinity.\nThe traces of requests causing this kind of long-tail phenomena are automatically listing from there.\nFigure 8: Query parameters to search for traces based on latency.\nAre More Associations Available? SkyWalking provides more than just associations between between traces and metrics to help you find possible causal relationships and to avoid looking for the proverbial needle in a haystack.\nCurrently, SkyWalking 9.3.0 offers two more associations: metric-to-metric associations and event-to-metric associations.\nMetric-to-metric Associations There are dozens of metrics on the dashboard—which is great for getting a complete picture of application behavior. During a typical performance issue, the peaks of multiple metrics are affected simultaneously. But, trying to correlate peaks across all of these graphs can be difficult\u0026hellip;\nNow in SkyWalking 9.3.0, when you click the peak of one graph, the pop-out box lets you see associated metrics.\nFigure 9: SkyWalking\u0026rsquo;s option to view associated metrics.\nWhen you choose that option, all associated metrics graphs will show axis pointers (the dotted vertical lines) in all associated graphs like in Figure 10. This makes it easier to correlate the peaks in different graphs with each other. Often, these correlated peaks with have the same root cause.\nFigure 10: Axis pointers (vertical dotted lines) show associations between peaks across multiple metrics graphs.\nEvent-to-Metric Associations SkyWalking provides the event concept to associate possible service performance impacted by the infrastructure, such as new deployment even from k8s. Or, the anomaly had been detected by alerting or integrated AIOps engine.\nThe event to metrics association is also automatically, it could cover the time range of the event on the metric graphs(blue areas). If the area of event and peaks are matched, most likely this event covered this anomaly.\nFigure 11: SkyWalking\u0026rsquo;s event to metric association view.\nSkyWalking Makes it Easier and Faster to Find Root Causes SkyWalking now makes it easy to find associations between metrics, events, and traces, ultimately making it possible to identify root causes and fix problems fast. The associations we\u0026rsquo;ve discussed in this article are available out-of-box in the SkyWalking 9.3.0 release.\nFigure 12: Just click on the dots to see related traces and metrics associations.\nClick the dots on any metric graph, and you will see a View Related Traces item pop-out if this metric has logical mapping traces.\nConclusion In this blog, we took a look at the newly-added association feature between metrics and traces. With this new visualization, it\u0026rsquo;s now much easier to find key traces to identify root cause of issues.Associations in SkyWalking can go even deeper. Associations from metrics to traces is not the end of diagnosing system bottleneck. In the next post, we will introduce an eBPF powered trace enhancement where you’ll be able to see HTTP request and response details associated with tracing spans from network profiling. Stay tuned.\n","excerpt":"\u003cp\u003e\u003cimg src=\"banner.jpg\" alt=\"Banner\"\u003e\u003c/p\u003e\n\u003cp\u003eObservability for modern distributed applications work is critical for understanding how they …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/boost-root-cause-analysis-quickly-with-skywalking-new-trace-metrics-association-feature/","title":"Boost Root Cause Analysis Quickly With SkyWalking’s New Trace-Metrics Association Feature"},{"body":"\n现代分布式应用程序工作的可观测性对于了解它们在各种条件下的行为方式以及在出现问题时进行故障排除和解决至关重要。追踪、指标和日志被视为可观测性堆栈的基本部分。Trace 是分布式系统执行的足迹，而 metric 则是用时间轴上的数字衡量系统性能。本质上，它们从两个维度衡量性能。能够快速可视化追踪和相应指标之间的联系，可以快速诊断哪些流程与潜在的异常相关。SkyWalking 9.3.0 现在提供了这一强大的新功能。\nSkyWalking 项目从 tracing 开始，从 2018 年开始专注于 100% 基于采样的指标和拓扑分析。当用户面对时间序列指标的异常趋势时，比如折线图上的峰值，或者直方图显示 p95 和 p95 之间的差距较大，直接的问题是，为什么会出现这种情况？SkyWalking 的最新功能之一，trace 与 metric 关联，使得回答这个问题和解决根本原因更加容易。\n指标是如何生成的？ SkyWalking 提供了三种计算指标的方式：\n根据追踪跨度构建的指标，具体取决于跨度的层、种类和标签。 从日志中提取指标—— 一种基于关键词和标签的指标提取。 从成熟和主流的指标 / 仪表系统报告的指标，例如 OpenTelemetry、Prometheus 和 Zabbix。 Tracing 追踪应用程序服务之间的请求过程。大多数生成流量和性能相关指标的系统也会生成追踪数据，这些数据来自服务器端基于追踪的聚合或通过客户端 SDK。\n使用 SkyWalking 降低追踪索引的传统成本 Trace 数据和可视化对于开发人员和运维人员来说都是至关重要的故障排除工具，因为它们在定位问题边界方面非常有帮助。但是，由于传统上很难找到指标和痕迹之间的关联，团队已经将越来越多的标签添加到跨度中，并搜索各种组合。这种增加仪器和搜索的趋势需要增加基础设施投资来支持这种搜索。SkyWalking 的指标和追踪关联功能有助于降低索引和搜索该数据的成本。\n查找关联的 trace 在寻找 metric 和 trace 之间的关联时，我们处理的指标类型决定了它们与 trace 的关系。让我们回顾一下标准请求*率、错误和持续时间（RED）*指标，看看它是如何工作的。\n成功率指标 成功率由返回码、RPC 响应码或进程异常决定。当成功率下降时，在这个服务或 Pod 的 trace 中寻找错误是第一个寻找线索的地方。\n图 1：SkyWalking 9.3.0 仪表板的成功率图表，带有在特定时间查看相关 trace 的选项。\n从成功率的峰值向下探索，SkyWalking 列出了在这一特定分钟内收集的所有 trace 及其错误状态（图 2）：\n图 2：SkyWalking 显示具有错误状态的相关追踪。\n可以从 trace 中找到对 /test 的请求，并且 span 的标记指示 HTTP 请求的 404 响应代码。\n图 3：显示 URI 不存在的 http://frontend/test 请求的详细视图。\n通过查看 trace 数据，很明显成功率的下降是由对不存在的 URI 的请求引起的。\n平均响应时间 平均响应时间指标提供了服务性能的一般概览。当平均响应时间不稳定时，这通常意味着系统面临严重的性能影响。\n图 4：SkyWalking 用于搜索相关 trace 的查询 UI，显示超过特定持续时间阈值的请求的 trace。\n当您从该指标向下探索时，该查询条件（图 4）将揭示该特定分钟内服务的最慢 trace。请注意，至少 168ms 作为条件自动添加，以避免扫描数据库中的大量行。\nApdex Apdex（应用程序性能指数）是根据设定的阈值衡量响应时间的指标。它测量令人满意的响应时间与不令人满意的响应时间的比率（图 5）。响应时间是从资产请求到完成交付回请求者的时间。\n图 5：Apdex 公式\n用户定义响应时间容忍阈值 T。在 T 或更短时间内处理的所有响应都使用户满意。\n例如，如果 T 为 1.2 秒，响应在 0.5 秒内完成，则用户会感到满意。所有大于 1.2 秒的响应都会让用户不满意。超过 4.8 秒的响应会让用户感到沮丧。\n当 Apdex 分数下降时，我们需要从两个角度寻找相关的 trace：慢速和错误状态的 trace。SkyWalking 的新相关追踪功能提供了一种直接从 Apdex 图表查看两者（图 6）的快速方法。\n图 6：显示 Apdex 图中的慢速 trace 和错误状态 trace\n服务响应时间 百分位指标百分位图（图 7）提供 p50、p75、p90、p95 和 p99 延迟排名，以衡量服务性能的长尾问题。\n图 7：服务响应时间百分位图有助于突出服务性能的长尾问题。\n这个百分位数图显示了一个典型的长尾问题。P99 延迟比 P95 慢四倍。当我们使用关联时，我们会看到 P95 - P99 和 P99 - Infinity 之间具有延迟的 trace。\n造成这种长尾现象的请求 trace，就是从那里自动列出来的。\n图 8：用于根据延迟搜索 trace 的查询参数。\n是否有更多关联可用？ SkyWalking 提供的不仅仅是 trace 和 metric 之间的关联，还可以帮助您找到可能的因果关系，避免大海捞针。\n目前，SkyWalking 9.3.0 提供了两种关联：metric-to-metric 关联和 event-to-metric 关联。\nMetric-to-metric 关联 仪表板上有许多指标 —— 这对于全面了解应用程序行为非常有用。在典型的性能问题中，多个指标的峰值会同时受到影响。但是，尝试关联所有这些图表中的峰值可能很困难……\n现在在 SkyWalking 9.3.0 中，当你点击一个图表的峰值时，弹出框可以让你看到相关的指标。\n图 9：SkyWalking 用于查看相关指标的选项。\n当您选择该选项时，所有关联的指标图表将在所有关联的图表中显示轴指针（垂直虚线），如图 10 所示。这使得将不同图表中的峰值相互关联起来变得更加容易。通常，这些相关的峰值具有相同的根本原因。\n图 10：轴指针（垂直虚线）显示多个指标图中峰值之间的关联。\nEvent-to-metric 关联 SkyWalking 提供了事件概念来关联可能受基础设施影响的服务性能，例如来自 Kubernetes 的新部署。或者，已通过警报或集成 AIOps 引擎检测到异常。\n事件到指标的关联也是自动的，它可以覆盖指标图上事件的时间范围（蓝色区域）。如果事件区域和峰值匹配，则很可能该事件覆盖了该异常。\n图 11：SkyWalking 的事件与指标关联视图。\nSkyWalking 使查找根本原因变得更加容易和快速 SkyWalking 现在可以轻松找到指标、事件和追踪之间的关联，最终可以确定根本原因并快速解决问题。我们在本文中讨论的关联在 SkyWalking 9.3.0 版本中开箱即用。\n图 12：只需单击圆点即可查看相关 trace 和 metric 关联。\n单击任何指标图上的点，如果该指标具有逻辑映射，您将看到一个查看相关 trace 弹出窗口。\n结论 在这篇博客中，我们了解了 metric 和 trace 之间新增的关联功能。有了这个新的可视化，现在可以更容易地找到关键 trace 来识别问题的根本原因。SkyWalking 中的关联可以更深入。从 metric 到 trace 的关联并不是诊断系统瓶颈的终点。在下一篇文章中，我们将介绍 eBPF 支持的追踪增强功能，您将看到与网络分析中的追踪跨度相关的 HTTP 请求和响应详细信息。敬请关注。\n","excerpt":"\u003cp\u003e\u003cimg src=\"banner.jpg\" alt=\"Banner\"\u003e\u003c/p\u003e\n\u003cp\u003e现代分布式应用程序工作的可观测性对于了解它们在各种条件下的行为方式以及在出现问题时进行故障排除和解决至关重要。追踪、指标和日志被视为可观测性堆栈的基本部分。Trace 是分布式系统执行的足迹，而 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/boost-root-cause-analysis-quickly-with-skywalking-new-trace-metrics-association-feature/","title":"SkyWalking 推出 trace-metric 关联功能助力快速根源问题排查"},{"body":"In cloud native applications, a request often needs to be processed through a series of APIs or backend services, some of which are parallel and some serial and located on different platforms or nodes. How do we determine the service paths and nodes a call goes through to help us troubleshoot the problem? This is where distributed tracing comes into play.\nThis article covers:\nHow distributed tracing works How to choose distributed tracing software How to use distributed tracing in Istio How to view distributed tracing data using Bookinfo and SkyWalking as examples Distributed Tracing Basics Distributed tracing is a method for tracing requests in a distributed system to help users better understand, control, and optimize distributed systems. There are two concepts used in distributed tracing: TraceID and SpanID. You can see them in Figure 1 below.\nTraceID is a globally unique ID that identifies the trace information of a request. All traces of a request belong to the same TraceID, and the TraceID remains constant throughout the trace of the request. SpanID is a locally unique ID that identifies a request’s trace information at a certain time. A request generates different SpanIDs at different periods, and SpanIDs are used to distinguish trace information for a request at different periods. TraceID and SpanID are the basis of distributed tracing. They provide a uniform identifier for request tracing in distributed systems and facilitate users’ ability to query, manage, and analyze the trace information of requests.\nFigure 1: Trace and span\nThe following is the process of distributed tracing:\nWhen a system receives a request, the distributed tracing system assigns a TraceID to the request, which is used to chain together the entire chain of invocations. The distributed trace system generates a SpanID and ParentID for each service call within the system for the request, which is used to record the parent-child relationship of the call; a Span without a ParentID is used as the entry point of the call chain. TraceID and SpanID are to be passed during each service call. When viewing a distributed trace, query the full process of a particular request by TraceID. How Istio Implements Distributed Tracing Istio’s distributed tracing is based on information collected by the Envoy proxy in the data plane. After a service request is intercepted by Envoy, Envoy adds tracing information as headers to the request forwarded to the destination workload. The following headers are relevant for distributed tracing:\nAs TraceID: x-request-id Used to establish parent-child relationships for Span in the LightStep trace: x-ot-span-context\u0026lt;/li Used for Zipkin, also for Jaeger, SkyWalking, see b3-propagation: x-b3-traceid x-b3-traceid x-b3-spanid x-b3-parentspanid x-b3-sampled x-b3-flags b3 For Datadog: x-datadog-trace-id x-datadog-parent-id x-datadog-sampling-priority For SkyWalking: sw8 For AWS X-Ray: x-amzn-trace-id For more information on how to use these headers, please see the Envoy documentation.\nRegardless of the language of your application, Envoy will generate the appropriate tracing headers for you at the Ingress Gateway and forward these headers to the upstream cluster. However, in order to utilize the distributed tracing feature, you must modify your application code to attach the tracing headers to upstream requests. Since neither the service mesh nor the application can automatically propagate these headers, you can integrate the agent for distributed tracing into the application or manually propagate these headers in the application code itself. Once the tracing headers are propagated to all upstream requests, Envoy will send the tracing data to the tracer’s back-end processing, and then you can view the tracing data in the UI.\nFor example, look at the code of the Productpage service in the Bookinfo application. You can see that it integrates the Jaeger client library and synchronizes the header generated by Envoy with the HTTP requests to the Details and Reviews services in the getForwardHeaders (request) function.\ndef getForwardHeaders(request): headers = {} # Using Jaeger agent to get the x-b3-* headers span = get_current_span() carrier = {} tracer.inject( span_context=span.context, format=Format.HTTP_HEADERS, carrier=carrier) headers.update(carrier) # Dealing with the non x-b3-* header manually if \u0026#39;user\u0026#39; in session: headers[\u0026#39;end-user\u0026#39;] = session[\u0026#39;user\u0026#39;] incoming_headers = [ \u0026#39;x-request-id\u0026#39;, \u0026#39;x-ot-span-context\u0026#39;, \u0026#39;x-datadog-trace-id\u0026#39;, \u0026#39;x-datadog-parent-id\u0026#39;, \u0026#39;x-datadog-sampling-priority\u0026#39;, \u0026#39;traceparent\u0026#39;, \u0026#39;tracestate\u0026#39;, \u0026#39;x-cloud-trace-context\u0026#39;, \u0026#39;grpc-trace-bin\u0026#39;, \u0026#39;sw8\u0026#39;, \u0026#39;user-agent\u0026#39;, \u0026#39;cookie\u0026#39;, \u0026#39;authorization\u0026#39;, \u0026#39;jwt\u0026#39;, ] for ihdr in incoming_headers: val = request.headers.get(ihdr) if val is not None: headers[ihdr] = val return headers For more information, the Istio documentation provides answers to frequently asked questions about distributed tracing in Istio.\nHow to Choose A Distributed Tracing System Distributed tracing systems are similar in principle. There are many such systems on the market, such as Apache SkyWalking, Jaeger, Zipkin, Lightstep, Pinpoint, and so on. For our purposes here, we will choose three of them and compare them in several dimensions. Here are our inclusion criteria:\nThey are currently the most popular open-source distributed tracing systems. All are based on the OpenTracing specification. They support integration with Istio and Envoy. Items Apache SkyWalking Jaeger Zipkin Implementations Language-based probes, service mesh probes, eBPF agent, third-party instrumental libraries (Zipkin currently supported) Language-based probes Language-based probes Database ES, H2, MySQL, TiDB, Sharding-sphere, BanyanDB ES, MySQL, Cassandra, Memory ES, MySQL, Cassandra, Memory Supported Languages Java, Rust, PHP, NodeJS, Go, Python, C++, .Net, Lua Java, Go, Python, NodeJS, C#, PHP, Ruby, C++ Java, Go, Python, NodeJS, C#, PHP, Ruby, C++ Initiator Personal Uber Twitter Governance Apache Foundation CNCF CNCF Version 9.3.0 1.39.0 2.23.19 Stars 20.9k 16.8k 15.8k Although Apache SkyWalking’s agent does not support as many languages as Jaeger and Zipkin, SkyWalking’s implementation is richer and compatible with Jaeger and Zipkin trace data, and development is more active, so it is one of the best choices for building a telemetry platform.\nDemo Refer to the Istio documentation to install and configure Apache SkyWalking.\nEnvironment Description The following is the environment for our demo:\nKubernetes 1.24.5 Istio 1.16 SkyWalking 9.1.0 Install Istio Before installing Istio, you can check the environment for any problems:\n$ istioctl experimental precheck ✔ No issues found when checking the cluster. Istio is safe to install or upgrade! To get started, check out https://istio.io/latest/docs/setup/getting-started/ Then install Istio and configure the destination for sending tracing messages as SkyWalking:\n# Initial Istio Operator istioctl operator init # Configure tracing destination kubectl apply -f - \u0026lt;\u0026lt;EOF apiVersion: install.istio.io/v1alpha1 kind: IstioOperator metadata: namespace: istio-system name: istio-with-skywalking spec: meshConfig: defaultProviders: tracing: - \u0026#34;skywalking\u0026#34; enableTracing: true extensionProviders: - name: \u0026#34;skywalking\u0026#34; skywalking: service: tracing.istio-system.svc.cluster.local port: 11800 EOF Deploy Apache SkyWalking Istio 1.16 supports distributed tracing using Apache SkyWalking. Install SkyWalking by executing the following code:\nkubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.16/samples/addons/extras/skywalking.yaml It will install the following components under the istio-system namespace:\nSkyWalking Observability Analysis Platform (OAP): Used to receive trace data, supports SkyWalking native data formats, Zipkin v1 and v2 and Jaeger format. UI: Used to query distributed trace data. For more information about SkyWalking, please refer to the SkyWalking documentation.\nDeploy the Bookinfo Application Execute the following command to install the bookinfo application:\nkubectl label namespace default istio-injection=enabled kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml Launch the SkyWalking UI:\nistioctl dashboard skywalking Figure 2 shows all the services available in the bookinfo application:\nFigure 2: SkyWalking General Service page\nYou can also see information about instances, endpoints, topology, tracing, etc. For example, Figure 3 shows the service topology of the bookinfo application:\nFigure 3: Topology diagram of the Bookinfo application\nTracing views in SkyWalking can be displayed in a variety of formats, including list, tree, table, and statistics. See Figure 4:\nFigure 4: SkyWalking General Service trace supports multiple display formats\nTo facilitate our examination, set the sampling rate of the trace to 100%:\nkubectl apply -f - \u0026lt;\u0026lt;EOF apiVersion: telemetry.istio.io/v1alpha1 kind: Telemetry metadata: name: mesh-default namespace: istio-system spec: tracing: - randomSamplingPercentage: 100.00 EOF Important: It’s generally not good practice to set the sampling rate to 100% in a production environment. To avoid the overhead of generating too many trace logs in production, please adjust the sampling strategy (sampling percentage).\nUninstall After experimenting, uninstall Istio and SkyWalking by executing the following command.\nsamples/bookinfo/platform/kube/cleanup.sh istioctl unintall --purge kubectl delete namespace istio-system Understanding the Bookinfo Tracing Information Navigate to the General Service tab in the Apache SkyWalking UI, and you can see the trace information for the most recent istio-ingressgateway service, as shown in Figure 5. Click on each span to see the details.\nFigure 5: The table view shows the basic information about each span.\nSwitching to the list view, you can see the execution order and duration of each span, as shown in Figure 6:\nFigure 6: List display\nYou might want to know why such a straightforward application generates so much span data. Because after we inject the Envoy proxy into the pod, every request between services will be intercepted and processed by Envoy, as shown in Figure 7:\nFigure 7: Envoy intercepts requests to generate a span\nThe tracing process is shown in Figure 8:\nFigure 8: Trace of the Bookinfo application\nWe give each span a label with a serial number, and the time taken is indicated in parentheses. For illustration purposes, we have summarized all spans in the table below.\nNo. Endpoint Total Duration (ms) Component Duration (ms) Current Service Description 1 /productpage 190 0 istio-ingressgateway Envoy Outbound 2 /productpage 190 1 istio-ingressgateway Ingress -\u0026gt; Productpage network transmission 3 /productpage 189 1 productpage Envoy Inbound 4 /productpage 188 21 productpage Application internal processing 5 /details/0 8 1 productpage Envoy Outbound 6 /details/0 7 3 productpage Productpage -\u0026gt; Details network transmission 7 /details/0 4 0 details Envoy Inbound 8 /details/0 4 4 details Application internal processing 9 /reviews/0 159 0 productpage Envoy Outbound 10 /reviews/0 159 14 productpage Productpage -\u0026gt; Reviews network transmission 11 /reviews/0 145 1 reviews Envoy Inbound 12 /reviews/0 144 109 reviews Application internal processing 13 /ratings/0 35 2 reviews Envoy Outbound 14 /ratings/0 33 16 reviews Reviews -\u0026gt; Ratings network transmission 15 /ratings/0 17 1 ratings Envoy Inbound 16 /ratings/0 16 16 ratings Application internal processing From the above information, it can be seen that:\nThe total time consumed for this request is 190 ms. In Istio sidecar mode, each traffic flow in and out of the application container must pass through the Envoy proxy once, each time taking 0 to 2 ms. Network requests between Pods take between 1 and 16ms. This is because the data itself has errors and the start time of the Span is not necessarily equal to the end time of the parent Span. We can see that the most time-consuming part is the Reviews application, which takes 109 ms so that we can optimize it for that application. Summary Distributed tracing is an indispensable tool for analyzing performance and troubleshooting modern distributed applications. In this tutorial, we’ve seen how, with just a few minor changes to your application code to propagate tracing headers, Istio makes distributed tracing simple to use. We’ve also reviewed Apache SkyWalking as one of the best distributed tracing systems that Istio supports. It is a fully functional platform for cloud native application analytics, with features such as metrics and log collection, alerting, Kubernetes monitoring, service mesh performance diagnosis using eBPF, and more.\nIf you’re new to service mesh and Kubernetes security, we have a bunch of free online courses available at Tetrate Academy that will quickly get you up to speed with Istio and Envoy.\nIf you’re looking for a fast way to get to production with Istio, check out Tetrate Istio Distribution (TID). TID is Tetrate’s hardened, fully upstream Istio distribution, with FIPS-verified builds and support available. It’s a great way to get started with Istio knowing you have a trusted distribution to begin with, have an expert team supporting you, and also have the option to get to FIPS compliance quickly if you need to.\nOnce you have Istio up and running, you will probably need simpler ways to manage and secure your services beyond what’s available in Istio, that’s where Tetrate Service Bridge comes in. You can learn more about how Tetrate Service Bridge makes service mesh more secure, manageable, and resilient here, or contact us for a quick demo.\n","excerpt":"\u003cp\u003eIn cloud native applications, a request often needs to be processed through a series of APIs or …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/how-to-use-skywalking-for-distributed-tracing-in-istio/","title":"How to Use SkyWalking for Distributed Tracing in Istio?"},{"body":"在云原生应用中，一次请求往往需要经过一系列的 API 或后台服务处理才能完成，这些服务有些是并行的，有些是串行的，而且位于不同的平台或节点。那么如何确定一次调用的经过的服务路径和节点以帮助我们进行问题排查？这时候就需要使用到分布式追踪。\n本文将向你介绍：\n分布式追踪的原理 如何选择分布式追踪软件 在 Istio 中如何使用分布式追踪 以 Bookinfo 和 SkyWalking 为例说明如何查看分布式追踪数据 分布式追踪基础 分布式追踪是一种用来跟踪分布式系统中请求的方法，它可以帮助用户更好地理解、控制和优化分布式系统。分布式追踪中用到了两个概念：TraceID 和 SpanID。\nTraceID 是一个全局唯一的 ID，用来标识一个请求的追踪信息。一个请求的所有追踪信息都属于同一个 TraceID，TraceID 在整个请求的追踪过程中都是不变的； SpanID 是一个局部唯一的 ID，用来标识一个请求在某一时刻的追踪信息。一个请求在不同的时间段会产生不同的 SpanID，SpanID 用来区分一个请求在不同时间段的追踪信息； TraceID 和 SpanID 是分布式追踪的基础，它们为分布式系统中请求的追踪提供了一个统一的标识，方便用户查询、管理和分析请求的追踪信息。\n下面是分布式追踪的过程：\n当一个系统收到请求后，分布式追踪系统会为该请求分配一个 TraceID，用于串联起整个调用链； 分布式追踪系统会为该请求在系统内的每一次服务调用生成一个 SpanID 和 ParentID，用于记录调用的父子关系，没有 ParentID 的 Span 将作为调用链的入口； 每个服务调用过程中都要传递 TraceID 和 SpanID； 在查看分布式追踪时，通过 TraceID 查询某次请求的全过程； Istio 如何实现分布式追踪 Istio 中的分布式追踪是基于数据平面中的 Envoy 代理实现的。服务请求在被劫持到 Envoy 中后，Envoy 在转发请求时会附加大量 Header，其中与分布式追踪相关的有：\n作为 TraceID：x-request-id 用于在 LightStep 追踪系统中建立 Span 的父子关系：x-ot-span-context 用于 Zipkin，同时适用于 Jaeger、SkyWalking，详见 b3-propagation： x-b3-traceid x-b3-spanid x-b3-parentspanid x-b3-sampled x-b3-flags b3 用于 Datadog： x-datadog-trace-id x-datadog-parent-id x-datadog-sampling-priority 用于 SkyWalking：sw8 用于 AWS X-Ray：x-amzn-trace-id 关于这些 Header 的详细用法请参考 Envoy 文档 。\nEnvoy 会在 Ingress Gateway 中为你产生用于追踪的 Header，不论你的应用程序使用何种语言开发，Envoy 都会将这些 Header 转发到上游集群。但是，你还要对应用程序代码做一些小的修改，才能为使用分布式追踪功能。这是因为应用程序无法自动传播这些 Header，可以在程序中集成分布式追踪的 Agent，或者在代码中手动传播这些 Header。Envoy 会将追踪数据发送到 tracer 后端处理，然后就可以在 UI 中查看追踪数据了。\n例如在 Bookinfo 应用中的 Productpage 服务，如果你查看它的代码可以发现，其中集成了 Jaeger 客户端库，并在 getForwardHeaders (request) 方法中将 Envoy 生成的 Header 同步给对 Details 和 Reviews 服务的 HTTP 请求：\ndef getForwardHeaders(request): headers = {} # 使用 Jaeger agent 获取 x-b3-* header span = get_current_span() carrier = {} tracer.inject( span_context=span.context, format=Format.HTTP_HEADERS, carrier=carrier) headers.update(carrier) # 手动处理非 x-b3-* header if \u0026#39;user\u0026#39; in session: headers[\u0026#39;end-user\u0026#39;] = session[\u0026#39;user\u0026#39;] incoming_headers = [ \u0026#39;x-request-id\u0026#39;, \u0026#39;x-ot-span-context\u0026#39;, \u0026#39;x-datadog-trace-id\u0026#39;, \u0026#39;x-datadog-parent-id\u0026#39;, \u0026#39;x-datadog-sampling-priority\u0026#39;, \u0026#39;traceparent\u0026#39;, \u0026#39;tracestate\u0026#39;, \u0026#39;x-cloud-trace-context\u0026#39;, \u0026#39;grpc-trace-bin\u0026#39;, \u0026#39;sw8\u0026#39;, \u0026#39;user-agent\u0026#39;, \u0026#39;cookie\u0026#39;, \u0026#39;authorization\u0026#39;, \u0026#39;jwt\u0026#39;, ] for ihdr in incoming_headers: val = request.headers.get(ihdr) if val is not None: headers[ihdr] = val return headers 关于 Istio 中分布式追踪的常见问题请见 Istio 文档 。\n分布式追踪系统如何选择 分布式追踪系统的原理类似，市面上也有很多这样的系统，例如 Apache SkyWalking 、Jaeger 、Zipkin 、LightStep 、Pinpoint 等。我们将选择其中三个，从多个维度进行对比。之所以选择它们是因为：\n它们是当前最流行的开源分布式追踪系统； 都是基于 OpenTracing 规范； 都支持与 Istio 及 Envoy 集成； 类别 Apache SkyWalking Jaeger Zipkin 实现方式 基于语言的探针、服务网格探针、eBPF agent、第三方指标库（当前支持 Zipkin） 基于语言的探针 基于语言的探针 数据存储 ES、H2、MySQL、TiDB、Sharding-sphere、BanyanDB ES、MySQL、Cassandra、内存 ES、MySQL、Cassandra、内存 支持语言 Java、Rust、PHP、NodeJS、Go、Python、C++、.NET、Lua Java、Go、Python、NodeJS、C#、PHP、Ruby、C++ Java、Go、Python、NodeJS、C#、PHP、Ruby、C++ 发起者 个人 Uber Twitter 治理方式 Apache Foundation CNCF CNCF 版本 9.3.0 1.39.0 2.23.19 Star 数量 20.9k 16.8k 15.8k 分布式追踪系统对比表（数据截止时间 2022-12-07）\n虽然 Apache SkyWalking 的 Agent 支持的语言没有 Jaeger 和 Zipkin 多，但是 SkyWalking 的实现方式更丰富，并且与 Jaeger、Zipkin 的追踪数据兼容，开发更为活跃，且为国人开发，中文资料丰富，是构建遥测平台的最佳选择之一。\n实验 参考 Istio 文档 来安装和配置 Apache SkyWalking。\n环境说明 以下是我们实验的环境：\nKubernetes 1.24.5 Istio 1.16 SkyWalking 9.1.0 安装 Istio 安装之前可以先检查下环境是否有问题:\n$ istioctl experimental precheck ✔ No issues found when checking the cluster. Istio is safe to install or upgrade! To get started, check out https://istio.io/latest/docs/setup/getting-started/ 然后安装 Istio 同时配置发送追踪信息的目的地为 SkyWalking：\n# 初始化 Istio Operator istioctl operator init # 安装 Istio 并配置使用 SkyWalking kubectl apply -f - \u0026lt;\u0026lt;EOF apiVersion: install.istio.io/v1alpha1 kind: IstioOperator metadata: namespace: istio-system name: istio-with-skywalking spec: meshConfig: defaultProviders: tracing: - \u0026#34;skywalking\u0026#34; enableTracing: true extensionProviders: - name: \u0026#34;skywalking\u0026#34; skywalking: service: tracing.istio-system.svc.cluster.local port: 11800 EOF 部署 Apache SkyWalking Istio 1.16 支持使用 Apache SkyWalking 进行分布式追踪，执行下面的代码安装 SkyWalking：\nkubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.16/samples/addons/extras/skywalking.yaml 它将在 istio-system 命名空间下安装：\nSkyWalking OAP (Observability Analysis Platform) ：用于接收追踪数据，支持 SkyWalking 原生数据格式，Zipkin v1 和 v2 以及 Jaeger 格式。 UI ：用于查询分布式追踪数据。 关于 SkyWalking 的详细信息请参考 SkyWalking 文档 。\n部署 Bookinfo 应用 执行下面的命令安装 bookinfo 示例：\nkubectl label namespace default istio-injection=enabled kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml 打开 SkyWalking UI：\nistioctl dashboard skywalking SkyWalking 的 General Service 页面展示了 bookinfo 应用中的所有服务。\n你还可以看到实例、端点、拓扑、追踪等信息。例如下图展示了 bookinfo 应用的服务拓扑。\nSkyWalking 的追踪视图有多种显示形式，如列表、树形、表格和统计。\nSkyWalking 通用服务追踪支持多种显示样式\n为了方便我们检查，将追踪的采样率设置为 100%：\nkubectl apply -f - \u0026lt;\u0026lt;EOF apiVersion: telemetry.istio.io/v1alpha1 kind: Telemetry metadata: name: mesh-default namespace: istio-system spec: tracing: - randomSamplingPercentage: 100.00 EOF 卸载 在实验完后，执行下面的命令卸载 Istio 和 SkyWalking：\nsamples/bookinfo/platform/kube/cleanup.sh istioctl unintall --purge kubectl delete namespace istio-system Bookinfo demo 追踪信息说明 在 Apache SkyWalking UI 中导航到 General Service 分页，查看最近的 istio-ingressgateway 服务的追踪信息，表视图如下所示。图中展示了此次请求所有 Span 的基本信息，点击每个 Span 可以查看详细信息。\n切换为列表视图，可以看到每个 Span 的执行顺序及持续时间，如下图所示。\n你可能会感到困惑，为什么这么简单的一个应用会产生如此多的 Span 信息？因为我们为 Pod 注入了 Envoy 代理之后，每个服务间的请求都会被 Envoy 拦截和处理，如下图所示。\n整个追踪流程如下图所示。\n图中给每一个 Span 标记了序号，并在括号里注明了耗时。为了便于说明我们将所有 Span 汇总在下面的表格中。\n序号 方法 总耗时（ms） 组件耗时（ms） 当前服务 说明 1 /productpage 190 0 istio-ingressgateway Envoy Outbound 2 /productpage 190 1 istio-ingressgateway Ingress -\u0026gt; Productpage 网络传输 3 /productpage 189 1 productpage Envoy Inbound 4 /productpage 188 21 productpage 应用内部处理 5 /details/0 8 1 productpage Envoy Outbound 6 /details/0 7 3 productpage Productpage -\u0026gt; Details 网络传输 7 /details/0 4 0 details Envoy Inbound 8 /details/0 4 4 details 应用内部 9 /reviews/0 159 0 productpage Envoy Outbound 10 /reviews/0 159 14 productpage Productpage -\u0026gt; Reviews 网络传输 11 /reviews/0 145 1 reviews Envoy Inbound 12 /reviews/0 144 109 reviews 应用内部处理 13 /ratings/0 35 2 reviews Envoy Outbound 14 /ratings/0 33 16 reviews Reviews -\u0026gt; Ratings 网络传输 15 /ratings/0 17 1 ratings Envoy Inbound 16 /ratings/0 16 16 ratings 应用内部处理 从以上信息可以发现：\n本次请求总耗时 190ms； 在 Istio sidecar 模式下，每次流量在进出应用容器时都需要经过一次 Envoy 代理，每次耗时在 0 到 2 ms； 在 Pod 间的网络请求耗时在 1 到 16ms 之间； 将耗时做多的调用链 Ingress Gateway -\u0026gt; Productpage -\u0026gt; Reviews -\u0026gt; Ratings 上的所有耗时累计 182 ms，小于请求总耗时 190ms，这是因为数据本身有误差，以及 Span 的开始时间并不一定等于父 Span 的结束时间，如果你在 SkyWalking 的追踪页面，选择「列表」样式查看追踪数据（见图 2）可以更直观的发现这个问题； 我们可以查看到最耗时的部分是 Reviews 应用，耗时 109ms，因此我们可以针对该应用进行优化； 总结 只要对应用代码稍作修改就可以在 Istio 很方便的使用分布式追踪功能。在 Istio 支持的众多分布式追踪系统中，Apache SkyWalking 是其中的佼佼者。它不仅支持分布式追踪，还支持指标和日志收集、报警、Kubernetes 和服务网格监控，使用 eBPF 诊断服务网格性能 等功能，是一个功能完备的云原生应用分析平台。本文中为了方便演示，将追踪采样率设置为了 100%，在生产使用时请根据需要调整采样策略（采样百分比），防止产生过多的追踪日志。\n如果您不熟悉服务网格和 Kubernetes 安全性，我们在 Tetrate Academy 提供了一系列免费在线课程，可以让您快速了解 Istio 和 Envoy。\n如果您正在寻找一种快速将 Istio 投入生产的方法，请查看 Tetrate Istio Distribution (TID)。TID 是 Tetrate 的强化、完全上游的 Istio 发行版，具有经过 FIPS 验证的构建和支持。这是开始使用 Istio 的好方法，因为您知道您有一个值得信赖的发行版，有一个支持您的专家团队，并且如果需要，还可以选择快速获得 FIPS 合规性。\n一旦启动并运行 Istio，您可能需要更简单的方法来管理和保护您的服务，而不仅仅是 Istio 中可用的方法，这就是 Tetrate Service Bridge 的用武之地。您可以在这里详细了解 Tetrate Service Bridge 如何使服务网格更安全、更易于管理和弹性，或联系我们进行快速演示。\n","excerpt":"\u003cp\u003e在云原生应用中，一次请求往往需要经过一系列的 API 或后台服务处理才能完成，这些服务有些是并行的，有些是串行的，而且位于不同的平台或节点。那么如何确定一次调用的经过的服务路径和节点以帮助我们进行问题 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/how-to-use-skywalking-for-distributed-tracing-in-istio/","title":"如何在 Istio 中使用 SkyWalking 进行分布式追踪？"},{"body":"Introduction Apache SkyWalking is an open source APM tool for monitoring and troubleshooting distributed systems, especially designed for microservices, cloud native and container-based (Docker, Kubernetes, Mesos) architectures. It provides distributed tracing, service mesh observability, metric aggregation and visualization, and alarm.\nIn this article, I will introduce how to quickly set up Apache SkyWalking on AWS EKS and RDS/Aurora, as well as a couple of sample services, monitoring services to observe SkyWalking itself.\nPrerequisites AWS account AWS CLI Terraform kubectl We can use the AWS web console or CLI to create all resources needed in this tutorial, but it can be too tedious and hard to debug when something goes wrong. So in this artical I will use Terraform to create all AWS resources, deploy SkyWalking, sample services, and load generator services (Locust).\nArchitecture The demo architecture is as follows:\ngraph LR subgraph AWS subgraph EKS subgraph istio-system namespace direction TB OAP[[SkyWalking OAP]] UI[[SkyWalking UI]] Istio[[istiod]] end subgraph sample namespace Service0[[Service0]] Service1[[Service1]] ServiceN[[Service ...]] end subgraph locust namespace LocustMaster[[Locust Master]] LocustWorkers0[[Locust Worker 0]] LocustWorkers1[[Locust Worker 1]] LocustWorkersN[[Locust Worker ...]] end end RDS[[RDS/Aurora]] end OAP --\u0026gt; RDS Service0 -. telemetry data -.-\u0026gt; OAP Service1 -. telemetry data -.-\u0026gt; OAP ServiceN -. telemetry data -.-\u0026gt; OAP UI --query--\u0026gt; OAP LocustWorkers0 -- traffic --\u0026gt; Service0 LocustWorkers1 -- traffic --\u0026gt; Service0 LocustWorkersN -- traffic --\u0026gt; Service0 Service0 --\u0026gt; Service1 --\u0026gt; ServiceN LocustMaster --\u0026gt; LocustWorkers0 LocustMaster --\u0026gt; LocustWorkers1 LocustMaster --\u0026gt; LocustWorkersN User --\u0026gt; LocustMaster As shown in the architecture diagram, we need to create the following AWS resources:\nEKS cluster RDS instance or Aurora cluster Sounds simple, but there are a lot of things behind the scenes, such as VPC, subnets, security groups, etc. You have to configure them correctly to make sure the EKS cluster can connect to RDS instance/Aurora cluster otherwise the SkyWalking won\u0026rsquo;t work. Luckily, Terraform can help us to create and destroy all these resources automatically.\nI have created a Terraform module to create all AWS resources needed in this tutorial, you can find it in the GitHub repository.\nCreate AWS resources First, we need to clone the GitHub repository and cd into the folder:\ngit clone https://github.com/kezhenxu94/oap-load-test.git Then, we need to create a file named terraform.tfvars to specify the AWS region and other variables:\ncat \u0026gt; terraform.tfvars \u0026lt;\u0026lt;EOF aws_access_key = \u0026#34;\u0026#34; aws_secret_key = \u0026#34;\u0026#34; cluster_name = \u0026#34;skywalking-on-aws\u0026#34; region = \u0026#34;ap-east-1\u0026#34; db_type = \u0026#34;rds-postgresql\u0026#34; EOF If you have already configured the AWS CLI, you can skip the aws_access_key and aws_secret_key variables. To install SkyWalking with RDS postgresql, set the db_type to rds-postgresql, to install SkyWalking with Aurora postgresql, set the db_type to aurora-postgresql.\nThere are a lot of other variables you can configure, such as tags, sample services count, replicas, etc., you can find them in the variables.tf.\nThen, we can run the following commands to initialize the Terraform module and download the required providers, then create all AWS resources:\nterraform init terraform apply -var-file=terraform.tfvars Type yes to confirm the creation of all AWS resources, or add the -auto-approve flag to the terraform apply to skip the confirmation:\nterraform apply -var-file=terraform.tfvars -auto-approve Now what you need to do is to wait for the creation of all AWS resources to complete, it may take a few minutes. You can check the progress of the creation in the AWS web console, and check the deployment progress of the services inside the EKS cluster.\nGenerate traffic Besides creating necessary AWS resources, the Terraform module also deploys SkyWalking, sample services, and Locust load generator services to the EKS cluster.\nYou can access the Locust web UI to generate traffic to the sample services:\nopen http://$(kubectl get svc -n locust -l app=locust-master -o jsonpath=\u0026#39;{.items[0].status.loadBalancer.ingress[0].hostname}\u0026#39;):8089 The command opens the browser to the Locust web UI, you can configure the number of users and hatch rate to generate traffic.\nObserve SkyWalking You can access the SkyWalking web UI to observe the sample services.\nFirst you need to forward the SkyWalking UI port to local\nkubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=skywalking -l component=ui -o name) 8080:8080 And then open the browser to http://localhost:8080 to access the SkyWalking web UI.\nObserve RDS/Aurora You can also access the RDS/Aurora web console to observe the performance of RDS/Aurora instance/Aurora cluste.\nTest Results Test 1: SkyWalking with EKS and RDS PostgreSQL Service Traffic RDS Performance SkyWalking Performance Test 2: SkyWalking with EKS and Aurora PostgreSQL Service Traffic RDS Performance SkyWalking Performance Clean up When you are done with the demo, you can run the following command to destroy all AWS resources:\nterraform destroy -var-file=terraform.tfvars -auto-approve ","excerpt":"\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eApache SkyWalking is an open source APM tool for monitoring and troubleshooting …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2022-12-13-how-to-run-apache-skywalking-on-aws-eks-rds/","title":"How to run Apache SkyWalking on AWS EKS and RDS/Aurora"},{"body":"介绍 Apache SkyWalking 是一个开源的 APM 工具，用于监控分布式系统和排除故障，特别是为微服务、云原生和基于容器（Docker、Kubernetes、Mesos）的架构而设计。它提供分布式跟踪、服务网格可观测性、指标聚合和可视化以及警报。\n在本文中，我将介绍如何在 AWS EKS 和 RDS/Aurora 上快速设置 Apache SkyWalking，以及几个示例服务，监控服务以观察 SkyWalking 本身。\n先决条件 AWS 账号 AWS CLI Terraform kubectl 我们可以使用 AWS Web 控制台或 CLI 来创建本教程所需的所有资源，但是当出现问题时，它可能过于繁琐且难以调试。因此，在本文中，我将使用 Terraform 创建所有 AWS 资源、部署 SkyWalking、示例服务和负载生成器服务 (Locust)。\n架构 演示架构如下：\ngraph LR subgraph AWS subgraph EKS subgraph istio-system namespace direction TB OAP[[SkyWalking OAP]] UI[[SkyWalking UI]] Istio[[istiod]] end subgraph sample namespace Service0[[Service0]] Service1[[Service1]] ServiceN[[Service ...]] end subgraph locust namespace LocustMaster[[Locust Master]] LocustWorkers0[[Locust Worker 0]] LocustWorkers1[[Locust Worker 1]] LocustWorkersN[[Locust Worker ...]] end end RDS[[RDS/Aurora]] end OAP --\u0026gt; RDS Service0 -. telemetry data -.-\u0026gt; OAP Service1 -. telemetry data -.-\u0026gt; OAP ServiceN -. telemetry data -.-\u0026gt; OAP UI --query--\u0026gt; OAP LocustWorkers0 -- traffic --\u0026gt; Service0 LocustWorkers1 -- traffic --\u0026gt; Service0 LocustWorkersN -- traffic --\u0026gt; Service0 Service0 --\u0026gt; Service1 --\u0026gt; ServiceN LocustMaster --\u0026gt; LocustWorkers0 LocustMaster --\u0026gt; LocustWorkers1 LocustMaster --\u0026gt; LocustWorkersN User --\u0026gt; LocustMaster 如架构图所示，我们需要创建以下 AWS 资源：\nEKS 集群 RDS 实例或 Aurora 集群 听起来很简单，但背后有很多东西，比如 VPC、子网、安全组等。你必须正确配置它们以确保 EKS 集群可以连接到 RDS 实例 / Aurora 集群，否则 SkyWalking 不会不工作。幸运的是，Terraform 可以帮助我们自动创建和销毁所有这些资源。\n我创建了一个 Terraform 模块来创建本教程所需的所有 AWS 资源，您可以在 GitHub 存储库中找到它。\n创建 AWS 资源 首先，我们需要将 GitHub 存储库克隆 cd 到文件夹中：\ngit clone https://github.com/kezhenxu94/oap-load-test.git 然后，我们需要创建一个文件 terraform.tfvars 来指定 AWS 区域和其他变量：\ncat \u0026gt; terraform.tfvars \u0026lt;\u0026lt;EOF aws_access_key = \u0026#34;\u0026#34; aws_secret_key = \u0026#34;\u0026#34; cluster_name = \u0026#34;skywalking-on-aws\u0026#34; region = \u0026#34;ap-east-1\u0026#34; db_type = \u0026#34;rds-postgresql\u0026#34; EOF 如果您已经配置了 AWS CLI，则可以跳过 aws_access_key 和 aws_secret_key 变量。要使用 RDS postgresql 安装 SkyWalking，请将 db_type 设置为 rds-postgresql，要使用 Aurora postgresql 安装 SkyWalking，请将 db_type 设置为 aurora-postgresql。\n您可以配置许多其他变量，例如标签、示例服务计数、副本等，您可以在 variables.tf 中找到它们。\n然后，我们可以运行以下命令来初始化 Terraform 模块并下载所需的提供程序，然后创建所有 AWS 资源：\nterraform init terraform apply -var-file=terraform.tfvars 键入 yes 以确认所有 AWS 资源的创建，或将标志 -auto-approve 添加到 terraform apply 以跳过确认：\nterraform apply -var-file=terraform.tfvars -auto-approve 现在你需要做的就是等待所有 AWS 资源的创建完成，这可能需要几分钟的时间。您可以在 AWS Web 控制台查看创建进度，也可以查看 EKS 集群内部服务的部署进度。\n产生流量 除了创建必要的 AWS 资源外，Terraform 模块还将 SkyWalking、示例服务和 Locust 负载生成器服务部署到 EKS 集群。\n您可以访问 Locust Web UI 以生成到示例服务的流量：\nopen http://$(kubectl get svc -n locust -l app=locust-master -o jsonpath=\u0026#39;{.items[0].status.loadBalancer.ingress[0].hostname}\u0026#39;):8089 该命令将浏览器打开到 Locust web UI，您可以配置用户数量和孵化率以生成流量。\n观察 SkyWalking 您可以访问 SkyWalking Web UI 来观察示例服务。\n首先需要将 SkyWalking UI 端口转发到本地：\nkubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=skywalking -l component=ui -o name) 8080:8080 然后在浏览器中打开 http://localhost:8080 访问 SkyWalking web UI。\n观察 RDS/Aurora 您也可以访问 RDS/Aurora web 控制台，观察 RDS/Aurora 实例 / Aurora 集群的性能。\n试验结果 测试 1：使用 EKS 和 RDS PostgreSQL 的 SkyWalking 服务流量 RDS 性能 SkyWalking 性能 测试 2：使用 EKS 和 Aurora PostgreSQL 的 SkyWalking 服务流量 RDS 性能 SkyWalking 性能 清理 完成演示后，您可以运行以下命令销毁所有 AWS 资源：\nterraform destroy -var-file=terraform.tfvars -auto-approve ","excerpt":"\u003ch2 id=\"介绍\"\u003e介绍\u003c/h2\u003e\n\u003cp\u003eApache SkyWalking 是一个开源的 APM 工具，用于监控分布式系统和排除故障，特别是为微服务、云原生和基于容器（Docker、Kubernetes、Mesos）的架构而设计。它提 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2022-12-13-how-to-run-apache-skywalking-on-aws-eks-rds/","title":"如何在 AWS EKS 和 RDS/Aurora 上运行 Apache SkyWalking"},{"body":"As an application performance monitoring tool for distributed systems, Apache SkyWalking observes metrics, logs, traces, and events in the service mesh.\nSkyWalking OAP’s dataflow processing architecture boasts high performance and is capable of dealing with massive data traffic in real-time. However, storing, updating, and querying massive amounts of data poses a great challenge to its backend storage system.\nBy default, SkyWalking provides storage methods including H2, OpenSearch, ElasticSearch, MySQL, TiDB, PostgreSQL, and BanyanDB. Among them, MySQL storage is suited to a single machine and table (MySQL cluster capability depends on your technology selection). Nevertheless, in the context of high-traffic business systems, the storage of monitoring data is put under great pressure and query performance is lowered.\nBased on MySQL storage, SkyWalking v9.3.0 provides a new storage method: MySQL-Sharding. It supports database and table sharding features thanks to ShardingSphere-Proxy, which is a mature solution for dealing with relational databases’ massive amounts of data.\n1. Architecture Deployment SkyWalking will only interact with ShardingSphere-Proxy instead of directly connecting to the database. The connection exposed by each MySQL node is a data source managed by ShardingSphere-Proxy. ShardingSphere-Proxy will establish a virtual logical database based on the configuration and then carry out database and table sharding and routing according to the OAP provided data sharding rules. SkyWalking OAP creates data sharding rules and performs DDL and DML on a virtual logical database just like it does with MySQL. 2. Application Scenario Applicable to scenarios where MySQL is used for storage, but the single-table mode cannot meet the performance requirements created by business growth.\n3. How Does Data Sharding Work with SkyWalking? Data sharding defines the data Model in SkyWalking with the annotation @SQLDatabase.Sharding.\n@interface Sharding { ShardingAlgorithm shardingAlgorithm(); String dataSourceShardingColumn() default \u0026#34;\u0026#34;; String tableShardingColumn() default \u0026#34;\u0026#34;; } Note:\nshardingAlgorithm: Table sharding algorithm dataSourceShardingColumn: Database sharding key tableShardingColumn: Table sharding key\nSkyWalking selects database sharding key, table sharding key and table sharding algorithm based on @SQLDatabase.Sharding, in order to dynamically generate sharding rules for each table. Next, it performs rule definition by operating ShardingSphere-Proxy via DistSQL. ShardingSphere-Proxy carries out data sharding based on the rule definition.\n3.1 Database Sharding Method SkyWalking adopts a unified method to carry out database sharding. The number of databases that need to be sharded requires modulo by the hash value of the database sharding key, which should be the numeric suffix of the routing target database. Therefore, the routing target database is:\nds_{dataSourceShardingColumn.hashcode() % dataSourceList.size()} For example, we now have dataSourceList = ds_0…ds_n. If {dataSourceShardingColumn.hashcode() % dataSourceList.size() = 2}, all the data will be routed to the data source node ds_2.\n3.2 Table Sharding Method The table sharding algorithm mainly shards according to the data owing to the TTL mechanism. According to TTL, there will be one sharding table per day:\n{tableName = logicTableName_timeSeries (data)} To ensure that data within the TTL can be written and queried, the time series will generate the current date:\n{timeSeries = currentDate - TTL +1...currentDate + 1} For example, if TTL=3 and currentDate=20220907, sharding tables will be: logicTableName_20220905 logicTableName_20220906 logicTableName_20220907 logicTableName_20220908\nSkyWalking provides table sharding algorithms for different data models:\nAlgorithm Name Sharding Description Time Precision Requirements for Sharding Key Typical Application Data Model NO_SHARDING No table sharding and single-table mode is maintained. N/A Data model with a small amount of data and no need for sharding. TIME_RELATIVE_ID_SHARDING_ALGORITHM Shard by day using time_bucket in the ID column. time_bucket can be accurate to seconds, minutes, hours, or days in the same table. Various metrics. TIME_SEC_RANGE_SHARDING_ALGORITHM Shard by day using time_bucket column. time_bucket must be accurate to seconds. SegmentRecordLogRecord, etc. TIME_MIN_RANGE_SHARDING_ALGORITHM Shard by day using time_bucket column. time_bucket must be accurate to minutes. EndpointTraffic TIME_BUCKET_SHARDING_ALGORITHM Shard by day using time_bucket column. time_bucket can be accurate to seconds, minutes, hours, and days in the same table. Service, Instance, Endpoint and other call relations such as ServiceRelationServerSideMetrics 4. TTL Mechanism For sharding tables, delete the physical table deadline \u0026gt;= timeSeries according to TTL.\n{deadline = new DateTime().plusDays(-ttl)} TTL timer will delete the expired tables according to the current date while updating sharding rules according to the new date and informing ShardingSphere-Proxy to create new sharding tables.\nFor a single table, use the previous method and delete the row record of deadline \u0026gt;=time_bucket.\n5. Examples of Sharding Data Storage Next, we’ll take segment (Record type) and service_resp_time (Metrics type) as examples to illustrate the data storage logic and physical distribution. Here, imagine MySQL has two nodes ds_0 and ds_1.\nNote:\nThe following storage table structure is just a simplified version as an example, and does not represent the real SkyWalking table structure.\n5.1 segment The sharding configuration is as follows:\n@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.TIME_SEC_RANGE_SHARDING_ALGORITHM, dataSourceShardingColumn = service_id, tableShardingColumn = time_bucket) The logical database, table structures and actual ones are as follows:\n5.2 service_resp_time The sharding configuration is as follows:\n@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.TIME_RELATIVE_ID_SHARDING_ALGORITHM, tableShardingColumn = id, dataSourceShardingColumn = entity_id) The logical database and table structures and actual ones are as follows:\n6. How to Use ShardingSphere-Proxy? 6.1 Manual Deployment Here we take the deployment of a single-node SkyWalking OAP and ShardingSphere-Proxy 5.1.2 as an example. Please refer to the relevant documentation for the cluster deployment.\nPrepare the MySQL cluster. Deploy, install and configure ShardingSphere-Proxy: conf/server.yaml and props.proxy-hint-enabled must be true. Refer to the link for the complete configuration.\nconf/config-sharding.yaml configures logical database and dataSources list. The dataSource name must be prefixed with ds_ and start with ds_0. For details about the configuration, please refer to this page.\nDeploy, install and configure SkyWalking OAP: Set up OAP environment variables: ${SW_STORAGE:mysql-sharding}，\nConfigure the connection information based on the actual deployment: ${SW_JDBC_URL} ${SW_DATA_SOURCE_USER} ${SW_DATA_SOURCE_PASSWORD}\nNote:\nConnection information must correspond to ShardingSphere-Proxy virtual database.\nConfigure the data source name configured by conf/config-sharding.yaml in ShardingSphere-Proxy to ${SW_JDBC_SHARDING_DATA_SOURCES} and separate names with commas. Start the MySQL cluster. Start ShardingSphere-Proxy. Start SkyWalking OAP. 6.2 Running Demo with Docker Our GitHub repository provides a complete and operational demo based on Docker, allowing you to quickly grasp the operation’s effectiveness. The deployment includes the following:\nOne OAP service. The TTL of Metrics and Record data set to 2 days. One sharding-proxy service with version 5.1.2. Its external port is 13307 and the logical database name is swtest. Two MySQL services. Their external ports are 3306 and 3307 respectively and they are configured as ds_0 and ds_1 in sharding-proxy’s conf/config-sharding.yaml. One provider service (simulated business programs used to verify trace and metrics and other data). Its external port is 9090. One consumer service (simulated business programs used to verify trace and metrics and other data). Its external port is 9092. Download the demo program locally and run it directly in the directory skywalking-mysql-sharding-demo.\ndocker-compose up -d Note:\nThe first startup may take some time to pull images and create all the tables.\nOnce all the services are started, database tools can be used to check the creation of sharding-proxy logical tables and the actual physical sharding table in the two MySQL databases. Additionally, you can also connect the sharding-proxy logical database to view the data query routing. For example:\nPREVIEW SELECT * FROM SEGMENT The result is as follows:\nThe simulated business program provided by the demo can simulate business requests by requesting the consumer service to verify various types of data distribution:\ncurl http://127.0.0.1:9092/info 7. Conclusion In this blog, we introduced SkyWalking’s new storage feature, MySQL sharding, which leverage ShardingSphere-Proxy and covered details of its deployment architecture, application scenarios, sharding logic, and TTL mechanism. We’ve also provided sample data and deployment steps to help get started.\nSkyWalking offers a variety of storage options to fit many use cases. If you need a solution to store large volumes of telemetry data in a relational database, the new MySQL sharding feature is worth a look. For more information on the SkyWalking 9.3.0 release and where to get it, check out the release notes.\n","excerpt":"\u003cp\u003eAs an application performance monitoring tool for distributed systems, Apache SkyWalking observes …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/skywalkings-new-storage-feature-based-on-shardingsphere-proxy-mysql-sharding/","title":"SkyWalking's New Storage Feature Based on ShardingSphere-Proxy: MySQL-Sharding"},{"body":"SkyWalking NodeJS 0.6.0 is released. Go to downloads page to find release tars.\nAdd missing build doc by @kezhenxu94 in https://github.com/apache/skywalking-nodejs/pull/92 Fix invalid url error in axios plugin by @kezhenxu94 in https://github.com/apache/skywalking-nodejs/pull/93 Ignore no requests if ignoreSuffix is empty by @michaelzangl in https://github.com/apache/skywalking-nodejs/pull/94 Escape HTTP method in regexp by @michaelzangl in https://github.com/apache/skywalking-nodejs/pull/95 docs: grammar improvements by @BFergerson in https://github.com/apache/skywalking-nodejs/pull/97 fix: entry span url in endponts using Express middleware/router objects by @BFergerson in https://github.com/apache/skywalking-nodejs/pull/96 chore: use openapi format for endpoint uris by @BFergerson in https://github.com/apache/skywalking-nodejs/pull/98 AWS DynamoDB, Lambda, SQS and SNS plugins, webpack by @tom-pytel in https://github.com/apache/skywalking-nodejs/pull/100 Fix nits by @wu-sheng in https://github.com/apache/skywalking-nodejs/pull/101 Update AxiosPlugin for v1.0+ by @tom-pytel in https://github.com/apache/skywalking-nodejs/pull/102 ","excerpt":"\u003cp\u003eSkyWalking NodeJS 0.6.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAdd missing build …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-nodejs-0-6-0/","title":"Release Apache SkyWalking for NodeJS 0.6.0"},{"body":"SkyWalking 9.3.0 is released. Go to downloads page to find release tars.\nMetrics Association Dashboard Pop-up Trace Query APISIX Dashboard Use Sharding MySQL as the Database Virtual Cache Performance Virtual MQ Performance Project Bump up the embedded swctl version in OAP Docker image. OAP Server Add component ID(133) for impala JDBC Java agent plugin and component ID(134) for impala server. Use prepareStatement in H2SQLExecutor#getByIDs.(No function change). Bump up snakeyaml to 1.32 for fixing CVE. Fix DurationUtils.convertToTimeBucket missed verify date format. Enhance LAL to support converting LogData to DatabaseSlowStatement. [Breaking Change] Change the LAL script format(Add layer property). Adapt ElasticSearch 8.1+, migrate from removed APIs to recommended APIs. Support monitoring MySQL slow SQLs. Support analyzing cache related spans to provide metrics and slow commands for cache services from client side Optimize virtual database, fix dynamic config watcher NPE when default value is null Remove physical index existing check and keep template existing check only to avoid meaningless retry wait in no-init mode. Make sure instance list ordered in TTL processor to avoid TTL timer never runs. Support monitoring PostgreSQL slow SQLs. [Breaking Change] Support sharding MySQL database instances and tables by Shardingsphere-Proxy. SQL-Database requires removing tables log_tag/segment_tag/zipkin_query before OAP starts, if bump up from previous releases. Fix meter functions avgHistogram, avgHistogramPercentile, avgLabeled, sumHistogram having data conflict when downsampling. Do sorting readLabeledMetricsValues result forcedly in case the storage(database) doesn\u0026rsquo;t return data consistent with the parameter list. Fix the wrong watch semantics in Kubernetes watchers, which causes heavy traffic to API server in some Kubernetes clusters, we should use Get State and Start at Most Recent semantic instead of Start at Exact because we don\u0026rsquo;t need the changing history events, see https://kubernetes.io/docs/reference/using-api/api-concepts/#semantics-for-watch. Unify query services and DAOs codes time range condition to Duration. [Breaking Change]: Remove prometheus-fetcher plugin, please use OpenTelemetry to scrape Prometheus metrics and set up SkyWalking OpenTelemetry receiver instead. BugFix: histogram metrics sent to MAL should be treated as OpenTelemetry style, not Prometheus style: (-infinity, explicit_bounds[i]] for i == 0 (explicit_bounds[i-1], explicit_bounds[i]] for 0 \u0026lt; i \u0026lt; size(explicit_bounds) (explicit_bounds[i-1], +infinity) for i == size(explicit_bounds) Support Golang runtime metrics analysis. Add APISIX metrics monitoring Support skywalking-client-js report empty service version and page path , set default version as latest and default page path as /(root). Fix the error fetching data (/browser_app_page_pv0) : Can't split endpoint id into 2 parts. [Breaking Change] Limit the max length of trace/log/alarm tag\u0026rsquo;s key=value, set the max length of column tags in tableslog_tag/segment_tag/alarm_record_tag and column query in zipkin_query and column tag_value in tag_autocomplete to 256. SQL-Database requires altering these columns\u0026rsquo; length or removing these tables before OAP starts, if bump up from previous releases. Optimize the creation conditions of profiling task. Lazy load the Kubernetes metadata and switch from event-driven to polling. Previously we set up watchers to watch the Kubernetes metadata changes, this is perfect when there are deployments changes and SkyWalking can react to the changes in real time. However when the cluster has many events (such as in large cluster or some special Kubernetes engine like OpenShift), the requests sent from SkyWalking becomes unpredictable, i.e. SkyWalking might send massive requests to Kubernetes API server, causing heavy load to the API server. This PR switches from the watcher mechanism to polling mechanism, SkyWalking polls the metadata in a specified interval, so that the requests sent to API server is predictable (~10 requests every interval, 3 minutes), and the requests count is constant regardless of the cluster\u0026rsquo;s changes. However with this change SkyWalking can\u0026rsquo;t react to the cluster changes in time, but the delay is acceptable in our case. Optimize the query time of tasks in ProfileTaskCache. Fix metrics was put into wrong slot of the window in the alerting kernel. Support sumPerMinLabeled in MAL. Bump up jackson databind, snakeyaml, grpc dependencies. Support export Trace and Log through Kafka. Add new config initialization mechanism of module provider. This is a ModuleManager lib kernel level change. [Breaking Change] Support new records query protocol, rename the column named service_id to entity_id for support difference entity. Please re-create top_n_database_statement index/table. Remove improper self-obs metrics in JvmMetricsHandler(for Kafka channel). gRPC stream canceling code is not logged as an error when the client cancels the stream. The client cancels the stream when the pod is terminated. [Breaking Change] Change the way of loading MAL rules(support pattern). Move k8s relative MAL files into /otel-rules/k8s. [Breaking Change] Refactor service mesh protobuf definitions and split TCP-related metrics to individual definition. Add TCP{Service,ServiceInstance,ServiceRelation,ServiceInstanceRelation} sources and split TCP-related entities out from original Service,ServiceInstance,ServiceRelation,ServiceInstanceRelation. [Breaking Change] TCP-related source names are changed, fields of TCP-related sources are changed, please refer to the latest oal/tcp.oal file. Do not log error logs when failed to create ElasticSearch index because the index is created already. Add virtual MQ analysis for native traces. Support Python runtime metrics analysis. Support sampledTrace in LAL. Support multiple rules with different names under the same layer of LAL script. (Optimization) Reduce the buffer size(queue) of MAL(only) metric streams. Set L1 queue size as 1/20, L2 queue size as 1/2. Support monitoring MySQL/PostgreSQL in the cluster mode. [Breaking Change] Migrate to BanyanDB v0.2.0. Adopt new OR logical operator for, MeasureIDs query BanyanDBProfileThreadSnapshotQueryDAO query Multiple Event conditions query Metrics query Simplify Group check and creation Partially apply UITemplate changes Support index_only Return CompletableFuture\u0026lt;Void\u0026gt; directly from BanyanDB client Optimize data binary parse methods in *LogQueryDAO Support different indexType Support configuration for TTL and (block|segment) intervals Elasticsearch storage: Provide system environment variable(SW_STORAGE_ES_SPECIFIC_INDEX_SETTINGS) and support specify the settings (number_of_shards/number_of_replicas) for each index individually. Elasticsearch storage: Support update index settings (number_of_shards/number_of_replicas) for the index template after rebooting. Optimize MQ Topology analysis. Use entry span\u0026rsquo;s peer from the consumer side as source service when no producer instrumentation(no cross-process reference). Refactor JDBC storage implementations to reuse logics. Fix ClassCastException in LoggingConfigWatcher. Support span attached event concept in Zipkin and SkyWalking trace query. Support span attached events on Zipkin lens UI. Force UTF-8 encoding in JsonLogHandler of kafka-fetcher-plugin. Fix max length to 512 of entity, instance and endpoint IDs in trace, log, profiling, topN tables(JDBC storages). The value was 200 by default. Add component IDs(135, 136, 137) for EventMesh server and client-side plugins. Bump up Kafka client to 2.8.1 to fix CVE-2021-38153. Remove lengthEnvVariable for Column as it never works as expected. Add LongText to support longer logs persistent as a text type in ElasticSearch, instead of a keyword, to avoid length limitation. Fix wrong system variable name SW_CORE_ENABLE_ENDPOINT_NAME_GROUPING_BY_OPENAPI. It was opaenapi. Fix not-time-series model blocking OAP boots in no-init mode. Fix ShardingTopologyQueryDAO.loadServiceRelationsDetectedAtServerSide invoke backend miss parameter serviceIds. Changed system variable SW_SUPERDATASET_STORAGE_DAY_STEP to SW_STORAGE_ES_SUPER_DATASET_DAY_STEP to be consistent with other ES storage related variables. Fix ESEventQueryDAO missing metric_table boolQuery criteria. Add default entity name(_blank) if absent to avoid NPE in the decoding. This caused Can't split xxx id into 2 parts. Support dynamic config the sampling strategy in network profiling. Zipkin module support BanyanDB storage. Zipkin traces query API, sort the result set by start time by default. Enhance the cache mechanism in the metric persistent process. This cache only worked when the metric is accessible(readable) from the database. Once the insert execution is delayed due to the scale, the cache loses efficacy. It only works for the last time update per minute, considering our 25s period. Fix ID conflicts for all JDBC storage implementations. Due to the insert delay, the JDBC storage implementation would still generate another new insert statement. [Breaking Change] Remove core/default/enableDatabaseSession config. [Breaking Change] Add @BanyanDB.TimestampColumn to identify which column in Record is providing the timestamp(milliseconds) for BanyanDB, since BanyanDB stream requires a timestamp in milliseconds. For SQL-Database: add new column timestamp for tables profile_task_log/top_n_database_statement, requires altering this column or removing these tables before OAP starts, if bump up from previous releases. Fix Elasticsearch storage: In No-Sharding Mode, add specific analyzer to the template before index creation to avoid update index error. Internal API: remove undocumented ElasticSearch API usage and use documented one. Fix BanyanDB.ShardingKey annotation missed in the generated OAL metrics classes. Fix Elasticsearch storage: Query sortMetrics missing transform real index column name. Rename BanyanDB.ShardingKey to BanyanDB.SeriesID. Self-Observability: Add counters for metrics reading from DB or cached. Dashboard:Metrics Persistent Cache Count. Self-Observability: Fix GC Time calculation. Fix Elasticsearch storage: In No-Sharding Mode, column\u0026rsquo;s property indexOnly not applied and cannot be updated. Update the trace_id field as storage only(cannot be queried) in top_n_database_statement, top_n_cache_read_command, top_n_cache_read_command index. UI Fix: tab active incorrectly, when click tab space Add impala icon for impala JDBC Java agent plugin. (Webapp)Bump up snakeyaml to 1.31 for fixing CVE-2022-25857 [Breaking Change]: migrate from Spring Web to Armeria, now you should use the environment variable name SW_OAP_ADDRESS to change the OAP backend service addresses, like SW_OAP_ADDRESS=localhost:12800,localhost:12801, and use environment variable SW_SERVER_PORT to change the port. Other Spring-related configurations don\u0026rsquo;t take effect anymore. Polish the endpoint list graph. Fix styles for an adaptive height. Fix setting up a new time range after clicking the refresh button. Enhance the process topology graph to support dragging nodes. UI-template: Fix metrics calculation in general-service/mesh-service/faas-function top-list dashboard. Update MySQL dashboard to visualize collected slow SQLs. Add virtual cache dashboard. Remove responseCode fields of all OAL sources, as well as examples to avoid user\u0026rsquo;s confusion. Remove All from the endpoints selector. Enhance menu configurations to make it easier to change. Update PostgreSQL dashboard to visualize collected slow SQLs. Add Golang runtime metrics and cpu/memory used rate panels in General-Instance dashboard. Add gateway apisix menu. Query logs with the specific service ID. Bump d3-color from 3.0.1 to 3.1.0. Add Golang runtime metrics and cpu/memory used rate panels in FaaS-Instance dashboard. Revert logs on trace widget. Add a sub-menu for virtual mq. Add readRecords to metric types. Verify dashboard names for new dashboards. Associate metrics with the trace widget on dashboards. Fix configuration panel styles. Remove a un-use icon. Support labeled value on the service/instance/endpoint list widgets. Add menu for virtual MQ. Set selector props and update configuration panel styles. Add Python runtime metrics and cpu/memory utilization panels to General-Instance and Fass-Instance dashboards. Enhance the legend of metrics graph widget with the summary table. Add apache eventMesh logo file. Fix conditions for trace profiling. Fix tag keys list and duration condition. Fix typo. Fix condition logic for trace tree data. Enhance tags component to search tags with the input value. Fix topology loading style. Fix update metric processor for the readRecords and remove readSampledRecords from metrics selector. Add trace association for FAAS dashboards. Visualize attached events on the trace widget. Add HTTP/1.x metrics and HTTP req/resp body collecting tabs on the network profiling widget. Implement creating tasks ui for network profiling widget. Fix entity types for ProcessRelation. Add trace association for general service dashboards. Documentation Add metadata-uid setup doc about Kubernetes coordinator in the cluster management. Add a doc for adding menus to booster UI. Move general good read blogs from Agent Introduction to Academy. Add re-post for blog Scaling with Apache SkyWalking in the academy list. Add re-post for blog Diagnose Service Mesh Network Performance with eBPF in the academy list. Add Security Notice doc. Add new docs for Report Span Attached Events data collecting protocol. Add new docs for Record query protocol Update Server Agents and Compatibility for PHP agent. Add docs for profiling. Update the network profiling documentation. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking 9.3.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"metrics-association\"\u003eMetrics Association …\u003c/h4\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-9.3.0/","title":"Release Apache SkyWalking APM 9.3.0"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/shardingsphere/","title":"ShardingSphere"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/shardingsphere-proxy/","title":"ShardingSphere-Proxy"},{"body":"Apache SkyWalking 作为一个分布式系统的应用性能监控工具，它观察服务网格中的指标、日志、痕迹和事件。其中 SkyWalking OAP 高性能的数据流处理架构能够实时处理庞大的数据流量，但是这些海量数据的存储更新和后续查询对后端存储系统带来了挑战。\nSkyWalking 默认已经提供了多种存储支持包括 H2、OpenSearch、ElasticSearch、MySQL、TiDB、PostgreSQL、BanyanDB。其中 MySQL 存储提供的是针对单机和单表的存储方式（MySQL 的集群能力需要自己选型提供），在面对高流量的业务系统时，监控数据的存储存在较大压力，同时影响查询性能。\n在 MySQL 存储基础上 SkyWalking v9.3.0 提供了一种新的存储方式 MySQL-Sharding，它提供了基于 ShardingSphere-Proxy 的分库分表特性，而分库分表是关系型数据库面对大数据量处理的成熟解决方案。\n部署架构 SkyWalking 使用 ShardingSphere-Proxy 的部署方式如下图所示。\nSkyWalking OAP 由直连数据库的方式变成只与 ShardingSphere-Proxy 进行交互； 每一个 MySQL 节点暴露的连接都是一个数据源，由 ShardingSphere-Proxy 进行统一管理； ShardingSphere-Proxy 会根据配置建立一个虚拟逻辑数据库，根据 OAP 提供的分库分表规则进行库表分片和路由； SkyWalking OAP 负责生成分库分表规则并且像操作 MySQL 一样对虚拟逻辑库执行 DDL 和 DML； 适用场景 希望使用 MySQL 作为存储，随着业务规模的增长，单表模式已经无法满足性能需要。\nSkyWalking 分库分表逻辑 分库分表逻辑通过注解 @SQLDatabase.Sharding 对 SkyWalking 中的数据模型 Model 进行定义：\n@interface Sharding { ShardingAlgorithm shardingAlgorithm(); String dataSourceShardingColumn() default \u0026#34;\u0026#34;; String tableShardingColumn() default \u0026#34;\u0026#34;; } 其中：\nshardingAlgorithm：表分片算法\ndataSourceShardingColumn：分库键\ntableShardingColumn：分表键\nSkyWalking 根据注解 @SQLDatabase.Sharding 选择分库键、分表键以及表分片算法对每个表动态生成分片规则通过 DistSQL 操作 Shardingsphere-Proxy 执行规则定义 Shardingsphere-Proxy 根据规则定义进行数据分片。\n分库方式 SkyWalking 对于分库采用统一的方式，路由目标库的数字后缀使用分库键的哈希值取模需要分库的数据库数量，所以路由目标库为：\nds_{dataSourceShardingColumn.hashcode() % dataSourceList.size()} 例如我们有 dataSourceList = ds_0...ds_n，如果\n{dataSourceShardingColumn.hashcode() % dataSourceList.size() = 2} 那么所有数据将会路由到 ds_2 这个数据源节点上。\n分表方式 由于 TTL 机制的存在，分表算法主要根据时间的日期进行分片，分片表的数量是根据 TTL 每天一个表：\n分片表名 = 逻辑表名_时间序列（日期）：{tableName =logicTableName_timeSeries}\n为保证在 TTL 有效期内的数据能够被写入和查询，时间序列将生成当前日期\n{timeSeries = currentDate - TTL +1...currentDate + 1} 例如：如果 TTL=3, currentDate = 20220907，则分片表为:\nlogicTableName_20220905 logicTableName_20220906 logicTableName_20220907 logicTableName_20220908 SkyWalking 提供了多种不同的分表算法用于不同的数据模型：\n算法名称 分片说明 分片键时间精度要求 典型应用数据模型 NO_SHARDING 不做任何表分片，保持单表模式 / 数据量小无需分片的数据模型 TIME_RELATIVE_ID_SHARDING_ALGORITHM 使用 ID 列中的 time_bucket 按天分片 time_bucket 的精度可以是同一表中的秒、分、小时和天 各类 Metrics 指标 TIME_SEC_RANGE_SHARDING_ALGORITHM 使用 time_bucket 列按天分片 time_bucket 的精度必须是秒 SegmentRecordLogRecord 等 TIME_MIN_RANGE_SHARDING_ALGORITHM 使用 time_bucket 列按天分片 time_bucket 的精度必须是分钟 EndpointTraffic TIME_BUCKET_SHARDING_ALGORITHM 使用 time_bucket 列按天分片 time_bucket 的精度可以是同一个表中的秒、分、小时和天 Service、Instance、Endpoint 调用关系等如 ServiceRelationServerSideMetrics TTL 机制 对于进行分片的表根据 TTL 直接删除 deadline \u0026gt;= timeSeries 的物理表 {deadline = new DateTime().plusDays(-ttl)} TTL 定时器在根据当前日期删除过期表的同时也会根据新日期更新分片规则，通知 ShardingSphere-Proxy 创建新的分片表 对于单表的延续之前的方式，删除 deadline \u0026gt;= time_bucket 的行记录 分片数据存储示例 下面以 segment（Record 类型）和 service_resp_time（Metrics 类型）两个为例说明数据存储的逻辑和物理分布。这里假设 MySQL 为 ds_0 和 ds_1 两个节点。\n注意：以下的存储表结构仅为简化后的存储示例，不表示 SkyWalking 真实的表结构。\nsegment 分片配置为：\n@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.TIME_SEC_RANGE_SHARDING_ALGORITHM, dataSourceShardingColumn = service_id, tableShardingColumn = time_bucket) 逻辑库表结构和实际库表如下图：\nservice_resp_time 分片配置为：\n@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.TIME_RELATIVE_ID_SHARDING_ALGORITHM, tableShardingColumn = id, dataSourceShardingColumn = entity_id) 逻辑库表结构和实际库表如下图：\n如何使用 你可以选择手动或使用 Docker 来运行 Demo。\n手动部署 这里以单节点 SkyWalking OAP 和 Shardingsphere-Proxy 5.1.2 部署为例，集群部署请参考其他相关文档。\n准备好 MySQL 集群\n部署安装并配置 Shardingsphere-Proxy：\nconf/server.yaml，props.proxy-hint-enabled 必须为 true，完整配置可参考这里。 conf/config-sharding.yaml，配置逻辑数据库和 dataSources 列表，dataSource 的名称必须以 ds_为前缀，并且从 ds_0 开始，完整配置可参考这里。 部署安装并配置 SkyWalking OAP：\n设置 OAP 环境变量 ${SW_STORAGE:mysql-sharding} 根据实际部署情况配置连接信息： ${SW_JDBC_URL} ${SW_DATA_SOURCE_USER} ${SW_DATA_SOURCE_PASSWORD} 注意：连接信息需对应 Shardingsphere-Proxy 虚拟数据库。\n将 Shardingsphere-Proxy 中 conf/config-sharding.yaml 配置的数据源名称配置在 ${SW_JDBC_SHARDING_DATA_SOURCES} 中，用 , 分割\n启动 MySQL 集群\n启动 Shardingsphere-Proxy\n启动 SkyWalking OAP\n使用 Docker 运行 Demo GitHub 资源库提供了一个基于 Docker 完整可运行的 demo：skywalking-mysql-sharding-demo，可以快速尝试实际运行效果。\n其中部署包含：\noap 服务 1 个，Metrics 和 Record 数据的 TTL 均设为 2 天 sharding-proxy 服务 1 个版本为 5.1.2，对外端口为 13307，创建的逻辑库名称为 swtest mysql 服务 2 个，对外端口分别为 3306，3307，在 sharding-proxy 的 conf/config-sharding.yaml 中配置为 ds_0 和 ds_1 provider 服务 1 个（模拟业务程序用于验证 trace 和 metrics 等数据），对外端口为 9090 consumer 服务 1 个（模拟业务程序用于验证 trace 和 metrics 等数据），对外端口为 9092 将 Demo 程序获取到本地后，在 skywalking-mysql-sharding-demo 目录下直接运行：\ndocker-compose up -d 注意：初次启动由于拉取镜像和新建所有表可能需要一定的时间。\n所有服务启动完成之后可以通过数据库工具查看 sharding-proxy 逻辑表创建情况，以及两个 MySQL 库中实际的物理分片表创建情况。也可以连接 sharding-proxy 逻辑库 swtest 查看数据查询路由情况，如：\nPREVIEW SELECT * FROM SEGMENT 显示结果如下：\nDemo 提供的模拟业务程序可以通过请求 consumer 服务模拟业务请求，用于验证各类型数据分布：\ncurl http://127.0.0.1:9092/info 总结 在这篇文章中我们详细介绍了 SkyWalking 基于 ShardingSphere-Proxy 的 MySQL-Sharding 存储特性的部署架构、适应场景、核心分库分表逻辑以及 TTL 机制，并提供了运行后的数据存储示例和详细部署配置步骤以便大家快速理解上手。SkyWalking 提供了多种存储方式以供选择，如果你目前的需求如本文所述，欢迎使用该新特性。\n","excerpt":"\u003cp\u003eApache SkyWalking 作为一个分布式系统的应用性能监控工具，它观察服务网格中的指标、日志、痕迹和事件。其中 SkyWalking OAP 高性能的数据流处理架构能够实时处理庞大的数据流量 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/skywalking-shardingsphere-proxy/","title":"SkyWalking 基于 ShardingSphere-Proxy 的 MySQL-Sharding 分库分表的存储特性介绍"},{"body":"SkyWalking Kubernetes Helm Chart 4.4.0 is released. Go to downloads page to find release tars.\n[Breaking Change]: remove .Values.oap.initEs, there is no need to use this to control whether to run init job anymore, SkyWalking Helm Chart automatically delete the init job when installing/upgrading. [Breaking Change]: remove files/config.d mechanism and use values.yaml files to put the configurations to override default config files in the /skywalking/config folder, using files/config.d is very limited and you have to clone the source codes if you want to use this mechanism, now you can simply use our Docker Helm Chart to install. Refactor oap init job, and support postgresql storage. Upgrade ElasticSearch Helm Chart dependency version. ","excerpt":"\u003cp\u003eSkyWalking Kubernetes Helm Chart 4.4.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e[ …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-kubernetes-helm-chart-4.4.0/","title":"Release Apache SkyWalking Kubernetes Helm Chart 4.4.0"},{"body":"SkyWalking PHP 0.2.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed Update PECL user by @heyanlong in https://github.com/apache/skywalking-php/pull/12 Start up 0.2.0 by @heyanlong in https://github.com/apache/skywalking-php/pull/13 Update compiling project document. by @jmjoy in https://github.com/apache/skywalking-php/pull/14 Add PDO plugin, and switch unix datagram to stream. by @jmjoy in https://github.com/apache/skywalking-php/pull/15 Update readme about creating issue. by @jmjoy in https://github.com/apache/skywalking-php/pull/17 Fix package.xml role error by @heyanlong in https://github.com/apache/skywalking-php/pull/16 Add swoole support. by @jmjoy in https://github.com/apache/skywalking-php/pull/19 Add .fleet to .gitignore by @heyanlong in https://github.com/apache/skywalking-php/pull/20 [Feature] Add Mysql Improved Extension by @heyanlong in https://github.com/apache/skywalking-php/pull/18 Add predis plugin. by @jmjoy in https://github.com/apache/skywalking-php/pull/21 Take care of PDO false and DSN tailing semicolons. by @phanalpha in https://github.com/apache/skywalking-php/pull/22 Add container by @heyanlong in https://github.com/apache/skywalking-php/pull/23 Save PDO exceptions. by @phanalpha in https://github.com/apache/skywalking-php/pull/24 Update minimal supported PHP version to 7.2. by @jmjoy in https://github.com/apache/skywalking-php/pull/25 Utilize UnixListener for the worker process to accept reports. by @phanalpha in https://github.com/apache/skywalking-php/pull/26 Kill the worker on module shutdown. by @phanalpha in https://github.com/apache/skywalking-php/pull/28 Add plugin for memcached. by @jmjoy in https://github.com/apache/skywalking-php/pull/27 Upgrade rust mini version to 1.65. by @jmjoy in https://github.com/apache/skywalking-php/pull/30 Add plugin for phpredis. by @jmjoy in https://github.com/apache/skywalking-php/pull/29 Add missing request_id. by @jmjoy in https://github.com/apache/skywalking-php/pull/31 Adapt virtual cache. by @jmjoy in https://github.com/apache/skywalking-php/pull/32 Fix permission denied of unix socket. by @jmjoy in https://github.com/apache/skywalking-php/pull/33 Bump to 0.2.0. by @jmjoy in https://github.com/apache/skywalking-php/pull/34 New Contributors @phanalpha made their first contribution in https://github.com/apache/skywalking-php/pull/22 Full Changelog: https://github.com/apache/skywalking-php/compare/v0.1.0...v0.2.0\nPECL https://pecl.php.net/package/skywalking_agent/0.2.0\n","excerpt":"\u003cp\u003eSkyWalking PHP 0.2.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-php-0-2-0/","title":"Release Apache SkyWalking PHP 0.2.0"},{"body":"This is an official annoucement from SkyWalking team.\nDue to the Plan to End-of-life(EOL) all v8 releases in Nov. 2022 had been posted in 3 months, SkyWalking community doesn\u0026rsquo;t received any objection or a proposal about releasing a new patch version.\nNow, it is time to end the v8 series. All documents of v8 are not going to be hosted on the website. You only could find the artifacts and source codes from the Apache\u0026rsquo;s archive repository. The documents of each version are included in /docs/ folder in the source tars.\nThe SkyWalking community would reject the bug reports and release proposal due to its End-of-life(EOL) status. v9 provides more powerful features and covers all capabilities of the latest v8. Recommend upgrading to the latest.\nV8 was a memorable and significative release series, which makes the project globally adopted. It brought dev community scale up to over 500 contributors.\nWe want to highlight and thank all those contributors and end users again. You made today\u0026rsquo;s SkyWalking.\nWelcome more contributors and users to join the community, to contribute your ideas, experiences, and feedback. We need you to improve and enhance the project to a higher level.\n","excerpt":"\u003cp\u003eThis is an official annoucement from SkyWalking team.\u003c/p\u003e\n\u003cp\u003eDue to the \u003ca href=\"../deprecate-v8/index.md\"\u003ePlan to End-of-life(EOL) all v8 …\u003c/a\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/events/v8-eol/","title":"SkyWalking v8 OAP server End-of-life(EOL)"},{"body":"SkyWalking BanyanDB 0.2.0 is released. Go to downloads page to find release tars.\nFeatures Command line tool: bydbctl. Retention controller. Full-text searching. TopN aggregation. Add RESTFul style APIs based on gRPC gateway. Add \u0026ldquo;exists\u0026rdquo; endpoints to the schema registry. Support tag-based CRUD of the property. Support index-only tags. Support logical operator(and \u0026amp; or) for the query. Bugs \u0026ldquo;metadata\u0026rdquo; syncing pipeline complains about an \u0026ldquo;unknown group\u0026rdquo;. \u0026ldquo;having\u0026rdquo; semantic inconsistency. \u0026ldquo;tsdb\u0026rdquo; leaked goroutines. Chores \u0026ldquo;tsdb\u0026rdquo; structure optimization. Merge the primary index into the LSM-based index Remove term metadata. Memory parameters optimization. Bump go to 1.19. ","excerpt":"\u003cp\u003eSkyWalking BanyanDB 0.2.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"features\"\u003eFeatures\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eCommand …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-0-2-0/","title":"Release Apache SkyWalking BanyanDB 0.2.0"},{"body":"SkyWalking Java Agent 8.13.0 is released. Go to downloads page to find release tars. Changes by Version\n8.13.0 This release begins to adopt SkyWalking 9.3.0+ Virtual Cache Analysis,Virtual MQ Analysis\nSupport set-type in the agent or plugin configurations Optimize ConfigInitializer to output warning messages when the config value is truncated. Fix the default value of the Map field would merge rather than override by new values in the config. Support to set the value of Map/List field to an empty map/list. Add plugin to support Impala JDBC 2.6.x. Update guava-cache, jedis, memcached, ehcache plugins to adopt uniform tags. Fix Apache ShenYu plugin traceId empty string value. Add plugin to support brpc-java-3.x Update compose-start-script.template to make compatible with new version docker compose Bump up grpc to 1.50.0 to fix CVE-2022-3171 Polish up nats plugin to unify MQ related tags Correct the duration of the transaction span for Neo4J 4.x. Plugin-test configuration.yml dependencies support docker service command field Polish up rabbitmq-5.x plugin to fix missing broker tag on consumer side Polish up activemq plugin to fix missing broker tag on consumer side Enhance MQ plugin relative tests to check key tags not blank. Add RocketMQ test scenarios for version 4.3 - 4.9. No 4.0 - 4.2 release images for testing. Support mannual propagation of tracing context to next operators for webflux. Add MQ_TOPIC and MQ_BROKER tags for RocketMQ consumer\u0026rsquo;s span. Polish up Pulsar plugins to remove unnecessary dynamic value , set peer at consumer side Polish Kafka plugin to set peer at the consumer side. Polish NATS plugin to set peer at the consumer side. Polish ActiveMQ plugin to set peer at the consumer side. Polish RabbitMQ plugin to set peer at the consumer side. Documentation Update configuration doc about overriding default value as empty map/list accordingly. Update plugin dev tags for cache relative tags. Add plugin dev docs for virtual database tags. Add plugin dev docs for virtual MQ tags. Add doc about kafka plugin Manual APIs. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking Java Agent 8.13.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nChanges by …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-java-agent-8-13-0/","title":"Release Apache SkyWalking Java Agent 8.13.0"},{"body":"SkyWalking Client JS 0.9.0 is released. Go to downloads page to find release tars.\nFix custom configurations when the page router changed for SPA. Fix reporting data by navigator.sendbeacon when pages is closed. Bump dependencies. Add Security Notice. Support adding custom tags to spans. Validate custom parameters for register. ","excerpt":"\u003cp\u003eSkyWalking Client JS 0.9.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eFix custom …\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-client-js-0-9-0/","title":"Release Apache SkyWalking Client JS 0.9.0"},{"body":"I am excited to announce a new SkyWalking committer, Yueqin Zhang(GitHub ID, yswdqz). Yueqin entered the SkyWalking community on Jul. 3rd[1], 2022, for the first time. Later, I knew he was invited by Yihao Chen, our committer, who is running an open-source program for students who can\u0026rsquo;t join Summer 2022 due to SkyWalking having limited slots.\nHis first PR[2] for Issue #7420 took 20 days to propose. I believe he took incredibly hard work in his own time. For every PMC member, we all were there. Purely following documents and existing codes to build a new feature is always not easy to start.\nAfter that, we had several private talks, he asked for more possible directions to join the community deeper. Then, I am honored to witness a great landscape extension in SkyWalking feature territory, SkyWalking adopts OpenTelemetry features quickly, and is powered by our powerful MAL and v9 kernel/UI, He built MySQL and PostgreSQL server monitoring, metrics, and slow SQLs collecting(through enhancing LAL with a new layer concept), under a new menu, .\nIt is unbelievable to see his contributions in the main repo, 8 PRs[3], LOC 4,857++, 1,627\u0026ndash;\nMeanwhile, this story continues, he is trying to build A lightweight and APM-oriented SQL parser module[4] under my mentoring. This would be another challenging idea, but also very useful to enhance existing virtual database perf. analyzing.\nI believe this would not be the end for the moment between SkyWalking and him.\nWelcome to join the team.\nReferrer \u0026amp; PMC member, Sheng Wu.\n[1] https://github.com/apache/skywalking/issues/7420#issuecomment-1173061870 [2] https://github.com/apache/skywalking-java/pull/286 [3] https://github.com/apache/skywalking/commits?author=yswdqz [4] https://github.com/apache/skywalking/issues/9661 ","excerpt":"\u003cp\u003eI am excited to announce a new SkyWalking committer, Yueqin Zhang(GitHub ID, yswdqz).\nYueqin entered …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-yueqin-zhang-as-new-committer/","title":"Welcome Yueqin Zhang as a new committer"},{"body":"SkyWalking PHP 0.1.0 is released. Go to downloads page to find release tars.\nWhat's Changed [docs] Update README by @heyanlong in https://github.com/apache/skywalking-php/pull/1 Remove the CI limit first, in order to run CI. by @jmjoy in https://github.com/apache/skywalking-php/pull/3 Setup CI. by @jmjoy in https://github.com/apache/skywalking-php/pull/5 Implementation, with curl support. By @jmjoy in https://github.com/apache/skywalking-php/pull/4 Turn off Swoole support, and fix Makefile. By @jmjoy in https://github.com/apache/skywalking-php/pull/6 Update docs by @heyanlong in https://github.com/apache/skywalking-php/pull/7 Add PECL support. By @jmjoy in https://github.com/apache/skywalking-php/pull/8 Support macOS by replace ipc-channel with socket pair, upgrade dependencies and improve CI. by @jmjoy in https://github.com/apache/skywalking-php/pull/9 Add compile and release docs. By @jmjoy in https://github.com/apache/skywalking-php/pull/10 Update official documentation link. By @jmjoy in https://github.com/apache/skywalking-php/pull/11 New Contributors @heyanlong made their first contribution in https://github.com/apache/skywalking-php/pull/1 @jmjoy made their first contribution in https://github.com/apache/skywalking-php/pull/3 Full Changelog: https://github.com/apache/skywalking-php/commits/v0.1.0\nPECL https://pecl.php.net/package/skywalking_agent/0.1.0\n","excerpt":"\u003cp\u003eSkyWalking PHP 0.1.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat's Changed\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e[docs] …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-php-0-1-0/","title":"Release Apache SkyWalking PHP 0.1.0"},{"body":"Yanlong He (GitHub: heyanlong) is a SkyWalking committer for years. He was working on skyapm-php for years to support the SkyWalking ecosystem. That PHP agent has significant contributions for SkyWalking\u0026rsquo;s users adoption in the PHP landscape. Yanlong keeps active in supporting and maintaining the project to help the community.\nJiemin Xia (GitHub: jmjoy) is a new committer voted in July 2022. He is super active in this year. He took over the maintaince capatbilify from Rei Shimizu, who is too busy in his daily work. He leads on the Rust SDK, and is also a release manager for the Rust SDK.\nRecently, both of them are working with Yanlong He to build a new skywalking PHP agent.\nWe are having our PHP agent v0.1.0 for the community.\nSkyWalking PHP Agent\nNotice, SkyAPM PHP is going to be archived and replaced by SkyWalking PHP agent according to its project maintainer, Yanlong He. Our community would work more closely forward the new PHP agent together.\nLet\u0026rsquo;s welcome and congrats to our 31st and 32nd PMC members, Yanlong He and Jiemin Xia. We are honored to have you.\n","excerpt":"\u003cp\u003eYanlong He (GitHub: \u003ca href=\"https://github.com/heyanlong\"\u003eheyanlong\u003c/a\u003e) is a SkyWalking committer for years.\nHe was working on \u003ca href=\"https://github.com/SkyAPM/SkyAPM-php-sdk\"\u003eskyapm-php\u003c/a\u003e for …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-heyanlong-xiajiemin-join-the-pmc/","title":"Welcome Yanlong He and Jiemin Xia to join the PMC"},{"body":"Background This article will show how to use Apache SkyWalking with eBPF to make network troubleshooting easier in a service mesh environment.\nApache SkyWalking is an application performance monitor tool for distributed systems. It observes metrics, logs, traces, and events in the service mesh environment and uses that data to generate a dependency graph of your pods and services. This dependency graph can provide quick insights into your system, especially when there\u0026rsquo;s an issue.\nHowever, when troubleshooting network issues in SkyWalking\u0026rsquo;s service topology, it is not always easy to pinpoint where the error actually is. There are two reasons for the difficulty:\nTraffic through the Envoy sidecar is not easy to observe. Data from Envoy\u0026rsquo;s Access Log Service (ALS) shows traffic between services (sidecar-to-sidecar), but not metrics on communication between the Envoy sidecar and the service it proxies. Without that information, it is more difficult to understand the impact of the sidecar. There is a lack of data from transport layer (OSI Layer 4) communication. Since services generally use application layer (OSI Layer 7) protocols such as HTTP, observability data is generally restricted to application layer communication. However, the root cause may actually be in the transport layer, which is typically opaque to observability tools. Access to metrics from Envoy-to-service and transport layer communication can make it easier to diagnose service issues. To this end, SkyWalking needs to collect and analyze transport layer metrics between processes inside Kubernetes pods - a task well suited to eBPF. We investigated using eBPF for this purpose and present our results and a demo below.\nMonitoring Kubernetes Networks with eBPF With its origins as the Extended Berkeley Packet Filter, eBPF is a general purpose mechanism for injecting and running your own code into the Linux kernel and is an excellent tool for monitoring network traffic in Kubernetes Pods. In the next few sections, we'll provide an overview of how to use eBPF for network monitoring as background for introducing Skywalking Rover, a metrics collector and profiler powered by eBPF to diagnose CPU and network performance.\nHow Applications and the Network Interact Interactions between the application and the network can generally be divided into the following steps from higher to lower levels of abstraction:\nUser Code: Application code uses high-level network libraries in the application stack to exchange data across the network, like sending and receiving HTTP requests. Network Library: When the network library receives a network request, it interacts with the language API to send the network data. Language API: Each language provides an API for operating the network, system, etc. When a request is received, it interacts with the system API. In Linux, this API is called syscalls. Linux API: When the Linux kernel receives the request through the API, it communicates with the socket to send the data, which is usually closer to an OSI Layer 4 protocol, such as TCP, UDP, etc. Socket Ops: Sending or receiving the data to/from the NIC. Our hypothesis is that eBPF can monitor the network. There are two ways to implement the interception: User space (uprobe) or Kernel space (kprobe). The table below summarizes the differences.\nPros Cons uprobe •\tGet more application-related contexts, such as whether the current request is HTTP or HTTPS.•\tRequests and responses can be intercepted by a single method •\tData structures can be unstable, so it is more difficult to get the desired data. •\tImplementation may differ between language/library versions. •\tDoes not work in applications without symbol tables. kprobe •\tAvailable for all languages. •\tThe data structure and methods are stable and do not require much adaptation. •\tEasier correlation with underlying data, such as getting the destination address of TCP, OSI Layer 4 protocol metrics, etc. •\tA single request and response may be split into multiple probes. •\tContextual information is not easy to get for stateful requests. For example header compression in HTTP/2. For the general network performance monitor, we chose to use the kprobe (intercept the syscalls) for the following reasons:\nIt\u0026rsquo;s available for applications written in any programming language, and it\u0026rsquo;s stable, so it saves a lot of development/adaptation costs. It can be correlated with metrics from the system level, which makes it easier to troubleshoot. As a single request and response are split into multiple probes, we can use technology to correlate them. For contextual information, It\u0026rsquo;s usually used in OSI Layer 7 protocol network analysis. So, if we just monitor the network performance, then they can be ignored. Kprobes and network monitoring Following the network syscalls of Linux documentation, we can implement network monitoring by intercepting two types of methods: socket operations and send/receive methods.\nSocket Operations When accepting or connecting with another socket, we can get the following information:\nConnection information: Includes the remote address from the connection which helps us to understand which pod is connected. Connection statics: Includes basic metrics from sockets, such as round-trip time (RTT), lost packet count in TCP, etc. Socket and file descriptor (FD) mapping: Includes the relationship between the Linux file descriptor and socket object. It is useful when sending and receiving data through a Linux file descriptor. Send/Receive The interface related to sending or receiving data is the focus of performance analysis. It mainly contains the following parameters:\nSocket file descriptor: The file descriptor of the current operation corresponding to the socket. Buffer: The data sent or received, passed as a byte array. Based on the above parameters, we can analyze the following data:\nBytes: The size of the packet in bytes. Protocol: The protocol analysis according to the buffer data, such as HTTP, MySQL, etc. Execution Time: The time it takes to send/receive the data. At this point (Figure 1) we can analyze the following steps for the whole lifecycle of the connection:\nConnect/Accept: When the connection is created. Transform: Sending and receiving data on the connection. Close: When the connection is closed. Figure 1\nProtocol and TLS The previous section described how to analyze connections using send or receive buffer data. For example, following the HTTP/1.1 message specification to analyze the connection. However, this does not work for TLS requests/responses.\nFigure 2\nWhen TLS is in use, the Linux Kernel transmits data encrypted in user space. In the figure above, The application usually transmits SSL data through a third-party library (such as OpenSSL). For this case, the Linux API can only get the encrypted data, so it cannot recognize any higher layer protocol. To decrypt inside eBPF, we need to follow these steps:\nRead unencrypted data through uprobe: Compatible multiple languages, using uprobe to capture the data that is not encrypted before sending or after receiving. In this way, we can get the original data and associate it with the socket. Associate with socket: We can associate unencrypted data with the socket. OpenSSL Use case For example, the most common way to send/receive SSL data is to use OpenSSL as a shared library, specifically the SSL_read and SSL_write methods to submit the buffer data with the socket.\nFollowing the documentation, we can intercept these two methods, which are almost identical to the API in Linux. The source code of the SSL structure in OpenSSL shows that the Socket FD exists in the BIO object of the SSL structure, and we can get it by the offset.\nIn summary, with knowledge of how OpenSSL works, we can read unencrypted data in an eBPF function.\nIntroducing SkyWalking Rover, an eBPF-based Metrics Collector and Profiler SkyWalking Rover introduces the eBPF network profiling feature into the SkyWalking ecosystem. It\u0026rsquo;s currently supported in a Kubernetes environment, so must be deployed inside a Kubernetes cluster. Once the deployment is complete, SkyWalking Rover can monitor the network for all processes inside a given Pod. Based on the monitoring data, SkyWalking can generate the topology relationship diagram and metrics between processes.\nTopology Diagram The topology diagram can help us understand the network access between processes inside the same Pod, and between the process and external environment (other Pod or service). Additionally, it can identify the data direction of traffic based on the line flow direction.\nIn Figure 3 below, all nodes within the hexagon are the internal process of a Pod, and nodes outside the hexagon are externally associated services or Pods. Nodes are connected by lines, which indicate the direction of requests or responses between nodes (client or server). The protocol is indicated on the line, and it\u0026rsquo;s either HTTP(S), TCP, or TCP(TLS). Also, we can see in this figure that the line between Envoy and Python applications is bidirectional because Envoy intercepts all application traffic.\nFigure 3\nMetrics Once we recognize the network call relationship between processes through the topology, we can select a specific line and view the TCP metrics between the two processes.\nThe diagram below (Figure 4) shows the metrics of network monitoring between two processes. There are four metrics in each line. Two on the left side are on the client side, and two on the right side are on the server side. If the remote process is not in the same Pod, only one side of the metrics is displayed.\nFigure 4\nThe following two metric types are available:\nCounter: Records the total number of data in a certain period. Each counter contains the following data: a. Count: Execution count. b. Bytes: Packet size in bytes. c. Execution time: Execution duration. Histogram: Records the distribution of data in the buckets. Based on the above data types, the following metrics are exposed:\nName Type Unit Description Write Counter and histogram Millisecond The socket write counter. Read Counter and histogram Millisecond The socket read counter. Write RTT Counter and histogram Microsecond The socket write round trip time (RTT) counter. Connect Counter and histogram Millisecond The socket connect/accept with another server/client counter. Close Counter and histogram Millisecond The socket with other socket counter. Retransmit Counter Millisecond The socket retransmit package counter. Drop Counter Millisecond The socket drop package counter. Demo In this section, we demonstrate how to perform network profiling in the service mesh. To follow along, you will need a running Kubernetes environment.\nNOTE: All commands and scripts are available in this GitHub repository.\nInstall Istio Istio is the most widely deployed service mesh, and comes with a complete demo application that we can use for testing. To install Istio and the demo application, follow these steps:\nInstall Istio using the demo configuration profile. Label the default namespace, so Istio automatically injects Envoy sidecar proxies when we\u0026rsquo;ll deploy the application. Deploy the bookinfo application to the cluster. Deploy the traffic generator to generate some traffic to the application. export ISTIO_VERSION=1.13.1 # install istio istioctl install -y --set profile=demo kubectl label namespace default istio-injection=enabled # deploy the bookinfo applications kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/bookinfo-gateway.yaml kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/destination-rule-all.yaml kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/virtual-service-all-v1.yaml # generate traffic kubectl apply -f https://raw.githubusercontent.com/mrproliu/skywalking-network-profiling-demo/main/resources/traffic-generator.yaml Install SkyWalking The following will install the storage, backend, and UI needed for SkyWalking:\ngit clone https://github.com/apache/skywalking-kubernetes.git cd skywalking-kubernetes cd chart helm dep up skywalking helm -n istio-system install skywalking skywalking \\ --set fullnameOverride=skywalking \\ --set elasticsearch.minimumMasterNodes=1 \\ --set elasticsearch.imageTag=7.5.1 \\ --set oap.replicas=1 \\ --set ui.image.repository=apache/skywalking-ui \\ --set ui.image.tag=9.2.0 \\ --set oap.image.tag=9.2.0 \\ --set oap.envoy.als.enabled=true \\ --set oap.image.repository=apache/skywalking-oap-server \\ --set oap.storageType=elasticsearch \\ --set oap.env.SW_METER_ANALYZER_ACTIVE_FILES=\u0026#39;network-profiling\u0026#39; Install SkyWalking Rover SkyWalking Rover is deployed on every node in Kubernetes, and it automatically detects the services in the Kubernetes cluster. The network profiling feature has been released in the version 0.3.0 of SkyWalking Rover. When a network monitoring task is created, the SkyWalking rover sends the data to the SkyWalking backend.\nkubectl apply -f https://raw.githubusercontent.com/mrproliu/skywalking-network-profiling-demo/main/resources/skywalking-rover.yaml Start the Network Profiling Task Once all deployments are completed, we must create a network profiling task for a specific instance of the service in the SkyWalking UI.\nTo open SkyWalking UI, run:\nkubectl port-forward svc/skywalking-ui 8080:80 --namespace istio-system Currently, we can select the specific instances that we wish to monitor by clicking the Data Plane item in the Service Mesh panel and the Service item in the Kubernetes panel.\nIn the figure below, we have selected an instance with a list of tasks in the network profiling tab. When we click the start button, the SkyWalking Rover starts monitoring this instance\u0026rsquo;s network.\nFigure 5\nDone! After a few seconds, you will see the process topology appear on the right side of the page.\nFigure 6\nWhen you click on the line between processes, you can see the TCP metrics between the two processes.\nFigure 7\nConclusion In this article, we detailed a problem that makes troubleshooting service mesh architectures difficult: lack of context between layers in the network stack. These are the cases when eBPF begins to really help with debugging/productivity when existing service mesh/envoy cannot. Then, we researched how eBPF could be applied to common communication, such as TLS. Finally, we demo the implementation of this process with SkyWalking Rover.\nFor now, we have completed the performance analysis for OSI layer 4 (mostly TCP). In the future, we will also introduce the analysis for OSI layer 7 protocols like HTTP.\nGet Started with Istio To get started with service mesh today, Tetrate Istio Distro is the easiest way to install, manage, and upgrade Istio. It provides a vetted upstream distribution of Istio that\u0026rsquo;s tested and optimized for specific platforms by Tetrate plus a CLI that facilitates acquiring, installing, and configuring multiple Istio versions. Tetrate Istio Distro also offers FIPS certified Istio builds for FedRAMP environments.\nFor enterprises that need a unified and consistent way to secure and manage services and traditional workloads across complex, heterogeneous deployment environments, we offer Tetrate Service Bridge, our flagship edge-to-workload application connectivity platform built on Istio and Envoy.\nContact us to learn more.\nAdditional Resources SkyWalking Github Repo SkyWalking Rover Github Repo SkyWalking Rover Documentation Pinpoint Service Mesh Critical Performance impact by using eBPF blog post Apache SkyWalking with Native eBPF Agent presentation eBPF hook overview ","excerpt":"\u003ch2 id=\"background\"\u003eBackground\u003c/h2\u003e\n\u003cp\u003eThis article will show how to use \u003ca href=\"https://github.com/apache/skywalking\"\u003eApache SkyWalking\u003c/a\u003e with \u003ca href=\"https://ebpf.io/what-is-ebpf/\"\u003eeBPF\u003c/a\u003e to make network …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/diagnose-service-mesh-network-performance-with-ebpf/","title":"Diagnose Service Mesh Network Performance with eBPF"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/ebpf/","title":"EBPF"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/performance/","title":"Performance"},{"body":"本文将展示如何利用 Apache SkyWalking 与 eBPF，使服务网格下的网络故障排除更加容易。\nApache SkyWalking 是一个分布式系统的应用性能监控工具。它观察服务网格中的指标、日志、痕迹和事件，并使用这些数据来生成 pod 和服务的依赖图。这个依赖关系图可以帮助你快速系统，尤其是在出现问题的时候。\n然而，在排除 SkyWalking 服务拓扑中的网络问题时，确定错误的实际位置有时候并不容易。造成这种困难的原因有两个：\n通过 Envoy sidecar 的流量并不容易观察：来自 Envoy 的访问日志服务（ALS）的数据显示了服务之间的流量（sidecar-to-sidecar），但没有关于 Envoy sidecar 和它代理的服务之间的通信指标。如果没有这些信息，就很难理解 sidecar 的影响。 缺乏来自传输层（OSI 第 4 层）通信的数据：由于服务通常使用应用层（OSI 第 7 层）协议，如 HTTP，可观测性数据通常被限制在应用层通信中。然而，根本原因可能实际上是在传输层，而传输层对可观测性工具来说通常是不透明的。 获取 Envoy-to-service 和传输层通信的指标，可以更容易诊断服务问题。为此，SkyWalking 需要收集和分析 Kubernetes pod 内进程之间的传输层指标 —— 这项任务很适合 eBPF。我们调查了为此目的使用 eBPF 的情况，并在下面介绍了我们的结果和演示。\n用 eBPF 监控 Kubernetes 网络 eBPF 起源于 Extended Berkeley Packet Filter，是一种通用的机制，可以在 Linux 内核中注入和运行自己的代码，是监测 Kubernetes Pod 中网络流量的优秀工具。在接下来的几节中，我们将概述如何使用 eBPF 进行网络监控，作为介绍 Skywalking Rover 的背景，这是一个由 eBPF 驱动的指标收集器和分析器，用于诊断 CPU 和网络性能。\n应用程序和网络如何相互作用 应用程序和网络之间的互动一般可分为以下步骤，从较高的抽象层次到较低的抽象层次：\n用户代码：应用程序代码使用应用程序堆栈中的高级网络库，在网络上交换数据，如发送和接收 HTTP 请求。 网络库：当网络库收到网络请求时，它与语言 API 进行交互以发送网络数据。 语言 API：每种语言都提供了一个操作网络、系统等的 API。当收到一个请求时，它与系统的 API 进行交互。在 Linux 中，这个 API 被称为系统调用（syscalls）。 Linux API：当 Linux 内核通过 API 收到请求时，它与套接字进行通信以发送数据，这通常更接近于 OSI 第四层协议，如 TCP、UDP 等。 Socket Ops：向 / 从网卡发送或接收数据。 我们的假设是，eBPF 可以监控网络。有两种方法可以实现拦截：用户空间（uprobe）或内核空间（kprobe）。下表总结了两者的区别。\n方式 优点 缺点 uprobe • 获取更多与应用相关的上下文，例如当前请求是 HTTP 还是 HTTPS。 • 请求和响应可以通过一个方法来截获。 • 数据结构可能是不稳定的，所以更难获得所需的数据。 • 不同语言/库版本的实现可能不同。 • 在没有符号表的应用程序中不起作用。 kprobe • 可用于所有语言。 • 数据结构和方法很稳定，不需要太多调整。 • 更容易与底层数据相关联，如获得 TCP 的目标地址、OSI 第四层协议指标等。 • 一个单一的请求和响应可能被分割成多个 probe。 • 对于有状态的请求，上下文信息不容易得到。例如 HTTP/2 中的头压缩。 对于一般的网络性能监控，我们选择使用 kprobe（拦截系统调用），原因如下：\n它可用于用任何编程语言编写的应用程序，而且很稳定，所以可以节省大量的开发 / 适应成本。 它可以与系统层面的指标相关联，这使得故障排除更加容易。 由于一个请求和响应被分割成多个 probe，我们可以利用技术将它们关联起来。 对于背景信息，它通常用于 OSI 第七层协议网络分析。因此，如果我们只是监测网络性能，那么它们可以被忽略。 Kprobes 和网络监控 按照 Linux 文档中的网络系统调用，我们可以通过两类拦截方法实现网络监控：套接字操作和发送 / 接收方法。\n套接字操作 当接受或与另一个套接字连接时，我们可以得到以下信息：\n连接信息：包括来自连接的远程地址，这有助于我们了解哪个 pod 被连接。 连接统计 ：包括来自套接字的基本指标，如往返时间（RTT）、TCP 的丢包数等。 套接字和文件描述符（FD）的映射：包括 Linux 文件描述符和套接字对象之间的关系。在通过 Linux 文件描述符发送和接收数据时，它很有用。 发送 / 接收 与发送或接收数据有关的接口是性能分析的重点。它主要包含以下参数：\nSocket 文件描述符：当前操作对应的套接字的文件描述符。 缓冲区：发送或接收的数据，以字节数组形式传递。 基于上述参数，我们可以分析以下数据：\n字节：数据包的大小，以字节为单位。 协议：根据缓冲区的数据进行协议分析，如 HTTP、MySQL 等。 执行时间：发送 / 接收数据所需的时间。 在这一点上（图 1），我们可以分析出连接的整个生命周期的以下步骤：\n连接 / 接受：当连接被创建时。 转化：在连接上发送和接收数据。 关闭：当连接被关闭时。 图 1\n协议和 TLS 上一节描述了如何使用发送或接收缓冲区数据来分析连接。例如，遵循 HTTP/1.1 消息规范来分析连接。然而，这对 TLS 请求 / 响应不起作用。\n图 2\n当使用 TLS 时，Linux 内核在用户空间中传输加密的数据。在上图中，应用程序通常通过第三方库（如 OpenSSL）传输 SSL 数据。对于这种情况，Linux API 只能得到加密的数据，所以它不能识别任何高层协议。为了在 eBPF 内部解密，我们需要遵循以下步骤：\n通过 uprobe 读取未加密的数据：兼容多种语言，使用 uprobe 来捕获发送前或接收后没有加密的数据。通过这种方式，我们可以获得原始数据并将其与套接字联系起来。 与套接字关联：我们可以将未加密的数据与套接字关联。 OpenSSL 用例 例如，发送 / 接收 SSL 数据最常见的方法是使用 OpenSSL 作为共享库，特别是 SSL_read 和 SSL_write 方法，以提交缓冲区数据与套接字。\n按照文档，我们可以截获这两种方法，这与 Linux 中的 API 几乎相同。OpenSSL 中 SSL 结构的源代码显示， Socket FD 存在于 SSL 结构的 BIO 对象中，我们可以通过 offset 得到它。\n综上所述，通过对 OpenSSL 工作原理的了解，我们可以在一个 eBPF 函数中读取未加密的数据。\nSkyWalking Rover—— 基于 eBPF 的指标收集器和分析器 SkyWalking Rover 在 SkyWalking 生态系统中引入了 eBPF 网络分析功能。目前已在 Kubernetes 环境中得到支持，所以必须在 Kubernetes 集群内部署。部署完成后，SkyWalking Rover 可以监控特定 Pod 内所有进程的网络。基于监测数据，SkyWalking 可以生成进程之间的拓扑关系图和指标。\n拓扑结构图 拓扑图可以帮助我们了解同一 Pod 内的进程之间以及进程与外部环境（其他 Pod 或服务）之间的网络访问情况。此外，它还可以根据线路的流动方向来确定流量的数据方向。\n在下面的图 3 中，六边形内的所有节点都是一个 Pod 的内部进程，六边形外的节点是外部关联的服务或 Pod。节点由线连接，表示节点之间的请求或响应方向（客户端或服务器）。线条上标明了协议，它是 HTTP (S)、TCP 或 TCP (TLS)。另外，我们可以在这个图中看到，Envoy 和 Python 应用程序之间的线是双向的，因为 Envoy 拦截了所有的应用程序流量。\n图 3\n度量 一旦我们通过拓扑结构认识到进程之间的网络调用关系，我们就可以选择一个特定的线路，查看两个进程之间的 TCP 指标。\n下图（图4）显示了两个进程之间网络监控的指标。每行有四个指标。左边的两个是在客户端，右边的两个是在服务器端。如果远程进程不在同一个 Pod 中，则只显示一边的指标。\n图 4\n有以下两种度量类型。\n计数器（Counter）：记录一定时期内的数据总数。每个计数器包含以下数据。 计数：执行次数。 字节：数据包大小，以字节为单位。 执行时间：执行时间。 柱状图（Histogram）：记录数据在桶中的分布。 基于上述数据类型，暴露了以下指标：\n名称 类型 单位 描述 Write 计数器和柱状图 毫秒 套接字写计数器。 Read 计数器和柱状图 毫秒 套接字读计数器。 Write RTT 计数器和柱状图 微秒 套接字写入往返时间（RTT）计数器。 Connect 计数器和柱状图 毫秒 套接字连接/接受另一个服务器/客户端的计数器。 Close 计数器和柱状图 毫秒 有其他套接字的计数器。 Retransmit 计数器 毫秒 套接字重发包计数器 Drop 计数器 毫秒 套接字掉包计数器。 演示 在本节中，我们将演示如何在服务网格中执行网络分析。要跟上进度，你需要一个正在运行的 Kubernetes 环境。\n注意：所有的命令和脚本都可以在这个 GitHub 资源库中找到。\n安装 Istio Istio是最广泛部署的服务网格，并附带一个完整的演示应用程序，我们可以用来测试。要安装 Istio 和演示应用程序，请遵循以下步骤：\n使用演示配置文件安装 Istio。 标记 default 命名空间，所以当我们要部署应用程序时，Istio 会自动注入 Envoy 的 sidecar 代理。 将 bookinfo 应用程序部署到集群上。 部署流量生成器，为应用程序生成一些流量。 export ISTIO_VERSION=1.13.1 # 安装 istio istioctl install -y --set profile=demo kubectl label namespace default istio-injection=enabled # 部署 bookinfo 应用程序 kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/bookinfo-gateway.yaml kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/destination-rule-all.yaml kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/virtual-service-all-v1.yaml # 产生流量 kubectl apply -f https://raw.githubusercontent.com/mrproliu/skywalking-network-profiling-demo/main/resources/traffic-generator.yaml 安装 SkyWalking 下面将安装 SkyWalking 所需的存储、后台和用户界面。\ngit clone https://github.com/apache/skywalking-kubernetes.git cd skywalking-kubernetes cd chart helm dep up skywalking helm -n istio-system install skywalking skywalking \\ --set fullnameOverride=skywalking \\ --set elasticsearch.minimumMasterNodes=1 \\ --set elasticsearch.imageTag=7.5.1 \\ --set oap.replicas=1 \\ --set ui.image.repository=apache/skywalking-ui \\ --set ui.image.tag=9.2.0 \\ --set oap.image.tag=9.2.0 \\ --set oap.envoy.als.enabled=true \\ --set oap.image.repository=apache/skywalking-oap-server \\ --set oap.storageType=elasticsearch \\ --set oap.env.SW_METER_ANALYZER_ACTIVE_FILES=\u0026#39;network-profiling\u0026#39; 安装 SkyWalking Rover SkyWalking Rover 部署在 Kubernetes 的每个节点上，它自动检测 Kubernetes 集群中的服务。网络剖析功能已经在 SkyWalking Rover 的 0.3.0 版本中发布。当网络监控任务被创建时，SkyWalking Rover 会将数据发送到 SkyWalking 后台。\nkubectl apply -f https://raw.githubusercontent.com/mrproliu/skywalking-network-profiling-demo/main/resources/skywalking-rover.yaml 启动网络分析任务 一旦所有部署完成，我们必须在 SkyWalking UI 中为服务的特定实例创建一个网络分析任务。\n要打开 SkyWalking UI，请运行：\nkubectl port-forward svc/skywalking-ui 8080:80 --namespace istio-system 目前，我们可以通过点击服务网格面板中的数据平面项目和 Kubernetes 面板中的服务项目来选择我们想要监控的特定实例。\n在下图中，我们选择了一个实例，在网络剖析标签里有一个任务列表。当我们点击启动按钮时，SkyWalking Rover 开始监测这个实例的网络。\n图 5\n完成 几秒钟后，你会看到页面的右侧出现进程拓扑结构。\n图 6\n当你点击进程之间的线时，你可以看到两个进程之间的 TCP 指标。\n图 7\n总结 在这篇文章中，我们详细介绍了一个使服务网格故障排除困难的问题：网络堆栈中各层之间缺乏上下文。这些情况下，当现有的服务网格 /envoy 不能时，eBPF 开始真正帮助调试 / 生产。然后，我们研究了如何将 eBPF 应用于普通的通信，如 TLS。最后，我们用 SkyWalking Rover 演示了这个过程的实现。\n目前，我们已经完成了对 OSI 第四层（主要是 TCP）的性能分析。在未来，我们还将介绍对 OSI 第 7 层协议的分析，如 HTTP。\n开始使用 Istio 开始使用服务网格，Tetrate Istio Distro 是安装、管理和升级 Istio 的最简单方法。它提供了一个经过审查的 Istio 上游发布，由 Tetrate 为特定平台进行测试和优化，加上一个 CLI，方便获取、安装和配置多个 Istio 版本。Tetrate Istio Distro 还为 FedRAMP 环境提供 FIPS 认证的 Istio 构建。\n对于需要以统一和一致的方式在复杂的异构部署环境中保护和管理服务和传统工作负载的企业，我们提供 Tetrate Service Bridge，这是我们建立在 Istio 和 Envoy 上的旗舰工作负载应用连接平台。\n联系我们以了解更多。\n其他资源 SkyWalking Github Repo SkyWalking Rover Github Repo SkyWalking Rover 文件 通过使用 eBPF 博文准确定位服务网格关键性能影响 Apache SkyWalking 与本地 eBPF 代理的介绍 eBPF hook概述 ","excerpt":"\u003cp\u003e本文将展示如何利用 \u003ca href=\"https://github.com/apache/skywalking\"\u003eApache SkyWalking\u003c/a\u003e 与 \u003ca href=\"https://ebpf.io/what-is-ebpf/\"\u003eeBPF\u003c/a\u003e，使服务网格下的网络故障排除更加容易。\u003c/p\u003e\n\u003cp\u003eApache SkyWalking 是一个分布式系统的应用性能监控工具。它观察服务网格中的指 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/diagnose-service-mesh-network-performance-with-ebpf/","title":"使用 eBPF 诊断服务网格网络性能"},{"body":"SkyWalking CLI 0.11.0 is released. Go to downloads page to find release tars.\nAdd .github/scripts to release source tarball by @kezhenxu94 in https://github.com/apache/skywalking-cli/pull/140 Let the eBPF profiling could performs by service level by @mrproliu in https://github.com/apache/skywalking-cli/pull/141 Add the sub-command for estimate the process scale by @mrproliu in https://github.com/apache/skywalking-cli/pull/142 feature: update install.sh version regex by @Alexxxing in https://github.com/apache/skywalking-cli/pull/143 Update the commands relate to the process by @mrproliu in https://github.com/apache/skywalking-cli/pull/144 Add layer to event related commands by @fgksgf in https://github.com/apache/skywalking-cli/pull/145 Add layer to events.graphql by @fgksgf in https://github.com/apache/skywalking-cli/pull/146 Add layer field to alarms.graphql by @fgksgf in https://github.com/apache/skywalking-cli/pull/147 Upgrade crypto lib to fix cve by @kezhenxu94 in https://github.com/apache/skywalking-cli/pull/148 Remove layer field in the instance and process commands by @mrproliu in https://github.com/apache/skywalking-cli/pull/149 Remove duration flag in profiling ebpf schedules by @mrproliu in https://github.com/apache/skywalking-cli/pull/150 Remove total field in trace list and logs list commands by @mrproliu in https://github.com/apache/skywalking-cli/pull/152 Remove total field in event list, browser logs, alarm list commands. by @mrproliu in https://github.com/apache/skywalking-cli/pull/153 Add aggregate flag in profiling ebpf analysis commands by @mrproliu in https://github.com/apache/skywalking-cli/pull/154 event: fix event query should query all types by default by @kezhenxu94 in https://github.com/apache/skywalking-cli/pull/155 Fix a possible lint error and update CI lint version by @JarvisG495 in https://github.com/apache/skywalking-cli/pull/156 Add commands for support network profiling by @mrproliu in https://github.com/apache/skywalking-cli/pull/158 Add the components field in the process relation by @mrproliu in https://github.com/apache/skywalking-cli/pull/159 Trim license headers in query string by @kezhenxu94 in https://github.com/apache/skywalking-cli/pull/160 Bump up dependency swck version to fix CVE by @kezhenxu94 in https://github.com/apache/skywalking-cli/pull/161 Bump up swck dependency for transitive dep upgrade by @kezhenxu94 in https://github.com/apache/skywalking-cli/pull/162 Add the sub-commands for query sorted metrics/records by @mrproliu in https://github.com/apache/skywalking-cli/pull/163 Add compatibility documentation by @mrproliu in https://github.com/apache/skywalking-cli/pull/164 Overhaul licenses, prepare for 0.11.0 by @kezhenxu94 in https://github.com/apache/skywalking-cli/pull/165 ","excerpt":"\u003cp\u003eSkyWalking CLI 0.11.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAdd \u003ccode\u003e.github/scripts …\u003c/code\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cli-0-11-0/","title":"Release Apache SkyWalking CLI 0.11.0"},{"body":"SkyWalking Kubernetes Helm Chart 4.3.0 is released. Go to downloads page to find release tars.\nFix hasSuffix replace hasPrefix by @geffzhang in https://github.com/apache/skywalking-kubernetes/pull/86 Add \u0026ldquo;pods/log\u0026rdquo; permission to OAP so on-demand Pod log can work by @kezhenxu94 in https://github.com/apache/skywalking-kubernetes/pull/87 add .Values.oap.initEs to work with ES initial by @williamyao1982 in https://github.com/apache/skywalking-kubernetes/pull/88 Remove Istio adapter, add changelog for 4.3.0 by @kezhenxu94 in https://github.com/apache/skywalking-kubernetes/pull/89 Bump up helm chart version by @kezhenxu94 in https://github.com/apache/skywalking-kubernetes/pull/90 ","excerpt":"\u003cp\u003eSkyWalking Kubernetes Helm Chart 4.3.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eFix …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-kubernetes-helm-chart-4.3.0/","title":"Release Apache SkyWalking Kubernetes Helm Chart 4.3.0"},{"body":"SkyWalking Cloud on Kubernetes 0.7.0 is released. Go to downloads page to find release tars.\nFeatures Replace go-bindata with embed lib. Add the OAPServerConfig CRD, webhooks and controller. Add the OAPServerDynamicConfig CRD, webhooks and controller. Add the SwAgent CRD, webhooks and controller. [Breaking Change] Remove the way to configure the agent through Configmap. Bugs Fix the error in e2e testing. Fix status inconsistent with CI. Bump up prometheus client version to fix cve. Chores Bump several dependencies of adapter. Update license eye version. Bump up SkyWalking OAP to 9.0.0. Bump up the k8s api of the e2e environment to v1.21.10. ","excerpt":"\u003cp\u003eSkyWalking Cloud on Kubernetes 0.7.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars. …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cloud-on-kubernetes-0-7-0/","title":"Release Apache SkyWalking Cloud on Kubernetes 0.7.0"},{"body":"SkyWalking Rover 0.3.0 is released. Go to downloads page to find release tars.\nFeatures Support NETWORK Profiling. Let the logger as a configurable module. Support analyze the data of OpenSSL, BoringSSL library, GoTLS, NodeTLS in NETWORK Profiling. Enhancing the kubernetes process finder. Bug Fixes Fixed reading process paths incorrect when running as a container. Fix the crash caused by multiple profiling tasks. Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Rover 0.3.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-rover-0-3-0/","title":"Release Apache SkyWalking Rover 0.3.0"},{"body":"SkyWalking Java Agent 8.12.0 is released. Go to downloads page to find release tars. Changes by Version\n8.12.0 Fix Shenyu plugin\u0026rsquo;s NPE in reading trace ID when IgnoredTracerContext is used in the context. Update witness class in elasticsearch-6.x-plugin, avoid throw NPE. Fix onHalfClose using span operation name /Request/onComplete instead of the wrong name /Request/onHalfClose. Add plugin to support RESTeasy 4.x. Add plugin to support hutool-http 5.x. Add plugin to support Tomcat 10.x. Save http status code regardless of it\u0026rsquo;s status. Upgrade byte-buddy to 1.12.13, and adopt byte-buddy APIs changes. Upgrade gson to 2.8.9. Upgrade netty-codec-http2 to 4.1.79.Final. Fix race condition causing agent to not reconnect after network error Force the injected high-priority classes in order to avoid NoClassDefFoundError. Plugin to support xxl-job 2.3.x. Add plugin to support Micronaut(HTTP Client/Server) 3.2.x-3.6.x Add plugin to support NATS Java client 2.14.x-2.15.x Remove inappropriate dependency from elasticsearch-7.x-plugin Upgrade jedis plugin to support 3.x(stream),4.x Documentation Add a section in Bootstrap-plugins doc, introducing HttpURLConnection Plugin compatibility. Update Plugin automatic test framework, fix inconsistent description about configuration.yml. Update Plugin automatic test framework, add expected data format of the log items. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking Java Agent 8.12.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nChanges by …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-java-agent-8-12-0/","title":"Release Apache SkyWalking Java Agent 8.12.0"},{"body":"This is an official annoucement from SkyWalking team.\nSkyWalking backend server and UI released significant 9.2.0 at Sep. 2nd, 2022. With the new added Layer concept, the ebpf agent, wider middleware server monitoring(Such as MySQL and PostgreSQL servers) powered by OpenTelemetry ecosystem, SkyWalking v9 has been much more powerful than the last v8 version(8.9.1).\nFrom now, we have resolved all found critical bugs since 9.0.0 release which could block the v8 users to upgrade. v9 releases also provide the as same compatibility as the 8.9.1 release. So, end users would not have a block when they apply to upgrade. (We don\u0026rsquo;t provide storage structure compatibility as usually, users should use an empty database to initialize for a new version.)\nAnd more importantly, we are confident that, v9 could provide a stable and higher performance APM in the product environment.\nThe 8.9.1 release was released at Dec., 2021. Since then, there is no one contributed any code, and there is no committer requested to begin a new iteration or plan to run a patch release. From the project management committee perspective, the 8.x had became inactive.\nWe are going to wait for another 3 month to official end 8.x series\u0026rsquo; life.\nNotice, this could be changed if there are at least 3 committers supporting to work on further 8.x releases officially, and provide a release plan.\n","excerpt":"\u003cp\u003eThis is an official annoucement from SkyWalking team.\u003c/p\u003e\n\u003cp\u003eSkyWalking backend server and UI released …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/deprecate-v8/","title":"Plan to End-of-life(EOL) all v8 releases in Nov. 2022"},{"body":"SkyWalking 9.2.0 is released. Go to downloads page to find release tars.\neBPF Network Profiling for K8s Pod Event and Metrics Association MySQL Server Monitoring PostgreSQL Server Monitoring Project [Critical] Fix a low performance issue of metrics persistent in the ElasticSearch storage implementation. One single metric could have to wait for an unnecessary 7~10s(System Env Variable SW_STORAGE_ES_FLUSH_INTERVAL) since 8.8.0 - 9.1.0 releases. Upgrade Armeria to 1.16.0, Kubernetes Java client to 15.0.1. OAP Server Add more entities for Zipkin to improve performance. ElasticSearch: scroll id should be updated when scrolling as it may change. Mesh: fix only last rule works when multiple rules are defined in metadata-service-mapping.yaml. Support sending alarm messages to PagerDuty. Support Zipkin kafka collector. Add VIRTUAL detect type to Process for Network Profiling. Add component ID(128) for Java Hutool plugin. Add Zipkin query exception handler, response error message for illegal arguments. Fix a NullPointerException in the endpoint analysis, which would cause missing MQ-related LocalSpan in the trace. Add forEach, processRelation function to MAL expression. Add expPrefix, initExp in MAL config. Add component ID(7015) for Python Bottle plugin. Remove legacy OAL percentile functions, p99, p95, p90, p75, p50 func(s). Revert #8066. Keep all metrics persistent even it is default value. Skip loading UI templates if folder is empty or doesn\u0026rsquo;t exist. Optimize ElasticSearch query performance by using _mGet and physical index name rather than alias in these scenarios, (a) Metrics aggregation (b) Zipkin query (c) Metrics query (d) Log query Support the NETWORK type of eBPF Profiling task. Support sumHistogram in MAL. [Breaking Change] Make the eBPF Profiling task support to the service instance level, index/table ebpf_profiling_task is required to be re-created when bump up from previous releases. Fix race condition in Banyandb storage Support SUM_PER_MIN downsampling in MAL. Support sumHistogramPercentile in MAL. Add VIRTUAL_CACHE to Layer, to fix conjectured Redis server, which icon can\u0026rsquo;t show on the topology. [Breaking Change] Elasticsearch storage merge all metrics/meter and records(without super datasets) indices into one physical index template metrics-all and records-all on the default setting. Provide system environment variable(SW_STORAGE_ES_LOGIC_SHARDING) to shard metrics/meter indices into multi-physical indices as the previous versions(one index template per metric/meter aggregation function). In the current one index mode, users still could choose to adjust ElasticSearch\u0026rsquo;s shard number(SW_STORAGE_ES_INDEX_SHARDS_NUMBER) to scale out. More details please refer to New ElasticSearch storage option explanation in 9.2.0 and backend-storage.md [Breaking Change] Index/table ebpf_profiling_schedule added a new column ebpf_profiling_schedule_id, the H2/Mysql/Tidb/Postgres storage users are required to re-created it when bump up from previous releases. Fix Zipkin trace query the max size of spans. Add tls and https component IDs for Network Profiling. Support Elasticsearch column alias for the compatibility between storage logicSharding model and no-logicSharding model. Support MySQL monitoring. Support PostgreSQL monitoring. Fix query services by serviceId error when Elasticsearch storage SW_STORAGE_ES_QUERY_MAX_SIZE \u0026gt; 10000. Support sending alarm messages to Discord. Fix query history process data failure. Optimize TTL mechanism for Elasticsearch storage, skip executed indices in one TTL rotation. Add Kubernetes support module to share codes between modules and reduce calls to Kubernetes API server. Bump up Kubernetes Java client to fix cve. Adapt OpenTelemetry native metrics protocol. [Breaking Change] rename configuration folder from otel-oc-rules to otel-rules. [Breaking Change] rename configuration field from enabledOcRules to enabledOtelRules and environment variable name from SW_OTEL_RECEIVER_ENABLED_OC_RULES to SW_OTEL_RECEIVER_ENABLED_OTEL_RULES. [Breaking Change] Fix JDBC TTL to delete additional tables data. SQL Database requires removing segment,segment_tag, logs, logs_tag, alarms, alarms_tag, zipkin_span, zipkin_query before OAP starts. SQL Database: add @SQLDatabase.ExtraColumn4AdditionalEntity to support add an extra column from parent to an additional table. Add component ID(131) for Java Micronaut plugin Add component ID(132) for Nats java client plugin UI Fix query conditions for the browser logs. Implement a URL parameter to activate tab index. Fix clear interval fail when switch autoRefresh to off. Optimize log tables. Fix log detail pop-up page doesn\u0026rsquo;t work. Optimize table widget to hide the whole metric column when no metric is set. Implement the Event widget. Remove event menu. Fix span detail text overlap. Add Python Bottle Plugin Logo. Implement an association between widgets(line, bar, area graphs) with time. Fix tag dropdown style. Hide the copy button when db.statement is empty. Fix legend metrics for topology. Dashboard: Add metrics association. Dashboard: Fix FaaS-Root document link and topology service relation dashboard link. Dashboard: Fix Mesh-Instance metric Throughput. Dashboard: Fix Mesh-Service-Relation metric Throughput and Proxy Sidecar Internal Latency in Nanoseconds (Client Response). Dashboard: Fix Mesh-Instance-Relation metric Throughput. Enhance associations for the Event widget. Add event widgets in dashboard where applicable. Fix dashboard list search box not work. Fix short time range. Fix event widget incompatibility in Safari. Refactor the tags component to support searching for tag keys and values. Implement the log widget and the trace widget associate with each other, remove log tables on the trace widget. Add log widget to general service root. Associate the event widget with the trace and log widget. Add the MySQL layer and update layer routers. Fix query order for trace list. Add a calculation to convert seconds to days. q* Add Spring Sleuth dashboard to general service instance. Support the process dashboard and create the time range text widget. Fix picking calendar with a wrong time range and setting a unique value for dashboard grid key. Add PostgreSQL to Database sub-menu. Implement the network profiling widget. Add Micronaut icon for Java plugin. Add Nats icon for Java plugin. Bump moment and @vue/cli-plugin-e2e-cypress. Add Network Profiling for Service Mesh DP instance and K8s pod panels. Documentation Fix invalid links in release docs. Clean up doc about event metrics. Add a table for metric calculations in the UI doc. Add an explanation for alerting kernel and its in-memory window mechanism. Add more docs for widget details. Update alarm doc introduce configuration property key Fix dependency license\u0026rsquo;s NOTICE and binary jar included issues in the source release. Add eBPF CPU profiling doc. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking 9.2.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"ebpf-network-profiling-for-k8s-pod\"\u003eeBPF Network Profiling for …\u003c/h4\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-9.2.0/","title":"Release Apache SkyWalking APM 9.2.0"},{"body":"SkyWalking Rust 0.4.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed Publish release doc. by @wu-sheng in https://github.com/apache/skywalking-rust/pull/31 Set up CI and approval requirements by @wu-sheng in https://github.com/apache/skywalking-rust/pull/32 Move skywalking_proto mod to single files. by @jmjoy in https://github.com/apache/skywalking-rust/pull/33 Polish the release doc. by @wu-sheng in https://github.com/apache/skywalking-rust/pull/34 Add serde support for protobuf generated struct. by @jmjoy in https://github.com/apache/skywalking-rust/pull/35 Improve LogReporter and fix tests. by @jmjoy in https://github.com/apache/skywalking-rust/pull/36 Split tracer inner segment sender and receiver into traits. by @jmjoy in https://github.com/apache/skywalking-rust/pull/37 Switch to use nightly rustfmt. by @jmjoy in https://github.com/apache/skywalking-rust/pull/38 Change Span to refer to SpanStack, rather than TracingContext. by @jmjoy in https://github.com/apache/skywalking-rust/pull/39 Adjust the trace structure. by @jmjoy in https://github.com/apache/skywalking-rust/pull/40 Add logging. by @jmjoy in https://github.com/apache/skywalking-rust/pull/41 Upgrade dependencies. by @jmjoy in https://github.com/apache/skywalking-rust/pull/42 Add feature vendored, to auto build protoc. by @jmjoy in https://github.com/apache/skywalking-rust/pull/43 Add metrics. by @jmjoy in https://github.com/apache/skywalking-rust/pull/44 Add more GH labels as new supports by @wu-sheng in https://github.com/apache/skywalking-rust/pull/45 Bump to 0.4.0. by @jmjoy in https://github.com/apache/skywalking-rust/pull/46 Fix trace id is not transmitted. by @jmjoy in https://github.com/apache/skywalking-rust/pull/47 ","excerpt":"\u003cp\u003eSkyWalking Rust 0.4.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-rust-0-4-0/","title":"Release Apache SkyWalking Rust 0.4.0"},{"body":" 目录 开篇 为什么需要全链路监控 为什么选择SkyWalking 预研 POC 优化 未来 1、开篇 自从SkyWalking开始在公司推广，时不时会在排查问题的人群中听到这样的话：“你咋还没接SkyWalking？接入后，一眼就看出是哪儿的问题了\u0026hellip;\u0026quot;，正如同事所说的，在许多情况下，SkyWalking就是这么秀。作为实践者，我非常感谢SkyWalking，因为这款国产全链路监控产品给公司的的伙伴们带来了实实在在的帮助；也特别感谢公司的领导和同事们，正因为他们的支持和帮助，才让这套SkyWalking（V8.5.0）系统从起初的有用进化到现在的好用；从几十亿的Segment储能上限、几十秒的查询耗时，优化到千亿级的Segment储能、毫秒级的查询耗时。\n小提示：\nSkyWalking迭代速度很快，公司使用的是8.5.0版本，其新版本的性能肯定有改善。 Segment是SkyWalking中提出的概念，表示一次请求在某个服务内的执行链路片段的合集，一个请求在多个服务中先后产生的Segment串起来构成一个完整的Trace，如下图所示： SkyWalking的这次实践，截止到现在有一年多的时间，回顾总结一下这段历程中的些许积累和收获，愿能反哺社区，给有需求的道友提供个案例借鉴；也希望能收获到专家们的指导建议，把项目做得更好。因为安全约束，要把有些内容和谐掉，但也努力把这段历程中那些**靓丽的风景，**尽可能完整的呈现给大家。\n2、为什么需要全链路监控 随着微服务架构的演进，单体应用按照服务维度进行拆分，组织架构也随之演进以横向、纵向维度拆分；一个业务请求的执行轨迹，也从单体应用时期一个应用实例内一个接口，变成多个服务实例的多个接口；对应到组织架构，可能跨越多个BU、多个Owner。虽然微服务架构高内聚低耦合的优势是不言而喻的，但是低耦合也有明显的副作用，它在现实中给跨部门沟通、协作带来额外的不可控的开销；因此开发者尤其是终端业务侧的架构师、管理者，特别需要一些可以帮助理解系统拓扑和用于分析性能问题的工具，便于在架构调整、性能检测和发生故障时，缩减沟通协作方面的精力和时间耗费，快速定位并解决问题。\n我所在的平安健康互联网股份有限公司（文中简称公司），是微服务架构的深度实践者。公司用互联网技术搭建医疗服务平台，致力于构筑专业的医患桥梁，提供专业、全面、高品质、一站式企业健康管理服务。为了进一步提高系统服务质量、提升问题响应效率，部门在21年结合自身的一些情况，决定对现行的全链路监控系统进行升级，目的与以下网络中常见的描述基本一致：\n快速发现问题 判断故障影响范围 梳理服务依赖并判断依赖的合理性 分析链路性能并实施容量规划 3、为什么选择SkyWalking 在做技术选型时，网络中搜集的资料显示，谷歌的 Dapper系统，算是链路追踪领域的始祖。受其公开论文中提出的概念和理念的影响，一些优秀的企业、个人先后做出不少非常nice的产品，有些还在社区开源共建，如：韩国的Pinpoint，Twitter的Zipkin，Uber的Jaeger及中国的SkyWalking 等，我司选型立项的过程中综合考虑的因素较多，这里只归纳一下SkyWalking吸引我们的2个优势：\n产品的完善度高：\njava生态，功能丰富 社区活跃，迭代迅速 链路追踪、拓扑分析的能力强：\n插件丰富，探针无侵入。 采用先进的流式拓扑分析设计 “好东西不需要多说,实际行动告诉你“，这句话我个人非常喜欢，关于SkyWalking的众多的优点，网络上可以找到很多，此处先不逐一比较、赘述了。\n4、预研 当时最新版本8.5.0，梳理分析8.x的发布记录后，评估此版本的核心功能是蛮稳定的，于是基于此版本开始了SkyWalking的探索之旅。当时的认知是有限的，串行思维模型驱使我将关注的问题聚焦在架构原理是怎样、有什么副作用这2个方面：\n架构和原理：\nagent端 主要关注 Java Agent的机制、SkyWalking Agent端的配置、插件的工作机制、数据采集及上报的机制。 服务端 主要关注 角色和职责、模块和配置、数据接收的机制、指标构建的机制、指标聚合的机制及指标存储的机制。 存储端 主要关注 数据量，存储架构要求以及资源评估。 副作用：\n功能干扰 性能损耗 4.1 架构和原理 SkyWalking社区很棒，官网文档和官方出版的书籍有较系统化的讲解，因为自己在APM系统以及Java Agent方面有一些相关的经验沉淀，通过在这两个渠道的学习，对Agent端和OAP(服务端)很快便有了较系统化的认知。在做系统架构选型时，评估数据量会比较大（成千上万的JVM实例数，每天采集的Segment数量可能是50-100亿的级别），所以传输通道选择Kafka、存储选择Elasticsearch，如此简易版的架构以及数据流转如下图所示：\n这里有几处要解释一下：\nAgent上报数据给OAP端，有grpc通道和kafka通道，当时就盲猜grpc通道可能撑不住，所以选择kafka通道来削峰；kafka通道是在8.x里加入的。 千亿级的数据用ES来做存储肯定是可以的。 图中L1聚合的意思是：SkyWalking OAP服务端 接收数据后，构建metric并完成metric 的Level-1聚合，这里简称L1聚合。 图中L2聚合的意思是：服务端 基于metric的Level-1聚合结果，再做一次聚合，即Level-2聚合，这里简称L2聚合。后续把纯Mixed角色的集群拆成了两个集群。 4.2 副作用 对于质量团队和接入方来说，他们最关注的问题是，接入SkyWalking后：\n是否对应用有功能性干扰 在运行期能带来哪些性能损耗 这两个问题从3个维度来得到答案：\n网络资料显示：\nAgent带来的性能损耗在5%以内 未搜到功能性干扰相关的资料（盲猜没有这方面问题） 实现机制评估：\n字节码增强机制是JVM提供的机制，SkyWalking使用的字节码操控框架ByteBuddy也是成熟稳定的；通过自定义ClassLoader来加载管理插件类，不会产生冲突和污染。 Agent内插件开发所使用的AOP机制是基于模板方法模式实现的，风控很到位，即使插件的实现逻辑有异常也不影响用户逻辑的执行； 插件采集数据跟上报逻辑之间用了一个轻量级的无锁环形队列进行解耦，算是一种保护机制；这个队列在MPSC场景下性能还不错；队列采用满时丢弃的策略，不会有积压阻塞和OOM。 性能测试验证\n测试的老师针对dubbo、http 这两种常规RPC通信场景，进行压力测试和稳定性测试，结果与网络资料描述一致，符合预期。 5、POC 在POC阶段，接入几十个种子应用，在非生产环境试点观察，同时完善插件补全链路，对接公司的配置中心，对接发布系统，完善自监控.全面准备达到推广就绪状态。\n5.1 对接发布系统 为了对接公司的发布系统，方便系统的发布，将SkyWalking应用拆分为4个子应用：\n应用 介绍 Webapp Skywalking的web端 Agent Skywalking的Agent端 OAP-Receiver skywakling的服务端，角色是Mixed或Receiver OAP-Aggregator skywalking的服务端，角色是Aggregator 这里有个考虑，暂定先使用纯Mixed角色的单集群，有性能问题时就试试 Receiver+Aggregator双角色集群模式，最终选哪种视效果而定。\nSkyWalking Agent端是基于Java Agent机制实现的，采用的是启动挂载模式；启动挂载需在启动脚本里加入挂载Java Agent的逻辑，发布系统实现这个功能需要注意2点：\n启动脚本挂载SkyWalking Agent的环节，尽量让用户无感知。 发布系统在挂载Agent的时候，给Agent指定应用名称和所属分组信息。 SkyWalking Agent的发布和升级也由发布系统来负责；Agent的升级采用了灰度管控的方案，控制的粒度是应用级和实例级两种：\n按照应用灰度，可给应用指定使用什么版本的Agent 按照应用的实例灰度，可给应用指定其若干实例使用什么版本的Agent 5.2 完善插件补全链路 针对公司OLTP技术栈，量身定制了插件套，其中大部分在开源社区的插件库中有，缺失的部分通过自研快速补齐。\n这些插件给各组件的核心环节埋点，采集数据上报给SkyWalking后，Web端的【追踪】页面就能勾勒出丰满完美的请求执行链路；这对架构师理解真实架构，测试同学验证逻辑变更和分析性能损耗，开发同学精准定位问题都非常的有帮助。这里借官方在线Demo的截图一用（抱歉后端程序员，五毛特效都没做出来，丰满画面还请自行脑补）\n友情小提示：移除不用的插件对程序编译打包和减少应用启动耗时很有帮助。\n5.3压测稳测 测试的老师，针对SkyWalking Agent端的插件套，设计了丰富的用例，压力测试和稳定性测试的结果都符合预期；每家公司的标准不尽一致，此处不再赘述。\n5.4 对接自研的配置中心 把应用中繁杂的配置交给配置中心来管理是非常必要的，配置中心既能提供启动时的静态配置，又能管理运行期的动态配置，而且外部化配置的机制特别容易满足容器场景下应用的无状态化要求。啰嗦一下，举2个例子：\n调优时，修改参数的值不用来一遍开发到测试再到生产的发布。 观测系统状态，修改日志配置后不需要来一遍开发到测试再到生产的发布。 Skywaling在外接配置中心这块儿，适配了市面中主流的配置中心产品。而公司的配置中心是自研的，需要对接一下，得益于SkyWalking提供的模块化管理机制，只用扩展一个模块即可。\n在POC阶段，梳理服务端各模块的功能，能感受到其配置化做的不错，配置项很丰富，管控的粒度也很细；在POC阶段几乎没有变动，除了对Webapp模块的外部化配置稍作改造，与配置中心打通以便在配置中心管理 Webapp模块中Ribbon和Hystrix的相关配置。\n5.5完善自监控 自监控是说监控SkyWalking系统内各模块的运转情况：\n组件 监控方案 说明 kafka kafka-manager 它俩是老搭档了 Agent端 Skywalking Agent端会发心跳信息给服务端，可在Web端看到Agent的信息 OAP集群 prometheus 指标还算丰富，感觉缺的可以自己补充 ES集群 prometheus 指标还算丰富 完善自监控后的架构如下图所示：\n5.6 自研Native端SDK 公司移动端的应用很核心，也要使用链路追踪的功能，社区缺了这块，于是基于SkyWalking的协议，移动端的伙伴们自研了一套SDK，弥补了Native端链路数据的缺失，也在后来的秒开页面指标统计中发挥了作用。随着口口相传，不断有团队提出需求、加入建设，所以也在持续迭代中；内容很多，这里先不展开。\n5.7 小结 POC阶段数据量不大，主要是发现系统的各种功能性问题，查缺补漏。\n6、优化 SkyWalking的正式推广采用的是城市包围农村的策略；公司的核心应用作为第一批次接入，这个策略有几个好处：\n核心应用的监管是重中之重，优先级默认最高。 核心应用的上下游应用，会随着大家对SkyWalking依赖的加深，而逐步自主接入。 当然安全是第一位的，无论新系统多好、多厉害，其引入都需遵守安全稳定的前提要求。既要安全又要快速还要方便，于是基于之前Agent灰度接入的能力，在发布系统中增加应用Owner自助式灰度接入和快速卸载SkyWalking Agent的能力，即应用负责人可自主选择哪个应用接入，接入几个实例，倘若遇到问题仅通过重启即可完成快速卸载；这个能力在推广的前期发挥了巨大的作用；毕竟安全第一，信任也需逐步建立。\n随着应用的接入、使用，我们也逐渐遇到了一些问题，这里按照时间递增的顺序将问题和优化效果快速的介绍给大家，更多技术原理的内容计划在【SkyWalking(v8.5.0)调优系列】补充。开始之前有几个事项要说明：\n下文中提到的数字仅代表我司的情况，标注的Segment数量是处理这个问题的那段时间的情况，并不是说达到这个数量才开始出现这个现象。 这些数值以及当时的现象，受到宿主机配置、Segment数据的大小、存储处理能力等多种因素的影响；请关注调整的过程和效果，不必把数字和现象对号入座哈。 6.1 启动耗时： 问题： 有同事反馈应用启动变慢，排查发现容器中多数应用启动的总耗时，在接入SkyWalking前是2秒，接入后变成了16秒以上，公司很多核心应用的实例数很多，这样的启动损耗对它们的发布影响太大。\n优化： 记录启动耗时并随着其他启动数据上报到服务端，方便查看对比。 优化Kafka Reporter的启动过程，将启动耗时减少了3-4秒。 优化类匹配和增强环节（重点）后，容器中的应用启动总耗时从之前16秒以上降低到了3秒内。 梳理Kafka 启动和上报的过程中，顺带调整了Agent端的数据上报到kafka的分区选择策略，将一个JVM实例中的数据全部发送到同一个的分区中，如此在L1层的聚合就完成了JVM实例级的Metric聚合，需注意调整Kafka分片数来保证负载均衡。 6.2 kafka积压-6亿segment/天 问题： SkyWalking OAP端消费慢，导致Kafka中Segment积压。未能达到能用的目标。\n优化： 从SkyWalking OAP端的监控指标中没有定位出哪个环节的问题，把服务端单集群拆为双集群，即把 Mixed角色的集群 ，修改为 Receiver 角色（接收和L1聚合）的集群 ，并加入 Aggregation角色（L2聚合）的集群，调整成了双集群模式，数据流传如下图所示：\n6.3 kafka积压-8亿segment/天 问题： SkyWalking OAP端消费慢，导致Kafka中Segment积压，监控指标能看出是在ES存储环节慢，未能达到能用的目标。\n优化： 优化segment保存到ES的批处理过程，调整BulkProcessor的线程数和批处理大小。 优化metrics保存到ES的批处理过程，调整批处理的时间间隔、线程数、批处理大小以及刷盘时间。 6.4 kafka积压-20亿segment/天 问题： Aggregation集群的实例持续Full GC，Receiver集群通过grpc 给Aggregation集群发送metric失败。未能达到能用的目标。\n优化： 增加ES节点、分片，效果不明显。 ES集群有压力，但无法精准定位出是什么数据的什么操作引发的。采用分治策略，尝试将数据拆分，从OAP服务端读写逻辑调整，将ES单集群拆分为 trace集群 和 metric集群；之后对比ES的监控指标明确看出是metric集群读写压力太大。 优化Receiver集群metric的L1聚合，完成1分钟的数据聚合后，再提交给Aggregation集群做L2聚合。 Aggregation集群metric的L2 聚合是基于db实现的，会有 空读-写-再读-累加-更新写 这样的逻辑，每次写都会有读，调整逻辑是：提升读的性能，优化缓存机制减少读的触发；调整间隔，避免触发累加和更新。 将metric批量写ES操作调整成BulkProcessor。 ES的metric集群 使用SSD存储，增加节点数和分片数。 这一次的持续优化具有里程碑式的意义，Kafka消费很快，OAP各机器的Full GC没了，ES的各方面指标也很稳定；接下来开始优化查询，提升易用性。\n6.5 trace查询慢-25亿segment/天 问题： Web端【追踪】页中的查询都很慢，仅保存了15天的数据，按照traceId查询耗时要20多秒，按照条件查询trace列表的耗时更糟糕；这给人的感受就是“一肚子墨水倒不出来”，未能达到好用的目标。\n优化： ES查询优化方面的信息挺多，但通过百度筛选出解决此问题的有效方案，就要看咱家爱犬的品类了；当时搜集整理了并尝试了N多优化条款，可惜没有跟好运偶遇，结论是颜值不可靠。言归正传，影响读写性能的基本要素有3个：读写频率，数据规模，硬件性能；trace的情况从这三个维度来套一套模板：\n要素 trace的情况 备注 读写频率 宏观来看是写多读少的状况 数据规模 按照每天50亿个segment来算，半个月是750亿，1个月是1500亿。 硬件性能 普通硬盘速度一般 这个分析没有得出具有指导意义的结论，读写频率这里粒度太粗，用户的使用情况跟时间也有紧密的关系，情况大概是：\n当天的数据是读多写多（当天不断有新数据写入，基于紧急响应的需求，问题出现时可能是近实时的排查处理）。 前一天的数据是读多写少（一般也会有问题隔天密集上报的情况，0点后会有前一天数据延迟到达的情况）。 再早的话无新数据写入，数据越早被读的概率也越小。 基于以上分析，增加时间维度并细化更多的参考因素后，分析模型变成了这样：\n要素 当天 当天-1 当天-2 ~ 当天-N 写频率 多 少 无 读（查询）频率 多 多 少 读响应速度要求 快 快 慢点也行 数据规模 50亿 50亿 50亿* (N-2) 宿主机性能要求 高 高 次高 硬盘速度要求 高(SSD) 高(SSD) 次高(机械) 硬件成本 高 高 次高 期望成本 低 低 低 从上表可以看出，整体呈现出hot-warm数据架构的需求之势，近1-2天为hot数据，之前的为warm数据；恰好ES7提供了hot-warm架构支持，按照hot-warm改造后架构如下图所示：\n恰逢公司ES中台调优版的ES发布，其内置的ZSTD压缩算法 空间压缩效果非常显著。 对 trace集群进行hot-warm架构调整，查询耗时从20多秒变成了2-3秒，效果是非常明显的。 从查询逻辑进一步调整，充分利用ES的数据分片、路由机制，把全量检索调整为精准检索，即降低检索时需要扫描的数据量，把2-3秒优化到毫秒。 这里要炫一个5毛特效，这套机制下，Segment数据即使是保留半年的，按照TraceId查询的耗时也是毫秒。\n至此完成了查询千亿级Trace数据只要毫秒级耗时的阶段性优化。\n6.6 仪表盘和拓扑查询慢 问题： Web端的【拓扑】页，在开始只有几十个应用的时候，虽然很慢，但还是能看到数据，随着应用增多后，【拓扑】页面数据请求一直是超时(配置的60s超时)的，精力有限，先通过功能降级把这个页面隐藏了；【仪表盘】的指标查询也非常的慢，未能达到好用的目标。\n优化： Web端的【仪表盘】页和【拓扑】页是对SkyWalking里metric数据的展现，metric数据同trace数据一样满足hot-warm的特征。\nmetric集群采用hot-warm架构调整，之后仪表盘中的查询耗时也都减小为毫秒级。 【拓扑】页接口依然是超时(60s)，对拓扑这里做了几个针对性的调整： 把内部的循环调用合并，压缩调用次数。 去除非必要的查询。 拆分隔离通用索引中的数据，避免互相干扰。 全量检索调整为精准检索，即降低检索时需要扫描的数据量。 至此完成了拓扑页数据查询毫秒级耗时的阶段性优化。\n6.7 小结 SkyWalking调优这个阶段，恰逢上海疫情封城，既要为生存抢菜，又要翻阅学习着各种ES原理、调优的文档资料，一行一行反复的品味思考SkyWalking相关的源码，尝试各种方案去优化它，梦中都在努力提升它的性能。疫情让很多人变得焦虑烦躁，但以我的感受来看在系统的性能压力下疫情不值一提。凡事贵在坚持，时间搞定了诸多困难，调优的效果是很显著的。\n可能在业务价值驱动的价值观中这些技术优化不产生直接业务价值，顶多是五毛特效，但从其他维度来看它价值显著：\n对个人来说，技术有提升。 对团队来说，实战练兵提升战力，团队协作加深友情；特别感谢ES中台这段时间的鼎力支持！ 对公司来说，易用性的提升将充分发挥SkyWalking的价值，在问题发生时，给到同事们切实、高效的帮助，使得问题可以被快速响应；须知战争拼的是保障。 这期间其实也是有考虑过其他的2个方案的：\n使用降低采样率的兜底方案；但为了得到更准确的指标数据，以及后续其他的规划而坚持了全采样。 采用ClickHouse优化存储；因为公司有定制优化的ES版本，所以就继续在ES上做存储优化，刚好借此机会验证一下。后续【全链路结构化日志】的存储会使用ClickHouse。 这个章节将内容聚焦在落地推广时期技术层面的准备和调优，未描述团队协调、推广等方面的情况；因每个公司情况不同，所以并未提及；但其实对多数公司来说，有些项目的推广比技术本身可能难度更大，这个项目也遇到过一些困难，PM去推广是既靠能力又靠颜值， 以后有机会再与大家探讨。\n7、未来 H5、Native以及后端应用都在持续接入中，相应的SDK也在不断的迭代；目前正在基于已建立的链路通道，完善【全链路业务状态追踪】和【全链路结构化日志追踪】，旨在给运营、客服、运维、开发等服务在一线的同事们提供多视角一站式的观测平台，全方位提升系统服务质量、提高问题响应速度。\n","excerpt":"\u003cimg src=\"pic1.png\"\u003e\n\u003ch3 id=\"目录\"\u003e目录\u003c/h3\u003e\n\u003col\u003e\n\u003cli\u003e开篇\u003c/li\u003e\n\u003cli\u003e为什么需要全链路监控\u003c/li\u003e\n\u003cli\u003e为什么选择SkyWalking\u003c/li\u003e\n\u003cli\u003e预研\u003c/li\u003e\n\u003cli\u003ePOC\u003c/li\u003e\n\u003cli\u003e优化\u003c/li\u003e\n\u003cli\u003e未来\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch3 id=\"1开篇\"\u003e1、开篇\u003c/h3\u003e\n\u003cp\u003e自从SkyWalking开始在公司推广，时不时会在排查问题的人群中听到这样的话：“你咋还没接 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2022-08-30-pingan-jiankang/","title":"SkyWalking on the way - 平安健康千亿级的全链路追踪系统的建设与实践"},{"body":"Observability essential when working with distributed systems. Built on 3 pillars of metrics, logging and tracing, having the right tools in place to quickly identify and determine the root cause of an issue in production is imperative. In this Kongcast interview, we explore the benefits of having observability and demo the use of Apache SkyWalking. We walk through the capabilities that SkyWalking offers out of the box and debug a common HTTP 500 error using the tool.\nAndrew Kew is interviewed by Viktor Gamov, a developer advocate at Kong Inc\nAndrew is a highly passionate technologist with over 16 valuable years experience in building server side and cloud applications. Having spent the majority of his time in the Financial Services domain, his meritocratic rise to CTO of an Algorithmic Trading firm allowed him to not only steer the business from a technology standpoint, but build robust and scalable trading algorithms. His mantra is \u0026ldquo;right first time\u0026rdquo;, thus ensuring the projects or clients he is involved in are left in a better place than they were before he arrived.\nHe is the founder of a boutique software consultancy in the United Kingdom, QuadCorps Ltd, working in the API and Integration Ecosystem space and is currently on a residency programme at Kong Inc as a senior field engineer and technical account manager working across many of their enterprise strategic accounts.\n","excerpt":"\u003cp\u003eObservability essential when working with distributed systems. Built on 3 pillars of metrics, …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2022-08-11-kongcast-20-distributed-tracing-using-skywalking-kong/","title":"[Video] Distributed tracing demo using Apache SkyWalking and Kong API Gateway"},{"body":"SkyWalking Rust 0.3.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed Update README.md by @wu-sheng in https://github.com/apache/skywalking-rust/pull/24 Improve errors. by @jmjoy in https://github.com/apache/skywalking-rust/pull/25 Add tracer. by @jmjoy in https://github.com/apache/skywalking-rust/pull/26 Move e2e to workspace. by @jmjoy in https://github.com/apache/skywalking-rust/pull/27 Auto finalize context and span when dropped. by @jmjoy in https://github.com/apache/skywalking-rust/pull/28 Add context capture and continued methods. by @jmjoy in https://github.com/apache/skywalking-rust/pull/29 Bump to 0.3.0. by @jmjoy in https://github.com/apache/skywalking-rust/pull/30 ","excerpt":"\u003cp\u003eSkyWalking Rust 0.3.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-rust-0-3-0/","title":"Release Apache SkyWalking Rust 0.3.0"},{"body":"SkyWalking NodeJS 0.5.1 is released. Go to downloads page to find release tars.\nSkyWalking NodeJS 0.5.1 is a patch release that fixed a vulnerability(CVE-2022-36127) in all previous versions \u0026lt;=0.5.0, we recommend all users who are using versions \u0026lt;=0.5.0 should upgrade to this version.\nThe vulnerability could cause NodeJS services that has this agent installed to be unavailable if the header includes an illegal SkyWalking header, such as\nOAP is unhealthy and the downstream service\u0026rsquo;s agent can\u0026rsquo;t establish the connection. Some sampling mechanism is activated in downstream agents. ","excerpt":"\u003cp\u003eSkyWalking NodeJS 0.5.1 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cp\u003eSkyWalking NodeJS …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-nodejs-0-5-1/","title":"[CVE-2022-36127] Release Apache SkyWalking for NodeJS 0.5.1"},{"body":"SkyWalking Eyes 0.4.0 is released. Go to downloads page to find release tars.\nReorganize GHA by header and dependency. (#123) Add rust cargo support for dep command. (#121) Support license expression in dep check. (#120) Prune npm packages before listing all dependencies (#119) Add support for multiple licenses in the header config section (#118) Add excludes to license resolve config (#117) maven: set group:artifact as dependency name and extend functions in summary template (#116) Stablize summary context to perform consistant output (#115) Add custom license urls for identification (#114) Lazy initialize GitHub client for comment (#111) Make license identifying threshold configurable (#110) Use Google\u0026rsquo;s licensecheck to identify licenses (#107) dep: short circuit if user declare dep license (#108) ","excerpt":"\u003cp\u003eSkyWalking Eyes 0.4.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eReorganize GHA by …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-eyes-0-4-0/","title":"Release Apache SkyWalking Eyes 0.4.0"},{"body":"SkyWalking NodeJS 0.5.0 is released. Go to downloads page to find release tars.\nBump up grpc-node to 1.6.7 to fix CVE-2022-25878 (#85) Fix issue #9165 express router entry duplicated (#84) Fix skywalking s3 upload error #8824 (#82) Improved ignore path regex (#81) Upgrade data collect protocol (#78) Fix wrong instance properties (#77) Fix wrong command in release doc (#76) ","excerpt":"\u003cp\u003eSkyWalking NodeJS 0.5.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eBump up grpc-node …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-nodejs-0-5-0/","title":"Release Apache SkyWalking for NodeJS 0.5.0"},{"body":"SkyWalking Infra E2E 1.2.0 is released. Go to downloads page to find release tars.\nFeatures Expand kind file path with system environment. Support shutdown service during setup phase in compose mode. Expand kind file path with system environment. Support arbitrary os and arch. Support docker-compose v2 container naming. Support installing via go install and add install doc. Add retry when delete kind cluster. Upgrade to go1.18. Bug Fixes Fix the problem of parsing verify.retry.interval without setting value. Documentation Make trigger.times parameter doc more clear. Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Infra E2E 1.2.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eExpand …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-infra-e2e-1-2-0/","title":"Release Apache SkyWalking Infra E2E 1.2.0"},{"body":"SkyWalking Python 0.8.0 is released. Go to downloads page to find release tars.\nFeature:\nUpdate mySQL plugin to support two different parameter keys. (#186) Add a SW_AGENT_LOG_REPORTER_SAFE_MODE option to control the HTTP basic auth credential filter (#200) Plugins:\nAdd Psycopg(3.x) support (#168) Add MySQL support (#178) Add FastAPI support (#181) Drop support for flask 1.x due to dependency issue in Jinja2 and EOL (#195) Add Bottle support (#214) Fixes:\nSpans now correctly reference finished parents (#161) Remove potential password leak from Aiohttp outgoing url (#175) Handle error when REMOTE_PORT is missing in Flask (#176) Fix sw-rabbitmq TypeError when there are no headers (#182) Fix agent bootstrap traceback not shown in sw-python CLI (#183) Fix local log stack depth overridden by agent log formatter (#192) Fix typo that cause user sitecustomize.py not loaded (#193) Fix instance property wrongly shown as UNKNOWN in OAP (#194) Fix multiple components inconsistently named on SkyWalking UI (#199) Fix SW_AGENT_LOGGING_LEVEL not properly set during startup (#196) Unify the http tag name with other agents (#208) Remove namespace to instance properties and add pid property (#205) Fix the properties are not set correctly (#198) Improved ignore path regex (#210) Fix sw_psycopg2 register_type() (#211) Fix psycopg2 register_type() second arg default (#212) Enhance Traceback depth (#206) Set spans whose http code \u0026gt; 400 to error (#187) Docs:\nAdd a FAQ doc on how to use with uwsgi (#188) Others:\nRefactor current Python agent docs to serve on SkyWalking official website (#162) Refactor SkyWalking Python to use the CLI for CI instead of legacy setup (#165) Add support for Python 3.10 (#167) Move flake configs all together (#169) Introduce another set of flake8 extensions (#174) Add E2E test coverage for trace and logging (#199) Now Log reporter cause_exception_depth traceback limit defaults to 10 Enable faster CI by categorical parallelism (#170) ","excerpt":"\u003cp\u003eSkyWalking Python 0.8.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eFeature:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eUpdate …\u003c/li\u003e\u003c/ul\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-python-0-8-0/","title":"Release Apache SkyWalking Python 0.8.0"},{"body":"SkyWalking Satellite 1.0.1 is released. Go to downloads page to find release tars.\nFeatures Bug Fixes Fix metadata messed up when transferring Log data. Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Satellite 1.0.1 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003ch4 id=\"bug-fixes\"\u003eBug …\u003c/h4\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-satellite-1-0-1/","title":"Release Apache SkyWalking Satellite 1.0.1"},{"body":"Content Background Apache SkyWalking observes metrics, logs, traces, and events for services deployed into the service mesh. When troubleshooting, SkyWalking error analysis can be an invaluable tool helping to pinpoint where an error occurred. However, performance problems are more difficult: It’s often impossible to locate the root cause of performance problems with pre-existing observation data. To move beyond the status quo, dynamic debugging and troubleshooting are essential service performance tools. In this article, we\u0026rsquo;ll discuss how to use eBPF technology to improve the profiling feature in SkyWalking and analyze the performance impact in the service mesh.\nTrace Profiling in SkyWalking Since SkyWalking 7.0.0, Trace Profiling has helped developers find performance problems by periodically sampling the thread stack to let developers know which lines of code take more time. However, Trace Profiling is not suitable for the following scenarios:\nThread Model: Trace Profiling is most useful for profiling code that executes in a single thread. It is less useful for middleware that relies heavily on async execution models. For example Goroutines in Go or Kotlin Coroutines. Language: Currently, Trace Profiling is only supported in Java and Python, since it’s not easy to obtain the thread stack in the runtimes of some languages such as Go and Node.js. Agent Binding: Trace Profiling requires Agent installation, which can be tricky depending on the language (e.g., PHP has to rely on its C kernel; Rust and C/C++ require manual instrumentation to make install). Trace Correlation: Since Trace Profiling is only associated with a single request it can be hard to determine which request is causing the problem. Short Lifecycle Services: Trace Profiling doesn\u0026rsquo;t support short-lived services for (at least) two reasons: It\u0026rsquo;s hard to differentiate system performance from class code manipulation in the booting stage. Trace profiling is linked to an endpoint to identify performance impact, but there is no endpoint to match these short-lived services. Fortunately, there are techniques that can go further than Trace Profiling in these situations.\nIntroduce eBPF We have found that eBPF — a technology that can run sandboxed programs in an operating system kernel and thus safely and efficiently extend the capabilities of the kernel without requiring kernel modifications or loading kernel modules — can help us fill gaps left by Trace Profiling. eBPF is a trending technology because it breaks the traditional barrier between user and kernel space. Programs can now inject bytecode that runs in the kernel, instead of having to recompile the kernel to customize it. This is naturally a good fit for observability.\nIn the figure below, we can see that when the system executes the execve syscalls, the eBPF program is triggered, and the current process runtime information is obtained by using function calls.\nUsing eBPF technology, we can expand the scope of Skywalking\u0026rsquo;s profiling capabilities:\nGlobal Performance Analysis: Before eBPF, data collection was limited to what agents can observe. Since eBPF programs run in the kernel, they can observe all threads. This is especially useful when you are not sure whether a performance problem is caused by a particular request. Data Content: eBPF can dump both user and kernel space thread stacks, so if a performance issue happens in kernel space, it’s easier to find. Agent Binding: All modern Linux kernels support eBPF, so there is no need to install anything. This means it is an orchestration-free vs an agent model. This reduces friction caused by built-in software which may not have the correct agents installed, such as Envoy in a Service Mesh. Sampling Type: Unlike Trace Profiling, eBPF is event-driven and, therefore, not constrained by interval polling. For example, eBPF can trigger events and collect more data depending on a transfer size threshold. This can allow the system to triage and prioritize data collection under extreme load. eBPF Limitations While eBPF offers significant advantages for hunting performance bottlenecks, no technology is perfect. eBPF has a number of limitations described below. Fortunately, since SkyWalking does not require eBPF, the impact is limited.\nLinux Version Requirement: eBPF programs require a Linux kernel version above 4.4, with later kernel versions offering more data to be collected. The BCC has documented the features supported by different Linux kernel versions, with the differences between versions usually being what data can be collected with eBPF. Privileges Required: All processes that intend to load eBPF programs into the Linux kernel must be running in privileged mode. As such, bugs or other issues in such code may have a big impact. Weak Support for Dynamic Language: eBPF has weak support for JIT-based dynamic languages, such as Java. It also depends on what data you want to collect. For Profiling, eBPF does not support parsing the symbols of the program, which is why most eBPF-based profiling technologies only support static languages like C, C++, Go, and Rust. However, symbol mapping can sometimes be solved through tools provided by the language. For example, in Java, perf-map-agent can be used to generate the symbol mapping. However, dynamic languages don\u0026rsquo;t support the attach (uprobe) functionality that would allow us to trace execution events through symbols. Introducing SkyWalking Rover SkyWalking Rover introduces the eBPF profiling feature into the SkyWalking ecosystem. The figure below shows the overall architecture of SkyWalking Rover. SkyWalking Rover is currently supported in Kubernetes environments and must be deployed inside a Kubernetes cluster. After establishing a connection with the SkyWalking backend server, it saves information about the processes on the current machine to SkyWalking. When the user creates an eBPF profiling task via the user interface, SkyWalking Rover receives the task and executes it in the relevant C, C++, Golang, and Rust language-based programs.\nOther than an eBPF-capable kernel, there are no additional prerequisites for deploying SkyWalking Rover.\nCPU Profiling with Rover CPU profiling is the most intuitive way to show service performance. Inspired by Brendan Gregg‘s blog post, we\u0026rsquo;ve divided CPU profiling into two types that we have implemented in Rover:\nOn-CPU Profiling: Where threads are spending time running on-CPU. Off-CPU Profiling: Where time is spent waiting while blocked on I/O, locks, timers, paging/swapping, etc. Profiling Envoy with eBPF Envoy is a popular proxy, used as the data plane by the Istio service mesh. In a Kubernetes cluster, Istio injects Envoy into each service’s pod as a sidecar where it transparently intercepts and processes incoming and outgoing traffic. As the data plane, any performance issues in Envoy can affect all service traffic in the mesh. In this scenario, it’s more powerful to use eBPF profiling to analyze issues in production caused by service mesh configuration.\nDemo Environment If you want to see this scenario in action, we\u0026rsquo;ve built a demo environment where we deploy an Nginx service for stress testing. Traffic is intercepted by Envoy and forwarded to Nginx. The commands to install the whole environment can be accessed through GitHub.\nOn-CPU Profiling On-CPU profiling is suitable for analyzing thread stacks when service CPU usage is high. If the stack is dumped more times, it means that the thread stack occupies more CPU resources.\nWhen installing Istio using the demo configuration profile, we found there are two places where we can optimize performance:\nZipkin Tracing: Different Zipkin sampling percentages have a direct impact on QPS. Access Log Format: Reducing the fields of the Envoy access log can improve QPS. Zipkin Tracing Zipkin with 100% sampling In the default demo configuration profile, Envoy is using 100% sampling as default tracing policy. How does that impact the performance?\nAs shown in the figure below, using the on-CPU profiling, we found that it takes about 16% of the CPU overhead. At a fixed consumption of 2 CPUs, its QPS can reach 5.7K.\nDisable Zipkin tracing At this point, we found that if Zipkin is not necessary, the sampling percentage can be reduced or we can even disable tracing. Based on the Istio documentation, we can disable tracing when installing the service mesh using the following command:\nistioctl install -y --set profile=demo \\ --set \u0026#39;meshConfig.enableTracing=false\u0026#39; \\ --set \u0026#39;meshConfig.defaultConfig.tracing.sampling=0.0\u0026#39; After disabling tracing, we performed on-CPU profiling again. According to the figure below, we found that Zipkin has disappeared from the flame graph. With the same 2 CPU consumption as in the previous example, the QPS reached 9K, which is an almost 60% increase. Tracing with Throughput With the same CPU usage, we\u0026rsquo;ve discovered that Envoy performance greatly improves when the tracing feature is disabled. Of course, this requires us to make trade-offs between the number of samples Zipkin collects and the desired performance of Envoy (QPS).\nThe table below illustrates how different Zipkin sampling percentages under the same CPU usage affect QPS.\nZipkin sampling % QPS CPUs Note 100% (default) 5.7K 2 16% used by Zipkin 1% 8.1K 2 0.3% used by Zipkin disabled 9.2K 2 0% used by Zipkin Access Log Format Default Log Format In the default demo configuration profile, the default Access Log format contains a lot of data. The flame graph below shows various functions involved in parsing the data such as request headers, response headers, and streaming the body.\nSimplifying Access Log Format Typically, we don’t need all the information in the access log, so we can often simplify it to get what we need. The following command simplifies the access log format to only display basic information:\nistioctl install -y --set profile=demo \\ --set meshConfig.accessLogFormat=\u0026#34;[%START_TIME%] \\\u0026#34;%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\\\u0026#34; %RESPONSE_CODE%\\n\u0026#34; After simplifying the access log format, we found that the QPS increased from 5.7K to 5.9K. When executing the on-CPU profiling again, the CPU usage of log formatting dropped from 2.4% to 0.7%.\nSimplifying the log format helped us to improve the performance.\nOff-CPU Profiling Off-CPU profiling is suitable for performance issues that are not caused by high CPU usage. For example, when there are too many threads in one service, using off-CPU profiling could reveal which threads spend more time context switching.\nWe provide data aggregation in two dimensions:\nSwitch count: The number of times a thread switches context. When the thread returns to the CPU, it completes one context switch. A thread stack with a higher switch count spends more time context switching. Switch duration: The time it takes a thread to switch the context. A thread stack with a higher switch duration spends more time off-CPU. Write Access Log Enable Write Using the same environment and settings as before in the on-CPU test, we performed off-CPU profiling. As shown below, we found that access log writes accounted for about 28% of the total context switches. The \u0026ldquo;__write\u0026rdquo; shown below also indicates that this method is the Linux kernel method.\nDisable Write SkyWalking implements Envoy\u0026rsquo;s Access Log Service (ALS) feature which allows us to send access logs to the SkyWalking Observability Analysis Platform (OAP) using the gRPC protocol. Even by disabling the access logging, we can still use ALS to capture/aggregate the logs. We\u0026rsquo;ve disabled writing to the access log using the following command:\nistioctl install -y --set profile=demo --set meshConfig.accessLogFile=\u0026#34;\u0026#34; After disabling the Access Log feature, we performed the off-CPU profiling. File writing entries have disappeared as shown in the figure below. Envoy throughput also increased from 5.7K to 5.9K.\nConclusion In this article, we\u0026rsquo;ve examined the insights Apache Skywalking\u0026rsquo;s Trace Profiling can give us and how much more can be achieved with eBPF profiling. All of these features are implemented in skywalking-rover. In addition to on- and off-CPU profiling, you will also find the following features:\nContinuous profiling, helps you automatically profile without manual intervention. For example, when Rover detects that the CPU exceeds a configurable threshold, it automatically executes the on-CPU profiling task. More profiling types to enrich usage scenarios, such as network, and memory profiling. ","excerpt":"\u003ch3 id=\"content\"\u003eContent\u003c/h3\u003e\n\u003ch1 id=\"background\"\u003eBackground\u003c/h1\u003e\n\u003cp\u003e\u003ca href=\"https://skywalking.apache.org/\"\u003eApache SkyWalking\u003c/a\u003e observes metrics, logs, traces, and events for services …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2022-07-05-pinpoint-service-mesh-critical-performance-impact-by-using-ebpf/","title":"Pinpoint Service Mesh Critical Performance Impact by using eBPF"},{"body":"SkyWalking Rust 0.2.0 is released. Go to downloads page to find release tars.\nWhat\u0026rsquo;s Changed add a description to compile in README.md by @Shikugawa in https://github.com/apache/skywalking-rust/pull/16 Update NOTICE to 2022 by @wu-sheng in https://github.com/apache/skywalking-rust/pull/17 fix ignore /e2e/target folder by @tisonkun in https://github.com/apache/skywalking-rust/pull/18 Remove Cargo.lock, update dependencies, update submodule, disable build grpc server api. by @jmjoy in https://github.com/apache/skywalking-rust/pull/19 Enhance Trace Context machenism. by @jmjoy in https://github.com/apache/skywalking-rust/pull/20 chore(typo): fix typo in context/propagation/context.rs by @CherishCai in https://github.com/apache/skywalking-rust/pull/21 Feature(tonic-build): set tonic-build.build_server(false), do not build Server code. by @CherishCai in https://github.com/apache/skywalking-rust/pull/22 Rename crate name skywalking_rust to skywalking? by @jmjoy in https://github.com/apache/skywalking-rust/pull/23 ","excerpt":"\u003cp\u003eSkyWalking Rust 0.2.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch2 id=\"whats-changed\"\u003eWhat\u0026rsquo;s Changed …\u003c/h2\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-rust-0-2-0/","title":"Release Apache SkyWalking Rust 0.2.0"},{"body":"B站视频地址\n","excerpt":"\u003cp\u003e\u003ca href=\"https://www.bilibili.com/video/BV1Cg411X71x\"\u003eB站视频地址\u003c/a\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2022-06-23-more-than-tracing-logging-metrics/","title":"阿里云 - 可观测技术峰会 2022 - More than Tracing Logging Metrics"},{"body":"SkyWalking Java Agent 8.11.0 is released. Go to downloads page to find release tars. Changes by Version\n8.11.0 Fix cluster and namespace value duplicated(namespace value) in properties report. Add layer field to event when reporting. Remove redundant shade.package property. Add servicecomb-2.x plugin and Testcase. Fix NPE in gateway plugin when the timer triggers webflux webclient call. Add an optional plugin, trace-sampler-cpu-policy-plugin, which could disable trace collecting in high CPU load. Change the dateformat of logs to yyyy-MM-dd HH:mm:ss.SSS(was yyyy-MM-dd HH:mm:ss:SSS). Fix NPE in elasticsearch plugin. Grpc plugin support trace client async generic call(without grpc stubs), support Method type: UNARY、SERVER_STREAMING. Enhance Apache ShenYu (incubating) plugin: support trace grpc,sofarpc,motan,tars rpc proxy. Add primary endpoint name to log events. Fix Span not finished in gateway plugin when the gateway request timeout. Support -Dlog4j2.contextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector in gRPC log report. Fix tcnative libraries relocation for aarch64. Add plugin.jdbc.trace_sql_parameters into Configuration Discovery Service. Fix argument type name of Array in postgresql-8.x-plugin from java.lang.String[] to [Ljava.lang.String; Add type name checking in ArgumentTypeNameMatch and ReturnTypeNameMatch Highlight ArgumentTypeNameMatch and ReturnTypeNameMatch type naming rule in docs/en/setup/service-agent/java-agent/Java-Plugin-Development-Guide.md Fix FileWriter scheduled task NPE Optimize gRPC Log reporter to set service name for the first element in the streaming.(No change for Kafka reporter) All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking Java Agent 8.11.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nChanges by …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-java-agent-8-11-0/","title":"Release Apache SkyWalking Java Agent 8.11.0"},{"body":"SkyWalking Rover 0.2.0 is released. Go to downloads page to find release tars.\nFeatures Support OFF_CPU Profiling. Introduce the BTFHub module. Update to using frequency mode to ON_CPU Profiling. Add logs in the profiling module logical. Bug Fixes Fix docker based process could not be detected. Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Rover 0.2.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-rover-0-2-0/","title":"Release Apache SkyWalking Rover 0.2.0"},{"body":"SkyWalking 9.1.0 is released. Go to downloads page to find release tars.\neBPF agent(skywalking rover) is integrated in the first time BanyanDB(skywalking native database) is integrated and passed MVP phase. On-demand logs are provided first time in skywalking for all mesh services and k8s deployment as a zero cost log solution Zipkin alternative is being official, and Zipkin\u0026rsquo;s HTTP APIs are supported as well as lens UI. Changes by Version Project [IMPORTANT] Remove InfluxDB 1.x and Apache IoTDB 0.X as storage options, check details at here. Remove converter-moshi 2.5.0, influx-java 2.15, iotdb java 0.12.5, thrift 0.14.1, moshi 1.5.0, msgpack 0.8.16 dependencies. Remove InfluxDB and IoTDB relative codes and E2E tests. Upgrade OAP dependencies zipkin to 2.23.16, H2 to 2.1.212, Apache Freemarker to 2.3.31, gRPC-java 1.46.0, netty to 4.1.76. Upgrade Webapp dependencies, spring-cloud-dependencies to 2021.0.2, logback-classic to 1.2.11 [IMPORTANT] Add BanyanDB storage implementation. Notice BanyanDB is currently under active development and SHOULD NOT be used in production cluster. OAP Server Add component definition(ID=127) for Apache ShenYu (incubating). Fix Zipkin receiver: Decode spans error, missing Layer for V9 and wrong time bucket for generate Service and Endpoint. [Refactor] Move SQLDatabase(H2/MySQL/PostgreSQL), ElasticSearch and BanyanDB specific configurations out of column. Support BanyanDB global index for entities. Log and Segment record entities declare this new feature. Remove unnecessary analyzer settings in columns of templates. Many were added due to analyzer\u0026rsquo;s default value. Simplify the Kafka Fetch configuration in cluster mode. [Breaking Change] Update the eBPF Profiling task to the service level, please delete index/table: ebpf_profiling_task, process_traffic. Fix event can\u0026rsquo;t split service ID into 2 parts. Fix OAP Self-Observability metric GC Time calculation. Set SW_QUERY_MAX_QUERY_COMPLEXITY default value to 1000 Webapp module (for UI) enabled compression. [Breaking Change] Add layer field to event, report an event without layer is not allowed. Fix ES flush thread stops when flush schedule task throws exception, such as ElasticSearch flush failed. Fix ES BulkProcessor in BatchProcessEsDAO was initialized multiple times and created multiple ES flush schedule tasks. HTTPServer support the handler register with allowed HTTP methods. [Critical] Revert Enhance DataCarrier#MultipleChannelsConsumer to add priority to avoid consuming issues. Fix the problem that some configurations (such as group.id) did not take effect due to the override order when using the kafkaConsumerConfig property to extend the configuration in Kafka Fetcher. Remove build time from the OAP version. Add data-generator module to run OAP in testing mode, generating mock data for testing. Support receive Kubernetes processes from gRPC protocol. Fix the problem that es index(TimeSeriesTable, eg. endpoint_traffic, alarm_record) didn\u0026rsquo;t create even after rerun with init-mode. This problem caused the OAP server to fail to start when the OAP server was down for more than a day. Support autocomplete tags in traces query. [Breaking Change] Replace all configurations **_JETTY_** to **_REST_**. Add the support eBPF profiling field into the process entity. E2E: fix log test miss verify LAL and metrics. Enhance Converter mechanism in kernel level to make BanyanDB native feature more effective. Add TermsAggregation properties collect_mode and execution_hint. Add \u0026ldquo;execution_hint\u0026rdquo;: \u0026ldquo;map\u0026rdquo;, \u0026ldquo;collect_mode\u0026rdquo;: \u0026ldquo;breadth_first\u0026rdquo; for aggregation and topology query to improve 5-10x performance. Clean up scroll contexts after used. Support autocomplete tags in logs query. Enhance Deprecated MetricQuery(v1) getValues querying to asynchronous concurrency query Fix the pod match error when the service has multiple selector in kubernetes environment. VM monitoring adapts the 0.50.0 of the opentelemetry-collector. Add Envoy internal cost metrics. Remove Layer concept from ServiceInstance. Remove unnecessary onCompleted on gRPC onError callback. Remove Layer concept form Process. Update to list all eBPF profiling schedulers without duration. Storage(ElasticSearch): add search options to tolerate inexisting indices. Fix the problem that MQ has the wrong Layer type. Fix NoneStream model has wrong downsampling(was Second, should be Minute). SQL Database: provide @SQLDatabase.AdditionalEntity to support create additional tables from a model. [Breaking Change] SQL Database: remove SQL Database config maxSizeOfArrayColumn and numOfSearchableValuesPerTag. [Breaking Change] SQL Database: move Tags list from Segment,Logs,Alarms to their additional table. [Breaking Change] Remove total field in Trace, Log, Event, Browser log, and alarm list query. Support OFF_CPU eBPF Profiling. Fix SumAggregationBuilder#build should use the SumAggregation rather than MaxAggregation. Add TiDB, OpenSearch, Postgres storage optional to Trace and eBPF Profiling E2E testing. Add OFF CPU eBPF Profiling E2E Testing. Fix searchableTag as rpc.status_code and http.status_code. status_code had been removed. Fix scroll query failure exception. Add profileDataQueryBatchSize config in Elasticsearch Storage. Add APIs to query Pod log on demand. Remove OAL for events. Simplify the format index name logical in ES storage. Add instance properties extractor in MAL. Support Zipkin traces collect and zipkin traces query API. [Breaking Change] Zipkin receiver mechanism changes and traces do not stream into OAP Segment anymore. UI General service instance: move Thread Pool from JVM to Overview, fix JVM GC Count calculation. Add Apache ShenYu (incubating) component LOGO. Show more metrics on service/instance/endpoint list on the dashboards. Support average values of metrics on the service/list/endpoint table widgets, with pop-up linear graph. Fix viewLogs button query no data. Fix UTC when page loads. Implement the eBPF profile widget on dashboard. Optimize the trace widget. Avoid invalid query for topology metrics. Add the alarm and log tag tips. Fix spans details and task logs. Verify query params to avoid invalid queries. Mobile terminal adaptation. Fix: set dropdown for the Tab widget, init instance/endpoint relation selectors, update sankey graph. Add eBPF Profiling widget into General service, Service Mesh and Kubernetes tabs. Fix jump to endpoint-relation dashboard template. Fix set graph options. Remove the Layer filed from the Instance and Process. Fix date time picker display when set hour to 0. Implement tags auto-complete for Trace and Log. Support multiple trees for the flame graph. Fix the page doesn\u0026rsquo;t need to be re-rendered when the url changes. Remove unexpected data for exporting dashboards. Fix duration time. Remove the total field from query conditions. Fix minDuration and maxDuration for the trace filter. Add Log configuration for the browser templates. Fix query conditions for the browser logs. Add Spanish Translation. Visualize the OFF CPU eBPF profiling. Add Spanish language to UI. Sort spans with startTime or spanId in a segment. Visualize a on-demand log widget. Fix activate the correct tab index after renaming a Tabs name. FaaS dashboard support on-demand log (OpenFunction/functions-framework-go version \u0026gt; 0.3.0). Documentation Add eBPF agent into probe introduction. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking 9.1.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eeBPF agent(skywalking …\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-9.1.0/","title":"Release Apache SkyWalking APM 9.1.0"},{"body":"SkyWalking BanyanDB 0.1.0 is released. Go to downloads page to find release tars.\nFeatures BanyanD is the server of BanyanDB TSDB module. It provides the primary time series database with a key-value data module. Stream module. It implements the stream data model\u0026rsquo;s writing. Measure module. It implements the measure data model\u0026rsquo;s writing. Metadata module. It implements resource registering and property CRUD. Query module. It handles the querying requests of stream and measure. Liaison module. It\u0026rsquo;s the gateway to other modules and provides access endpoints to clients. gRPC based APIs Document API reference Installation instrument Basic concepts Testing UT E2E with Java Client and OAP ","excerpt":"\u003cp\u003eSkyWalking BanyanDB 0.1.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"features\"\u003eFeatures\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eBanyanD …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-0-1-0/","title":"Release Apache SkyWalking BanyanDB 0.1.0"},{"body":"SkyWalking BanyanDB 0.1.0 is released. Go to downloads page to find release tars.\nFeatures Support Measure, Stream and Property Query and Write APIs Support Metadata Management APIs for Measure, Stream, IndexRule and IndexRuleBinding Chores Set up GitHub actions to check code styles, licenses, and tests. ","excerpt":"\u003cp\u003eSkyWalking BanyanDB 0.1.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-banyandb-java-client-0-1-0/","title":"Release Apache SkyWalking BanyanDB Java Client 0.1.0"},{"body":"SkyWalking Rover 0.1.0 is released. Go to downloads page to find release tars.\nFeatures Support detect processes in scanner or kubernetes mode. Support profiling C, C++, Golang, and Rust service. Bug Fixes Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Rover 0.1.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-rover-0-1-0/","title":"Release Apache SkyWalking Rover 0.1.0"},{"body":"SkyWalking Satellite 1.0.0 is released. Go to downloads page to find release tars.\nFeatures Add the compat protocol receiver for the old version of agents. Support transmit the native eBPF Process and Profiling protocol. Change the name of plugin that is not well-named. Bug Fixes Fix Metadata lost in the Native Meter protocol. Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Satellite 1.0.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd the …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-satellite-1-0-0/","title":"Release Apache SkyWalking Satellite 1.0.0"},{"body":"SkyWalking Eyes 0.3.0 is released. Go to downloads page to find release tars.\nDependency License\nFix license check in go library testify (#93) License Header\nfix command supports more languages: Add comment style for cmake language (#86) Add comment style for hcl (#89) Add mpl-2.0 header template (#87) Support fix license header for tcl files (#102) Add python docstring comment style (#100) Add comment style for makefile \u0026amp; editorconfig (#90) Support config license header comment style (#97) Trim leading and trailing newlines before rewrite license header cotent (#94) Replace already existing license header based on pattern (#98) [docs] add the usage for config the license header comment style (#99) Project\nObtain default github token in github actions (#82) Add tests for bare spdx license header content (#92) Add github action step summary for better experience (#104) Adds an option to the action to run in fix mode (#84) Provide --summary flag to generate the license summary file (#103) Add .exe suffix to windows binary (#101) Fix wrong file path and exclude binary files in src release (#81) Use t.tempdir to create temporary test directory (#95) Config: fix incorrect log message (#91) [docs] correct spelling mistakes (#96) ","excerpt":"\u003cp\u003eSkyWalking Eyes 0.3.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eDependency License …\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-eyes-0-3-0/","title":"Release Apache SkyWalking Eyes 0.3.0"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/apache-shenyu-incubating/","title":"Apache ShenYu (Incubating)"},{"body":"目录 SkyWalking和ShenYu介绍 ApacheShenYu插件实现原理 给gRPC插件增加泛化调用追踪并保持兼容 ShenYu网关可观测性实践 总结 1.SkyWalking和ShenYu介绍 1.1 SkyWalking SkyWalking是一个针对微服务、分布式系统、云原生的应用性能监控(APM)和可观测性分析平台(OAP), 拥有强大的功能，提供了多维度应用性能分析手段，包含分布式拓扑图、应用性能指标、分布式链路追踪、日志关联分析和告警。同时还拥有非常丰富的生态。广泛应用于各个公司和开源项目。\n1.2 Apache ShenYu (incubating) Apache ShenYu (incubating)是一个高性能，多协议，易扩展，响应式的API网关。 兼容各种主流框架体系，支持热插拔，用户可以定制化开发，满足用户各种场景的现状和未来需求，经历过大规模场景的锤炼。 支持丰富的协议：Http、Spring Cloud、gRPC、Dubbo、SOFARPC、Motan、Tars等等。\n2.ApacheShenYu插件实现原理 ShenYu的异步和以往接触的异步有一点不一样，是一种全链路异步，每一个插件的执行都是异步的，并且线程切换并不是单一固定的情况(和各个插件实现有关)。 网关会发起各种协议类型的服务调用，现有的SkyWalking插件发起服务调用的时候会创建ExitSpan(同步或异步). 网关接收到请求会创建异步的EntrySpan。 异步的EntrySpan需要和同步或异步的ExitSpan串联起来，否则链路会断。 串联方案有2种：\n快照传递： 将创建EntrySpan之后的快照通过某种方式传递到创建ExitSpan的线程中。\n目前这种方式应用在异步的WebClient插件中，该插件能接收异步快照。ShenYu代理Http服务或SpringCloud服务便是通过快照传递实现span串联。 LocalSpan中转： 其它RPC类插件不像异步WebClient那样可以接收快照实现串联。尽管你可以改动其它RPC插件让其接收快照实现串联，但不推荐也没必要， 因为可以通过在创建ExitSpan的线程中，创建一个LocalSpan就可以实现和ExitSpan串联，然后将异步的EntrySpan和LocalSpan通过快照传递的方式串联。这样实现完全可以不改动原先插件的代码。 span连接如下图所示:\n也许你会问是否可以在一个通用的插件里面创建LocalSpan,而不是ShenYu RPC插件分别创建一个？ 答案是不行，因为需要保证LocalSpan和ExitSpan在同一个线程，而ShenYu是全链路异步. 在实现上创建LocalSpan的代码是复用的。\n3. 给gRPC插件增加泛化调用追踪并保持兼容 现有的SkyWalking gRPC插件只支持通过存根的方式发起的调用。而对于网关而言并没有proto文件，网关采取的是泛化调用(不通过存根)，所以追踪rpc请求，你会发现链路会在网关节点断掉。 在这种情况下，需要让gRPC插件支持泛化调用，而同时需要保持兼容，不影响原先的追踪方式。实现上通过判断请求参数是否是动态消息(DynamicMessage)，如果不是则走原先通过存根的追踪逻辑， 如果是则走泛化调用追踪逻辑。另外的兼容则是在gRPC新旧版本的差异，以及获取服务端IP各种情况的兼容，感兴趣的可以看看源码。\n4. ShenYu网关可观测性实践 上面讲解了SkyWalking ShenYu插件的实现原理，下面部署应用看下效果。SkyWalking功能强大，除了了链路追踪需要开发插件外，其它功能强大功能开箱即用。 这里只描述链路追踪和应用性能剖析部分，如果想体验SkyWalking功能的强大，请参考SkyWalking官方文档。\n版本说明：\nskywalking-java: 8.11.0-SNAPSHOT源码构建。说明：shenyu插件会在8.11.0版本发布，可能会在5月或6月初步发布它。Java代理正处于常规发布阶段。 skywalking: 9.0.0 V9 版本 用法说明:\nSkyWalking的设计非常易用，配置和激活插件请参考官方文档。\nSkyWalking Documentation SkyWalking Java Agent Documentation 4.1 向网关发起请求 通过postman客户端或者其它方式向网关发起各种服务请求\n4.2 请求拓扑图 4.3 请求链路(以gRPC为例) 正常链路： 异常链路： 点击链路节点变可以看到对应的节点信息和异常信息\n服务提供者span 网关请求span 4.4 服务指标监控 服务指标监控 4.5 网关后台指标监控 数据库监控: 线程池和连接池监控 4.6 JVM监控 4.7 接口分析 4.8 异常日志和异常链路分析 日志配置见官方文档\n日志监控 异常日志对应的分布式链路追踪详情 5. 总结 SkyWalking在可观测性方面对指标、链路追踪、日志有着非常全面的支持，功能强大，简单易用，专为大型分布式系统、微服务、云原生、容器架构而设计，拥有丰富的生态。 使用SkyWalking为Apache ShenYu (incubating)提供强大的可观测性支持，让ShenYu如虎添翼。最后，如果你对高性能响应式网关感兴趣，可以关注 Apache ShenYu (incubating) 。 同时感谢SkyWalking这么优秀的开源软件对行业所作的贡献。\n","excerpt":"\u003ch3 id=\"目录\"\u003e目录\u003c/h3\u003e\n\u003col\u003e\n\u003cli\u003e\u003ca href=\"#1.-SkyWalking%E5%92%8CShenYu%E4%BB%8B%E7%BB%8D\"\u003eSkyWalking和ShenYu介绍\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#2.-ApacheShenYu%E6%8F%92%E4%BB%B6%E5%AE%9E%E7%8E%B0%E5%8E%9F%E7%90%86\"\u003eApacheShenYu插件实现原理\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#3.-%E7%BB%99gRPC%E6%8F%92%E4%BB%B6%E5%A2%9E%E5%8A%A0%E6%B3%9B%E5%8C%96%E8%B0%83%E7%94%A8%E8%BF%BD%E8%B8%AA%E5%B9%B6%E4%BF%9D%E6%8C%81%E5%85%BC%E5%AE%B9\"\u003e给gRPC插件增加泛化调用追踪并保持兼容\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#4.-ShenYu%E7%BD%91%E5%85%B3%E5%8F%AF%E8%A7%82%E6%B5%8B%E6%80%A7%E5%AE%9E%E8%B7%B5\"\u003eShenYu网关可观测性实践\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#5.-%E6%80%BB%E7%BB%93\"\u003e总结\u003c/a\u003e\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch2 id=\"1skywalking和shenyu介绍\"\u003e1.SkyWalking和 …\u003c/h2\u003e","ref":"https://skywalking.apache.org/zh/2022-05-08-apache-shenyuincubating-integrated-skywalking-practice-observability/","title":"Apache ShenYu (incubating)插件实现原理和可观测性实践"},{"body":"Content Introduction of SkyWalking and ShenYu Apache ShenYu plugin implementation principle Adding generalized call tracking to the gRPC plugin and keeping it compatible ShenYu Gateway Observability Practice Summary 1. Introduction of SkyWalking and ShenYu 1.1 SkyWalking SkyWalking is an Application Performance Monitoring (APM) and Observability Analysis Platform (OAP) for microservices, distributed systems, and cloud natives, Has powerful features that provide a multi-dimensional means of application performance analysis, including distributed topology diagrams, application performance metrics, distributed link tracing, log correlation analysis and alerts. Also has a very rich ecology. Widely used in various companies and open source projects.\n1.2 Apache ShenYu (incubating) Apache ShenYu (incubating) High-performance,multi-protocol,extensible,responsive API Gateway. Compatible with a variety of mainstream framework systems, support hot plug, users can customize the development, meet the current situation and future needs of users in a variety of scenarios, experienced the temper of large-scale scenes. Rich protocol support: Http, Spring Cloud, gRPC, Dubbo, SOFARPC, Motan, Tars, etc.\n2. Apache ShenYu plugin implementation principle ShenYu\u0026rsquo;s asynchrony is a little different from previous exposure to asynchrony, it is a full-link asynchrony, the execution of each plug-in is asynchronous, and thread switching is not a single fixed situation (and the individual plug-in implementation is related). The gateway initiates service calls of various protocol types, and the existing SkyWalking plugins create ExitSpan (synchronous or asynchronous) when they initiate service calls. The gateway receives the request and creates an asynchronous EntrySpan. The asynchronous EntrySpan needs to be concatenated with the synchronous or asynchronous ExitSpan, otherwise the link will be broken.\nThere are 2 types of tandem solutions：\nSnapshot Delivery:\nPass the snapshot after creating the EntrySpan to the thread that created the ExitSpan in some way.\nCurrently this approach is used in the asynchronous WebClient plugin, which can receive asynchronous snapshots. shenYu proxy Http service or SpringCloud service is to achieve span concatenation through snapshot passing. LocalSpan transit:\nOther RPC class plugins do not receive snapshots for concatenation like Asynchronous WebClient. Although you can modify other RPC plugins to receive snapshots for concatenation, it is not recommended or necessary to do so. This can be achieved by creating a LocalSpan in the thread where the ExitSpan is created, and then connecting the asynchronous EntrySpan and LocalSpan by snapshot passing. This can be done without changing the original plugin code. The span connection is shown below:\nYou may ask if it is possible to create LocalSpan inside a generic plugin, instead of creating one separately for ShenYu RPC plugin? The answer is no, because you need to ensure that LocalSpan and ExitSpan are in the same thread, and ShenYu is fully linked asynchronously. The code to create LocalSpan is reused in the implementation.\n3. Adding generalized call tracking to the gRPC plugin and keeping it compatible The existing SkyWalking gRPC plugin only supports calls initiated by way of stubs. For the gateway there is no proto file, the gateway takes generalized calls (not through stubs), so tracing RPC requests, you will find that the link will break at the gateway node. In this case, it is necessary to make the gRPC plugin support generalized calls, while at the same time needing to remain compatible and not affect the original tracing method. This is achieved by determining whether the request parameter is a DynamicMessage, and if it is not, then the original tracing logic through the stub is used. If not, then the original tracing logic via stubs is used, and if not, then the generalized call tracing logic is used. The other compatibility is the difference between the old and new versions of gRPC, as well as the compatibility of various cases of obtaining server-side IP, for those interested in the source code.\n4. ShenYu Gateway Observability Practice The above explains the principle of SkyWalking ShenYu plug-in implementation, the following deployment application to see the effect. SkyWalking powerful, in addition to the link tracking requires the development of plug-ins, other powerful features out of the box. Here only describe the link tracking and application performance analysis part, if you want to experience the power of SkyWalking features, please refer to the SkyWalking official documentation.\nVersion description:\nskywalking-java: 8.11.0-SNAPSHOT source code build. Note: The shenyu plugin will be released in version 8.11.0, and will probably release it initially in May or June. the Java agent is in the regular release phase. skywalking: 9.0.0 V9 version Usage instructions:\nSkyWalking is designed to be very easy to use. Please refer to the official documentation for configuring and activating the shenyu plugin.\nSkyWalking Documentation SkyWalking Java Agent Documentation 4.1 Sending requests to the gateway Initiate various service requests to the gateway via the postman client or other means.\n4.2 Request Topology Diagram 4.3 Request Trace (in the case of gRPC) Normal Trace： Abnormal Trace： Click on the link node to see the corresponding node information and exception information\nService Provider Span Gateway request span 4.4 Service Metrics Monitoring 4.5 Gateway background metrics monitoring Database Monitoring: Thread pool and connection pool monitoring: 4.6 JVM Monitoring 4.7 Endpoint Analysis 4.8 Exception log and exception link analysis See official documentation for log configuration\nLog monitoring Distributed link trace details corresponding to exception logs 5. Summary SkyWalking has very comprehensive support for metrics, link tracing, and logging in observability, and is powerful, easy to use, and designed for large distributed systems, microservices, cloud-native, container architectures, and has a rich ecosystem. Using SkyWalking to provide powerful observability support for Apache ShenYu (incubating) gives ShenYu a boost. Finally, if you are interested in high-performance responsive gateways, you can follow Apache ShenYu (incubating). Also, thanks to SkyWalking such an excellent open source software to the industry contributions.\n","excerpt":"\u003ch3 id=\"content\"\u003eContent\u003c/h3\u003e\n\u003col\u003e\n\u003cli\u003e\u003ca href=\"#1.-Introduction-of-SkyWalking-and-ShenYu\"\u003eIntroduction of SkyWalking and ShenYu\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#2.-Apache-ShenYu-plugin-implementation-principle\"\u003eApache ShenYu plugin implementation principle\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#3.-Adding-generalized-call-tracking-to-the-gRPC-plugin-and-keeping-it-compatible\"\u003eAdding …\u003c/a\u003e\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/blog/2022-05-08-apache-shenyuincubating-integrated-skywalking-practice-observability/","title":"Apache ShenYu(incubating) plugin implementation principles and observability practices"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/logging/","title":"Logging"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/metrics/","title":"Metrics"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/observability/","title":"Observability"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/skywalking/","title":"SkyWalking"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/tracing/","title":"Tracing"},{"body":"SkyWalking Kubernetes Event Exporter 1.0.0 is released. Go to downloads page to find release tars.\nAdd Apache SkyWalking exporter to export events into SkyWalking OAP. Add console exporter for debugging purpose. ","excerpt":"\u003cp\u003eSkyWalking Kubernetes Event Exporter 1.0.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars. …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-kubernetes-event-exporter-1.0.0/","title":"Release Apache SkyWalking Kubernetes Event Exporter 1.0.0"},{"body":"content: Introduction Features Install SWCK Deploy a demo application Verify the injector Concluding remarks 1. Introduction 1.1 What\u0026rsquo;s SWCK? SWCK is a platform for the SkyWalking user, provisions, upgrades, maintains SkyWalking relevant components, and makes them work natively on Kubernetes.\nIn fact, SWCK is an operator developed based on kubebuilder, providing users with Custom Resources ( CR ) and controllers for managing resources ( Controller ), all CustomResourceDefinitions（CRDs）are as follows:\nJavaAgent OAP UI Storage Satellite Fetcher 1.2 What\u0026rsquo;s the java agent injector? For a java application, users need to inject the java agent into the application to get metadata and send it to the SkyWalking backend. To make users use the java agent more natively, we propose the java agent injector to inject the java agent sidecar into a pod. The java agent injector is actually a Kubernetes Mutation Webhook Controller. The controller intercepts pod events and applies mutations to the pod if annotations exist within the request.\n2. Features Transparent. User’s applications generally run in normal containers while the java agent runs in the init container, and both belong to the same pod. Each container in the pod mounts a shared memory volume that provides a storage path for the java agent. When the pod starts, the java agent in the init container will run before the application container, and the injector will store the java agent file in the shared memory volume. When the application container starts, the injector injects the agent file into the application by setting the JVM parameter. Users can inject the java agent in this way without rebuilding the container image containing the java agent.\nConfigurability. The injector provides two ways to configure the java agent: global configuration and custom configuration. The default global configuration is stored in the configmap, you can update it as your own global configuration, such as backend_service. In addition, you can also set custom configuration for some applications via annotation, such as “service_name”. For more information, please see java-agent-injector.\nObservability. For each injected java agent, we provide CustomDefinitionResources called JavaAgent to observe the final agent configuration. Please refer to javaagent to get more details.\n3. Install SWCK In the next steps, we will show how to build a stand-alone Kubernetes cluster and deploy the 0.6.1 version of SWCK on the platform.\n3.1 Tool Preparation Firstly, you need to install some tools as follows:\nkind, which is used to create a stand-alone Kubernetes cluster. kubectl, which is used to communicate with the Kubernetes cluster. 3.2 Install stand-alone Kubernetes cluster After installing kind , you could use the following command to create a stand-alone Kubernetes cluster.\nNotice! If your terminal is configured with a proxy, you need to close it before the cluster is created to avoid some errors.\n$ kind create cluster --image=kindest/node:v1.19.1 After creating a cluster, you can get the pods as below.\n$ kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-f9fd979d6-57xpc 1/1 Running 0 7m16s kube-system coredns-f9fd979d6-8zj8h 1/1 Running 0 7m16s kube-system etcd-kind-control-plane 1/1 Running 0 7m23s kube-system kindnet-gc9gt 1/1 Running 0 7m16s kube-system kube-apiserver-kind-control-plane 1/1 Running 0 7m23s kube-system kube-controller-manager-kind-control-plane 1/1 Running 0 7m23s kube-system kube-proxy-6zbtb 1/1 Running 0 7m16s kube-system kube-scheduler-kind-control-plane 1/1 Running 0 7m23s local-path-storage local-path-provisioner-78776bfc44-jwwcs 1/1 Running 0 7m16s 3.3 Install certificates manger(cert-manger) The certificates of SWCK are distributed and verified by the certificate manager. You need to install the cert-manager through the following command.\n$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.yaml Verify whether cert-manager is installed successfully.\n$ kubectl get pod -n cert-manager NAME READY STATUS RESTARTS AGE cert-manager-7dd5854bb4-slcmd 1/1 Running 0 73s cert-manager-cainjector-64c949654c-tfmt2 1/1 Running 0 73s cert-manager-webhook-6bdffc7c9d-h8cfv 1/1 Running 0 73s 3.4 Install SWCK The java agent injector is a component of the operator, so please follow the next steps to install the operator first.\nGet the deployment yaml file of SWCK and deploy it. $ curl -Ls https://archive.apache.org/dist/skywalking/swck/0.6.1/skywalking-swck-0.6.1-bin.tgz | tar -zxf - -O ./config/operator-bundle.yaml | kubectl apply -f - Check SWCK as below. $ kubectl get pod -n skywalking-swck-system NAME READY STATUS RESTARTS AGE skywalking-swck-controller-manager-7f64f996fc-qh8s9 2/2 Running 0 94s 3.5 Install Skywalking components — OAPServer and UI Deploy the OAPServer and UI in the default namespace. $ kubectl apply -f https://raw.githubusercontent.com/apache/skywalking-swck/master/operator/config/samples/default.yaml Check the OAPServer. $ kubectl get oapserver NAME INSTANCES RUNNING ADDRESS default 1 1 default-oap.default Check the UI. $ kubectl get ui NAME INSTANCES RUNNING INTERNALADDRESS EXTERNALIPS PORTS default 1 1 default-ui.default [80] 4. Deploy a demo application In the third step, we have installed SWCK and related Skywalking components. Next, we will show how to use the java agent injector in SWCK through two java application examples in two ways: global configuration and custom configuration.\n4.1 Set the global configuration When we have installed SWCK, the default configuration is the configmap in the system namespace, we can get it as follows.\n$ kubectl get configmap skywalking-swck-java-agent-configmap -n skywalking-swck-system -oyaml apiVersion: v1 data: agent.config: |- # The service name in UI agent.service_name=${SW_AGENT_NAME:Your_ApplicationName} # Backend service addresses. collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:127.0.0.1:11800} # Please refer to https://skywalking.apache.org/docs/skywalking-java/latest/en/setup/service-agent/java-agent/configurations/#table-of-agent-configuration-properties to get more details. In the cluster created by kind, the backend_service may not be correct, we need to use the real OAPServer\u0026rsquo;s address default-oap.default to replace the default 127.0.0.1, so we can edit the configmap as follow.\n$ kubectl edit configmap skywalking-swck-java-agent-configmap -n skywalking-swck-system configmap/skywalking-swck-java-agent-configmap edited $ kubectl get configmap skywalking-swck-java-agent-configmap -n skywalking-swck-system -oyaml apiVersion: v1 data: agent.config: |- # The service name in UI agent.service_name=${SW_AGENT_NAME:Your_ApplicationName} # Backend service addresses. collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:default-oap.default:11800} # Please refer to https://skywalking.apache.org/docs/skywalking-java/latest/en/setup/service-agent/java-agent/configurations/#table-of-agent-configuration-properties to get more details. 4.2 Set the custom configuration In some cases, we need to use the Skywalking component to monitor different java applications, so the agent configuration of different applications may be different, such as the name of the application, and the plugins that the application needs to use, etc. Next, we will take two simple java applications developed based on spring boot and spring cloud gateway as examples for a detailed description. You can use the source code to build the image.\n# build the springboot and springcloudgateway image $ git clone https://github.com/dashanji/swck-spring-cloud-k8s-demo $ cd swck-spring-cloud-k8s-demo \u0026amp;\u0026amp; make # check the image $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE gateway v0.0.1 51d16251c1d5 48 minutes ago 723MB app v0.0.1 62f4dbcde2ed 48 minutes ago 561MB # load the image into the cluster $ kind load docker-image app:v0.0.1 \u0026amp;\u0026amp; kind load docker-image gateway:v0.0.1 4.3 deploy spring boot application Create the springboot-system namespace. $ kubectl create namespace springboot-system Label the springboot-systemnamespace to enable the java agent injector. $ kubectl label namespace springboot-system swck-injection=enabled Deploy the corresponding deployment file springboot.yaml for the spring boot application, which uses annotation to override the default agent configuration, such as service_name. Notice! Before using the annotation to override the agent configuration, you need to add strategy.skywalking.apache.org/agent.Overlay: \u0026quot;true\u0026quot; to make the override take effect.\napiVersion: apps/v1 kind: Deployment metadata: name: demo-springboot namespace: springboot-system spec: selector: matchLabels: app: demo-springboot template: metadata: labels: swck-java-agent-injected: \u0026#34;true\u0026#34; # enable the java agent injector app: demo-springboot annotations: strategy.skywalking.apache.org/agent.Overlay: \u0026#34;true\u0026#34; # enable the agent overlay agent.skywalking.apache.org/agent.service_name: \u0026#34;backend-service\u0026#34; spec: containers: - name: springboot imagePullPolicy: IfNotPresent image: app:v0.0.1 command: [\u0026#34;java\u0026#34;] args: [\u0026#34;-jar\u0026#34;,\u0026#34;/app.jar\u0026#34;] --- apiVersion: v1 kind: Service metadata: name: demo namespace: springboot-system spec: type: ClusterIP ports: - name: 8085-tcp port: 8085 protocol: TCP targetPort: 8085 selector: app: demo-springboot Deploy a spring boot application in the springboot-system namespace. $ kubectl apply -f springboot.yaml Check for deployment. $ kubectl get pod -n springboot-system NAME READY STATUS RESTARTS AGE demo-springboot-7c89f79885-dvk8m 1/1 Running 0 11s Get the finnal injected java agent configuration through JavaAgent. $ kubectl get javaagent -n springboot-system NAME PODSELECTOR SERVICENAME BACKENDSERVICE app-demo-springboot-javaagent app=demo-springboot backend-service default-oap.default:11800 4.4 deploy spring cloud gateway application Create the gateway-system namespace. $ kubectl create namespace gateway-system Label the gateway-systemnamespace to enable the java agent injector. $ kubectl label namespace gateway-system swck-injection=enabled Deploy the corresponding deployment file springgateway.yaml for the spring cloud gateway application, which uses annotation to override the default agent configuration, such as service_name. In addition, when using spring cloud gateway, we need to add the spring cloud gateway plugin to the agent configuration. Notice! Before using the annotation to override the agent configuration, you need to add strategy.skywalking.apache.org/agent.Overlay: \u0026quot;true\u0026quot; to make the override take effect.\napiVersion: apps/v1 kind: Deployment metadata: labels: app: demo-gateway name: demo-gateway namespace: gateway-system spec: selector: matchLabels: app: demo-gateway template: metadata: labels: swck-java-agent-injected: \u0026#34;true\u0026#34; app: demo-gateway annotations: strategy.skywalking.apache.org/agent.Overlay: \u0026#34;true\u0026#34; agent.skywalking.apache.org/agent.service_name: \u0026#34;gateway-service\u0026#34; optional.skywalking.apache.org: \u0026#34;cloud-gateway-3.x\u0026#34; # add spring cloud gateway plugin spec: containers: - image: gateway:v0.0.1 name: gateway command: [\u0026#34;java\u0026#34;] args: [\u0026#34;-jar\u0026#34;,\u0026#34;/gateway.jar\u0026#34;] --- apiVersion: v1 kind: Service metadata: name: service-gateway namespace: gateway-system spec: type: ClusterIP ports: - name: 9999-tcp port: 9999 protocol: TCP targetPort: 9999 selector: app: demo-gateway Deploy a spring cloud gateway application in the gateway-system namespace. $ kubectl apply -f springgateway.yaml Check for deployment. $ kubectl get pod -n gateway-system NAME READY STATUS RESTARTS AGE demo-gateway-5bb77f6d85-9j7c6 1/1 Running 0 15s Get the finnal injected java agent configuration through JavaAgent. $ kubectl get javaagent -n gateway-system NAME PODSELECTOR SERVICENAME BACKENDSERVICE app-demo-gateway-javaagent app=demo-gateway gateway-service default-oap.default:11800 5. Verify the injector After completing the above steps, we can view detailed state of the injected pod, like the injected agent container. # get all injected pod $ kubectl get pod -A -lswck-java-agent-injected=true NAMESPACE NAME READY STATUS RESTARTS AGE gateway-system demo-gateway-5bb77f6d85-lt4z7 1/1 Running 0 69s springboot-system demo-springboot-7c89f79885-lkb5j 1/1 Running 0 75s # view detailed state of the injected pod [demo-springboot] $ kubectl describe pod -l app=demo-springboot -n springboot-system ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- ... Normal Created 91s kubelet,kind-control-plane Created container inject-skywalking-agent Normal Started 91s kubelet,kind-control-plane Started container inject-skywalking-agent ... Normal Created 90s kubelet,kind-control-plane Created container springboot Normal Started 90s kubelet,kind-control-plane Started container springboot # view detailed state of the injected pod [demo-gateway] $ kubectl describe pod -l app=demo-gateway -n gateway-system ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- ... Normal Created 2m20s kubelet,kind-control-plane Created container inject-skywalking-agent Normal Started 2m20s kubelet,kind-control-plane Started container inject-skywalking-agent ... Normal Created 2m20s kubelet,kind-control-plane Created container gateway Normal Started 2m20s kubelet,kind-control-plane Started container gateway Now we can expose the service and watch the data displayed on the web. First of all, we need to get the gateway service and the ui service as follows. $ kubectl get service service-gateway -n gateway-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service-gateway ClusterIP 10.99.181.145 \u0026lt;none\u0026gt; 9999/TCP 9m19s $ kubectl get service default-ui NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default-ui ClusterIP 10.111.39.250 \u0026lt;none\u0026gt; 80/TCP 82m Then open two terminals to expose the service: service-gateway、default-ui. $ kubectl port-forward service/service-gateway -n gateway-system 9999:9999 Forwarding from 127.0.0.1:9999 -\u0026gt; 9999 Forwarding from [::1]:9999 -\u0026gt; 9999 $ kubectl port-forward service/default-ui 8090:80 Forwarding from 127.0.0.1:8090 -\u0026gt; 8080 Forwarding from [::1]:8090 -\u0026gt; 8080 Use the following commands to access the spring boot demo 10 times through the spring cloud gateway service. $ for i in {1..10}; do curl http://127.0.0.1:9999/gateway/hello \u0026amp;\u0026amp; echo \u0026#34;\u0026#34;; done Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! We can see the Dashboard by accessing http://127.0.0.1:8090. All services\u0026rsquo; topology is shown below. We can see the trace information of gateway-service. We can see the trace information of backend-service. 6. Concluding remarks If your application is deployed in the Kubernetes platform and requires Skywalking to provide monitoring services, SWCK can help you deploy, upgrade and maintain the Skywalking components in the Kubernetes cluster. In addition to this blog, you can also view swck document and Java agent injector documentation for more information. If you find this project useful, please give SWCK a star! If you have any questions, welcome to ask in Issues or Discussions.\n","excerpt":"\u003ch3 id=\"content\"\u003econtent:\u003c/h3\u003e\n\u003col\u003e\n\u003cli\u003e\u003ca href=\"#1.-Introduction\"\u003eIntroduction\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#2.-Features\"\u003eFeatures\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#3.-Install-SWCK\"\u003eInstall SWCK\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#4.-Deploy-a-demo-application\"\u003eDeploy a demo application\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#5.-Verify-the-injector\"\u003eVerify the injector …\u003c/a\u003e\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/blog/2022-04-19-how-to-use-the-java-agent-injector/","title":"How to use the java agent injector?"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/user-manual/","title":"User Manual"},{"body":"目录 介绍 主要特点 安装SWCK 部署demo应用 验证注入器 结束语 1. 介绍 1.1 SWCK 是什么？ SWCK是部署在 Kubernetes 环境中，为 Skywalking 用户提供服务的平台，用户可以基于该平台使用、升级和维护 SkyWalking 相关组件。\n实际上，SWCK 是基于 kubebuilder 开发的Operator，为用户提供自定义资源（ CR ）以及管理资源的控制器（ Controller ），所有的自定义资源定义（CRD）如下所示：\nJavaAgent OAP UI Storage Satellite Fetcher 1.2 java 探针注入器是什么？ 对于 java 应用来说，用户需要将 java 探针注入到应用程序中获取元数据并发送到 Skywalking 后端。为了让用户在 Kubernetes 平台上更原生地使用 java 探针，我们提供了 java 探针注入器，该注入器能够将 java 探针通过 sidecar 方式注入到应用程序所在的 pod 中。 java 探针注入器实际上是一个Kubernetes Mutation Webhook控制器，如果请求中存在 annotations ，控制器会拦截 pod 事件并将其应用于 pod 上。\n2. 主要特点 透明性。用户应用一般运行在普通容器中而 java 探针则运行在初始化容器中，且两者都属于同一个 pod 。该 pod 中的每个容器都会挂载一个共享内存卷，为 java 探针提供存储路径。在 pod 启动时，初始化容器中的 java 探针会先于应用容器运行，由注入器将其中的探针文件存放在共享内存卷中。在应用容器启动时，注入器通过设置 JVM 参数将探针文件注入到应用程序中。用户可以通过这种方式实现 java 探针的注入，而无需重新构建包含 java 探针的容器镜像。 可配置性。注入器提供两种方式配置 java 探针：全局配置和自定义配置。默认的全局配置存放在 configmap 中，用户可以根据需求修改全局配置，比如修改 backend_service 的地址。此外，用户也能通过 annotation 为特定应用设置自定义的一些配置，比如不同服务的 service_name 名称。详情可见 java探针说明书。 可观察性。每个 java 探针在被注入时，用户可以查看名为 JavaAgent 的 CRD 资源，用于观测注入后的 java 探针配置。详情可见 JavaAgent说明。 3. 安装SWCK 在接下来的几个步骤中，我们将演示如何从0开始搭建单机版的 Kubernetes 集群，并在该平台部署0.6.1版本的 SWCK。\n3.1 工具准备 首先，你需要安装一些必要的工具，如下所示：\nkind，用于创建单机版 Kubernetes集群。 kubectl，用于和Kubernetes 集群交互。 3.2 搭建单机版 Kubernetes集群 在安装完 kind 工具后，可通过如下命令创建一个单机集群。\n注意！如果你的终端配置了代理，在运行以下命令之前最好先关闭代理，防止一些意外错误的发生。\n$ kind create cluster --image=kindest/node:v1.19.1 在集群创建完毕后，可获得如下的pod信息。\n$ kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-f9fd979d6-57xpc 1/1 Running 0 7m16s kube-system coredns-f9fd979d6-8zj8h 1/1 Running 0 7m16s kube-system etcd-kind-control-plane 1/1 Running 0 7m23s kube-system kindnet-gc9gt 1/1 Running 0 7m16s kube-system kube-apiserver-kind-control-plane 1/1 Running 0 7m23s kube-system kube-controller-manager-kind-control-plane 1/1 Running 0 7m23s kube-system kube-proxy-6zbtb 1/1 Running 0 7m16s kube-system kube-scheduler-kind-control-plane 1/1 Running 0 7m23s local-path-storage local-path-provisioner-78776bfc44-jwwcs 1/1 Running 0 7m16s 3.3 安装证书管理器(cert-manger) SWCK 的证书都是由证书管理器分发和验证，需要先通过如下命令安装证书管理器cert-manger。\n$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.yaml 验证 cert-manger 是否安装成功。\n$ kubectl get pod -n cert-manager NAME READY STATUS RESTARTS AGE cert-manager-7dd5854bb4-slcmd 1/1 Running 0 73s cert-manager-cainjector-64c949654c-tfmt2 1/1 Running 0 73s cert-manager-webhook-6bdffc7c9d-h8cfv 1/1 Running 0 73s 3.4 安装SWCK java 探针注入器是 SWCK 中的一个组件，首先需要按照如下步骤安装 SWCK：\n输入如下命令获取 SWCK 的 yaml 文件并部署在 Kubernetes 集群中。 $ curl -Ls https://archive.apache.org/dist/skywalking/swck/0.6.1/skywalking-swck-0.6.1-bin.tgz | tar -zxf - -O ./config/operator-bundle.yaml | kubectl apply -f - 检查 SWCK 是否正常运行。 $ kubectl get pod -n skywalking-swck-system NAME READY STATUS RESTARTS AGE skywalking-swck-controller-manager-7f64f996fc-qh8s9 2/2 Running 0 94s 3.5 安装 Skywalking 组件 — OAPServer 和 UI 在 default 命名空间中部署 OAPServer 组件和 UI 组件。 $ kubectl apply -f https://raw.githubusercontent.com/apache/skywalking-swck/master/operator/config/samples/default.yaml 查看 OAPServer 组件部署情况。 $ kubectl get oapserver NAME INSTANCES RUNNING ADDRESS default 1 1 default-oap.default 查看 UI 组件部署情况。 $ kubectl get ui NAME INSTANCES RUNNING INTERNALADDRESS EXTERNALIPS PORTS default 1 1 default-ui.default [80] 4. 部署demo应用 在第3个步骤中，我们已经安装好 SWCK 以及相关的 Skywalking 组件，接下来按照全局配置以及自定义配置两种方式，通过两个 java 应用实例，分别演示如何使用 SWCK 中的 java 探针注入器。\n4.1 设置全局配置 当 SWCK 安装完成后，默认的全局配置就会以 configmap 的形式存储在系统命令空间中，可通过如下命令查看。\n$ kubectl get configmap skywalking-swck-java-agent-configmap -n skywalking-swck-system -oyaml apiVersion: v1 data: agent.config: |- # The service name in UI agent.service_name=${SW_AGENT_NAME:Your_ApplicationName} # Backend service addresses. collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:127.0.0.1:11800} # Please refer to https://skywalking.apache.org/docs/skywalking-java/latest/en/setup/service-agent/java-agent/configurations/#table-of-agent-configuration-properties to get more details. 在 kind 创建的 Kubernetes 集群中， SkyWalking 后端地址和 configmap 中指定的地址可能不同，我们需要使用真正的 OAPServer 组件的地址 default-oap.default 来代替默认的 127.0.0.1 ，可通过修改 configmap 实现。\n$ kubectl edit configmap skywalking-swck-java-agent-configmap -n skywalking-swck-system configmap/skywalking-swck-java-agent-configmap edited $ kubectl get configmap skywalking-swck-java-agent-configmap -n skywalking-swck-system -oyaml apiVersion: v1 data: agent.config: |- # The service name in UI agent.service_name=${SW_AGENT_NAME:Your_ApplicationName} # Backend service addresses. collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:default-oap.default:11800} # Please refer to https://skywalking.apache.org/docs/skywalking-java/latest/en/setup/service-agent/java-agent/configurations/#table-of-agent-configuration-properties to get more details. 4.2 设置自定义配置 在实际使用场景中，我们需要使用 Skywalking 组件监控不同的 java 应用，因此不同应用的探针配置可能有所不同，比如应用的名称、应用需要使用的插件等。为了支持自定义配置，注入器提供 annotation 来覆盖默认的全局配置。接下来我们将分别以基于 spring boot 以及 spring cloud gateway 开发的两个简单java应用为例进行详细说明，你可以使用这两个应用的源代码构建镜像。\n# build the springboot and springcloudgateway image $ git clone https://github.com/dashanji/swck-spring-cloud-k8s-demo $ cd swck-spring-cloud-k8s-demo \u0026amp;\u0026amp; make # check the image $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE gateway v0.0.1 51d16251c1d5 48 minutes ago 723MB app v0.0.1 62f4dbcde2ed 48 minutes ago 561MB # load the image into the cluster $ kind load docker-image app:v0.0.1 \u0026amp;\u0026amp; kind load docker-image gateway:v0.0.1 4.3 部署 spring boot 应用 创建 springboot-system 命名空间。 $ kubectl create namespace springboot-system 给 springboot-system 命名空间打上标签使能 java 探针注入器。 $ kubectl label namespace springboot-system swck-injection=enabled 接下来为 spring boot 应用对应的部署文件 springboot.yaml ，其中使用了 annotation 覆盖默认的探针配置，比如 service_name ，将其覆盖为 backend-service 。 需要注意的是，在使用 annotation 覆盖探针配置之前，需要增加 strategy.skywalking.apache.org/agent.Overlay: \u0026quot;true\u0026quot; 来使覆盖生效。\napiVersion: apps/v1 kind: Deployment metadata: name: demo-springboot namespace: springboot-system spec: selector: matchLabels: app: demo-springboot template: metadata: labels: swck-java-agent-injected: \u0026#34;true\u0026#34; # enable the java agent injector app: demo-springboot annotations: strategy.skywalking.apache.org/agent.Overlay: \u0026#34;true\u0026#34; # enable the agent overlay agent.skywalking.apache.org/agent.service_name: \u0026#34;backend-service\u0026#34; spec: containers: - name: springboot imagePullPolicy: IfNotPresent image: app:v0.0.1 command: [\u0026#34;java\u0026#34;] args: [\u0026#34;-jar\u0026#34;,\u0026#34;/app.jar\u0026#34;] --- apiVersion: v1 kind: Service metadata: name: demo namespace: springboot-system spec: type: ClusterIP ports: - name: 8085-tcp port: 8085 protocol: TCP targetPort: 8085 selector: app: demo-springboot 在 springboot-system 命名空间中部署 spring boot 应用。 $ kubectl apply -f springboot.yaml 查看部署情况。 $ kubectl get pod -n springboot-system NAME READY STATUS RESTARTS AGE demo-springboot-7c89f79885-dvk8m 1/1 Running 0 11s 通过 JavaAgent 查看最终注入的 java 探针配置。 $ kubectl get javaagent -n springboot-system NAME PODSELECTOR SERVICENAME BACKENDSERVICE app-demo-springboot-javaagent app=demo-springboot backend-service default-oap.default:11800 4.4 部署 spring cloud gateway 应用 创建 gateway-system 命名空间。 $ kubectl create namespace gateway-system 给 gateway-system 命名空间打上标签使能 java 探针注入器。 $ kubectl label namespace gateway-system swck-injection=enabled 接下来为 spring cloud gateway 应用对应的部署文件 springgateway.yaml ，其中使用了 annotation 覆盖默认的探针配置，比如 service_name ，将其覆盖为 gateway-service 。此外，在使用 spring cloud gateway 时，我们需要在探针配置中添加 spring cloud gateway 插件。 需要注意的是，在使用 annotation 覆盖探针配置之前，需要增加 strategy.skywalking.apache.org/agent.Overlay: \u0026quot;true\u0026quot; 来使覆盖生效。\napiVersion: apps/v1 kind: Deployment metadata: labels: app: demo-gateway name: demo-gateway namespace: gateway-system spec: selector: matchLabels: app: demo-gateway template: metadata: labels: swck-java-agent-injected: \u0026#34;true\u0026#34; app: demo-gateway annotations: strategy.skywalking.apache.org/agent.Overlay: \u0026#34;true\u0026#34; agent.skywalking.apache.org/agent.service_name: \u0026#34;gateway-service\u0026#34; optional.skywalking.apache.org: \u0026#34;cloud-gateway-3.x\u0026#34; # add spring cloud gateway plugin spec: containers: - image: gateway:v0.0.1 name: gateway command: [\u0026#34;java\u0026#34;] args: [\u0026#34;-jar\u0026#34;,\u0026#34;/gateway.jar\u0026#34;] --- apiVersion: v1 kind: Service metadata: name: service-gateway namespace: gateway-system spec: type: ClusterIP ports: - name: 9999-tcp port: 9999 protocol: TCP targetPort: 9999 selector: app: demo-gateway 在 gateway-system 命名空间中部署 spring cloud gateway 应用。 $ kubectl apply -f springgateway.yaml 查看部署情况。 $ kubectl get pod -n gateway-system NAME READY STATUS RESTARTS AGE demo-gateway-758899c99-6872s 1/1 Running 0 15s 通过 JavaAgent 获取最终注入的java探针配置。 $ kubectl get javaagent -n gateway-system NAME PODSELECTOR SERVICENAME BACKENDSERVICE app-demo-gateway-javaagent app=demo-gateway gateway-service default-oap.default:11800 5. 验证注入器 当完成上述步骤后，我们可以查看被注入pod的详细状态，比如被注入的agent容器。 # get all injected pod $ kubectl get pod -A -lswck-java-agent-injected=true NAMESPACE NAME READY STATUS RESTARTS AGE gateway-system demo-gateway-5bb77f6d85-lt4z7 1/1 Running 0 69s springboot-system demo-springboot-7c89f79885-lkb5j 1/1 Running 0 75s # view detailed state of the injected pod [demo-springboot] $ kubectl describe pod -l app=demo-springboot -n springboot-system ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- ... Normal Created 91s kubelet,kind-control-plane Created container inject-skywalking-agent Normal Started 91s kubelet,kind-control-plane Started container inject-skywalking-agent ... Normal Created 90s kubelet,kind-control-plane Created container springboot Normal Started 90s kubelet,kind-control-plane Started container springboot # view detailed state of the injected pod [demo-gateway] $ kubectl describe pod -l app=demo-gateway -n gateway-system ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- ... Normal Created 2m20s kubelet,kind-control-plane Created container inject-skywalking-agent Normal Started 2m20s kubelet,kind-control-plane Started container inject-skywalking-agent ... Normal Created 2m20s kubelet,kind-control-plane Created container gateway Normal Started 2m20s kubelet,kind-control-plane Started container gateway 现在我们可以将服务绑定在某个端口上并通过 web 浏览器查看采样数据。首先，我们需要通过以下命令获取gateway服务和ui服务的信息。 $ kubectl get service service-gateway -n gateway-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service-gateway ClusterIP 10.99.181.145 \u0026lt;none\u0026gt; 9999/TCP 9m19s $ kubectl get service default-ui NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default-ui ClusterIP 10.111.39.250 \u0026lt;none\u0026gt; 80/TCP 82m 接下来分别启动2个终端将service-gateway 以及 default-ui 绑定到本地端口上，如下所示： $ kubectl port-forward service/service-gateway -n gateway-system 9999:9999 Forwarding from 127.0.0.1:9999 -\u0026gt; 9999 Forwarding from [::1]:9999 -\u0026gt; 9999 $ kubectl port-forward service/default-ui 8090:80 Forwarding from 127.0.0.1:8090 -\u0026gt; 8080 Forwarding from [::1]:8090 -\u0026gt; 8080 使用以下命令通过spring cloud gateway 网关服务暴露的端口来访问 spring boot 应用服务。 $ for i in {1..10}; do curl http://127.0.0.1:9999/gateway/hello \u0026amp;\u0026amp; echo \u0026#34;\u0026#34;; done Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! 我们可以在 web 浏览器中输入 http://127.0.0.1:8090 来访问探针采集到的数据。 所有服务的拓扑图如下所示。 查看 gateway-service 网关服务的 trace 信息。 查看 backend-service 应用服务的 trace 信息。 6. 结束语 如果你的应用部署在 Kubernetes 平台中，且需要 Skywalking 提供监控服务， SWCK 能够帮助你部署、升级和维护 Kubernetes 集群中的 Skywalking 组件。除了本篇博客外，你还可以查看 SWCK文档 以及 java探针注入器文档 获取更多的信息。如果你觉得这个项目好用，请给 SWCK 一个star! 如果你有任何疑问，欢迎在Issues或者Discussions中提出。\n","excerpt":"\u003ch3 id=\"目录\"\u003e目录\u003c/h3\u003e\n\u003col\u003e\n\u003cli\u003e\u003ca href=\"#1.-%E4%BB%8B%E7%BB%8D\"\u003e介绍\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#2.-%E4%B8%BB%E8%A6%81%E7%89%B9%E7%82%B9\"\u003e主要特点\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#3.-%E5%AE%89%E8%A3%85SWCK\"\u003e安装SWCK\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#4.-%E9%83%A8%E7%BD%B2demo%E5%BA%94%E7%94%A8\"\u003e部署demo应用\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#5.-%E9%AA%8C%E8%AF%81%E6%B3%A8%E5%85%A5%E5%99%A8\"\u003e验证注入器\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#6.-%E7%BB%93%E6%9D%9F%E8%AF%AD\"\u003e结束语\u003c/a\u003e\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch2 id=\"1-介绍\"\u003e1. 介绍\u003c/h2\u003e\n\u003ch3 id=\"11-swck-是什么\"\u003e1.1 SWCK 是什么？\u003c/h3\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/apache/skywalking-swck\"\u003eSWCK\u003c/a\u003e是部署在 Kubernetes 环境中，为 Skywalking 用户提供 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2022-04-19-how-to-use-the-java-agent-injector/","title":"如何使用java探针注入器?"},{"body":"\nApache SkyWalking 是中国首个，也是目前唯一的个人开源的 Apache 顶级项目。\n作为一个针对分布式系统的应用性能监控 APM 和可观测性分析平台， SkyWalking 提供了媲美商业APM/监控的功能。\nCSDN云原生系列在线峰会第4期，特邀SkyWalking创始人、Apache基金会首位中国董事、Tetrate创始工程师吴晟担任出品人，推出SkyWalking峰会。\nSkyWalking峰会在解读SkyWalking v9新特性的同时，还将首发解密APM的专用数据库BanyanDB，以及分享SkyWalking在原生eBPF探针、监控虚拟机和Kubernetes、云原生函数计算可观测性等方面的应用实践。\n峰会议程：\n14:00-14:30 开场演讲：SkyWalking v9解析 吴晟 Tetrate 创始工程师、Apache 基金会首位中国董事\n14:30-15:00 首发解密：APM的专用数据库BanyanDB\n高洪涛 Tetrate 创始工程师\n15:00-15:30 SkyWalking 原生eBPF探针展示\n刘晗 Tetrate 工程师\n15:30-16:00 Apache SkyWalking MAL实践-监控虚拟机和Kubernetes\n万凯 Tetrate 工程师\n16:00-16:30 SkyWalking助力云原生函数计算可观测\n霍秉杰 青云科技 资深架构师\n峰会视频 B站视频地址\n","excerpt":"\u003cp\u003e\u003cimg src=\"https://img-blog.csdnimg.cn/9dfd03251da64946becdf67f6a10ead7.png\" alt=\"\"\u003e\u003c/p\u003e\n\u003cp\u003eApache SkyWalking 是中国首个，也是目前唯一的个人开源的 Apache 顶级项目。\u003c/p\u003e\n\u003cp\u003e作为一个针对分布式系统的应用性能监控 APM 和可观测性分析平台， SkyWalking 提供了媲 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2022-04-18-meeting/","title":"Apache SkyWalking 2022 峰会"},{"body":"SkyWalking Java Agent 8.10.0 is released. Go to downloads page to find release tars. Changes by Version\n8.10.0 [Important] Namespace represents a subnet, such as kubernetes namespace, or 172.10... Make namespace concept as a part of service naming format. [Important] Add cluster concept, also as a part of service naming format. The cluster name would be Add as {@link #SERVICE_NAME} suffix. Add as exit span\u0026rsquo;s peer, ${CLUSTER} / original peer Cross Process Propagation Header\u0026rsquo;s value addressUsedAtClient[index=8] (Target address of this request used on the client end). Support Undertow thread pool metrics collecting. Support Tomcat thread pool metric collect. Remove plugin for ServiceComb Java Chassis 0.x Add Guava EventBus plugin. Fix Dubbo 3.x plugin\u0026rsquo;s tracing problem. Fix the bug that maybe generate multiple trace when invoke http request by spring webflux webclient. Support Druid Connection pool metrics collecting. Support HikariCP Connection pool metrics collecting. Support Dbcp2 Connection pool metrics collecting. Ignore the synthetic constructor created by the agent in the Spring patch plugin. Add witness class for vertx-core-3.x plugin. Add witness class for graphql plugin. Add vertx-core-4.x plugin. Renamed graphql-12.x-plugin to graphql-12.x-15.x-plugin and graphql-12.x-scenario to graphql-12.x-15.x-scenario. Add graphql-16plus plugin. [Test] Support to configure plugin test base images. [Breaking Change] Remove deprecated agent.instance_properties configuration. Recommend agent.instance_properties_json. The namespace and cluster would be reported as instance properties, keys are namespace and cluster. Notice, if instance_properties_json includes these two keys, they would be overrided by the agent core. [Breaking Change] Remove the namespace from cross process propagation key. Make sure the parent endpoint in tracing context from existing first ENTRY span, rather than first span only. Fix the bug that maybe causing memory leak and repeated traceId when use gateway-2.1.x-plugin or gateway-3.x-plugin. Fix Grpc 1.x plugin could leak context due to gRPC cancelled. Add JDK ThreadPoolExecutor Plugin. Support default database(not set through JDBC URL) in mysql-5.x plugin. Documentation Add link about java agent injector. Update configurations doc, remove agent.instance_properties[key]=value. Update configurations doc, add agent.cluster and update agent.namespace. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking Java Agent 8.10.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nChanges by …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-java-agent-8-10-0/","title":"Release Apache SkyWalking Java Agent 8.10.0"},{"body":"Introduction The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it. - Mark Weiser\nMark Weiser prophetically argued in the late 1980s, that the most far-reaching technologies are those which vanish into thin air. According to Weiser, \u0026ldquo;Whenever people learn something sufficiently well, they cease to be aware of it.\u0026rdquo; This disappearing act, as Weiser claimed, is not limited to technology but rather human psychology. It is this very experience that allows us to escape lower-level thinking into higher-level thinking. For once we are no longer impeded by mundane details, we are then free to focus on new goals.\nThis realization becomes more relevant as APMs become increasingly popular. As more applications are deployed with APMs, the number of abstract representations of the underlying source code also increases. While this provides great value to many non-development roles within an organization, it does pose additional challenges to those in development roles who must translate these representations into concepts they can work with (i.e. source code). Weiser sums this difficultly up rather succinctly when he states that \u0026ldquo;Programmers should no more be asked to work without access to source code than auto-mechanics should be asked to work without looking at the engine.\u0026rdquo;\nStill, APMs collect more information only to produce a plethora of new abstract representations. In this article, we will introduce a new concept in Source++, the open-source live-coding platform, specifically designed to allow developers to monitor production applications more intuitively.\nLive Views And we really don\u0026rsquo;t understand even yet, hundreds of metrics later, what make a program easier to understand or modify or reuse or borrow. I don\u0026rsquo;t think we\u0026rsquo;ll find out by looking away from programs to their abstract interfaces. The answers are in the source code. - Mark Weiser\nAs APMs move from the \u0026ldquo;nice to have\u0026rdquo; category to the \u0026ldquo;must-have\u0026rdquo; category, there is a fundamental feature holding them back from ubiquity. They must disappear from consciousness. As developers, we should feel no impulse to open our browsers to better understand the underlying source code. The answers are literally in the source code. Instead, we should improve our tools so the source code conveniently tells us what we need to know. Think of how simple life could be if failing code always indicated how and why it failed. This is the idea behind Source++.\nIn our last blog post, we discussed Extending Apache SkyWalking with non-breaking breakpoints. In that post, we introduced a concept called Live Instruments, which developers can use to easily debug live production applications without leaving their IDE. Today, we will discuss how existing SkyWalking installations can be integrated into your IDE via a new concept called Live Views. Unlike Live Instruments, which are designed for debugging live applications, Live Views are designed for increasing application comprehension and awareness. This is accomplished through a variety of commands which are input into the Live Command Palette.\nLive Command Palette The Live Command Palette (LCP) is a contextual command prompt, included in the Source++ JetBrains Plugin, that allows developers to control and query live applications from their IDE. Opened via keyboard shortcut (Ctrl+Shift+S), the LCP allows developers to easily view metrics relevant to the source code they\u0026rsquo;re currently viewing. The following Live View commands are currently supported:\nCommand: view (overview/activity/traces/logs) The view commands display contextual popups with live operational data of the current source code. These commands allow developers to view traditional SkyWalking operational data filtered down to the relevant metrics.\nCommand: watch log The watch log command allows developers to follow individual log statements of a running application in real-time. This command allows developers to negate the need for manually scrolling through the logs to find instances of a specific log statement.\nCommand: (show/hide) quick stats The show quick stats command displays live endpoint metrics for a quick idea of an endpoint\u0026rsquo;s activity. Using this command, developers can quickly assess the status of an endpoint and determine if the endpoint is performing as expected.\nFuture Work A good tool is an invisible tool. By invisible, I mean that the tool does not intrude on your consciousness; you focus on the task, not the tool. Eyeglasses are a good tool \u0026ndash; you look at the world, not the eyeglasses. - Mark Weiser\nSource++ aims to extend SkyWalking in such a way that SkyWalking itself becomes invisible. To accomplish this, we plan to support custom developer commands. Developers will be able to build customized commands for themselves, as well as commands to share with their team. These commands will recognize context, types, and conditions allowing for a wide possibility of operations. As more commands are added, developers will be able to expose everything SkyWalking has to offer while focusing on what matters most, the source code.\nIf you find these features useful, please consider giving Source++ a try. You can install the plugin directly from your JetBrains IDE, or through the JetBrains Marketplace. If you have any issues or questions, please open an issue. Feedback is always welcome!\n","excerpt":"\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cblockquote\u003e\n\u003cp\u003eThe most profound technologies are those that disappear. They weave themselves into …\u003c/p\u003e\u003c/blockquote\u003e","ref":"https://skywalking.apache.org/blog/2022-04-14-integrating-skywalking-with-source-code/","title":"Integrating Apache SkyWalking with source code"},{"body":"Read this post in original language: English\n介绍 最具影响力的技术是那些消失的技术。他们交织在日常生活中，直到二者完全相融。 - 马克韦瑟\n马克韦瑟在 1980 年代后期预言，影响最深远的技术是那些消失在空气中的技术。\n“当人们足够熟知它，就不会再意识到它。”\n正如韦瑟所说，这种消失的现象不只源于技术，更是人类的心理。 正是这种经验使我们能够摆脱对底层的考量，进入更高层次的思考。 一旦我们不再被平凡的细枝末节所阻碍，我们就可以自如地专注于新的目标。\n随着 APM(应用性能管理系统) 变得越来越普遍，这种认识变得更加重要。随着更多的应用程序开始使用 APM 部署，底层源代码抽象表示的数量也在同步增加。 虽然这为组织内的许多非开发角色提供了巨大的价值，但它确实也对开发人员提出了额外的挑战 - 他们必须将这些表示转化为可操作的概念（即源代码）。 对此，韦瑟相当简洁的总结道,“就像不应要求汽车机械师在不查看引擎的情况下工作一样，我们不应要求程序员在不访问源代码的情况下工作”。\n尽管如此，APM 收集更多信息只是为了产生充足的新抽象表示。 在本文中，我们将介绍开源实时编码平台 Source++ 中的一个新概念，旨在让开发人员更直观地监控生产应用程序。\n实时查看 我们尚且不理解在收集了数百个指标之后，是什么让程序更容易理解、修改、重复使用或借用。 我不认为我们能够通过原理程序本身而到它们的抽象接口中找到答案。答案就在源代码之中。 - 马克韦瑟\n随着 APM 从“有了更好”转变为“必须拥有”，有一个基本特性阻碍了它们的普及。 它们必须从意识中消失。作为开发人员，我们不应急于打开浏览器以更好地理解底层源代码，答案就在源代码中。 相反，我们应该改进我们的工具，以便源代码直观地告诉我们需要了解的内容。 想想如果失败的代码总是表明它是如何以及为什么失败的，生活会多么简单。这就是 Source++ 背后的理念。\n在我们的上一篇博客中，我们讨论了不间断断点 Extending Apache SkyWalking。 我们介绍了一个名为 Live Instruments(实时埋点) 的概念，开发人员可以使用它轻松调试实时生产应用程序，而无需离开他们的开发环境。 而今天，我们将讨论如何通过一个名为 Live Views（实时查看）的新概念将现有部署的 SkyWalking 集成到您的 IDE 中。 与专为调试实时应用程序而设计的 Live Instruments (实时埋点) 不同，Live Views（实时查看）旨在提高对应用程序的理解和领悟。 这将通过输入到 Live Command Palette (实时命令面板) 中的各种命令来完成。\n实时命令面板 Live Command Palette (LCP) 是一个当前上下文场景下的命令行面板，这个组件包含在 Source++ JetBrains 插件中，它允许开发人员从 IDE 中直接控制和对实时应用程序发起查询。\nLCP 通过键盘快捷键 (Ctrl+Shift+S) 打开，允许开发人员轻松了解与他们当前正在查看的源代码相关的运行指标。\n目前 LCP 支持以下实时查看命令：\n命令：view（overview/activity/traces/Logs）- 查看 总览/活动/追踪/日志 view 查看命令会展示一个与当前源码的实时运维数据关联的弹窗。 这些命令允许开发人员查看根据相关指标过滤的传统 SkyWalking 的运维数据。\n命令：watch log - 实时监听日志 本日志命令允许开发人员实时跟踪正在运行的应用程序的每一条日志。 通过此命令开发人员无需手动查阅大量日志就可以查找特定日志语句的实例。\n命令：(show/hide) quick stats （显示/隐藏）快速统计 show quick stats 显示快速统计命令显示实时端点指标，以便快速了解端点的活动。 使用此命令，开发人员可以快速评估端点的状态并确定端点是否按预期正常运行。\n未来的工作 好工具是无形的。我所指的无形，是指这个工具不会侵入你的意识； 你专注于任务，而不是工具。 眼镜就是很好的工具——你看的是世界，而不是眼镜。 - 马克韦瑟\nSource++ 旨在扩展 SkyWalking，使 SkyWalking 本身变得无需感知。 为此，我们计划支持自定义的开发人员命令。 开发人员将能够构建自定义命令，以及与团队共享的命令。 这些命令将识别上下文、类型和条件，从而允许广泛的操作。 随着更多命令的添加，开发人员将能够洞悉 SkyWalking 所提供的所有功能，同时专注于最重要的源码。\n如果您觉得这些功能有用，请考虑尝试使用 Source++。 您可以通过 JetBrains Marketplace 或直接从您的 JetBrains IDE 安装插件。 如果您有任何疑问，请到这提 issue。\n欢迎随时反馈！\n","excerpt":"\u003cp\u003e\u003csub\u003eRead this post in original language: \u003ca href=\"https://skywalking.apache.org/blog/2022-04-14-integrating-skywalking-with-source-code/\"\u003eEnglish\u003c/a\u003e\u003c/sub\u003e\u003c/p\u003e\n\u003ch2 id=\"介绍\"\u003e介绍\u003c/h2\u003e\n\u003cblockquote\u003e\n\u003cp\u003e最具影响力的技术是那些消失的技术。他们交织在日常生活中，直到二者完全相融。 - 马克韦瑟\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003e马克韦瑟在 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2022-04-14-integrating-skywalking-with-source-code/","title":"将 Apache SkyWalking 与源代码集成"},{"body":"随着无人驾驶在行业的不断发展和技术的持续革新，规范化、常态化的真无人运营逐渐成为事实标准，而要保障各个场景下的真无人业务运作，一个迫切需要解决的现状就是业务链路长，出现问题难以定位。本文由此前于 KubeSphere 直播上的分享整理而成，主要介绍 SkyWalking 的基本概念和使用方法，以及在无人驾驶领域的一系列实践。\nB站视频地址\n行业背景 驭势科技（UISEE）是国内领先的无人驾驶公司。致力于为全行业、全场景提供 AI 驾驶服务,做赋能出行和物流新生态的 AI 驾驶员。早在三年前， 驭势科技已在机场和厂区领域实现了“去安全员” 无人驾驶常态化运营的重大突破，落地“全场景、真无人、全天候”的自动驾驶技术，并由此迈向大规模商用。要保证各个场景下没有安全员参与的业务运作，我们在链路追踪上做了一系列实践。\n对于无人驾驶来说，从云端到车端的链路长且复杂，任何一层出问题都会导致严重的后果；然而在如下图所示的链路中，准确迅速地定位故障服务并不容易，经常遇到多个服务层层排查的情况。我们希望做到的事情，就是在出现问题以后，能够尽快定位到源头，从而快速解决问题，以绝后患。\n前提条件 SkyWalking 简介 Apache SkyWalking 是一个开源的可观察性平台，用于收集、分析、聚集和可视化来自服务和云原生基础设施的数据。SkyWalking 通过简单的方法，提拱了分布式系统的清晰视图，甚至跨云。它是一个现代的 APM（Application Performence Management），专门为云原生、基于容器的分布式系统设计。它在逻辑上被分成四个部分。探针、平台后端、存储和用户界面。\n探针收集数据并根据 SkyWalking 的要求重新格式化（不同的探针支持不同的来源）。 平台后端支持数据聚合、分析以及从探针接收数据流的过程，包括 Tracing、Logging、Metrics。 存储系统通过一个开放/可插拔接口容纳 SkyWalking 数据。用户可以选择一个现有的实现，如 ElasticSearch、H2、MySQL、TiDB、InfluxDB，或实现自定义的存储。 UI是一个高度可定制的基于网络的界面，允许 SkyWalking 终端用户可视化和管理 SkyWalking 数据。 综合考虑了对各语言、各框架的支持性、可观测性的全面性以及社区环境等因素，我们选择了 SkyWalking 进行链路追踪。\n链路追踪简介 关于链路追踪的基本概念，可以参看吴晟老师翻译的 OpenTracing 概念和术语 以及 OpenTelemetry。在这里，择取几个重要的概念供大家参考：\nTrace：代表一个潜在的分布式的存在并行数据或者并行执行轨迹的系统。一个 Trace 可以认为是多个 Span 的有向无环图（DAG）。简单来说，在微服务体系下，一个 Trace 代表从第一个服务到最后一个服务经历的一系列的服务的调用链。 Span：在服务中埋点时，最需要关注的内容。一个 Span 代表系统中具有开始时间和执行时长的逻辑运行单元。举例来说，在一个服务发出请求时，可以认为是一个 Span 的开始；在这个服务接收到上游服务的返回值时，可以认为是这个 Span 的结束。Span 之间通过嵌套或者顺序排列建立逻辑因果关系。在 SkyWalking 中，Span 被区分为： LocalSpan：服务内部调用方法时创建的 Span 类型 EntrySpan：请求进入服务时会创建的 Span 类型（例如处理其他服务对于本服务接口的调用） ExitSpan：请求离开服务时会创建的 Span 类型（例如调用其他服务的接口） TraceSegment：SkyWalking 中的概念，介于 Trace 和 Span 之间，是一条 Trace 的一段，可以包含多个 Span。一个 TraceSegment 记录了一个线程中的执行过程，一个 Trace 由一个或多个 TraceSegment 组成，一个 TraceSegment 又由一个或多个 Span 组成。 SpanContext：代表跨越进程上下文，传递到下级 Span 的状态。一般包含 Trace ID、Span ID 等信息。 Baggage：存储在 SpanContext 中的一个键值对集合。它会在一条追踪链路上的所有 Span 内全局传输，包含这些 Span 对应的 SpanContext。Baggage 会随着 Trace 一同传播。 SkyWalking 中，上下文数据通过名为 sw8 的头部项进行传递，值中包含 8 个字段，由 - 进行分割（包括 Trace ID，Parent Span ID 等等） 另外 SkyWalking 中还提供名为 sw8-correlation 的扩展头部项，可以传递一些自定义的信息 快速上手 以 Go 为例，介绍如何使用 SkyWalking 在服务中埋点。\n部署 我们选择使用 Helm Chart 在 Kubernetes 中进行部署。\nexport SKYWALKING_RELEASE_NAME=skywalking # change the release name according to your scenario export SKYWALKING_RELEASE_NAMESPACE=default # change the namespace to where you want to install SkyWalking export REPO=skywalking helm repo add ${REPO} https://apache.jfrog.io/artifactory/skywalking-helm helm install \u0026#34;${SKYWALKING_RELEASE_NAME}\u0026#34; ${REPO}/skywalking -n \u0026#34;${SKYWALKING_RELEASE_NAMESPACE}\u0026#34; \\ --set oap.image.tag=8.8.1 \\ --set oap.storageType=elasticsearch \\ --set ui.image.tag=8.8.1 \\ --set elasticsearch.imageTag=6.8.6 埋点 部署完以后，需要在服务中进行埋点，以生成 Span 数据：主要的方式即在服务的入口和出口创建 Span。在代码中，首先我们会创建一个 Reporter，用于向 SkyWalking 后端发送数据。接下来，我们需要创建一个名为 \u0026quot;example\u0026quot; 的 Tracer 实例。此时，我们就可以使用 Tracer 实例来创建 Span。 在 Go 中，主要利用 context.Context 来创建以及传递 Span。\nimport \u0026#34;github.com/SkyAPM/go2sky\u0026#34; // configure to export to OAP server r, err := reporter.NewGRPCReporter(\u0026#34;oap-skywalking:11800\u0026#34;) if err != nil { log.Fatalf(\u0026#34;new reporter error %v \\n\u0026#34;, err) } defer r.Close() tracer, err := go2sky.NewTracer(\u0026#34;example\u0026#34;, go2sky.WithReporter(r)) 服务内部 在下面的代码片段中，通过 context.background() 生成的 Context 创建了一个 Root Span，同时在创建该 Span 的时候，也会产生一个跟这 个 Span 相关联的 Context。利用这个新的 Context，就可以创建一个与 Root Span 相关联的 Child Span。\n// create root span span, ctx, err := tracer.CreateLocalSpan(context.Background()) // create sub span w/ context above subSpan, newCtx, err := tracer.CreateLocalSpan(ctx) 服务间通信 在服务内部，我们会利用 Context 传的递来进行 Span 的创建。但是如果是服务间通信的话，这也是链路追踪最为广泛的应用场景，肯定是没有办法直接传递 Context 参数的。这种情况下，应该怎么做呢？一般来说，SkyWalking 会把 Context 中与当前 Span 相关的键值对进行编码，后续在服务通信时进行传递。例如，在 HTTP 协议中，一般利用请求头进行链路传递。再例如 gRPC 协议，一般想到的就是利用 Metadata 进行传递。\n在服务间通信的时候，我们会利用 EntrySpan 和 ExitSpan 进行链路的串联。以 HTTP 请求为例，在创建 EntrySpan 时，会从请求头中获取到 Span 上下文信息。而在 ExitSpan 中，则在请求中注入了上下文。这里的上下文是经过了 SkyWalking 编码后的字符串，以便在服务间进行传递。除了传递 Span 信息，也可以给 Span 打上 Tag 进行标记。例如，记录 HTTP 请求的方法，URL 等等，以便于后续数据的可视化。\n//Extract context from HTTP request header `sw8` span, ctx, err := tracer.CreateEntrySpan(r.Context(), \u0026#34;/api/login\u0026#34;, func(key string) (string, error) { return r.Header.Get(key), nil }) // Some operation ... // Inject context into HTTP request header `sw8` span, err := tracer.CreateExitSpan(req.Context(), \u0026#34;/service/validate\u0026#34;, \u0026#34;tomcat-service:8080\u0026#34;, func(key, value string) error { req.Header.Set(key, value) return nil }) // tags span.Tag(go2sky.TagHTTPMethod, req.Method) span.Tag(go2sky.TagURL, req.URL.String()) 但是，我们可能也会用到一些不那么常用的协议，比如说 MQTT 协议。在这些情况下，应该如何传递上下文呢？关于这个问题，我们在自定义插件的部分做了实践。\nUI 经过刚才的埋点以后，就可以在 SkyWalking 的 UI 界面看到调用链。SkyWalking 官方提供了一个 Demo 页面，有兴趣可以一探究竟：\nUI http://demo.skywalking.apache.org\nUsername skywalking Password skywalking\n插件体系 如上述埋点的方式，其实是比较麻烦的。好在 SkyWalking 官方提供了很多插件，一般情况下，直接接入插件便能达到埋点效果。SkyWalking 官方为多种语言都是提供了丰富的插件，对一些主流框架都有插件支持。由于我们部门使用的主要是 Go 和 Python 插件，下文中便主要介绍这两种语言的插件。同时，由于我们的链路复杂，用到的协议较多，不可避免的是也需要开发一些自定义插件。下图中整理了 Go 与 Python 插件的主要思想，以及我们开发的各框架协议自定义插件的研发思路。\n官方插件 Go · Gin 插件 Gin 是 Go 的 Web 框架，利用其中间件，可以进行链路追踪。由于是接收请求，所以需要在中间件中，创建一个 EntrySpan，同时从请求头中获取 Span 的上下文的信息。获取到上下文信息以后，还需要再进行一步操作：把当前请求请求的上下文 c.Request.Context(), 设置成为刚才创建完 EntrySpan 时生成的 Context。这样一来，这个请求的 Context 就会携带有 Span 上下文信息，可以用于在后续的请求处理中进行后续传递。\nfunc Middleware(engine *gin.Engine, tracer *go2sky.Tracer) gin.HandlerFunc { return func(c *gin.Context) { span, ctx, err := tracer.CreateEntrySpan(c.Request.Context(), getOperationName(c), func(key string) (string, error) { return c.Request.Header.Get(key), nil }) // some operation c.Request = c.Request.WithContext(ctx) c.Next() span.End() } } Python · requests Requests 插件会直接修改 Requests 库中的request函数，把它替换成 SkyWalking 自定义的_sw_request函数。在这个函数中，创建了 ExitSpan，并将 ExitSpan 上下文注入到请求头中。在服务安装该插件后，实际调用 Requests 库进行请求的时候，就会携带带有上下文的请求体进行请求。\ndef install(): from requests import Session _request = Session.request def _sw_request(this: Session, method, url, other params...): span = get_context().new_exit_span(op=url_param.path or \u0026#39;/\u0026#39;, peer=url_param.netloc, component=Component.Requests) with span: carrier = span.inject() span.layer = Layer.Http if headers is None: headers = {} for item in carrier: headers[item.key] = item.val span.tag(TagHttpMethod(method.upper())) span.tag(TagHttpURL(url_param.geturl())) res = _request(this, method, url, , other params...n) # some operation return res Session.request = _sw_request 自定义插件 Go · Gorm Gorm 框架是 Go 的 ORM 框架。我们自己在开发的时候经常用到这个框架，因此希望能对通过 Gorm 调用数据库的链路进行追踪。\nGorm 有自己的插件体系，会在数据库的操作前调用BeforeCallback函数，数据库的操作后调用AfterCallback函数。于是在BeforeCallback中，我们创建 ExitSpan，并在AfterCallback里结束先前在BeforeCallback中创建的 ExitSpan。\nfunc (s *SkyWalking) BeforeCallback(operation string) func(db *gorm.DB) { // some operation return func(db *gorm.DB) { tableName := db.Statement.Table operation := fmt.Sprintf(\u0026#34;%s/%s\u0026#34;, tableName, operation) span, err := tracer.CreateExitSpan(db.Statement.Context, operation, peer, func(key, value string) error { return nil }) // set span from db instance\u0026#39;s context to pass span db.Set(spanKey, span) } } 需要注意的是，因为 Gorm 的插件分为 Before 与 After 两个 Callback，所以需要在两个回调函数间传递 Span，这样我们才可以在AfterCallback中结束当前的 Span。\nfunc (s *SkyWalking) AfterCallback() func(db *gorm.DB) { // some operation return func(db *gorm.DB) { // get span from db instance\u0026#39;s context spanInterface, _ := db.Get(spanKey) span, ok := spanInterface.(go2sky.Span) if !ok { return } defer span.End() // some operation } } Python · MQTT 在 IoT 领域，MQTT 是非常常用的协议，无人驾驶领域自然也相当依赖这个协议。\n以 Publish 为例，根据官方插件的示例，我们直接修改 paho.mqtt 库中的publish函数，改为自己定义的_sw_publish函数。在自定义函数中，创建 ExitSpan，并将上下文注入到 MQTT 的 Payload 中。\ndef install(): from paho.mqtt.client import Client _publish = Client.publish Client.publish = _sw_publish_func(_publish) def _sw_publish_func(_publish): def _sw_publish(this, topic, payload=None, qos=0, retain=False, properties=None): # some operation with get_context().new_exit_span(op=\u0026#34;EMQX/Topic/\u0026#34; + topic + \u0026#34;/Producer\u0026#34; or \u0026#34;/\u0026#34;, peer=peer) as span: carrier = span.inject() span.layer = Layer.MQ span.component = Component.RabbitmqProducer payload = {} if payload is None else json.loads(payload) payload[\u0026#39;headers\u0026#39;] = {} for item in carrier: payload[\u0026#39;headers\u0026#39;][item.key] = item.val # ... return _sw_publish 可能这个方式不是特别优雅：因为我们目前使用 MQTT 3.1 版本，此时尚未引入 Properties 属性（类似于请求头）。直到 MQTT 5.0，才对此有相关支持。我们希望在升级到 MQTT 5.0 以后，能够将上下文注入到 Properties 中进行传递。\n无人驾驶领域的实践 虽然这些插件基本上涵盖了所有的场景，但是链路追踪并不是只要接入插件就万事大吉。在一些复杂场景下，尤其无人驾驶领域的链路追踪，由于微服务架构中涉及的语言环境、中间件种类以及业务诉求通常都比较丰富，导致在接入全链路追踪的过程中，难免遇到各种主观和客观的坑。下面选取了几个典型例子和大家分享。\n【问题一】Kong 网关的插件链路接入 我们的请求在进入服务之前，都会通过 API 网关 Kong，同时我们在 Kong 中定义了一个自定义权限插件，这个插件会调用权限服务接口进行授权。如果只是单独单纯地接入 SkyWalking Kong 插件，对于权限服务的调用无法在调用链中体现。所以我们的解决思路是，直接地在权限插件里进行埋点，而不是使用官方的插件，这样就可以把对于权限服务的调用也纳入到调用链中。\n【问题二】 Context 传递 我们有这样一个场景：一个服务，使用 Gin Web 框架，同时在处理 HTTP 请求时调用上游服务的 gRPC 接口。起初以为只要接入 Gin 的插件以及 gRPC 的插件，这个场景的链路就会轻松地接上。但是结果并不如预期。\n最后发现，Gin 提供一个 Contextc；同时对于某一个请求，可以通过c.Request.Context()获取到请求的 ContextreqCtx，二者不一致；接入 SkyWalking 提供的 Gin 插件后，修改的是reqCtx，使其包含 Span 上下文信息；而现有服务，在 gRPC 调用时传入的 Context 是c，所以一开始 HTTP -\u0026gt; gRPC 无法连接。最后通过一个工具函数，复制了reqCtx的键值对到c后，解决了这个问题。\n【问题三】官方 Python·Redis 插件 Pub/Sub 断路 由于官方提供了 Python ·Redis 插件，所以一开始认为，安装了 Redis 插件，对于一切 Redis 操作，都能互相连接。但是实际上，对于 Pub/Sub 操作，链路会断开。\n查看代码后发现，对于所有的 Redis 操作，插件都创建一个 ExitSpan；也就是说该插件其实仅适用于 Redis 作缓存等情况；但是在我们的场景中，需要进行 Pub/Sub 操作。这导致两个操作都会创建 ExitSpan，而使链路无法相连。通过改造插件，在 Pub 时创建 ExitSpan，在 Sub 时创建 EntrySpan 后，解决该问题。\n【问题四】MQTT Broker 的多种 DataBridge 接入 一般来说，对 MQTT 的追踪链路是 Publisher -\u0026gt; Subscriber，但是在我们的使用场景中，存在 MQTT broker 接收到消息后，通过规则引擎调用其他服务接口这种特殊场景。这便不是 Publisher -\u0026gt; Subscriber，而是 Publisher -\u0026gt; HTTP。\n我们希望能够从 MQTT Payload 中取出 Span 上下文，再注入到 HTTP 的请求头中。然而规则引擎调用接口时，没有办法自定义请求头，所以我们最后的做法是，约定好参数名称，将上下文放到请求体中，在服务收到请求后，从请求体中提取 Context。\n【问题五】Tracing 与 Logging 如何结合 很多时候，只有 Tracing 信息，对于问题排查来说可能还是不充分的，我们非常的期望也能够把 Tracing 和 Logging 进行结合。\n如上图所示，我们会把所有服务的 Tracing 的信息发送到 SkyWalking，同时也会把这个服务产生的日志通过 Fluent Bit 以及 Fluentd 发送到 ElasticSearch。对于这种情况，我们只需要在日志中去记录 Span 的上下文，比如记录 Trace ID 或者 Span ID 等，就可以在 Kibana 里面去进行对于 Trace ID 的搜索，来快速的查看同一次调用链中的日志。\n当然，SkyWalking 它本身也提供了自己的日志收集和分析机制，可以利用 Fluentd 或者 Fluent Bit 等向 SkyWalking 后端发送日志（我们选用了 Fluentd）。当然，像 SkyWalking 后端发送日志的时候，也要符合其日志协议，即可在 UI 上查看相应日志。\n本文介绍了 SkyWalking 的使用方法、插件体系以及实践踩坑等，希望对大家有所帮助。总结一下，SkyWalking 的使用的确是有迹可循的，一般来说我们只要接入插件，基本上可以涵盖大部分的场景，达到链路追踪的目的。但是也要注意，很多时候需要具体问题具体分析，尤其是在链路复杂的情况下，很多地方还是需要根据不同场景来进行一些特殊处理。\n最后，我们正在使用的 FaaS 平台 OpenFunction 近期也接入了 SkyWalking 作为其 链路追踪的解决方案：\nOpenFunction 提供了插件体系，并预先定义了 SkyWalking pre/post 插件；编写函数时，用户无需手动埋点，只需在 OpenFunction 配置文件中简单配置，即可开启 SkyWalking 插件，达到链路追踪的目的。\n在感叹 OpenFunction 动作迅速的同时，也能够看到 SkyWalking 已成为链路追踪领域的首要选择之一。\n参考资料 OpenTracing 文档：https://wu-sheng.gitbooks.io/opentracing-io/content/pages/spec.html SkyWalking 文档：https://skywalking.apache.org/docs/main/latest/readme/ SkyWalking GitHub：https://github.com/apache/skywalking SkyWalking go2sky GitHub：https://github.com/SkyAPM/go2sky SkyWalking Python GitHub：https://github.com/apache/skywalking-python SkyWalking Helm Chart：https://github.com/apache/skywalking-kubernetes SkyWalking Solution for OpenFunction https://openfunction.dev/docs/best-practices/skywalking-solution-for-openfunction/ ","excerpt":"\u003cp\u003e随着无人驾驶在行业的不断发展和技术的持续革新，规范化、常态化的真无人运营逐渐成为事实标准，而要保障各个场景下的真无人业务运作，一个迫切需要解决的现状就是业务链路长，出现问题难以定位。本文由此前于 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2022-04-13-skywalking-in-autonomous-driving/","title":"SkyWalking 在无人驾驶领域的实践"},{"body":"SkyWalking Client JS 0.8.0 is released. Go to downloads page to find release tars.\nFix fmp metric. Add e2e tese based on skywaling-infra-e2e. Update metric and events. Remove ServiceTag by following SkyWalking v9 new layer model. ","excerpt":"\u003cp\u003eSkyWalking Client JS 0.8.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eFix fmp metric. …\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-client-js-0-8-0/","title":"Release Apache SkyWalking Client JS 0.8.0"},{"body":"SkyWalking 9.0.0 is released. Go to downloads page to find release tars.\nSkyWalking v9 is the next main stream of the OAP and UI.\nStarting from v9, SkyWalking introduces the new core concept Layer. A layer represents an abstract framework in computer science, such as Operating System(OS_LINUX layer), Kubernetes(k8s layer). All detected instances belong to a layer to represent the running environment of this instance, the service would have one or multiple layer definitions according to its instances.\nRocketBot UI has officially been replaced by the Booster UI.\nChanges by Version Project Upgrade log4j2 to 2.17.1 for CVE-2021-44228, CVE-2021-45046, CVE-2021-45105 and CVE-2021-44832. This CVE only effects on JDK if JNDI is opened in default. Notice, using JVM option -Dlog4j2.formatMsgNoLookups=true or setting the LOG4J_FORMAT_MSG_NO_LOOKUPS=”true” environment variable also avoids CVEs. Upgrade maven-wrapper to 3.1.0, maven to 3.8.4 for performance improvements and ARM more native support. Exclude unnecessary libs when building under JDK 9+. Migrate base Docker image to eclipse-temurin as adoptopenjdk is deprecated. Add E2E test under Java 17. Upgrade protoc to 3.19.2. Add Istio 1.13.1 to E2E test matrix for verification. Upgrade Apache parent pom version to 25. Use the plugin version defined by the Apache maven parent. Upgrade maven-dependency-plugin to 3.2.0. Upgrade maven-assembly-plugin to 3.3.0. Upgrade maven-failsafe-plugin to 2.22.2. Upgrade maven-surefire-plugin to 2.22.2. Upgrade maven-jar-plugin to 3.2.2. Upgrade maven-enforcer-plugin to 3.0.0. Upgrade maven-compiler-plugin to 3.10.0. Upgrade maven-resources-plugin to 3.2.0. Upgrade maven-source-plugin to 3.2.1. Update codeStyle.xml to fix incompatibility on M1\u0026rsquo;s IntelliJ IDEA 2021.3.2. Update frontend-maven-plugin to 1.12 and npm to 16.14.0 for booster UI build. Improve CI with the GHA new feature \u0026ldquo;run failed jobs\u0026rdquo;. Fix ./mvnw compile not work if ./mvnw install is not executed at least once. Add JD_PRESERVE_LINE_FEEDS=true in official code style file. Upgrade OAP dependencies gson(2.9.0), guava(31.1), jackson(2.13.2), protobuf-java(3.18.4), commons-io(2.7), postgresql(42.3.3). Remove commons-pool and commons-dbcp from OAP dependencies(Not used before). Upgrade webapp dependencies gson(2.9.0), spring boot(2.6.6), jackson(2.13.2.2), spring cloud(2021.0.1), Apache httpclient(4.5.13). OAP Server Fix potential NPE in OAL string match and a bug when right-hand-side variable includes double quotes. Bump up Armeria version to 1.14.1 to fix CVE. Polish ETCD cluster config environment variables. Add the analysis of metrics in Satellite MetricsService. Fix Can't split endpoint id into 2 parts bug for endpoint ID. In the TCP in service mesh observability, endpoint name doesn\u0026rsquo;t exist in TCP traffic. Upgrade H2 version to 2.0.206 to fix CVE-2021-23463 and GHSA-h376-j262-vhq6. Extend column name override mechanism working for ValueColumnMetadata. Introduce new concept Layer and removed NodeType. More details refer to v9-version-upgrade. Fix query sort metrics failure in H2 Storage. Bump up grpc to 1.43.2 and protobuf to 3.19.2 to fix CVE-2021-22569. Add source layer and dest layer to relation. Follow protocol grammar fix GCPhrase -\u0026gt; GCPhase. Set layer to mesh relation. Add FAAS to SpanLayer. Adjust e2e case for V9 core. Support ZGC GC time and count metric collecting. Sync proto buffers files from upstream Envoy (Related to https://github.com/envoyproxy/envoy/pull/18955). Bump up GraphQL related dependencies to latest versions. Add normal to V9 service meta query. Support scope=ALL catalog for metrics. Bump up H2 to 2.1.210 to fix CVE-2022-23221. E2E: Add normal field to Service. Add FreeSql component ID(3017) of dotnet agent. E2E: verify OAP cluster model data aggregation. Fix SelfRemoteClient self observing metrics. Add env variables SW_CLUSTER_INTERNAL_COM_HOST and SW_CLUSTER_INTERNAL_COM_PORT for cluster selectors zookeeper ,consul,etcd and nacos. Doc update: configuration-vocabulary,backend-cluster about env variables SW_CLUSTER_INTERNAL_COM_HOST and SW_CLUSTER_INTERNAL_COM_PORT. Add Python MysqlClient component ID(7013) with mapping information. Support Java thread pool metrics analysis. Fix IoTDB Storage Option insert null index value. Set the default value of SW_STORAGE_IOTDB_SESSIONPOOL_SIZE to 8. Bump up iotdb-session to 0.12.4. Bump up PostgreSQL driver to fix CVE. Add Guava EventBus component ID(123) of Java agent. Add OpenFunction component ID(5013). Expose configuration responseTimeout of ES client. Support datasource metric analysis. [Breaking Change] Keep the endpoint avg resp time meter name the same with others scope. (This may break 3rd party integration and existing alarm rule settings) Add Python FastAPI component ID(7014). Support all metrics from MAL engine in alarm core, including Prometheus, OC receiver, meter receiver. Allow updating non-metrics templates when structure changed. Set default connection timeout of ElasticSearch to 3000 milliseconds. Support ElasticSearch 8 and add it into E2E tests. Disable indexing for field alarm_record.tags_raw_data of binary type in ElasticSearch storage. Fix Zipkin receiver wrong condition for decoding gzip. Add a new sampler (possibility) in LAL. Unify module name receiver_zipkin to receiver-zipkin, remove receiver_jaeger from application.yaml. Introduce the entity of Process type. Set the length of event#parameters to 2000. Limit the length of Event#parameters. Support large service/instance/networkAddressAlias list query by using ElasticSearch scrolling API, add metadataQueryBatchSize to configure scrolling page size. Change default value of metadataQueryMaxSize from 5000 to 10000 Replace deprecated Armeria API BasicToken.of with AuthToken.ofBasic. Implement v9 UI template management protocol. Implement process metadata query protocol. Expose more ElasticSearch health check related logs to help to diagnose Health check fails. reason: No healthy endpoint. Add source event generated metrics to SERVICE_CATALOG_NAME catalog. [Breaking Change] Deprecate All from OAL source. [Breaking Change] Remove SRC_ALL: 'All' from OAL grammar tree. Remove all_heatmap and all_percentile metrics. Fix ElasticSearch normal index couldn\u0026rsquo;t apply mapping and update. Enhance DataCarrier#MultipleChannelsConsumer to add priority for the channels, which makes OAP server has a better performance to activate all analyzers on default. Activate receiver-otel#enabledOcRules receiver with k8s-node,oap,vm rules on default. Activate satellite,spring-sleuth for agent-analyzer#meterAnalyzerActiveFiles on default. Activate receiver-zabbix receiver with agent rule on default. Replace HTTP server (GraphQL, agent HTTP protocol) from Jetty with Armeria. [Breaking Change] Remove configuration restAcceptorPriorityDelta (env var: SW_RECEIVER_SHARING_JETTY_DELTA , SW_CORE_REST_JETTY_DELTA). [Breaking Change] Remove configuration graphql/path (env var: SW_QUERY_GRAPHQL_PATH). Add storage column attribute indexOnly, support ElasticSearch only index and not store some fields. Add indexOnly=true to SegmentRecord.tags, AlarmRecord.tags, AbstractLogRecord.tags, to reduce unnecessary storage. [Breaking Change] Remove configuration restMinThreads (env var: SW_CORE_REST_JETTY_MIN_THREADS , SW_RECEIVER_SHARING_JETTY_MIN_THREADS). Refactor the core Builder mechanism, new storage plugin could implement their own converter and get rid of hard requirement of using HashMap to communicate between data object and database native structure. [Breaking Change] Break all existing 3rd-party storage extensions. Remove hard requirement of BASE64 encoding for binary field. Add complexity limitation for GraphQL query to avoid malicious query. Add Column.shardingKeyIdx for column definition for BanyanDB. Sharding key is used to group time series data per metric of one entity in one place (same sharding and/or same row for column-oriented database). For example, ServiceA\u0026#39;s traffic gauge, service call per minute, includes following timestamp values, then it should be sharded by service ID [ServiceA(encoded ID): 01-28 18:30 values-1, 01-28 18:31 values-2, 01-28 18:32 values-3, 01-28 18:32 values-4] BanyanDB is the 1st storage implementation supporting this. It would make continuous time series metrics stored closely and compressed better. NOTICE, this sharding concept is NOT just for splitting data into different database instances or physical files. Support ElasticSearch template mappings properties parameters and _source update. Implement the eBPF profiling query and data collect protocol. [Breaking Change] Remove Deprecated responseCode from sources, including Service, ServiceInstance, Endpoint Enhance endpoint dependency analysis to support cross threads cases. Refactor span analysis code structures. Remove isNotNormal service requirement when use alias to merge service topology from client side. All RPCs\u0026rsquo; peer services from client side are always normal services. This cause the topology is not merged correctly. Fix event type of export data is incorrect, it was EventType.TOTAL always. Reduce redundancy ThreadLocal in MAL core. Improve MAL performance. Trim tag\u0026rsquo;s key and value in log query. Refactor IoTDB storage plugin, add IoTDBDataConverter and fix ModifyCollectionInEnhancedForLoop bug. Bump up iotdb-session to 0.12.5. Fix the configuration of Aggregation and GC Count metrics for oap self observability E2E: Add verify OAP eBPF Profiling. Let multiGet could query without tag value in the InfluxDB storage plugin. Adjust MAL for V9, remove some groups, add a new Service function for the custom delimiter. Add service catalog DatabaseSlowStatement. Add Error Prone Annotations dependency to suppress warnings, which are not errors. UI [Breaking Change] Introduce Booster UI, remove RocketBot UI. [Breaking Change] UI Templates have been redesigned totally. GraphQL query is minimal compatible for metadata and metrics query. Remove unused jars (log4j-api.jar) in classpath. Bump up netty version to fix CVE. Add Database Connection pool metric. Re-implement UI template initialization for Booster UI. Add environment variable SW_ENABLE_UPDATE_UI_TEMPLATE to control user edit UI template. Add the Self Observability template of the SkyWalking Satellite. Add the template of OpenFunction observability. Documentation Reconstruction doc menu for v9. Update backend-alarm.md doc, support op \u0026ldquo;=\u0026rdquo; to \u0026ldquo;==\u0026rdquo;. Update backend-meter.md doc . Add \u0026lt;STAM: Enhancing Topology Auto Detection For A Highly Distributed and Large-Scale Application System\u0026gt; paper. Add Academy menu for recommending articles. Remove All source relative document and examples. Update Booster UI\u0026rsquo;s dependency licenses. Add profiling doc, and remove service mesh intro doc(not necessary). Add a doc for virtual database. Rewrite UI introduction. Update k8s-monitoring, backend-telemetry and v9-version-upgrade doc for v9. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking 9.0.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSkyWalking v9 is the next …\u003c/strong\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-9.0.0/","title":"Release Apache SkyWalking APM 9.0.0"},{"body":"SkyWalking CLI 0.10.0 is released. Go to downloads page to find release tars.\nFeatures Allow setting start and end with relative time (#128) Add some commands for the browser (#126) Add the sub-command service layer to query services according to layer (#133) Add the sub-command layer list to query layer list (#133) Add the sub-command instance get to query single instance (#134) Add the sub-command endpoint get to query single endpoint info (#134) Change the GraphQL method to the v9 version according to the server version (#134) Add normal field to Service entity (#136) Add the command process for query Process metadata (#137) Add the command profiling ebpf for process ebpf profiling (#138) Support getprofiletasklogs query (#125) Support query list alarms (#127) [Breaking Change] Update the command profile as a sub-command profiling trace, and update profiled-analyze command to analysis (#138) profiling ebpf/trace analysis generates the profiling graph HTML on default and saves it to the current work directory (#138) Bug Fixes Fix quick install (#131) Set correct go version in publishing snapshot docker image (#124) Stop build kit container after finishing (#130) Chores Add cross platform build targets (#129) Update download host (#132) ","excerpt":"\u003cp\u003eSkyWalking CLI 0.10.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"features\"\u003eFeatures\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eAllow …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cli-0-10-0/","title":"Release Apache SkyWalking CLI 0.10.0"},{"body":"SkyWalking is an open-source APM system, including monitoring, tracing, and diagnosing capabilities for distributed systems in Cloud Native architecture. It covers monitoring for Linux, Kubernetes, Service Mesh, Serverless/Function-as-a-Service, agent-attached services, and browsers. With data covering traces, metrics, logs, and events, SkyWalking is a full-stack observability APM system.\nOpen Source Promotion Plan is a summer program organized and long-term supported by Open Source Software Supply Chain Promotion Plan. It aims to encourage college students to actively participate in developing and maintaining open-source software and promote the vigorous development of an excellent open-source software community.\nApache SkyWalking has been accepted in OSPP 2022\nProject Description Difficulty Mentor / E-mail Expectation Tech. Requirements Repository SkyAPM-PHP Add switches for monitoring items Advanced Level Yanlong He / heyanlong@apache.org Complete project development work C++, GO, PHP https://github.com/SkyAPM/SkyAPM-php-sdk SkyWalking-Infra-E2E Optimize verifier Normal Level Huaxi Jiang / hoshea@apache.org 1. Continue to verify cases when other cases fail 2. Merge retry outputs 3. Prettify verify results\u0026rsquo; output Go https://github.com/apache/skywalking-infra-e2e SkyWalking Metrics anomaly detection with machine learning Advanced Level Yihao Chen / yihaochen@apache.org An MVP version of ML-powered metrics anomaly detection using dynamic baselines and thresholds Python, Java https://github.com/apache/skywalking SkyWalking Python Collect PVM metrics and send the metrics to OAP backend, configure dashboard in UI Normal Level Zhenxu Ke / kezhenxu94@apache.org Core Python VM metrics should be collected and displayed in SkyWalking. Python https://github.com/apache/skywalking-python issue SkyWalking BanyanDB Command line tools for BanyanDB Normal Level Hongtao Gao / hanahmily@apache.org Command line tools should access relevant APIs to manage resources and online data. Go https://github.com/apache/skywalking-banyandb SkyWalking SWCK CRD and controller for BanyanDB Advance Level Ye Cao / dashanji@apache.org CRD and controller provision BanyanDB as the native Storage resource. Go https://github.com/apache/skywalking-swck SkyAPM-Go2sky Collect golang metrics such as gc, goroutines and threads, and send the the metrics to OAP backend, configure dashboard in UI Normal Level Wei Zhang / zhangwei24@apache.org Core golang metrics should be collected and displayed in SkyWalking. Go https://github.com/SkyAPM/go2sky SkyWalking Collect system metrics such as system_load, cpu_usage, mem_usage from telegraf and send the metrics to OAP backend, configure dashboard in UI Normal Level Haoyang Liu / liuhaoyangzz@apache.org System metrics should be collected and displayed in SkyWalking. Java https://github.com/apache/skywalking Mentors could submit pull requests to update the above list.\nContact the community You could send emails to mentor\u0026rsquo;s personal email to talk about the project and details. The official mail list of the community is dev@skywalking.apache.org. You need to subscribe to the mail list to get all replies. Send mail to dev-suscribe@skywalking.apache.org and follow the replies.\n","excerpt":"\u003cp\u003e\u003cstrong\u003eSkyWalking\u003c/strong\u003e is an open-source APM system, including monitoring, tracing, and diagnosing capabilities …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/summer-ospp-2022/readme/","title":"Open Source Promotion Plan 2022 -- Project List"},{"body":"如果要讨论提高自己系统设计能力的方式，我想大多数人都会选择去阅读优秀开源项目的源代码。近年来我参与了多个监控服务的开发工作，并在工作中大量地使用了 SkyWalking 并对其进行二次开发。在这个过程中，我发现 SkyWalking 天然的因其国产的身份，整套源代码地组织和设计非常符合国人的编程思维。由此我录制了本套课程，旨在和大家分享我的一些浅薄的心得和体会。\n本套课程分为两个阶段，分别讲解 Agent 端和 OAP 端地设计和实现。每个阶段的内容都是以启动流程作为讲解主线，逐步展开相关的功能模块。除了对 SKyWalking 本身内容进行讲解，课程还针对 SKyWalking 使用到的一些较为生僻的知识点进行了补充讲解（如 synthetic、NBAC 机制、自定义类加载器等），以便于大家更清晰地掌握课程内容。\nSkyWalking8.7.0 源码分析 - 视频课程直达链接\n目前课程已更新完 Agent 端的讲解，目录如下：\n01-开篇和源码环境准备 02-Agent 启动流程 03-Agent 配置加载流程 04-自定义类加载器 AgentClassLoader 05-插件定义体系 07-插件加载 06-定制 Agent 08-什么是 synthetic 09-NBAC 机制 10-服务加载 11-witness 组件版本识别 12-Transform 工作流程 13-静态方法插桩 14-构造器和实例方法插桩 15-插件拦截器加载流程(非常重要) 16-运行时插件效果的字节码讲解 17-JDK 类库插件工作原理 18-服务-GRPCChanelService 19-服务-ServiceManagementClient 20-服务-CommandService 21-服务-SamplingService 22-服务-JVMService 23-服务-KafkaXxxService 24-服务-StatusCheckService 25-链路基础知识 26-链路 ID 生成 27-TraceSegment 28-Span 基本概念 29-Span 完整模型 30-StackBasedTracingSpan 31-ExitSpan 和 LocalSpan 32-链路追踪上下文 TracerContext 33-上下文适配器 ContextManager 34-DataCarrier-Buffer 35-DataCarrier-全解 36-链路数据发送到 OAP B站视频地址\n","excerpt":"\u003cp\u003e如果要讨论提高自己系统设计能力的方式，我想大多数人都会选择去阅读优秀开源项目的源代码。近年来我参与了多个监控服务的开发工作，并在工作中大量地使用了 SkyWalking 并对其进行二次开发。在这个过程 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2022-03-25-skywalking-source-code-analyzation/","title":"[视频] SkyWalking 8.7.0 源码分析"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/course/","title":"Course"},{"body":"SkyWalking NodeJS 0.4.0 is released. Go to downloads page to find release tars.\nFix mysql2 plugin install error. (#74) Update IORedis Plugin, fill dbinstance tag as host if condition.select doesn\u0026rsquo;t exist. (#73) Experimental AWS Lambda Function support. (#70) Upgrade dependencies to fix vulnerabilities. (#68) Add lint pre-commit hook and migrate to eslint. (#66, #67) Bump up gRPC version, and use its new release repository. (#65) Regard baseURL when in Axios Plugin. (#63) Add an API to access the trace id. (#60) Use agent test tool snapshot Docker image instead of building in CI. (#59) Wrapped IORedisPlugin call in try/catch. (#58) ","excerpt":"\u003cp\u003eSkyWalking NodeJS 0.4.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eFix mysql2 plugin …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-nodejs-0-4-0/","title":"Release Apache SkyWalking for NodeJS 0.4.0"},{"body":"大约二十年前我刚开始进入互联网的世界的时候，支撑起整个网络的基础设施，就包括了 Apache 软件基金会（ASF）治下的软件。\nApache Httpd 是开启这个故事的软件，巅峰时期有超过七成的市场占有率，即使是在今天 NGINX 等新技术蓬勃发展的时代，也有三成左右的市场占有率。由 Linux、Apache Httpd、MySQL 和 PHP 组成的 LAMP 技术栈，是开源吞噬软件应用的第一场大型胜利。\n我从 2018 年参与 Apache Flink 开始正式直接接触到成立于 1999 年，如今已经有二十年以上历史的 Apache 软件基金会，并在一年后的 2019 年成为 Apache Flink 项目 Committer 队伍的一员，2020 年成为 Apache Curator 项目 PMC（项目管理委员会）的一员。今年，经由姜宁老师推荐，成为了 Apache Members 之一，也就是 Apache 软件基金会层面的正式成员。\n我想系统性地做一个开源案例库已经很久了。无论怎么分类筛选优秀的开源共同体，The Apache Community 都是无法绕开的。然而，拥有三百余个开源软件项目的 Apache 软件基金会，并不是一篇文章就能讲清楚的案例。本文也没有打算写成一篇长文顾及方方面面，而是启发于自己的新角色，回顾过去近五年在 Apache Community 当中的经历和体验，简单讨论 Apache 的理念，以及这些理念是如何落实到基金会组织、项目组织以及每一个参与者的日常生活事务当中的。\n不过，尽管对讨论的对象做了如此大幅度的缩减，由我自己来定义什么是 Apache 的理念未免也太容易有失偏颇。幸运的是，Apache Community 作为优秀的开源共同体，当然做到了我在《共同创造价值》一文中提到的回答好“我能为你做什么”以及“我应该怎么做到”的问题。Apache Community 的理念之一就是 Open Communications 即开放式讨论，由此产生的公开材料以及基于公开材料整理的文档汗牛充栋。这既是研究 Apache Community 的珍贵材料，也为还原和讨论一个真实的 Apache Community 提出了不小的挑战。\n无论如何，本文将以 Apache 软件基金会在 2020 年发布的纪录片 Trillions and Trillions Served 为主线，结合其他文档和文字材料来介绍 Apache 的理念。\n以人为本 纪录片一开始就讲起了 Apache Httpd 项目的历史，当初的 Apache Group 是基于一个源代码共享的 Web Server 建立起来的邮件列表上的一群人。软件开发当初的印象如同科学研究，因此交流源码在近似科学共同体的开源共同体当中是非常自然的。\n如同 ASF 的联合创始人 Brian Behlendorf 所说，每当有人解决了一个问题或者实现了一个新功能，他出于一种朴素的分享精神，也就是“为什么不把补丁提交回共享的源代码当中呢”的念头，基于开源软件的协作就这样自然发生了。纪录片中有一位提到，她很喜欢 Apache 这个词和 a patchy software 的谐音，共享同一个软件的补丁（patches）就是开源精神最早诞生的形式。\n这是 Apache Community 的根基，我们将会看到这种朴素精神经过发展形成了一个怎样的共同体，在共同体的发展过程当中，这样的根基又是如何深刻地影响了 Apache 理念的方方面面。\nApache Group 的工作模式还有一个重要的特征，那就是每个人都是基于自己的需求修复缺陷或是新增功能，在邮件列表上交流和提交补丁的个人，仅仅只是代表他个人，而没有一个“背后的组织”或者“背后的公司”。因此，ASF 的 How it Works 文档中一直强调，在基金会当中的个体，都只是个体（individuals），或者称之为志愿者（volunteers）。\n我在某公司的分享当中提到过，商业产品可以基于开源软件打造，但是当公司的雇员出现在社群当中的时候，他应该保持自己志愿者的身份。这就像是开源软件可以被用于生产环境或者严肃场景，例如航空器的发射和运行离不开 Linux 操作系统，但是开源软件本身是具有免责条款的。商业公司或专业团队提供服务保障，而开源软件本身是 AS IS 的。同样，社群成员本人可以有商业公司雇员的身份，但是他在社群当中，就是一个志愿者。\n毫无疑问，这种论调当即受到了质疑，因为通常的认知里，我就是拿了公司的钱，就是因为在给这家公司打工，才会去关注这个项目，你非要说我是一个志愿者，我还就真不是一个志愿者，你怎么说？\n其实这个问题，同样在 How it Works 文档中已经有了解答。\nAll participants in ASF projects are volunteers and nobody (not even members or officers) is paid directly by the foundation to do their job. There are many examples of committers who are paid to work on projects, but never by the foundation itself. Rather, companies or institutions that use the software and want to enhance it or maintain it provide the salary.\n我当时基于这样的认识，给到质疑的回答是，如果你不想背负起因为你是员工，因此必须响应社群成员的 issue 或 PR 等信息，那么你可以试着把自己摆在一个 volunteer 的角度来观察和参与社群。实际上，你并没有这样的义务，即使公司要求你必须回答，那也是公司的规定，而不是社群的要求。如果你保持着这样的认识和心态，那么社群于你而言，才有可能是一个跨越职业生涯不同阶段的归属地，而不是工作的附庸。\n社群从来不会从你这里索取什么，因为你的参与本身也是自愿的。其他社群成员会感谢你的参与，并且如果相处得好，这会是一个可爱的去处。社群不是你的敌人，不要因为公司下达了离谱的社群指标而把怒火发泄在社群和社群成员身上。压力来源于公司，作为社群成员的你本来可以不用承受这些。\nApache Community 对个体贡献者组成社群这点有多么重视呢？只看打印出来不过 10 页 A4 纸的 How it Works 文档，volunteer 和 individuals 两个词加起来出现了 19 次。The Apache Way 文档中强调的社群特征就包括了 Independence 一条，唯一并列的另一个是经常被引用的 Community over code 原则。甚至，有一个专门的 Project independence 文档讨论了 ASF 治下的项目如何由个体志愿者开发和维护，又为何因此是中立和非商业性的。\nINDIVIDUALS COMPOSE THE ASF 集中体现了 ASF 以人为本的理念。实际上，不止上面提到的 Independence 强调了社群成员个体志愿者的属性，Community over code 这一原则也在强调 ASF 关注围绕开源软件聚集起来的人，包括开发者、用户和其他各种形式的参与者。人是维持社群常青的根本，在后面具体讨论 The Apache Way 的内容的时候还会展开。\n上善若水 众所周知，Apache License 2.0 (APL-2.0) 是所谓的宽容式软件协议。也就是说，不同于 GPL 3.0 这样的 Copyleft 软件协议要求衍生作品需要以相同的条款发布，其中包括开放源代码和自由修改从而使得软件源代码总是可以获取和修改的，Apache License 在协议内容当中仅保留了著作权和商标，并要求保留软件作者的任何声明（NOTICE）。\nASF 在软件协议上的理念是赋予最大程度的使用自由，鼓励用户和开发者参与到共同体当中来，鼓励与上游共同创造价值，共享补丁。“鼓励”而不是“要求”，是 ASF 和自由软件基金会（Free Software Foundation, FSF）最主要的区别。\n这一倾向可以追溯到 Apache Group 建立的基础。Apache Httpd 派生自伊利诺伊大学的 NCSA Httpd 项目，由于使用并开发这个 web server 的人以邮件列表为纽带聚集在一起，通过交换补丁来开发同一个项目。在项目的发起人 Robert McCool 等大学生毕业以后，Apache Group 的发起人们接过这个软件的维护和开发工作。当时他们看到的软件协议，就是一个 MIT License 精神下的宽容式软件协议。自然而然地，Apache Group 维护 Apache Httpd 的时候，也就继承了这个协议。\n后来，Apache Httpd 打下了 web server 的半壁江山，也验证了这一模式的可靠性。虽然有些路径依赖的嫌疑，但是 ASF 凭借近似“上善若水”的宽容理念，在二十年间成功创造了数以百亿计美元价值的三百多个软件项目。\n纪录片中 ASF 的元老 Ted Dunning 提到，在他早期创造的软件当中，他会在宽容式软件协议之上，添加一个商用的例外条款。这就像是著名开源领域律师 Heather Meeker 起草的 The Commons Clause 附加条款。\nWithout limiting other conditions in the License, the grant of rights under the License will not include, and the License does not grant to you, the right to Sell the Software.\n附加 The Commons Clause 条款的软件都不是符合 OSD 定义的开源软件，也不再是原来的协议了。NebulaGraph 曾经在附加 The Commons Clause 条款的情况下声称自己是 APL-2.0 协议许可的软件，当时的 ASF 董事吴晟就提 issue (vesoft-inc/nebula#3247) 指出这一问题。NebulaGraph 于是删除了所有 The Commons Clause 的字样，保证无误地以 APL-2.0 协议许可该软件。\nTed Dunning 随后提到，这样的附加条款实际上严重影响了软件的采用。他意识到自己实际上并不想为此打官司，因此加上这样的条款对他而言是毫无意义的。Ted Dunning 于是去掉了附加条款，而这使得使用他的软件的条件能够简单的被理解，从而需要这些软件的用户能够大规模的采用。“水利万物而不争”，反而是不去强迫和约束用户行为的做法，为软件赢得了更多贡献。\n我仍然很敬佩采用 GPL 系列协议发布高质量软件的开发者，Linux 和 GCC 这样的软件的成功改变了世人对软件领域的自由的认识。然而，FSF 自己也认识到需要提出修正的 LGPL 来改进应用程序以外的软件的发布和采用，例如基础库。\nAPL-2.0 的思路与之不同，它允许任何人以任何形式使用、修改和分发软件，因此 ASF 治下的项目，以及 Linux Foundation 治下采用 APL-2.0 的项目，以及更多个人或组织采用 APL-2.0 的项目，共同构成了强大的开源软件生态，涵盖了应用软件，基础库，开发工具和框架等等各个方面。事实证明，“鼓励”而不是“要求”用户秉持 upstream first 的理念，尽可能参与到开源共同体并交换知识和补丁，共同创造价值，是能够制造出高质量的软件，构建出繁荣的社群和生态的。\n匠人精神 Apache Community 关注开发者的需要。\nApache Group 成立 ASF 的原因，是在 Apache Httpd 流行起来以后，商业公司和社会团体开始寻求和这个围绕项目形成的群体交流。然而，缺少一个正式的法律实体让组织之间的往来缺乏保障和流程。因此，如同纪录片当中提到的，ASF 成立的主要原因，是为了支撑 Apache Httpd 项目。只不过当初的创始成员们很难想到的是，ASF 最终支撑了数百个开源项目。\n不同于 Linux Foundation 是行业联盟，主要目的是为了促进其成员的共同商业利益，ASF 主要服务于开发者，由此支撑开源项目的开发以及开源共同体的发展。\n举例来说，进入 ASF 孵化器的项目都能够在 ASF Infra 的支持下运行自己的 apache.org 域名的网站，将代码托管在 ASF 仓库中上，例如 Apache GitBox Repositories 和 Apache GitHub Organization 等。这些仓库上运行着自由取用的开发基础设施，例如持续集成和持续发布的工具和资源等等。ASF 还维护了自己的邮件列表和文件服务器等一系列资源，以帮助开源项目建立起自己的共同体和发布自己的构件。\n反观 Linux Foundation 的主要思路，则是关注围绕项目聚集起来的供应商，以行业联盟的形式举办联合市场活动扩大影响，协调谈判推出行业标准等等。典型地，例如 CNCF 一直致力于定义云上应用开发的标准，容器虚拟化技术的标准。上述 ASF Infra 关注的内容和资源，则大多需要项目开发者自己解决，这些开发者往往主要为一个或若干个供应商工作，他们解决的方式通常也是依赖供应商出力。\n当然，上面的对比只是为了说明区别，并无优劣之分，也不相互对立。ASF 的创始成员 Brian Behlendorf 同时是 Linux Foundation 下 Open Source Security Foundation 的经理，以及 Hyperledger 的执行董事。\nASF 关注开发者的需要，体现出 Apache Community 及其成员对开发者的人文关怀。纪录片中谈到 ASF 治下项目的开发体验时，几乎每个人的眼里都有光。他们谈论着匠人精神，称赞知识分享，与人合作，以及打磨技艺的愉快经历。实际上，要想从 Apache 孵化器中成功毕业，相当部分的 mentor 关注的是围绕开源软件形成的共同体，能否支撑开源软件长久的发展和采用，这其中就包括共同体成员是否能够沉下心来做技术，而不是追求花哨的数字指标和人头凑数。\n讲几个具体的开发者福利。\n每个拥有 @apache.org 邮箱的人，即成为 ASF 治下项目 Committer 或 ASF Member 的成员，JetBrains 会提供免费的全家桶订阅授权码。我从 2019 年成为 Apache Flink 项目的 Committer 以后，已经三年沉浸在 IDEA 和 CLion 的包容下，成为彻底使用 IDE 主力开发的程序员了。\nApache GitHub Organization 下的 GitHub Actions 资源是企业级支持，这部分开销也是由 ASF 作为非营利组织募资和运营得到的资金支付的。基本上，如果你的项目成为 Apache 孵化器项目或顶级项目，那么和 GitHub Actions 集成的 CI 体验是非常顺畅的。Apache SkyWalking 只算主仓库就基于 GitHub Actions 运行了十多个端到端测试作业，Apache Pulsar 也全面基于 GitHub Actions 集成了自己的 CI 作业。\n提到匠人精神，一个隐形的开发者福利，其实是 ASF 的成员尤其是孵化器的 mentor 大多是经验非常丰富的开发者。软件开发不只是写代码，Apache Community 成员之间相互帮助，能够帮你跟上全世界最前沿的开发实践。如何提问题，如何做项目管理，如何发布软件，这些平日里在学校在公司很难有机会接触的知识和实践机会，在 Apache Community 当中只要你积极承担责任，都是触手可得的。\n当然，如何写代码也是开发当中最常交流的话题。我深入接触 Maven 开始于跟 Flink Community 的 Chesnay Schepler 的交流。我对 Java 开发的理解，分布式系统开发的知识，很大程度上也得到了 Apache Flink 和 Apache ZooKeeper 等项目的成员的帮助，尤其是 Till Rohrmann 和 Enrico Olivelli 几位。上面提到的 Ted Dunning 开始攻读博士的时候，我还没出生。但是我在项目当中用到 ZooKeeper 的 multi 功能并提出疑问和改进想法的时候，也跟他有过一系列的讨论。\n谈到技艺就会想起人，这也是 ASF 一直坚持以人为本带来的社群风气。\n我跟姜宁老师在一年前认识，交流 The Apache Way 期间萌生出相互认同。姜宁老师在 Apache 孵化器当中帮助众多项目理解 The Apache Way 并予以实践，德高望重。在今年的 ASF Members 年会当中，姜宁老师也被推举为 ASF Board 的一员。\n我跟吴晟老师在去年认识。他经常会强调开发者尤其是没有强烈公司背景的开发者的视角，多次提到这些开发者是整个开源生态的重要组成部分。他作为 PMC Chair 的 Apache SkyWalking 项目相信“没有下一个版本的计划，只知道会有下一个版本”，这是最佳实践的传播，也是伴随技术的文化理念的传播。SkyWalking 项目出于自己需要，也出于为开源世界添砖加瓦的动机创建的 SkyWalking Eyes 项目，被广泛用在不止于 ASF 治下项目，而是整个开源世界的轻量级的软件协议审计和 License Header 检查上。\n主要贡献在 Apache APISIX 的琚致远同学今年也被推选成为 Apache Members 的一员。他最让我印象深刻的是在 APISIX 社群当中积极讨论社群建设的议题，以及作为 APISIX 发布的 GSoC 项目的 mentor 帮助在校学生接触开源，实践开源，锻炼技艺。巧合的是，他跟我年龄相同，于是我痛失 Youngest Apache Member 的噱头，哈哈。\n或许，参与 Apache Community 就是这样的一种体验。并不是什么复杂的叙事，只是找到志同道合的人做出好的软件。我希望能够为提升整个软件行业付出自己的努力，希望我（参与）制造的软件创造出更大的价值，这里的人看起来大都也有相似的想法，这很好。仅此而已。\n原本还想聊聊 The Apache Way 的具体内容，还有介绍 Apache Incubator 这个保持 Apache Community 理念常青，完成代际传承的重要机制，但是到此为止似乎也很好。Apache Community 的故事和经验很难用一篇文章讲完，这两个话题就留待以后再写吧。\n","excerpt":"\u003cp\u003e大约二十年前我刚开始进入互联网的世界的时候，支撑起整个网络的基础设施，就包括了 Apache 软件基金会（ASF）治下的软件。\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://httpd.apache.org/\"\u003eApache Httpd\u003c/a\u003e 是开启这个故事的软件，巅峰时期有超过七成的市场 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2022-03-14-the-apache-community/","title":"我眼中的 The Apache Way"},{"body":"SkyWalking Client Rust 0.1.0 is released. Go to downloads page to find release tars.\n","excerpt":"\u003cp\u003eSkyWalking Client Rust 0.1.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-client-rust-0-1-0/","title":"Release Apache SkyWalking Client Rust 0.1.0"},{"body":"SkyWalking Java Agent 8.9.0 is released. Go to downloads page to find release tars. Changes by Version\n8.9.0 Support Transaction and fix duplicated methods enhancements for jedis-2.x plugin. Add ConsumerWrapper/FunctionWrapper to support CompletableFuture.x.thenAcceptAsync/thenApplyAsync. Build CLI from Docker instead of source codes, add alpine based Docker image. Support set instance properties in json format. Upgrade grpc-java to 1.42.1 and protoc to 3.17.3 to allow using native Mac osx-aarch_64 artifacts. Add doc about system environment variables to configurations.md Avoid ProfileTaskChannelService.addProfilingSnapshot throw IllegalStateException(Queue full) Increase ProfileTaskChannelService.snapshotQueue default size from 50 to 4500 Support 2.8 and 2.9 of pulsar client. Add dubbo 3.x plugin. Fix TracePathMatcher should match pattern \u0026ldquo;**\u0026rdquo; with paths end by \u0026ldquo;/\u0026rdquo; Add support returnedObj expression for apm-customize-enhance-plugin Fix the bug that httpasyncclient-4.x-plugin puts the dirty tracing context in the connection context Compatible with the versions after dubbo-2.7.14 Follow protocol grammar fix GCPhrase -\u0026gt; GCPhase. Support ZGC GC time and count metric collect. (Require 9.0.0 OAP) Support configuration for collecting redis parameters for jedis-2.x and redisson-3.x plugin. Migrate base images to Temurin and add images for ARM. (Plugin Test) Fix compiling issues in many plugin tests due to they didn\u0026rsquo;t lock the Spring version, and Spring 3 is incompatible with 2.x APIs and JDK8 compiling. Support ShardingSphere 5.0.0 Bump up gRPC to 1.44.0, fix relative CVEs. Documentation Add a FAQ, Why is -Djava.ext.dirs not supported?. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking Java Agent 8.9.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nChanges by …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-java-agent-8-9-0/","title":"Release Apache SkyWalking Java Agent 8.9.0"},{"body":"Apache SkyWalking is an open-source APM for a distributed system, Apache Software Foundation top-level project.\nOn Jan. 28th, we received a License violation report from one of the committers (anonymously). They have a cloud service called Application Performance Monitoring - Distributed Tracing (应用性能监控全链路版). At the Java service monitoring section, it provides this agent download link\nwget https://datarangers.com.cn/apminsight/repo/v2/download/java-agent/apminsight-java-agent_latest.tar.gz\nWe downloaded it at 23:15 Jan. 28th UTC+8(Beijing), and archived it at here\nWe have confirmed this is a distribution of SkyWalking Java agent.\nWe listed several pieces of evidence to prove this here, every reader could compare with the official SkyWalking source codes\nThe first and the easiest one is agent.config file, which is using the same config keys, and the same config format. This is the Volcengine\u0026rsquo;s version, and check SkyWalking agent.config In the apmplus-agent.jar, Volcengine\u0026rsquo;s agent core jar, you could easily find several core classes exactly as same as SkyWalking\u0026rsquo;s. The ComponentsDefine class is unchanged, even with component ID and name. This is Volcengine\u0026rsquo;s version, and check SkyWalking\u0026rsquo;s version\nThe whole code names, package names, and hierarchy structure are all as same as SkyWalking 6.x version. This is the Volcengine package hierarchy structure, and check the SkyWalking\u0026rsquo;s version\nVolcengine Inc.\u0026rsquo;s team changed all package names, removed the Apache Software Foundation\u0026rsquo;s header, and don\u0026rsquo;t keep Apache Software Foundation and Apache SkyWalking\u0026rsquo;s LICENSE and NOTICE file in their redistribution.\nAlso, we can\u0026rsquo;t find anything on their website to declare they are distributing SkyWalking.\nAll above have proved they are violating the Apache 2.0 License, and don\u0026rsquo;t respect Apache Software Foundation and Apache SkyWalking\u0026rsquo;s IP and Branding.\nWe have contacted their legal team, and wait for their official response.\nResolution On Jan. 30th night, UTC+8, 2022. We received a response from Volcengine\u0026rsquo;s APMPlus team. They admitted their violation behaviors, and made the following changes.\nVolcengine\u0026rsquo;s APMPlus service page was updated on January 30th and stated that the agent is a fork version(re-distribution) of Apache SkyWalking agent. Below is the screenshot of Volcengine\u0026rsquo;s APMPlus product page. Volcengine\u0026rsquo;s APMPlus agent distributions were also updated and include SkyWalking\u0026rsquo;s License and NOTICE now. Below is the screenshot of Volcengine\u0026rsquo;s APMPlus latest agent, you could download from the product page. We keep a copy of their Jan. 30th 2022 at here. Volcengine\u0026rsquo;s APMPlus team had restored all license headers of SkyWalking in the agent, and the modifications of the project files are also listed in \u0026ldquo;SkyWalking-NOTICE\u0026rdquo;, which you could download from the product page. We have updated the status to the PMC mail list. This license violation issue has been resolved for now.\nAppendix Inquiries of committers Q: I hope Volcengine Inc. can give a reason for this license issue, not just an afterthought PR. This will not only let us know where the issue is but also avoid similar problems in the future.\nA(apmplus apmplus@volcengine.com):\nThe developers neglected this repository during submitting compliance assessment. Currently, APMPlus team had introduced advanced tools provided by the company for compliance assessment, and we also strengthened training for our developers. In the future, the compliance assessment process will be further improved from tool assessment and manual assessment. ","excerpt":"\u003cp\u003e\u003ca href=\"https://skywalking.apache.org\"\u003eApache SkyWalking\u003c/a\u003e is an open-source APM for a distributed system, Apache Software Foundation …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2022-01-28-volcengine-violates-aplv2/","title":"[Resolved][License Issue] Volcengine Inc.(火山引擎) violates the Apache 2.0 License when using SkyWalking."},{"body":"Background In the Apache SkyWalking ecosystem, the OAP obtains metrics, traces, logs, and event data through SkyWalking Agent, Envoy, or other data sources. Under the gRPC protocol, it transmits data by communicating with a single server node. Only when the connection is broken, the reconnecting policy would be used based on DNS round-robin mode. When new services are added at runtime or the OAP load is kept high due to increased traffic of observed services, the OAP cluster needs to scale out for increased traffic. The load of the new OAP node would be less due to all existing agents having connected to previous nodes. Even without scaling, the load of OAP nodes would be unbalanced, because the agent would keep the connection due to random policy at the booting stage. In these cases, it would become a challenge to keep up the health status of all nodes, and be able to scale out when needed.\nIn this article, we mainly discuss how to solve this challenge in SkyWalking.\nHow to Load Balance SkyWalking mainly uses the gRPC protocol for data transmission, so this article mainly introduces load balancing in the gRPC protocol.\nProxy Or Client-side Based on the gRPC official Load Balancing blog, there are two approaches to load balancing:\nClient-side: The client perceives multiple back-end services and uses a load-balancing algorithm to select a back-end service for each RPC. Proxy: The client sends the message to the proxy server, and the proxy server load balances the message to the back-end service. From the perspective of observability system architecture:\nPros Cons Client-side High performance because of the elimination of extra hop Complex client (cluster awareness, load balancing, health check, etc.)Ensure each data source to be connected provides complex client capabilities Proxy Simple Client Higher latency We choose Proxy mode for the following reasons:\nObservable data is not very time-sensitive, a little latency caused by transmission is acceptable. A little extra hop is acceptable and there is no impact on the client-side. As an observability platform, we cannot/should not ask clients to change. They make their own tech decisions and may have their own commercial considerations. Transmission Policy In the proxy mode, we should determine the transmission path between downstream and upstream.\nDifferent data protocols require different processing policies. There are two transmission policies:\nSynchronous: Suitable for protocols that require data exchange in the client, such as SkyWalking Dynamic Configuration Service. This type of protocol provides real-time results. Asynchronous batch: Used when the client doesn’t care about the upstream processing results, but only the transmitted data (e.g., trace report, log report, etc.) The synchronization policy requires that the proxy send the message to the upstream server when receiving the client message, and synchronously return the response data to the downstream client. Usually, only a few protocols need to use the synchronization policy.\nAs shown below, after the client sends the request to the Proxy, the proxy would send the message to the server synchronously. When the proxy receives the result, it returns to the client.\nThe asynchronous batch policy means that the data is sent to the upstream server in batches asynchronously. This policy is more common because most protocols in SkyWalking are primarily based on data reporting. We think using the queue as a buffer could have a good effect. The asynchronous batch policy is executed according to the following steps:\nThe proxy receives the data and wraps it as an Event object. An event is added into the queue. When the cycle time is reached or when the queue elements reach the fixed number, the elements in the queue will parallel consume and send to the OAP. The advantage of using queues is:\nSeparate data receiving and sending to reduce the mutual influence. The interval quantization mechanism can be used to combine events, which helps to speed up sending events to the OAP. Using multi-threaded consumption queue events can make fuller use of network IO. As shown below, after the proxy receives the message, the proxy would wrap the message as an event and push it to the queue. The message sender would take batch events from the queue and send them to the upstream OAP.\nRouting Routing algorithms are used to route messages to a single upstream server node.\nThe Round-Robin algorithm selects nodes in order from the list of upstream service nodes. The advantage of this algorithm is that the number of times each node is selected is average. When the size of the data is close to the same, each upstream node can handle the same quantity of data content.\nWith the Weight Round-Robin, each upstream server node has a corresponding routing weight ratio. The difference from Round-Robin is that each upstream node has more chances to be routed according to its weight. This algorithm is more suitable to use when the upstream server node machine configuration is not the same.\nThe Fixed algorithm is a hybrid algorithm. It can ensure that the same data is routed to the same upstream server node, and when the upstream server scales out, it still maintains routing to the same node; unless the upstream node does not exist, it will reroute. This algorithm is mainly used in the SkyWalking Meter protocol because this protocol needs to ensure that the metrics of the same service instance are sent to the same OAP node. The Routing steps are as follows:\nGenerate a unique identification string based on the data content, as short as possible. The amount of data is controllable. Get the upstream node of identity from LRU Cache, and use it if it exists. According to the identification, generate the corresponding hash value, and find the upstream server node from the upstream list. Save the mapping relationship between the upstream server node and identification to LRU Cache. The advantage of this algorithm is to bind the data with the upstream server node as much as possible, so the upstream server can better process continuous data. The disadvantage is that it takes up a certain amount of memory space to save the corresponding relationship.\nAs shown below, the image is divided into two parts:\nThe left side represents that the same data content always is routed to the same server node. The right side represents the data routing algorithm. Get the number from the data, and use the remainder algorithm to obtain the position. We choose to use a combination of Round-Robin and Fixed algorithm for routing:\nThe Fixed routing algorithm is suitable for specific protocols, mainly used when passing metrics data to the SkyWalking Meter protocol The Round-Robin algorithm is used by default. When the SkyWalking OAP cluster is deployed, the configuration of the nodes needs to be as much the same as possible, so there would be no need to use the Weight Round-Robin algorithm. How to balance the load balancer itself? Proxy still needs to deal with the load balancing problem from client to itself, especially when deploying a Proxy cluster in a production environment.\nThere are three ways to solve this problem:\nConnection management: Use the max_connection config on the client-side to specify the maximum connection duration of each connection. For more information, please read the proposal. Cluster awareness: The proxy has cluster awareness, and actively disconnects the connection when the load is unbalanced to allow the client to re-pick up the proxy. Resource limit+HPA: Restrict the connection resource situation of each proxy, and no longer accept new connections when the resource limit is reached. And use the HPA mechanism of Kubernetes to dynamically scale out the number of the proxy. Connection management Cluster awareness Resource Limit+HPA Pros Simple to use Ensure that the number of connections in each proxy is relatively Simple to use Cons Each client needs to ensure that data is not lostThe client is required to accept GOWAY responses May cause a sudden increase in traffic on some nodesEach client needs to ensure that data is not lost Traffic will not be particularly balanced in each instance We choose Limit+HPA for these reasons:\nEasy to config and use the proxy and easy to understand based on basic data metrics. No data loss due to broken connection. There is no need for the client to implement any other protocols to prevent data loss, especially when the client is a commercial product. The connection of each node in the proxy cluster does not need to be particularly balanced, as long as the proxy node itself is high-performance. SkyWalking-Satellite We have implemented this Proxy in the SkyWalking-Satellite project. It’s used between Client and SkyWalking OAP, effectively solving the load balancing problem.\nAfter the system is deployed, the Satellite would accept the traffic from the Client, and the Satellite will perceive all the nodes of the OAP through Kubernetes Label Selector or manual configuration, and load balance the traffic to the upstream OAP node.\nAs shown below, a single client still maintains a connection with a single Satellite, Satellite would establish the connection with each OAP, and load balance message to the OAP node.\nWhen scaling Satellite, we need to deploy the SWCK adapter and configure the HPA in Kubernetes. SWCK is a platform for the SkyWalking users, provisions, upgrades, maintains SkyWalking relevant components, and makes them work natively on Kubernetes.\nAfter deployment is finished, the following steps would be performed:\nRead metrics from OAP: HPA requests the SWCK metrics adapter to dynamically read the metrics in the OAP. Scaling the Satellite: Kubernetes HPA senses that the metrics values are in line with expectations, so the Satellite would be scaling automatically. As shown below, use the dotted line to divide the two parts. HPA uses SWCK Adapter to read the metrics in the OAP. When the threshold is met, HPA would scale the Satellite deployment.\nExample In this section, we will demonstrate two cases:\nSkyWalking Scaling: After SkyWalking OAP scaling, the traffic would auto load balancing through Satellite. Satellite Scaling: Satellite’s own traffic load balancing. NOTE: All commands could be accessed through GitHub.\nSkyWalking Scaling We will use the bookinfo application to demonstrate how to integrate Apache SkyWalking 8.9.1 with Apache SkyWalking-Satellite 0.5.0, and observe the service mesh through the Envoy ALS protocol.\nBefore starting, please make sure that you already have a Kubernetes environment.\nInstall Istio Istio provides a very convenient way to configure the Envoy proxy and enable the access log service. The following step:\nInstall the istioctl locally to help manage the Istio mesh. Install Istio into the Kubernetes environment with a demo configuration profile, and enable the Envoy ALS. Transmit the ALS message to the satellite. The satellite we will deploy later. Add the label into the default namespace so Istio could automatically inject Envoy sidecar proxies when you deploy your application later. # install istioctl export ISTIO_VERSION=1.12.0 curl -L https://istio.io/downloadIstio | sh - sudo mv $PWD/istio-$ISTIO_VERSION/bin/istioctl /usr/local/bin/ # install istio istioctl install -y --set profile=demo \\ --set meshConfig.enableEnvoyAccessLogService=true \\ --set meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-system-satellite.skywalking-system:11800 # enbale envoy proxy in default namespace kubectl label namespace default istio-injection=enabled Install SWCK SWCK provides convenience for users to deploy and upgrade SkyWalking related components based on Kubernetes. The automatic scale function of Satellite also mainly relies on SWCK. For more information, you could refer to the official documentation.\n# Install cert-manager kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.yaml # Deploy SWCK mkdir -p skywalking-swck \u0026amp;\u0026amp; cd skywalking-swck wget https://dlcdn.apache.org/skywalking/swck/0.6.1/skywalking-swck-0.6.1-bin.tgz tar -zxvf skywalking-swck-0.6.1-bin.tgz cd config kubectl apply -f operator-bundle.yaml Deploy Apache SkyWalking And Apache SkyWalking-Satellite We have provided a simple script to deploy the skywalking OAP, UI, and Satellite.\n# Create the skywalking components namespace kubectl create namespace skywalking-system kubectl label namespace skywalking-system swck-injection=enabled # Deploy components kubectl apply -f https://raw.githubusercontent.com/mrproliu/sw-satellite-demo-scripts/5821a909b647f7c8f99c70378e197630836f45f7/resources/sw-components.yaml Deploy Bookinfo Application export ISTIO_VERSION=1.12.0 kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml kubectl wait --for=condition=Ready pods --all --timeout=1200s kubectl port-forward service/productpage 9080 Next, please open your browser and visit http://localhost:9080. You should be able to see the Bookinfo application. Refresh the webpage several times to generate enough access logs.\nThen, you can see the topology and metrics of the Bookinfo application on SkyWalking WebUI. At this time, you can see that the Satellite is working!\nDeploy Monitor We need to install OpenTelemetry Collector to collect metrics in OAPs and analyze them.\n# Add OTEL collector kubectl apply -f https://raw.githubusercontent.com/mrproliu/sw-satellite-demo-scripts/5821a909b647f7c8f99c70378e197630836f45f7/resources/otel-collector-oap.yaml kubectl port-forward -n skywalking-system service/skywalking-system-ui 8080:80 Next, please open your browser and visit http://localhost:8080/ and create a new item on the dashboard. The SkyWalking Web UI pictured below shows how the data content is applied.\nScaling OAP Scaling the number of OAPs by deployment.\nkubectl scale --replicas=3 -n skywalking-system deployment/skywalking-system-oap Done! After a period of time, you will see that the number of OAPs becomes 3, and the ALS traffic is balanced to each OAP.\nSatellite Scaling After we have completed the SkyWalking Scaling, we would carry out the Satellite Scaling demo.\nDeploy SWCK HPA SWCK provides an adapter to implement the Kubernetes external metrics to adapt the HPA through reading the metrics in SkyWalking OAP. We expose the metrics service in Satellite to OAP and configure HPA Resource to auto-scaling the Satellite.\nInstall the SWCK adapter into the Kubernetes environment:\nkubectl apply -f skywalking-swck/config/adapter-bundle.yaml Create the HPA resource, and limit each Satellite to handle a maximum of 10 connections:\nkubectl apply -f https://raw.githubusercontent.com/mrproliu/sw-satellite-demo-scripts/5821a909b647f7c8f99c70378e197630836f45f7/resources/satellite-hpa.yaml Then, you could see we have 9 connections in one satellite. One envoy proxy may establish multiple connections to the satellite.\n$ kubectl get HorizontalPodAutoscaler -n skywalking-system NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE hpa-demo Deployment/skywalking-system-satellite 9/10 1 3 1 5m18s Scaling Application The scaling application could establish more connections to the satellite, to verify whether the HPA is in effect.\nkubectl scale --replicas=3 deployment/productpage-v1 deployment/details-v1 Done! By default, Satellite will deploy a single instance and a single instance will only accept 11 connections. HPA resources limit one Satellite to handle 10 connections and use a stabilization window to make Satellite stable scaling up. In this case, we deploy the Bookinfo application in 10+ instances after scaling, which means that 10+ connections will be established to the Satellite.\nSo after HPA resources are running, the Satellite would be automatically scaled up to 2 instances. You can learn about the calculation algorithm of replicas through the official documentation. Run the following command to view the running status:\n$ kubectl get HorizontalPodAutoscaler -n skywalking-system --watch NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE hpa-demo Deployment/skywalking-system-satellite 11/10 1 3 1 3m31s hpa-demo Deployment/skywalking-system-satellite 11/10 1 3 1 4m20s hpa-demo Deployment/skywalking-system-satellite 11/10 1 3 2 4m38s hpa-demo Deployment/skywalking-system-satellite 11/10 1 3 2 5m8s hpa-demo Deployment/skywalking-system-satellite 6/10 1 3 2 5m23s By observing the “number of connections” metric, we would be able to see that when the number of connections of each gRPC exceeds 10 connections, then the satellite automatically scales through the HPA rule. As a result, the connection number is down to normal status (in this example, less than 10)\nswctl metrics linear --name satellite_service_grpc_connect_count --service-name satellite::satellite-service ","excerpt":"\u003ch2 id=\"background\"\u003eBackground\u003c/h2\u003e\n\u003cp\u003eIn the Apache SkyWalking ecosystem, the OAP obtains metrics, traces, logs, and event data …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2022-01-24-scaling-with-apache-skywalking/","title":"Scaling with Apache SkyWalking"},{"body":"SkyWalking Cloud on Kubernetes 0.6.1 is released. Go to downloads page to find release tars.\nBugs Fix could not deploy metrics adapter to GKE ","excerpt":"\u003cp\u003eSkyWalking Cloud on Kubernetes 0.6.1 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eBugs …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cloud-on-kubernetes-0-6-1/","title":"Release Apache SkyWalking Cloud on Kubernetes 0.6.1"},{"body":"随着业务与用户量的持续发展，系统的瓶颈也逐渐出现。尤其在一些节假日、突发的营销活动中，访问量激增可能会导致系统性能下降，甚至造成系统瘫痪。 全链路压测可以很好的帮助我们预先演练高峰流量，从而提前模拟出系统的执行情况，帮助我们预估系统容量。当流量真正来临时，也可以更从容面对。 Apache SkyWalking 联合 Apache APISIX 及 Apache ShardingSphere，三大顶级开源社区通力合作，共同打造生产级可用的全链路压测解决方案，CyborgFlow。\n介绍 CyborgFlow 是一款面向生产级可用的全链路压测解决方案。总共由三个组件组成，如下图所示。\nFlow Gateway: 压测流量网关。当流量到达该组件时，则会将请求认定为压测流量，并将压测流量标识传递至上游服务。 Database Shadow: 数据库中间件。当数据库中间件感知到当前流量为压测流量时，则会将数据库操作路由至影子表中进行操作。 Agent/Dashboard: 分布式监控系统。与业务系统紧密结合，当感知到压测请求后，自动将其标识传递至上游，无需业务代码改造。并且利用分析能力，构建Dashboard来便于查看流量情况。 以此，便覆盖了单个请求的完整生命周期，在网关层构建压测标识，到业务系统透传标识，最终将请求与影子表交互。同时整个流程拥有完整的监控分析。\n原理 依托于三大社区合作，让这一切变得简单易用。下图为全链路压测系统的运行原理，橙色和蓝色分别代表正常流量和压测流量。\nFlow Gateway Flow Gateway 作为压测流量网关，主要负责接收流量，并传递压测流量表示至上游。\n添加 skywalking插件 构建链路入口。 依据 proxy-rewrite插件 将压测流量标识注入到上游的请求头中。 Agent/Dashboard 该组件中则分为两部分内容说明。\nAgent Agent与业务程序拥有相同生命周期，负责压测流量标识在各个业务系统之间传递，并与 Database Shadow 交互。\nSkyWalking Agent通过读取从Flow Gateway传递的压测流量标识，利用 透传协议 将该标识在应用之间传递。 当准备进行数据库调用时，则通过判断是否包含压测流量标识来决定是否SQL调用时追加压测流量标识(/* cyborg-flow: true */)。 当检测到当前请求包含压测流量标识后，将该数据与Trace绑定，用于Dashboard数据分析。 Dashboard Dashboard 用于压测过程进行中的监控数据分析，并最终以图表的方式进行展示。\n接收来自Agent中上报的Trace数据，并依据OAL中的Tag过滤器(.filter(tags contain \u0026quot;cyborg-flow:true\u0026quot;))来生成压测与非压测的指标数据。 利用指标数据便可以在Dashboard中创建图表进行观察。 Database Shadow Database Shadow 作为 Proxy 在业务程序与数据库中间完成数据交互，当检测到压测流量时则会将SQL传递至影子表中处理。\n检测下游传递的数据库语句中是否包含压测流量标识(/* cyborg-flow: true */)，存在时则将SQL交给由用户配置的影子表中处理。 快速上手 下面将带你快速将Cyborg Flow集成至你的项目中。相关组件的下载请至 Github Release 中下载，目前已发布 0.1.0 版本。\n部署 Database Shadow 解压缩cyborg-database-shadow.tar.gz。 将 conf/config-shadow.yaml 文件中的业务数据库与影子数据库配置为自身业务中的配置。 启动 Database Shadow服务，启动脚本位于bin/start.sh中。 如需了解更详细的部署参数配置，请参考 官方文档 。\n部署 Cyborg Dashboard 解压缩cyborg-dashboard.tar.gz。 启动后端与UI界面服务，用于链路数据解析与界面展示，启动脚本位于bin/startup.sh中。 接下来就可以通过打开浏览器并访问http://localhost:8080/，此页面为Cyborg Dashboard界面，由于目前尚未部署任何业务程序，所以暂无任何数据。 如需了解更详细的部署参数配置，请参考 后端服务 与 UI界面服务 的安装文档。\n部署 Cyborg Agent 到业务程序中 解压缩cyborg-agent.tar.gz. 修改config/agent.config中的collector.backend_service为 Cyborg Dashboard 中后端地址(默认为11800端口)，用于将监控数据上报至 Cyborg Dashboard 。 修改业务程序中与数据库的链接，将其更改为 Database Shadow 中的配置。默认访问端口为3307，用户名密码均为root。 当程序启动时，增加该参数到启动命令中：-jar path/to/cyborg-agent/skywalking-agent.jar。 如需了解更详细的部署参数配置，请参考 Agent安装文档 。\n部署 Flow Gateway 参考 Flow Gateway 快速开始 进行下载 Apache APISIX 并配置相关插件。 基于 APISIX 创建路由文档 进行路由创建。 完成！ 最后，通过Flow Gateway访问业务系统资源，便完成了一次压测流量请求。\n压测流量最终访问至影子表进行数据操作。 如下图所示，通过观察 Cyborg Dashboard 便可以得知压测与非压测请求的执行情况。 总结 在本文中，我们详细介绍了Cyborg Flow中的各个组件的功能、原理，最终搭配快速上手来快速将该系统与自己的业务系统结合。 如果在使用中有任何问题，欢迎来共同讨论。\n","excerpt":"\u003cp\u003e随着业务与用户量的持续发展，系统的瓶颈也逐渐出现。尤其在一些节假日、突发的营销活动中，访问量激增可能会导致系统性能下降，甚至造成系统瘫痪。\n全链路压测可以很好的帮助我们预先演练高峰流量，从而提前模拟出 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2022-01-18-cyborg-flow/","title":"Cyborg Flow X SkyWalking: 生产环境全链路压测"},{"body":"SkyWalking Cloud on Kubernetes 0.6.0 is released. Go to downloads page to find release tars.\nFeatures Add the Satellite CRD, webhooks and controller Bugs Update release images to set numeric user id Fix the satellite config not support number error Use env JAVA_TOOL_OPTIONS to replace AGENT_OPTS Chores Add stabilization windows feature in satellite HPA documentation ","excerpt":"\u003cp\u003eSkyWalking Cloud on Kubernetes 0.6.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars. …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cloud-on-kubernetes-0-6-0/","title":"Release Apache SkyWalking Cloud on Kubernetes 0.6.0"},{"body":"SkyWalking Kong Agent 0.2.0 is released. Go to downloads page to find release tars.\nEstablish the SkyWalking Kong Agent. ","excerpt":"\u003cp\u003eSkyWalking Kong Agent 0.2.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eEstablish the …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-kong-0-2-0/","title":"Release Apache SkyWalking Kong 0.2.0"},{"body":"SkyWalking Satellite 0.5.0 is released. Go to downloads page to find release tars.\nFeatures Make the gRPC client client_pem_path and client_key_path as an optional config. Remove prometheus-server sharing server plugin. Support let the telemetry metrics export to prometheus or metricsService. Add the resource limit when gRPC server accept connection. Bug Fixes Fix the gRPC server enable TLS failure. Fix the native meter protocol message load balance bug. Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Satellite 0.5.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eMake …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-satellite-0-5-0/","title":"Release Apache SkyWalking Satellite 0.5.0"},{"body":"SkyWalking LUA Nginx 0.6.0 is released. Go to downloads page to find release tars.\nfix: skywalking_tracer:finish() will not be called in some case such as upstream timeout. ","excerpt":"\u003cp\u003eSkyWalking LUA Nginx 0.6.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003efix: …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-lua-nginx-0.6.0/","title":"Release Apache SkyWalking LUA Nginx 0.6.0"},{"body":"\nChaos Mesh is an open-source cloud-native chaos engineering platform. You can use Chaos Mesh to conveniently inject failures and simulate abnormalities that might occur in reality, so you can identify potential problems in your system. Chaos Mesh also offers a Chaos Dashboard which allows you to monitor the status of a chaos experiment. However, this dashboard cannot let you observe how the failures in the experiment impact the service performance of applications. This hinders us from further testing our systems and finding potential problems.\nApache SkyWalking is an open-source application performance monitor (APM), specially designed to monitor, track, and diagnose cloud native, container-based distributed systems. It collects events that occur and then displays them on its dashboard, allowing you to observe directly the type and number of events that have occurred in your system and how different events impact the service performance.\nWhen you use SkyWalking and Chaos Mesh together during chaos experiments, you can observe how different failures impact the service performance.\nThis tutorial will show you how to configure SkyWalking and Chaos Mesh. You’ll also learn how to leverage the two systems to monitor events and observe in real time how chaos experiments impact applications’ service performance.\nPreparation Before you start to use SkyWalking and Chaos Mesh, you have to:\nSet up a SkyWalking cluster according to the SkyWalking configuration guide. Deploy Chao Mesh using Helm. Install JMeter or other Java testing tools (to increase service loads). Configure SkyWalking and Chaos Mesh according to this guide if you just want to run a demo. Now, you are fully prepared, and we can cut to the chase.\nStep 1: Access the SkyWalking cluster After you install the SkyWalking cluster, you can access its user interface (UI). However, no service is running at this point, so before you start monitoring, you have to add one and set the agents.\nIn this tutorial, we take Spring Boot, a lightweight microservice framework, as an example to build a simplified demo environment.\nCreate a SkyWalking demo in Spring Boot by referring to this document. Execute the command kubectl apply -f demo-deployment.yaml -n skywalking to deploy the demo. After you finish deployment, you can observe the real-time monitoring results at the SkyWalking UI.\nNote: Spring Boot and SkyWalking have the same default port number: 8080. Be careful when you configure the port forwarding; otherise, you may have port conflicts. For example, you can set Spring Boot’s port to 8079 by using a command like kubectl port-forward svc/spring-boot-skywalking-demo 8079:8080 -n skywalking to avoid conflicts.\nStep 2: Deploy SkyWalking Kubernetes Event Exporter SkyWalking Kubernetes Event Exporter is able to watch, filter, and send Kubernetes events into the SkyWalking backend. SkyWalking then associates the events with the system metrics and displays an overview about when and how the metrics are affected by the events.\nIf you want to deploy SkyWalking Kubernetes Event Explorer with one line of commands, refer to this document to create configuration files in YAML format and then customize the parameters in the filters and exporters. Now, you can use the command kubectl apply to deploy SkyWalking Kubernetes Event Explorer.\nStep 3: Use JMeter to increase service loads To better observe the change in service performance, you need to increase the service loads on Spring Boot. In this tutorial, we use JMeter, a widely adopted Java testing tool, to increase the service loads.\nPerform a stress test on localhost:8079 using JMeter and add five threads to continuously increase the service loads.\nOpen the SkyWalking Dashboard. You can see that the access rate is 100%, and that the service loads reach about 5,300 calls per minute (CPM).\nStep 4: Inject failures via Chaos Mesh and observe results After you finish the three steps above, you can use the Chaos Dashboard to simulate stress scenarios and observe the change in service performance during chaos experiments.\nThe following sections describe how service performance varies under the stress of three chaos conditions:\nCPU load: 10%; memory load: 128 MB\nThe first chaos experiment simulates low CPU usage. To display when a chaos experiment starts and ends, click the switching button on the right side of the dashboard. To learn whether the experiment is Applied to the system or Recovered from the system, move your cursor onto the short, green line.\nDuring the time period between the two short, green lines, the service load decreases to 4,929 CPM, but returns to normal after the chaos experiment ends.\nCPU load: 50%; memory load: 128 MB\nWhen the application’s CPU load increases to 50%, the service load decreases to 4,307 CPM.\nCPU load: 100%; memory load: 128 MB\nWhen the CPU usage is at 100%, the service load decreases to only 40% of what it would be if no chaos experiments were taking place.\nBecause the process scheduling under the Linux system does not allow a process to occupy the CPU all the time, the deployed Spring Boot Demo can still handle 40% of the access requests even in the extreme case of a full CPU load.\nSummary By combining SkyWalking and Chaos Mesh, you can clearly observe when and to what extent chaos experiments affect application service performance. This combination of tools lets you observe the service performance in various extreme conditions, thus boosting your confidence in your services.\nChaos Mesh has grown a lot in 2021 thanks to the unremitting efforts of all PingCAP engineers and community contributors. In order to continue to upgrade our support for our wide variety of users and learn more about users’ experience in Chaos Engineering, we’d like to invite you to take this survey and give us your valuable feedback.\nIf you want to know more about Chaos Mesh, you’re welcome to join the Chaos Mesh community on GitHub or our Slack discussions (#project-chaos-mesh). If you find any bugs or missing features when using Chaos Mesh, you can submit your pull requests or issues to our GitHub repository.\n","excerpt":"\u003cp\u003e\u003cimg src=\"chaos-mesh-skywalking-banner.png\" alt=\"Chaos Mesh + SkyWalking: Better Observability for Chaos Engineering\"\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/chaos-mesh/chaos-mesh\"\u003eChaos Mesh\u003c/a\u003e is an open-source cloud-native \u003ca href=\"https://en.wikipedia.org/wiki/Chaos_engineering\"\u003echaos engineering\u003c/a\u003e platform. You can use Chaos Mesh to …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2021-12-21-better-observability-for-chaos-engineering/","title":"Chaos Mesh + SkyWalking: Better Observability for Chaos Engineering"},{"body":"SkyWalking Cloud on Kubernetes 0.5.0 is released. Go to downloads page to find release tars.\nFeatures Add E2E test cases to verify OAPServer, UI, Java agent and Storage components. Bugs Fix operator role patch issues Fix invalid CSR signername Fix bug in the configmap controller Chores Bump up KubeBuilder to V3 Bump up metric adapter server to v1.21.0 Split mono-project to two independent projects ","excerpt":"\u003cp\u003eSkyWalking Cloud on Kubernetes 0.5.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars. …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cloud-on-kubernetes-0-5-0/","title":"Release Apache SkyWalking Cloud on Kubernetes 0.5.0"},{"body":"We Can integrate Skywalking to Java Application by Java Agent TEC.， In typical application, the system runs Java Web applications at the backend of the load balancer, and the most commonly used load balancer is nginx. What should we do if we want to bring it under surveillance? Fortunately, skywalking has provided Nginx agent。 During the integration process, it is found that the examples on the official website only support openresty. For openresty, common modules such as luajit and Lua nginx module have been integrated. Adding skywalking related configurations according to the examples on the official website can take effect. However, when configured for nginx startup, many errors will be reported. We may not want to change a load balancer (nginx to openresty) in order to use skywalking. Therefore, we must solve the integration problem between skywalking and nginx.\nNote: openresty is a high-performance web development platform based on nginx + Lua, which solves the short board that is not easy to program in nginx.\nBased on Skywalking-8.7.0 and Nginx-1.20.1\nUpgrade of nginx: The agent plug-in of nginx is written based on Lua, so nginx needs to add support for Lua, Lua nginx module It just provides this function. The Lua nginx module depends on luajit Therefore, first we need to install luajit. In the environment, it is best to choose version 2.1.\nFor nginx, you need to compile the necessary modules yourself. It depends on the following two modules:\nlua-nginx-module The version is lua-nginx-module-0.10.21rc1\nngx_devel_kit The version using ngx_devel_kit-0.3.1\nCompile nginx parameters\nconfigure arguments: --add-module=/path/to/ngx_devel_kit-0.3.1 --add-module=/path/to/lua-nginx-module-0.10.21rc1 --with-ld-opt=-Wl,-rpath,/usr/local/LuaJIT/lib The following is for skywalking-nginx-lua-0.3.0 and 0.3.0+ are described separately.\nskywalking-nginx-lua-0.3.0 After testing, skywalking-nginx-lua-0.3.0 requires the following Lua related modules\nlua-resty-core https://github.com/openresty/lua-resty-core lua-resty-lrucache https://github.com/openresty/lua-resty-lrucache lua-cjson https://github.com/openresty/lua-cjson The dependent Lua modules are as follows:\nlua_package_path \u0026#34;/path/to/lua-resty-core/lua-resty-core-master/lib/?.lua;/path/to/lua-resty-lrucache-0.11/lib/?.lua;/path/to/skywalking-nginx-lua-0.3.0/lib/?.lua;;\u0026#34;; In the process of make \u0026amp; \u0026amp; make install, Lua cjson needs to pay attention to:\nModify a path in makefile\nLUA_INCLUDE_DIR ?= /usr/local/LuaJIT/include/luajit-2.0\nReference: https://blog.csdn.net/ymeputer/article/details/50146143 skywalking-nginx-lua-0.3.0+ For skywalking-nginx-lua-0.3.0+, tablepool support needs to be added, but it seems that cjson is not required\nlua-resty-core https://github.com/openresty/lua-resty-core lua-resty-lrucache https://github.com/openresty/lua-resty-lrucache lua-tablepool https://github.com/openresty/lua-tablepool lua_ package_ path \u0026#34;/path/to/lua-resty-core/lua-resty-core-master/lib/?.lua;/path/to/lua-resty-lrucache-0.11/lib/?.lua;/path/to/lua-tablepool-master/lib/?.lua;/path/to/skywalking-nginx-lua-master/lib/?.lua;;\u0026#34;; tablepool introduces two APIs according to its official documents table new and table. Clear requires luajit2.1, there is a paragraph in the skywalking-nginx-lua document that says you can use \u0026lsquo;require (\u0026ldquo;skywalking. Util\u0026rdquo;) disable_ Tablepool() ` disable tablepool\nWhen you start nginx, you will be prompted to install openresty\u0026rsquo;s own [luajit version]（ https://github.com/openresty/luajit2 )\ndetected a LuaJIT version which is not OpenResty\u0026#39;s; many optimizations will be disabled and performance will be compromised (see https://github.com/openresty/luajit2 for OpenResty\u0026#39;s LuaJIT or, even better, consider using the OpenResty releases from https://openresty.org/en/download.html ) here is successful configuration:\nhttp { lua_package_path \u0026#34;/path/to/lua-resty-core/lua-resty-core-master/lib/?.lua;/path/to/lua-resty-lrucache-0.11/lib/?.lua;/path/to/lua-tablepool-master/lib/?.lua;/path/to/skywalking-nginx-lua-master/lib/?.lua;;\u0026#34;; # Buffer represents the register inform and the queue of the finished segment lua_shared_dict tracing_buffer 100m; # Init is the timer setter and keeper # Setup an infinite loop timer to do register and trace report. init_worker_by_lua_block { local metadata_buffer = ngx.shared.tracing_buffer -- Set service name metadata_buffer:set(\u0026#39;serviceName\u0026#39;, \u0026#39;User Service Name\u0026#39;) -- Instance means the number of Nginx deployment, does not mean the worker instances metadata_buffer:set(\u0026#39;serviceInstanceName\u0026#39;, \u0026#39;User Service Instance Name\u0026#39;) -- type \u0026#39;boolean\u0026#39;, mark the entrySpan include host/domain metadata_buffer:set(\u0026#39;includeHostInEntrySpan\u0026#39;, false) -- set random seed require(\u0026#34;skywalking.util\u0026#34;).set_randomseed() require(\u0026#34;skywalking.client\u0026#34;):startBackendTimer(\u0026#34;http://127.0.0.1:12800\u0026#34;) -- If there is a bug of this `tablepool` implementation, we can -- disable it in this way -- require(\u0026#34;skywalking.util\u0026#34;).disable_tablepool() skywalking_tracer = require(\u0026#34;skywalking.tracer\u0026#34;) } server { listen 8090; location /ingress { default_type text/html; rewrite_by_lua_block { ------------------------------------------------------ -- NOTICE, this should be changed manually -- This variable represents the upstream logic address -- Please set them as service logic name or DNS name -- -- Currently, we can not have the upstream real network address ------------------------------------------------------ skywalking_tracer:start(\u0026#34;upstream service\u0026#34;) -- If you want correlation custom data to the downstream service -- skywalking_tracer:start(\u0026#34;upstream service\u0026#34;, {custom = \u0026#34;custom_value\u0026#34;}) } -- Target upstream service proxy_pass http://127.0.0.1:8080/backend; body_filter_by_lua_block { if ngx.arg[2] then skywalking_tracer:finish() end } log_by_lua_block { skywalking_tracer:prepareForReport() } } } } Original post：https://www.cnblogs.com/kebibuluan/p/14440228.html\n","excerpt":"\u003cp\u003eWe Can integrate Skywalking to Java Application by Java Agent TEC.， In typical application, the …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2021-12-13-skywalking-nginx-agent-integration/","title":"How to integrate skywalking-nginx-lua to Nginx?"},{"body":"SkyWalking 8.9.1 is released. Go to downloads page to find release tars.\nChanges by Version\nProject Upgrade log4j2 to 2.15.0 for CVE-2021-44228. This CVE only effects on JDK versions below 6u211, 7u201, 8u191 and 11.0.1 according to the post. Notice, using JVM option -Dlog4j2.formatMsgNoLookups=true also avoids CVE if your JRE opened JNDI in default. ","excerpt":"\u003cp\u003eSkyWalking 8.9.1 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cp\u003eChanges by Version\u003c/p\u003e\n\u003ch4 id=\"project\"\u003eProject …\u003c/h4\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-8-9-1/","title":"Release Apache SkyWalking APM 8.9.1"},{"body":"In the field of observability, the three main directions of data collection and analysis, Metrics, Logger and Tracing, are usually used to achieve insight into the operational status of applications.\nApache APISIX has integrated Apache SkyWaling Tracing capabilities as early as version 1.4, with features such as error logging and access log collection added in subsequent versions. Now with Apache SkyWalking\u0026rsquo;s support for Metrics, it enables Apache APISIX to implement a one-stop observable solution in integrated mode, covering both logging, metrics and call tracing.\nFeature Development Background Those of you who are familiar with Apache APISIX should know that Apache APISIX produces two types of logs during operation, namely the access log and the error log.\nAccess logs record detailed information about each request and are logs generated within the scope of the request, so they can be directly associated with Tracing. Error logs, on the other hand, are Apache APISIX runtime output log messages, which are application-wide logs, but cannot be 100% associated with requests.\nAt present, Apache APISIX provides very rich log processing plug-ins, including TCP/HTTP/Kafka and other collection and reporting plug-ins, but they are weakly associated with Tracing. Take Apache SkyWalking as an example. We extract the SkyWalking Tracing Conetxt Header from the log records of Apache APISIX and export it to the file system, and then use the log processing framework (fluentbit) to convert the logs into a log format acceptable to SkyWalking. The Tracing Context is then parsed and extracted to obtain the Tracing ID to establish a connection with the Trace.\nObviously, the above way of handling the process is tedious and complicated, and requires additional conversion of log formats. For this reason, in PR#5500 we have implemented the Apache SkyWalking access log into the Apache APISIX plug-in ecosystem to make it easier for users to collect and process logs using Apache SkyWalking in Apache APISIX.\nIntroduction of the New Plugins SkyWalking Logger Pulgin The SkyWalking Logger plugin parses the SkyWalking Tracing Context Header and prints the relevant Tracing Context information to the log, thus enabling the log to be associated with the call chain.\nBy using this plug-in, Apache APISIX can get the SkyWalking Tracing Context and associate it with Tracing even if the SkyWalking Tracing plug-in is not turned on, if Apache SkyWalking is already integrated downstream.\nThe above Content is the log content, where the Apache APISIX metadata configuration is used to collect request-related information. You can later modify the Log Format to customize the log content by Plugin Metadata, please refer to the official documentation.\nHow to Use When using this plugin, since the SkyWalking plugin is \u0026ldquo;not enabled\u0026rdquo; by default, you need to manually modify the plugins section in the conf/default-apisix.yaml file to enable the plugin.\nplugins: ... - error-log-logger ... Then you can use the SkyWalking Tracing plug-in to get the tracing data directly, so you can verify that the Logging plug-in-related features are enabled and working properly.\nStep 1: Create a route Next, create a route and bind the SkyWalking Tracing plugin and the SkyWalking Logging plugin. More details of the plugin configuration can be found in the official Apache APISIX documentation.\ncurl -X PUT \u0026#39;http://192.168.0.108:9080/apisix/admin/routes/1001\u0026#39; \\ -H \u0026#39;X-API-KEY: edd1c9f034335f136f87ad84b625c8f1\u0026#39; \\ -H \u0026#39;Content-Type: application/json\u0026#39; \\ -d \u0026#39;{ \u0026#34;uri\u0026#34;: \u0026#34;/get\u0026#34;, \u0026#34;plugins\u0026#34;: { \u0026#34;skywalking\u0026#34;: { \u0026#34;sample_ratio\u0026#34;: 1 }, \u0026#34;skywalking-logger\u0026#34;: { \u0026#34;endpoint_addr\u0026#34;: \u0026#34;http://127.0.0.1:12800\u0026#34; } }, \u0026#34;upstream\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;roundrobin\u0026#34;, \u0026#34;nodes\u0026#34;: { \u0026#34;httpbin.org:80\u0026#34;: 1 } } }\u0026#39; Step 2: Log Processing On the Apache SkyWalking side, you can use LAL (Logger Analysis Language) scripts for log processing, such as Tag extraction, SkyWalking metadata correction, and so on.\nThe main purpose of Tag extraction here is to facilitate subsequent retrieval and to add dependencies to the Metrics statistics. The following code can be used to configure the SkyWalking LAL script to complete the Tag extraction. For more information on how to use the SkyWalking LAL script, please refer to the official Apache SkyWalking documentation.\n# The default LAL script to save all logs, behaving like the versions before 8.5.0. rules: - name: default dsl: | filter { json { abortOnFailure false } extractor { tag routeId: parsed.route_id tag upstream: parsed.upstream tag clientIp: parsed.client_ip tag latency: parsed.latency } sink { } } After configuring the above LAL script in SkyWalking OAP Server the following log will be displayed.\nDetails of the expanded log are as follows.\nAs you can see from the above, displaying routeId, upstream and clientIp as key-value pairs is much easier than searching directly in the log body. This is because the Tag format not only supports log display format and search, but also generates information such as Metrics using MAL statistics.\nSkyWalking Error Logger Plugin The error-log-logger plug-in now supports the SkyWalking log format, and you can now use the http-error-log plug-in to quickly connect Apache APISIX error logs to Apache SkyWalking. Currently, error logs do not have access to SkyWalking Tracing Context information, and therefore cannot be directly associated with SkyWalking Tracing.\nThe main reason for the error log to be integrated into SkyWalking is to centralize the Apache APISIX log data and to make it easier to view all observable data within SkyWalking.\nHow to Use Since the error-log-logger plugin is \u0026ldquo;not enabled\u0026rdquo; by default, you still need to enable the plugin in the way mentioned above.\nplugins: ... - error-log-logger ... Step 1: Bind the route After enabling, you need to bind the plugin to routes or global rules. Here we take \u0026ldquo;bind routes\u0026rdquo; as an example.\ncurl -X PUT \u0026#39;http://192.168.0.108:9080/apisix/admin/plugin_metadata/error-log-logger\u0026#39; \\ -H \u0026#39;X-API-KEY: edd1c9f034335f136f87ad84b625c8f1\u0026#39; \\ -H \u0026#39;Content-Type: application/json\u0026#39; \\ -d \u0026#39;{ \u0026#34;inactive_timeout\u0026#34;: 10, \u0026#34;level\u0026#34;: \u0026#34;ERROR\u0026#34;, \u0026#34;skywalking\u0026#34;: { \u0026#34;endpoint_addr\u0026#34;: \u0026#34;http://127.0.0.1:12800/v3/logs\u0026#34; } }\u0026#39; Note that the endpoint_addr is the SkyWalking OAP Server address and needs to have the URI (i.e. /v3/logs).\nStep 2: LAL Processing In much the same way as the Access Log processing, the logs are also processed by LAL when they reach SkyWalking OAP Server. Therefore, we can still use the SkyWalking LAL script to analyze and process the log messages.\nIt is important to note that the Error Log message body is in text format. If you are extracting tags, you will need to use regular expressions to do this. Unlike Access Log, which handles the message body in a slightly different way, Acces Log uses JSON format and can directly reference the fields of the JSON object using JSON parsing, but the rest of the process is largely the same.\nTags can also be used to optimize the display and retrieval for subsequent metrics calculations using SkyWalking MAL.\nrules: - name: apisix-errlog dsl: | filter { text { regexp \u0026#34;(?\u0026lt;datetime\u0026gt;\\\\d{4}/\\\\d{2}/\\\\d{2} \\\\d{2}:\\\\d{2}:\\\\d{2}) \\\\[(?\u0026lt;level\u0026gt;\\\\w+)\\\\] \\\\d+\\\\#\\\\d+:( \\\\*\\\\d+ \\\\[(?\u0026lt;module\u0026gt;\\\\w+)\\\\] (?\u0026lt;position\u0026gt;.*\\\\.lua:\\\\d+): (?\u0026lt;function\u0026gt;\\\\w+\\\\(\\\\)):)* (?\u0026lt;msg\u0026gt;.+)\u0026#34; } extractor { tag level: parsed.level if (parsed?.module) { tag module: parsed.module tag position: parsed.position tag function: parsed.function } } sink { } } After the LAL script used by SkyWalking OAP Server, some of the Tags will be extracted from the logs, as shown below.\nSummary This article introduces two logging plug-ins for Apache APISIX that integrate with SkyWalking to provide a more convenient operation and environment for logging in Apache APISIX afterwards.\nWe hope that through this article, you will have a fuller understanding of the new features and be able to use Apache APISIX for centralized management of observable data more conveniently in the future.\n","excerpt":"\u003cp\u003eIn the field of observability, the three main directions of data collection and analysis, Metrics, …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2021-12-08-apisix-integrate-skywalking-plugin/apisix-integrate-skywalking-plugin/","title":"Apache APISIX Integrates with SkyWalking to Create a Full Range of Log Processing"},{"body":"This document is one of the outcomes of Apache IoTDB - Apache SkyWalking Adapter in Summer 2021 of Open Source Promotion Plan. The design and development work is under the guidance of @jixuan1989 from IoTDB and @wu-sheng from SkyWalking. Thanks for their guidance and the help from community.\nStart with SkyWalking Showcase Before using SkyWalking Showcase to quick start with IoTDB, please ensure your have make installed and Docker daemon running.\nPlease run the command below.\ngit clone https://github.com/LIU-WEI-git/skywalking-showcase.git cd skywalking-showcase make deploy.docker FEATURE_FLAGS=single-node.iotdb,agent The former variable single-node.iotdb will deploy only one single node of SkyWalking OAP-v8.9.0, and SkyWalking RocketBot UI-v8.9.0, IoTDB-v0.12.3 as storage. The latter variable agent will deploy micro-services with SkyWalking agent enabled, which include agents for Java, NodeJS server, browser, Python.\nThese shell command maybe take a long while. After pulling and running docker image, please visit http://localhost:9999/. Then you will see the SkyWalking UI and data from OAP backend.\nIf you want to use more functions of SkyWalking Showcase, please visit its official document and clone official repository.\nStart Manually If you want to download and run IoTDB and SkyWalking manually, here is the guidance.\nInstall and Run IoTDB Apache IoTDB (Database for Internet of Things) is an IoT native database with high performance for data management and analysis, deployable on the edge and the cloud. It is a time-series database storage option for SkyWalking now. Please ensure your IoTDB server version \u0026gt;= 0.12.3 and a single node version is sufficient. For more installation details, please see official document: IoTDB Quick Start and IoTDB Download Page. You could download it from Docker Hub as well.\nThere is some connection tools for IoTDB\nCommand Line Interface(CLI)\nIf iotdb-cli connects successfully, you will see _____ _________ ______ ______ |_ _| | _ _ ||_ _ `.|_ _ \\ | | .--.|_/ | | \\_| | | `. \\ | |_) | | | / .\u0026#39;`\\ \\ | | | | | | | __\u0026#39;. _| |_| \\__. | _| |_ _| |_.\u0026#39; /_| |__) | |_____|\u0026#39;.__.\u0026#39; |_____| |______.\u0026#39;|_______/ version x.x.x IoTDB\u0026gt; login successfully IoTDB\u0026gt; IoTDB-Grafana\nIoTDB-Grafana is a connector which we developed to show time series data in IoTDB by reading data from IoTDB and sends to Grafana. Zeppelin-IoTDB\nYou could enable Zeppelin to operate IoTDB via SQL. For more ecosystem integration, please visit official documents.\nWe will use iotdb-cli in the next examples.\nRun SkyWalking OAP Server There are some SkyWalking official documents which will help you start. Please ensure your SkyWalking version \u0026gt;= 8.9.0. We recommend you download SkyWalking OAP distributions from its official download page or pull docker images.\nSkyWalking Download Page SkyWalking Backend Setup SkyWalking UI Setup Before starting SkyWalking backend, please edit /config/application.yml, set storage.selector: ${SW_STORAGE:iotdb} or set environment variable SW_STORAGE=iotdb. All config options about IoTDB is following, please edit it or not according to your local environment:\nstorage: selector: ${SW_STORAGE:iotdb} iotdb: host: ${SW_STORAGE_IOTDB_HOST:127.0.0.1} rpcPort: ${SW_STORAGE_IOTDB_RPC_PORT:6667} username: ${SW_STORAGE_IOTDB_USERNAME:root} password: ${SW_STORAGE_IOTDB_PASSWORD:root} storageGroup: ${SW_STORAGE_IOTDB_STORAGE_GROUP:root.skywalking} sessionPoolSize: ${SW_STORAGE_IOTDB_SESSIONPOOL_SIZE:16} fetchTaskLogMaxSize: ${SW_STORAGE_IOTDB_FETCH_TASK_LOG_MAX_SIZE:1000} # the max number of fetch task log in a request Visit IoTDB Server and Query SkyWalking Data There are some official document about data model and IoTDB-SQL language:\nData Model and Terminology DDL (Data Definition Language) DML (Data Manipulation Language) Maintenance Command Example Model and Insert SQL Before giving any example, we set time display type as long (CLI: set time_display_type=long).\nIn our design, we choose id, entity_id, node_type, service_id, service_group, trace_id as indexes and fix their appearance order. The value of these indexed fields store in the path with double quotation mark wrapping, just like \u0026quot;value\u0026quot;.\nThere is a model named service_traffic with fields id, time_bucket, name, node_type, service_group. In order to see its data, we could use a query SQL: select * from root.skywalking.service_traffic align by device. root.skywalking is the default storage group and align by device could return a more friendly result. The query result is following:\nTime Device name 1637919540000 root.skywalking.service_traffic.\u0026ldquo;YXBwbGljYXRpb24tZGVtbw==.1\u0026rdquo;.\u0026ldquo;0\u0026rdquo;.\u0026quot;\u0026quot; application-demo 1637919600000 root.skywalking.service_traffic.\u0026ldquo;YXBwbGljYXRpb24tZGVtby1teXNxbA==.1\u0026rdquo;.\u0026ldquo;0\u0026rdquo;.\u0026quot;\u0026quot; application-demo-mysql Another example model is service_cpm which has fields id, service_id, total, value. Query its data with select * from root.skywalking.service_cpm align by device. The result is following:\nTime Device total value 1637919540000 root.skywalking.service_cpm.\u0026ldquo;202111261739_YXBwbGljYXRpb24tZGVtbw==.1\u0026rdquo;.\u0026ldquo;YXBwbGljYXRpb24tZGVtbw==.1\u0026rdquo; 2 2 1637919600000 root.skywalking.service_cpm.\u0026ldquo;202111261740_YXBwbGljYXRpb24tZGVtby1teXNxbA==.1\u0026rdquo;.\u0026ldquo;YXBwbGljYXRpb24tZGVtby1teXNxbA==.1\u0026rdquo; 1 1 1637917200000 root.skywalking.service_cpm.\u0026ldquo;2021112617_YXBwbGljYXRpb24tZGVtbw==.1\u0026rdquo;.\u0026ldquo;YXBwbGljYXRpb24tZGVtbw==.1\u0026rdquo; 2 0 For the first data of service_traffic, the mapping between fields and values is following. Notice, all time_bucket are converted to timestamp(also named time in IoTDB) and the value of all indexed fields are stored in the Device path.\nField Value id(indexed) YXBwbGljYXRpb24tZGVtbw==.1 time(converted from time_bucket) 1637919540000 name application-demo node_type(indexed) 0 service_group(indexed) (empty string) You could use the SQL below to insert example data.\ncreate storage group root.skywalking insert into root.skywalking.service_traffic.\u0026#34;YXBwbGljYXRpb24tZGVtbw==.1\u0026#34;.\u0026#34;0\u0026#34;.\u0026#34;\u0026#34;(timestamp, name) values(1637919540000, \u0026#34;application-demo\u0026#34;) insert into root.skywalking.service_traffic.\u0026#34;YXBwbGljYXRpb24tZGVtby1teXNxbA==.1\u0026#34;.\u0026#34;0\u0026#34;.\u0026#34;\u0026#34;(timestamp, name) values(1637919600000, \u0026#34;application-demo-mysql\u0026#34;) insert into root.skywalking.service_cpm.\u0026#34;202111261739_YXBwbGljYXRpb24tZGVtbw==.1\u0026#34;.\u0026#34;YXBwbGljYXRpb24tZGVtbw==.1\u0026#34;(timestamp, total, value) values(1637919540000, 2, 2) insert into root.skywalking.service_cpm.\u0026#34;202111261740_YXBwbGljYXRpb24tZGVtby1teXNxbA==.1\u0026#34;.\u0026#34;YXBwbGljYXRpb24tZGVtby1teXNxbA==.1\u0026#34;(timestamp, total, value) values(1637919600000, 1, 1) insert into root.skywalking.service_cpm.\u0026#34;2021112617_YXBwbGljYXRpb24tZGVtbw==.1\u0026#34;.\u0026#34;YXBwbGljYXRpb24tZGVtbw==.1\u0026#34;(timestamp, total, value) values(1637917200000, 2, 0) Query SQL Now, let\u0026rsquo;s show some query examples.\nFilter Query\nIf you want to query name field of service_traffic, the query SQL is select name from root.skywalking.service_traffic align by device. If you want to query service_traffic with id = YXBwbGljYXRpb24tZGVtbw==.1, the query SQL is select * from root.skywalking.service_traffic.\u0026quot;YXBwbGljYXRpb24tZGVtbw==.1\u0026quot; align by device. If you want to query service_traffic with name = application-demo, the query SQL is select * from root.skywalking.service_traffic where name = \u0026quot;application-demo\u0026quot; align by device. Combining the above three, the query SQL is select name from root.skywalking.service_traffic.\u0026quot;YXBwbGljYXRpb24tZGVtbw==.1\u0026quot; where name = \u0026quot;application-demo\u0026quot; align by device. Fuzzy Query\nIf you want to query service_traffic with name contains application, the query SQL is select * from root.skywalking.service_traffic.*.*.* where name like '%application%' align by device. Aggregate Query\nIoTDB only supports group by time and group by level. The former please refer to Down-Frequency Aggregate Query and the latter please refer to Aggregation By Level. Here is an example about group by level: select sum(total) from root.skywalking.service_cpm.*.* group by level = 3. We couldn\u0026rsquo;t get a expected result since our design make the data of one model spread across multiple devices. So we don\u0026rsquo;t recommend using group by level to query SkyWalking backend data. You could refer to the Discussion #3907 in IoTDB community for more details.\nSort Query\nIoTDB only supports order by time, but we could use its select function which contains top_k and bottom_k to get top/bottom k data. For example, select top_k(total, \u0026quot;k\u0026quot;=\u0026quot;3\u0026quot;) from root.skywalking.service_cpm.*.*. We don\u0026rsquo;t recommend using this to query SkyWalking backend data since its result is not friendly. You could refer to the Discussion #3888 in IoTDB community for more details.\nPagination Query\nWe could use limit and offset to paginate the query result. Please refer to Row and Column Control over Query Results.\nDelete\nDelete storage group: delete storage group root.skywalking Delete timeseries: delete timeseries root.skywalking.service_cpm.*.*.total delete timeseries root.skywalking.service_cpm.\u0026quot;202111261739_YXBwbGljYXRpb24tZGVtbw==.1\u0026quot;.\u0026quot;YXBwbGljYXRpb24tZGVtbw==.1\u0026quot;.total Delete data: delete from root.skywalking.service_traffic delete from root.skywalking.service_traffic where time \u0026lt; 1637919540000 ","excerpt":"\u003cp\u003eThis document is one of the outcomes of \u003ca href=\"https://summer.iscas.ac.cn/#/org/prodetail/210070771\"\u003eApache IoTDB - Apache SkyWalking Adapter\u003c/a\u003e in \u003ca href=\"https://summer.iscas.ac.cn/#/homepage\"\u003eSummer 2021 of …\u003c/a\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2021-12-08-application-guide-of-iotdb-storage-option/","title":"The Application Guide of Apache IoTDB Storage Option"},{"body":"Non-breaking breakpoints are breakpoints specifically designed for live production environments. With non-breaking breakpoints, reproducing production bugs locally or in staging is conveniently replaced with capturing them directly in production.\nLike regular breakpoints, non-breaking breakpoints can be:\nplaced almost anywhere added and removed at will set to fire on specific conditions expose internal application state persist as long as desired (even between application reboots) The last feature is especially useful given non-breaking breakpoints can be left in production for days, weeks, and even months at a time while waiting to capture behavior that happens rarely and unpredictably.\nHow do non-breaking breakpoints work? If you\u0026rsquo;re familiar with general distributed tracing concepts, such as \u0026ldquo;traces\u0026rdquo; and \u0026ldquo;spans\u0026rdquo;, then you\u0026rsquo;re already broadly familiar with how non-breaking breakpoints work. Put simply, non-breaking breakpoints are small fragments of code added during runtime that, upon the proper conditions, save a portion of the application\u0026rsquo;s current state, and resume normal execution. In SkyWalking, this can be implemented by simply opening a new local span, adding some tags, and closing the local span.\nWhile this process is relatively simple, the range of functionality that can be achieved through this technique is quite impressive. Save the current and global variables to create a non-breaking breakpoint; add the ability to format log messages to create just-in-time logging; add the ability to trigger metric telemetry to create real-time KPI monitoring. If you keep moving in this direction, you eventually enter the realm of live debugging/coding, and this is where Source++ comes in.\nLive Coding Platform Source++ is an open-source live coding platform designed for production environments, powered by Apache SkyWalking. Using Source++, developers can add breakpoints, logs, metrics, and distributed tracing to live production software in real-time on-demand, right from their IDE or CLI. While capable of stand-alone deployment, the latest version of Source++ makes it easier than ever to integrate into existing Apache SkyWalking installations. This process can be completed in a few minutes and is easy to customize for your specific needs.\nFor a better idea of how Source++ works, take a look at the following diagram:\nIn this diagram, blue components represent existing SkyWalking architecture, black components represent new Source++ architecture, and the red arrows show how non-breaking breakpoints make their way from production to IDEs. A process that is facilitated by Source++ components: Live Probe, Live Processors, Live Platform, and Live Interface.\nLive Probe The Live Probe is currently available for JVM and Python applications. It runs alongside the SkyWalking agent and is responsible for dynamically adding and removing code fragments based on valid instrumentation requests from developers. These code fragments in turn make use of the SkyWalking agent\u0026rsquo;s internal APIs to facilitate production instrumentation.\nLive Processors Live Processors are responsible for finding, extracting, and transforming data found in distributed traces produced via live probes. They run alongside SkyWalking collectors and implement additional post-processing logic, such as PII redaction. Live processors work via uniquely identifiable tags (prefix spp.) added previously by live probes.\nOne could easily view a non-breaking breakpoint ready for processing using Rocketbot, however, it will look like this:\nEven though the above does not resemble what\u0026rsquo;s normally thought of as a breakpoint, the necessary information is there. With live processors added to your SkyWalking installation, this data is refined and may be viewed more traditionally via live interfaces.\nLive Platform The Live Platform is the core part of the Source++ architecture. Unlike the live probe and processors, the live platform does not have a direct correlation with SkyWalking components. It is a standalone server responsible for validating and distributing production breakpoints, logs, metrics, and traces. Each component of the Source++ architecture (probes, processors, interfaces) communicates with each other through the live platform. It is important to ensure the live platform is accessible to all of these components.\nLive Interface Finally, with all the previous parts installed, we\u0026rsquo;re now at the component software developers will find the most useful. A Live Interface is what developers use to create, manage, and view non-breaking breakpoints, and so on. There are a few live interfaces available:\nJetBrains Plugin CLI With the Live Instrument Processor enabled, and the JetBrains Plugin installed, non-breaking breakpoints appear as such:\nThe above should be a sight far more familiar to software developers. Beyond the fact that you can\u0026rsquo;t step through execution, non-breaking breakpoints look and feel just like regular breakpoints.\nFor more details and complete setup instructions, please visit:\nhttps://github.com/sourceplusplus/deploy-skywalking ","excerpt":"\u003cp\u003eNon-breaking breakpoints are breakpoints specifically designed for live production environments. …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2021-12-06-extend-skywalking-with-nbb/","title":"Extending Apache SkyWalking with non-breaking breakpoints"},{"body":"SkyWalking Kubernetes Helm Chart 4.2.0 is released. Go to downloads page to find release tars.\nFix Can\u0026rsquo;t evaluate field Capabilities in type interface{}. Update the document let that all docker images use the latest version. Fix missing nodes resource permission when the OAP using k8s-mesh analyzer. Fix bug that customized config files are not loaded into es-init job. Add skywalking satellite support. ","excerpt":"\u003cp\u003eSkyWalking Kubernetes Helm Chart 4.2.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eFix …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-kubernetes-helm-chart-4.2.0/","title":"Release Apache SkyWalking Kubernetes Helm Chart 4.2.0"},{"body":"SkyWalking Satellite 0.4.0 is released. Go to downloads page to find release tars.\nFeatures Support partition queue. Using byte array to transmit the ALS streaming, Native tracing segment and log, reducing en/decoding cpu usage. Support using the new ALS protocol to transmit the Envoy accesslog. Support transmit the Native Meter Batch protocol. Bug Fixes Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Satellite 0.4.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-satellite-0-4-0/","title":"Release Apache SkyWalking Satellite 0.4.0"},{"body":"SkyWalking 8.9.0 is released. Go to downloads page to find release tars.\nChanges by Version\nProject E2E tests immigrate to e2e-v2. Support JDK 16 and 17. Add Docker images for arm64 architecture. OAP Server Add component definition for Jackson. Fix that zipkin-receiver plugin is not packaged into dist. Upgrade Armeria to 1.12, upgrade OpenSearch test version to 1.1.0. Add component definition for Apache-Kylin. Enhance get generation mechanism of OAL engine, support map type of source\u0026rsquo;s field. Add tag(Map) into All, Service, ServiceInstance and Endpoint sources. Fix funcParamExpression and literalExpression can\u0026rsquo;t be used in the same aggregation function. Support cast statement in the OAL core engine. Support (str-\u0026gt;long) and (long) for string to long cast statement. Support (str-\u0026gt;int) and (int) for string to int cast statement. Support Long literal number in the OAL core engine. Support literal string as parameter of aggregation function. Add attributeExpression and attributeExpressionSegment in the OAL grammar tree to support map type for the attribute expression. Refactor the OAL compiler context to improve readability. Fix wrong generated codes of hashCode and remoteHashCode methods for numeric fields. Support != null in OAL engine. Add Message Queue Consuming Count metric for MQ consuming service and endpoint. Add Message Queue Avg Consuming Latency metric for MQ consuming service and endpoint. Support -Inf as bucket in the meter system. Fix setting wrong field when combining Events. Support search browser service. Add getProfileTaskLogs to profile query protocol. Set SW_KAFKA_FETCHER_ENABLE_NATIVE_PROTO_LOG, SW_KAFKA_FETCHER_ENABLE_NATIVE_JSON_LOG default true. Fix unexpected deleting due to TTL mechanism bug for H2, MySQL, TiDB and PostgreSQL. Add a GraphQL query to get OAP version, display OAP version in startup message and error logs. Fix TimeBucket missing in H2, MySQL, TiDB and PostgreSQL bug, which causes TTL doesn\u0026rsquo;t work for service_traffic. Fix TimeBucket missing in ElasticSearch and provide compatible storage2Entity for previous versions. Fix ElasticSearch implementation of queryMetricsValues and readLabeledMetricsValues doesn\u0026rsquo;t fill default values when no available data in the ElasticSearch server. Fix config yaml data type conversion bug when meets special character like !. Optimize metrics of minute dimensionality persistence. The value of metrics, which has declaration of the default value and current value equals the default value logically, the whole row wouldn\u0026rsquo;t be pushed into database. Fix max function in OAL doesn\u0026rsquo;t support negative long. Add MicroBench module to make it easier for developers to write JMH test. Upgrade Kubernetes Java client to 14.0.0, supports GCP token refreshing and fixes some bugs. Change SO11Y metric envoy_als_in_count to calculate the ALS message count. Support Istio 1.10.3, 1.11.4, 1.12.0 release.(Tested through e2e) Add filter mechanism in MAL core to filter metrics. Fix concurrency bug in MAL increase-related calculation. Fix a null pointer bug when building SampleFamily. Fix the so11y latency of persistence execution latency not correct in ElasticSearch storage. Add MeterReportService collectBatch method. Add OpenSearch 1.2.0 to test and verify it works. Upgrade grpc-java to 1.42.1 and protoc to 3.17.3 to allow using native Mac osx-aarch_64 artifacts. Fix TopologyQuery.loadEndpointRelation bug. Support using IoTDB as a new storage option. Add customized envoy ALS protocol receiver for satellite transmit batch data. Remove logback dependencies in IoTDB plugin. Fix StorageModuleElasticsearchProvider doesn\u0026rsquo;t watch on trustStorePath. Fix a wrong check about entity if GraphQL at the endpoint relation level. UI Optimize endpoint dependency. Show service name by hovering nodes in the sankey chart. Add Apache Kylin logo. Add ClickHouse logo. Optimize the style and add tips for log conditions. Fix the condition for trace table. Optimize profile functions. Implement a reminder to clear cache for dashboard templates. Support +/- hh:mm in TimeZone setting. Optimize global settings. Fix current endpoint for endpoint dependency. Add version in the global settings popup. Optimize Log page style. Avoid some abnormal settings. Fix query condition of events. Documentation Enhance documents about the data report and query protocols. Restructure documents about receivers and fetchers. Remove general receiver and fetcher docs Add more specific menu with docs to help users to find documents easier. Add a guidance doc about the logic endpoint. Link Satellite as Load Balancer documentation and compatibility with satellite. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking 8.9.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cp\u003eChanges by Version\u003c/p\u003e\n\u003ch4 id=\"project\"\u003eProject …\u003c/h4\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-8-9-0/","title":"Release Apache SkyWalking APM 8.9.0"},{"body":"Chaos Mesh 是一个开源的云原生混沌工程平台，借助 Chaos Mesh，用户可以很方便地对服务注入异常故障，并配合 Chaos Dashboard 实现对整个混沌实验运行状况的监测 。然而，对混沌实验运行情况的监控并不能告诉我们应用服务性能的变化。从系统可观测性的角度来说，我们可能无法单纯通过混沌实验的动态了解故障的全貌，这也阻碍了我们对系统和故障的进一步了解，调试。\nApache SkyWalking 是一个开源的 APM (Application Performance Monitor) 系统，可以对云原生服务提供监控、跟踪、诊断等功能。SkyWalking 支持收集 Event（事件），可在 Dashboard 中查看分布式系统中发生了哪些事件，并可以直观地观测到不同 Event 对服务性能造成的影响，和 Chaos Mesh 结合使用，便可为混沌实验造成的服务影响提供监控。\n本教程将分享如何通过将 SkyWalking 和 Chaos Mesh 结合，运用 Event 信息监控，实时了解混沌实验对应用服务性能造成的影响。\n准备工作 创建 Skywalking 集群，具体可以参考 SkyWalking Readme。 部署 Chaos Mesh，推荐使用 helm 安装。 安装 Java 测试工具 JMeter （其他工具亦可，仅用于增加服务负载） 如果仅作为 Demo 使用，可以参考 chaos-mesh-on-skywalking 这个仓库进行配置 Step 1 - 访问 SkyWalking 集群 安装 SkyWalking 后，就可以访问它的UI了，但因为还没有服务进行监控，这里还需要添加服务并进行 Agent 埋点设置。本文选用轻量级微服务框架 Spring Boot 作为埋点对象搭建一个简易 Demo 环境。\n可以参考 chaos-mesh-on-skywalking 仓库中的 demo-deployment.yaml 文件创建。之后使用 kubectl apply -f demo-deployment.yaml -n skywalking 进行部署。部署成功后即可在SkyWalking-UI 中看到实时监控的服务信息。\n注意：因为 Spring Boot 的端口也是8080，在端口转发时要避免和 **SkyWalking **的端口冲突，比如使用 kubectl port-forward svc/spring-boot-skywalking-demo 8079:8080 -n skywalking 。\nStep 2 - 部署 SkyWalking Kubernetes Event Exporter SkyWalking Kubernetes Event Exporter 可以用来监控和过滤 Kubernetes 集群中的 Event ，通过设置过滤条件筛选出需要的 Event，并将这些 Event 发送到 SkyWalking 后台， 这样就可以通过 SkyWalking 观察到你的 Kubernetes 集群中的Event 何时影响到服务的各项指标了。如果想要一条命令部署，可以参考此配置创建 yaml 文件 ，设置 filters 和 exporters 的参数后，使用 kubectl apply 进行部署。\nStep 3 - 使用 JMeter 对服务加压 为了达到更好的观察效果，需要先对 Spring Boot 增加服务负载，本文选择使用 JMeter 这一使用广泛的 Java 压力测试工具来对服务加压。\n通过 JMeter 对 localhost:8079 进行压测，添加5个线程持续进行加压。 通过 SkyWalking Dashboard 可以看到，目前访问成功率为100%，服务负载大约在5300 CPM (Calls Per Minute）。\nStep 4 - Chaos Mesh 注入故障，观察效果 做好了这些准备工便可以使用 Chaos Dashboard 进行压力场景模拟，并在实验进程中观察服务性能的变化。\n以下使用不同 Stress Chaos 配置，观测对应服务性能变化：\nCPU 负载10%，内存负载128 MB 。\n混沌实验开始和结束的时间点标记可以通过右侧开关显示在在图表中，将鼠标移至短线出可以看到是实验的 Applied 或 Recovered。可以看到两个绿色短线之间的时间段里，服务处理调用的的性能降低，为4929 CPM，在实验结束后，性能恢复正常。\nCPU load 增加到50%，发现服务负载进一步降低至4307 CPM。\n极端情况下 CPU 负载达到100%，服务负载降至无混沌实验时的40% 。\n因为 Linux 系统下的进程调度并不会让某个进程一直占据 CPU，所以即使实在 CPU 满载的极端情况下，该部署的 Spring Boot Demo 仍可以处理40%的访问请求。\n小结 通过 SkyWalking 与 Chaos Mesh 的结合，我们可以清晰的观察到服务在何时受到混沌实验的影响，在注入混沌后服务的表现性能又将如何。SkyWalking 与 Chaos Mesh 的结合使得我们轻松地观察到了服务在各种极端情况下的表现，增强了我们对服务的信心。\nChaos Mesh 在 2021 年成长了许多。为了更多地了解用户在实践混沌工程方面的经验，以便持续完善和提升对用户的支持，社区发起了 Chaos Mesh 用户问卷调查，点击【阅读原文】参与调查，谢谢！\nhttps://www.surveymonkey.com/r/X78WQPC\n欢迎大家加入 Chaos Mesh 社区，加入 CNCF Slack (slack.cncf.io) 底下的 Chaos Mesh 频道: project-chaos-mesh，一起参与到项目的讨论与开发中来！大家在使用过程发现 Bug 或缺失什么功能，也可以直接在 GitHub (https://github.com/chaos-mesh) 上提 Issue 或 PR。\n","excerpt":"\u003cp\u003e\u003ca href=\"https://github.com/chaos-mesh/chaos-mesh\"\u003eChaos Mesh\u003c/a\u003e 是\u003cstrong\u003e一个开源的云原生混沌工程\u003c/strong\u003e平台，借助 Chaos Mesh，用户可以很方便地对服务注入异常故障，并配合 Chaos Dashboard 实现对整个混沌实验运行状况的监测 。然而， …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2021-11-29-better-observability-for-chaos-engineering/","title":"Chaos Mesh X SkyWalking: 可观测的混沌工程"},{"body":"This plugin is one of the outcomes of Apache IoTDB - Apache SkyWalking Adapter in Summer 2021 of Open Source Promotion Plan. The design and development work is under the guidance of @jixuan1989 from IoTDB and @wu-sheng from SkyWalking. Thanks for their guidance and the help from community.\nIoTDB Storage Plugin Setup IoTDB is a time-series database from Apache, which is one of the storage plugin options. If you want to use iotdb as SkyWalking backend storage, please refer to the following configuration.\nIoTDB storage plugin is still in progress. Its efficiency will improve in the future.\nstorage: selector: ${SW_STORAGE:iotdb} iotdb: host: ${SW_STORAGE_IOTDB_HOST:127.0.0.1} rpcPort: ${SW_STORAGE_IOTDB_RPC_PORT:6667} username: ${SW_STORAGE_IOTDB_USERNAME:root} password: ${SW_STORAGE_IOTDB_PASSWORD:root} storageGroup: ${SW_STORAGE_IOTDB_STORAGE_GROUP:root.skywalking} sessionPoolSize: ${SW_STORAGE_IOTDB_SESSIONPOOL_SIZE:16} fetchTaskLogMaxSize: ${SW_STORAGE_IOTDB_FETCH_TASK_LOG_MAX_SIZE:1000} # the max number of fetch task log in a request All connection related settings, including host, rpcPort, username, and password are found in application.yml. Please ensure the IoTDB version \u0026gt;= 0.12.3.\nIoTDB Introduction Apache IoTDB (Database for Internet of Things) is an IoT native database with high performance for data management and analysis, deployable on the edge and the cloud. It is a time-series database donated by Tsinghua University to Apache Foundation.\nThe Data Model of IoTDB We can use the tree structure to understand the data model of iotdb. If divided according to layers, from high to low is: Storage Group \u0026ndash; (LayerName) \u0026ndash; Device \u0026ndash; Measurement. From the top layer to a certain layer below it is called a Path. The top layer is Storage Group (must start with root), the penultimate layer is Device, and the bottom layer is Measurement. There can be many layers in the middle, and each layer is called a LayerName. For more information, please refer to the Data Model and Terminology in the official document of the version 0.12.x.\nThe Design of IoTDB Storage Plugin The Data Model of SkyWalking Each storage model of SkyWalking can be considered as a Model, which contains multiple Columns. Each Column has ColumnName and ColumnType attributes, representing the name and type of Column respectively. Each Column named ColumnName stores multiple Value of the ColumnType. From a relational database perspective, Model is a relational table and Column is the field in a relational table.\nSchema Design Since each LayerName of IoTDB is stored in memory, it can be considered as an index, and this feature can be fully utilized to improve IoTDB query performance. The default storage group is root.skywalking, it will occupy the first and the second layer of the path. The model name is stored at the next layer of the storage group (the third layer of the path), such as root.skywalking.model_name.\nSkyWalking has its own index requirement, but it isn\u0026rsquo;t applicable to IoTDB. Considering query frequency and referring to the implementation of the other storage options, we choose id, entity_id, node_type, service_id, service_group, trace_id as indexes and fix their appearance order in the path. The value of these indexed columns will occupy the last few layers of the path. If we don\u0026rsquo;t fix their order, we cannot map their value to column, since we only store their value in the path but don\u0026rsquo;t store their column name. The other columns are treated as Measurements.\nThe mapping from SkyWalking data model to IoTDB data model is below.\nSkyWalking IoTDB Database Storage Group (1st and 2nd layer of the path) Model LayerName (3rd layer of the path) Indexed Column stored in memory through hard-code Indexed Column Value LayerName (after 3rd layer of the path) Non-indexed Column Measurement Non-indexed Value the value of Measurement For general example There are model1(column11, column12), model2(column21, column22, column23), model3(column31). Underline indicates that the column requires to be indexed. In this example, modelx_name refers to the name of modelx, columnx_name refers to the name of columnx and columnx_value refers to the value of columnx.\nBefore these 3 model storage schema, here are some points we need to know.\nIn order to avoid the value of indexed column contains dot(.), all of them should be wrapped in double quotation mark since IoTDB use dot(.) as the separator in the path. We use align by device in query SQL to get a more friendly result. For more information about align by device, please see DML (Data Manipulation Language) and Query by device alignment. The path of them is following:\nThe Model with index: root.skywalking.model1_name.column11_value.column12_name root.skywalking.model2_name.column21_value.column22_value.column23_name The Model without index: root.skywalking.model3_name.column31_Name Use select * from root.skywalking.modelx_name align by device respectively to get their schema and data. The SQL result is following:\nTime Device column12_name 1637494020000 root.skywalking.model1_name.\u0026ldquo;column11_value\u0026rdquo; column12_value Time Device column23_name 1637494020000 root.skywalking.model2_name.\u0026ldquo;column21_value\u0026rdquo;.\u0026ldquo;column22_value\u0026rdquo; column23_value Time Device column31_name 1637494020000 root.skywalking.model3_name column31_value For specific example Before 5 typical examples, here are some points we need to know.\nThe indexed columns and their order: id, entity_id, node_type, service_id, service_group, trace_id. Other columns are treated as non indexed and stored as Measurement. The storage entity extends Metrics or Record contains a column time_bucket. The time_bucket column in SkyWalking Model can be converted to the timestamp of IoTDB when inserting data. We don\u0026rsquo;t need to store time_bucket separately. In the next examples, we won\u0026rsquo;t list time_bucket anymore. The Time in query result corresponds to the timestamp in insert SQL and API. Metadata: service_traffic\nservice_traffic entity has 4 columns: id, name, node_type, service_group. When service_traffic entity includes a row with timestamp 1637494020000, the row should be as following: (Notice: the value of service_group is null.) id name node_type service_group ZTJlLXNlcnZpY2UtcHJvdmlkZXI=.1 e2e-service-provider 0 And the row stored in IoTDB should be as following: (Query SQL: select from root.skywalking.service_traffic align by device)\nTime Device name 1637494020000 root.skywalking.service_traffic.\u0026ldquo;ZTJlLXNlcnZpY2UtcHJvdmlkZXI=.1\u0026rdquo;.\u0026ldquo;0\u0026rdquo;.\u0026ldquo;null\u0026rdquo; e2e-service-provider The value of id, node_type and service_group are stored in the path in the specified order. Notice: If those index value is null, it will be transformed to a string \u0026ldquo;null\u0026rdquo;.\nMetrics: service_cpm\nservice_cpm entity has 4 columns: id, service_id, total, value.\nWhen service_cpm entity includes a row with timestamp 1637494020000, the row should be as following: id service_id total value 202111211127_ZTJlLXNlcnZpY2UtY29uc3VtZXI=.1 ZTJlLXNlcnZpY2UtY29uc3VtZXI=.1 4 4 And the row stored in IoTDB should be as following: (Query SQL: select from root.skywalking.service_cpm align by device)\nTime Device total value 1637494020000 root.skywalking.service_cpm.\u0026ldquo;202111211127_ZTJlLXNlcnZpY2UtY29uc3VtZXI=.1\u0026rdquo;.\u0026ldquo;ZTJlLXNlcnZpY2UtY29uc3VtZXI=.1\u0026rdquo; 4 4 The value of id and service_id are stored in the path in the specified order.\nTrace segment: segment\nsegment entity has 10 columns at least: id, segment_id, trace_id, service_id, service_instance_id, endpoint_id, start_time, latency, is_error, data_binary. In addition, it could have variable number of tags.\nWhen segment entity includes 2 rows with timestamp 1637494106000 and 1637494134000, these rows should be as following. The db.type and db.instance are two tags. The first data has two tags, and the second data doesn\u0026rsquo;t have tag. id segment_id trace_id service_id service_instance_id endpoint_id start_time latency is_error data_binary db.type db.instance id_1 segment_id_1 trace_id_1 service_id_1 service_instance_id_1 endpoint_id_1 1637494106515 1425 0 data_binary_1 sql testdb id_2 segment_id_2 trace_id_2 service_id_2 service_instance_id_2 endpoint_id_2 2637494106765 1254 0 data_binary_2 And these row stored in IoTDB should be as following: (Query SQL: select from root.skywalking.segment align by device)\nTime Device start_time data_binary latency endpoint_id is_error service_instance_id segment_id \u0026ldquo;db.type\u0026rdquo; \u0026ldquo;db.instance\u0026rdquo; 1637494106000 root.skywalking.segment.\u0026ldquo;id_1\u0026rdquo;.\u0026ldquo;service_id_1\u0026rdquo;.\u0026ldquo;trace_id_1\u0026rdquo; 1637494106515 data_binary_1 1425 endpoint_id_1 0 service_instance_id_1 segment_id_1 sql testdb 1637494106000 root.skywalking.segment.\u0026ldquo;id_2\u0026rdquo;.\u0026ldquo;service_id_2\u0026rdquo;.\u0026ldquo;trace_id_2\u0026rdquo; 1637494106765 data_binary_2 1254 endpoint_id_2 0 service_instance_id_2 segment_id_2 null null The value of id, service_id and trace_id are stored in the path in the specified order. Notice: If the measurement contains dot(.), it will be wrapped in double quotation mark since IoTDB doesn\u0026rsquo;t allow it. In order to align, IoTDB will append null value for those data without tag in some models.\nLog\nlog entity has 12 columns at least: id, unique_id, service_id, service_instance_id, endpoint_id, trace_id, trace_segment_id, span_id, content_type, content, tags_raw_data, timestamp. In addition, it could have variable number of tags. When log entity includes a row with timestamp 1637494052000, the row should be as following and the level is a tag. id unique_id service_id service_instance_id endpoint_id trace_id trace_segment_id span_id content_type content tags_raw_data timestamp level id_1 unique_id_1 service_id_1 service_instance_id_1 endpoint_id_1 trace_id_1 trace_segment_id_1 0 1 content_1 tags_raw_data_1 1637494052118 INFO And the row stored in IoTDB should be as following: (Query SQL: select from root.skywalking.log align by device)\nTime Device unique_id content_type span_id tags_raw_data \u0026ldquo;timestamp\u0026rdquo; level service_instance_id content trace_segment_id 1637494052000 root.skywalking.\u0026ldquo;id_1\u0026rdquo;.\u0026ldquo;service_id_1\u0026rdquo;.\u0026ldquo;trace_id_1\u0026rdquo; unique_id_1 1 0 tags_raw_data_1 1637494052118 INFO service_instance_id_1 content_1 trace_segment_id_1 The value of id, service_id and trace_id are stored in the path in the specified order. Notice: If the measurement named timestamp, it will be wrapped in double quotation mark since IoTDB doesn\u0026rsquo;t allow it.\nProfiling snapshots: profile_task_segment_snapshot\nprofile_task_segment_snapshot entity has 6 columns: id, task_id, segment_id, dump_time, sequence, stack_binary. When profile_task_segment_snapshot includes a row with timestamp 1637494131000, the row should be as following. id task_id segment_id dump_time sequence stack_binary id_1 task_id_1 segment_id_1 1637494131153 0 stack_binary_1 And the row stored in IoTDB should be as following: (Query SQL: select from root.skywalking.profile_task_segment_snapshot align by device)\nTime Device sequence dump_time stack_binary task_id segment_id 1637494131000 root.skywalking.profile_task_segment_snapshot.\u0026ldquo;id_1\u0026rdquo; 0 1637494131153 stack_binary_1 task_id_1 segment_id_1 The value of id is stored in the path in the specified order.\nQuery In this design, part of the data is stored in memory through LayerName, so data from the same Model is spread across multiple devices. Queries often need to cross multiple devices. But in this aspect, IoTDB\u0026rsquo;s support is not perfect in cross-device aggregation query, sort query and pagination query. In some cases, we have to use a violence method that query all data meets the condition and then aggregate, sort or paginate them. So it might not be efficient. For detailed descriptions, please refer to the Discussion submitted in IoTDB community below.\nDiscussion: 一个有关排序查询的问题（A problem about sort query）#3888 一个有关聚合查询的问题（A problem about aggregation query）#3907 Query SQL for the general example above:\n-- query all data in model1 select * from root.skywalking.model1_name align by device; -- query the data in model2 with column22_value=\u0026#34;test\u0026#34; select * from root.skywalking.model2_name.*.\u0026#34;test\u0026#34; align by device; -- query the sum of column23 in model2 and group by column21 select sum(column23) from root.skywalking.model2_name.*.* group by level = 3; iotdb-cli is a useful tools to connect and visit IoTDB server. More information please refer Command Line Interface(CLI)\n","excerpt":"\u003cp\u003eThis plugin is one of the outcomes of \u003ca href=\"https://summer.iscas.ac.cn/#/org/prodetail/210070771\"\u003eApache IoTDB - Apache SkyWalking Adapter\u003c/a\u003e in \u003ca href=\"https://summer.iscas.ac.cn/#/homepage\"\u003eSummer 2021 of …\u003c/a\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2021-11-23-design-of-iotdb-storage-option/","title":"The Design of Apache IoTDB Storage Option"},{"body":"SkyWalking Infra E2E 1.1.0 is released. Go to downloads page to find release tars.\nFeatures Support using setup.init-system-environment to import environment. Support body and headers in http trigger. Add install target in makefile. Stop trigger when cleaning up. Change interval setting to Duration style. Add reasonable default cleanup.on. Support float value compare when type not match Support reuse verify.cases. Ignore trigger when not set. Support export KUBECONFIG to the environment. Support using setup.kind.import-images to load local docker images. Support using setup.kind.expose-ports to declare the resource port for host access. Support save pod/container std log on the Environment. Bug Fixes Fix that trigger is not continuously triggered when running e2e trigger. Migrate timeout config to Duration style and wait for node ready in KinD setup. Remove manifest only could apply the default namespace resource. Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Infra E2E 1.1.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-infra-e2e-1-1-0/","title":"Release Apache SkyWalking Infra E2E 1.1.0"},{"body":"SkyWalking Cloud on Kubernetes 0.4.0 is released. Go to downloads page to find release tars.\nSupport special characters in the metric selector of HPA metric adapter.\nAdd the namespace to HPA metric name.\nFeatures\nAdd Java agent injector. Add JavaAgent and Storage CRDs of the operator. Vulnerabilities\nCVE-2021-3121: An issue was discovered in GoGo Protobuf before 1.3.2. plugin/unmarshal/unmarshal.go lacks certain index validation CVE-2020-29652: A nil pointer dereference in the golang.org/x/crypto/ssh component through v0.0.0-20201203163018-be400aefbc4c for Go allows remote attackers to cause a denial of service against SSH servers. Chores\nBump up GO to 1.17. Bump up k8s api to 0.20.11. Polish documents. Bump up SkyWalking OAP to 8.8.1. ","excerpt":"\u003cp\u003eSkyWalking Cloud on Kubernetes 0.4.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars. …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cloud-on-kubernetes-0-4-0/","title":"Release Apache SkyWalking Cloud on Kubernetes 0.4.0"},{"body":"SkyWalking Satellite 0.3.0 is released. Go to downloads page to find release tars.\nFeatures Support load-balance GRPC client with the static server list. Support load-balance GRPC client with the Kubernetes selector. Support transmit Envoy ALS v2/v3 protocol. Support transmit Envoy Metrics v2/v3 protocol. Bug Fixes Fix errors when converting meter data from histogram and summary.#75 Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Satellite 0.3.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-satellite-0-3-0/","title":"Release Apache SkyWalking Satellite 0.3.0"},{"body":"SkyWalking Java Agent 8.8.0 is released. Go to downloads page to find release tars. Changes by Version\n8.8.0 Split Java agent from the main monorepo. It is a separate repository and going to release separately. Support JDK 8-17 through upgrading byte-buddy to 1.11.18. Upgrade JDK 11 in dockerfile and remove unused java_opts. DataCarrier changes a #consume API to add properties as a parameter to initialize consumer when use Class\u0026lt;? extends IConsumer\u0026lt;T\u0026gt;\u0026gt; consumerClass. Support Multiple DNS period resolving mechanism Modify Tags.STATUS_CODE field name to Tags.HTTP_RESPONSE_STATUS_CODE and type from StringTag to IntegerTag, add Tags.RPC_RESPONSE_STATUS_CODE field to hold rpc response code value. Fix kafka-reporter-plugin shade package conflict Add all config items to agent.conf file for convenient containerization use cases. Advanced Kafka Producer configuration enhancement. Support mTLS for gRPC channel. fix the bug that plugin record wrong time elapse for lettuce plugin fix the bug that the wrong db.instance value displayed on Skywalking-UI when existing multi-database-instance on same host port pair. Add thrift plugin support thrift TMultiplexedProcessor. Add benchmark result for exception-ignore plugin and polish plugin guide. Provide Alibaba Druid database connection pool plugin. Provide HikariCP database connection pool plugin. Fix NumberFormat exception in jdbc-commons plugin when MysqlURLParser parser jdbcurl Provide Alibaba Fastjson parser/generator plugin. Provide Jackson serialization and deserialization plugin. Fix a tracing context leak of SpringMVC plugin, when an internal exception throws due to response can\u0026rsquo;t be found. Make GRPC log reporter sharing GRPC channel with other reporters of agent. Remove config items of agent.conf, plugin.toolkit.log.grpc.reporter.server_host, plugin.toolkit.log.grpc.reporter.server_port, and plugin.toolkit.log.grpc.reporter.upstream_timeout. rename plugin.toolkit.log.grpc.reporter.max_message_size to log.max_message_size. Implement Kafka Log Reporter. Add config item of agnt.conf, plugin.kafka.topic_logging. Add plugin to support Apache HttpClient 5. Format SpringMVC \u0026amp; Tomcat EntrySpan operation name to METHOD:URI. Make HTTP method in the operation name according to runtime, rather than previous code-level definition, which used to have possibilities including multiple HTTP methods. Fix the bug that httpasyncclient-4.x-plugin does not take effect every time. Add plugin to support ClickHouse JDBC driver. Fix version compatibility for JsonRPC4J plugin. Add plugin to support Apache Kylin-jdbc 2.6.x 3.x 4.x Fix instrumentation v2 API doesn\u0026rsquo;t work for constructor instrumentation. Add plugin to support okhttp 2.x Optimize okhttp 3.x 4.x plugin to get span time cost precisely Adapt message header properties of RocketMQ 4.9.x Documentation All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking Java Agent 8.8.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nChanges by …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-java-agent-8-8-0/","title":"Release Apache SkyWalking Java Agent 8.8.0"},{"body":"SkyWalking CLI 0.9.0 is released. Go to downloads page to find release tars.\nFeatures Add the sub-command dependency instance to query instance relationships (#117) Bug Fixes fix: multiple-linear command\u0026rsquo;s labels type can be string type (#122) Add missing dest-service-id dest-service-name to metrics linear command (#121) Fix the wrong name when getting destInstance flag (#118) Chores Upgrade Go version to 1.16 (#120) Migrate tests to infra-e2e, overhaul the flags names (#119) Publish Docker snapshot images to ghcr (#116) Remove dist directory when build release source tar (#115) ","excerpt":"\u003cp\u003eSkyWalking CLI 0.9.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch3 id=\"features\"\u003eFeatures\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eAdd the …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cli-0-9-0/","title":"Release Apache SkyWalking CLI 0.9.0"},{"body":"SkyWalking Eyes 0.2.0 is released. Go to downloads page to find release tars.\nDependency License\nSupport resolving go.mod for Go Support resolving pom.xml for maven (#50) Support resolving jars\u0026rsquo; licenses (#53) Support resolving npm dependencies\u0026rsquo; licenses (#48) Support saving dependencies\u0026rsquo; licenses (#69) Add dependency check to check dependencies license compatibilities (#58) License Header\nfix command supports more languages: Add support for plantuml (#42) Add support for PHP (#40) Add support for Twig template language (#39) Add support for Smarty template language (#38) Add support for MatLab files (#37) Add support for TypeScript language files (#73) Add support for nextflow files (#65) Add support for perl files (#63) Add support for ini extension (#24) Add support for R files (#64) Add support for .rst files and allow fixing header of a single file (#25) Add support for Rust files (#29) Add support for bat files (#32) Remove .tsx from XML language extensions Honor Python\u0026rsquo;s coding directive (#68) Fix file extension conflict between RenderScript and Rust (#66) Add comment type to cython declaration (#62) header fix: respect user configured license content (#60) Expose license-location-threshold as config item (#34) Fix infinite recursive calls when containing symbolic files (#33) defect: avoid crash when no comment style is found (#23) Project\nEnhance license identification (#79) Support installing via go install (#76) Speed up the initialization phase (#75) Resolve absolute path in .gitignore to relative path (#67) Reduce img size and add npm env (#59) Make the config file and log level in GitHub Action configurable (#56, #57) doc: add a PlantUML activity diagram of header fixing mechanism (#41) Fix bug: license file is not found but reported message is nil (#49) Add all well-known licenses and polish normalizers (#47) Fix compatibility issues in Windows (#44) feature: add reasonable default config to allow running in a new repo without copying config file (#28) chore: only build linux binary when building inside docker (#26) chore: upgrade to go 1.16 and remove go-bindata (#22) Add documentation about how to use via docker image (#20) ","excerpt":"\u003cp\u003eSkyWalking Eyes 0.2.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eDependency License …\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-eyes-0-2-0/","title":"Release Apache SkyWalking Eyes 0.2.0"},{"body":"SkyWalking Client JS 0.7.0 is released. Go to downloads page to find release tars.\nSupport setting time interval to report segments. Fix segments report only send once. Fix apache/skywalking#7335. Fix apache/skywalking#7793. Fix firstReportedError for SPA. ","excerpt":"\u003cp\u003eSkyWalking Client JS 0.7.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eSupport setting …\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-client-js-0-7-0/","title":"Release Apache SkyWalking Client JS 0.7.0"},{"body":"SkyWalking 8.8.1 is released. Go to downloads page to find release tars.\nThis is a bugfix version that fixes several important bugs in previous version 8.8.0.\nChanges OAP Server Fix wrong (de)serializer of ElasticSearch client for OpenSearch storage. Fix that traces query with tags will report error. Replace e2e simple cases to e2e-v2. Fix endpoint dependency breaking. UI Delete duplicate calls for endpoint dependency. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking 8.8.1 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eThis is a bugfix version …\u003c/strong\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-8-8-1/","title":"Release Apache SkyWalking APM 8.8.1"},{"body":"Kai Wan has been involved in SkyWalking for over half a year since the first PR(Dec 21, 2020). He majorly focuses on the Service Mesh and metrics analysis engine(MAL). And recently add the support of OpenAPI specification into SkyWalking.\nHe learnd fast, and dedicates hours every day on the project, and has finished 37 PRs 11,168 LOC++ 1,586 LOC\u0026ndash;. In these days, he is working with PMC and infra-e2e team to upgrade our main repository\u0026rsquo;s test framework to the NGET(Next Generation E2E Test framework).\nIt is our honor to have him join the team.\n","excerpt":"\u003cp\u003e\u003ca href=\"https://github.com/wankai123\"\u003eKai Wan\u003c/a\u003e has been involved in SkyWalking for over half a year since the first PR(Dec 21, 2020).\nHe …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-kai-wan-to-join-the-pmc/","title":"Welcome Kai Wan (万凯) to join the PMC"},{"body":"SkyWalking 8.8.0 is released. Go to downloads page to find release tars.\nThis is a first OAP server + UI release, Java agent will be release independently. Check the latest compatibility document to find suitable agent releases.\nChanges by Version\nProject Split javaagent into skywalking-java repository. https://github.com/apache/skywalking-java Merge Dockerfiles from apache/skywalking-docker into this codebase. OAP Server Fix CVE-2021-35515, CVE-2021-35516, CVE-2021-35517, CVE-2021-36090. Upgrade org.apache.commons:commons-compress to 1.21. kubernetes java client upgrade from 12.0.1 to 13.0.0 Add event http receiver Support Metric level function serviceRelation in MAL. Support envoy metrics binding into the topology. Fix openapi-definitions folder not being read correctly. Trace segment wouldn\u0026rsquo;t be recognized as a TopN sample service. Add through #4694 experimentally, but it caused performance impact. Remove version and endTime in the segment entity. Reduce indexing payload. Fix mapper_parsing_exception in ElasticSearch 7.14. Support component IDs for Go-Kratos framework. [Break Change] Remove endpoint name in the trace query condition. Only support query by endpoint id. Fix ProfileSnapshotExporterTest case on OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9), MacOS. [Break Change] Remove page path in the browser log query condition. Only support query by page path id. [Break Change] Remove endpoint name in the backend log query condition. Only support query by endpoint id. [Break Change] Fix typo for a column page_path_id(was pate_path_id) of storage entity browser_error_log. Add component id for Python falcon plugin. Add rpcStatusCode for rpc.status_code tag. The responseCode field is marked as deprecated and replaced by httpResponseStatusCode field. Remove the duplicated tags to reduce the storage payload. Add a new API to test log analysis language. Harden the security of Groovy-based DSL, MAL and LAL. Fix distinct in Service/Instance/Endpoint query is not working. Support collection type in dynamic configuration core. Support zookeeper grouped dynamic configurations. Fix NPE when OAP nodes synchronize events with each other in cluster mode. Support k8s configmap grouped dynamic configurations. Add desc sort function in H2 and ElasticSearch implementations of IBrowserLogQueryDAO Support configure sampling policy by configuration module dynamically and static configuration file trace-sampling-policy-settings.yml for service dimension on the backend side. Dynamic configurations agent-analyzer.default.sampleRate and agent-analyzer.default.slowTraceSegmentThreshold are replaced by agent-analyzer.default.traceSamplingPolicy. Static configurations agent-analyzer.default.sampleRate and agent-analyzer.default.slowTraceSegmentThreshold are replaced by agent-analyzer.default.traceSamplingPolicySettingsFile. Fix dynamic configuration watch implementation current value not null when the config is deleted. Fix LoggingConfigWatcher return watch.value would not consistent with the real configuration content. Fix ZookeeperConfigWatcherRegister.readConfig() could cause NPE when data.getData() is null. Support nacos grouped dynamic configurations. Support for filter function filtering of int type values. Support mTLS for gRPC channel. Add yaml file suffix limit when reading ui templates. Support consul grouped dynamic configurations. Fix H2MetadataQueryDAO.searchService doesn\u0026rsquo;t support auto grouping. Rebuilt ElasticSearch client on top of their REST API. Fix ElasticSearch storage plugin doesn\u0026rsquo;t work when hot reloading from secretsManagementFile. Support etcd grouped dynamic configurations. Unified the config word namespace in the project. Switch JRE base image for dev images. Support apollo grouped dynamic configurations. Fix ProfileThreadSnapshotQuery.queryProfiledSegments adopts a wrong sort function Support gRPC sync grouped dynamic configurations. Fix H2EventQueryDAO doesn\u0026rsquo;t sort data by Event.START_TIME and uses a wrong pagination query. Fix LogHandler of kafka-fetcher-plugin cannot recognize namespace. Improve the speed of writing TiDB by batching the SQL execution. Fix wrong service name when IP is node IP in k8s-mesh. Support dynamic configurations for openAPI endpoint name grouping rule. Add component definition for Alibaba Druid and HikariCP. Fix Hour and Day dimensionality metrics not accurate, due to the cache read-then-clear mechanism conflicts with low down metrics flush period added in 8.7.0. Fix Slow SQL sampling not accurate, due to TopN works conflict with cache read-then-clear mechanism. The persistent cache is only read when necessary. Add component definition for Alibaba Fastjson. Fix entity(service/instance/endpoint) names in the MAL system(prometheus, native meter, open census, envoy metric service) are not controlled by core\u0026rsquo;s naming-control mechanism. Upgrade netty version to 4.1.68.Final avoid cve-2021-37136. UI Fix not found error when refresh UI. Update endpointName to endpointId in the query trace condition. Add Python falcon icon on the UI. Fix searching endpoints with keywords. Support clicking the service name in the chart to link to the trace or log page. Implement the Log Analysis Language text regexp debugger. Fix fetching nodes and calls with serviceIds on the topology side. Implement Alerts for query errors. Fixes graph parameter of query for topology metrics. Documentation Add a section in Log Collecting And Analysis doc, introducing the new Python agent log reporter. Add one missing step in otel-receiver doc about how to activate the default receiver. Reorganize dynamic configuration doc. Add more description about meter configurations in backend-meter doc. Fix typo in endpoint-grouping-rules doc. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking 8.8.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eThis is a first OAP server …\u003c/strong\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-8-8-0/","title":"Release Apache SkyWalking APM 8.8.0"},{"body":"SkyWalking CLI 0.8.0 is released. Go to downloads page to find release tars.\nFeatures\nAdd profile command Add logs command Add dependency command Support query events protocol Support auto-completion for bash and powershell Bug Fixes\nFix missing service instance name in trace command Chores\nOptimize output by adding color to help information Set display style explicitly for commands in the test script Set different default display style for different commands Add scripts for quick install Update release doc and add scripts for release split into multiple workflows to speed up CI ","excerpt":"\u003cp\u003eSkyWalking CLI 0.8.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eFeatures\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAdd profile …\u003c/li\u003e\u003c/ul\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cli-0-8-0/","title":"Release Apache SkyWalking CLI 0.8.0"},{"body":"SkyWalking Satellite 0.2.0 is released. Go to downloads page to find release tars.\nFeatures Set MAXPROCS according to real cpu quota. Update golangci-lint version to 1.39.0. Update protoc-gen-go version to 1.26.0. Add prometheus-metrics-fetcher plugin. Add grpc client plugin. Add nativelog-grpc-forwarder plugin. Add meter-grpc-forwarder plugin. Support native management protocol. Support native tracing protocol. Support native profile protocol. Support native CDS protocol. Support native JVM protocol. Support native Meter protocol. Support native Event protocol. Support native protocols E2E testing. Add Prometheus service discovery in Kubernetes. Bug Fixes Fix the data race in mmap queue. Fix channel blocking in sender module. Fix pipes.sender.min_flush_events config could not support min number. Remove service name and instance name labels from Prometheus fetcher. Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Satellite 0.2.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSet …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-satellite-0-2-0/","title":"Release Apache SkyWalking Satellite 0.2.0"},{"body":"SkyWalking Python 0.7.0 is released. Go to downloads page to find release tars.\nFeature:\nSupport collecting and reporting logs to backend (#147) Support profiling Python method level performance (#127 Add a new sw-python CLI that enables agent non-intrusive integration (#156) Add exponential reconnection backoff strategy when OAP is down (#157) Support ignoring traces by http method (#143) NoopSpan on queue full, propagation downstream (#141) Support agent namespace. (#126) Support secure connection option for GRPC and HTTP (#134) Plugins:\nAdd Falcon Plugin (#146) Update sw_pymongo.py to be compatible with cluster mode (#150) Add Python celery plugin (#125) Support tornado5+ and tornado6+ (#119) Fixes:\nRemove HTTP basic auth credentials from log, stacktrace, segment (#152) Fix @trace decorator not work (#136) Fix grpc disconnect, add SW_AGENT_MAX_BUFFER_SIZE to control buffer queue size (#138) Others:\nChore: bump up requests version to avoid license issue (#142) Fix module wrapt as normal install dependency (#123) Explicit component inheritance (#132) Provide dockerfile \u0026amp; images for easy integration in containerized scenarios (#159) ","excerpt":"\u003cp\u003eSkyWalking Python 0.7.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eFeature:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eSupport …\u003c/li\u003e\u003c/ul\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-python-0-7-0/","title":"Release Apache SkyWalking Python 0.7.0"},{"body":"SkyWalking Infra E2E 1.0.0 is released. Go to downloads page to find release tars.\nFeatures Support using docker-compose to setup the environment. Support using the HTTP request as trigger. Support verify test case by command-line or file with retry strategy. Support GitHub Action. Bug Fixes Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Infra E2E 1.0.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-infra-e2e-1-0-0/","title":"Release Apache SkyWalking Infra E2E 1.0.0"},{"body":"The Java Agent of Apache SkyWalking has supported profiling since v7.0.0, and it enables users to troubleshoot the root cause of performance issues, and now we bring it into Python Agent. In this blog, we will show you how to use it, and we will introduce the mechanism of profiling.\nHow to use profiling in Python Agent This feature is released in Python Agent at v0.7.0. It is turned on by default, so you don\u0026rsquo;t need any extra configuration to use it. You can find the environment variables about it here.\nHere are the demo codes of an intentional slow application.\nimport time def method1(): time.sleep(0.02) return \u0026#39;1\u0026#39; def method2(): time.sleep(0.02) return method1() def method3(): time.sleep(0.02) return method2() if __name__ == \u0026#39;__main__\u0026#39;: import socketserver from http.server import BaseHTTPRequestHandler class SimpleHTTPRequestHandler(BaseHTTPRequestHandler): def do_POST(self): method3() time.sleep(0.5) self.send_response(200) self.send_header(\u0026#39;Content-Type\u0026#39;, \u0026#39;application/json\u0026#39;) self.end_headers() self.wfile.write(\u0026#39;{\u0026#34;song\u0026#34;: \u0026#34;Despacito\u0026#34;, \u0026#34;artist\u0026#34;: \u0026#34;Luis Fonsi\u0026#34;}\u0026#39;.encode(\u0026#39;ascii\u0026#39;)) PORT = 19090 Handler = SimpleHTTPRequestHandler with socketserver.TCPServer((\u0026#34;\u0026#34;, PORT), Handler) as httpd: httpd.serve_forever() We can start it with SkyWalking Python Agent CLI without changing any application code now, which is also the latest feature of v0.7.0. We just need to add sw-python run before our start command(i.e. sw-python run python3 main.py), to start the application with python agent attached. More information about sw-python can be found there.\nThen, we should add a new profile task for the / endpoint from the SkyWalking UI, as shown below.\nWe can access it by curl -X POST http://localhost:19090/, after that, we can view the result of this profile task on the SkyWalking UI.\nThe mechanism of profiling When a request lands on an application with the profile function enabled, the agent begins the profiling automatically if the request’s URI is as required by the profiling task. A new thread is spawned to fetch the thread dump periodically until the end of request.\nThe agent sends these thread dumps, called ThreadSnapshot, to SkyWalking OAPServer, and the OAPServer analyzes those ThreadSnapshot(s) and gets the final result. It will take a method invocation with the same stack depth and code signature as the same operation, and estimate the execution time of each method from this.\nLet\u0026rsquo;s demonstrate how this analysis works through the following example. Suppose we have such a program below and we profile it at 10ms intervals.\ndef main(): methodA() def methodA(): methodB() def methodB(): methodC() methodD() def methodC(): time.sleep(0.04) def methodD(): time.sleep(0.06) The agent collects a total of 10 ThreadSnapShot(s) over the entire time period(Diagram A). The first 4 snapshots represent the thread dumps during the execution of function C, and the last 6 snapshots represent the thread dumps during the execution of function D. After the analysis of OAPServer, we can see the result of this profile task on the SkyWalking Rocketbot UI as shown in the right of the diagram. With this result, we can clearly see the function call relationship and the time consumption situation of this program.\nDiagram A You can read more details of profiling theory from this blog.\nWe hope you enjoy the profile in the Python Agent, and if so, you can give us a star on Python Agent and SkyWalking on GitHub.\n","excerpt":"\u003cp\u003eThe Java Agent of Apache SkyWalking has supported profiling since \u003ca href=\"https://github.com/apache/skywalking/releases/tag/v7.0.0\"\u003ev7.0.0\u003c/a\u003e, and it enables users to …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2021-09-12-skywalking-python-profiling/","title":"SkyWalking Python Agent Supports Profiling Now"},{"body":"SkyWalking Kubernetes Helm Chart 4.1.0 is released. Go to downloads page to find release tars.\nAdd missing service account to init job. Improve notes.txt and nodePort configuration. Improve ingress compatibility. Fix bug that customized config files are not loaded into es-init job. Add imagePullSecrets and node selector. Fix istio adapter description. Enhancement: allow mounting binary data files. ","excerpt":"\u003cp\u003eSkyWalking Kubernetes Helm Chart 4.1.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAdd …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-kubernetes-helm-chart-4.1.0/","title":"Release Apache SkyWalking Kubernetes Helm Chart 4.1.0"},{"body":"GOUP hosted a webinar, and invited Sheng Wu to introduce Apache SkyWalking. This is a 1.5 hours presentation including the full landscape of Apache SkyWalking 8.x.\nChapter04 Session10 - Apache Skywalking by Sheng Wu ","excerpt":"\u003cp\u003e\u003ca href=\"https://www.linkedin.com/company/goupaz/\"\u003eGOUP\u003c/a\u003e hosted a webinar, and invited \u003ca href=\"https://twitter.com/wusheng1108\"\u003eSheng Wu\u003c/a\u003e to introduce\nApache SkyWalking. This is a 1.5 hours …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2021-08-01-skywalking-8-intro/","title":"[Webinar] SkyWalking 8.x Introduction"},{"body":"SkyWalking 8.7.0 is released. Go to downloads page to find release tars. Changes by Version\nProject Extract dependency management to a bom. Add JDK 16 to test matrix. DataCarrier consumer add a new event notification, call nothingToConsume method if the queue has no element to consume. Build and push snapshot Docker images to GitHub Container Registry, this is only for people who want to help to test the master branch codes, please don\u0026rsquo;t use in production environments. Java Agent Supports modifying span attributes in async mode. Agent supports the collection of JVM arguments and jar dependency information. [Temporary] Support authentication for log report channel. This feature and grpc channel is going to be removed after Satellite 0.2.0 release. Remove deprecated gRPC method, io.grpc.ManagedChannelBuilder#nameResolverFactory. See gRPC-java 7133 for more details. Add Neo4j-4.x plugin. Correct profile.duration to profile.max_duration in the default agent.config file. Fix the response time of gRPC. Support parameter collection for SqlServer. Add ShardingSphere-5.0.0-beta plugin. Fix some method exception error. Fix async finish repeatedly in spring-webflux-5.x-webclient plugin. Add agent plugin to support Sentinel. Move ehcache-2.x plugin as an optional plugin. Support guava-cache plugin. Enhance the compatibility of mysql-8.x-plugin plugin. Support Kafka SASL login module. Fix gateway plugin async finish repeatedly when fallback url configured. Chore: polish methods naming for Spring-Kafka plugins. Remove plugins for ShardingSphere legacy version. Update agent plugin for ElasticJob GA version Remove the logic of generating instance name in KafkaServiceManagementServiceClient class. Improve okhttp plugin performance by optimizing Class.getDeclaredField(). Fix GRPCLogClientAppender no context warning. Fix spring-webflux-5.x-webclient-plugin NPE. OAP-Backend Disable Spring sleuth meter analyzer by default. Only count 5xx as error in Envoy ALS receiver. Upgrade apollo core caused by CVE-2020-15170. Upgrade kubernetes client caused by CVE-2020-28052. Upgrade Elasticsearch 7 client caused by CVE-2020-7014. Upgrade jackson related libs caused by CVE-2018-11307, CVE-2018-14718 ~ CVE-2018-14721, CVE-2018-19360 ~ CVE-2018-19362, CVE-2019-14379, CVE-2019-14540, CVE-2019-14892, CVE-2019-14893, CVE-2019-16335, CVE-2019-16942, CVE-2019-16943, CVE-2019-17267, CVE-2019-17531, CVE-2019-20330, CVE-2020-8840, CVE-2020-9546, CVE-2020-9547, CVE-2020-9548, CVE-2018-12022, CVE-2018-12023, CVE-2019-12086, CVE-2019-14439, CVE-2020-10672, CVE-2020-10673, CVE-2020-10968, CVE-2020-10969, CVE-2020-11111, CVE-2020-11112, CVE-2020-11113, CVE-2020-11619, CVE-2020-11620, CVE-2020-14060, CVE-2020-14061, CVE-2020-14062, CVE-2020-14195, CVE-2020-24616, CVE-2020-24750, CVE-2020-25649, CVE-2020-35490, CVE-2020-35491, CVE-2020-35728 and CVE-2020-36179 ~ CVE-2020-36190. Exclude log4j 1.x caused by CVE-2019-17571. Upgrade log4j 2.x caused by CVE-2020-9488. Upgrade nacos libs caused by CVE-2021-29441 and CVE-2021-29442. Upgrade netty caused by CVE-2019-20444, CVE-2019-20445, CVE-2019-16869, CVE-2020-11612, CVE-2021-21290, CVE-2021-21295 and CVE-2021-21409. Upgrade consul client caused by CVE-2018-1000844, CVE-2018-1000850. Upgrade zookeeper caused by CVE-2019-0201, zookeeper cluster coordinator plugin now requires zookeeper server 3.5+. Upgrade snake yaml caused by CVE-2017-18640. Upgrade embed tomcat caused by CVE-2020-13935. Upgrade commons-lang3 to avoid potential NPE in some JDK versions. OAL supports generating metrics from events. Support endpoint name grouping by OpenAPI definitions. Concurrent create PrepareRequest when persist Metrics Fix CounterWindow increase computing issue. Performance: optimize Envoy ALS analyzer performance in high traffic load scenario (reduce ~1cpu in ~10k RPS). Performance: trim useless metadata fields in Envoy ALS metadata to improve performance. Fix: slowDBAccessThreshold dynamic config error when not configured. Performance: cache regex pattern and result, optimize string concatenation in Envy ALS analyzer. Performance: cache metrics id and entity id in Metrics and ISource. Performance: enhance persistent session mechanism, about differentiating cache timeout for different dimensionality metrics. The timeout of the cache for minute and hour level metrics has been prolonged to ~5 min. Performance: Add L1 aggregation flush period, which reduce the CPU load and help young GC. Support connectTimeout and socketTimeout settings for ElasticSearch6 and ElasticSearch7 storages. Re-implement storage session mechanism, cached metrics are removed only according to their last access timestamp, rather than first time. This makes sure hot data never gets removed unexpectedly. Support session expired threshold configurable. Fix InfluxDB storage-plugin Metrics#multiGet issue. Replace zuul proxy with spring cloud gateway 2.x. in webapp module. Upgrade etcd cluster coordinator and dynamic configuration to v3.x. Configuration: Allow configuring server maximum request header size and ES index template order. Add thread state metric and class loaded info metric to JVMMetric. Performance: compile LAL DSL statically and run with type checked. Add pagination to event query protocol. Performance: optimize Envoy error logs persistence performance. Support envoy cluster manager metrics. Performance: remove the synchronous persistence mechanism from batch ElasticSearch DAO. Because the current enhanced persistent session mechanism, don\u0026rsquo;t require the data queryable immediately after the insert and update anymore. Performance: share flushInterval setting for both metrics and record data, due to synchronous persistence mechanism removed. Record flush interval used to be hardcoded as 10s. Remove syncBulkActions in ElasticSearch storage option. Increase the default bulkActions(env, SW_STORAGE_ES_BULK_ACTIONS) to 5000(from 1000). Increase the flush interval of ElasticSearch indices to 15s(from 10s) Provide distinct for elements of metadata lists. Due to the more aggressive asynchronous flush, metadata lists have more chances including duplicate elements. Don\u0026rsquo;t need this as indicate anymore. Reduce the flush period of hour and day level metrics, only run in 4 times of regular persistent period. This means default flush period of hour and day level metrics are 25s * 4. Performance: optimize IDs read of ElasticSearch storage options(6 and 7). Use the physical index rather than template alias name. Adjust index refresh period as INT(flushInterval * 2/3), it used to be as same as bulk flush period. At the edge case, in low traffic(traffic \u0026lt; bulkActions in the whole period), there is a possible case, 2 period bulks are included in one index refresh rebuild operation, which could cause version conflicts. And this case can\u0026rsquo;t be fixed through core/persistentPeriod as the bulk fresh is not controlled by the persistent timer anymore. The core/maxSyncOperationNum setting(added in 8.5.0) is removed due to metrics persistence is fully asynchronous. The core/syncThreads setting(added in 8.5.0) is removed due to metrics persistence is fully asynchronous. Optimization: Concurrency mode of execution stage for metrics is removed(added in 8.5.0). Only concurrency of prepare stage is meaningful and kept. Fix -meters metrics topic isn\u0026rsquo;t created with namespace issue Enhance persistent session timeout mechanism. Because the enhanced session could cache the metadata metrics forever, new timeout mechanism is designed for avoiding this specific case. Fix Kafka transport topics are created duplicated with and without namespace issue Fix the persistent session timeout mechanism bug. Fix possible version_conflict_engine_exception in bulk execution. Fix PrometheusMetricConverter may throw an IllegalArgumentException when convert metrics to SampleFamily Filtering NaN value samples when build SampleFamily Add Thread and ClassLoader Metrics for the self-observability and otel-oc-rules Simple optimization of trace sql query statement. Avoid \u0026ldquo;select *\u0026rdquo; query method Introduce dynamical logging to update log configuration at runtime Fix Kubernetes ConfigMap configuration center doesn\u0026rsquo;t send delete event Breaking Change: emove qps and add rpm in LAL UI Fix the date component for log conditions. Fix selector keys for duplicate options. Add Python celery plugin. Fix default config for metrics. Fix trace table for profile ui. Fix the error of server response time in the topology. Fix chart types for setting metrics configure. Fix logs pages number. Implement a timeline for Events in a new page. Fix style for event details. Documentation Add FAQ about Elasticsearch exception type=version_conflict_engine_exception since 8.7.0 Add Self Observability service discovery (k8s). Add sending Envoy Metrics to OAP in envoy 1.19 example and bump up to Envoy V3 api. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking 8.7.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nChanges by Version\u003c/p\u003e\n\u003ch4 id=\"project\"\u003eProject …\u003c/h4\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-8-7-0/","title":"Release Apache SkyWalking APM 8.7.0"},{"body":"SkyWalking Client JS 0.6.0 is released. Go to downloads page to find release tars.\nSeparate production and development environments when building. Upgrade packages to fix vulnerabilities. Fix headers could be null . Fix catching errors for http requests. Fix the firstReportedError is calculated with more types of errors. ","excerpt":"\u003cp\u003eSkyWalking Client JS 0.6.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eSeparate …\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-client-js-0-6-0/","title":"Release Apache SkyWalking Client JS 0.6.0"},{"body":"\nSkyWalking is an open source APM (application performance monitor) system, especially designed for microservices, cloud native, and container-based architectures.\nFrom 2020, it has dominated the open source APM market in China, and expanded aggressively in North American, Europe and Asia\u0026rsquo;s other countries.\nWith over 6 years (2015-2021) of development, driven by the global open source community, SkyWalking now provides full stack observability covering metrics, tracing and logging, plus event detector, which are built based on various native and ecosystem solutions.\nLanguage agent-based(Java, Dot Net, Golang, PHP, NodeJS, Python, C++, LUA) in-process monitoring, is as powerful as commercial APM vendors\u0026rsquo; agents. Mostly auto-instrumentation, and good interactivity. Service Mesh Observability, working closely with Envoy and Istio teams. Transparent integration of popular metrics ecosystem. Accept metrics from Prometheus SDK, OpenTelemetry collectors, Zabbix agents, etc. Log collection with analysis capability from FluentD, Fluent-bit, Filebeat, etc. agents. Infrastructure monitoring, such as Linux and k8s, is out of the box. The SkyWalking ecosystem was started by very few people. The community drives the project to cover real scenarios, from tracing to the whole APM field. Even today, more professional open source developers, powered by the vendors behind them, are bringing the project to a different level.\nTypically and most attractively, SkyWalking is going to build the first known open source APM specific database in the world, at least providing\nTime series-based database engine. Support traces/logs and metrics in the database core level. High performance with cluster mode and HPA. Reasonable resource cost. We nearly doubled the number of contributors in the last year, from ~300 to over 500. The whole community is very energetic. Here, we want to thank our 47 committers(28 PMC members included), listed here, and over 400 other contributors.\nWe together built this humongous Apache Top Level project, and proved the stronge competitiveness of an open-source project.\nThis is a hard-won and impressive achievement. We won\u0026rsquo;t stop here. The trend is there, the ground is solid. We are going to build the top-level APM system relying on our open-source community.\n500 Contributors List GitHub 1095071913 182148432** 295198088** 394102339** 437376068** 50168383 55846420** 826245622** 844067874 Ahoo-Wang AirTrioa AlexanderWert AlseinX AngryMills Ax1an BFergerson BZFYS CalvinKirs CharlesMaster ChaunceyLin5152 CommissarXia Cvimer DeadLion Doublemine Du-fei ElderJames EvanLjp FatihErdem FeynmanZhou Fine0830 FingerLiu FrankyXu Gallardot GerryYuan HackerRookie HarryFQ Heguoya Hen1ng HendSame Humbertzhang IanCao IluckySi Indifer J-Cod3r JaredTan95 Jargon96 Jijun JoeKerouac JohnNiang Johor03 Jozdortraz Jtrust Just-maple KangZhiDong LazyLei LiWenGu Lin1997 Linda-pan LiteSun Liu-XinYuan MiracleDx Miss-you MoGuGuai-hzr MrYzys O-ll-O Patrick0308 QHWG67 Qiliang QuanjieDeng RandyAbernethy RedzRedz Runrioter SataQiu ScienJus SevenBlue2018 ShaoHans Shikugawa SoberChina SummerOfServenteen Switch-vov TJ666 Technoboy- TerrellChen TeslaCN TheRealHaui TinyAllen TomMD ViberW Videl WALL-E WeihanLi WildWolfBang WillemJiang Wooo0 XhangUeiJong Xlinlin YczYanchengzhe Yebemeto YoungHu YunaiV YunfengGao Z-Beatles ZS-Oliver ZhHong ZhuoSiChen a198720 a1vin-tian a526672351 acurtain adamni135 adermxzs adriancole** aeolusheath agile6v aix3 aiyanbo ajanthan alexkarezin alonelaval amogege amwyyyy andyliyuze andyzzl aoxls arugal ascrutae ascrutae** augustowebd aviaviavi bai-yang beckhampu beckjin beiwangnull bigflybrother bootsrc bostin brucewu-fly buxingzhe buzuotaxuan bwh12398** c feng c1ay candyleer carllhw carlvine500 carrypann cheenursn cheetah012 chenbeitang chenglei** chengshiwen chenmudu chenpengfei chenvista chess-equality chestarss chidaodezhongsheng chopin-d clevertension clk1st cngdkxw cnlangzi codeglzhang codelipenghui coder-yqj coki230 compilerduck constanine coolbeevip crystaldust cui-liqiang cuiweiwei cutePanda123 cyberdak cyejing cyhii dafu-wu dagmom dalekliuhan** darcydai dengliming devkanro devon-ye dickens7 dimaaan dingdongnigetou dio divyakumarjain dmsolr dominicqi donbing007 dsc6636926 dvsv2 dzx2018 echooymxq efekaptan elk-g emschu eoeac evanljp** evanxuhe feelwing1314 fgksgf fredster33 fuhuo fulmicoton fushiqinghuan111 geektcp geomonlin ggndnn gitter-badger givingwu glongzh gnr163 gonedays grissom-grissom grissomsh guodongq guyukou gxthrj gy09535 gzshilu hailin0 hanahmily haotian2015 haoyann hardzhang harvies heihaozi hepyu heyanlong hi-sb honganan horber hsoftxl huangyoje huliangdream huohuanhuan iluckysi innerpeacez itsvse jasper-zsh jbampton jialong121 jinlongwang jjlu521016 jjtyro jmjoy jsbxyyx justeene juzhiyuan jy00464346 kaanid kagaya85 karott kayleyang kevinyyyy kezhenxu94 kikupotter kilingzhang killGC kkl129 klboke ksewen kuaikuai kun-song kylixs landonzeng langke93 langyan1022 langyizhao lazycathome leemove leizhiyuan libinglong lijial lilien1010 limfriend linkinshi linliaoy liqiangz liu-junchi liufei** liuhaoXD liuhaoyang liuweiyi** liuyanggithup liuzhengyang liweiv lixin40** lizl9** lkxiaolou llissery louis-zhou lpcy lpf32 lsyf lucperkins lujiajing1126 lunamagic1978 lunchboxav lxin96** lxliuxuankb lytscu lyzhang1999 mage3k makefriend8 makingtime mantuliu maolie margauxcabrera masterxxo maxiaoguang64 me** membphis mestarshine mgsheng michaelsembwever mikkeschiren ming_flycash** minquan.chen** misaya momo0313 moonming mrproliu mrproliu** muyun12 nacx neatlife neeuq nic-chen nickwongwong nikitap492 nileblack nisiyong novayoung oatiz oflebbe olzhy onecloud360 osiriswd panniyuyu peng-yongsheng pengweiqhca potiuk probeyang purgeyao qijianbo010 qinhang3 qiuyu-d qjgszzx qq362220083 qqeasonchen qxo ralphgj raybi-asus refactor2 remicollet rlenferink rootsongjc rovast ruibaby s00373198 scolia sdanzo seifeHu sergicastro shiluo34 sikelangya simonlei sk163 snakorse songzhendong songzhian songzhian** sonxy spacewander stalary stenio2011 stevehu stone-wlg sungitly surechen swartz-k sxzaihua tangxqa tanjunchen tankilo tanzhen** taskmgr tbdpmi terranhu terrymanu tevahp thanq thebouv tianyk tianyuak tincopper tinyu0 tom-pytel tristaZero tristan-tsl trustin tsuilouis tuohai666 tzsword-2020 tzy1316106836 vcjmhg viktoryi vision-ken viswaramamoorthy wallezhang wang-yeliang wang_weihan** wangrzneu wankai123 wbpcode web-xiaxia webb2019 weiqiang-w weiqiang333 wendal wengangJi wenjianzhang whfjam whl12345 willseeyou wilsonwu wind2008hxy wingwong-knh withlin wl4g wqr2016 wu-sheng wuguangkuo wujun8 wuwen5 wuxingye x22x22 xbkaishui xcaspar xdRight xiaoweiyu** xiaoxiangmoe xiaoy00 xinfeingxia85 xingren23 xinzhuxiansheng xonze xuanyu66 xuchangjunjx xudianyang yanbw yanfch yang-xiaodong yangxb2010000 yanickxia yanmaipian yanmingbi yantaowu yaojingguo yaowenqiang yazong ychandu ycoe yimeng yu199195 yuqichou yushuqiang** yuyujulin yxudong yymoth zaunist zaygrzx zcai2 zeaposs zhang98722 zhanghao001 zhangjianweibj zhangkewei zhangsean zhangxin** zhaoyuguang zhe1926 zhentaoJin zhongjianno1** zhousiliang163 zhuCheer zhyyu zifeihan zijin-m zkscpqm zoidbergwill zoumingzm zouyx zpf1989 zshit zxbu zygfengyuwuzu ","excerpt":"\u003cp\u003e\u003cimg src=\"500-mark.png\" alt=\"\"\u003e\u003c/p\u003e\n\u003cp\u003eSkyWalking is an open source APM (application performance monitor) system, especially designed for …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2021-07-12-500-contributors-mark/","title":"[Community win] SkyWalking achieved 500 contributors milestone."},{"body":"时间：2021 年 6 月 26 日\n地点：北京市海淀区西格玛大厦 B1 多功能厅\n视频回放：见 Bilibili\nApache SkyWalking Landscape 吴晟 Sheng Wu. Tetrate Founding Engineer, Apache Software Foundation board director. SkyWalking founder. SkyWalking 2020-2021 年发展和后续计划\n微服务可观测性分析平台的探索与实践 凌若川 腾讯高级工程师 可观测性分析平台作为云原生时代微服务系统基础组件，开放性与性能是决定平台价值的核心要素。 复杂微服务应用场景与海量多维链路数据，对可观测性分析平台在开放性设计和各环节高性能实现带来诸多挑战。 本次分享中将重点梳理腾讯云微服务团队在构建云原生可观测性分析平台过程中遇到的挑战，介绍我们在架构设计与实现方面的探索与实践。\n云原生时代微服务可观测性平台面临的性能与可用性挑战 腾讯云在构建高性能微服务可观测性分析平台的探索与实践 微服务可观测性分析平台架构的下一阶段演进方向展望 BanyanDB 数据模型背后的逻辑 高洪涛 Hongtao Gao. Tetrate SRE, SkyWalking PMC, Apache ShardingSphere PMC. BanyanDB 作为为处理 Apache SkyWalking 产生的 trace，log 和 metric 的数据而特别设计的数据库，其背后数据模型的抉择是非常与众不同的。 在本次分享中，我将根据 RUM 猜想来讨论为什么 BanyanDB 使用的数据模型对于 APM 数据而言是更加高效和可靠的。\n通过本次分享，观众可以：\n理解数据库设计的取舍 了解 BanyanDB 的数据模型 认识到该模型对于 APM 类数据有特定的优势 Apache SkyWalking 如何做前端监控 范秋霞 Qiuxia Fan，Tetrate FE SRE，SkyWalking PMC. Apache SkyWalking 对前端进行了监控与跟踪，分别有 Metric, Log, Trace 三部分。本次分享我会介绍页面性能指标的收集与计算，同时用案列进行分析，也会讲解 Log 的采集方法以及 Source Map 错误定位的实施。最后介绍浏览器端 Requets 的跟踪方法。\n通过本次分享，观众可以：\n了解页面的性能指标以及收集计算方法 了解前端如何做错误日志收集 如何对页面请求进行跟踪以及跟踪的好处 一名普通工程师，该如何正确的理解开源精神？ 王晔倞 Yeliang Wang. API7 Partner / Product VP. 开源精神，那也许是一种给于和获取的平衡，有给于才能有获取，有获取才会有给于的动力。无需指责别人只会获取，我们应该懂得开源是一种创造方式，一个没有创造欲和创造力的人加入开源也是无用的。\n通过本次分享，观众可以：\n为什么国内一些程序员会对开源产生误解？ 了解 “开源≠自由≠非商业” 的来龙去脉。 一名普通工程师，如何高效地向开源社区做贡献？ 可观测性技术生态和 OpenTelemetry 原理及实践 陈一枭 腾讯. OpenTelemetry docs-cn maintainer、Tencent OpenTelemetry OTeam 创始人 综述云原生可观测性技术生态，介绍 OpenTracing，OpenMetrics，OpenTelemetry 等标准演进。介绍 OpenTelemetry 存在价值意义，介绍 OpenTelemetry 原理及其整体生态规划。介绍腾讯在 OpenTelemetry 方面的实践。\n本次分享内容如下：\n云原生可观测性技术简介 OpenTelemetry 及其它规范简介 OpenTelemetry 原理 OpenTelemetry 在腾讯的应用及实践 Apache SkyWalking 事件采集系统更快定位故障 柯振旭 Zhenxu Ke，Tetrate SRE, Apache SkyWalking PMC. Apache Incubator PMC. Apache Dubbo committer. 通过本次分享，听众可以：\n了解 SkyWalking 的事件采集系统； 了解上报事件至 SkyWalking 的多种方式； 学习如何利用 SkyWalking 采集的事件结合 metrics，分析目标系统的性能问题； 可观测性自动注入技术原理探索与实践 詹启新 Tencnet OpenTelemetry Oteam PMC 在可观测领域中自动注入已经成为重要的组成部分之一，其优异简便的使用方式并且可同时覆盖到链路、指标、日志，大大降低了接入成本及运维成本，属于友好的一种接入方式； 本次分享将介绍 Java 中的字节码注入技术原理，及在可观测领域的应用实践\n常用的自动注入技术原理简介 介绍可观测性在 Java 落地的要点 opentelemetry-java-instrumentation 的核心原理及实现 opentelemetry 自动注入的应用实践 如何利用 Apache APISIX 提升 Nginx 的可观测性 金卫 Wei Jin, API7 Engineer Apache SkyWalking committer. Apache apisix-ingress-controller Founder. Apache APISIX PMC. 在云原生时代，动态和可观测性是 API 网关的标准特性。Apache APISIX 不仅覆盖了 Nginx 的传统功能，在可观测性上也和 SkyWalking 深度合作，大大提升了服务治理能力。本次分享会介绍如何无痛的提升 Nginx 的可观测性和 APISIX 在未来可观测性方面的规划。\n通过本次分享，观众可以：\n通过 Apache APISIX 实现观测性的几种手段. 了解 Apache APISIX 高效且易用的秘诀. 结合 Apache skywalking 进一步提升可观测性. ","excerpt":"\u003cp\u003e时间：2021 年 6 月 26 日\u003c/p\u003e\n\u003cp\u003e地点：北京市海淀区西格玛大厦 B1 多功能厅\u003c/p\u003e\n\u003cp\u003e视频回放：见 \u003ca href=\"https://space.bilibili.com/390683219/channel/detail?cid=190669\"\u003eBilibili\u003c/a\u003e\u003c/p\u003e\n\u003ch4 id=\"apache-skywalking-landscape\"\u003e\u003ca href=\"https://www.bilibili.com/video/BV1HV411W7sr\"\u003eApache SkyWalking Landscape\u003c/a\u003e\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e吴晟 Sheng Wu. …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/zh/skywalking-day-2021/","title":"[视频] SkyWalking Day 2021 演讲视频"},{"body":"SkyWalking CLI 0.7.0 is released. Go to downloads page to find release tars.\nFeatures\nAdd GitHub Action for integration of event reporter Bug Fixes\nFix metrics top can\u0026rsquo;t infer the scope automatically Chores\nUpgrade dependency crypto Refactor project to use goapi Move parseScope to pkg Update release doc ","excerpt":"\u003cp\u003eSkyWalking CLI 0.7.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eFeatures\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAdd GitHub …\u003c/li\u003e\u003c/ul\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cli-0-7-0/","title":"Release Apache SkyWalking CLI 0.7.0"},{"body":"SkyWalking 8.6.0 is released. Go to downloads page to find release tars. Changes by Version\nProject Add OpenSearch as storage option. Upgrade Kubernetes Java client dependency to 11.0. Fix plugin test script error in macOS. Java Agent Add trace_segment_ref_limit_per_span configuration mechanism to avoid OOM. Improve GlobalIdGenerator performance. Add an agent plugin to support elasticsearch7. Add jsonrpc4j agent plugin. new options to support multi skywalking cluster use same kafka cluster(plugin.kafka.namespace) resolve agent has no retries if connect kafka cluster failed when bootstrap Add Seata in the component definition. Seata plugin hosts on Seata project. Extended Kafka plugin to properly trace consumers that have topic partitions directly assigned. Support Kafka consumer 2.8.0. Support print SkyWalking context to logs. Add MessageListener enhancement in pulsar plugin. fix a bug that spring-mvc set an error endpoint name if the controller class annotation implements an interface. Add an optional agent plugin to support mybatis. Add spring-cloud-gateway-3.x optional plugin. Add okhttp-4.x plugin. Fix NPE when thrift field is nested in plugin thrift Fix possible NullPointerException in agent\u0026rsquo;s ES plugin. Fix the conversion problem of float type in ConfigInitializer. Fixed part of the dynamic configuration of ConfigurationDiscoveryService that does not take effect under certain circumstances. Introduce method interceptor API v2 Fix ClassCast issue for RequestHolder/ResponseHolder. fixed jdk-threading-plugin memory leak. Optimize multiple field reflection operation in Feign plugin. Fix trace-ignore-plugin TraceIgnorePathPatterns can\u0026rsquo;t set empty value OAP-Backend BugFix: filter invalid Envoy access logs whose socket address is empty. Fix K8s monitoring the incorrect metrics calculate. Loop alarm into event system. Support alarm tags. Support WeLink as a channel of alarm notification. Fix: Some defensive codes didn\u0026rsquo;t work in PercentileFunction combine. CVE: fix Jetty vulnerability. https://nvd.nist.gov/vuln/detail/CVE-2019-17638 Fix: MAL function would miss samples name after creating new samples. perf: use iterator.remove() to remove modulesWithoutProvider Support analyzing Envoy TCP access logs and persist error TCP logs. Fix: Envoy error logs are not persisted when no metrics are generated Fix: Memory leakage of low version etcd client. fix-issue Allow multiple definitions as fallback in metadata-service-mapping.yaml file and k8sServiceNameRule. Fix: NPE when configmap has no data. Fix: Dynamic Configuration key slowTraceSegmentThreshold not work Fix: != is not supported in oal when parameters are numbers. Include events of the entity(s) in the alarm. Support native-json format log in kafka-fetcher-plugin. Fix counter misuse in the alarm core. Alarm can\u0026rsquo;t be triggered in time. Events can be configured as alarm source. Make the number of core worker in meter converter thread pool configurable. Add HTTP implementation of logs reporting protocol. Make metrics exporter still work even when storage layer failed. Fix Jetty HTTP TRACE issue, disable HTTP methods except POST. CVE: upgrade snakeyaml to prevent billion laughs attack in dynamic configuration. polish debug logging avoids null value when the segment ignored. UI Add logo for kong plugin. Add apisix logo. Refactor js to ts for browser logs and style change. When creating service groups in the topology, it is better if the service names are sorted. Add tooltip for dashboard component. Fix style of endpoint dependency. Support search and visualize alarms with tags. Fix configurations on dashboard. Support to configure the maximum number of displayed items. After changing the durationTime, the topology shows the originally selected group or service. remove the no use maxItemNum for labeled-value metric, etc. Add Azure Functions logo. Support search Endpoint use keyword params in trace view. Add a function which show the statistics infomation during the trace query. Remove the sort button at the column of Type in the trace statistics page. Optimize the APISIX icon in the topology. Implement metrics templates in the topology. Visualize Events on the alarm page. Update duration steps in graphs for Trace and Log. Documentation Polish k8s monitoring otel-collector configuration example. Print SkyWalking context to logs configuration example. Update doc about metrics v2 APIs. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking 8.6.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nChanges by Version\u003c/p\u003e\n\u003ch4 id=\"project\"\u003eProject …\u003c/h4\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-8-6-0/","title":"Release Apache SkyWalking APM 8.6.0"},{"body":" Abstract Apache SkyWalking hosts SkyWalkingDay Conference 2021 in June 26th, jointly with Tencent and Tetrate.\nWe are going to share SkyWalking\u0026rsquo;s roadmap, features, product experiences and open source culture.\nWelcome to join us.\nVenue Addr./地址 北京市海淀区西格玛大厦B1多功能厅\nDate June 26th.\nRegistration For Free Register for onsite or online\nSessions 10:00 - 10:20 Apache SkyWalking Landscape 吴晟 Sheng Wu. Tetrate Founding Engineer, Apache Software Foundation board director. SkyWalking founder. SkyWalking 2020-2021年发展和后续计划\n10:20 - 10:50 微服务可观测性分析平台的探索与实践 凌若川 腾讯高级工程师 可观测性分析平台作为云原生时代微服务系统基础组件，开放性与性能是决定平台价值的核心要素。 复杂微服务应用场景与海量多维链路数据，对可观测性分析平台在开放性设计和各环节高性能实现带来诸多挑战。 本次分享中将重点梳理腾讯云微服务团队在构建云原生可观测性分析平台过程中遇到的挑战，介绍我们在架构设计与实现方面的探索与实践。\n云原生时代微服务可观测性平台面临的性能与可用性挑战 腾讯云在构建高性能微服务可观测性分析平台的探索与实践 微服务可观测性分析平台架构的下一阶段演进方向展望 10:50 - 11:20 BanyanDB数据模型背后的逻辑 高洪涛 Hongtao Gao. Tetrate SRE, SkyWalking PMC, Apache ShardingSphere PMC. BanyanDB作为为处理Apache SkyWalking产生的trace，log和metric的数据而特别设计的数据库，其背后数据模型的抉择是非常与众不同的。 在本次分享中，我将根据RUM猜想来讨论为什么BanyanDB使用的数据模型对于APM数据而言是更加高效和可靠的。\n通过本次分享，观众可以：\n理解数据库设计的取舍 了解BanyanDB的数据模型 认识到该模型对于APM类数据有特定的优势 11:20 - 11:50 Apache SkyWalking 如何做前端监控 范秋霞 Qiuxia Fan，Tetrate FE SRE，SkyWalking PMC. Apache SkyWalking对前端进行了监控与跟踪，分别有Metric, Log, Trace三部分。本次分享我会介绍页面性能指标的收集与计算，同时用案列进行分析，也会讲解Log的采集方法以及Source Map错误定位的实施。最后介绍浏览器端Requets的跟踪方法。\n通过本次分享，观众可以：\n了解页面的性能指标以及收集计算方法 了解前端如何做错误日志收集 如何对页面请求进行跟踪以及跟踪的好处 午休 13:30 - 14:00 一名普通工程师，该如何正确的理解开源精神？ 王晔倞 Yeliang Wang. API7 Partner / Product VP. 开源精神，那也许是一种给于和获取的平衡，有给于才能有获取，有获取才会有给于的动力。无需指责别人只会获取，我们应该懂得开源是一种创造方式，一个没有创造欲和创造力的人加入开源也是无用的。\n通过本次分享，观众可以：\n为什么国内一些程序员会对开源产生误解？ 了解 “开源≠自由≠非商业” 的来龙去脉。 一名普通工程师，如何高效地向开源社区做贡献？ 14:00 - 14:30 可观测性技术生态和OpenTelemetry原理及实践 陈一枭 腾讯. OpenTelemetry docs-cn maintainer、Tencent OpenTelemetry OTeam创始人 综述云原生可观测性技术生态，介绍OpenTracing，OpenMetrics，OpenTelemetry等标准演进。介绍OpenTelemetry存在价值意义，介绍OpenTelemetry原理及其整体生态规划。介绍腾讯在OpenTelemetry方面的实践。\n本次分享内容如下：\n云原生可观测性技术简介 OpenTelemetry及其它规范简介 OpenTelemetry原理 OpenTelemetry在腾讯的应用及实践 14:30 - 15:10 利用 Apache SkyWalking 事件采集系统更快定位故障 柯振旭 Zhenxu Ke，Tetrate SRE, Apache SkyWalking PMC. Apache Incubator PMC. Apache Dubbo committer. 通过本次分享，听众可以：\n了解 SkyWalking 的事件采集系统； 了解上报事件至 SkyWalking 的多种方式； 学习如何利用 SkyWalking 采集的事件结合 metrics，分析目标系统的性能问题； 15:10 - 15:30 茶歇 15:30 - 16:00 可观测性自动注入技术原理探索与实践 詹启新 Tencnet OpenTelemetry Oteam PMC 在可观测领域中自动注入已经成为重要的组成部分之一，其优异简便的使用方式并且可同时覆盖到链路、指标、日志，大大降低了接入成本及运维成本，属于友好的一种接入方式； 本次分享将介绍Java中的字节码注入技术原理，及在可观测领域的应用实践\n常用的自动注入技术原理简介 介绍可观测性在Java落地的要点 opentelemetry-java-instrumentation的核心原理及实现 opentelemetry自动注入的应用实践 16:00 - 16:30 如何利用 Apache APISIX 提升 Nginx 的可观测性 金卫 Wei Jin, API7 Engineer Apache SkyWalking committer. Apache apisix-ingress-controller Founder. Apache APISIX PMC. 在云原生时代，动态和可观测性是 API 网关的标准特性。Apache APISIX 不仅覆盖了 Nginx 的传统功能，在可观测性上也和 SkyWalking 深度合作，大大提升了服务治理能力。本次分享会介绍如何无痛的提升 Nginx 的可观测性和 APISIX 在未来可观测性方面的规划。\n通过本次分享，观众可以：\n通过 Apache APISIX 实现观测性的几种手段. 了解 Apache APISIX 高效且易用的秘诀. 结合 Apache skywalking 进一步提升可观测性. 16:35 抽奖，结束 Sponsors Tencent Tetrate SegmentFault 思否 Anti-harassment policy SkyWalkingDay is dedicated to providing a harassment-free experience for everyone. We do not tolerate harassment of participants in any form. Sexual language and imagery will also not be tolerated in any event venue. Participants violating these rules may be sanctioned or expelled without a refund, at the discretion of the event organizers. Our anti-harassment policy can be found at Apache website.\nContact Us Send mail to dev@skywalking.apache.org.\n","excerpt":"\u003cimg src=\"skywalkingday.png\"\u003e\n\u003ch2 id=\"abstract\"\u003eAbstract\u003c/h2\u003e\n\u003cp\u003eApache SkyWalking hosts SkyWalkingDay Conference 2021 in June 26th, jointly with Tencent …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/skywalkingday-2021/","title":"SkyWalkingDay Conference 2021, relocating at Beijing"},{"body":"SkyWalking NodeJS 0.3.0 is released. Go to downloads page to find release tars.\nAdd ioredis plugin. (#53) Endpoint cold start detection and marking. (#52) Add mysql2 plugin. (#54) Add AzureHttpTriggerPlugin. (#51) Add Node 15 into test matrix. (#45) Segment reference and reporting overhaul. (#50) Add http ignore by method. (#49) Add secure connection option. (#48) BugFix: wrong context during many async spans. (#46) Add Node Mongoose Plugin. (#44) ","excerpt":"\u003cp\u003eSkyWalking NodeJS 0.3.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAdd ioredis plugin. …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-nodejs-0-3-0/","title":"Release Apache SkyWalking for NodeJS 0.3.0"},{"body":"SkyWalking Client JS 0.5.1 is released. Go to downloads page to find release tars.\nAdd noTraceOrigins option. Fix wrong URL when using relative path. Catch frames errors. Get response.body as a stream with the fetch API. Support reporting multiple logs. Support typescript project. ","excerpt":"\u003cp\u003eSkyWalking Client JS 0.5.1 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eAdd …\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-client-js-0-5-1/","title":"Release Apache SkyWalking Client JS 0.5.1"},{"body":"SkyWalking Kong Agent 0.1.1 is released. Go to downloads page to find release tars.\nEstablish the SkyWalking Kong Agent. ","excerpt":"\u003cp\u003eSkyWalking Kong Agent 0.1.1 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eEstablish the …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-kong-0-1-1/","title":"Release Apache SkyWalking Kong 0.1.1"},{"body":"B站视频地址\n","excerpt":"\u003cp\u003e\u003ca href=\"https://www.bilibili.com/video/BV1BQ4y1o7rA\"\u003eB站视频地址\u003c/a\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2021-05-09-summer-2021-asf20/","title":"[视频] 大咖说开源 第二季 第4期 | Apache软件基金会20年"},{"body":"We posted our Response to Elastic 2021 License Change blog 4 months ago. It doesn\u0026rsquo;t have a big impact in the short term, but because of the incompatibility between SSPL and Apache 2.0, we lost the chance of upgrading the storage server, which concerns the community and our users. So, we have to keep looking for a new option as a replacement.\nThere was an open source project, Open Distro for Elasticsearch, maintained by the AWS team. It is an Apache 2.0-licensed distribution of Elasticsearch enhanced with enterprise security, alerting, SQL, and more. After Elastic relicensed its projects, we talked with their team, and they have an agenda to take over the community leadship and keep maintaining Elasticsearch, as it was licensed by Apache 2.0. So, they are good to fork and continue.\nOn April 12th, 2021, AWS announced the new project, OpenSearch, driven by the community, which is initialized from people of AWS, Red Hat, SAP, Capital One, and Logz.io. Read this Introducing OpenSearch blog for more detail.\nOnce we had this news in public, we begin to plan the process of evaluating and testing OpenSearch as SkyWalking\u0026rsquo;s storage option. Read our issue.\nToday, we are glad to ANNOUNCE, OpenSearch could replace ElastcSearch as the storage, and it is still licensed under Apache 2.0.\nThis has been merged in the main stream, and you can find it in the dev doc already.\nOpenSearch OpenSearch storage shares the same configurations as Elasticsearch 7. In order to activate Elasticsearch 7 as storage, set storage provider to elasticsearch7. Please download the apache-skywalking-bin-es7.tar.gz if you want to use OpenSearch as storage.\nSkyWalking community will keep our eyes on the OpenSearch project, and look forward to their first GA release.\nNOTE: we have to add a warning NOTICE to the Elasticsearch storage doc:\nNOTICE: Elastic announced through their blog that Elasticsearch will be moving over to a Server Side Public License (SSPL), which is incompatible with Apache License 2.0. This license change is effective from Elasticsearch version 7.11. So please choose the suitable Elasticsearch version according to your usage.\n","excerpt":"\u003cp\u003eWe posted our \u003ca href=\"/blog/2021-01-17-elastic-change-license/\"\u003e\u003cstrong\u003eResponse to Elastic 2021 License Change\u003c/strong\u003e\u003c/a\u003e blog 4 months ago. It doesn\u0026rsquo;t have a big …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2021-05-09-opensearch-supported/","title":"OpenSearch, a new storage option to avoid ElasticSearch's SSPL"},{"body":"Hailin Wang(GitHub ID, hailin0) began his SkyWalking journey since Aug 23rd, 2020.\nHe is very active on the code contributions and brought several important features into the SkyWalking ecosystem.\nHe is on the 33rd of the contributor in the main repository[1], focuses on plugin contributions, and logs ecosystem integration, see his code contributions[2]. And also, he started a new and better way to make other open-source projects integrating with SkyWalking.\nHe used over 2 months to make the SkyWalking agent and its plugins as a part of Apache DolphinScheduler\u0026rsquo;s default binary distribution[3], see this PR[4]. This kind of example has affected further community development. Our PMC member, Yuguang Zhao, is using this way to ship our agent and plugins into the Seata project[5]. With SkyWalking\u0026rsquo;s growing, I would not doubt that this kind of integration would be more.\nThe SkyWalking accepts him as a new committer.\nWelcome Hailin Wang join the committer team.\n[1] https://github.com/apache/skywalking/graphs/contributors [2] https://github.com/apache/skywalking/commits?author=hailin0 [3] https://github.com/apache/dolphinscheduler/tree/1.3.6-prepare/ext/skywalking [4] https://github.com/apache/incubator-dolphinscheduler/pull/4852 [5] https://github.com/seata/seata/pull/3652\n","excerpt":"\u003cp\u003eHailin Wang(GitHub ID, hailin0) began his SkyWalking journey since Aug 23rd, 2020.\u003c/p\u003e\n\u003cp\u003eHe is very active …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-hailin-wang-as-new-committer/","title":"Welcome Hailin Wang as new committer"},{"body":"SkyWalking LUA Nginx 0.5.0 is released. Go to downloads page to find release tars.\nAdapt to Kong agent. Correct the version format luarock. ","excerpt":"\u003cp\u003eSkyWalking LUA Nginx 0.5.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAdapt to Kong …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-lua-nginx-0.5.0/","title":"Release Apache SkyWalking LUA Nginx 0.5.0"},{"body":"SkyWalking 8.5.0 is released. Go to downloads page to find release tars. Changes by Version\nProject Incompatible Change. Indices and templates of ElasticSearch(6/7, including zipkin-elasticsearch7) storage option have been changed. Update frontend-maven-plugin to 1.11.0, for Download node x64 binary on Apple Silicon. Add E2E test for VM monitoring that metrics from Prometheus node-exporter. Upgrade lombok to 1.18.16. Add Java agent Dockerfile to build Docker image for Java agent. Java Agent Remove invalid mysql configuration in agent.config. Add net.bytebuddy.agent.builder.AgentBuilder.RedefinitionStrategy.Listener to show detail message when redefine errors occur. Fix ClassCastException of log4j gRPC reporter. Fix NPE when Kafka reporter activated. Enhance gRPC log appender to allow layout pattern. Fix apm-dubbo-2.7.x-plugin memory leak due to some Dubbo RpcExceptions. Fix lettuce-5.x-plugin get null host in redis sentinel mode. Fix ClassCastException by making CallbackAdapterInterceptor to implement EnhancedInstance interface in the spring-kafka plugin. Fix NullPointerException with KafkaProducer.send(record). Support config agent.span_limit_per_segment can be changed in the runtime. Collect and report agent starting / shutdown events. Support jedis pipeline in jedis-2.x-plugin. Fix apm-toolkit-log4j-2.x-activation no trace Id in async log. Replace hbase-1.x-plugin with hbase-1.x-2.x-plugin to adapt hbase client 2.x Remove the close_before_method and close_after_method parameters of custom-enhance-plugin to avoid memory leaks. Fix bug that springmvc-annotation-4.x-plugin, witness class does not exist in some versions. Add Redis command parameters to \u0026lsquo;db.statement\u0026rsquo; field on Lettuce span UI for displaying more info. Fix NullPointerException with ReactiveRequestHolder.getHeaders. Fix springmvc reactive api can\u0026rsquo;t collect HTTP statusCode. Fix bug that asynchttpclient plugin does not record the response status code. Fix spanLayer is null in optional plugin(gateway-2.0.x-plugin gateway-2.1.x-plugin). Support @Trace, @Tag and @Tags work for static methods. OAP-Backend Allow user-defined JAVA_OPTS in the startup script. Metrics combination API supports abandoning results. Add a new concept \u0026ldquo;Event\u0026rdquo; and its implementations to collect events. Add some defensive codes for NPE and bump up Kubernetes client version to expose exception stack trace. Update the timestamp field type for LogQuery. Support Zabbix protocol to receive agent metrics. Update the Apdex metric combine calculator. Enhance MeterSystem to allow creating metrics with same metricName / function / scope. Storage plugin supports postgresql. Fix kubernetes.client.openapi.ApiException. Remove filename suffix in the meter active file config. Introduce log analysis language (LAL). Fix alarm httpclient connection leak. Add sum function in meter system. Remove Jaeger receiver. Remove the experimental Zipkin span analyzer. Upgrade the Zipkin Elasticsearch storage from 6 to 7. Require Zipkin receiver must work with zipkin-elasticsearch7 storage option. Fix DatabaseSlowStatementBuilder statement maybe null. Remove fields of parent entity in the relation sources. Save Envoy http access logs when error occurs. Fix wrong service_instance_sla setting in the topology-instance.yml. Fix wrong metrics name setting in the self-observability.yml. Add telemetry data about metrics in, metrics scraping, mesh error and trace in metrics to zipkin receiver. Fix tags store of log and trace on h2/mysql/pg storage. Merge indices by Metrics Function and Meter Function in Elasticsearch Storage. Fix receiver don\u0026rsquo;t need to get itself when healthCheck Remove group concept from AvgHistogramFunction. Heatmap(function result) doesn\u0026rsquo;t support labels. Support metrics grouped by scope labelValue in MAL, no need global same labelValue as before. Add functions in MAL to filter metrics according to the metric value. Optimize the self monitoring grafana dashboard. Enhance the export service. Add function retagByK8sMeta and opt type K8sRetagType.Pod2Service in MAL for k8s to relate pods and services. Using \u0026ldquo;service.istio.io/canonical-name\u0026rdquo; to replace \u0026ldquo;app\u0026rdquo; label to resolve Envoy ALS service name. Support k8s monitoring. Make the flushing metrics operation concurrent. Fix ALS K8SServiceRegistry didn\u0026rsquo;t remove the correct entry. Using \u0026ldquo;service.istio.io/canonical-name\u0026rdquo; to replace \u0026ldquo;app\u0026rdquo; label to resolve Envoy ALS service name. Append the root slash(/) to getIndex and getTemplate requests in ES(6 and 7) client. Fix disable statement not working. This bug exists since 8.0.0. Remove the useless metric in vm.yaml. UI Update selector scroller to show in all pages. Implement searching logs with date. Add nodejs 14 compiling. Fix trace id by clear search conditions. Search endpoints with keywords. Fix pageSize on logs page. Update echarts version to 5.0.2. Fix instance dependency on the topology page. Fix resolved url for vue-property-decorator. Show instance attributes. Copywriting grammar fix. Fix log pages tags column not updated. Fix the problem that the footer and topology group is shaded when the topology radiation is displayed. When the topology radiation chart is displayed, the corresponding button should be highlighted. Refactor the route mapping, Dynamically import routing components, Improve first page loading performance. Support topology of two mutually calling services. Implement a type of table chart in the dashboard. Support event in the dashboard. Show instance name in the trace view. Fix groups of services in the topography. Documentation Polish documentation due to we have covered all tracing, logging, and metrics fields. Adjust documentation about Zipkin receiver. Add backend-infrastructure-monitoring doc. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking 8.5.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nChanges by Version\u003c/p\u003e\n\u003ch4 id=\"project\"\u003eProject …\u003c/h4\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-8-5-0/","title":"Release Apache SkyWalking APM 8.5.0"},{"body":"SkyWalking Cloud on Kubernetes 0.3.0 is released. Go to downloads page to find release tars.\nSupport special characters in the metric selector of HPA metric adapter. Add the namespace to HPA metric name. ","excerpt":"\u003cp\u003eSkyWalking Cloud on Kubernetes 0.3.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars. …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cloud-on-kubernetes-0-3-0/","title":"Release Apache SkyWalking Cloud on Kubernetes 0.3.0"},{"body":"SkyWalking NodeJS 0.2.0 is released. Go to downloads page to find release tars.\nAdd AMQPLib plugin (RabbitMQ). (#34) Add MongoDB plugin. (#33) Add PgPlugin - PosgreSQL. (#31) Add MySQLPlugin to plugins. (#30) Add http protocol of host to http plugins. (#28) Add tag http.method to plugins. (#26) Bugfix: child spans created on immediate cb from op. (#41) Bugfix: async and preparing child entry/exit. (#36) Bugfix: tsc error of dist lib. (#24) Bugfix: AxiosPlugin async() / resync(). (#21) Bugfix: some requests of express / axios are not close correctly. (#20) Express plugin uses http wrap explicitly if http plugin disabled. (#42) ","excerpt":"\u003cp\u003eSkyWalking NodeJS 0.2.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAdd AMQPLib plugin …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-nodejs-0-2-0/","title":"Release Apache SkyWalking for NodeJS 0.2.0"},{"body":"SkyWalking Python 0.6.0 is released. Go to downloads page to find release tars.\nFixes: Segment data loss when gRPC timing out. (#116) sw_tornado plugin async handler status set correctly. (#115) sw_pymysql error when connection haven\u0026rsquo;t db. (#113) ","excerpt":"\u003cp\u003eSkyWalking Python 0.6.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eFixes:\n\u003cul\u003e\n\u003cli\u003eSegment …\u003c/li\u003e\u003c/ul\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-python-0-6-0/","title":"Release Apache SkyWalking Python 0.6.0"},{"body":"\nOrigin: End-User Tracing in a SkyWalking-Observed Browser - The New Stack\nApache SkyWalking: an APM (application performance monitor) system, especially designed for microservices, cloud native, and container-based (Docker, Kubernetes, Mesos) architectures.\nskywalking-client-js: a lightweight client-side JavaScript exception, performance, and tracing library. It provides metrics and error collection to the SkyWalking backend. It also makes the browser the starting point for distributed tracing.\nBackground Web application performance affects the retention rate of users. If a page load time is too long, the user will give up. So we need to monitor the web application to understand performance and ensure that servers are stable, available and healthy. SkyWalking is an APM tool and the skywalking-client-js extends its monitoring to include the browser, providing performance metrics and error collection to the SkyWalking backend.\nPerformance Metrics The skywalking-client-js uses [window.performance] (https://developer.mozilla.org/en-US/docs/Web/API/Window/performance) for performance data collection. From the MDN doc, the performance interface provides access to performance-related information for the current page. It\u0026rsquo;s part of the High Resolution Time API, but is enhanced by the Performance Timeline API, the Navigation Timing API, the User Timing API, and the Resource Timing API. In skywalking-client-js, all performance metrics are calculated according to the Navigation Timing API defined in the W3C specification. We can get a PerformanceTiming object describing our page using the window.performance.timing property. The PerformanceTiming interface contains properties that offer performance timing information for various events that occur during the loading and use of the current page.\nWe can better understand these attributes when we see them together in the figure below from W3C:\nThe following table contains performance metrics in skywalking-client-js.\nMetrics Name Describe Calculating Formulae Note redirectTime Page redirection time redirectEnd - redirectStart If the current document and the document that is redirected to are not from the same origin, set redirectStart, redirectEnd to 0 ttfbTime Time to First Byte responseStart - requestStart According to Google Development dnsTime Time to DNS query domainLookupEnd - domainLookupStart tcpTime Time to TCP link connectEnd - connectStart transTime Time to content transfer responseEnd - responseStart sslTime Time to SSL secure connection connectEnd - secureConnectionStart Only supports HTTPS resTime Time to resource loading loadEventStart - domContentLoadedEventEnd Represents a synchronized load resource in pages fmpTime Time to First Meaningful Paint - Listen for changes in page elements. Traverse each new element, and calculate the total score of these elements. If the element is visible, the score is 1 * weight; if the element is not visible, the score is 0 domAnalysisTime Time to DOM analysis domInteractive - responseEnd fptTime First Paint Time responseEnd - fetchStart domReadyTime Time to DOM ready domContentLoadedEventEnd - fetchStart loadPageTime Page full load time loadEventStart - fetchStart ttlTime Time to interact domInteractive - fetchStart firstPackTime Time to first package responseStart - domainLookupStart Skywalking-client-js collects those performance metrics and sends them to the OAP (Observability Analysis Platform) server , which aggregates data on the back-end side that is then shown in visualizations on the UI side. Users can optimize the page according to these data.\nException Metrics There are five kinds of errors that can be caught in skywalking-client-js:\nThe resource loading error is captured by window.addeventlistener ('error ', callback, true) window.onerror catches JS execution errors window.addEventListener('unhandledrejection', callback) is used to catch the promise errors the Vue errors are captured by Vue.config.errorHandler the Ajax errors are captured by addEventListener('error', callback); addEventListener('abort', callback); addEventListener('timeout', callback); in send callback. The Skywalking-client-js traces error data to the OAP server, finally visualizing data on the UI side. For an error overview of the App, there are several metrics for basic statistics and trends of errors, including the following metrics.\nApp Error Count, the total number of errors in the selected time period. App JS Error Rate, the proportion of PV with JS errors in a selected time period to total PV. All of Apps Error Count, Top N Apps error count ranking. All of Apps JS Error Rate, Top N Apps JS error rate ranking. Error Count of Versions in the Selected App, Top N Error Count of Versions in the Selected App ranking. Error Rate of Versions in the Selected App, Top N JS Error Rate of Versions in the Selected App ranking. Error Count of the Selected App, Top N Error Count of the Selected App ranking. Error Rate of the Selected App, Top N JS Error Rate of the Selected App ranking. For pages, we use several metrics for basic statistics and trends of errors, including the following metrics:\nTop Unstable Pages / Error Rate, Top N Error Count pages of the Selected version ranking. Top Unstable Pages / Error Count, Top N Error Count pages of the Selected version ranking. Page Error Count Layout, data display of different errors in a period of time. User Metrics SkyWalking browser monitoring also provides metrics about how the visitors use the monitored websites, such as PV(page views), UV(unique visitors), top N PV(page views), etc.\nIn SPAs (single page applications), the page will be refreshed only once. The traditional method only reports PV once after the page loading, but cannot count the PV of each sub-page, and can\u0026rsquo;t make other types of logs aggregate by sub-page.\nSkyWalking browser monitoring provides two processing methods for SPA pages:\nEnable SPA automatic parsing. This method is suitable for most single page application scenarios with URL hash as the route. In the initialized configuration item, set enableSPA to true, which will turn on the page\u0026rsquo;s hashchange event listener (trigger re reporting PV), and use URL hash as the page field in other data reporting.\nManual reporting. This method can be used in all single page application scenarios. This method can be used if the first method is not usable. The following example provides a set page method to manually update the page name when data is reported. When this method is called, the page PV will be re reported by default:\napp.on(\u0026#39;routeChange\u0026#39;, function (to) { ClientMonitor.setPerformance({ collector: \u0026#39;http://127.0.0.1:8080\u0026#39;, service: \u0026#39;browser-app\u0026#39;, serviceVersion: \u0026#39;1.0.0\u0026#39;, pagePath: to.path, autoTracePerf: true, enableSPA: true, }); }); Let\u0026rsquo;s take a look at the result found in the following image. It shows the most popular applications and versions, and the changes of PV over a period of time.\nMake the browser the starting point for distributed tracing SkyWalking browser monitoring intercepts HTTP requests to trace segments and spans. It supports tracking these following modes of HTTP requests: XMLHttpRequest and fetch. It also supports tracking libraries and tools based on XMLHttpRequest and fetch - such as Axios, SuperAgent, OpenApi, and so on.\nLet’s see how the SkyWalking browser monitoring intercepts HTTP requests:\nAfter this, use window.addEventListener('xhrReadyStateChange', callback) and set the readyState value tosw8 = xxxx in the request header. At the same time, reporting requests information to the back-end side. Finally, we can view trace data on the trace page. The following graphic is from the trace page:\nTo see how we listen for fetch requests, let’s see the source code of fetch\nAs you can see, it creates a promise and a new XMLHttpRequest object. Because the code of the fetch is built into the browser, it must monitor the code execution first. Therefore, when we add listening events, we can\u0026rsquo;t monitor the code in the fetch. Just after monitoring the code execution, let\u0026rsquo;s rewrite the fetch:\nimport { fetch } from \u0026#39;whatwg-fetch\u0026#39;; window.fetch = fetch; In this way, we can intercept the fetch request through the above method.\nAdditional Resources End-User Tracing in a SkyWalking-Observed Browser. ","excerpt":"\u003cp\u003e\u003cimg src=\"aircraft.jpg\" alt=\"\"\u003e\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eOrigin: \u003ca href=\"https://thenewstack.io/end-user-tracing-in-a-skywalking-observed-browser\"\u003eEnd-User Tracing in a SkyWalking-Observed Browser - The New Stack\u003c/a\u003e\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/apache/skywalking\"\u003eApache SkyWalking\u003c/a\u003e: an …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/end-user-tracing-in-a-skywalking-observed-browser/","title":"End-User Tracing in a SkyWalking-Observed Browser"},{"body":"\nSourceMarker is an open-source continuous feedback IDE plugin built on top of Apache SkyWalking, a popular open-source APM system with monitoring, tracing, and diagnosing capabilities for distributed software systems. SkyWalking, a truly holistic system, provides the means for automatically producing, storing, and querying software operation metrics. It requires little to no code changes to implement and is lightweight enough to be used in production. By itself, SkyWalking is a formidable force in the realm of continuous monitoring technology.\nSourceMarker, leveraging the continuous monitoring functionality provided by SkyWalking, creates continuous feedback technology by automatically linking software operation metrics to source code and displaying feedback directly inside of the IDE. While currently only supporting JetBrains-based IDEs and JVM-based programming languages, SourceMarker may be extended to support any number of programming languages and IDEs. Using SourceMarker, software developers can understand and validate software operation inside of their IDE. Instead of charts that indicate the health of the application, software developers can view the health of individual source code components and interpret software operation metrics from a much more familiar perspective. Such capabilities improve productivity as time spent continuously context switching from development to monitoring would be eliminated.\nLogging The benefits of continuous feedback technology are immediately apparent with the ability to view and search logs directly from source code. Instead of tailing log files or viewing logs through the browser, SourceMarker allows software developers to navigate production logs just as easily as they navigate source code. By using the source code as the primary perspective for navigating logs, SourceMarker allows software developers to view logs specific to any package, class, method, or line directly from the context of the source code which resulted in those logs.\nTracing Furthermore, continuous feedback technology offers software developers a deeper understanding of software by explicitly tying the implicit software operation to source code. Instead of visualizing software traces as Gantt charts, SourceMarker allows software developers to step through trace stacks while automatically resolving trace tags and logs. With SourceMarker, software developers can navigate production software traces in much the same way one debugs local applications.\nAlerting Most importantly, continuous feedback technology keeps software developers aware of production software operation. Armed with an APM-powered IDE, every software developer can keep track of the behavior of any method, class, package, and even the entire application itself. Moreover, this allows for source code to be the medium through which production bugs are made evident, thereby creating the feasibility of source code with the ability to self-diagnose and convey its own health.\nDownload SourceMarker SourceMarker aims to bridge the theoretical and empirical practices of software development through continuous feedback. The goal is to make developing software with empirical data feel natural and intuitive, creating more complete software developers that understand the entire software development cycle.\nhttps://github.com/sourceplusplus/sourcemarker This project is still early in its development, so if you think of any ways to improve SourceMarker, please let us know.\n","excerpt":"\u003cp\u003e\u003cimg src=\"SM_IDE-APM.gif\" alt=\"Alt Text\"\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://sourcemarker.dev\"\u003eSourceMarker\u003c/a\u003e is an open-source continuous feedback IDE plugin built on top of Apache SkyWalking, a …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2021-03-16-continuous-feedback/","title":"SourceMarker: Continuous Feedback for Developers"},{"body":"SkyWalking LUA Nginx 0.4.1 is released. Go to downloads page to find release tars.\nfix: missing constants in the rockspsec. ","excerpt":"\u003cp\u003eSkyWalking LUA Nginx 0.4.1 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003efix: missing …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-lua-nginx-0.4.1/","title":"Release Apache SkyWalking LUA Nginx 0.4.1"},{"body":"SkyWalking LUA Nginx 0.4.0 is released. Go to downloads page to find release tars.\nAdd a global field \u0026lsquo;includeHostInEntrySpan\u0026rsquo;, type \u0026lsquo;boolean\u0026rsquo;, mark the entrySpan include host/domain. Add destroyBackendTimer to stop reporting metrics. Doc: set random seed in init_worker phase. Local cache some variables and reuse them in Lua module. Enable local cache and use tablepool to reuse the temporary table. ","excerpt":"\u003cp\u003eSkyWalking LUA Nginx 0.4.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAdd a global …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-lua-nginx-0.4.0/","title":"Release Apache SkyWalking LUA Nginx 0.4.0"},{"body":"SkyWalking Client JS 0.4.0 is released. Go to downloads page to find release tars.\nUpdate stack and message in logs. Fix wrong URL when using relative path in xhr. ","excerpt":"\u003cp\u003eSkyWalking Client JS 0.4.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eUpdate stack and …\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-client-js-0-4-0/","title":"Release Apache SkyWalking Client JS 0.4.0"},{"body":"SkyWalking Satellite 0.1.0 is released. Go to downloads page to find release tars.\nFeatures Build the Satellite core structure. Add prometheus self telemetry. Add kafka client plugin. Add none-fallbacker plugin. Add timer-fallbacker plugin. Add nativelog-kafka-forwarder plugin. Add memory-queue plugin. Add mmap-queue plugin. Add grpc-nativelog-receiver plugin. Add http-nativelog-receiver plugin. Add grpc-server plugin. Add http-server plugin. Add prometheus-server plugin. Bug Fixes Issues and PR All issues are here All and pull requests are here ","excerpt":"\u003cp\u003eSkyWalking Satellite 0.1.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eBuild …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-satellite-0-1-0/","title":"Release Apache SkyWalking Satellite 0.1.0"},{"body":"Juntao Zhang leads and finished the re-build process of the whole skywalking website. Immigrate to the whole automatic website update, super friendly to users. Within the re-building process, he took several months contributions to bring the document of our main repository to host on the SkyWalking website, which is also available for host documentations of other repositories. We were waiting for this for years.\nJust in the website repository, he has 3800 LOC contributions through 26 commits.\nWe are honored to have him on the PMC team.\n","excerpt":"\u003cp\u003e\u003ca href=\"https://github.com/Jtrust\"\u003eJuntao Zhang\u003c/a\u003e leads and finished the re-build process of the whole skywalking website. Immigrate to …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-juntao-zhang-to-join-the-pmc/","title":"Welcome Juntao Zhang (张峻滔) to join the PMC"},{"body":"\nOrigin: Observe VM Service Meshes with Apache SkyWalking and the Envoy Access Log Service - The New Stack\nApache SkyWalking: an APM (application performance monitor) system, especially designed for microservices, cloud native, and container-based (Docker, Kubernetes, Mesos) architectures.\nEnvoy Access Log Service: Access Log Service (ALS) is an Envoy extension that emits detailed access logs of all requests going through Envoy.\nBackground In the previous post, we talked about the observability of service mesh under Kubernetes environment, and applied it to the bookinfo application in practice. We also mentioned that, in order to map the IP addresses into services, SkyWalking needs access to the service metadata from a Kubernetes cluster, which is not available for services deployed in virtual machines (VMs). In this post, we will introduce a new analyzer in SkyWalking that leverages Envoy’s metadata exchange mechanism to decouple with Kubernetes. The analyzer is designed to work in Kubernetes environments, VM environments, and hybrid environments. If there are virtual machines in your service mesh, you might want to try out this new analyzer for better observability, which we will demonstrate in this tutorial.\nHow it works The mechanism of how the analyzer works is the same as what we discussed in the previous post. What makes VMs different from Kubernetes is that, for VM services, there are no places where we can fetch the metadata to map the IP addresses into services.\nThe basic idea we present in this article is to carry the metadata along with Envoy’s access logs, which is called metadata-exchange mechanism in Envoy. When Istio pilot-agent starts an Envoy proxy as a sidecar of a service, it collects the metadata of that service from the Kubernetes platform, or a file on the VM where that service is deployed, and injects the metadata into the bootstrap configuration of Envoy. Envoy will carry the metadata transparently when emitting access logs to the SkyWalking receiver.\nBut how does Envoy compose a piece of a complete access log that involves the client side and server side? When a request goes out from Envoy, a plugin of istio-proxy named \u0026ldquo;metadata-exchange\u0026rdquo; injects the metadata into the http headers (with a prefix like x-envoy-downstream-), and the metadata is propagated to the server side. The Envoy sidecar of the server side receives the request and parses the headers into metadata, and puts the metadata into the access log, keyed by wasm.downstream_peer. The server side Envoy also puts its own metadata into the access log keyed by wasm.upstream_peer. Hence the two sides of a single request are completed.\nWith the metadata-exchange mechanism, we can use the metadata directly without any extra query.\nExample In this tutorial, we will use another demo application Online Boutique that consists of 10+ services so that we can deploy some of them in VMs and make them communicate with other services deployed in Kubernetes.\nTopology of Online Boutique In order to cover as many cases as possible, we will deploy CheckoutService and PaymentService on VM and all the other services on Kubernetes, so that we can cover the cases like Kubernetes → VM (e.g. Frontend → CheckoutService), VM → Kubernetes (e.g. CheckoutService → ShippingService), and VM → VM ( e.g. CheckoutService → PaymentService).\nNOTE: All the commands used in this tutorial are accessible on GitHub.\ngit clone https://github.com/SkyAPMTest/sw-als-vm-demo-scripts cd sw-als-vm-demo-scripts Make sure to init the gcloud SDK properly before moving on. Modify the GCP_PROJECT in file env.sh to your own project name. Most of the other variables should be OK to work if you keep them intact. If you would like to use ISTIO_VERSION \u0026gt;/= 1.8.0, please make sure this patch is included.\nPrepare Kubernetes cluster and VM instances 00-create-cluster-and-vms.sh creates a new GKE cluster and 2 VM instances that will be used through the entire tutorial, and sets up some necessary firewall rules for them to communicate with each other.\nInstall Istio and SkyWalking 01a-install-istio.sh installs Istio Operator with spec resources/vmintegration.yaml. In the YAML file, we enable the meshExpansion that supports VM in mesh. We also enable the Envoy access log service and specify the address skywalking-oap.istio-system.svc.cluster.local:11800 to which Envoy emits the access logs. 01b-install-skywalking.sh installs Apache SkyWalking and sets the analyzer to mx-mesh.\nCreate files to initialize the VM 02-create-files-to-transfer-to-vm.sh creates necessary files that will be used to initialize the VMs. 03-copy-work-files-to-vm.sh securely transfers the generated files to the VMs with gcloud scp command. Now use ./ssh.sh checkoutservice and ./ssh.sh paymentservice to log into the two VMs respectively, and cd to the ~/work directory, execute ./prep-checkoutservice.sh on checkoutservice VM instance and ./prep-paymentservice.sh on paymentservice VM instance. The Istio sidecar should be installed and started properly. To verify that, use tail -f /var/logs/istio/istio.log to check the Istio logs. The output should be something like:\n2020-12-12T08:07:07.348329Z\tinfo\tsds\tresource:default new connection 2020-12-12T08:07:07.348401Z\tinfo\tsds\tSkipping waiting for gateway secret 2020-12-12T08:07:07.348401Z\tinfo\tsds\tSkipping waiting for gateway secret 2020-12-12T08:07:07.568676Z\tinfo\tcache\tRoot cert has changed, start rotating root cert for SDS clients 2020-12-12T08:07:07.568718Z\tinfo\tcache\tGenerateSecret default 2020-12-12T08:07:07.569398Z\tinfo\tsds\tresource:default pushed key/cert pair to proxy 2020-12-12T08:07:07.949156Z\tinfo\tcache\tLoaded root cert from certificate ROOTCA 2020-12-12T08:07:07.949348Z\tinfo\tsds\tresource:ROOTCA pushed root cert to proxy 2020-12-12T20:12:07.384782Z\tinfo\tsds\tresource:default pushed key/cert pair to proxy 2020-12-12T20:12:07.384832Z\tinfo\tsds\tDynamic push for secret default The dnsmasq configuration address=/.svc.cluster.local/{ISTIO_SERVICE_IP_STUB} also resolves the domain names ended with .svc.cluster.local to Istio service IP, so that you are able to access the Kubernetes services in the VM by fully qualified domain name (FQDN) such as httpbin.default.svc.cluster.local.\nDeploy demo application Because we want to deploy CheckoutService and PaymentService manually on VM, resources/google-demo.yaml removes the two services from the original YAML . 04a-deploy-demo-app.sh deploys the other services on Kubernetes. Then log into the 2 VMs, run ~/work/deploy-checkoutservice.sh and ~/work/deploy-paymentservice.sh respectively to deploy CheckoutService and PaymentService.\nRegister VMs to Istio Services on VMs can access the services on Kubernetes by FQDN, but that’s not the case when the Kubernetes services want to talk to the VM services. The mesh has no idea where to forward the requests such as checkoutservice.default.svc.cluster.local because checkoutservice is isolated in the VM. Therefore, we need to register the services to the mesh. 04b-register-vm-with-istio.sh registers the VM services to the mesh by creating a \u0026ldquo;dummy\u0026rdquo; service without running Pods, and a WorkloadEntry to bridge the \u0026ldquo;dummy\u0026rdquo; service with the VM service.\nDone! The demo application contains a load generator service that performs requests repeatedly. We only need to wait a few seconds, and then open the SkyWalking web UI to check the results.\nexport POD_NAME=$(kubectl get pods --namespace istio-system -l \u0026#34;app=skywalking,release=skywalking,component=ui\u0026#34; -o jsonpath=\u0026#34;{.items[0].metadata.name}\u0026#34;) echo \u0026#34;Visit http://127.0.0.1:8080 to use your application\u0026#34; kubectl port-forward $POD_NAME 8080:8080 --namespace istio-system Navigate the browser to http://localhost:8080 . The metrics, topology should be there.\nTroubleshooting If you face any trouble when walking through the steps, here are some common problems and possible solutions:\nVM service cannot access Kubernetes services? It’s likely the DNS on the VM doesn’t correctly resolve the fully qualified domain names. Try to verify that with nslookup istiod.istio-system.svc.cluster.local. If it doesn’t resolve to the Kubernetes CIDR address, recheck the step in prep-checkoutservice.sh and prep-paymentservice.sh. If the DNS works correctly, try to verify that Envoy has fetched the upstream clusters from the control plane with curl http://localhost:15000/clusters. If it doesn’t contain the target service, recheck prep-checkoutservice.sh.\nServices are normal but nothing on SkyWalking WebUI? Check the SkyWalking OAP logs via kubectl -n istio-system logs -f $(kubectl get pod -A -l \u0026quot;app=skywalking,release=skywalking,component=oap\u0026quot; -o name) and WebUI logs via kubectl -n istio-system logs -f $(kubectl get pod -A -l \u0026quot;app=skywalking,release=skywalking,component=ui\u0026quot; -o name) to see whether there are any error logs . Also, make sure the time zone at the bottom-right of the browser is set to UTC +0.\nAdditional Resources Observe a Service Mesh with Envoy ALS. ","excerpt":"\u003cp\u003e\u003cimg src=\"stone-arch.jpg\" alt=\"\"\u003e\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eOrigin: \u003ca href=\"https://thenewstack.io/observe-virtual-machine-service-meshes-with-apache-skywalking-and-the-envoy-access-log-service\"\u003eObserve VM Service Meshes with Apache SkyWalking and the Envoy Access Log Service - The …\u003c/a\u003e\u003c/p\u003e\u003c/blockquote\u003e","ref":"https://skywalking.apache.org/blog/obs-service-mesh-vm-with-sw-and-als/","title":"Observe VM Service Meshes with Apache SkyWalking and the Envoy Access Log Service"},{"body":"When using SkyWalking java agent, people usually propagate context easily. They even do not need to change the business code. However, it becomes harder when you want to propagate context between threads when using ThreadPoolExecutor. You can use the RunnableWrapper in the maven artifact org.apache.skywalking:apm-toolkit-trace. This way you must change your code. The developer manager usually don\u0026rsquo;t like this because there may be lots of projects, or lots of runnable code. If they don\u0026rsquo;t use SkyWalking some day, the code added will be superfluous and inelegant.\nIs there a way to propagate context without changing the business code? Yes.\nSkywalking java agent enhances a class by add a field and implement an interface. The ThreadPoolExecutor is a special class that is used widely. We even don\u0026rsquo;t know when and where it is loaded. Most JVMs do not allow changes in the class file format for classes that have been loaded previously. So SkyWalking should not enhance the ThreadPoolExecutor successfully by retransforming when the ThreadPoolExecutor has been loaded. However, we can apply advice to the ThreadPoolExecutor#execute method and wrap the Runnable param using our own agent, then enhance the wrapper class by SkyWalking java agent. An advice do not change the layout of a class.\nNow we should decide how to do this. You can use the RunnableWrapper in the maven artifact org.apache.skywalking:apm-toolkit-trace to wrap the param, but you need to face another problem. This RunnableWrapper has a plugin whose active condition is checking if there is @TraceCrossThread. Agent core uses net.bytebuddy.pool.TypePool.Default.WithLazyResolution.LazyTypeDescription to find the annotations of a class. The LazyTypeDescription finds annotations by using a URLClassLoader with no urls if the classloader is null(bootstrap classloader). So it can not find the @TraceCrossThread class unless you change the LocationStrategy of SkyWalking java agent builder.\nIn this project, I write my own wrapper class, and simply add a plugin with a name match condition. Next, Let me show you how these two agents work together.\nMove the plugin to the skywalking \u0026ldquo;plugins\u0026rdquo; directory.\nAdd this agent after the SkyWalking agent since the wrapper class should not be loaded before SkyWalking agent instrumentation have finished. For example,\njava -javaagent:/path/to/skywalking-agent.jar -javaagent:/path/to/skywalking-tool-agent-v1.0.0.jar \u0026hellip;\nWhen our application runs\nSkyWalking java agent adds a transformer by parsing the plugin for enhancing the wrapper class in the tool agent. The tool agent loads the wrapper class into bootstrap classloader. This triggers the previous transformer. The tool agent applies an advice to the ThreadPoolExecutor class, wrapping the java.lang.Runnable param of \u0026ldquo;execute\u0026rdquo; method with the wrapper class. Now SkyWalking propagates the context with the wrapper class. Enjoy tracing with ThreadPoolExecutor in SkyWalking!\n","excerpt":"\u003cp\u003eWhen using SkyWalking java agent, people usually propagate context easily. They even do not need to …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2021-02-09-skywalking-trace-threadpool/","title":"Apache SkyWalking: How to propagate context between threads when using ThreadPoolExecutor"},{"body":"SkyWalking CLI 0.6.0 is released. Go to downloads page to find release tars.\nFeatures\nSupport authorization when connecting to the OAP Add install command and manifest sub-command Add event command and report sub-command Bug Fixes\nFix the bug that can\u0026rsquo;t query JVM instance metrics Chores\nSet up a simple test with GitHub Actions Reorganize the project layout Update year in NOTICE Add missing license of swck Use license-eye to check license header ","excerpt":"\u003cp\u003eSkyWalking CLI 0.6.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eFeatures\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eSupport …\u003c/li\u003e\u003c/ul\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cli-0-6-0/","title":"Release Apache SkyWalking CLI 0.6.0"},{"body":"\nOrigin: Tetrate.io blog\nBackground Apache SkyWalking\u0026ndash; the APM tool for distributed systems\u0026ndash; has historically focused on providing observability around tracing and metrics, but service performance is often affected by the host. The newest release, SkyWalking 8.4.0, introduces a new feature for monitoring virtual machines. Users can easily detect possible problems from the dashboard\u0026ndash; for example, when CPU usage is overloaded, when there’s not enough memory or disk space, or when the network status is unhealthy, etc.\nHow it works SkyWalking leverages Prometheus and OpenTelemetry for collecting metrics data as we did for Istio control panel metrics; Prometheus is mature and widely used, and we expect to see increased adoption of the new CNCF project, OpenTelemetry. The SkyWalking OAP Server receives these metrics data of OpenCensus format from OpenTelemetry. The process is as follows:\nPrometheus Node Exporter collects metrics data from the VMs. OpenTelemetry Collector fetches metrics from Node Exporters via Prometheus Receiver, and pushes metrics to SkyWalking OAP Server via the OpenCensus GRPC Exporter. The SkyWalking OAP Server parses the expression with MAL to filter/calculate/aggregate and store the results. The expression rules are in /config/otel-oc-rules/vm.yaml. We can now see the data on the SkyWalking WebUI dashboard. What to monitor SkyWalking provides default monitoring metrics including:\nCPU Usage (%) Memory RAM Usage (MB) Memory Swap Usage (MB) CPU Average Used CPU Load Memory RAM (total/available/used MB) Memory Swap (total/free MB) File System Mount point Usage (%) Disk R/W (KB/s) Network Bandwidth Usage (receive/transmit KB/s) Network Status (tcp_curr_estab/tcp_tw/tcp_alloc/sockets_used/udp_inuse) File fd Allocated The following is how it looks when we monitor Linux:\nHow to use To enable this feature, we need to install Prometheus Node Exporter and OpenTelemetry Collector and activate the VM monitoring rules in SkyWalking OAP Server.\nInstall Prometheus Node Exporter wget https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz tar xvfz node_exporter-1.0.1.linux-amd64.tar.gz cd node_exporter-1.0.1.linux-amd64 ./node_exporter In linux Node Exporter exposes metrics on port 9100 by default. When it is running, we can get the metrics from the /metrics endpoint. Use a web browser or command curl to verify.\ncurl http://localhost:9100/metrics We should see all the metrics from the output like:\n# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles. # TYPE go_gc_duration_seconds summary go_gc_duration_seconds{quantile=\u0026#34;0\u0026#34;} 7.7777e-05 go_gc_duration_seconds{quantile=\u0026#34;0.25\u0026#34;} 0.000113756 go_gc_duration_seconds{quantile=\u0026#34;0.5\u0026#34;} 0.000127199 go_gc_duration_seconds{quantile=\u0026#34;0.75\u0026#34;} 0.000147778 go_gc_duration_seconds{quantile=\u0026#34;1\u0026#34;} 0.000371894 go_gc_duration_seconds_sum 0.292994058 go_gc_duration_seconds_count 2029 ... Note: We only need to install Node Exporter, rather than Prometheus server. If you want to get more information about Prometheus Node Exporter see: https://prometheus.io/docs/guides/node-exporter/\nInstall OpenTelemetry Collector We can quickly install a OpenTelemetry Collector instance by using docker-compose with the following steps:\nCreate a directory to store the configuration files, like /usr/local/otel. Create docker-compose.yaml and otel-collector-config.yaml in this directory represented below: docker-compose.yaml\nversion: \u0026#34;2\u0026#34; services: # Collector otel-collector: # Specify the image to start the container from image: otel/opentelemetry-collector:0.19.0 # Set the otel-collector configfile command: [\u0026#34;--config=/etc/otel-collector-config.yaml\u0026#34;] # Mapping the configfile to host directory volumes: - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml ports: - \u0026#34;13133:13133\u0026#34; # health_check extension - \u0026#34;55678\u0026#34; # OpenCensus receiver otel-collector-config.yaml\nextensions: health_check: # A receiver is how data gets into the OpenTelemetry Collector receivers: # Set Prometheus Receiver to collects metrics from targets # It’s supports the full set of Prometheus configuration prometheus: config: scrape_configs: - job_name: \u0026#39;otel-collector\u0026#39; scrape_interval: 10s static_configs: # Replace the IP to your VMs‘s IP which has installed Node Exporter - targets: [ \u0026#39;vm1:9100\u0026#39; ] - targets: [ \u0026#39;vm2:9100\u0026#39; ] - targets: [ ‘vm3:9100\u0026#39; ] processors: batch: # An exporter is how data gets sent to different systems/back-ends exporters: # Exports metrics via gRPC using OpenCensus format opencensus: endpoint: \u0026#34;docker.for.mac.host.internal:11800\u0026#34; # The OAP Server address insecure: true logging: logLevel: debug service: pipelines: metrics: receivers: [prometheus] processors: [batch] exporters: [logging, opencensus] extensions: [health_check] In this directory use command docker-compose to start up the container: docker-compose up -d After the container is up and running, you should see metrics already exported in the logs:\n... Metric #165 Descriptor: -\u0026gt; Name: node_network_receive_compressed_total -\u0026gt; Description: Network device statistic receive_compressed. -\u0026gt; Unit: -\u0026gt; DataType: DoubleSum -\u0026gt; IsMonotonic: true -\u0026gt; AggregationTemporality: AGGREGATION_TEMPORALITY_CUMULATIVE DoubleDataPoints #0 Data point labels: -\u0026gt; device: ens4 StartTime: 1612234754364000000 Timestamp: 1612235563448000000 Value: 0.000000 DoubleDataPoints #1 Data point labels: -\u0026gt; device: lo StartTime: 1612234754364000000 Timestamp: 1612235563448000000 Value: 0.000000 ... If you want to get more information about OpenTelemetry Collector see: https://opentelemetry.io/docs/collector/\nSet up SkyWalking OAP Server To activate the oc handler and vm relevant rules, set your environment variables:\nSW_OTEL_RECEIVER=default SW_OTEL_RECEIVER_ENABLED_OC_RULES=vm Note: If there are other rules already activated , you can add vm with use , as a separator.\nSW_OTEL_RECEIVER_ENABLED_OC_RULES=vm,oap Start the SkyWalking OAP Server.\nDone! After all of the above steps are completed, check out the SkyWalking WebUI. Dashboard VM provides the default metrics of all observed virtual machines. Note: Clear the browser local cache if you used it to access deployments of previous SkyWalking versions.\nAdditional Resources Read more about the SkyWalking 8.4 release highlights. Get more SkyWalking updates on Twitter. ","excerpt":"\u003cp\u003e\u003cimg src=\"apache-skywalking.jpeg\" alt=\"\"\u003e\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eOrigin: \u003ca href=\"https://www.tetrate.io/blog/skywalking-8-4-provides-infrastucture-monitoring-for-vms/\"\u003eTetrate.io blog\u003c/a\u003e\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"background\"\u003eBackground\u003c/h2\u003e\n\u003cp\u003eApache SkyWalking\u0026ndash; the APM tool for distributed …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2021-02-07-infrastructure-monitoring/","title":"SkyWalking 8.4 provides infrastructure monitoring"},{"body":"\nOrigin: Tetrate.io blog\nThe Apache SkyWalking team today announced the 8.4 release is generally available. This release fills the gap between all previous versions of SkyWalking and the logging domain area. The release also advances SkyWalking’s capabilities for infrastructure observability, starting with virtual machine monitoring.\nBackground SkyWalking has historically focused on the tracing and metrics fields of observability. As its features for tracing, metrics and service level monitoring have become more and more powerful and stable, the SkyWalking team has started to explore new scenarios covered by observability. Because service performance is reflected in the logs, and is highly impacted by the infrastructure on which it runs, SkyWalking brings these two fields into the 8.4 release. This release blog briefly introduces the two new features as well as some other notable changes.\nLogs Metrics, tracing, and logging are considered the three pillars of observability [1]. SkyWalking had the full features of metrics and tracing prior to 8.4; today, as 8.4 is released, the last piece of the jigsaw is now in place.\nFigure 1: Logs Collected By SkyWalking\nFigure 2: Logs Collected By SkyWalking\nThe Java agent firstly provides SDKs to enhance the widely-used logging frameworks, log4j (1.x and 2.x) [2] and logback [3], and send the logs to the SkyWalking backend (OAP). The latter is able to collect logs from wherever the protocol is implemented. This is not a big deal, but when it comes to the correlation between logs and traces, the traditional solution is to print the trace IDs in the logs, and pick the IDs in the error logs to query the related traces. SkyWalking just simplifies the workflow by correlating the logs and traces natively. Navigating between traces and their related logs is as simple as clicking a button.\nFigure 3: Correlation Between Logs and Traces\nInfrastructure Monitoring SkyWalking is known as an application performance monitoring tool. One of the most important factors that impacts the application’s performance is the infrastructure on which the application runs. In the 8.4 release, we added the monitoring metrics of virtual machines into the dashboard.\nFigure 4: VM Metrics\nFundamental metrics such as CPU Used, Memory Used, Disk Read / Write and Network Usage are available on the dashboard. And as usual, those metrics are also available to be configured as alarm triggers when needed.\nDynamic Configurations at Agent Side Dynamic configuration at the backend side has long existed in SkyWalking for several versions. Now, it finally comes to the agent side! Prior to 8.4, you’d have to restart the target services when you modify some configuration items of the agent \u0026ndash; for instance, sampling rate (agent side), ignorable endpoint paths, etc. Now, say goodbye to rebooting. Modifying configurations is not the only usage of the dynamic configuration mechanism. The latter gives countless possibilities to the agent side in terms of dynamic behaviours, e.g. enabling / disabling plugins, enabling / disabling the whole agent, etc. Just imagine!\nGrouped Service Topology This enhancement is from the UI. SkyWalking backend supports grouping the services by user-defined dimensions. In a real world use case, the services are usually grouped by business group or department. When a developer opens the topology map, out of hundreds of services, he or she may just want to focus on the services in charge. The grouped service topology comes to the rescue: one can now choose to display only services belonging to a specified group.\nFigure 5: Grouped Service Topology\nOther Notable Enhancements Agent: resolves domain names to look up backend service IP addresses. Backend: meter receiver supports meter analysis language (MAL). Backend: several CVE fixes. Backend: supports Envoy {AccessLog,Metrics}Service API V3 and adopts MAL. Links [1] https://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html [2] https://logging.apache.org/log4j/2.x/ [3] http://logback.qos.ch Additional Resources Read more about the SkyWalking 8.4 release highlights. Get more SkyWalking updates on Twitter. ","excerpt":"\u003cp\u003e\u003cimg src=\"heading.png\" alt=\"\"\u003e\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eOrigin: \u003ca href=\"https://www.tetrate.io/blog/skywalking-8-4/\"\u003eTetrate.io blog\u003c/a\u003e\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eThe Apache SkyWalking team today announced the 8.4 release is generally …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/skywalking8-4-release/","title":"Apache SkyWalking 8.4: Logs, VM Monitoring, and Dynamic Configurations at Agent Side"},{"body":"SkyWalking 8.4.0 is released. Go to downloads page to find release tars. Changes by Version\nProject Incompatible with previous releases when use H2/MySQL/TiDB storage options, due to support multiple alarm rules triggered for one entity. Chore: adapt create_source_release.sh to make it runnable on Linux. Add package to .proto files, prevent polluting top-level namespace in some languages; The OAP server supports previous agent releases, whereas the previous OAP server (\u0026lt;=8.3.0) won\u0026rsquo;t recognize newer agents since this version (\u0026gt;= 8.4.0). Add ElasticSearch 7.10 to test matrix and verify it works. Replace Apache RAT with skywalking-eyes to check license headers. Set up test of Envoy ALS / MetricsService under Istio 1.8.2 to verify Envoy V3 protocol Test: fix flaky E2E test of Kafka. Java Agent The operation name of quartz-scheduler plugin, has been changed as the quartz-scheduler/${className} format. Fix jdk-http and okhttp-3.x plugin did not overwrite the old trace header. Add interceptors of method(analyze, searchScroll, clearScroll, searchTemplate and deleteByQuery) for elasticsearch-6.x-plugin. Fix the unexpected RunningContext recreation in the Tomcat plugin. Fix the potential NPE when trace_sql_parameters is enabled. Update byte-buddy to 1.10.19. Fix thrift plugin trace link broken when intermediate service does not mount agent Fix thrift plugin collects wrong args when the method without parameter. Fix DataCarrier\u0026rsquo;s org.apache.skywalking.apm.commons.datacarrier.buffer.Buffer implementation isn\u0026rsquo;t activated in IF_POSSIBLE mode. Fix ArrayBlockingQueueBuffer\u0026rsquo;s useless IF_POSSIBLE mode list Support building gRPC TLS channel but CA file is not required. Add witness method mechanism in the agent plugin core. Add Dolphinscheduler plugin definition. Make sampling still works when the trace ignores plug-in activation. Fix mssql-plugin occur ClassCastException when call the method of return generate key. The operation name of dubbo and dubbo-2.7.x-plugin, has been changed as the groupValue/className.methodName format Fix bug that rocketmq-plugin set the wrong tag. Fix duplicated EnhancedInstance interface added. Fix thread leaks caused by the elasticsearch-6.x-plugin plugin. Support reading segmentId and spanId with toolkit. Fix RestTemplate plugin recording url tag with wrong port Support collecting logs and forwarding through gRPC. Support config agent.sample_n_per_3_secs can be changed in the runtime. Support config agent.ignore_suffix can be changed in the runtime. Support DNS periodic resolving mechanism to update backend service. Support config agent.trace.ignore_path can be changed in the runtime. Added support for transmitting logback 1.x and log4j 2.x formatted \u0026amp; un-formatted messages via gPRC OAP-Backend Make meter receiver support MAL. Support influxDB connection response format option. Fix some error when use JSON as influxDB response format. Support Kafka MirrorMaker 2.0 to replicate topics between Kafka clusters. Add the rule name field to alarm record storage entity as a part of ID, to support multiple alarm rules triggered for one entity. The scope id has been removed from the ID. Fix MAL concurrent execution issues. Fix group name can\u0026rsquo;t be queried in the GraphQL. Fix potential gRPC connection leak(not closed) for the channels among OAP instances. Filter OAP instances(unassigned in booting stage) of the empty IP in KubernetesCoordinator. Add component ID for Python aiohttp plugin requester and server. Fix H2 in-memory database table missing issues Add component ID for Python pyramid plugin server. Add component ID for NodeJS Axios plugin. Fix searchService method error in storage-influxdb-plugin. Add JavaScript component ID. Fix CVE of UninstrumentedGateways in Dynamic Configuration activation. Improve query performance in storage-influxdb-plugin. Fix the uuid field in GRPCConfigWatcherRegister is not updated. Support Envoy {AccessLog,Metrics}Service API V3. Adopt the MAL in Envoy metrics service analyzer. Fix the priority setting doesn\u0026rsquo;t work of the ALS analyzers. Fix bug that endpoint-name-grouping.yml is not customizable in Dockerized case. Fix bug that istio version metric type on UI template mismatches the otel rule. Improve ReadWriteSafeCache concurrency read-write performance Fix bug that if use JSON as InfluxDB.ResponseFormat then NumberFormatException maybe occur. Fix timeBucket not taking effect in EqualsAndHashCode annotation of some relationship metrics. Fix SharingServerConfig\u0026rsquo;s propertie is not correct in the application.yml, contextPath -\u0026gt; restConnextPath. Istio control plane: remove redundant metrics and polish panel layout. Fix bug endpoint name grouping not work due to setting service name and endpoint name out of order. Fix receiver analysis error count metrics. Log collecting and query implementation. Support Alarm to feishu. Add the implementation of ConfigurationDiscovery on the OAP side. Fix bug in parseInternalErrorCode where some error codes are never reached. OAL supports multiple values when as numeric. Add node information from the Openensus proto to the labels of the samples, to support the identification of the source of the Metric data. Fix bug that the same sample name in one MAL expression caused IllegalArgumentException in Analyzer.analyse. Add the text analyzer for querying log in the es storage. Chore: Remove duplicate codes in Envoy ALS handler. Remove the strict rule of OAL disable statement parameter. Fix a legal metric query adoption bug. Don\u0026rsquo;t support global level metric query. Add VM MAL and ui-template configration, support Prometheus node-exporter VM metrics that pushed from OpenTelemetry-collector. Remove unused log query parameters. UI Fix un-removed tags in trace query. Fix unexpected metrics name on single value component. Don\u0026rsquo;t allow negative value as the refresh period. Fix style issue in trace table view. Separation Log and Dashboard selector data to avoid conflicts. Fix trace instance selector bug. Fix Unnecessary sidebar in tooltips for charts. Refactor dashboard query in a common script. Implement refreshing data for topology by updating date. Implement group selector in the topology. Fix all as default parameter for services selector. Add icon for Python aiohttp plugin. Add icon for Python pyramid plugin. Fix topology render all services nodes when groups changed. Fix rk-footer utc input\u0026rsquo;s width. Update rk-icon and rewrite rk-header svg tags with rk-icon. Add icon for http type. Fix rk-footer utc without local storage. Sort group names in the topology. Add logo for Dolphinscheduler. Fix dashboard wrong instance. Add a legend for the topology. Update the condition of unhealthy cube. Fix: use icons to replace buttons for task list in profile. Fix: support = in the tag value in the trace query page. Add envoy proxy component logo. Chore: set up license-eye to check license headers and add missing license headers. Fix prop for instances-survey and endpoints-survey. Fix envoy icon in topology. Implement the service logs on UI. Change the flask icon to light version for a better view of topology dark theme. Implement viewing logs on trace page. Fix update props of date component. Fix query conditions for logs. Fix style of selectors to word wrap. Fix logs time. Fix search ui for logs. Documentation Update the documents of backend fetcher and self observability about the latest configurations. Add documents about the group name of service. Update docs about the latest UI. Update the document of backend trace sampling with the latest configuration. Update kafka plugin support version to 2.6.1. Add FAQ about Fix compiling on Mac M1 chip. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking 8.4.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nChanges by Version\u003c/p\u003e\n\u003ch4 id=\"project\"\u003eProject …\u003c/h4\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-8-4-0/","title":"Release Apache SkyWalking APM 8.4.0"},{"body":"Background The verifier is an important part of the next generation End-to-End Testing framework (NGE2E), which is responsible for verifying whether the actual output satisfies the expected template.\nDesign Thinking We will implement the verifier with Go template, plus some enhancements. Firstly, users need to write a Go template file with provided functions and actions to describe how the expected data looks like. Then the verifer renders the template with the actual data object. Finally, the verifier compares the rendered output with the actual data. If the rendered output is not the same with the actual output, it means the actual data is inconsist with the expected data. Otherwise, it means the actual data match the expected data. On failure, the verifier will also print out what are different between expected and actual data.\nBranches / Actions The verifier inherits all the actions from the standard Go template, such as if, with, range, etc. In addition, we also provide some custom actions to satisfy our own needs.\nList Elements Match contains checks if the actual list contains elements that match the given template.\nExamples:\nmetrics: {{- contains .metrics }} - name: {{ notEmpty .name }} id: {{ notEmpty .id }} value: {{ gt .value 0 }} {{- end }} It means that the list metrics must contain an element whose name and id are not empty, and value is greater than 0.\nmetrics: {{- contains .metrics }} - name: p95 value: {{ gt .value 0 }} - name: p99 value: {{ gt .value 0 }} {{- end }} This means that the list metrics must contain an element named p95 with a value greater than 0, and an element named p95 with a value greater than 0. Besides the two element, the list metrics may or may not have other random elements.\nFunctions Users can use these provided functions in the template to describe the expected data.\nNot Empty notEmpty checks if the string s is empty.\nExample:\nid: {{ notEmpty .id }} Regexp match regexp checks if string s matches the regular expression pattern.\nExamples:\nlabel: {{ regexp .label \u0026#34;ratings.*\u0026#34; }} Base64 b64enc s returns the Base64 encoded string of s.\nExamples:\nid: {{ b64enc \u0026#34;User\u0026#34; }}.static-suffix # this evalutes the base64 encoded string of \u0026#34;User\u0026#34;, concatenated with a static suffix \u0026#34;.static-suffix\u0026#34; Result:\nid: VXNlcg==.static-suffix Full Example Here is an example of expected data:\n# expected.data.yaml nodes: - id: {{ b64enc \u0026#34;User\u0026#34; }}.0 name: User type: USER isReal: false - id: {{ b64enc \u0026#34;Your_ApplicationName\u0026#34; }}.1 name: Your_ApplicationName type: Tomcat isReal: true - id: {{ $h2ID := (index .nodes 2).id }}{{ notEmpty $h2ID }} # We assert that nodes[2].id is not empty and save it to variable `h2ID` for later use name: localhost:-1 type: H2 isReal: false calls: - id: {{ notEmpty (index .calls 0).id }} source: {{ b64enc \u0026#34;Your_ApplicationName\u0026#34; }}.1 target: {{ $h2ID }} # We use the previously assigned variable `h2Id` to asert that the `target` is equal to the `id` of the nodes[2] detectPoints: - CLIENT - id: {{ b64enc \u0026#34;User\u0026#34; }}.0-{{ b64enc \u0026#34;Your_ApplicationName\u0026#34; }}.1 source: {{ b64enc \u0026#34;User\u0026#34; }}.0 target: {{ b64enc \u0026#34;Your_ApplicationName\u0026#34; }}.1 detectPoints: - SERVER will validate this data:\n# actual.data.yaml nodes: - id: VXNlcg==.0 name: User type: USER isReal: false - id: WW91cl9BcHBsaWNhdGlvbk5hbWU=.1 name: Your_ApplicationName type: Tomcat isReal: true - id: bG9jYWxob3N0Oi0x.0 name: localhost:-1 type: H2 isReal: false calls: - id: WW91cl9BcHBsaWNhdGlvbk5hbWU=.1-bG9jYWxob3N0Oi0x.0 source: WW91cl9BcHBsaWNhdGlvbk5hbWU=.1 detectPoints: - CLIENT target: bG9jYWxob3N0Oi0x.0 - id: VXNlcg==.0-WW91cl9BcHBsaWNhdGlvbk5hbWU=.1 source: VXNlcg==.0 detectPoints: - SERVER target: WW91cl9BcHBsaWNhdGlvbk5hbWU=.1 # expected.data.yaml metrics: {{- contains .metrics }} - name: {{ notEmpty .name }} id: {{ notEmpty .id }} value: {{ gt .value 0 }} {{- end }} will validate this data:\n# actual.data.yaml metrics: - name: business-zone::projectA id: YnVzaW5lc3Mtem9uZTo6cHJvamVjdEE=.1 value: 1 - name: system::load balancer1 id: c3lzdGVtOjpsb2FkIGJhbGFuY2VyMQ==.1 value: 0 - name: system::load balancer2 id: c3lzdGVtOjpsb2FkIGJhbGFuY2VyMg==.1 value: 0 and will report an error when validating this data, because there is no element with a value greater than 0:\n# actual.data.yaml metrics: - name: business-zone::projectA id: YnVzaW5lc3Mtem9uZTo6cHJvamVjdEE=.1 value: 0 - name: system::load balancer1 id: c3lzdGVtOjpsb2FkIGJhbGFuY2VyMQ==.1 value: 0 - name: system::load balancer2 id: c3lzdGVtOjpsb2FkIGJhbGFuY2VyMg==.1 value: 0 The contains does an unordered list verification, in order to do list verifications including orders, you can simply use the basic ruls like this:\n# expected.data.yaml metrics: - name: p99 value: {{ gt (index .metrics 0).value 0 }} - name: p95 value: {{ gt (index .metrics 1).value 0 }} which expects the actual metrics list to be exactly ordered, with first element named p99 and value greater 0, second element named p95 and value greater 0.\n","excerpt":"\u003ch2 id=\"background\"\u003eBackground\u003c/h2\u003e\n\u003cp\u003eThe verifier is an important part of the next generation End-to-End Testing framework …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2021-02-01-e2e-verifier-design/","title":"[Design] The Verifier of NGE2E"},{"body":"SkyWalking Cloud on Kubernetes 0.2.0 is released. Go to downloads page to find release tars.\nIntroduce custom metrics adapter to SkyWalking OAP cluster for Kubernetes HPA autoscaling. Add RBAC files and service account to support Kubernetes coordination. Add default and validation webhooks to operator controllers. Add UI CRD to deploy skywalking UI server. Add Fetcher CRD to fetch metrics from other telemetry system, for example, Prometheus. ","excerpt":"\u003cp\u003eSkyWalking Cloud on Kubernetes 0.2.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars. …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cloud-on-kubernetes-0-2-0/","title":"Release Apache SkyWalking Cloud on Kubernetes 0.2.0"},{"body":"Apache SkyWalking is an open source APM for distributed system, Apache Software Foundation top-level project.\nAt Jan. 11th, 2021, we noticed the Tencent Cloud Service, Tencent Service Watcher - TSW, for first time. Due to the similar short name, which SkyWalking is also called SW in the community, we connected with the service team of Tencent Cloud, and kindly asked.\nThey used to replay, TSW is purely developed by Tencent team itself, which doesn\u0026rsquo;t have any code dependency on SkyWalking.. We didn\u0026rsquo;t push harder.\nBut one week later, Jan 18th, 2021, our V.P., Sheng got the report again from Haoyang SkyWalking PMC member, through WeChat DM(direct message),. He provided complete evidence to prove TSW actually re-distributed the SkyWalking\u0026rsquo;s Java agent. We keep one copy of their agent\u0026rsquo;s distribution(at Jan. 18th), you could be downloaded here.\nSome typically evidences are here\nServiceManager is copied and package-name changed in the TSW\u0026rsquo;s agent. ContextManager is copied and ackage-name changed in the TSW\u0026rsquo;s agent. At the same time, we checked their tsw-client-package.zip, it didn\u0026rsquo;t include the SkyWalking\u0026rsquo;s LICENSE and NOTICE. Also, they didn\u0026rsquo;t mention TSW agent is the re-ditribution SkyWalking on their website.\nWith all above information, we had enough reason to believe, from the tech perspective, they were violating the Apache 2.0 License.\nFrom the 18th Jan., 2021, we sent mail [Apache 2.0 License Violation] Tencent Cloud TSW service doesn't follow the Apache 2.0 License to brief the SkyWalking PMC, and took the following actions to connect with Tencent.\nMade direct call to Tencent Open Source Office. Connected with Tencent Cloud TVP program committee, as Sheng Wu(Our VP) is a Tencent Cloud TVP. Talked with the Tencent Cloud team lead. In all above channels, we provided the evidences of copy-redistribution hebaviors, requested them to revaluate their statements on the website, and follow the License\u0026rsquo;s requirements.\nResolution At Jan. 19th night, UTC+8, 2021. We received response from the Tencent cloud team. They admited their violation behaviors, and did following changes\nTencent Cloud TSW service page states, the agent is the fork version(re-distribution) of Apache SkyWalking agent. TSW agent distributions include the SkyWalking\u0026rsquo;s License and NOTICE. Below is the screenshot, you could download from their product page. We keep a copy of their Jan. 19th 2021 at here. We have updated the status to the PMC mail list. This license violation issue has been resolved for now.\nThe SkyWalking community and program management committee will keep our eyes on Tencent TSW. ","excerpt":"\u003cp\u003e\u003ca href=\"https://skywalking.apache.org\"\u003eApache SkyWalking\u003c/a\u003e is an open source APM for distributed system, Apache Software Foundation top-level …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2021-01-23-tencent-cloud-violates-aplv2/","title":"[Resolved][License Issue] Tencent Cloud TSW service violates the Apache 2.0 License when using SkyWalking."},{"body":" 第一节：开篇介绍 第二节：数字游戏（Number Game） 第三节：社区原则（Community “Principles”） 第四节：基金会原则（For public good） 第五节：一些不太好的事情 B站视频地址\n","excerpt":"\u003cul\u003e\n\u003cli\u003e第一节：开篇介绍\u003c/li\u003e\n\u003cli\u003e第二节：数字游戏（Number Game）\u003c/li\u003e\n\u003cli\u003e第三节：社区原则（Community “Principles”）\u003c/li\u003e\n\u003cli\u003e第四节：基金会原则（For public good）\u003c/li\u003e\n\u003cli\u003e第五节：一些不太 …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/zh/2021-01-21-educate-community/","title":"[视频] 开放原子开源基金会2020年度峰会 - Educate community Over Support community"},{"body":"Elastic announced their license change, Upcoming licensing changes to Elasticsearch and Kibana.\nWe are moving our Apache 2.0-licensed source code in Elasticsearch and Kibana to be dual licensed under Server Side Public License (SSPL) and the Elastic License, giving users the choice of which license to apply. This license change ensures our community and customers have free and open access to use, modify, redistribute, and collaborate on the code. It also protects our continued investment in developing products that we distribute for free and in the open by restricting cloud service providers from offering Elasticsearch and Kibana as a service without contributing back. This will apply to all maintained branches of these two products and will take place before our upcoming 7.11 release. Our releases will continue to be under the Elastic License as they have been for the last three years.\nAlso, they provide the FAQ page for more information about the impact for the users, developers, and vendors.\nIn the perspective of Apache Software Foundation, SSPL has been confirmed as a Catalog X LICENSE(https://www.apache.org/legal/resolved.html#category-x), which means hard-dependency as a part of the core is not allowed. With that, we can\u0026rsquo;t only focus on it anymore. We need to consider other storage options. Right now, we still have InfluxDB, TiDB, H2 server still in Apache 2.0 licensed. Right now, we still have InfluxDB, TiDB, H2 server as storage options still in Apache 2.0 licensed.\nAs one optional plugin, we need to focus on the client driver license. Right now, we are only using ElasticSearch 7.5.0 and 6.3.2 drivers, which are both Apache 2.0 licensed. So, we are safe. For further upgrade, here is their announcement. They answer these typical cases in the FAQ page.\nI build a SaaS application using Elasticsearch as the backend, how does this affect me?\nThis source code license change should not affect you - you can use our default distribution or develop applications on top of it for free, under the Elastic License. This source-available license does not contain any copyleft provisions and the default functionality is free of charge. For a specific example, you can see our response to a question around this at Magento.\nOur users still could use, redistribute, sale the products/services, based on SkyWalking, even they are using self hosting Elastic Search unmodified server.\nI\u0026rsquo;m using Elasticsearch via APIs, how does this change affect me?\nThis change does not affect how you use client libraries to access Elasticsearch. Our client libraries remain licensed under Apache 2.0, with the exception of our Java High Level Rest Client (Java HLRC). The Java HLRC has dependencies on the core of Elasticsearch, and as a result this client library will be licensed under the Elastic License. Over time, we will eliminate this dependency and move the Java HLRC to be licensed under Apache 2.0. Until that time, for the avoidance of doubt, we do not consider using the Java HLRC as a client library in development of an application or library used to access Elasticsearch to constitute a derivative work under the Elastic License, and this will not have any impact on how you license the source code of your application using this client library or how you distribute it.\nThe client driver license incompatible issue will exist, we can\u0026rsquo;t upgrade the driver(s) until they release the Apache 2.0 licensed driver jars. But users are still safe to upgrade the drivers by themselves.\nApache SkyWalking will discuss the further actions here. If you have any question, welcome to ask. In the later 2021, we will begin to invest the posibility of creating SkyWalking\u0026rsquo;s observability database implementation.\n","excerpt":"\u003cp\u003e\u003ca href=\"https://elastic.co\"\u003eElastic\u003c/a\u003e announced their license change, \u003ca href=\"https://www.elastic.co/blog/licensing-change\"\u003e\u003cstrong\u003eUpcoming licensing changes to Elasticsearch and Kibana\u003c/strong\u003e.\u003c/a\u003e\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eWe …\u003c/p\u003e\u003c/blockquote\u003e","ref":"https://skywalking.apache.org/blog/2021-01-17-elastic-change-license/","title":"Response to Elastic 2021 License Change"},{"body":"SkyWalking Client JS 0.3.0 is released. Go to downloads page to find release tars.\nSupport tracing starting at the browser. Add traceSDKInternal SDK for tracing SDK internal RPC. Add detailMode SDK for tracing http method and url as tags in spans. Fix conditions of http status. ","excerpt":"\u003cp\u003eSkyWalking Client JS 0.3.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eSupport tracing …\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-client-js-0-3-0/","title":"Release Apache SkyWalking Client JS 0.3.0"},{"body":"SkyWalking Eyes 0.1.0 is released. Go to downloads page to find release tars.\nLicense Header Add check and fix command. check results can be reported to pull request as comments. fix suggestions can be filed on pull request as edit suggestions. ","excerpt":"\u003cp\u003eSkyWalking Eyes 0.1.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eLicense Header\n\u003cul\u003e\n\u003cli\u003eAdd …\u003c/li\u003e\u003c/ul\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-eyes-0-1-0/","title":"Release Apache SkyWalking Eyes 0.1.0"},{"body":"SkyWalking NodeJS 0.1.0 is released. Go to downloads page to find release tars.\nInitialize project core codes. Built-in http/https plugin. Express plugin. Axios plugin. ","excerpt":"\u003cp\u003eSkyWalking NodeJS 0.1.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eInitialize project …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-nodejs-0-1-0/","title":"Release Apache SkyWalking for NodeJS 0.1.0"},{"body":"SkyWalking Python 0.5.0 is released. Go to downloads page to find release tars.\nNew plugins\nPyramid Plugin (#102) AioHttp Plugin (#101) Sanic Plugin (#91) API and enhancements\n@trace decorator supports async functions Supports async task context Optimized path trace ignore Moved exception check to Span.__exit__ Moved Method \u0026amp; Url tags before requests Fixes:\nBaseExceptions not recorded as errors Allow pending data to send before exit sw_flask general exceptions handled Make skywalking logging Non-global Chores and tests\nMake tests really run on specified Python version Deprecate 3.5 as it\u0026rsquo;s EOL ","excerpt":"\u003cp\u003eSkyWalking Python 0.5.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eNew plugins …\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-python-0-5-0/","title":"Release Apache SkyWalking Python 0.5.0"},{"body":"Apache SkyWalking is an open source APM for distributed system. Provide tracing, service mesh observability, metrics analysis, alarm and visualization.\nJust 11 months ago, on Jan. 20th, 2020, SkyWalking hit the 200 contributors mark. With the growth of the project and the community, SkyWalking now includes over 20 sub(ecosystem) projects covering multiple language agents and service mesh, integration with mature open source projects, like Prometheus, Spring(Sleuth), hundreds of libraries to support all tracing/metrics/logs fields. In the past year, the number of contributors grows super astoundingly , and all its metrics point to its community vibrancy. Many corporate titans are already using SkyWalking in a large-scale production environment, including, Alibaba, Huawei, Baidu, Tencent, etc.\nRecently, our SkyWalking main repository overs 300 contributors.\nOur website has thousands of views from most countries in the world every week.\nAlthough we know that, the metrics like GitHub stars and the numbers of open users and contributors, are not a determinant of vibrancy, they do show the trend, we are very proud to share the increased numbers here, too.\nWe double those numbers and are honored with the development of our community.\nThank you, all of our contributors. Not just these 300 contributors of the main repository, or nearly 400 contributors in all repositories, counted by GitHub. There are countless people contributing codes to SkyWalking\u0026rsquo;s subprojects, ecosystem projects, and private fork versions; writing blogs and guidances, translating documents, books, and presentations; setting up learning sessions for new users; convincing friends to join the community as end-users, contributors, even committers. Companies behinds those contributors support their employees to work with the community to provide feedback and contribute the improvements and features upstream. Conference organizers share the stages with speakers from the SkyWalking community.\nSkyWalking can’t make this happen without your help. You made this community extraordinary.\nAt this crazy distributed computing and cloud native age, we as a community could make DEV, OPS, and SRE teams\u0026rsquo; work easier by locating the issue(s) in the haystack quicker than before, like why we named the project as SkyWalking, we will have a clear site line when you stand on the glass bridge Skywalk at Grand Canyon West.\n376 Contributors counted by GitHub account are following. Dec. 22st, 2020. Generated by a tool deveoped by Yousa\n1095071913 50168383 Ahoo-Wang AirTrioa AlexanderWert AlseinX Ax1an BFergerson BZFYS CharlesMaster ChaunceyLin5152 CommissarXia Cvimer Doublemine ElderJames EvanLjp FatihErdem FeynmanZhou Fine0830 FingerLiu Gallardot GerryYuan HackerRookie Heguoya Hen1ng Humbertzhang IanCao IluckySi Indifer J-Cod3r JaredTan95 Jargon96 Jijun JohnNiang Jozdortraz Jtrust Just-maple KangZhiDong LazyLei LiWenGu Liu-XinYuan Miss-you O-ll-O Patrick0308 QHWG67 Qiliang RandyAbernethy RedzRedz Runrioter SataQiu ScienJus SevenPointOld ShaoHans Shikugawa SoberChina SummerOfServenteen TJ666 TerrellChen TheRealHaui TinyAllen TomMD ViberW Videl WALL-E WeihanLi WildWolfBang WillemJiang Wooo0 XhangUeiJong Xlinlin YczYanchengzhe YoungHu YunaiV ZhHong ZhuoSiChen ZS-Oliver a198720 a526672351 acurtain adamni135 adermxzs adriancole aeolusheath agile6v aix3 aiyanbo ajanthan alexkarezin alonelaval amogege amwyyyy arugal ascrutae augustowebd bai-yang beckhampu beckjin beiwangnull bigflybrother bostin brucewu-fly c1ay candyleer carlvine500 carrypann cheenursn cheetah012 chenpengfei chenvista chess-equality chestarss chidaodezhongsheng chopin-d clevertension clk1st cngdkxw codeglzhang codelipenghui coder-yqj coki230 coolbeevip crystaldust cui-liqiang cuiweiwei cyberdak cyejing dagmom dengliming devkanro devon-ye dimaaan dingdongnigetou dio dmsolr dominicqi donbing007 dsc6636926 duotai dvsv2 dzx2018 echooymxq efekaptan eoeac evanxuhe feelwing1314 fgksgf fuhuo geektcp geomonlin ggndnn gitter-badger glongzh gnr163 gonedays grissom-grissom grissomsh guodongq guyukou gxthrj gzshilu hailin0 hanahmily haotian2015 haoyann hardzhang harvies hepyu heyanlong hi-sb honganan hsoftxl huangyoje huliangdream huohuanhuan innerpeacez itsvse jasonz93 jialong121 jinlongwang jjlu521016 jjtyro jmjoy jsbxyyx justeene juzhiyuan jy00464346 kaanid karott kayleyang kevinyyyy kezhenxu94 kikupotter kilingzhang killGC klboke ksewen kuaikuai kun-song kylixs landonzeng langke93 langyan1022 langyizhao lazycathome leemove leizhiyuan libinglong lilien1010 limfriend linkinshi linliaoy liuhaoXD liuhaoyang liuyanggithup liuzhengyang liweiv lkxiaolou llissery louis-zhou lpf32 lsyf lucperkins lujiajing1126 lunamagic1978 lunchboxav lxliuxuankb lytscu lyzhang1999 magic-akari makingtime maolie masterxxo maxiaoguang64 membphis mestarshine mgsheng michaelsembwever mikkeschiren mm23504570 momo0313 moonming mrproliu muyun12 nacx neatlife neeuq nic-chen nikitap492 nileblack nisiyong novayoung oatiz oflebbe olzhy onecloud360 osiriswd peng-yongsheng pengweiqhca potiuk purgeyao qijianbo010 qinhang3 qiuyu-d qqeasonchen qxo raybi-asus refactor2 remicollet rlenferink rootsongjc rovast scolia sdanzo seifeHu shiluo34 sikelangya simonlei sk163 snakorse songzhendong songzhian sonxy spacewander stalary stenio2011 stevehu stone-wlg sungitly surechen swartz-k sxzaihua tanjunchen tankilo taskmgr tbdpmi terranhu terrymanu tevahp thanq thebouv tianyuak tincopper tinyu0 tom-pytel tristaZero tristan-tsl trustin tsuilouis tuohai666 tzsword-2020 tzy1316106836 vcjmhg vision-ken viswaramamoorthy wankai123 wbpcode web-xiaxia webb2019 weiqiang333 wendal wengangJi wenjianzhang whfjam wind2008hxy withlin wqr2016 wu-sheng wuguangkuo wujun8 wuxingye x22x22 xbkaishui xcaspar xiaoxiangmoe xiaoy00 xinfeingxia85 xinzhuxiansheng xudianyang yanbw yanfch yang-xiaodong yangxb2010000 yanickxia yanmaipian yanmingbi yantaowu yaowenqiang yazong ychandu ycoe yimeng yu199195 yuqichou yuyujulin yymoth zaunist zaygrzx zcai2 zeaposs zhang98722 zhanghao001 zhangjianweibj zhangkewei zhangsean zhaoyuguang zhentaoJin zhousiliang163 zhuCheer zifeihan zkscpqm zoidbergwill zoumingzm zouyx zshit zxbu zygfengyuwuzu ","excerpt":"\u003cp\u003eApache SkyWalking is an open source APM for distributed system. Provide tracing, service mesh …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2021-01-01-300-contributors-mark/","title":"Celebrate SkyWalking single repository hits the 300 contributors mark"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/open-source-contribution/","title":"Open Source Contribution"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/open-source-promotion-plan/","title":"Open Source Promotion Plan"},{"body":"Ke Zhang (a.k.a. HumbertZhang) mainly focuses on the SkyWalking Python agent, he had participated in the \u0026ldquo;Open Source Promotion Plan - Summer 2020\u0026rdquo; and completed the project smoothly, and won the award \u0026ldquo;Most Potential Students\u0026rdquo; that shows his great willingness to continuously contribute to our community.\nUp to date, he has submitted 8 PRs in the Python agent repository, 7 PRs in the main repo, all in total include ~2000 LOC.\nAt Dec. 13th, 2020, the project management committee (PMC) passed the proposal of promoting him as a new committer. He has accepted the invitation at the same day.\nWelcome to join the committer team, Ke Zhang!\n","excerpt":"\u003cp\u003eKe Zhang (a.k.a. \u003ca href=\"https://github.com/HumbertZhang\"\u003eHumbertZhang\u003c/a\u003e) mainly focuses on the SkyWalking Python agent, he had participated in …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-ke-zhang-as-new-committer/","title":"Welcome Ke Zhang (张可) as new committer"},{"body":"今年暑假期间我参加了开源软件供应链点亮计划—暑期 2020 的活动，在这个活动中，我主要参加了 Apache SkyWalking 的 Python Agent 的开发，最终项目顺利结项并获得了”最具潜力奖“，今天我想分享一下我参与这个活动以及开源社区的感受与收获。\n缘起 其实我在参加暑期 2020 活动之前就听说过 SkyWalking 了。我研究生的主要研究方向是微服务和云原生，组里的学长们之前就在使用 SkyWalking 进行一些研究工作，也是通过他们，我了解到了 OpenTracing, SkyWalking 等与微服务相关的 Tracing 工具以及 APM 等，当时我就在想如果有机会可以深度参加这些开源项目就好了。 巧的是，也正是在差不多的时候，本科的一个学长发给了我暑期 2020 活动的链接，我在其中惊喜的发现了 SkyWalking 项目。\n虽然说想要参与 SkyWalking 的开发，但是真的有了机会我却有一些不自信——这可是 Star 上万的 Apache 顶级项目。万幸的是在暑期 2020 活动中，每一个社区都提供了很多题目以供选择，想参与的同学可以提前对要做的事情有所了解，并可以提前做一些准备。我当时也仔细地浏览了项目列表，最终决定申请为 Python Agent 支持 Flask 或 Django 埋点的功能。当时主要考虑的是，我对 Python 语言比较熟悉，同时也有使用 Flask 等 web 框架进行开发的经验，我认为应该可以完成项目要求。为了能让心里更有底一些，我阅读了 Python Agent 的源码，写下了对项目需要做的工作的理解，并向项目的导师柯振旭发送了自荐邮件，最终被选中去完成这个项目。\n过程 被选中后我很激动，也把这份激动化作了参与开源的动力。我在进一步阅读源码，搭建本地环境后，用了三周左右的时间完成了 Django 项目的埋点插件的开发，毕竟我选择的项目是一个低难度的项目，而我在 Python web 方面也有一些经验。在这之后，我的导师和我进行了沟通，在我表达了想要继续做贡献的意愿之后，他给我建议了一些可以进一步进行贡献的方向，我也就继续参与 Python Agent 的开发。接下来，我陆续完成了 PyMongo 埋点插件, 插件版本检查机制, 支持使用 kafka 协议进行数据上报等功能。在提交了暑期 2020 活动的结项申请书后，我又继续参与了在端到端测试中增加对百分位数的验证等功能。\n在整个过程中，我遇到过很多问题，包括对问题认识不够清晰，功能的设计不够完善等等，但是通过与导师的讨论以及 Code Review，这些问题最终都迎刃而解了。此外他还经常会和我交流项目进一步发展方向，并给我以鼓励和肯定，在这里我想特别感谢我的导师在整个项目过程中给我的各种帮助。\n收获 参加暑期 2020 的活动带给我了很多收获，主要有以下几点：\n第一是让我真正参与到了开源项目中。在之前我只向在项目代码或文档中发现的 typo 发起过一些 Pull Request，但是暑期 2020 活动通过列出项目 ＋ 导师指导的方式，明确了所要做的事情，并提供了相应的指导，降低了参与开源的门槛，使得我们学生可以参与到项目的开发中来。\n第二是对我的专业研究方向也有很多启发，我的研究方向就是微服务与云原生相关，通过参与到 SkyWalking 的开发中使得我可以更好地理解研究问题中的一些概念，也让我更得心应手得使用 SkyWalking 来解决一些实际的问题。\n第三是通过参与 SkyWalking Python Agent 以及其他部分的开发，我的贡献得到了社区的承认，并在最近被邀请作为 Committer 加入了社区，这对我而言是很高的认可，也提升了我的自信心。\n​\t第四点就是我通过这个活动认识了不少新朋友，同时也开拓了我的视野，使得我对于开源项目与开源社区有了很多新的认识。\n建议 最后同样是我对想要参与开源社区，想要参与此类活动的同学们的一些建议：\n虽然奖金很吸引人，但是还是希望大家能抱着长期为项目进行贡献的心态来参与开源项目，以这样的心态参与开源可以让你更好地理解开源社区的运作方式，也可以让你更有机会参与完成激动人心的功能，你在一个东西上付出的时间精力越多，你能收获的往往也越多。 在申请项目的时候，可以提前阅读一下相关功能的源码，并结合自己的思考去写一份清晰明了的 proposal ，这样可以帮助你在申请人中脱颖而出。 在开始着手去完成一个功能之前，首先理清思路，并和自己的导师或了解这一部分的人进行沟通与确认，从而尽量避免在错误的方向上浪费太多时间。 ","excerpt":"\u003cp\u003e今年暑假期间我参加了开源软件供应链点亮计划—暑期 2020 的活动，在这个活动中，我主要参加了 Apache SkyWalking 的 Python Agent 的开发，最终项目顺利结项并获得了”最具 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2020-12-20-summer2020-activity-sharing2/","title":"暑期 2020 活动学生（张可）心得分享"},{"body":"背景 我是一个热爱编程、热爱技术的人，⼀直以来都向往着能参与到开源项⽬中锻炼⾃⼰，但当我面对庞大而复杂的项目代码时，却感到手足无措，不知该从何开始。⽽此次的“开源软件供应链点亮计划-暑期2020”活动则正好提供了这样⼀个机会：清晰的任务要求、开源社区成员作为导师提供指导以及一笔丰厚的奖金，让我顺利地踏上了开源这条道路。\n回顾 在“暑期2020”活动的这两个多月里，我为 SkyWalking 的命令行工具实现了一个 dashboard，此外在阅读项目源码的过程中，还发现并修复了几个 bug。到活动结束时，我共提交了11个 PR，贡献了两千多行改动，对 SkyWalking CLI 项目的贡献数量排名第二，还获得了“最具潜力奖”。\n我觉得之所以能够如此顺利地完成这个项⽬主要有两个原因。一方面，我选择的 SkyWalking CLI 项⽬当时最新的版本号为0.3.0，还处于起步阶段，代码量相对较少，⽽且项⽬结构非常清晰，文档也较为详细，这对于我理解整个项⽬⾮常有帮助，从⽽能够更快地上⼿。另一方面，我的项目导师非常认真负责，每次我遇到问题，导师都会及时地为我解答，然后我提交的 PR 也能够很快地被 review。⽽且导师不时会给予我肯定的评论与⿎励，这极⼤地提⾼了我的成就感，让我更加积极地投⼊到下⼀阶段的⼯作，形成⼀个正向的循环。\n收获 回顾整个参与过程，觉得自己收获颇多：\n首先，我学习到了很多可能在学校里接触不到的新技术，了解了开源项目是如何进行协作，开源社区是如何运转治理的，以及开源文化、Apache way 等知识，仿佛进入了一个崭新而精彩的世界。\n其次，我的编程能力得到了锻炼。因为开源项目对于代码的质量有较高的要求，因此我会在编程时有意识地遵守相关的规范，培养良好的编码习惯。然后在导师的 code review 中也学习到了一些编程技巧。\n此外，参与开源为我的科研带来了不少灵感。因为我的研究方向是智能软件工程，旨在将人工智能技术应用在软件工程的各个环节中，这需要我在实践中发现实际问题。而开源则提供了这样一个窗口，让我足不出户即可参与到软件项目的设计、开发、测试和发布等环节。\n最后也是本次活动最大的一个收获，我的贡献得到了社区的认可，被提名成为了 SkyWalking 社区的第一位学生 committer。\n建议 最后，对于将来想要参加此类活动的同学，附上我的一些建议：\n第一，选择活跃、知名的社区。社区对你的影响将是极其深远的，好的社区意味着成熟的协作流程、良好的氛围、严谨的代码规范，以及有更大几率遇到优秀的导师，这些对于你今后在开源方面的发展都是非常有帮助的。\n第二，以兴趣为导向来选择项目，同时要敢于走出舒适区。我最初在选择项目时，初步确定了两个，一个是低难度的 Python 项目，另一个是中等难度的 Go 项目。当时我很纠结：因为我对 Python 语言比较熟悉，选择一个低难度的项目是比较稳妥的，但是项目的代码我看的并不是很懂，具体要怎么做我完全没有头绪；而 Go 项目是一个命令行工具，我对这个比较感兴趣，且有一个大致的思路，但是我对 Go 语言并不是很熟悉，实践经验为零。最后凭借清晰具体的 proposal 我成功申请到了 Go 项目并顺利地完成了，还在实践中快速掌握了一门新的编程语言。\n这次的“暑期2020”活动虽已圆满结束，但我的开源之路才刚刚开始。\n","excerpt":"\u003ch2 id=\"背景\"\u003e背景\u003c/h2\u003e\n\u003cp\u003e我是一个热爱编程、热爱技术的人，⼀直以来都向往着能参与到开源项⽬中锻炼⾃⼰，但当我面对庞大而复杂的项目代码时，却感到手足无措，不知该从何开始。⽽此次的“开源软件供应链点亮计划-暑期2020”活动 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2020-12-19-summer2020-activity-sharing/","title":"暑期2020活动心得分享"},{"body":"NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It\u0026rsquo;s built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.\nGoal Keep the feature parity with the existing E2E framework in SkyWalking main repo; Support both docker-compose and KinD to orchestrate the tested services under different environments; Get rid of the heavy Java/Maven stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes; Non-Goal This framework is not involved with the build process, i.e. it won\u0026rsquo;t do something like mvn package or docker build, the artifacts (.tar, docker images) should be ready in an earlier process before this; This project doesn\u0026rsquo;t take the plugin tests into account, at least for now; This project doesn\u0026rsquo;t mean to add/remove any new/existing test case to/from the main repo; This documentation won\u0026rsquo;t cover too much technical details of how to implement the framework, that should go into an individual documentation; Design Before diving into the design details, let\u0026rsquo;s take a quick look at how the end user might use NGE2E.\nAll the following commands are mock, and are open to debate.\nTo run a test case in a directory /path/to/the/case/directory\ne2e run /path/to/the/case/directory # or cd /path/to/the/case/directory \u0026amp;\u0026amp; e2e run This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:\nNOTE: because all the options can be loaded from a configuration file, so as long as a configuration file (say e2e.yaml) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);\nSet Up e2e setup --env=compose --file=docker-compose.yaml --wait-for=service/health e2e setup --env=kind --file=kind.yaml --manifests=bookinfo.yaml,gateway.yaml --wait-for=pod/ready e2e setup # If configuration file e2e.yaml is present --env: the environment, may be compose or kind, represents docker-compose and KinD respectively; --file: the docker-compose.yaml or kind.yaml file that declares how to set up the environment; --manifests: for KinD, the resources files/directories to apply (using kubectl apply -f); --command: a command to run after the environment is started, this may be useful when users need to install some extra tools or apply resources from command line, like istioctl install --profile=demo; --wait-for: can be specified multiple times to give a list of conditions to be met; wait until the given conditions are met; the most frequently-used strategy should be --wait-for=service/health, --wait-for=deployments/available, etc. that make the e2e setup command to wait for all conditions to be met; other possible strategies may be something like --wait-for=\u0026quot;log:Started Successfully\u0026quot;, --wait-for=\u0026quot;http:localhost:8080/healthcheck\u0026quot;, etc. if really needed; Trigger Inputs e2e trigger --interval=3s --times=0 --action=http --url=\u0026#34;localhost:8080/users\u0026#34; e2e trigger --interval=3s --times=0 --action=cmd --cmd=\u0026#34;curl localhost:8080/users\u0026#34; e2e trigger # If configuration file e2e.yaml is present --interval=3s: trigger the action every 3 seconds; --times=0: how many times to trigger the action, 0=infinite; --action=http: the action of the trigger, i.e. \u0026ldquo;perform an http request as an input\u0026rdquo;; --action=cmd: the action of the trigger, i.e. \u0026ldquo;execute the cmd as an input\u0026rdquo;; Query Output swctl service ls this is a project-specific step, different project may use different tools to query the actual output, for SkyWalking, it uses swctl to query the actual output.\nVerify e2e verify --actual=actual.data.yaml --expected=expected.data.yaml e2e verify --query=\u0026#34;swctl service ls\u0026#34; --expected=expected.data.yaml e2e verify # If configuration file e2e.yaml is present --actual: the actual data file, only YAML file format is supported;\n--expected: the expected data file, only YAML file format is supported;\n--query: the query to get the actual data, the query result must have the same format as --actual and --expected;\nThe --query option will get the output into a temporary file and use the --actual under the hood;\nCleanup e2e cleanup --env=compose --file=docker-compose.yaml e2e cleanup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml e2e cleanup # If configuration file e2e.yaml is present This step requires the same options in the setup step so that it can clean up all things necessarily.\nSummarize To summarize, the directory structure of a test case might be\ncase-name ├── agent-service # optional, an arbitrary project that is used in the docker-compose.yaml if needed │ ├── Dockerfile │ ├── pom.xml │ └── src ├── docker-compose.yaml ├── e2e.yaml # see a sample below └── testdata ├── expected.endpoints.service1.yaml ├── expected.endpoints.service2.yaml └── expected.services.yaml or\ncase-name ├── kind.yaml ├── bookinfo │ ├── bookinfo.yaml │ └── bookinfo-gateway.yaml ├── e2e.yaml # see a sample below └── testdata ├── expected.endpoints.service1.yaml ├── expected.endpoints.service2.yaml └── expected.services.yaml a sample of e2e.yaml may be\nsetup: env: kind file: kind.yaml manifests: - path: bookinfo.yaml wait: # you can have multiple conditions to wait - namespace: bookinfo label-selector: app=product for: deployment/available - namespace: reviews label-selector: app=product for: deployment/available - namespace: ratings label-selector: app=product for: deployment/available run: - command: | # it can be a shell script or anything executable istioctl install --profile=demo -y kubectl label namespace default istio-injection=enabled wait: - namespace: istio-system label-selector: app=istiod for: deployment/available # OR # env: compose # file: docker-compose.yaml trigger: action: http interval: 3s times: 0 url: localhost:9090/users verify: - query: swctl service ls expected: expected.services.yaml - query: swctl endpoint ls --service=\u0026#34;YnVzaW5lc3Mtem9uZTo6cHJvamVjdEM=.1\u0026#34; expected: expected.projectC.endpoints.yaml then a single command should do the trick.\ne2e run Modules This project is divided into the following modules.\nController A controller command (e2e run) composes all the steps declared in the e2e.yaml, it should be progressive and clearly display which step is currently running. If it failed in a step, the error message should be as much comprehensive as possible. An example of the output might be\ne2e run ✔ Started Kind Cluster - Cluster Name ✔ Checked Pods Readiness - All pods are ready ? Generating Traffic - http localhost:9090/users (progress spinner) ✔ Verified Output - service ls (progress spinner) Verifying Output - endpoint ls ✘ Failed to Verify Output Data - endpoint ls \u0026lt;the diff content\u0026gt; ✔ Clean Up Compared with running the steps one by one, the controller is also responsible for cleaning up env (by executing cleanup command) no mater what status other commands are, even if they are failed, the controller has the following semantics in terms of setup and cleanup.\n// Java try { setup(); // trigger step // verify step // ... } finally { cleanup(); } // GoLang func run() { setup(); defer cleanup(); // trigger step // verify step // ... } Initializer The initializer is responsible for\nWhen env==compose\nStart the docker-compose services; Check the services\u0026rsquo; healthiness; Wait until all services are ready according to the interval, etc.; When env==kind\nStart the KinD cluster according to the config files; Apply the resources files (--manifests) or/and run the custom init command (--commands); Check the pods\u0026rsquo; readiness; Wait until all pods are ready according to the interval, etc.; Verifier According to scenarios we have at the moment, the must-have features are:\nMatchers\nExact match Not null Not empty Greater than 0 Regexp match At least one of list element match Functions\nBase64 encode/decode in order to help to identify simple bugs from the GitHub Actions workflow, there are some \u0026ldquo;nice to have\u0026rdquo; features:\nPrinting the diff content when verification failed is a super helpful bonus proved in the Python agent repo; Logging When a test case failed, all the necessary logs should be collected into a dedicated directory, which could be uploaded to the GitHub Artifacts for downloading and analysis;\nLogs through the entire process of a test case are:\nKinD clusters logs; Containers/pods logs; The logs from the NGE2E itself; More Planned Debugging Debugging the E2E locally has been a strong requirement and time killer that we haven\u0026rsquo;t solve up to date, though we have enhancements like https://github.com/apache/skywalking/pull/5198 , but in this framework, we will adopt a new method to \u0026ldquo;really\u0026rdquo; support debugging locally.\nThe most common case when debugging is to run the E2E tests, with one or more services forwarded into the host machine, where the services are run in the IDE or in debug mode.\nFor example, you may run the SkyWalking OAP server in an IDE and run e2e run, expecting the other services (e.g. agent services, SkyWalking WebUI, etc.) inside the containers to connect to your local OAP, instead of the one declared in docker-compose.yaml.\nFor Docker Desktop Mac/Windows, we can access the services running on the host machine inside containers via host.docker.internal, for Linux, it\u0026rsquo;s 172.17.0.1.\nOne possible solution is to add an option --debug-services=oap,other-service-name that rewrites all the router rules inside the containers from oap to host.docker.internal/172.17.0.1.\nCodeGen When adding new test case, a code generator would be of great value to eliminate the repeated labor and copy-pasting issues.\ne2e new \u0026lt;case-name\u0026gt; ","excerpt":"\u003cp\u003eNGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/e2e-design/","title":"[Design] NGE2E - Next Generation End-to-End Testing Framework"},{"body":"这篇文章暂时不讲告警策略, 直接看默认情况下激活的告警目标以及钉钉上的告警效果\nSkyWalking内置了很多默认的告警策略, 然后根据告警策略生成告警目标, 我们可以很容易的在界面上看到\n当我们想去让这些告警目标通知到我们时, 由于SkyWalking目前版本(8.3)已经自带了, 只需要简单配置一下即可\n我们先来钉钉群中创建机器人并勾选加签\n然后再修改告警部分的配置文件, 如果你是默认的配置文件(就像我一样), 你可以直接执行以下命令, 反之你也可以手动修改configs/alarm-settings.yml文件\ntee \u0026lt;your_skywalking_path\u0026gt;/configs/alarm-settings.yml \u0026lt;\u0026lt;-\u0026#39;EOF\u0026#39; dingtalkHooks: textTemplate: |- { \u0026#34;msgtype\u0026#34;: \u0026#34;text\u0026#34;, \u0026#34;text\u0026#34;: { \u0026#34;content\u0026#34;: \u0026#34;Apache SkyWalking Alarm: \\n %s.\u0026#34; } } webhooks: - url: https://oapi.dingtalk.com/robot/send?access_token=\u0026lt;access_token\u0026gt; secret: \u0026lt;加签值\u0026gt; EOF 最终效果如下\n参考文档:\nhttps://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-alarm.md\nhttps://ding-doc.dingtalk.com/doc#/serverapi2/qf2nxq/uKPlK\n谢谢观看, 后续我会在SkyWalking告警这块写更多实战文章\n","excerpt":"\u003cp\u003e这篇文章暂时不讲告警策略, 直接看默认情况下激活的告警目标以及钉钉上的告警效果\u003c/p\u003e\n\u003cp\u003eSkyWalking内置了很多默认的告警策略, 然后根据告警策略生成告警目标, 我们可以很容易的在界面上看到\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"image-20201213163408221.png\" alt=\"image-20201213163408221\"\u003e\u003c/p\u003e\n\u003cp\u003e当我们想 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2020-12-13-skywalking-alarm/","title":"SkyWalking报警发送到钉钉群"},{"body":"Gui Cao began the code contributions since May 3, 2020. In the past 6 months, his 23 pull requests(GitHub, zifeihan[1]) have been accepted, which includes 5k+ lines of codes.\nMeanwhile, he took part in the tech discussion, and show the interests to contribute more to the project.\nAt Dec. 4th, 2020, the project management committee(PMC) passed the proposal of promoting him as a new committer. He has accepted the invitation at the same day.\nWelcome Gui Cao join the committer team.\n[1] https://github.com/apache/skywalking/commits?author=zifeihan\n","excerpt":"\u003cp\u003eGui Cao began the code contributions since May 3, 2020.\nIn the past 6 months, his 23 pull …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-gui-cao-as-new-committer/","title":"Welcome Gui Cao as new committer"},{"body":"\nAuthor: Zhenxu Ke, Sheng Wu, and Tevah Platt. tetrate.io Original link, Tetrate.io blog Dec. 03th, 2020 Apache SkyWalking: an APM (application performance monitor) system, especially designed for microservices, cloud native, and container-based (Docker, Kubernetes, Mesos) architectures.\nEnvoy Access Log Service: Access Log Service (ALS) is an Envoy extension that emits detailed access logs of all requests going through Envoy.\nBackground Apache SkyWalking has long supported observability in service mesh with Istio Mixer adapter. But since v1.5, Istio began to deprecate Mixer due to its poor performance in large scale clusters. Mixer’s functionalities have been moved into the Envoy proxies, and is supported only through the 1.7 Istio release. On the other hand, Sheng Wu and Lizan Zhou presented a better solution based on the Apache SkyWalking and Envoy ALS on KubeCon China 2019, to reduce the performance impact brought by Mixer, while retaining the same observability in service mesh. This solution was initially implemented by Sheng Wu, Hongtao Gao, Lizan Zhou, and Dhi Aurrahman at Tetrate.io. If you are looking for a more efficient solution to observe your service mesh instead of using a Mixer-based solution, this is exactly what you need. In this tutorial, we will explain a little bit how the new solution works, and apply it to the bookinfo application in practice.\nHow it works From a perspective of observability, Envoy can be typically deployed in 2 modes, sidecar, and router. As a sidecar, Envoy mostly represents a single service to receive and send requests (2 and 3 in the picture below). While as a proxy, Envoy may represent many services (1 in the picture below).\nIn both modes, the logs emitted by ALS include a node identifier. The identifier starts with router~ (or ingress~) in router mode and sidecar~ in sidecar proxy mode.\nApart from the node identifier, there are several noteworthy properties in the access logs that will be used in this solution:\ndownstream_direct_remote_address: This field is the downstream direct remote address on which the request from the user was received. Note: This is always the physical peer, even if the remote address is inferred from for example the x-forwarded-for header, proxy protocol, etc.\ndownstream_remote_address: The remote/origin address on which the request from the user was received.\ndownstream_local_address: The local/destination address on which the request from the user was received.\nupstream_remote_address: The upstream remote/destination address that handles this exchange.\nupstream_local_address: The upstream local/origin address that handles this exchange.\nupstream_cluster: The upstream cluster that upstream_remote_address belongs to.\nWe will discuss more about the properties in the following sections.\nSidecar When serving as a sidecar, Envoy is deployed alongside a service, and delegates all the incoming/outgoing requests to/from the service.\nDelegating incoming requests: in this case, Envoy acts as a server side sidecar, and sets the upstream_cluster in form of inbound|portNumber|portName|Hostname[or]SidecarScopeID.\nThe SkyWalking analyzer checks whether either downstream_remote_address can be mapped to a Kubernetes service:\na. If there is a service (say Service B) whose implementation is running in this IP(and port), then we have a service-to-service relation, Service B -\u0026gt; Service A, which can be used to build the topology. Together with the start_time and duration fields in the access log, we have the latency metrics now.\nb. If there is no service that can be mapped to downstream_remote_address, then the request may come from a service out of the mesh. Since SkyWalking cannot identify the source service where the requests come from, it simply generates the metrics without source service, according to the topology analysis method. The topology can be built as accurately as possible, and the metrics detected from server side are still correct.\nDelegating outgoing requests: in this case, Envoy acts as a client-side sidecar, and sets the upstream_cluster in form of outbound|\u0026lt;port\u0026gt;|\u0026lt;subset\u0026gt;|\u0026lt;serviceFQDN\u0026gt;.\nClient side detection is relatively simpler than (1. Delegating incoming requests). If upstream_remote_address is another sidecar or proxy, we simply get the mapped service name and generate the topology and metrics. Otherwise, we have no idea what it is and consider it an UNKNOWN service.\nProxy role When Envoy is deployed as a proxy, it is an independent service itself and doesn\u0026rsquo;t represent any other service like a sidecar does. Therefore, we can build client-side metrics as well as server-side metrics.\nExample In this section, we will use the typical bookinfo application to demonstrate how Apache SkyWalking 8.3.0+ (the latest version up to Nov. 30th, 2020) works together with Envoy ALS to observe a service mesh.\nInstalling Kubernetes SkyWalking 8.3.0 supports the Envoy ALS solution under both Kubernetes environment and virtual machines (VM) environment, in this tutorial, we’ll only focus on the Kubernetes scenario, for VM solution, please stay tuned for our next blog, so we need to install Kubernetes before taking further steps.\nIn this tutorial, we will use the Minikube tool to quickly set up a local Kubernetes(v1.17) cluster for testing. In order to run all the needed components, including the bookinfo application, the SkyWalking OAP and WebUI, the cluster may need up to 4GB RAM and 2 CPU cores.\nminikube start --memory=4096 --cpus=2 Next, run kubectl get pods --namespace=kube-system --watch to check whether all the Kubernetes components are ready. If not, wait for the readiness before going on.\nInstalling Istio Istio provides a very convenient way to configure the Envoy proxy and enable the access log service. The built-in configuration profiles free us from lots of manual operations. So, for demonstration purposes, we will use Istio through this tutorial.\nexport ISTIO_VERSION=1.7.1 curl -L https://istio.io/downloadIstio | sh - sudo mv $PWD/istio-$ISTIO_VERSION/bin/istioctl /usr/local/bin/ istioctl install --set profile=demo kubectl label namespace default istio-injection=enabled Run kubectl get pods --namespace=istio-system --watch to check whether all the Istio components are ready. If not, wait for the readiness before going on.\nEnabling ALS The demo profile doesn’t enable ALS by default. We need to reconfigure it to enable ALS via some configuration.\nistioctl manifest install \\ --set meshConfig.enableEnvoyAccessLogService=true \\ --set meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-oap.istio-system:11800 The example command --set meshConfig.enableEnvoyAccessLogService=true enables the Envoy access log service in the mesh. And as we said earlier, ALS is essentially a gRPC service that emits requests logs. The config meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-oap.istio-system:11800 tells this gRPC service where to emit the logs, say skywalking-oap.istio-system:11800, where we will deploy the SkyWalking ALS receiver later.\nNOTE: You can also enable the ALS when installing Istio so that you don’t need to restart Istio after installation:\nistioctl install --set profile=demo \\ --set meshConfig.enableEnvoyAccessLogService=true \\ --set meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-oap.istio-system:11800 kubectl label namespace default istio-injection=enabled Deploying Apache SkyWalking The SkyWalking community provides a Helm Chart to make it easier to deploy SkyWalking and its dependent services in Kubernetes. The Helm Chart can be found at the GitHub repository.\n# Install Helm curl -sSLO https://get.helm.sh/helm-v3.0.0-linux-amd64.tar.gz sudo tar xz -C /usr/local/bin --strip-components=1 linux-amd64/helm -f helm-v3.0.0-linux-amd64.tar.gz # Clone SkyWalking Helm Chart git clone https://github.com/apache/skywalking-kubernetes cd skywalking-kubernetes/chart git reset --hard dd749f25913830c47a97430618cefc4167612e75 # Update dependencies helm dep up skywalking # Deploy SkyWalking helm -n istio-system install skywalking skywalking \\ --set oap.storageType=\u0026#39;h2\u0026#39;\\ --set ui.image.tag=8.3.0 \\ --set oap.image.tag=8.3.0-es7 \\ --set oap.replicas=1 \\ --set oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh \\ --set oap.env.JAVA_OPTS=\u0026#39;-Dmode=\u0026#39; \\ --set oap.envoy.als.enabled=true \\ --set elasticsearch.enabled=false We deploy SkyWalking to the namespace istio-system, so that SkyWalking OAP service can be accessed by skywalking-oap.istio-system:11800, to which we told ALS to emit their logs, in the previous step.\nWe also enable the ALS analyzer in the SkyWalking OAP: oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh. The analyzer parses the access logs and maps the IP addresses in the logs to the real service names in the Kubernetes, to build a topology.\nIn order to retrieve the metadata (such as Pod IP and service names) from a Kubernetes cluster for IP mappings, we also set oap.envoy.als.enabled=true, to apply for a ClusterRole that has access to the metadata.\nexport POD_NAME=$(kubectl get pods -A -l \u0026#34;app=skywalking,release=skywalking,component=ui\u0026#34; -o name) echo $POD_NAME kubectl -n istio-system port-forward $POD_NAME 8080:8080 Now navigate your browser to http://localhost:8080 . You should be able to see the SkyWalking dashboard. The dashboard is empty for now, but after we deploy the demo application and generate traffic, it should be filled up later.\nDeploying Bookinfo application Run:\nexport ISTIO_VERSION=1.7.1 kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/bookinfo-gateway.yaml kubectl wait --for=condition=Ready pods --all --timeout=1200s minikube tunnel Then navigate your browser to http://localhost/productpage. You should be able to see the typical bookinfo application. Refresh the webpage several times to generate enough access logs.\nDone! And you’re all done! Check out the SkyWalking WebUI again. You should see the topology of the bookinfo application, as well the metrics of each individual service of the bookinfo application.\nTroubleshooting Check all pods status: kubectl get pods -A. SkyWalking OAP logs: kubectl -n istio-system logs -f $(kubectl get pod -A -l \u0026quot;app=skywalking,release=skywalking,component=oap\u0026quot; -o name). SkyWalking WebUI logs: kubectl -n istio-system logs -f $(kubectl get pod -A -l \u0026quot;app=skywalking,release=skywalking,component=ui\u0026quot; -o name). Make sure the time zone at the bottom-right of the WebUI is set to UTC +0. Customizing Service Names The SkyWalking community brought more improvements to the ALS solution in the 8.3.0 version. You can decide how to compose the service names when mapping from the IP addresses, with variables service and pod. For instance, configuring K8S_SERVICE_NAME_RULE to the expression ${service.metadata.name}-${pod.metadata.labels.version} gets service names with version label such as reviews-v1, reviews-v2, and reviews-v3, instead of a single service reviews, see the PR.\nWorking ALS with VM Kubernetes is popular, but what about VMs? From what we discussed above, in order to map the IPs to services, SkyWalking needs access to the Kubernetes cluster, fetching service metadata and Pod IPs. But in a VM environment, there is no source from which we can fetch those metadata. In the next post, we will introduce another ALS analyzer based on the Envoy metadata exchange mechanism. With this analyzer, you are able to observe a service mesh in the VM environment. Stay tuned! If you want to have commercial support for the ALS solution or hybrid mesh observability, Tetrate Service Bridge, TSB is another good option out there.\nAdditional Resources KubeCon 2019 Recorded Video. Get more SkyWalking updates on the official website. Apache SkyWalking founder Sheng Wu, SkyWalking core maintainer Zhenxu Ke are Tetrate engineers, and Tevah Platt is a content writer for Tetrate. Tetrate helps organizations adopt open source service mesh tools, including Istio, Envoy, and Apache SkyWalking, so they can manage microservices, run service mesh on any infrastructure, and modernize their applications.\n","excerpt":"\u003cp\u003e\u003cimg src=\"canyonhorseshoe.jpg\" alt=\"\"\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAuthor: Zhenxu Ke, Sheng Wu, and Tevah Platt. tetrate.io\u003c/li\u003e\n\u003cli\u003eOriginal link, \u003ca href=\"https://www.tetrate.io/blog/observe-service-mesh-with-skywalking-and-envoy-access-log-service/\"\u003eTetrate.io blog\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eDec. 03th, …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/blog/2020-12-03-obs-service-mesh-with-sw-and-als/","title":"Observe Service Mesh with SkyWalking and Envoy Access Log Service"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/service-mesh/","title":"Service Mesh"},{"body":"\n如果你正在寻找在 Mixer 方案以外观察服务网格的更优解，本文正符合你的需要。\nApache Skywalking︰特别为微服务、云原生和容器化（Docker、Kubernetes、Mesos）架构而设计的 APM（应用性能监控）系统。\nEnvoy 访问日志服务︰访问日志服务（ALS）是 Envoy 的扩展组件，会将所有通过 Envoy 的请求的详细访问日志发送出来。\n背景 Apache SkyWalking 一直通过 Istio Mixer 的适配器，支持服务网格的可观察性。不过自从 v1.5 版本，由于 Mixer 在大型集群中差强人意的表现，Istio 开始弃用 Mixer。Mixer 的功能现已迁至 Envoy 代理，并获 Istio 1.7 版本支持。\n在去年的中国 KubeCon 中，吴晟和周礼赞基于 Apache SkyWalking 和 Envoy ALS，发布了新的方案：不再受制于 Mixer 带来的性能影响，也同时保持服务网格中同等的可观察性。这个方案最初是由吴晟、高洪涛、周礼赞和 Dhi Aurrahman 在 Tetrate.io 实现的。\n如果你正在寻找在 Mixer 方案之外，为你的服务网格进行观察的最优解，本文正是你当前所需的。在这个教程中，我们会解释此方案的运作逻辑，并将它实践到 bookinfo 应用上。\n运作逻辑 从可观察性的角度来说，Envoy 一般有两种部署模式︰Sidecar 和路由模式。 Envoy 代理可以代表多项服务（见下图之 1），或者当它作为 Sidecar 时，一般是代表接收和发送请求的单项服务（下图之 2 和 3）。\n在两种模式中，ALS 发放的日志都会带有一个节点标记符。该标记符在路由模式时，以 router~ （或 ingress~）开头，而在 Sidecar 代理模式时，则以 sidecar~ 开头。\n除了节点标记符之外，这个方案［1］所采用的访问日志也有几个值得一提的字段︰\ndownstream_direct_remote_address︰此字段是下游的直接远程地址，用作接收来自用户的请求。注意︰它永远是对端实体的地址，即使远程地址是从 x-forwarded-for header、代理协议等推断出来的。\ndownstream_remote_address︰远程或原始地址，用作接收来自用户的请求。\ndownstream_local_address︰本地或目标地址，用作接收来自用户的请求。\nupstream_remote_address︰上游的远程或目标地址，用作处理本次交换。\nupstream_local_address︰上游的本地或原始地址，用作处理本次交换。\nupstream_cluster︰upstream_remote_address 所属的上游集群。\n我们会在下面详细讲解各个字段。\nSidecar 当 Envoy 作为 Sidecar 的时候，会搭配服务一起部署，并代理来往服务的传入或传出请求。\n代理传入请求︰在此情况下，Envoy 会作为服务器端的 Sidecar，以 inbound|portNumber|portName|Hostname[or]SidecarScopeID 格式设定 upstream_cluster。\nSkyWalking 分析器会检查 downstream_remote_address 是否能够找到对应的 Kubernetes 服务。\n如果在此 IP（和端口）中有一个服务（例如服务 B）正在运行，那我们就会建立起服务对服务的关系（即服务 B → 服务 A），帮助建立拓扑。再配合访问日志中的 start_time 和 duration 两个字段，我们就可以获得延迟的指标数据了。\n如果没有任何服务可以和 downstream_remote_address 相对应，那请求就有可能来自网格以外的服务。由于 SkyWalking 无法识别请求的服务来源，在没有源服务的情况下，它简单地根据拓扑分析方法生成数据。拓扑依然可以准确地建立，而从服务器端侦测出来的指标数据也依然是正确的。\n代理传出请求︰在此情况下，Envoy 会作为客户端的 Sidecar，以 outbound|\u0026lt;port\u0026gt;|\u0026lt;subset\u0026gt;|\u0026lt;serviceFQDN\u0026gt; 格式设定 upstream_cluster。\n客户端的侦测相对来说比代理传入请求容易。如果 upstream_remote_address 是另一个 Sidecar 或代理的话，我们只需要获得它相应的服务名称，便可生成拓扑和指标数据。否则，我们没有办法理解它，只能把它当作 UNKNOWN 服务。\n代理角色 当 Envoy 被部署为前端代理时，它是独立的服务，并不会像 Sidecar 一样，代表任何其他的服务。所以，我们可以建立客户端以及服务器端的指标数据。\n演示范例 在本章，我们会使用典型的 bookinfo 应用，来演示 Apache SkyWalking 8.3.0+ （截至 2020 年 11 月 30 日的最新版本）如何与 Envoy ALS 合作，联手观察服务网格。\n安装 Kubernetes 在 Kubernetes 和虚拟机器（VM）的环境下，SkyWalking 8.3.0 均支持 Envoy ALS 的方案。在本教程中，我们只会演示在 Kubernetes 的情境，至于 VM 方案，请耐心期待我们下一篇文章。所以在进行下一步之前，我们需要先安装 Kubernetes。\n在本教程中，我们会使用 Minikube 工具来快速设立本地的 Kubernetes（v1.17 版本）集群用作测试。要运行所有必要组件，包括 bookinfo 应用、SkyWalking OAP 和 WebUI，集群需要动用至少 4GB 内存和 2 个 CPU 的核心。\nminikube start --memory=4096 --cpus=2 然后，运行 kubectl get pods --namespace=kube-system --watch，检查所有 Kubernetes 的组件是否已准备好。如果还没，在进行下一步前，请耐心等待准备就绪。\n安装 Istio Istio 为配置 Envoy 代理和实现访问日志服务提供了一个非常方便的方案。内建的配置设定档为我们省去了不少手动的操作。所以，考虑到演示的目的，我们会在本教程全程使用 Istio。\nexport ISTIO_VERSION=1.7.1 curl -L https://istio.io/downloadIstio | sh - sudo mv $PWD/istio-$ISTIO_VERSION/bin/istioctl /usr/local/bin/ istioctl install --set profile=demo kubectl label namespace default istio-injection=enabled 然后，运行 kubectl get pods --namespace=istio-system --watch，检查 Istio 的所有组件是否已准备好。如果还没，在进行下一步前，请耐心等待准备就绪。\n启动访问日志服务 演示的设定档没有预设启动 ALS，我们需要重新配置才能够启动 ALS。\nistioctl manifest install \\ --set meshConfig.enableEnvoyAccessLogService=true \\ --set meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-oap.istio-system:11800 范例指令 --set meshConfig.enableEnvoyAccessLogService=true 会在网格中启动访问日志服务。正如之前提到，ALS 本质上是一个会发放请求日志的 gRPC 服务。配置 meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-oap.istio-system:11800 会告诉这个gRPC 服务往哪里发送日志，这里是往 skywalking-oap.istio-system:11800 发送，稍后我们会部署 SkyWalking ALS 接收器到这个地址。\n注意︰\n你也可以在安装 Istio 时启动 ALS，那就不需要在安装后重新启动 Istio︰\nistioctl install --set profile=demo \\ --set meshConfig.enableEnvoyAccessLogService=true \\ --set meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-oap.istio-system:11800 kubectl label namespace default istio-injection=enabled 部署 Apache SkyWalking SkyWalking 社区提供了 Helm Chart ，让你更轻易地在 Kubernetes 中部署 SkyWalking 以及其依赖服务。 Helm Chart 可以在 GitHub 仓库找到。\n# Install Helm curl -sSLO https://get.helm.sh/helm-v3.0.0-linux-amd64.tar.gz sudo tar xz -C /usr/local/bin --strip-components=1 linux-amd64/helm -f helm-v3.0.0-linux-amd64.tar.gz # Clone SkyWalking Helm Chart git clone https://github.com/apache/skywalking-kubernetes cd skywalking-kubernetes/chart git reset --hard dd749f25913830c47a97430618cefc4167612e75 # Update dependencies helm dep up skywalking # Deploy SkyWalking helm -n istio-system install skywalking skywalking \\ --set oap.storageType=\u0026#39;h2\u0026#39;\\ --set ui.image.tag=8.3.0 \\ --set oap.image.tag=8.3.0-es7 \\ --set oap.replicas=1 \\ --set oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh \\ --set oap.env.JAVA_OPTS=\u0026#39;-Dmode=\u0026#39; \\ --set oap.envoy.als.enabled=true \\ --set elasticsearch.enabled=false 我们在 istio-system 的命名空间内部署 SkyWalking，使 SkyWalking OAP 服务可以使用地址 skywalking-oap.istio-system:11800 访问，在上一步中，我们曾告诉过 ALS 应往此处发放它们的日志。\n我们也在 SkyWalking OAP 中启动 ALS 分析器︰oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh。分析器会对访问日志进行分析，并解析日志中的 IP 地址和 Kubernetes 中的真实服务名称，以建立拓扑。\n为了从 Kubernetes 集群处获取元数据（例如 Pod IP 和服务名称），以识别相应的 IP 地址，我们还会设定 oap.envoy.als.enabled=true，用来申请一个对元数据有访问权的 ClusterRole。\nexport POD_NAME=$(kubectl get pods -A -l \u0026#34;app=skywalking,release=skywalking,component=ui\u0026#34; -o name) echo $POD_NAME kubectl -n istio-system port-forward $POD_NAME 8080:8080 现在到你的浏览器上访问 http://localhost:8080。你应该会看到 SkyWalking 的 Dashboard。 Dashboard 现在应该是空的，但稍后部署应用和生成流量后，它就会被填满。\n部署 Bookinfo 应用 运行︰\nexport ISTIO_VERSION=1.7.1 kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/bookinfo-gateway.yaml kubectl wait --for=condition=Ready pods --all --timeout=1200s minikube tunnel 现在到你的浏览器上进入 http://localhost/productpage。你应该会看到典型的 bookinfo 应用画面。重新整理该页面几次，以生成足够的访问日志。\n完成了！ 这样做，你就成功完成设置了！再查看 SkyWalking 的 WebUI，你应该会看到 bookinfo 应用的拓扑，以及它每一个单独服务的指标数据。\n疑难解答 检查所有 pod 的状态︰kubectl get pods -A。 SkyWalking OAP 的日志︰kubectl -n istio-system logs -f $(kubectl get pod -A -l \u0026quot;app=skywalking,release=skywalking,component=oap\u0026quot; -o name)。 SkyWalking WebUI 的日志︰kubectl -n istio-system logs -f $(kubectl get pod -A -l \u0026quot;app=skywalking,release=skywalking,component=ui\u0026quot; -o name)。 确保 WebUI 右下方的时区设定在 UTC +0。 自定义服务器名称 SkyWalking 社区在 ALS 方案的 8.3.0 版本中，作出了许多改善。你现在可以在映射 IP 地址时，决定如何用 service 和 pod 变量去自定义服务器的名称。例如，将 K8S_SERVICE_NAME_RULE 设置为 ${service.metadata.name}-${pod.metadata.labels.version}，就可以使服务名称带上版本的标签，类似 reviews-v1、reviews-v2 和 reviews- v3，而不再是单个服务 review［2］。\n在 VM 上使用 ALS Kubernetes 很受欢迎，可是 VM 呢？正如我们之前所说，为了替 IP 找到对应的服务，SkyWalking 需要对 Kubernetes 集群有访问权，以获得服务的元数据和 Pod 的 IP。可是在 VM 环境中，我们并没有来源去收集这些元数据。\n在下一篇文章，我们会介绍另外一个 ALS 分析器，它是建立于 Envoy 的元数据交换机制。有了这个分析器，你就可以在 VM 环境中观察服务网格了。万勿错过！\n如果你希望在 ALS 方案或是混合式网格可观察性上获得商业支持，TSB 会是一个好选项。\n额外资源\nKubeCon 2019 的录影视频。 在官方网站上获得更多有关 SkyWalking 的最新消息吧。 如有任何问题或反馈，发送邮件至 learn@tetrate.io。\nApache SkyWalking 创始人吴晟和 SkyWalking 的核心贡献者柯振旭都是 Tetrate 的工程师。 Tetrate 的内容创造者编辑与贡献于本文章。 Tetrate 帮助企业采用开源服务网格工具，包括 Istio、Envoy 和 Apache SkyWalking，让它们轻松管理微服务，在任何架构上运行服务网格，以至现代化他们的应用。\n［1］https://github.com/envoyproxy/envoy/blob/549164c42cae84b59154ca4c36009e408aa10b52/generated_api_shadow/envoy/data/accesslog/v2/accesslog.proto\n［2］https://github.com/apache/skywalking/pull/5722\n","excerpt":"\u003cp\u003e\u003cimg src=\"../../blog/2020-12-03-obs-service-mesh-with-sw-and-als/canyonhorseshoe.jpg\" alt=\"img\"\u003e\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003e如果你正在寻找在 Mixer 方案以外观察服务网格的更优解，本文正符合你的需要。\u003c/strong\u003e\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003e\u003cstrong\u003e\u003ca href=\"https://github.com/apache/skywalking\"\u003eApache Skywalking\u003c/a\u003e\u003c/strong\u003e︰特别为微服务、云原生和容器化 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/observe-service-mesh-with-skywalking-and-envoy-access-log-service/","title":"使用 SkyWalking 和 Envoy 访问日志服务对服务网格进行观察"},{"body":"SkyWalking 8.3.0 is released. Go to downloads page to find release tars.\nProject Test: ElasticSearch version 7.0.0 and 7.9.3 as storage are E2E tested. Test: Bump up testcontainers version to work around the Docker bug on MacOS. Java Agent Support propagate the sending timestamp in MQ plugins to calculate the transfer latency in the async MQ scenarios. Support auto-tag with the fixed values propagated in the correlation context. Make HttpClient 3.x, 4.x, and HttpAsyncClient 3.x plugins to support collecting HTTP parameters. Make the Feign plugin to support Java 14 Make the okhttp3 plugin to support Java 14 Polish tracing context related codes. Add the plugin for async-http-client 2.x Fix NPE in the nutz plugin. Provide Apache Commons DBCP 2.x plugin. Add the plugin for mssql-jtds 1.x. Add the plugin for mssql-jdbc 6.x -\u0026gt; 9.x. Fix the default ignore mechanism isn\u0026rsquo;t accurate enough bug. Add the plugin for spring-kafka 1.3.x. Add the plugin for Apache CXF 3.x. Fix okhttp-3.x and async-http-client-2.x did not overwrite the old trace header. OAP-Backend Add the @SuperDataset annotation for BrowserErrorLog. Add the thread pool to the Kafka fetcher to increase the performance. Add contain and not contain OPS in OAL. Add Envoy ALS analyzer based on metadata exchange. Add listMetrics GraphQL query. Add group name into services of so11y and istio relevant metrics Support keeping collecting the slowly segments in the sampling mechanism. Support choose files to active the meter analyzer. Support nested class definition in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Support sideCar.internalErrorCode in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Improve Kubernetes service registry for ALS analysis. Add health checker for cluster management Support the service auto grouping. Support query service list by the group name. Improve the queryable tags generation. Remove the duplicated tags to reduce the storage payload. Fix the threads of the Kafka fetcher exit if some unexpected exceptions happen. Fix the excessive timeout period set by the kubernetes-client. Fix deadlock problem when using elasticsearch-client-7.0.0. Fix storage-jdbc isExists not set dbname. Fix searchService bug in the InfluxDB storage implementation. Fix CVE in the alarm module, when activating the dynamic configuration feature. Fix CVE in the endpoint grouping, when activating the dynamic configuration feature. Fix CVE in the uninstrumented gateways configs, when activating the dynamic configuration feature. Fix CVE in the Apdex threshold configs, when activating the dynamic configuration feature. Make the codes and doc consistent in sharding server and core server. Fix that chunked string is incorrect while the tag contains colon. Fix the incorrect dynamic configuration key bug of endpoint-name-grouping. Remove unused min date timebucket in jdbc deletehistory logical Fix \u0026ldquo;transaction too large error\u0026rdquo; when use TiDB as storage. Fix \u0026ldquo;index not found\u0026rdquo; in trace query when use ES7 storage. Add otel rules to ui template to observe Istio control plane. Remove istio mixer Support close influxdb batch write model. Check SAN in the ALS (m)TLS process. UI Fix incorrect label in radial chart in topology. Replace node-sass with dart-sass. Replace serviceFilter with serviceGroup Removed \u0026ldquo;Les Miserables\u0026rdquo; from radial chart in topology. Add the Promise dropdown option Documentation Add VNode FAQ doc. Add logic endpoint section in the agent setup doc. Adjust configuration names and system environment names of the sharing server module Tweak Istio metrics collection doc. Add otel receiver. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking 8.3.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eTest: …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skwaylking-apm-8-3-0/","title":"Release Apache SkyWalking APM 8.3.0"},{"body":"Python 作为一门功能强大的编程语言，被广泛的应用于计算机行业之中； 在微服务系统架构盛行的今天，Python 以其丰富的软件生态和灵活的语言特性在服务端编程领域也占有重要的一席之地。 本次分享将阐述 Apache SkyWalking 在微服务架构中要解决的问题，展示如何使用 Apache SkyWalking 来近乎自动化地监控 Python 后端应用服务，并对 Apache SkyWalking 的 Python 语言探针的实现技术进行解读。\nB站视频地址\n","excerpt":"\u003cp\u003ePython 作为一门功能强大的编程语言，被广泛的应用于计算机行业之中； 在微服务系统架构盛行的今天，Python 以其丰富的软件生态和灵活的语言特性在服务端编程领域也占有重要的一席之地。  本次分享 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2020-11-30-pycon/","title":"[视频] PyCon China 2020 - Python 微服务应用性能监控"},{"body":"SkyWalking CLI 0.5.0 is released. Go to downloads page to find release tars.\nFeatures\nUse template files in yaml format instead Refactor metrics command to adopt metrics-v2 protocol Use goroutine to speed up dashboard global command Add metrics list command Bug Fixes\nAdd flags of instance, endpoint and normal for metrics command Fix the problem of unable to query database metrics Chores\nUpdate release guide doc Add screenshots for use cases in README.md Introduce generated codes into codebase ","excerpt":"\u003cp\u003eSkyWalking CLI 0.5.0 is released. Go to \u003ca href=\"https://skywalking.apache.org/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eFeatures\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eUse …\u003c/li\u003e\u003c/ul\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cli-0-5-0/","title":"Release Apache SkyWalking CLI 0.5.0"},{"body":"\nAuthor: Jiapeng Liu. Baidu. skywalking-satellite: The Sidecar Project of Apache SkyWalking Nov. 25th, 2020 A lightweight collector/sidecar which can be deployed close to the target monitored system, to collect metrics, traces, and logs. It also provides advanced features, such as local cache, format transformation, and sampling.\nDesign Thinking Satellite is a 2 level system to collect observability data from other core systems. So, the core element of the design is to guarantee data stability during Pod startup all the way to Pod shutdown avoiding alarm loss. All modules are designed as plugins, and if you have other ideas, you can add them yourself.\nSLO Single gatherer supports \u0026gt; 1000 ops (Based 0.5 Core,50M) At least once delivery.(Optional) Data stability: 99.999%.(Optional) Because they are influenced by the choice of plugins, some items in SLO are optional.\nRole Satellite would be running as a Sidecar. Although Daemonset mode would take up fewer resources, it will cause more troubles to the forwarding of agents. So we also want to use Sidecar mode by reducing the costs. But Daemonset mode would be also supported in the future plan.\nCore Modules The Satellite has 3 core modules which are Gatherer, Processor, and Sender.\nThe Gatherer module is responsible for fetching or receiving data and pushing the data to Queue. The Processor module is responsible for reading data from the queue and processing data by a series of filter chains. The Sender module is responsible for async processing and forwarding the data to the external services in the batch mode. After sending success, Sender would also acknowledge the offset of Queue in Gatherer. Detailed Structure The overall design is shown in detail in the figure below. We will explain the specific components one by one.\nGatherer Concepts The Gatherer has 4 components to support the data collection, which are Input, Collector, Worker, and Queue. There are 2 roles in the Worker, which are Fetcher and Receiver.\nThe Input is an abstraction of the input source, which is usually mapped to a configuration file. The Collector is created by the Source, but many collectors could be created by the same Source. For example, when a log path has been configured as the /var/*.log in an Input, the number of collectors is the same as the file number in this path. The Fetcher and Receiver is the real worker to collect data. The receiver interface is an abstraction, which has multiple implementations, such as gRPC receiver and HTTP receiver.Here are some specific use cases: Trace Receiver is a gRPC server for receiving trace data created by Skywalking agents. Log Receiver is also a gRPC server for receiving log data which is collected by Skywalking agents. (In the future we want Skywalking Agent to support log sending, and RPC-based log sending is more efficient and needs fewer resources than file reading. For example, the way of file reading will bring IO pressure and performance cost under multi-line splicing.) Log Fetcher is like Filebeat, which fits the common log collection scenario. This fetcher will have more responsibility than any other workers because it needs to record the offset and process the multi-line splicing. This feature will be implemented in the future. Prometheus Fetcher supports a new way to fetch Prometheus data and push the data to the upstream. \u0026hellip;\u0026hellip; The Queue is a buffer module to decouple collection and transmission. In the 1st release version, we will use persistent storage to ensure data stability. But the implementation is a plug-in design that can support pure memory queues later. The data flow We use the Trace Receiver as an example to introduce the data flow. Queue MmapQueue We have simplified the design of MmapQueue to reduce the resources cost on the memory and disk.\nConcepts There are 2 core concepts in MmapQueue.\nSegment: Segment is the real data store center, that provides large-space storage and does not reduce read and write performance as much as possible by using mmap. And we will avoid deleting files by reusing them. Meta: The purpose of meta is to find the data that the consumer needs. Segment One MmapQueue has a directory to store the whole data. The Queue directory is made up with many segments and 1 meta file. The number of the segments would be computed by 2 params, which are the max cost of the Queue and the cost of each segment. For example, If the max cost is 512M and each segment cost is 256K, the directory can hold up to 2000 files. Once capacity is exceeded, an coverage policy is adopted that means the 2000th would override the first file.\nEach segment in Queue will be N times the size of the page cache and will be read and written in an appended sequence rather than randomly. These would improve the performance of Queue. For example, each Segment is a 128k file, as shown in the figure below.\nMeta The Meta is a mmap file that only contains 56Bit. There are 5 concepts in the Meta.\nVersion: A version flag. Watermark Offset: Point to the current writing space. ID: SegmentID Offset: The offset in Segment. Writed Offset: Point to the latest refreshed data, that would be overridden by the write offset after period refresh. ID: SegmentID Offset: The offset in Segment. Reading Offset: Point to the current reading space. ID: SegmentID Offset: The offset in Segment. Committed Offset: Point to the latest committed offset , that is equal to the latest acked offset plus one. ID: SegmentID Offset: The offset in Segment. The following diagram illustrates the transformation process.\nThe publisher receives data and wants to write to Queue. The publisher would read Writing Offset to find a space and do plus one. After this, the publisher will write the data to the space. The consumer wants to read the data from Queue. The consumer would read Reading Offset to find the current read offset and do plus one. After this, the consumer will read the data from the space. On period flush, the flusher would override Watermark Offset by using Writing Offset. When the ack operation is triggered, Committed Offset would plus the batch size in the ack batch. When facing crash, Writing Offset and Reading Offset would be overridden by Watermark Offset and Committed Offset. That is because the Reading Offset and Writing Offset cannot guarantee at least once delivery. Mmap Performance Test The test is to verify the efficiency of mmap in low memory cost.\nThe rate of data generation: 7.5K/item 1043 item/s (Based on Aifanfan online pod.) The test structure is based on Bigqueue because of similar structure. Test tool: Go Benchmark Test Command: go test -bench BenchmarkEnqueue -run=none -cpu=1 Result On Mac(15-inch, 2018,16 GB 2400 MHz DDR4, 2.2 GHz Intel Core i7 SSD): BenchmarkEnqueue/ArenaSize-128KB/MessageSize-8KB/MaxMem-384KB 66501 21606 ns/op 68 B/op 1 allocs/op BenchmarkEnqueue/ArenaSize-128KB/MessageSize-8KB/MaxMem-1.25MB 72348 16649 ns/op 67 B/op 1 allocs/op BenchmarkEnqueue/ArenaSize-128KB/MessageSize-16KB/MaxMem-1.25MB 39996 33199 ns/op 103 B/op 1 allocs/op Result On Linux(INTEL Xeon E5-2450 V2 8C 2.5GHZ2,INVENTEC PC3L-10600 16G8,INVENTEC SATA 4T 7.2K*8): BenchmarkEnqueue/ArenaSize-128KB/MessageSize-8KB/MaxMem-384KB 126662\t12070 ns/op\t62 B/op\t1 allocs/op BenchmarkEnqueue/ArenaSize-128KB/MessageSize-8KB/MaxMem-1.25MB 127393\t12097 ns/op\t62 B/op\t1 allocs/op BenchmarkEnqueue/ArenaSize-128KB/MessageSize-16KB/MaxMem-1.25MB 63292\t23806 ns/op\t92 B/op\t1 allocs/op Conclusion: Based on the above tests, mmap is both satisfied at the write speed and at little memory with very low consumption when running as a sidecar. Processor The Processor has 3 core components, which are Consumer, Filter, and Context.\nThe Consumer is created by the downstream Queue. The consumer has its own read offset and committed offset, which is similar to the offset concept of Spark Streaming. Due to the particularity of APM data preprocessing, Context is a unique concept in the Satellite filter chain, which supports storing the intermediate event because the intermediate state event also needs to be sent in sometimes. The Filter is the core data processing part, which is similar to the processor of beats. Due to the context, the upstream/downstream filters would be logically coupling. Sender BatchConverter decouples the Processor and Sender by staging the Buffer structure, providing parallelization. But if BatchBuffer is full, the downstream processors would be blocked. Follower is a real send worker that has a client, such as a gRPC client or Kafka client, and a fallback strategy. Fallback strategy is an interface, we can add more strategies to resolve the abnormal conditions, such as Instability in the network, upgrade the oap cluster. When sent success, Committed Offset in Queue would plus the number of this batch. High Performance The scenario using Satellite is to collect a lot of APM data collection. We guarantee high performance by the following ways.\nShorten transmission path, that means only join 2 components,which are Queue and Processor, between receiving and forwarding. High Performance Queue. MmapQueue provides a big, fast and persistent queue based on memory mapped file and ring structure. Processor maintains a linear design, that could be functional processed in one go-routine to avoid too much goroutines switching. Stability Stability is a core point in Satellite. Stability can be considered in many ways, such as stable resources cost, stable running and crash recovery.\nStable resource cost In terms of resource cost, Memory and CPU should be a concern.\nIn the aspect of the CPU, we keep a sequence structure to avoid a large number of retries occurring when facing network congestion. And Satellite avoids keep pulling when the Queue is empty based on the offset design of Queue.\nIn the aspect of the Memory, we have guaranteed only one data caching in Satellite, that is Queue. For the queue structure, we also keep the size fixed based on the ring structure to maintain stable Memory cost. Also, MmapQueue is designed for minimizing memory consumption and providing persistence while keeping speed as fast as possible. Maybe supports some strategy to dynamically control the size of MmapQueue to process more extreme conditions in the future.\nStable running There are many cases of network congestion, such as the network problem on the host node, OAP cluster is under upgrating, and Kafka cluster is unstable. When facing the above cases, Follower would process fallback strategy and block the downstream processes. Once the failure strategy is finished, such that send success or give up this batch, the Follower would process the next batch.\nCrash Recovery The crash recovery only works when the user selects MmapQueue in Gatherer because of persistent file system design. When facing a crash, Reading Offset would be overridden by Committed Offset that ensure the at least once delivery. And Writed Offset would override Writing Offset that ensures the consumer always works properly and avoid encountering uncrossable defective data blocks.\nBuffer pool The Queue is to store fixed structure objects, object buffer pool would be efficient to reuse memory to avoid GC.\nackChan batch convertor Some metrics In Satellite, we should also collect its own monitoring metrics. The following metrics are necessary for Satellite.\ncpu memory go routine number gatherer_writing_offset gatherer_watermark_offset processor_reading_count sender_committed_offset sender_abandoned_count sender_retry_count Input and Output We will reuse this diagram to explain the input and output.\nInput Because the push-pull mode is both supported, Queue is a core component. Queue is designed to be a ring-shaped fixed capacity, that means the oldest data would be overridden by the latest data. If users find data loss, users should raise the ceiling of memory Queue. MmapQueue generally doesn\u0026rsquo;t face this problem unless the Sender transport is congested. Ouput If the BatchBuffer is full, the processor would be blocked. If the Channel is full, the downstream components would be blocked, such as BatchConvertor and Processor. When SenderWorker sends failure, the batch data would do a failure strategy that would block pulling data from the Channel. The strategy is a part of Sender,the operation mode is synchronous. Once the failure strategy is finished, such that send success or give up this batch, the Sendworker would keep pulling data from the Channel. Questions How to avoid keep pulling when the Queue is empty? If Watermark Offset is less than or equal to Reading Offset, a signal would be sent to the consumer to avoid keep pulling.\nWhy reusing files in Queue? The unified model is a ring in Queue, that limits fixed resources cost in memory or disk.In Mmap Queue, reusing files turns the delete operations into an overwrite operations, effectively reducing the creation and deletion behavior in files.\nWhat are the strategies for file creation and deletion in MmapQueue? As Satellite running, the number of the files in MmapQueue would keep growing until up to the maximum capacity. After this, the old files will be overridden by the new data to avoid file deletion. When the Pod died, all resources were recycled.\n","excerpt":"\u003cp\u003e\u003cimg src=\"Satellite.png\" alt=\"\"\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAuthor: \u003ca href=\"https://github.com/evanljp\"\u003eJiapeng Liu\u003c/a\u003e. Baidu.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/apache/skywalking-satellite\"\u003eskywalking-satellite\u003c/a\u003e: The Sidecar Project of Apache SkyWalking\u003c/li\u003e\n\u003cli\u003eNov. …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/blog/2020-11-25-skywalking-satellite-0.1.0-design/","title":"The first design of Satellite 0.1.0"},{"body":"SkyWalking Python 0.4.0 is released. Go to downloads page to find release tars.\nFeature: Support Kafka reporter protocol (#74) BugFix: Move generated packages into skywalking namespace to avoid conflicts (#72) BugFix: Agent cannot reconnect after server is down (#79) Test: Mitigate unsafe yaml loading (#76) ","excerpt":"\u003cp\u003eSkyWalking Python 0.4.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eFeature: Support …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-python-0-4-0/","title":"Release Apache SkyWalking Python 0.4.0"},{"body":"活动介绍 Apache SkyWalking 2020 开发者线下活动，社区创始人，PMC成员和Committer会亲临现场，和大家交流和分享项目中的使用经验。 以及邀请Apache Local Community 北京的成员一起分享Apache文化和Apache之道。\n日程安排 开场演讲 09：30-09：50 SkyWalking\u0026rsquo;s 2019-2020 and beyond\n吴晟，Tetrate.io创始工程师，Apache SkyWalking创始人\nB站视频地址\n上午 09：55-10：30 贝壳全链路跟踪实践\n赵禹光，赵禹光，贝壳找房监控技术负责人，Apache SkyWalking PMC成员\n10：35-11：15 SkyWalking在百度爱番番部门实践\n刘嘉鹏，百度，SkyWalking contributor\n11：15-11：55 非计算机背景的同学如何贡献开源\n缘于一位本科在读的社会学系的同学的问题，这让我反思我们开源community的定位和Open的程度，于是，适兕从生产、分发、消费的软件供应的角度，根据涉及到的角色，然后再反观现代大学教育体系的专业，进一步对一个开源项目和community需要的专业背景多样性进行一个阐述和探究。并以ALC Beijing为例进行一个事例性的说明。\n适兕，开源布道师，ALC Beijing member，开源之道主创，开源社教育组成员。\nB站视频地址\n下午 13：30-14：10 如何从 Apache SkyWalking 社区学习 Apache Way\n温铭，支流科技联合创始人＆CEO，Apache APISIX 项目 VP， Apache SkyWalking Committer\n14：10-14：50 Apache SkyWalking 在小米公司的应用\n宋振东，小米公司小米信息技术部 skywalking 研发负责人\n14：50-15：30 Istio全生命周期监控\n高洪涛，Tetrate.io创始工程师，Apache SkyWalking PMC成员\n15：30-15：45 茶歇\n15：45-16：25 针对HikariCP数据库连接池的监控\n张鑫 Apache SkyWalking PMC 成员\n16：25-17：00 SkyWalking 与 Nginx 的优化实践\n王院生 深圳支流科技创始人兼 CTO，Apache APISIX 创始人 \u0026amp; PMC成员\nB站视频地址\n","excerpt":"\u003ch1 id=\"活动介绍\"\u003e活动介绍\u003c/h1\u003e\n\u003cp\u003eApache SkyWalking 2020 开发者线下活动，社区创始人，PMC成员和Committer会亲临现场，和大家交流和分享项目中的使用经验。\n以及邀请Apache Local …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2020-11-23-devcon/","title":"[视频] SkyWalking DevCon 2020"},{"body":"The APM system provides the tracing or metrics for distributed systems or microservice architectures. Back to APM themselves, they always need backend storage to store the necessary massive data. What are the features required for backend storage? Simple, fewer dependencies, widely used query language, and the efficiency could be into your consideration. Based on that, traditional SQL databases (like MySQL) or NoSQL databases would be better choices. However, this topic will present another backend storage solution for the APM system viewing from NewSQL. Taking Apache Skywalking for instance, this talking will share how to make use of Apache ShardingSphere, a distributed database middleware ecosystem to extend the APM system\u0026rsquo;s storage capability.\nAs a senior DBA worked at JD.com, the responsibility is to develop the distributed database and middleware, and the automated management platform for database clusters. As a PMC of Apache ShardingSphere, I am willing to contribute to the OS community and explore the area of distributed databases and NewSQL.\n","excerpt":"\u003cp\u003eThe APM system provides the tracing or metrics for distributed systems or microservice …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2020-11-21-apachecon-obs-shardingsphere/","title":"[Video] Another backend storage solution for the APM system"},{"body":"Apache APISIX is a cloud-native microservices API gateway, delivering the ultimate performance, security, open-source and scalable platform for all your APIs and microservices. Apache SkyWalking: an APM(application performance monitor) system, especially designed for microservices, cloud-native and container-based (Docker, Kubernetes, Mesos) architectures. Through the powerful plug-in mechanism of Apache APISIX, Apache Skywalking is quickly supported, so that we can see the complete life cycle of requests from the edge to the internal service. Monitor and manage each request in a visual way, and improve the observability of the service.\n","excerpt":"\u003cp\u003eApache APISIX is a cloud-native microservices API gateway, delivering the ultimate performance, …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2020-11-21-apachecon-obs-apisix/","title":"[Video] Improve Apache APISIX observability with Apache SkyWalking"},{"body":"Today\u0026rsquo;s monitoring solutions are geared towards operational tasks, displaying behavior as time-series graphs inside dashboards and other abstractions. These abstractions are immensely useful but are largely designed for software operators, whose responsibilities require them to think in systems, rather than the underlying source code. This is problematic given that an ongoing trend of software development is the blurring boundaries between building and operating software. This trend makes it increasingly necessary for programming environments to not just support development-centric activities, but operation-centric activities as well. Such is the goal of the feedback-driven development approach. By combining IDE and APM technology, software developers can intuitively explore multiple dimensions of their software simultaneously with continuous feedback about their software from inception to production.\nBrandon Fergerson is an open-source software developer who does not regard himself as a specialist in the field of programming, but rather as someone who is a devoted admirer. He discovered the beauty of programming at a young age and views programming as an art and those who do it well to be artists. He has an affinity towards getting meta and combining that with admiration of programming, has found source code analysis to be exceptionally interesting. Lately, his primary focus involves researching and building AI-based pair programming technology.\n","excerpt":"\u003cp\u003eToday\u0026rsquo;s monitoring solutions are geared towards operational tasks, displaying behavior as …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2020-11-21-apachecon-obs-sourcemarker/","title":"[Video] SourceMarker - Continuous Feedback for Developers"},{"body":"Over the past few years, and coupled with the growing adoption of microservices, distributed tracing has emerged as one of the most commonly used monitoring and troubleshooting methodologies. New tracing tools are increasingly being introduced, driving adoption even further. One of these tools is Apache SkyWalking, a popular open-source tracing, and APM platform. This talk explores the history of the SkyWalking storage module, shows the evolution of distributed tracing storage layers, from the traditional relational database to document-based search engine. I hope that this talk contributes to the understanding of history and also that it helps to clarify the different types of storage that are available to organizations today.\nHongtao Gao is the engineer of tetrate.io and the former Huawei Cloud expert. One of PMC members of Apache SkyWalking and participates in some popular open-source projects such as Apache ShardingSphere and Elastic-Job. He has an in-depth understanding of distributed databases, container scheduling, microservices, ServicMesh, and other technologies.\n","excerpt":"\u003cp\u003eOver the past few years, and coupled with the growing adoption of microservices, distributed tracing …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2020-11-21-apachecon-obs-storage/","title":"[Video] The history of distributed tracing storage"},{"body":" 作者: 赵禹光 原文链接: 亲临百人盛况的Apache SkyWalking 2020 DevCon，看见了什么？ 2020 年 10 月 29 日 活动现场 2020年11月14日Apache SkyWalking 2020 DevCon由贝壳找房和tetrate赞助，Apache SkyWalking、云原生、Apache APISIX、Apache Pulsar 和 ALC Beijing 五大社区合作，在贝壳找房一年级会议室盛大举行，本次活动主要面对Apache SkyWalking的使用者、开发者和潜在用户。线上线下共有230多人报名。经统计，实际参加活动人数超过130人，近60%的人愿意抽出自己的休息时间，来交流学习Apache SkyWalking和开源文化。不难看见，在可预见的未来，中国的开源项目很快将进入下一个维度，那必定是更广的社区人员参与，更高技术知识体现，更强的线上稳定性和及时修复能力。\n活动历程： 09：30-09：50 SkyWalking\u0026rsquo;s 2019-2020 and beyond 吴晟老师本次分享：回顾2020年度SkyWalking发布的重要的新特性，出版的《Apache SkyWalking实战》图书，社区的进展，开源爱好者如何参与SkyWalking建设，和已知社区在主导的SkyWalking2021年孵化中的新特性。\n09：55-10：30 贝壳全链路跟踪实践 赵禹光老师（作者）本次分享：回顾了贝壳找房2018年至今，贝壳找房的全链路跟踪项目与SkyWalking的渊源，分享了SkyWalking在实践中遇到的问题，和解决方案。以及SkyWalking近10%的Committer都曾经或正在贝壳人店平台签中研发部，工作过的趣事。\n10：35-11：15 刘嘉鹏老师分享 SkyWalking在百度爱番番部门实践 刘嘉鹏老师本次分享：回顾了百度爱番番部门在使用SkyWalking的发展历程\u0026amp;现状，CRM SAAS产品在近1年使用SkyWalking实践经验，以及如何参与SkyWalking的贡献，并成为的Apache Committer。\n11：15-11：55 适兕老师分享 非计算机背景的同学如何贡献开源 适兕是国内很有名的开源布道师，本次分享从生产、分发、消费的软件供应的角度，根据涉及到的角色，然后再反观现代大学教育体系的专业，进一步对一个开源项目和community需要的专业背景多样性进行一个阐述和探究。并以ALC Beijing为例进行一个事例性的说明，非计算机背景的同学如何贡献开源。\n13：30-14：10 如何从 Apache SkyWalking 社区学习 Apache Way 14：10-14：50 Apache SkyWalking 在小米公司的应用 宋振东老师是小米信息技术部分布式链路追踪系统研发负责人，分别以小米公司，业务开发、架构师、SRE、Leader和QA等多个视角，回顾了SkyWalking在小米公司的应用实践。从APM的产品选型到实际落地，对其他公司准备使用SkyWalking落地，非常有借鉴意义。\n14：50-15：30 Istio全生命周期监控 高洪涛老师本次分享了SkyWalking和可观测云原生等非常前沿的知识布道，其中有，云原生在Logging、Metrics和Tracing的相关知识，Istio，K8S等方面的实践。对一些公司在前沿技术的落地，非常有借鉴意义。\n15：45-16：25 针对HikariCP数据库连接池的监控 张鑫老师本次分享了，以一个SkyWalking无法Tracing的实际线上故障的故事出发，讲述如何定位，和补充SkyWalking插件的不足，并将最后的实践贡献到社区。对大家参与开源很有帮助。\n16：25-17：00 SkyWalking 与 Nginx 的优化实践 王院生老师本次分享SkyWalking社区和APISIX社区合作，在Nginx插件的实践过程，对社区之间的如何开展合作，非常有借鉴意义，院生老师的工作\u0026amp;开源态度，很好的诠释Geek精神，也是我们互联网从业者需要学习恪守的。\nApache SkyWalking 2020 DevCon 讲师PPT Apache SkyWalking 2020 DevCon 讲师 PPT\nSkyWalking 后续发展计划 正如吴晟老师所说：No plan, open to the community，Apache SkyWalking是没有RoadMap。社区的后续发展，依赖于每个人在社区的贡献。与其期待，不如大胆设想，将自己的设计按照Apache Way贡献到SkyWalking，你就是下一个Apache SkyWalking Commiter，加入Member of SkyWalking大家庭，让社区因为你，而更加有活力。\n","excerpt":"\u003cul\u003e\n\u003cli\u003e作者: 赵禹光\u003c/li\u003e\n\u003cli\u003e原文链接: \u003ca href=\"https://mp.weixin.qq.com/s/5wYCYiP8oKs7V6BR1lyKxg\"\u003e亲临百人盛况的Apache SkyWalking 2020 DevCon，看见了什么？\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e2020 年 10 月 29 日\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"活动现场\"\u003e活动现场\u003c/h3\u003e\n\u003cp\u003e2020年11月14日Apache …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2020-11-21-what-do-we-see-at-the-apache-skywalking-2020-devcon-event/","title":"亲临百人盛况的Apache SkyWalking 2020 DevCon，看见了什么？"},{"body":"Sheng Wu is a founding engineer at tetrate.io, leads the observability for service mesh and hybrid cloud. A searcher, evangelist, and developer in the observability, distributed tracing, and APM. He is a member of the Apache Software Foundation. Love open source software and culture. Created the Apache SkyWalking project and being its VP and PMC member. Co-founder and PMC member of Apache ShardingSphere. Also as a PMC member of Apache Incubator and APISIX. He is awarded as Microsoft MVP, Alibaba Cloud MVP, Tencent Cloud TVP.\nIn the Apache FY2020 report, China is on the top of the download statistics. More China initiated projects joined the incubator, and some of them graduated as the Apache TLP. Sheng joined the Apache community since 2017, in the past 3 years, he witnessed the growth of the open-source culture and Apache way in China. Many developers have joined the ASF as new contributors, committers, foundation members. Chinese enterprises and companies paid more attention to open source contributions, rather than simply using the project like before. In the keynote, he would share the progress about China embracing the Apache culture, and willing of enhancing the whole Apache community.\n","excerpt":"\u003cp\u003eSheng Wu is a founding engineer at tetrate.io, leads the observability for service mesh and hybrid …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2020-11-21-apachecon-keynote/","title":"[Video] Apache grows in China"},{"body":"SkyWalking Client JS 0.2.0 is released. Go to downloads page to find release tars.\nBug Fixes Fixed a bug in sslTime calculate. Fixed a bug in server response status judgment. ","excerpt":"\u003cp\u003eSkyWalking Client JS 0.2.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eBug Fixes\n\u003cul\u003e\n\u003cli\u003eFixed …\u003c/li\u003e\u003c/ul\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-client-js-0-2-0/","title":"Release Apache SkyWalking Client JS 0.2.0"},{"body":"SkyWalking Cloud on Kubernetes 0.1.0 is released. Go to downloads page to find release tars.\nAdd OAPServer CRDs and controller. ","excerpt":"\u003cp\u003eSkyWalking Cloud on Kubernetes 0.1.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAdd …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cloud-on-kubernetes-0.1.0/","title":"Release Apache SkyWalking Cloud on Kubernetes 0.1.0"},{"body":"Based on his continuous contributions, Jiapeng Liu (a.k.a evanljp) has been voted as a new committer.\n","excerpt":"\u003cp\u003eBased on his continuous contributions, Jiapeng Liu (a.k.a \u003ca href=\"https://github.com/evanljp\"\u003eevanljp\u003c/a\u003e) has been voted as a new …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-jiapeng-liu-as-new-committer/","title":"Welcome Jiapeng Liu as new committer"},{"body":"SkyWalking Kubernetes Helm Chart 4.0.0 is released. Go to downloads page to find release tars.\nAllow overriding configurations files under /skywalking/config. Unify the usages of different SkyWalking versions. Add Values for init container in case of using private regestry. Add services, endpoints resources in ClusterRole. ","excerpt":"\u003cp\u003eSkyWalking Kubernetes Helm Chart 4.0.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars. …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-kubernetes-helm-chart-4.0.0/","title":"Release Apache SkyWalking Kubernetes Helm Chart 4.0.0"},{"body":"SkyWalking Client JS 0.1.0 is released. Go to downloads page to find release tars.\nSupport Browser Side Monitoring. Require SkyWalking APM 8.2+. ","excerpt":"\u003cp\u003eSkyWalking Client JS 0.1.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Browser …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-client-js-0-1-0/","title":"Release Apache SkyWalking Client JS 0.1.0"},{"body":"\nAuthor: Zhenxu Ke, Sheng Wu, Hongtao Gao, and Tevah Platt. tetrate.io Original link, Tetrate.io blog Oct. 29th, 2020 Apache SkyWalking, the observability platform, and open-source application performance monitor (APM) project, today announced the general availability of its 8.2 release. The release extends Apache SkyWalking’s functionalities and monitoring boundary to the browser side.\nBackground SkyWalking is an observability platform and APM tool that works with or without a service mesh, providing automatic instrumentation for microservices, cloud-native and container-based applications. The top-level Apache project is supported by a global community and is used by Alibaba, Huawei, Tencent, Baidu, ByteDance, and scores of others.\nBrowser side monitoring APM helps SRE and Engineering teams to diagnose system failures, or optimize the systems before they become intolerably slow. But is it enough to always make the users happy?\nIn 8.2.0, SkyWalking extends its monitoring boundary to the browser side, e.g., Chrome, or the network between Chrome and the backend service, or the codes running in the browser. With this, not only can we monitor the backend services and requests sent by the browser as usual, but also the front end rendering speed, error logs, etc., which are the most efficient metrics for capturing the experiences of our end users. (This does not currently extend to IoT devices, but this feature moves SkyWalking a step in that direction).\nWhat\u0026rsquo;s more, SkyWalking browser monitoring also provides data about how the users use products, such as PV(page views), UV(unique visitors), top N PV(page views), etc., which can give a product team clues for optimizing their products.\nQuery traces by tags In SkyWalking\u0026rsquo;s Span data model, there are many important fields that are already indexed and can be queried by the users, but for the sake of performance, querying by Span tags was not supported until now. In SkyWalking 8.2.0, we allow users to query traces by specified tags, which is extremely useful. For example, SRE engineers running tests on the product environment can tag the synthetic traffic and query by this tag later.\nMeter Analysis Language In 8.2.0, the meter system provides a functional analysis language called MAL(Meter Analysis Language) that allows users to analyze and aggregate meter data in the OAP streaming system. The result of an expression can be ingested by either the agent analyzer or OpenTelemetry/Prometheus analyzer.\nComposite Alert Rules Alerting is a good way to discover system failures in time. A common problem is that we configure too many triggers just to avoid missing any possible issue. Nobody likes to be woken up by alert messages at midnight, only to find out that the trigger is too sensitive. These kinds of alerts become noisy and don\u0026rsquo;t help at all.\nIn 8.2.0, users can now configure composite alert rules, where composite rules take multiple metrics dimensions into account. With composite alert rules, we can leverage as many metrics as needed to more accurately determine whether there’s a real problem or just an occasional glitch.\nCommon scenarios like successful rate \u0026lt; 90% but there are only 1~2 requests can now be resolved by a composite rule, such as traffic(calls per minute) \u0026gt; n \u0026amp;\u0026amp; successful rate \u0026lt; m%.\nOther Notable Enhancements The agent toolkit exposes some APIs for users to send customizable metrics. The agent exclude_plugins allows you to exclude some plugins; mount enables you to load a new set of plugins. More than 10 new plugins have been contributed to the agent. The alert system natively supports sending alert messages to Slack, WeChat, DingTalk. Additional Resources Read more about the SkyWalking 8.2 release highlights. Get more SkyWalking updates on Twitter. ","excerpt":"\u003cp\u003e\u003cimg src=\"0081Kckwly1gkl5m6kv3uj31lb0u0jum.jpg\" alt=\"\"\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAuthor: Zhenxu Ke, Sheng Wu, Hongtao Gao, and Tevah Platt. tetrate.io\u003c/li\u003e\n\u003cli\u003eOriginal link, \u003ca href=\"https://tetrate.io/blog/whats-new-with-apache-skywalking-8-2-browser-monitoring-and-more/\"\u003eTetrate.io …\u003c/a\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/blog/2020-10-29-skywalking8-2-release/","title":"Features in SkyWalking 8.2: Browser Side Monitoring; Query Traces by Tags; Meter Analysis Language"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/release-blog/","title":"Release Blog"},{"body":"\n作者: 柯振旭, 吴晟, 高洪涛, Tevah Platt. tetrate.io 原文链接: What\u0026rsquo;s new with Apache SkyWalking 8.2? Browser monitoring and more 2020 年 10 月 29 日 Apache SkyWalking，一个可观测性平台，也是一个开源的应用性能监视器（APM）项目，今日宣布 8.2 发行版全面可用。该发行版拓展了核心功能，并将其监控边界拓展到浏览器端。\n背景 SkyWalking 是一个观测平台和 APM 工具。它可以选择性的与 Service Mesh 协同工作，为微服务、云原生和基于容器的应用提供自动的指标。该项目是全球社区支持的 Apache 顶级项目，阿里巴巴、华为、腾讯、百度、字节跳动等许多公司都在使用。\n浏览器端监控 APM 可以帮助 SRE 和工程团队诊断系统故障，也能在系统异常缓慢之前优化它。但它是否足以让用户总是满意呢？\n在 8.2.0 版本中， SkyWalking 将它的监控边界拓展到了浏览器端，比如 Chrome ，或者 Chrome 和后端服务之间的网络。这样，我们不仅可以像以前一样监控浏览器发送给后端服务的与请求，还能看到前端的渲染速度、错误日志等信息——这些信息是获取最终用户体验的最有效指标。（目前此功能尚未拓展到物联网设备中，但这项功能使得 SkyWalking 向着这个方向前进了一步）\n此外，SkyWalking浏览器监视也提供以下数据: PV（page views，页面浏览量）， UV（unique visitors，独立访客数），浏览量前 N 的页面（Top N Page Views）等。这些数据可以为产品队伍优化他们的产品提供线索。\n按标签 (tag) 查询链路数据 在 SkyWalking 的 Span 数据模型中，已经有了许多被索引并可供用户查询的重要字段。但出于性能考虑，使用 Span 标签查询链路数据的功能直到现在才正式提供。在 SkyWalking 8.2.0 中，我们允许用户查询被特定标签标记的链路，这非常有用。SRE 工程师可以在生产环境中运行测试，将其打上仿真流量的标签，并稍后通过该标签查找它。\n指标分析语言 在 8.2.0 中，仪表系统提供了一项名为MAL（Meter Analysis Language，指标分析语言）的强大分析语言。该语言允许用户在 OAP 流系统中分析并聚合（aggregate）指标数据。 表达式的结果可以被 Agent 分析器或 OpenTelemetry/Prometheus 分析器获取。\n复合警报规则 警报是及时发现系统失效的有效方式。一个常见的问题是，为了避免错过任何可能的问题，我们通常会配置过多的触发器（triggers）。没有人喜欢半夜被警报叫醒，结果只是因为触发系统太敏感。这种警报很嘈杂并毫无帮助。\n在 8.2.0 版本中，用户选择可以配置考虑了多个度量维度的复合警报规则。使用复合报警规则，我们可以根据需要添加尽可能多的指标来更精确地判断是否存在真正的问题，或者只是一个偶发的小问题。\n一些常见的情况，如 成功率 \u0026lt; 90% 但只有 1~2 个请求，现在可以通过复合规则解决，如流量(即每分钟调用数) \u0026gt; n \u0026amp;\u0026amp; 成功率 \u0026lt; m%。\n其它值得注意的功能增强 agent-toolkit SDK 公开了某些 API，供用户发送自定义指标。 Agent exclude_plgins 配置允许您排除某些插件（plugins）; mount 配置使您能够加载一套新的插件。 社区贡献了超过 10 个新 Agent 插件。 报警系统原生支持发送消息到 Slack，企业微信，钉钉。 附加资源 阅读更多关于SkyWalkng 8.2 发行版重点.\n在推特上获取更多关于 SkyWalking 的更新。\nApache SkyWalking DevCon 报名信息 Apache SkyWalking DevCon 2020 开始报名了。 2020 年 11 月 14 日，欢迎大家来线下参加活动和交流, 或者报名观看线上直播。\n","excerpt":"\u003cp\u003e\u003cimg src=\"0081Kckwly1gkl5gnaa2ij31lb0u0jum.jpg\" alt=\"\"\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e作者: 柯振旭, 吴晟, 高洪涛, Tevah Platt. tetrate.io\u003c/li\u003e\n\u003cli\u003e原文链接: \u003ca href=\"https://www.tetrate.io/blog/whats-new-with-apache-skywalking-8-2-browser-monitoring-and-more/\"\u003eWhat\u0026rsquo;s new with Apache SkyWalking 8.2? …\u003c/a\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/zh/2020-10-29-skywalking8-2-release/","title":"SkyWalking 8.2.0 中的新特性: 浏览器端监控; 使用标签查询; 指标分析语言"},{"body":"SkyWalking 8.2.0 is released. Go to downloads page to find release tars.\nProject Support Browser monitoring. Add e2e test for ALS solution of service mesh observability. Support compiling(include testing) in JDK11. Support build a single module. Java Agent Support metrics plugin. Support slf4j logs of gRPC and Kafka(when agent uses them) into the agent log files. Add PROPERTIES_REPORT_PERIOD_FACTOR config to avoid the properties of instance cleared. Limit the size of traced SQL to avoid OOM. Support mount command to load a new set of plugins. Add plugin selector mechanism. Enhance the witness classes for MongoDB plugin. Enhance the parameter truncate mechanism of SQL plugins. Enhance the SpringMVC plugin in the reactive APIs. Enhance the SpringMVC plugin to collect HTTP headers as the span tags. Enhance the Kafka plugin, about @KafkaPollAndInvoke Enhance the configuration initialization core. Plugin could have its own plugins. Enhance Feign plugin to collect parameters. Enhance Dubbo plugin to collect parameters. Provide Thrift plugin. Provide XXL-job plugin. Provide MongoDB 4.x plugin. Provide Kafka client 2.1+ plugin. Provide WebFlux-WebClient plugin. Provide ignore-exception plugin. Provide quartz scheduler plugin. Provide ElasticJob 2.x plugin. Provide Spring @Scheduled plugin. Provide Spring-Kafka plugin. Provide HBase client plugin. Provide JSON log format. Move Spring WebFlux plugin to the optional plugin. Fix inconsistent logic bug in PrefixMatch Fix duplicate exit spans in Feign LoadBalancer mechanism. Fix the target service blocked by the Kafka reporter. Fix configurations of Kafka report don\u0026rsquo;t work. Fix rest template concurrent conflict. Fix NPE in the ActiveMQ plugin. Fix conflict between Kafka reporter and sampling plugin. Fix NPE in the log formatter. Fix span layer missing in certain cases, in the Kafka plugin. Fix error format of time in serviceTraffic update. Upgrade bytebuddy to 1.10.14 OAP-Backend Support Nacos authentication. Support labeled meter in the meter receiver. Separate UI template into multiple files. Provide support for Envoy tracing. Envoy tracer depends on the Envoy community. Support query trace by tags. Support composite alarm rules. Support alarm messages to DingTalk. Support alarm messages to WeChat. Support alarm messages to Slack. Support SSL for Prometheus fetcher and self telemetry. Support labeled histogram in the prometheus format. Support the status of segment based on entry span or first span only. Support the error segment in the sampling mechanism. Support SSL certs of gRPC server. Support labeled metrics in the alarm rule setting. Support to query all labeled data, if no explicit label in the query condition. Add TLS parameters in the mesh analysis. Add health check for InfluxDB storage. Add super dataset concept for the traces/logs. Add separate replicas configuration for super dataset. Add IN operator in the OAL. Add != operator in the OAL. Add like operator in the OAL. Add latest function in the prometheus analysis. Add more configurations in the gRPC server. Optimize the trace query performance. Optimize the CPU usage rate calculation, at least to be 1. Optimize the length of slow SQL column in the MySQL storage. Optimize the topology query, use client side component name when no server side mapping. Add component IDs for Python component. Add component ID range for C++. Fix Slack notification setting NPE. Fix some module missing check of the module manager core. Fix authentication doesn\u0026rsquo;t work in sharing server. Fix metrics batch persistent size bug. Fix trace sampling bug. Fix CLR receiver bug. Fix end time bug in the query process. Fix Exporter INCREMENT mode is not working. Fix an error when executing startup.bat when the log directory exists Add syncBulkActions configuration to set up the batch size of the metrics persistent. Meter Analysis Language. UI Add browser dashboard. Add browser log query page. Support query trace by tags. Fix JVM configuration. Fix CLR configuration. Document Add the document about SW_NO_UPSTREAM_REAL_ADDRESS. Update ALS setup document. Add Customization Config section for plugin development. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking 8.2.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Browser …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-8-2-0/","title":"Release Apache SkyWalking APM 8.2.0"},{"body":"高洪涛 美国ServiceMesh服务商tetrate创始工程师。原华为软件开发云技术专家。目前为Apache SkyWalking核心贡献者，参与该开源项目在软件开发云的商业化进程。曾任职当当网系统架构师，开源达人，曾参与Apache ShardingSphere，Elastic-Job等知名开源项目。对分布式数据库，容器调度，微服务，ServicMesh等技术有深入的了解。\n议题简介 定制化Operator模式在面向Kubernetes的云化平台建构中变得越来越流行。Apache SkyWalking社区已经开始尝试使用Operator模式去构建基于Kubernetes平台的PaaS云组件。本次分享给将会给听众带来该项目的初衷，实现与未来演进等相关内容。分享的内容包含：\n项目动机与设计理念 核心功能展示，包含SkyWalking核心组件的发布，更新与维护。 观测ServiceMesh，包含于Istio的自动集成。 目前的工作进展和对未来的规划。 B站视频地址\n","excerpt":"\u003ch3 id=\"高洪涛\"\u003e高洪涛\u003c/h3\u003e\n\u003cp\u003e美国ServiceMesh服务商tetrate创始工程师。原华为软件开发云技术专家。目前为Apache SkyWalking核心贡献者，参与该开源项目在软件开发云的商业化进程。曾任职当当网系统 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2020-10-25-coscon20-swck/","title":"[视频] Apache SkyWalking Cloud on Kubernetes"},{"body":"SkyWalking LUA Nginx 0.3.0 is released. Go to downloads page to find release tars.\nLoad the base64 module in utils, different ENV use different library. Add prefix skywalking, avoid conflicts with other lua libraries. Chore: only expose the method of setting random seed, it is optional. Coc: use correct code block type. CI: add upstream_status to tag http.status Add http.status ","excerpt":"\u003cp\u003eSkyWalking LUA Nginx 0.3.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eLoad the \u003ccode\u003ebase64 …\u003c/code\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-lua-nginx-0.3.0/","title":"Release Apache SkyWalking LUA Nginx 0.3.0"},{"body":"SkyWalking CLI 0.4.0 is released. Go to downloads page to find release tars.\nFeatures Add dashboard global command with auto-refresh Add dashboard global-metrics command Add traces search Refactor metrics thermodynamic command to adopt the new query protocol Bug Fixes Fix wrong golang standard time ","excerpt":"\u003cp\u003eSkyWalking CLI 0.4.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eFeatures\n\u003cul\u003e\n\u003cli\u003eAdd …\u003c/li\u003e\u003c/ul\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cli-0-4-0/","title":"Release Apache SkyWalking CLI 0.4.0"},{"body":"Huaxi Jiang (江华禧) (a.k.a. fgksgf) mainly focuses on the SkyWalking CLI project, he had participated in the \u0026ldquo;Open Source Promotion Plan - Summer 2020\u0026rdquo; and completed the project smoothly, and won the award \u0026ldquo;Most Potential Students\u0026rdquo; that shows his great willingness to continuously contribute to our community.\nUp to date, he has submitted 26 PRs in the CLI repository, 3 PRs in the main repo, all in total include ~4000 LOC.\nAt Sep. 28th, 2020, the project management committee (PMC) passed the proposal of promoting him as a new committer. He has accepted the invitation at the same day.\nWelcome to join the committer team, Huaxi!\n","excerpt":"\u003cp\u003eHuaxi Jiang (江华禧) (a.k.a. \u003ca href=\"https://github.com/fgksgf\"\u003efgksgf\u003c/a\u003e) mainly focuses on the SkyWalking CLI project, he had participated …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-huaxi-jiang-as-new-committer/","title":"Welcome Huaxi Jiang (江华禧) as new committer"},{"body":"SkyWalking Python 0.3.0 is released. Go to downloads page to find release tars.\nNew plugins\nUrllib3 Plugin (#69) Elasticsearch Plugin (#64) PyMongo Plugin (#60) Rabbitmq Plugin (#53) Make plugin compatible with Django (#52) API\nAdd process propagation (#67) Add tags to decorators (#65) Add Check version of packages when install plugins (#63) Add thread propagation (#62) Add trace ignore (#59) Support snapshot context (#56) Support correlation context (#55) Chores and tests\nTest: run multiple versions of supported libraries (#66) Chore: add pull request template for plugin (#61) Chore: add dev doc and reorganize the structure (#58) Test: update test health check (#57) Chore: add make goal to package release tar ball (#54) ","excerpt":"\u003cp\u003eSkyWalking Python 0.3.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eNew plugins …\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-python-0-3-0/","title":"Release Apache SkyWalking Python 0.3.0"},{"body":"吴晟 吴晟，Apache 基金会会员，Apache SkyWalking 创始人、项目 VP 和 PMC 成员，Apache 孵化器 PMC 成员，Apache ShardingSphere PMC成员，Apache APISIX PMC 成员，Apache ECharts (incubating) 和Apache DolphinScheduler (incubating) 孵化器导师，Zipkin 成员和贡献者。\n分享大纲 分布式追踪兴起的背景 SkyWalking和其他分布式追踪的异同 定位问题的流程和方法 性能剖析的由来、用途和优势 听众收获 听众能够全面的了解分布式追踪的技术背景，和技术原理。以及为什么这些年，分布式追踪和基于分布式追踪的APM系统，Apache SkyWalking，得到了广泛的使用、集成，甚至云厂商的支持。同时，除了针对追踪数据，我们应该关注更多的是，如何利用其产生的监控数据，定位系统的性能问题。以及它有哪些短板，应该如何弥补。\nB站视频地址\n","excerpt":"\u003ch2 id=\"吴晟\"\u003e吴晟\u003c/h2\u003e\n\u003cp\u003e吴晟，Apache 基金会会员，Apache SkyWalking 创始人、项目 VP 和 PMC 成员，Apache 孵化器 PMC 成员，Apache ShardingSphere PMC成 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2020-08-13-cloud-native-academy/","title":"[视频] 云原生学院 - 后分布式追踪时代的性能问题定位——方法级性能剖析"},{"body":"SkyWalking Chart 3.1.0 is released. Go to downloads page to find release tars.\nSupport SkyWalking 8.1.0 Support enable oap dynamic configuration through k8s configmap ","excerpt":"\u003cp\u003eSkyWalking Chart 3.1.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eSupport SkyWalking …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-chart-3-1-0-for-skywalking-8-1-0/","title":"Release Apache SkyWalking Chart 3.1.0 for SkyWalking 8.1.0"},{"body":" Author: Sheng Wu Original link, Tetrate.io blog SkyWalking, a top-level Apache project, is the open source APM and observability analysis platform that is solving the problems of 21st-century systems that are increasingly large, distributed, and heterogenous. It\u0026rsquo;s built for the struggles system admins face today: To identify and locate needles in a haystack of interdependent services, to get apples-to-apples metrics across polyglot apps, and to get a complete and meaningful view of performance.\nSkyWalking is a holistic platform that can observe microservices on or off a mesh, and can provide consistent monitoring with a lightweight payload.\nLet\u0026rsquo;s take a look at how SkyWalking evolved to address the problem of observability at scale, and grew from a pure tracing system to a feature-rich observability platform that is now used to analyze deployments that collect tens of billions of traces per day.\nDesigning for scale When SkyWalking was first initialized back in 2015, its primary use case was monitoring the first-generation distributed core system of China Top Telecom companies, China Unicom and China Mobile. In 2013-2014, the telecom companies planned to replace their old traditional monolithic applications with a distributed system. Supporting a super-large distributed system and scaleablity were the high-priority design goals from Day one. So, what matters at scale?\nPull vs. push Pull and push modes relate to the direction of data flow. If the agent collects data and pushes them to the backend for further analysis, we call it \u0026ldquo;push\u0026rdquo; mode. Debate over pull vs. push has gone on for a long time. The key for an observability system is to minimize the cost of the agent, and to be generally suitable for different kinds of observability data.\nThe agent would send the data out a short period after it is collected. Then, we would have less concern about overloading the local cache. One typical case would be endpoint (URI of HTTP, service of gRPC) metrics. Any service could easily have hundreds, even thousands of endpoints. An APM system must have these metrics analysis capabilities.\nFurthermore, metrics aren\u0026rsquo;t the only thing in the observability landscape; traces and logs are important too. SkyWalking is designed to provide a 100% sampling rate tracing capability in the production environment. Clearly, push mode is the only solution.\nAt the same time, using push mode natively doesn\u0026rsquo;t mean SkyWalking can\u0026rsquo;t do data pulling. In recent 8.x releases, SkyWalking supports fetching data from Prometheus-instrumented services for reducing the Non-Recurring Engineering of the end users. Also, pull mode is popular in the MQ based transport, typically as a Kafka consumer. The SkyWalking agent side uses the push mode, and the OAP server uses the pull mode.\nThe conclusion: push mode is the native way, but pull mode works in some special cases too.\nMetrics analysis isn\u0026rsquo;t just mathematical calculation Metrics rely on mathematical theories and calculations. Percentile is a good measure for identifying the long tail issue, and reasonable average response time and successful rate are good SLO(s). But those are not all. Distributed tracing provides not just traces with detailed information, but high values metrics that can be analyzed.\nThe service topology map is required from Ops and SRE teams for the NOC dashboard and confirmation of system data flow. SkyWalking uses the STAM (Streaming Topology Analysis Method) to analyze topology from the traces, or based on ALS (Envoy Access Log Service) in the service mesh environment. This topology and metrics of nodes (services) and lines (service relationships) can\u0026rsquo;t be pulled from simple metrics SDKs.\nAs with fixing the limitation of endpoint metrics collection, SkyWalking needs to do endpoint dependency analysis from trace data too. Endpoint dependency analysis provides more important and specific information, including upstream and downstream. Those dependency relationships and metrics help the developer team to locate the boundaries of a performance issue, to specific code blocks.\nPre-calculation vs. query stage calculation? Query stage calculation provides flexibility. Pre-calculation, in the analysis stage, provides better and much more stable performance. Recall our design principle: SkyWalking targets a large-scale distributed system. Query stage calculation was very limited in scope, and most metrics calculations need to be pre-defined and pre-calculated. The key of supporting large datasets is reducing the size of datasets in the design level. Pre-calculation allows the original data to be merged into aggregated results downstream, to be used in a query or even for an alert check.\nTTL of metrics is another important business enabler. With the near linear performance offered by queries because of pre-calculation, with a similar query infrastructure, organizations can offer higher TTL, thereby providing extended visibility of performance.\nSpeaking of alerts, query-stage calculation also means the alerting query is required to be based on the query engine. But in this case, when the dataset increasing, the query performance could be inconsistent. The same thing happens in a different metrics query.\nCases today Today, SkyWalking is monitoring super large-scale distributed systems in many large enterprises, including Alibaba, Huawei, Tencent, Baidu, China Telecom, and various banks and insurance companies. The online service companies have more traffic than the traditional companies, like banks and telecom suppliers.\nSkyWalking is the observability platform used for a variety of use cases for distributed systems that are super-large by many measures:\nLagou.com, an online job recruitment platform SkyWalking is observing \u0026gt;100 services, 500+ JVM instances SkyWalking collects and analyzes 4+ billion traces per day to analyze performance data, including metrics of 300k+ endpoints and dependencies Monitoring \u0026gt;50k traffic per second in the whole cluster Yonghui SuperMarket, online service SkyWalking analyzes at least 10+ billion (3B) traces with metrics per day SkyWalking\u0026rsquo;s second, smaller deployment, analyzes 200+ million traces per day Baidu, internet and AI company, Kubernetes deployment SkyWalking collects 1T+ traces a day from 1,400+ pods of 120+ services Continues to scale out as more services are added Beike Zhaofang(ke.com), a Chinese online property brokerage backed by Tencent Holdings and SoftBank Group Has used SkyWalking from its very beginning, and has two members in the PMC team. Deployments collect 16+ billion traces per day Ali Yunxiao, DevOps service on the Alibaba Cloud, SkyWalking collects and analyzes billions of spans per day SkyWalking keeps AliCloud\u0026rsquo;s 45 services and ~300 instances stable A department of Alibaba TMall, one of the largest business-to-consumer online retailers, spun off from Taobao A customized version of SkyWalking monitors billions of traces per day At the same time, they are building a load testing platform based on SkyWalking\u0026rsquo;s agent tech stack, leveraging its tracing and context propagation cabilities Conclusion SkyWalking\u0026rsquo;s approach to observability follows these principles:\nUnderstand the logic model: don\u0026rsquo;t treat observability as a mathematical tool. Identify dependencies first, then their metrics. Scaling should be accomplished easily and natively. Maintain consistency across different architectures, and in the performance of APM itself. Resources Read about the SkyWalking 8.1 release highlights. Get more SkyWalking updates on Twitter. Sign up to hear more about SkyWalking and observability from Tetrate. ","excerpt":"\u003cul\u003e\n\u003cli\u003eAuthor: Sheng Wu\u003c/li\u003e\n\u003cli\u003eOriginal link, \u003ca href=\"https://www.tetrate.io/blog/observability-at-scale-skywalking-it-is/\"\u003eTetrate.io blog\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSkyWalking, a top-level Apache project, is the …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2020-08-11-observability-at-scale/","title":"Observability at Scale: SkyWalking it is"},{"body":" 作者：吴晟 翻译：董旭 金蝶医疗 原文链接：Tetrate.io blog SkyWalking做为Apache的顶级项目，是一个开源的APM和可观测性分析平台，它解决了21世纪日益庞大、分布式和异构的系统的问题。它是为应对当前系统管理所面临的困难而构建的：就像大海捞针，SkyWalking可以在服务依赖复杂且多语言环境下，获取服务对应的指标，以及完整而有意义的性能视图。\nSkyWalking是一个非常全面的平台，无论你的微服务是否在服务网格(Service Mesh)架构下，它都可以提供高性能且一致性的监控。\n让我们来看看，SkyWalking是如何解决大规模集群的可观测性问题，并从一个纯粹的链路跟踪系统，发展成为一个每天分析百亿级跟踪数据，功能丰富的可观测性平台。\n为超大规模而生 SkyWalking的诞生，时间要追溯到2015年，当时它主要应用于监控顶级电信公司（例如：中国联通和中国移动）的第一代分布式核心系统。2013-2014年，这些电信公司计划用分布式系统取代传统的单体架构应用。从诞生那天开始，SkyWalking首要的设计目标，就是能够支持超大型分布式系统，并具有很好可扩展性。那么支撑超大规模系统要考虑什么呢？\n拉取vs推送 与数据流向息息相关的：拉取模式和推送模式。Agent（客户端）收集数据并将其推送到后端，再对数据进一步分析，我们称之为“推送”模式。究竟应该使用拉取还是推送？这个话题已经争论已久。关键因素取决于可观测性系统的目标，即：在Agent端花最小的成本，使其适配不同类型的可观测性数据。\nAgent收集数据后，可以在短时间内发送出去。这样，我们就不必担心本地缓存压力过大。举一个典型的例子，任意服务都可以轻松地拥有数百个甚至数千个端点指标（如：HTTP的URI，gRPC的服务）。那么APM系统就必须具有分析这些数量庞大指标的能力。\n此外，度量指标并不是可观测性领域中的唯一关注点，链路跟踪和日志也很重要。在生产环境下，SkyWalking为了能提供100%采样率的跟踪能力，数据推送模式是唯一可行的解决方案。\nSkyWalking即便使用了推送模式，同时也可进行数据拉取。在最近的8.x的发版本中，SkyWalking支持从已经集成Prometheus的服务中获取终端用户的数据，避免重复工程建设，减少资源浪费。另外，比较常见的是基于MQ的传输构建拉取模式，Kafka消费者就是一个比较典型的例子。SkyWalking的Agent端使用推送模式，OAP服务器端使用拉取模式。\n结论：SkyWalking的推送模式是原生方式，但拉取式模式也适用于某些特殊场景。\n度量指标分析并不仅仅是数学统计 度量指标依赖于数学理论和计算。Percentile（百分位数）是用于反映响应时间的长尾效应。服务具备合理的平均响应时间和成功率，说明服务的服务等级目标(SLO）很好。除此之外，分布式跟踪还为跟踪提供了详细的信息，以及可分析的高价值指标。\n运维团队（OPS）和系统稳定性（SRE）团队通过服务拓扑图，用来观察网络情况（当做NOC dashboard使用）、确认系统数据流。SkyWalking依靠trace（跟踪数据），使用STAM（Streaming Topology Analysis Method）方法进行分析拓扑结构。在服务网格环境下，使用ALS（Envoy Access Log Service）进行拓扑分析。节点（services）和线路（service relationships）的拓扑结构和度量指标数据，无法通过sdk轻而易举的拿到。\n为了解决端点度量指标收集的局限性，SkyWalking还要从跟踪数据中分析端点依赖关系，从而拿到链路上游、下游这些关键具体的信息。这些依赖关系和度量指标信息，有助于开发团队定位引起性能问题的边界，甚至代码块。\n预计算还是查询时计算？ 相比查询时计算的灵活性，预计算可以提供更好、更稳定的性能，这在分析场景下尤为重要。回想一下我们的设计原则：SkyWalking是为了一个大规模的分布式系统而设计。查询时计算的使用范围非常有限，大多数度量计算都需要预先定义和预先计算。支持大数据集的关键是：在设计阶段，要减小数据集。预计算允许将原始数据合并到下游的聚合结果中，用于查询，甚至用于警报检查。\n使用SkyWalking的另一个重要因素是：指标的有效期，TTL（Time To Live）。由于采用了预先计算，查询提供了近似线性的高性能。这也帮助“查询系统”这类基础设施系统，提供更好的性能扩展。\n关于警报，使用查询时计算方案，也意味着警报查询需要基于查询引擎。但在这种情况下，随着数据集增加，查询性能会随之下降，其他指标查询也是一样的结果。\n目前使用案例 如今，SkyWalking在许多大型企业的超大规模分布式系统中使用，包括阿里巴巴、华为、腾讯、百度、中国通讯企业以及多家银行和保险公司。上线SkyWalking公司的流量，比银行和电信运营商这种传统公司还要大。\n在很多行业中，SkyWalking是被应用于超大型分布式系统各种场景下的一个可观测性平台：\n拉勾网\nSkyWalking正在观测超过100个服务，500多个JVM实例\nSkyWalking每天收集和分析40多亿个跟踪数据，用来分析性能，其中包括30万个端点和依赖关系的指标\n在整个群集中监控\u0026gt;50k流量/秒\n永辉超市\nSkyWalking每天分析至少100多亿（3B）的跟踪数据\n其次，SkyWalking用较小的部署，每天分析2亿多个跟踪数据\n百度\nSkyWalking每天从1400多个pod中，从120多个服务收集1T以上的跟踪数据\n随着更多服务的增加，规模会持续增大\n贝壳找房(ke.com)\n很早就使用了SkyWalking，有两名成员已经成为PMC\nDeployments每天收集160多亿个跟踪数据\n阿里云效\nSkyWalking每天收集和分析数十亿个span\nSkyWalking使阿里云的45项服务和~300个实例保持稳定\n阿里巴巴天猫\nSkyWalking个性化定制版，每天监控数十亿跟踪数据\n与此同时，他们基于SkyWalking的Agent技术栈，利用其跟踪和上下文传播能力，正在构建一个全链路压测平台\n结论 SkyWalking针对可观测性遵循以下原则：\n理解逻辑模型：不要把可观测性当作数学统计工具。 首先确定依赖关系，然后确定它们的度量指标。 原生和方便的支撑大规模增长。 在不同的架构情况下，APM各方面表现依然保持稳定和一致。 资源 阅读SkyWalking 8.1发布亮点。 在Twitter上获取更多SkyWalking更新。 注册Tetrate以了解更多有关SkyWalking可观测性的信息。 ","excerpt":"\u003cul\u003e\n\u003cli\u003e作者：吴晟\u003c/li\u003e\n\u003cli\u003e翻译：董旭 金蝶医疗\u003c/li\u003e\n\u003cli\u003e原文链接：\u003ca href=\"https://www.tetrate.io/blog/observability-at-scale-skywalking-it-is/\"\u003eTetrate.io blog\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSkyWalking做为Apache的顶级项目，是一个开源的APM和可观测性分析平台，它解决了21世纪日益庞大、分布式和 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2020-08-11-observability-at-scale-skywalking-it-is/","title":"SkyWalking 为超大规模而生"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/use-case/","title":"Use Case"},{"body":" Author: Sheng Wu, Hongtao Gao, and Tevah Platt(Tetrate) Original link, Tetrate.io blog Apache SkyWalking, the observability platform, and open-source application performance monitor (APM) project, today announced the general availability of its 8.1 release that extends its functionalities and provides a transport layer to maintain the lightweight of the platform that observes data continuously.\nBackground SkyWalking is an observability platform and APM tool that works with or without a service mesh, providing automatic instrumentation for microservices, cloud-native and container-based applications. The top-level Apache project is supported by a global community and is used by Alibaba, Huawei, Tencent, Baidu, and scores of others.\nTransport traces For a long time, SkyWalking has used gRPC and HTTP to transport traces, metrics, and logs. They provide good performance and are quite lightweight, but people kept asking about the MQ as a transport layer because they want to keep the observability data continuously as much as possible. From SkyWalking’s perspective, the MQ based transport layer consumes more resources required in the deployment and the complexity of deployment and maintenance but brings more powerful throughput capacity between the agent and backend.\nIn 8.1.0, SkyWalking officially provides the typical MQ implementation, Kafka, to transport all observability data, including traces, metrics, logs, and profiling data. At the same time, the backend can support traditional gRPC and HTTP receivers, with the new Kafka consumer at the same time. Different users could choose the transport layer(s) according to their own requirements. Also, by referring to this implementation, the community could contribute various transport plugins for Apache Pulsar, RabbitMQ.\nAutomatic endpoint dependencies detection The 8.1 SkyWalking release offers automatic detection of endpoint dependencies. SkyWalking has long offered automatic endpoint detection, but endpoint dependencies, including upstream and downstream endpoints, are critical for Ops and SRE teams’ performance analysis. The APM system is expected to detect the relationships powered by the distributed tracing. While SkyWalking has been designed to include this important information at the beginning the latest 8.1 release offers a cool visualization about the dependency and metrics between dependent endpoints. It provides a new drill-down angle from the topology. Once you have the performance issue from the service level, you could check on instance and endpoint perspectives:\nSpringSleuth metrics detection In the Java field, the Spring ecosystem is one of the most widely used. Micrometer, the metrics API lib included in the Spring Boot 2.0, is now adopted by SkyWalking’s native meter system APIs and agent. For applications using Micrometer with the SkyWalking agent installed, all Micrometer collected metrics could then be shipped into SkyWalking OAP. With some configurations in the OAP and UI, all metrics are analyzed and visualized in the SkyWalking UI, with all other metrics detected by SkyWalking agents automatically.\nNotable enhancements The Java agent core is enhanced in this release. It could work better in the concurrency class loader case and is more compatible with another agent solution, such as Alibaba’s Arthas.\nWith the logic endpoint supported, the local span can be analyzed to get metrics. One span could carry the raw data of more than one endpoint’s performance. GraphQL, InfluxDB Java Client, and Quasar fiber libs are supported to be observed automatically. Kubernetes Configmap can now for the first time be used as the dynamic configuration center– a more cloud-native solution for k8s deployment environments. OAP supports health checks, especially including the storage health status. If the storage (e.g., ElasticSearch) is not available, you could get the unhealth status with explicit reasons through the health status query. Opencensus receiver supports ingesting OpenTelemetry/OpenCensus agent metrics by meter-system. Additional resources Read more about the SkyWalking 8.1 release highlights. Read more about SkyWalking from Tetrate on our blog. Get more SkyWalking updates on Twitter. Sign up to hear more about SkyWalking and observability from Tetrate. ","excerpt":"\u003cul\u003e\n\u003cli\u003eAuthor: Sheng Wu, Hongtao Gao, and Tevah Platt(Tetrate)\u003c/li\u003e\n\u003cli\u003eOriginal link, \u003ca href=\"https://www.tetrate.io/blog/skywalking8-1-release/\"\u003eTetrate.io blog\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cimg src=\"apache-skywalking.jpg\" alt=\"\"\u003e\u003c/p\u003e\n\u003cp\u003eApache …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2020-08-03-skywalking8-1-release/","title":"Features in SkyWalking 8.1: SpringSleuth metrics, endpoint dependency detection, Kafka transport traces and metrics"},{"body":"SkyWalking APM 8.1.0 is release. Go to downloads page to find release tars.\nProject Support Kafka as an optional trace, JVM metrics, profiling snapshots and meter system data transport layer. Support Meter system, including the native metrics APIs and the Spring Sleuth adoption. Support JVM thread metrics. Java Agent [Core] Fix the concurrency access bug in the Concurrency ClassLoader Case. [Core] Separate the config of the plugins from the core level. [Core] Support instrumented class cached in memory or file, to be compatible with other agents, such as Arthas. Add logic endpoint concept. Could analysis any span or tags flagged by the logic endpoint. Add Spring annotation component name for UI visualization only. Add support to trace Call procedures in MySQL plugin. Support GraphQL plugin. Support Quasar fiber plugin. Support InfluxDB java client plugin. Support brpc java plugin Support ConsoleAppender in the logback v1 plugin. Enhance vert.x endpoint names. Optimize the code to prevent mongo statements from being too long. Fix WebFlux plugin concurrency access bug. Fix ShardingSphere plugins internal conflicts. Fix duplicated Spring MVC endpoint. Fix lettuce plugin sometimes trace doesn‘t show span layer. Fix @Tag returnedObject bug. OAP-Backend Support Jetty Server advanced configurations. Support label based filter in the prometheus fetcher and OpenCensus receiver. Support using k8s configmap as the configuration center. Support OAP health check, and storage module health check. Support sampling rate in the dynamic configuration. Add endpoint_relation_sla and endpoint_relation_percentile for endpoint relationship metrics. Add components for Python plugins, including Kafka, Tornado, Redis, Django, PyMysql. Add components for Golang SDK. Add Nacos 1.3.1 back as an optional cluster coordinator and dynamic configuration center. Enhance the metrics query for ElasticSearch implementation to increase the stability. Reduce the length of storage entity names in the self-observability for MySQL and TiDB storage. Fix labels are missing in Prometheus analysis context. Fix column length issue in MySQL/TiDB storage. Fix no data in 2nd level aggregation in self-observability. Fix searchService bug in ES implementation. Fix wrong validation of endpoint relation entity query. Fix the bug caused by the OAL debug flag. Fix endpoint dependency bug in MQ and uninstrumented proxy cases. Fix time bucket conversion issue in the InfluxDB storage implementation. Update k8s client to 8.0.0 UI Support endpoint dependency graph. Support x-scroll of trace/profile page Fix database selector issue. Add the bar chart in the UI templates. Document Update the user logo wall. Add backend configuration vocabulary document. Add agent installation doc for Tomcat9 on Windows. Add istioctl ALS commands for the document. Fix TTL documentation. Add FAQ doc about thread instrumentation. CVE Fix fuzzy query sql injection in the MySQL/TiDB storage. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking APM 8.1.0 is release. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Kafka …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-8-1-0/","title":"Release Apache SkyWalking APM 8.1.0"},{"body":"Based on his continuous contributions, Wei Hua (a.k.a alonelaval) has been voted as a new committer.\n","excerpt":"\u003cp\u003eBased on his continuous contributions, Wei Hua (a.k.a \u003ca href=\"https://github.com/alonelaval\"\u003ealonelaval\u003c/a\u003e) has been voted as a new committer.\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-wei-hua-as-new-committer/","title":"Welcome Wei Hua as new committer"},{"body":"SkyWalking Python 0.2.0 is released. Go to downloads page to find release tars.\nPlugins:\nKafka Plugin (#50) Tornado Plugin (#48) Redis Plugin (#44) Django Plugin (#37) PyMsql Plugin (#35) Flask plugin (#31) API\nAdd ignore_suffix Config (#40) Add missing log method and simplify test codes (#34) Add content equality of SegmentRef (#30) Validate carrier before using it (#29) Chores and tests\nTest: print the diff list when validation failed (#46) Created venv builders for linux/windows and req flashers + use documentation (#38) ","excerpt":"\u003cp\u003eSkyWalking Python 0.2.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003ePlugins:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eKafka …\u003c/li\u003e\u003c/ul\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-python-0-2-0/","title":"Release Apache SkyWalking Python 0.2.0"},{"body":"SkyWalking CLI 0.3.0 is released. Go to downloads page to find release tars.\nCommand: health check command Command: Add trace command BugFix: Fix wrong metrics graphql path ","excerpt":"\u003cp\u003eSkyWalking CLI 0.3.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eCommand: health check …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cli-0-3-0/","title":"Release Apache SkyWalking CLI 0.3.0"},{"body":" Author: Srinivasan Ramaswamy, tetrate Original link, Tetrate.io blog Asking How are you is more profound than What are your symptoms Background Recently I visited my preferred doctor. Whenever I visit, the doctor greets me with a series of light questions: How’s your day? How about the week before? Any recent trips? Did I break my cycling record? How’s your workout regimen? _Finally _he asks, “Do you have any problems?\u0026quot; On those visits when I didn\u0026rsquo;t feel ok, I would say something like, \u0026ldquo;I\u0026rsquo;m feeling dull this week, and I\u0026rsquo;m feeling more tired towards noon….\u0026rdquo; It\u0026rsquo;s at this point that he takes out his stethoscope, his pulse oximeter, and blood pressure apparatus. Then, if he feels he needs a more in-depth insight, he starts listing out specific tests to be made.\nWhen I asked him if the first part of the discussion was just an ice-breaker, he said, \u0026ldquo;That\u0026rsquo;s the essential part. It helps me find out how you feel, rather than what your symptoms are.\u0026rdquo; So, despite appearances, our opening chat about life helped him structure subsequent questions on symptoms, investigations and test results.\nOn the way back, I couldn\u0026rsquo;t stop asking myself, \u0026ldquo;Shouldn\u0026rsquo;t we be managing our mesh this way, too?\u0026rdquo;\nIf I strike parallels between my own health check and a health check, “tests” would be log analysis, “investigations” would be tracing, and “symptoms” would be the traditional RED (Rate, Errors and Duration) metrics. That leaves the “essential part,” which is what we are talking about here: the Wellness Factor, primarily the health of our mesh.\nHealth in the context of service mesh We can measure the performance of any observed service through RED metrics. RED metrics offer immense value in understanding the performance, reliability, and throughput of every service. Compelling visualizations of these metrics across the mesh make monitoring the entire mesh standardized and scalable. Also, setting alerts based on thresholds for each of these metrics helps to detect anomalies as and when they arise.\nTo establish the context of any service and observe them, it\u0026rsquo;s ideal to visualize the mesh as a topology.\nA topology visualization of the mesh not only allows for picking any service and watching its metrics, but also gives vital information about service dependencies and the potential impact of a given service on the mesh.\nWhile RED metrics of each service offer tremendous insights, the user is more concerned with the overall responsiveness of the mesh rather than each of these services in isolation.\nTo describe the performance of any service, right from submitting the request to receiving a completed http response, we’d be measuring the user\u0026rsquo;s perception of responsiveness. This measure of response time compared with a set threshold is called Apdex. This Apdex is an indicator of the health of a service in the mesh.\nApdex Apdex is a measure of response time considered against a set threshold**. **It is the ratio of satisfactory response times and unsatisfactory response times to total response times.\nApdex is an industry standard to measure the satisfaction of users based on the response time of applications and services. It measures how satisfied your users are with your services, as traditional metrics such as average response time could get skewed quickly.\nSatisfactory response time indicates the number of times when the roundtrip response time of a particular service was less than this threshold. Unsatisfactory response time while meaning the opposite, is further categorized as Tolerating and Frustrating. Tolerating accommodates any performance that is up to four times the threshold, and anything over that or any errors encountered is considered Frustrating. The threshold mentioned here is an ideal roundtrip performance that we expect from any service. We could even start with an organization-wide limit of say, 500ms.\nThe Apdex score is a ratio of satisfied and tolerating requests to the total requests made.\nEach satisfied request counts as one request, while each tolerating request counts as half a satisfied request.\nAn Apdex score takes values from 0 to 1, with 0 being the worst possible score indicating that users were always frustrated, and ‘1’ as the best possible score (100% of response times were Satisfactory).\nA percentage representation of this score also serves as the Health Indicator of the service.\nThe Math The actual computation of this Apdex score is achieved through the following formula.\nSatisfiedCount + ( ToleratingCount / 2 ) Apdex Score = ------------------------------------------------------ TotalSamples A percentage representation of this score is known as the Health Indicator of a service.\nExample Computation During a 2-minute period, a host handles 200 requests.\nThe Apdex threshold T = 0.5 seconds (500ms).\n170 of the requests were handled within 500ms, so they are classified as Satisfied. 20 of the requests were handled between 500ms and 2 seconds (2000 ms), so they are classified as Tolerating. The remaining 10 were not handled properly or took longer than 2 seconds, so they are classified as Frustrated. The resulting Apdex score is 0.9: (170 + (20/2))/200 = 0.9.\nThe next level At the next level, we can attempt to improve our topology visualization by coloring nodes based on their health. Also, we can include health as a part of the information we show when the user taps on a service.\nApdex specifications recommend the following Apdex Quality Ratings by classifying Apdex Score as Excellent (0.94 - 1.00), Good (0.85 - 0.93), Fair (0.70 - 0.84), Poor (0.50 - 0.69) and Unacceptable (0.00 - 0.49).\nTo visualize this, let’s look at our topology using traffic light colors, marking our nodes as Healthy, At-Risk and Unhealthy, where Unhealthy indicates health that falls below 80%. A rate between 80% and 95% indicates At-Risk, and health at 95% and above is termed Healthy.\nLet’s incorporate this coloring into our topology visualization and take its usability to the next level. If implemented, we will be looking at something like this.\nMoving further Apdex provides tremendous visibility into customer satisfaction on the responsiveness of our services. Even more, by extending the implementation to the edges calling this service we get further insight into the health of the mesh itself.\nTwo services with similar Apdex scores offer the same customer satisfaction to the customer. However, the size of traffic that flows into the service can be of immense help in prioritizing between services to address. A service with higher traffic flow is an indication that this experience is impacting a significant number of users on the mesh.\nWhile health relates to a service, we can also analyze the interactions between two services and calculate the health of the interaction. This health calculation of every interaction on the mesh helps us establish a critical path, based on the health of all interactions in the entire topology.\nIn a big mesh, showing traffic as yet another number will make it more challenging to visualize and monitor. We can, with a bit of creativity, improve the entire visualization by rendering the edges that connect services with different thickness depending on the throughput of the service.\nAn unhealthy service participating in a high throughput transaction could lead to excessive consumption of resources. On the other hand, this visualization also offers a great tip to maximize investment in tuning services.\nTuning service that is a part of a high throughput transaction offers exponential benefits when compared to tuning an occasionally used service.\nIf we look at implementing such a visualization, which includes the health of interactions and throughput of such interactions, we would be looking at something like below :\nThe day is not far These capabilities are already available to users today as one of the UI features of Tetrate’s service mesh platform, using the highly configurable and performant observability and performance management framework: Apache SkyWalking (https://skywalking.apache.org), which monitors traffic across the mesh, aggregates RED metrics for both services and their interactions, continuously computes and monitors health of the services, and enables users to configure alerts and notifications when services cross specific thresholds, thereby having a comprehensive health visibility of the mesh.\nWith such tremendous visibility into our mesh performance, the day is not far when we at our NOC (Network Operations Center) for the mesh have this topology as our HUD (Heads Up Display).\nThis HUD, with the insights and patterns gathered over time, would predict situations and proactively prompt us on potential focus areas to improve customer satisfaction.\nThe visualization with rich historical data can also empower the Network Engineers to go back in time and look at the performance of the mesh on a similar day in the past.\nAn earnest implementation of such a visualization would be something like below :\nTo conclude With all the discussion so far, the health of a mesh is more about how our users feel, and what we can proactively do as service providers to sustain, if not enhance, the experience of our users.\nAs the world advances toward personalized medicine, we\u0026rsquo;re not far from a day when my doctor will text me: \u0026ldquo;How about feasting yourself with ice cream today and take the Gray Butte Trail to Mount Shasta!\u0026rdquo; Likewise, we can do more for our customers by having better insight into their overall wellness.\nTetrate’s approach to “service mesh health” is not only to offer management, monitoring and support but to make infrastructure healthy from the start to reduce the probability of incidents. Powered by the Istio, Envoy, and SkyWalking, Tetrate\u0026rsquo;s solutions enable consistent end-to-end observability, runtime security, and traffic management for any workload in any environment.\nOur customers deserve healthy systems! Please do share your thoughts on making service mesh an exciting and robust experience for our customers.\nReferences https://en.wikipedia.org/wiki/Apdex https://www.apdex.org/overview.html https://www.apdex.org/index.php/specifications/ https://skywalking.apache.org/ ","excerpt":"\u003cul\u003e\n\u003cli\u003eAuthor: Srinivasan Ramaswamy, tetrate\u003c/li\u003e\n\u003cli\u003eOriginal link, \u003ca href=\"https://www.tetrate.io/blog/the-apdex-score-for-measuring-service-mesh-health/\"\u003eTetrate.io blog\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"asking-how-are-you-is-more-profound-than-what-are-your-symptoms\"\u003eAsking \u003ccode\u003eHow are you\u003c/code\u003e is more …\u003c/h2\u003e","ref":"https://skywalking.apache.org/blog/2020-07-26-apdex-and-skywalking/","title":"The Apdex Score for Measuring Service Mesh Health"},{"body":" 作者: Srinivasan Ramaswamy, tetrate 翻译：唐昊杰，南京大学在读学生 校对：吴晟 Original link, Tetrate.io blog July. 26th, 2020 \u0026ldquo;你感觉怎么样\u0026rdquo; 比 \u0026ldquo;你的症状是什么\u0026rdquo; 更重要 背景 最近我拜访了我的医生。每次去看病，医生都会首先问我一连串轻快的问题，比如：你今天过得怎么样？上周过的怎么样？最近有什么出行吗？你打破了自己的骑车记录吗？你的锻炼计划实施如何？最后他会问：“你有什么麻烦吗？”如果这个时候我感觉自己不太好，我会说：“我这周感觉很沉闷，临近中午的时候感觉更累。”这时他就会拿出听诊器、脉搏血氧仪和血压仪。然后，如果他觉得自己需要更深入的了解情况，他就开始列出我需要做的具体检查。\n当我问他，最开始的讨论是否只是为了缓和氛围。他说：“这是必不可少的部分。它帮助我发现你感觉如何，而不是你的症状是什么。\u0026quot;。我们这样关于生活的开场聊天，帮助他组织了后续关于症状、调查和测试结果的问题。\n在回来的路上，我不停地问自己：“我们是不是也应该用这种方式管理我们的网格(service mesh)？”\n如果我把自己的健康检查和网格的健康检查进行类比，“医疗检查”就是日志分析，“调查”就是追踪，“症状”就是传统的RED指标（请求速率、请求错误和请求耗时）。那么根本的问题，就是我们在这里讨论的：健康因素（主要是网格的健康）。\n服务网格中的健康状况 我们可以通过RED指标来衡量任何被观察到的服务的性能。RED指标在了解每个服务的性能、可靠性和吞吐量方面提供了巨大的价值。这些指标在网格上的令人信服的可视化使得监控全部网格变得标准化和可扩展。此外，根据这些指标的阈值设置警报有助于在指标值异常的时候进行异常检测。\n为了建立任何服务的上下文环境并观察它们，理想的做法是将网格可视化为一个拓扑结构。\n网格的拓扑结构可视化不仅允许使用者挑选任意服务并观察其指标，还可以提供有关服务依赖和特定服务在网格上的潜在影响这些重要信息。\n虽然每个服务的RED指标为使用者提供了深刻的洞察能力，但使用者更关心网格的整体响应性，而非每个单独出来的服务的响应性。\n为了描述任意服务的性能（即从提交请求到收到完成了的http响应这段时间内的表现），我们会测量用户对响应性的感知。这种将响应时间与设定的阈值进行比较的衡量标准叫做Apdex。Apdex是衡量一个服务在网格中的健康程度的指标。\nApdex Apdex是根据设定的阈值和响应时间结合考虑的衡量标准。它是满意响应时间和不满意响应时间相对于总响应时间的比率。\nApdex是根据应用和服务的响应时间来衡量使用者满意程度的行业标准。它衡量的是用户对你的服务的满意程度，因为传统的指标（如平均响应时间）可能很快就会容易形成偏差。\n基于满意度的响应时间，表示特定服务的往返响应时间小于设定的阈值的次数。不满意响应时间虽然意思相反，但又进一步分为容忍型和失望型。容忍型包括了了任何响应时间不超过四倍阈值的表现，而任何超过四倍阈值或遇到了错误的表现都被认为是失望型。这里提到的阈值是我们对任意服务所期望的理想响应表现。我们可以设置一个全局范围的阈值，如，500ms。\nApdex得分是满意型请求和容忍型请求与做出的总请求的比率。\n每个_满意的请求_算作一个请求，而每个_容忍的请求_算作半个_满意_的请求。\n一个Apdex得分从0到1的范围内取值。0是最差的分数，表示用户总是感到失望；而'1\u0026rsquo;是最好的分数（100%的响应时间是令人满意的）。\n这个分数的百分比表示也可以用作服务的健康指标。\n数学表示 Apdex得分的实际计算是通过以下公式实现的：\n满意请求数 + ( 容忍请求数 / 2 ) Apdex 得分 = ------------------------------------------------------ 总请求数 此公示得到的百分率，即可视为服务的健康度。\n样例计算 在两分钟的采样时间内，主机处理200个请求。\nApdex阈值T设置为0.5秒（500ms）。\n*.\t170个请求在500ms内被处理完成，它们被分类为满意型。 *.\t20个请求在500ms和2秒间被处理，它们被分类为容忍型。 *.\t剩余的10个请求没有被正确处理或者处理时间超过了2秒，所以它们被分类为失望型。\n最终的Apdex得分是0.9，即（170 + （20 / 2））/ 200。\n深入使用 在接下来的层次，我们可以尝试通过根据节点的健康状况来着色节点以改进我们的拓扑可视化。此外，我们还可以在用户点击服务时将健康状况作为我们展示的信息的一部分。\nApdex规范推荐了以下Apdex质量评级，将Apdex得分分为优秀（0.94 - 1.00）、良好（0.85 - 0.93）、一般（0.70 - 0.84）、差（0.50 - 0.69）和不可接受（0.00 - 0.49）。\n为了可视化网格的健康状况，我们用交通灯的颜色将我们的节点标记为健康、有风险和不健康，其中不健康表示健康率低于80%。健康率在80%到95%之间的表示有风险，健康率在95%及以上的称为健康。\n让我们将这种着色融入到我们的拓扑可视化中，并将其可用性提升到一个新的水平。如果实施，我们将看到下图所示的情况。\n更进一步 Apdex为客户对我们服务响应性的满意度提供了可见性。更有甚者，通过将实施范围扩展到调用该服务的调用关系，我们可以进一步了解网格本身的健康状况。\n两个有着相似Apdex分数的服务，为客户提供了相同的客户满意度。然而，流入服务的流量大小对于优先处理哪一服务有着巨大的帮助。流量较高的服务表明这种服务体验影响了网格上更大量的使用者。\n虽然健康程度与单个服务有关，但我们也可以分析两个服务之间的交互并计算交互过程的健康程度。这种对网格上每一个交互的健康程度的计算，可以帮助我们根据整个拓扑结构中所有交互的健康程度，建立一个关键路径。\n在一个大的网格中，将流量展示为另一个数字将使可视化和监控更具挑战性。我们可以根据服务的吞吐量，通过用不同的粗细程度渲染连接服务的边来改善整个可视化的效果。\n一个位于高吞吐量事务的不健康的服务可能会导致资源的过度消耗。另一方面，这种可视化也为调整服务时获取最大化投资效果提供了一个很好的提示。\n与调整一个偶尔使用的服务相比，调整作为高吞吐量事务的一部分的那些服务会带来指数级的收益。\n实施这种包括了交互的健康状况和吞吐量的可视化，我们会看到下图所示的情况:\n这一天即将到来 目前，这些功能已经作为Tetrate服务网格平台的UI功能之一来提供给用户。该平台使用了高速可配置化、高性能的可观测性和监控性能管理平台：Apache SkyWalking (https://skywalking.apache.org)，SkyWalking可以监控整个网格的流量，为服务及它们的交互合计RED指标，持续计算和监控服务的健康状况，并使用户能够在服务超过特定阈值时配置报警和通知。这些功能使得SkyWalking对网格拥有全面的健康状况可见性。\n有了这样强大的网格性能可视性，我们将可以在为网格准备的网络运营中心使用这种拓扑结构作为我们的HUD（Heads Up Display）。\nHUD随着时间的推移收集了解到的信息和模式，并将预测各种情况和主动提示我们潜在的重点领域以提高客户满意度。\n丰富的历史数据的可视化也可以使网络工程师能够看看过去中类似的一天的网格表现。\n可视化效果如下图所示。\n总结 综合到目前为止的所有讨论，网格的健康状况更多地是关于用户的感受，以及我们作为服务提供商可以采取积极行动来维持（如果不能增强）用户的体验。\n着个人化医学的发展，现在距离我的医生给我发这样短信的日子并不遥远：“要不今天享用冰淇淋并且沿着灰色小山步道到达沙斯塔山！”相似的，我们可以通过更好地了解客户的整体健康状况为他们做更多的事情。\nTetrate的“服务网格健康程度”方法不仅提供了管理，监视和支持，而且从一开始就使基础架构保持健康以减少事故发生的可能性。在Istio，Envoy和SkyWalking的支持下，Tetrate的解决方案可为任何环境中的任何工作负载提供持续的端到端可观察性，运行时安全性和流量管理。\n我们的客户应该拥有健康的系统！请分享您对使用服务网格为我们的客户带来令人兴奋和强健的体验的想法。\n引用 https://en.wikipedia.org/wiki/Apdex https://www.apdex.org/overview.html https://www.apdex.org/index.php/specifications/ https://skywalking.apache.org/ ","excerpt":"\u003cul\u003e\n\u003cli\u003e作者: Srinivasan Ramaswamy, tetrate\u003c/li\u003e\n\u003cli\u003e翻译：唐昊杰，南京大学在读学生\u003c/li\u003e\n\u003cli\u003e校对：吴晟\u003c/li\u003e\n\u003cli\u003eOriginal link, \u003ca href=\"https://www.tetrate.io/blog/the-apdex-score-for-measuring-service-mesh-health/\"\u003eTetrate.io blog\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eJuly. 26th, …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/zh/2020-07-26-apdex-and-skywalking/","title":"度量服务网格健康度——Apdex得分"},{"body":"SkyWalking Python 0.1.0 is released. Go to downloads page to find release tars.\nAPI: agent core APIs, check the APIs and the examples Plugin: built-in libraries http, urllib.request and third-party library requests are supported. Test: agent test framework is setup, and the corresponding tests of aforementioned plugins are also added. ","excerpt":"\u003cp\u003eSkyWalking Python 0.1.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAPI: agent core …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-python-0-1-0/","title":"Release Apache SkyWalking Python 0.1.0"},{"body":"SkyWalking Chart 3.0.0 is released. Go to downloads page to find release tars.\nSupport SkyWalking 8.0.1 ","excerpt":"\u003cp\u003eSkyWalking Chart 3.0.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eSupport SkyWalking …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-chart-3-0-0-for-skywalking-8-0-1/","title":"Release Apache SkyWalking Chart 3.0.0 for SkyWalking 8.0.1"},{"body":"Apache SkyWalking 8.0.1 已发布。SkyWalking 是观察性分析平台和应用性能管理系统，提供分布式追踪、服务网格遥测分析、度量聚合和可视化一体化解决方案，支持 Java, .Net Core, PHP, NodeJS, Golang, LUA 语言探针，支持 Envoy + Istio 构建的 Service Mesh。\n与 8.0.0 相比，此版本包含一个热修复程序。\nOAP-Backend\n修复 no-init 模式在 Elasticsearch 存储中无法运行的错误 8.0.0 值得关注的变化：\n添加并实现了 v3 协议，旧版本与 8.x 不兼容 移除服务、实例、端点注册机制和 inventory 存储实体 (inventory storage entities) 提供新的 GraphQL 查询协议，同时支持旧协议（计划在今年年底移除） 支持 Prometheus 网络协议，可将 Prometheus 格式的指标传输到 SkyWalking 中 提供 Python agent 移除所有 inventory 缓存 提供 Apache ShardingSphere (4.0.0, 4.1.1) agent 插件 UI dashboard 100% 可配置，可采用后台定义的新指标 修复 H2/MySQL 实现中的 SQL 注入漏洞 Upgrade Nacos to avoid the FastJson CVE in high frequency. 升级 Nacos 以避免 FastJson CVE 升级 jasckson-databind 至 2.9.10 下载地址：http://skywalking.apache.org/downloads/\n","excerpt":"\u003cp\u003e\u003ca href=\"https://github.com/apache/skywalking/releases/tag/v8.0.1\"\u003eApache SkyWalking 8.0.1 已发布\u003c/a\u003e。SkyWalking 是观察性分析平台和应用性能管理系统，提供分布式追踪、服务网格遥测分析、度量聚合和可视化一体化解决方案，支持 Java, …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2020-06-21-skywalking8-0-1-release/","title":"Apache SkyWalking 8.0.1 发布"},{"body":"SkyWalking Nginx LUA 0.2.0 is release. Go to downloads page to find release tars.\nAdapt the new v3 protocol. Implement correlation protocol. Support batch segment report. ","excerpt":"\u003cp\u003eSkyWalking Nginx LUA 0.2.0 is release. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAdapt the new v3 …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-nginx-lua-0-2-0/","title":"Relase Apache SkyWalking Nginx LUA 0.2.0"},{"body":"SkyWalking APM 8.0.0 is release. Go to downloads page to find release tars.\nProject v3 protocol is added and implemented. All previous releases are incompatible with 8.x releases. Service, Instance, Endpoint register mechanism and inventory storage entities are removed. New GraphQL query protocol is provided, the legacy procotol is still supported(plan to remove at the end of this year). Support Prometheus network protocol. Metrics in Prometheus format could be transferred into SkyWalking. Python agent provided. All inventory caches have been removed. Apache ShardingSphere(4.1.0, 4.1.1) agent plugin provided. Java Agent Add MariaDB plugin. Vert.x plugin enhancement. More cases are covered. Support v3 extension header. Fix ElasticSearch 5.x plugin TransportClient error. Support Correlation protocol v1. Fix Finagle plugin bug, in processing Noop Span. Make CommandService daemon to avoid blocking target application shutting down gracefully. Refactor spring cloud gateway plugin and support tracing spring cloud gateway 2.2.x OAP-Backend Support meter system for Prometheus adoption. In future releases, we will add native meter APIs and MicroMeter(Sleuth) system. Support endpoint grouping. Add SuperDataSet annotation for storage entity. Add superDatasetIndexShardsFactor in the ElasticSearch storage, to provide more shards for @SuperDataSet annotated entites. Typically TraceSegment. Support alarm settings for relationship of service, instance, and endpoint level metrics. Support alarm settings for database(conjecture node in tracing scenario). Data Model could be added in the runtime, don\u0026rsquo;t depend on the bootstrap sequence anymore. Reduce the memory cost, due to no inventory caches. No buffer files in tracing and service mesh cases. New ReadWriteSafe cache implementation. Simplify codes. Provide default way for metrics query, even the metrics doesn\u0026rsquo;t exist. New GraphQL query protocol is provided. Support the metrics type query. Set up length rule of service, instance, and endpoint. Adjust the default jks for ElasticSearch to empty. Fix Apdex function integer overflow issue. Fix profile storage issue. Fix TTL issue. Fix H2 column type bug. Add JRE 8-14 test for the backend. UI UI dashboard is 100% configurable to adopt new metrics definited in the backend. Document Add v8 upgrade document. Make the coverage accurate including UT and e2e tests. Add miss doc about collecting parameters in the profiled traces. CVE Fix SQL Injection vulnerability in H2/MySQL implementation. Upgrade Nacos to avoid the FastJson CVE in high frequency. Upgrade jasckson-databind to 2.9.10. All issues and pull requests are here\n","excerpt":"\u003cp\u003eSkyWalking APM 8.0.0 is release. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003ev3 protocol is …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-8-0-0/","title":"Release Apache SkyWalking APM 8.0.0"},{"body":"可观察性平台和开源应用程序性能监控（APM）项目 Apache SkyWalking，今天刚宣布 8.0 的发布版本。素以强劲指标、追踪与服务网格能力见称的 SkyWalking ，在最新版本中的功能性延展到用户渴求已久的功能 —— 将指标功能和包括 Prometheus 的其他指标收集系统进行了融合。\n什么是 Apache SkyWalking？ SkyWalking 是可观察性平台和 APM 工具，可以选择是否搭载服务网格的使用，为微服务、云原生和容器化应用提供自动度量功能。顶尖的 Apache 项目由来自世界各地的社区人员支持，应用在阿里巴巴、华为、腾讯、百度和大量其他企业。SkyWalking 提供记录、监控和追踪功能，同时也得力于其架构而拥有数据收集终端、分析平台，还有用户界面。\n值得关注的优化包括： 用户界面 Dashboard 上提供百分百的自由度，用户可以任意进行配置，采用后台新定义的指标。 支持 Prometheus 导出格式。Prometheus 格式的指标可以转换至 SkyWalking。 SkyWalking 现已可以自主监控服务网格，为 Istio 和 Envoy 提供指标。 服务、实例、终端地址的注册机制，和库存存储实体已经被移除了。 无须修改原始码的前提下，为用户界面加入新的指标 对于 SkyWalking 的用户，8.0 版本的亮点将会是数据模型的更新，而且传播格式也针对更多语言进行优化。再加上引进了新的 MeterSystem ，除了可以同步运行传统追踪模式，用户还可自定义需要收集的指标。追踪和服务网格专注在拓扑和服务流量的指标上，而 MeterSystem 则汇报用户感兴趣的业务指标，例如是数据库存取性能、圣诞节期间的下单率，或者用户注册或下单的百分比。这些指标数据会在 SkyWalking 的用户界面 Dashboard 上以图像显示。指标的面板数据和拓扑图可以通过 Envoy 的指标绘制，而追踪分析也可以支持 Istio 的遥测。Dashboard 还支持以 JSON 格式导入、导出，而 Dashboard 上的自定义指标也支持设定指标名称、实体种类（服务、实例、终端地址或全部）、标记值等。用户界面模板上已详细描述了用户界面的逻辑和原型配置，以及它的 Dashboard、tab 和组件。\n观察任何配备了 Prometheus 的应用 在这次最新的社区发布中，SkyWalking 可以观察任何配备了 Prometheus 或者提供了 Prometheus 终端地址的应用。这项更新为很多想采用 SkyWalking 指标和追踪的用户节省了不少时间，现在你不再需要重新设置指标工具，就可以获得 Prometheus 数据。因为 Prometheus 更简单、更为人熟悉，是不少用户的不二选择。有了 8.0 版本，Prometheus 网络协议就能够读取所有已设定在 API 上的数据，另外 Prometheus 格式的指标也可转换至 SkyWalking 上。如此一来，通过图像方式展示，所有的指标和拓扑都能一目了然。同时，也支持 Prometheus 的 fetcher。\n监控你的网格 SkyWalking 现在不再只是监控服务或平台，而是监控整个网格。有了 8.0 版本，你除了能获取关于你的网格的指标（包括 Istio 和 Envoy 在内），同时也能通过 SkyWalking 监控自身的性能。因为当监控服务在观察业务集群的同时，它也能实现自我观察，确保运维团队拥有稳定可靠的平台。\n性能优化 最后，8.0 发布移除了注册机制，也不再需要使用独一无二的整数来代表实体。这项改变将大幅优化性能。想了解完整的更新功能列表，可以阅读在 SkyWalking 社区发布的公告页面。\n额外资源 追踪 Twitter 获取更多 SkyWalking 最新资讯 SkyWalking 未来的发布会加入原生指标 API 和融合 Micrometer (Sleuth) 指标集合。 ","excerpt":"\u003cp\u003e可观察性平台和开源应用程序性能监控（APM）项目 Apache SkyWalking，今天刚宣布 8.0 的发布版本。素以强劲指标、追踪与服务网格能力见称的 SkyWalking ，在最新版本中的功能 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/whats-new-in-skywalking-metersystem-and-mesh-monitoring-in-8-0/","title":"SkyWalking 的最新动向？8.0 版本的 MeterSystem 和网格监控"},{"body":"作者：宋净超、张伟\n日前，云原生网络代理 MOSN v0.12.0 发布，观察性分析平台和应用性能管理系统 SkyWalking 完成了与 MOSN 的集成，作为 MOSN 中的支持的分布式追踪系统之一，旨在实现在微服务和 Service Mesh 中的更强大的可观察性。\n背景 相比传统的巨石（Monolith）应用，微服务的一个主要变化是将应用中的不同模块拆分为了独立的进程。在微服务架构下，原来进程内的方法调用成为了跨进程的远程方法调用。相对于单一进程内的方法调用而言，跨进程调用的调试和故障分析是非常困难的，难以使用传统的代码调试程序或者日志打印来对分布式的调用过程进行查看和分析。\n如上图右边所示，微服务架构中系统中各个微服务之间存在复杂的调用关系。\n一个来自客户端的请求在其业务处理过程中经过了多个微服务进程。我们如果想要对该请求的端到端调用过程进行完整的分析，则必须将该请求经过的所有进程的相关信息都收集起来并关联在一起，这就是“分布式追踪”。\n以上关于分布式追踪的介绍引用自 Istio Handbook。\nMOSN 中 tracing 的架构 MOSN 的 tracing 框架由 Driver、Tracer 和 Span 三个部分组成。\nDriver 是 Tracer 的容器，管理注册的 Tracer 实例，Tracer 是 tracing 的入口，根据请求信息创建一个 Span，Span 存储当前跨度的链路信息。\n目前 MOSN tracing 有 SOFATracer 和 SkyWalking 两种实现。SOFATracer 支持 http1 和 xprotocol 协议的链路追踪，将 trace 数据写入本地日志文件中。SkyWalking 支持 http1 协议的链路追踪，使用原生的 Go 语言探针 go2sky 将 trace 数据通过 gRPC 上报到 SkyWalking 后端服务。\n快速开始 下面将使用 Docker 和 docker-compose 来快速开始运行一个集成了 SkyWalking 的分布式追踪示例，该示例代码请见 MOSN GitHub。\n准备 安装 docker 和 docker-compose。\n安装 docker\n安装 docker-compose\n需要一个编译好的 MOSN 程序，您可以下载 MOSN 源码自行编译，或者直接下载 MOSN v0.12.0 发行版以获取 MOSN 的运行时二进制文件。\n下面将以源码编译的方式演示 MOSN 如何与 SkyWalking 集成。\ncd ${projectpath}/cmd/mosn/main go build 获取示例代码目录。\n${targetpath} = ${projectpath}/examples/codes/trace/skywalking/http/ 将编译好的程序移动到示例代码目录。\nmv main ${targetpath}/ cd ${targetpath} 目录结构 下面是 SkyWalking 的目录结构。\n* skywalking └─── http │ main # 编译完成的 MOSN 程序 | server.go # 模拟的 Http Server | clint.go # 模拟的 Http Client | config.json # MOSN 配置 | skywalking-docker-compose.yaml # skywalking docker-compose 运行说明 启动 SkyWalking oap \u0026amp; ui。\ndocker-compose -f skywalking-docker-compose.yaml up -d 启动一个 HTTP Server。\ngo run server.go 启动 MOSN。\n./main start -c config.json 启动一个 HTTP Client。\ngo run client.go 打开 http://127.0.0.1:8080 查看 SkyWalking-UI，SkyWalking Dashboard 界面如下图所示。\n在打开 Dashboard 后请点击右上角的 Auto 按钮以使页面自动刷新。\nDemo 视频 下面来看一下该 Demo 的操作视频。\n清理 要想销毁 SkyWalking 后台运行的 docker 容器只需要下面的命令。\ncd ${projectpath}/examples/codes/trace/skywalking/http/ docker-compose -f skywalking-docker-compose.yaml down 未来计划 在今年五月份，SkyWalking 8.0 版本会进行一次全面升级，采用新的探针协议和分析逻辑，探针将更具互感知能力，更好的在 Service Mesh 下使用探针进行监控。同时，SkyWalking 将开放之前仅存在于内核中的 metrics 指标分析体系。Prmoetheus、Spring Cloud Sleuth、Zabbix 等常用的 metrics 监控方式，都会被统一的接入进来，进行分析。此外， SkyWalking 与 MOSN 社区将继续合作：支持追踪 Dubbo 和 SOFARPC，同时适配 sidecar 模式下的链路追踪。\n关于 MOSN MOSN 是一款使用 Go 语言开发的网络代理软件，由蚂蚁金服开源并经过几十万容器的生产级验证。 MOSN 作为云原生的网络数据平面，旨在为服务提供多协议、模块化、智能化、安全的代理能力。 MOSN 是 Modular Open Smart Network 的简称。 MOSN 可以与任何支持 xDS API 的 Service Mesh 集成，亦可以作为独立的四、七层负载均衡，API Gateway、云原生 Ingress 等使用。\nGitHub：https://github.com/mosn/mosn 官网：https://mosn.io 关于 Skywalking SkyWalking 是观察性分析平台和应用性能管理系统。提供分布式追踪、服务网格遥测分析、度量聚合和可视化一体化解决方案。支持 Java、.Net Core、PHP、NodeJS、Golang、LUA 语言探针，支持 Envoy/MOSN + Istio 构建的 Service Mesh。\nGitHub：https://github.com/apache/skywalking 官网：https://skywalking.apache.org 关于本文中的示例请参考 MOSN GitHub 和 MOSN 官方文档。\n","excerpt":"\u003cp\u003e作者：\u003ca href=\"https://jimmysong.io\"\u003e宋净超\u003c/a\u003e、\u003ca href=\"https://github.com/arugal\"\u003e张伟\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e日前，云原生网络代理 MOSN v0.12.0 发布，观察性分析平台和应用性能管理系统 SkyWalking 完成了与 MOSN 的集成，作为 MOSN 中的支持的分布式追踪系统之 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2020-04-28-skywalking-and-mosn/","title":"SkyWalking 支持云原生网络代理 MOSN 做分布式追踪"},{"body":"Based on his continuous contributions, Wei Zhang (a.k.a arugal) has been invited to join the PMC. Welcome aboard.\n","excerpt":"\u003cp\u003eBased on his continuous contributions, Wei Zhang (a.k.a \u003ca href=\"https://github.com/arugal\"\u003earugal\u003c/a\u003e) has been invited to join the PMC. …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-wei-zhang-to-join-the-pmc/","title":"Welcome Wei Zhang to join the PMC"},{"body":"目录：\n1. 概述 2. 搭建 SkyWalking 单机环境 3. 搭建 SkyWalking 集群环境 4. 告警 5. 注意事项 6. Spring Boot 使用示例 6. Spring Cloud 使用示例 作者：芋道源码 原文地址 1. 概述 1.1 概念 SkyWalking 是什么？\nFROM http://skywalking.apache.org/\n分布式系统的应用程序性能监视工具，专为微服务、云原生架构和基于容器（Docker、K8s、Mesos）架构而设计。\n提供分布式追踪、服务网格遥测分析、度量聚合和可视化一体化解决方案。\n1.2 功能列表 SkyWalking 有哪些功能？\nFROM http://skywalking.apache.org/\n多种监控手段。可以通过语言探针和 service mesh 获得监控是数据。 多个语言自动探针。包括 Java，.NET Core 和 Node.JS。 轻量高效。无需大数据平台，和大量的服务器资源。 模块化。UI、存储、集群管理都有多种机制可选。 支持告警。 优秀的可视化解决方案。 1.3 整体架构 SkyWalking 整体架构如何？\nFROM http://skywalking.apache.org/\n整个架构，分成上、下、左、右四部分：\n考虑到让描述更简单，我们舍弃掉 Metric 指标相关，而着重在 Tracing 链路相关功能。\n上部分 Agent ：负责从应用中，收集链路信息，发送给 SkyWalking OAP 服务器。目前支持 SkyWalking、Zikpin、Jaeger 等提供的 Tracing 数据信息。而我们目前采用的是，SkyWalking Agent 收集 SkyWalking Tracing 数据，传递给服务器。 下部分 SkyWalking OAP ：负责接收 Agent 发送的 Tracing 数据信息，然后进行分析(Analysis Core) ，存储到外部存储器( Storage )，最终提供查询( Query )功能。 右部分 Storage ：Tracing 数据存储。目前支持 ES、MySQL、Sharding Sphere、TiDB、H2 多种存储器。而我们目前采用的是 ES ，主要考虑是 SkyWalking 开发团队自己的生产环境采用 ES 为主。 左部分 SkyWalking UI ：负责提供控台，查看链路等等。 1.4 官方文档 在 https://github.com/apache/skywalking/tree/master/docs 地址下，提供了 SkyWalking 的英文文档。\n考虑到大多数胖友的英语水平和艿艿不相伯仲，再加上胖友一开始对 SkyWalking 比较陌生，所以比较推荐先阅读 https://github.com/SkyAPM/document-cn-translation-of-skywalking 地址，提供了 SkyWalking 的中文文档。\n考虑到胖友使用 SkyWalking 的目的，是实现分布式链路追踪的功能，所以最好去了解下相关的知识。这里推荐阅读两篇文章：\n《OpenTracing 官方标准 —— 中文版》 Google 论文 《Dapper，大规模分布式系统的跟踪系统》 2. 搭建 SkyWalking 单机环境 考虑到让胖友更快的入门，我们来搭建一个 SkyWalking 单机环境，步骤如下：\n第一步，搭建一个 Elasticsearch 服务。 第二步，下载 SkyWalking 软件包。 第三步，搭建一个 SkyWalking OAP 服务。 第四步，启动一个 Spring Boot 应用，并配置 SkyWalking Agent。 第五步，搭建一个 SkyWalking UI 服务。 仅仅五步，按照艿艿标题党的性格，应该给本文取个《10 分钟快速搭建 SkyWalking 服务》标题才对，哈哈哈。\n2.1 Elasticsearch 搭建 FROM https://www.elastic.co/cn/products/elasticsearch\nElasticsearch 是一个分布式、RESTful 风格的搜索和数据分析引擎，能够解决不断涌现出的各种用例。 作为 Elastic Stack 的核心，它集中存储您的数据，帮助您发现意料之中以及意料之外的情况。\n参考《Elasticsearch 极简入门》的「1. 单机部署」小节，搭建一个 Elasticsearch 单机服务。\n不过要注意，本文使用的是 Elasticsearch 7.5.1 版本。因为 SkyWalking 6.6.0 版本，增加了对 Elasticsearch 7.X 版本的支持。当然，如果胖友使用 Elasticsearch 6.X 版本也是可以的。\n2.2 下载 SkyWalking 软件包 对于 SkyWalking 的软件包，有两种方式获取：\n手动编译 官方包 一般情况下，我们建议使用官方包。手动编译，更多是尝鲜或者等着急修复的 BUG 的版本。\n2.2.1 官方包 在 http://skywalking.apache.org/downloads/ 下，我们下载操作系统对应的发布版。\n这里，我们选择 Binary Distribution for ElasticSearch 7 (Linux) 版本，因为艿艿是 Mac 环境，再加上想使用 Elasticsearch 7.X 版本作为存储。如果胖友想用 Elasticsearch 6.X 版本作为存储，记得下载 Binary Distribution (Linux) 版本。\n① 下载：\n# 创建目录 $ mkdir -p /Users/yunai/skywalking $ cd /Users/yunai/skywalking # 下载 $ wget http://mirror.bit.edu.cn/apache/skywalking/6.6.0/apache-skywalking-apm-es7-6.6.0.tar.gz ② 解压：\n# 解压 $ tar -zxvf apache-skywalking-apm-es7-6.6.0.tar.gz $ cd apache-skywalking-apm-bin-es7 $ ls -ls 4 drwxr-xr-x 8 root root 4096 Sep 9 15:09 agent # SkyWalking Agent 4 drwxr-xr-x 2 root root 4096 Sep 9 15:44 bin # 执行脚本 4 drwxr-xr-x 2 root root 4096 Sep 9 15:44 config # SkyWalking OAP Server 配置文件 32 -rwxr-xr-x 1 root root 28903 Sep 9 14:32 LICENSE 4 drwxr-xr-x 3 root root 4096 Sep 9 15:44 licenses 32 -rwxr-xr-x 1 root root 31850 Sep 9 14:32 NOTICE 16 drwxr-xr-x 2 root root 16384 Sep 9 15:22 oap-libs # SkyWalking OAP Server 4 -rw-r--r-- 1 root root 1978 Sep 9 14:32 README.txt 4 drwxr-xr-x 2 root root 4096 Sep 9 15:44 webapp # SkyWalking UI 2.2.2 手动编译 友情提示：如果胖友没有编译 SkyWalking 源码的诉求，可以跳过本小节。\n参考 How to build project 文章。\n需要前置安装如下：\nGIT JDK 8+ Maven ① 克隆代码：\n$ git clone https://github.com/apache/skywalking.git 因为网络问题，可能克隆会有点久。 ② 初始化子模块：\n$ cd skywalking $ git submodule init $ git submodule update ③ 编译\n$ ./mvnw clean package -DskipTests 编译过程，如果机子比较差，花费时间会比较久。 ④ 查看编译结果\n$ cd apm-dist # 编译结果目录 $ cd target $ tar -zxvf apache-skywalking-apm-bin.tar.gz # 解压 Linux 包 $ cd apache-skywalking-apm-bin $ ls -ls 4 drwxr-xr-x 8 root root 4096 Sep 9 15:09 agent # SkyWalking Agent 4 drwxr-xr-x 2 root root 4096 Sep 9 15:44 bin # 执行脚本 4 drwxr-xr-x 2 root root 4096 Sep 9 15:44 config # SkyWalking OAP Server 配置文件 32 -rwxr-xr-x 1 root root 28903 Sep 9 14:32 LICENSE 4 drwxr-xr-x 3 root root 4096 Sep 9 15:44 licenses 32 -rwxr-xr-x 1 root root 31850 Sep 9 14:32 NOTICE 16 drwxr-xr-x 2 root root 16384 Sep 9 15:22 oap-libs # SkyWalking OAP Server 4 -rw-r--r-- 1 root root 1978 Sep 9 14:32 README.txt 4 drwxr-xr-x 2 root root 4096 Sep 9 15:44 webapp # SkyWalking UI 2.3 SkyWalking OAP 搭建 ① 修改 OAP 配置文件\n友情提示：如果配置文件，适合 SkyWalking 6.X 版本。\n$ vi config/application.yml storage: elasticsearch7: nameSpace: ${SW_NAMESPACE:\u0026#34;elasticsearch\u0026#34;} clusterNodes: ${SW_STORAGE_ES_CLUSTER_NODES:localhost:9200} protocol: ${SW_STORAGE_ES_HTTP_PROTOCOL:\u0026#34;http\u0026#34;} # trustStorePath: ${SW_SW_STORAGE_ES_SSL_JKS_PATH:\u0026#34;../es_keystore.jks\u0026#34;} # trustStorePass: ${SW_SW_STORAGE_ES_SSL_JKS_PASS:\u0026#34;\u0026#34;} user: ${SW_ES_USER:\u0026#34;\u0026#34;} password: ${SW_ES_PASSWORD:\u0026#34;\u0026#34;} indexShardsNumber: ${SW_STORAGE_ES_INDEX_SHARDS_NUMBER:2} indexReplicasNumber: ${SW_STORAGE_ES_INDEX_REPLICAS_NUMBER:0} # Those data TTL settings will override the same settings in core module. recordDataTTL: ${SW_STORAGE_ES_RECORD_DATA_TTL:7} # Unit is day otherMetricsDataTTL: ${SW_STORAGE_ES_OTHER_METRIC_DATA_TTL:45} # Unit is day monthMetricsDataTTL: ${SW_STORAGE_ES_MONTH_METRIC_DATA_TTL:18} # Unit is month # Batch process setting, refer to https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.5/java-docs-bulk-processor.html bulkActions: ${SW_STORAGE_ES_BULK_ACTIONS:1000} # Execute the bulk every 1000 requests flushInterval: ${SW_STORAGE_ES_FLUSH_INTERVAL:10} # flush the bulk every 10 seconds whatever the number of requests concurrentRequests: ${SW_STORAGE_ES_CONCURRENT_REQUESTS:2} # the number of concurrent requests resultWindowMaxSize: ${SW_STORAGE_ES_QUERY_MAX_WINDOW_SIZE:10000} metadataQueryMaxSize: ${SW_STORAGE_ES_QUERY_MAX_SIZE:5000} segmentQueryMaxSize: ${SW_STORAGE_ES_QUERY_SEGMENT_SIZE:200} # h2: # driver: ${SW_STORAGE_H2_DRIVER:org.h2.jdbcx.JdbcDataSource} # url: ${SW_STORAGE_H2_URL:jdbc:h2:mem:skywalking-oap-db} # user: ${SW_STORAGE_H2_USER:sa} # metadataQueryMaxSize: ${SW_STORAGE_H2_QUERY_MAX_SIZE:5000} storage.elasticsearch7 配置项，设置使用 Elasticsearch 7.X 版本作为存储器。 这里，我们打开注释，并记得通过 nameSpace 设置 Elasticsearch 集群名。 storage.elasticsearch 配置项，设置使用 Elasticsearch 6.X 版本作为存储器。 这里，我们无需做任何改动。 如果胖友使用 Elasticsearch 6.X 版本作为存储器，记得设置这个配置项，而不是 storage.elasticsearch7 配置项。 storage.h2 配置项，设置使用 H2 作为存储器。 这里，我们需要手动注释掉，因为 H2 是默认配置的存储器。 友情提示：如果配置文件，适合 SkyWalking 7.X 版本。\n重点修改 storage 配置项，通过 storage.selector 配置项来设置具体使用的存储器。 storage.elasticsearch 配置项，设置使用 Elasticsearch 6.X 版本作为存储器。胖友可以主要修改 nameSpace、clusterNodes 两个配置项即可，设置使用的 Elasticsearch 的集群和命名空间。 storage.elasticsearch7 配置项，设置使用 Elasticsearch 7.X 版本作为存储器。 还有 MySQL、H2、InfluxDB 等等存储器的配置可以选择，胖友自己根据需要去选择哈~ ② 启动 SkyWalking OAP 服务\n$ bin/oapService.sh SkyWalking OAP started successfully! 是否真正启动成功，胖友打开 logs/skywalking-oap-server.log 日志文件，查看是否有错误日志。首次启动时，因为 SkyWalking OAP 会创建 Elasticsearch 的索引，所以会“疯狂”的打印日志。最终，我们看到如下日志，基本可以代表 SkyWalking OAP 服务启动成功：\n友情提示：因为首次启动会创建 Elasticsearch 索引，所以可能会比较慢。\n2020-01-02 18:22:53,635 - org.eclipse.jetty.server.Server - 444 [main] INFO [] - Started @35249ms 2.4 SkyWalking UI 搭建 ① 启动 SkyWalking UI 服务\nbin/webappService.sh SkyWalking Web Application started successfully! 是否真正启动成功，胖友打开 logs/logs/webapp.log 日志文件，查看是否有错误日志。最终，我们看到如下日志，基本可以代表 SkyWalking UI 服务启动成功：\n2020-01-02 18:27:02.824 INFO 48250 --- [main] o.a.s.apm.webapp.ApplicationStartUp : Started ApplicationStartUp in 7.774 seconds (JVM running for 8.316) 如果想要修改 SkyWalking UI 服务的参数，可以编辑 webapp/webapp.yml 配置文件。例如说：\nserver.port ：SkyWalking UI 服务端口。 collector.ribbon.listOfServers ：SkyWalking OAP 服务地址数组。因为 SkyWalking UI 界面的数据，是通过请求 SkyWalking OAP 服务来获得的。 ② 访问 UI 界面：\n浏览器打开 http://127.0.0.1:8080 。界面如下图：\n2.5 SkyWalking Agent 大多数情况下，我们在启动项目的 Shell 脚本上，通过 -javaagent 参数进行配置 SkyWalking Agent 。我们在 「2.3.1 Shell」 小节来看。\n考虑到偶尔我们需要在 IDE 中，也希望使用 SkyWalking Agent ，所以我们在 「2.3.2 IDEA」 小节来看。\n2.3.1 Shell ① Agent 软件包\n我们需要将 apache-skywalking-apm-bin/agent 目录，拷贝到 Java 应用所在的服务器上。这样，Java 应用才可以配置使用该 SkyWalking Agent。我们来看看 Agent 目录下有哪些：\n$ ls -ls total 35176 0 drwxr-xr-x@ 7 yunai staff 224 Dec 24 14:20 activations 0 drwxr-xr-x@ 4 yunai staff 128 Dec 24 14:21 bootstrap-plugins 0 drwxr-xr-x@ 3 yunai staff 96 Dec 24 14:12 config # SkyWalking Agent 配置 0 drwxr-xr-x@ 3 yunai staff 96 Jan 2 19:29 logs # SkyWalking Agent 日志 0 drwxr-xr-x@ 13 yunai staff 416 Dec 24 14:22 optional-plugins # 可选插件 0 drwxr-xr-x@ 68 yunai staff 2176 Dec 24 14:20 plugins # 插件 35176 -rw-r--r--@ 1 yunai staff 18006420 Dec 24 14:12 skywalking-agent.jar # SkyWalking Agent 关于 SkyWalking Agent 提供的插件列表，可以看看《SkyWalking 文档 —— 插件支持列表》。 因为艿艿是在本机测试，所以无需拷贝，SkyWalking Agent 目录是 /Users/yunai/skywalking/apache-skywalking-apm-bin-es7/agent/。\n考虑到方便胖友，艿艿这里提供了一个最简的 Spring Boot 应用 lab-39-demo-2.2.2.RELEASE.jar。对应 Github 仓库是 lab-39-demo。\n② 配置 Java 启动脚本\n# SkyWalking Agent 配置 export SW_AGENT_NAME=demo-application # 配置 Agent 名字。一般来说，我们直接使用 Spring Boot 项目的 `spring.application.name` 。 export SW_AGENT_COLLECTOR_BACKEND_SERVICES=127.0.0.1:11800 # 配置 Collector 地址。 export SW_AGENT_SPAN_LIMIT=2000 # 配置链路的最大 Span 数量。一般情况下，不需要配置，默认为 300 。主要考虑，有些新上 SkyWalking Agent 的项目，代码可能比较糟糕。 export JAVA_AGENT=-javaagent:/Users/yunai/skywalking/apache-skywalking-apm-bin-es7/agent/skywalking-agent.jar # SkyWalking Agent jar 地址。 # Jar 启动 java -jar $JAVA_AGENT -jar lab-39-demo-2.2.2.RELEASE.jar 通过环境变量，进行配置。 更多的变量，可以在 /work/programs/skywalking/apache-skywalking-apm-bin/agent/config/agent.config 查看。要注意，可能有些变量是被注释掉的，例如说 SW_AGENT_SPAN_LIMIT 对应的 agent.span_limit_per_segment 。 ③ 执行脚本：\n直接执行上述的 Shell 脚本，启动 Java 项目。在启动日志中，我们可以看到 SkyWalking Agent 被加载的日志。日志示例如下：\nDEBUG 2020-01-02 19:29:29:400 main AgentPackagePath : The beacon class location is jar:file:/Users/yunai/skywalking/apache-skywalking-apm-bin-es7/agent/skywalking-agent.jar!/org/apache/skywalking/apm/agent/core/boot/AgentPackagePath.class. INFO 2020-01-02 19:29:29:402 main SnifferConfigInitializer : Config file found in /Users/yunai/skywalking/apache-skywalking-apm-bin-es7/agent/config/agent.config. 同时，也可以在 /Users/yunai/skywalking/apache-skywalking-apm-bin-es7/agent/agent/logs/skywalking-api.log 查看对应的 SkyWalking Agent 日志。日志示例如下：\nDEBUG 2020-01-02 19:37:22:539 SkywalkingAgent-5-ServiceAndEndpointRegisterClient-0 ServiceAndEndpointRegisterClient : ServiceAndEndpointRegisterClient running, status:CONNECTED. 这里，我们看到 status:CONNECTED ，表示 SkyWalking Agent 连接 SkyWalking OAP 服务成功。 ④ 简单测试\n完事，可以去 SkyWalking UI 查看是否链路收集成功。\n1、首先，使用浏览器，访问下 http://127.0.0.1:8079/demo/echo 地址，请求下 Spring Boot 应用提供的 API。因为，我们要追踪下该链路。\n2、然后，继续使用浏览器，打开 http://127.0.0.1:8080/ 地址，进入 SkyWalking UI 界面。如下图所示：\n这里，我们会看到 SkyWalking 中非常重要的三个概念：\n服务(Service) ：表示对请求提供相同行为的一系列或一组工作负载。在使用 Agent 或 SDK 的时候，你可以定义服务的名字。如果不定义的话，SkyWalking 将会使用你在平台（例如说 Istio）上定义的名字。\n这里，我们可以看到 Spring Boot 应用的服务为 \u0026quot;demo-application\u0026quot;，就是我们在环境变量 SW_AGENT_NAME 中所定义的。\n服务实例(Service Instance) ：上述的一组工作负载中的每一个工作负载称为一个实例。就像 Kubernetes 中的 pods 一样, 服务实例未必就是操作系统上的一个进程。但当你在使用 Agent 的时候, 一个服务实例实际就是操作系统上的一个真实进程。\n这里，我们可以看到 Spring Boot 应用的服务为 {agent_name}-pid:{pid}@{hostname}，由 Agent 自动生成。关于它，我们在「5.1 hostname」小节中，有进一步的讲解，胖友可以瞅瞅。\n端点(Endpoint) ：对于特定服务所接收的请求路径, 如 HTTP 的 URI 路径和 gRPC 服务的类名 + 方法签名。\n这里，我们可以看到 Spring Boot 应用的一个端点，为 API 接口 /demo/echo。\n3、之后，点击「拓扑图」菜单，进入查看拓扑图的界面。如下图所示：\n4、再之后，点击「追踪」菜单，进入查看链路数据的界面。如下图所示：\n2.3.2 IDEA 我们统一使用 IDEA 作为开发 IDE ，所以忽略 Eclipse 的配置方式。\n具体参考下图，比较简单：\n3. 搭建 SkyWalking 集群环境 在生产环境下，我们一般推荐搭建 SkyWalking 集群环境。😈 当然，如果公司比较抠门，也可以在生产环境下使用 SkyWalking 单机环境，毕竟 SkyWalking 挂了之后，不影响业务的正常运行。\n搭建一个 SkyWalking 集群环境，步骤如下：\n第一步，搭建一个 Elasticsearch 服务的集群。 第二步，搭建一个注册中心的集群。目前 SkyWalking 支持 Zookeeper、Kubernetes、Consul、Nacos 作为注册中心。 第三步，搭建一个 SkyWalking OAP 服务的集群，同时参考《SkyWalking 文档 —— 集群管理》，将 SkyWalking OAP 服务注册到注册中心上。 第四步，启动一个 Spring Boot 应用，并配置 SkyWalking Agent。另外，在设置 SkyWaling Agent 的 SW_AGENT_COLLECTOR_BACKEND_SERVICES 地址时，需要设置多个 SkyWalking OAP 服务的地址数组。 第五步，搭建一个 SkyWalking UI 服务的集群，同时使用 Nginx 进行负载均衡。另外，在设置 SkyWalking UI 的 collector.ribbon.listOfServers 地址时，也需要设置多个 SkyWalking OAP 服务的地址数组。 😈 具体的搭建过程，并不复杂，胖友自己去尝试下。\n4. 告警 在 SkyWaling 中，已经提供了告警功能，具体可见《SkyWalking 文档 —— 告警》。\n默认情况下，SkyWalking 已经内置告警规则。同时，我们可以参考告警规则，进行自定义。\n在满足 SkyWalking 告警规则的触发规则时，我们在 SkyWaling UI 的告警界面，可以看到告警内容。如下图所示：\n同时，我们自定义 Webhook ，对接 SkyWalking 的告警请求。而具体的邮箱、钉钉等告警方式，需要自己进行开发。至于自定义 WebHook 如何实现，可以参考：\nJava 语言： 《基于 SkyWalking 的分布式跟踪系统 - 异常告警》 Go 语言： dingding-notify-for-skywalking infra-skywalking-webhook 5. 注意事项 5.1 hostname 配置 在 SkyWalking 中，每个被监控的实例的名字，会包含 hostname 。格式为：{agent_name}-pid:{pid}@{hostname} ，例如说：\u0026quot;scrm-scheduler-pid:27629@iZbp1e2xlyvr7fh67qi59oZ\u0026quot; 。\n因为有些服务器未正确设置 hostname ，所以我们一定要去修改，不然都不知道是哪个服务器上的实例（😈 鬼知道 \u0026quot;iZbp1e2xlyvr7fh67qi59oZ\u0026quot; 一串是哪个服务器啊）。\n修改方式如下：\n1、修改 /etc/hosts 的 hostname ：\n127.0.0.1 localhost ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.80.62.151 pre-app-01 # 就是这个，其中 10.80.62.151 是本机内网 IP ，pre-app-01 是 hostname 。 2、修改本机 hostname ：\n参考 《CentOS7 修改主机名（hostname）》\n$ hostname pre-app-01 # 其中 pre-app-01 就是你希望的 hostname 。 $ hostnamectl set-hostname pre-app-01 # 其中 pre-app-01 就是你希望的 hostname 。 6. Spring Boot 使用示例 在 《芋道 Spring Boot 链路追踪 SkyWalking 入门》 中，我们来详细学习如何在 Spring Boot 中，整合并使用 SkyWalking 收集链路数据。😈 相比「2.5 SkyWaling Agent」来说，我们会提供更加丰富的示例哟。\n7. Spring Cloud 使用示例 在 《芋道 Spring Cloud 链路追踪 SkyWalking 入门》 中，我们来详细学习如何在 Spring Cloud 中，整合并使用 SkyWalking 收集链路数据。😈 相比「2.5 SkyWaling Agent」来说，我们会提供更加丰富的示例哟。\n666. 彩蛋 本文仅仅是简单的 SkyWalking 入门文章，如果胖友想要更好的使用 SkyWalking，推荐通读下《SkyWalking 文档》。\n想要进一步深入的胖友，也可以阅读如下资料：\n《SkyWalking 源码解析》 《APM 巅峰对决：Apache Skywalking P.K. Pinpoint》 《SkyWalking 官方 —— 博客合集》 😈 最后弱弱的问一句，上完 SkyWaling 之后，有没发现自己系统各种地方慢慢慢！嘻嘻。\n","excerpt":"\u003cp\u003e目录：\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#\"\u003e1. 概述\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#\"\u003e2. 搭建 SkyWalking 单机环境\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#\"\u003e3. 搭建 SkyWalking 集群环境\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#\"\u003e4. 告警\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#\"\u003e5. 注意事项\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#\"\u003e6. Spring Boot 使用示例\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#\"\u003e6. Spring …\u003c/a\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/zh/2020-04-19-skywalking-quick-start/","title":"SkyWalking 极简入门"},{"body":"This post originally appears on The New Stack\nThis post introduces a way to automatically profile code in production with Apache SkyWalking. We believe the profile method helps reduce maintenance and overhead while increasing the precision in root cause analysis.\nLimitations of the Distributed Tracing In the early days, metrics and logging systems were the key solutions in monitoring platforms. With the adoption of microservice and distributed system-based architecture, distributed tracing has become more important. Distributed tracing provides relevant service context, such as system topology map and RPC parent-child relationships.\nSome claim that distributed tracing is the best way to discover the cause of performance issues in a distributed system. It’s good at finding issues at the RPC abstraction, or in the scope of components instrumented with spans. However, it isn’t that perfect.\nHave you been surprised to find a span duration longer than expected, but no insight into why? What should you do next? Some may think that the next step is to add more instrumentation, more spans into the trace, thinking that you would eventually find the root cause, with more data points. We’ll argue this is not a good option within a production environment. Here’s why:\nThere is a risk of application overhead and system overload. Ad-hoc spans measure the performance of specific scopes or methods, but picking the right place can be difficult. To identify the precise cause, you can “instrument” (add spans to) many suspicious places. The additional instrumentation costs more CPU and memory in the production environment. Next, ad-hoc instrumentation that didn’t help is often forgotten, not deleted. This creates a valueless overhead load. In the worst case, excess instrumentation can cause performance problems in the production app or overload the tracing system. The process of ad-hoc (manual) instrumentation usually implies at least a restart. Trace instrumentation libraries, like Zipkin Brave, are integrated into many framework libraries. To instrument a method’s performance typically implies changing code, even if only an annotation. This implies a re-deploy. Even if you have the way to do auto instrumentation, like Apache SkyWalking, you still need to change the configuration and reboot the app. Otherwise, you take the risk of GC caused by hot dynamic instrumentation. Injecting instrumentation into an uninstrumented third party library is hard and complex. It takes more time and many won’t know how to do this. Usually, we don’t have code line numbers in the distributed tracing. Particularly when lambdas are in use, it can be difficult to identify the line of code associated with a span. Regardless of the above choices, to dive deeper requires collaboration with your Ops or SRE team, and a shared deep level of knowledge in distributed tracing. Regardless of the above choices, to dive deeper requires collaboration with your Ops or SRE team, and a shared deep level of knowledge in distributed tracing.\nProfiling in Production Introduction To reuse distributed tracing to achieve method scope precision requires an understanding of the above limitations and a different approach. We called it PROFILE.\nMost high-level languages build and run on a thread concept. The profile approach takes continuous thread dumps. We merge the thread dumps to estimate the execution time of every method shown in the thread dumps. The key for distributed tracing is the tracing context, identifiers active (or current) for the profiled method. Using this trace context, we can weave data harvested from profiling into existing traces. This allows the system to automate otherwise ad-hoc instrumentation. Let’s dig deeper into how profiling works:\nWe consider a method invocation with the same stack depth and signature (method, line number etc), the same operation. We derive span timestamps from the thread dumps the same operation is in. Let’s put this visually:\nAbove, represents 10 successive thread dumps. If this method is in dumps 4-8, we assume it started before dump 4 and finished after dump 8. We can’t tell exactly when the method started and stopped. but the timestamps of thread dumps are close enough.\nTo reduce overhead caused by thread dumps, we only profile methods enclosed by a specific entry point, such as a URI or MVC Controller method. We identify these entry points through the trace context and the APM system.\nThe profile does thread dump analysis and gives us:\nThe root cause, precise to the line number in the code. Reduced maintenance as ad-hoc instrumentation is obviated. Reduced overload risk caused by ad-hoc instrumentation. Dynamic activation: only when necessary and with a very clear profile target. Implementing Precise Profiling with Apache SkyWalking 7 Distributed profiling is built-into Apache SkyWalking application performance monitoring (APM). Let’s demonstrate how the profiling approach locates the root cause of the performance issue.\nfinal CountDownLatchcountDownLatch= new CountDownLatch(2); threadPool.submit(new Task1(countDownLatch)); threadPool.submit(new Task2(countDownLatch)); try { countDownLatch.await(500, TimeUnit.MILLISECONDS); } catch (InterruptedExceptione) { } Task1 and Task2 have a race condition and unstable execution time: they will impact the performance of each other and anything calling them. While this code looks suspicious, it is representative of real life. People in the OPS/SRE team are not usually aware of all code changes and who did them. They only know something in the new code is causing a problem.\nTo make matters interesting, the above code is not always slow: it only happens when the condition is locked. In SkyWalking APM, we have metrics of endpoint p99/p95 latency, so, we are easy to find out the p99 of this endpoint is far from the avg response time. However, this is not the same as understanding the cause of the latency. To locate the root cause, add a profile condition to this endpoint: duration greater than 500ms. This means faster executions will not add profiling load.\nThis is a typical profiled trace segment (part of the whole distributed trace) shown on the SkyWalking UI. We now notice the “service/processWithThreadPool” span is slow as we expected, but why? This method is the one we added the faulty code to. As the UI shows that method, we know the profiler is working. Now, let’s see what the profile analysis result say.\nThis is the profile analysis stack view. We see the stack element names, duration (include/exclude the children) and slowest methods have been highlighted. It shows clearly, “sun.misc.Unsafe.park” costs the most time. If we look for the caller, it is the code we added: CountDownLatch.await.\nThe Limitations of the Profile Method No diagnostic tool can fit all cases, not even the profile method.\nThe first consideration is mistaking a repeatedly called method for a slow method. Thread dumps are periodic. If there is a loop of calling one method, the profile analysis result would say the target method is slow because it is captured every time in the dump process. There could be another reason. A method called many times can also end up captured in each thread dump. Even so, the profile did what it is designed for. It still helps the OPS/SRE team to locate the code having the issue.\nThe second consideration is overhead, the impact of repeated thread dumps is real and can’t be ignored. In SkyWalking, we set the profile dump period to at least 10ms. This means we can’t locate method performance issues if they complete in less than 10ms. SkyWalking has a threshold to control the maximum parallel degree as well.\nUnderstanding the above keeps distributed tracing and APM systems useful for your OPS/SRE team.\nHow to Try This Everything we discussed, including the Apache SkyWalking Java Agent, profile analysis code, and UI, could be found in our GitHub repository. We hope you enjoyed this new profile method, and love Apache SkyWalking. If so, give us a star on GitHub to encourage us.\nSkyWalking 7 has just been released. You can contact the project team through the following channels:\nFollow SkyWalking twitter. Subscribe mailing list: dev@skywalking.apache.org. Send to dev-subscribe@kywalking.apache.org to subscribe to the mail list. Co-author Sheng Wu is a Tetrate founding engineer and the founder and VP of Apache SkyWalking. He is solving the problem of observability for large-scale service meshes in hybrid and multi-cloud environments.\nAdrian Cole works in the Spring Cloud team at VMware, mostly on Zipkin\nHan Liu is a tech expert at Lagou. He is an Apache SkyWalking committer\n","excerpt":"\u003cp\u003e\u003cem\u003eThis post originally appears on \u003ca href=\"https://thenewstack.io/apache-skywalking-use-profiling-to-fix-the-blind-spot-of-distributed-tracing/\"\u003eThe New Stack\u003c/a\u003e\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eThis post introduces a way to automatically profile …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2020-04-13-apache-skywalking-profiling/","title":"Apache SkyWalking: Use Profiling to Fix the Blind Spot of Distributed Tracing"},{"body":"SkyWalking Chart 2.0.0 is released. Go to downloads page to find release tars.\nSupport SkyWalking 7.0.0 Support set ES user/password Add CI for release ","excerpt":"\u003cp\u003eSkyWalking Chart 2.0.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eSupport SkyWalking …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-chart-2-0-0-for-skywalking-7-0-0/","title":"Release Apache SkyWalking Chart 2.0.0 for SkyWalking 7.0.0"},{"body":"SkyWalking APM 7.0.0 is release. Go to downloads page to find release tars.\nUpgrade JDK minimal JDK requirement to JDK8 Support profiling code level performance Don\u0026rsquo;t support SkyWalking v5 agent in-wire and out-wire protocol. V6 is required. ","excerpt":"\u003cp\u003eSkyWalking APM 7.0.0 is release. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eUpgrade JDK minimal JDK …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-7-0-0/","title":"Release Apache SkyWalking APM 7.0.0"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/agent/","title":"Agent"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/java/","title":"Java"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/profiling/","title":"Profiling"},{"body":" 作者：吴晟，刘晗 原文地址 在本文中，我们详细介绍了代码级的性能剖析方法，以及我们在 Apache SkyWalking 中的实践。希望能够帮助大家在线定位系统性能短板，缓解系统压力。\n分布式链路追踪的局限性 在传统的监控系统中，我们如果想要得知系统中的业务是否正常，会采用进程监控、日志收集分析等方式来对系统进行监控。当机器或者服务出现问题时，则会触发告警及时通知负责人。通过这种方式，我们可以得知具体哪些服务出现了问题。但是这时我们并不能得知具体的错误原因出在了哪里，开发人员或者运维人员需要到日志系统里面查看错误日志，甚至需要到真实的业务服务器上查看执行情况来解决问题。\n如此一来，仅仅是发现问题的阶段，可能就会耗费相当长的时间；另外，发现问题但是并不能追溯到问题产生具体原因的情况，也常有发生。这样反反复复极其耗费时间和精力，为此我们便有了基于分布式追踪的 APM 系统。\n通过将业务系统接入分布式追踪中，我们就像是给程序增加了一个放大镜功能，可以清晰看到真实业务请求的整体链路，包括请求时间、请求路径，甚至是操作数据库的语句都可以看得一清二楚。通过这种方式，我们结合告警便可以快速追踪到真实用户请求的完整链路信息，并且这些数据信息完全是持久化的，可以随时进行查询，复盘错误的原因。\n然而随着我们对服务监控理解的加深，我们发现事情并没有那么简单。在分布式链路追踪中我们有这样的两个流派：代码埋点和字节码增强。无论使用哪种方式，底层逻辑一定都逃不过面向切面这个基础逻辑。因为只有这样才可以做到大面积的使用。这也就决定了它只能做到框架级别和 RPC 粒度的监控。这时我们可能依旧会遇到程序执行缓慢或者响应时间不稳定等情况，但无法具体查询到原因。这时候，大家很自然的会考虑到增加埋点粒度，比如对所有的 Spring Bean 方法、甚至主要的业务层方法都加上埋点。但是这种思路会遇到不小的挑战：\n第一，增加埋点时系统开销大，埋点覆盖不够全面。通过这种方式我们确实可以做到具体业务场景具体分析。但随着业务不断迭代上线，弊端也很明显：大量的埋点无疑会加大系统资源的开销，造成 CPU、内存使用率增加，更有可能拖慢整个链路的执行效率。虽然每个埋点消耗的性能很小，在微秒级别，但是因为数量的增加，甚至因为业务代码重用造成重复埋点或者循环使用，此时的性能开销已经无法忽略。\n第二，动态埋点作为一项埋点技术，和手动埋点的性能消耗上十分类似，只是减少的代码修改量，但是因为通用技术的特别，上一个挑战中提到的循环埋点和重复使用的场景甚至更为严重。比如选择所有方法或者特定包下的所有方法埋点，很可能造成系统性能彻底崩溃。\n第三，即使我们通过合理设计和埋点，解决了上述问题，但是 JDK 函数是广泛使用的，我们很难限制对 JDK API 的使用场景。对 JDK 过多方法、特别是非 RPC 方法的监控会造成系统的巨大延迟风险。而且有一些基础类型和底层工具类，是很难通过字节码进行增强的。当我们的 SDK 使用不当或者出现 bug 时，我们无法具体得知真实的错误原因。\n代码级性能剖析方法 方法介绍 基于以上问题，在系统性能监控方法上，我们提出了代码级性能剖析这种在线诊断方法。这种方法基于一个高级语言编程模型共性，即使再复杂的系统，再复杂的业务逻辑，都是基于线程去进行执行的，而且多数逻辑是在单个线程状态下执行的。\n代码级性能剖析就是利用方法栈快照，并对方法执行情况进行分析和汇总。并结合有限的分布式追踪 span 上下文，对代码执行速度进行估算。\n性能剖析激活时，会对指定线程周期性的进行线程栈快照，并将所有的快照进行汇总分析，如果两个连续的快照含有同样的方法栈，则说明此栈中的方法大概率在这个时间间隔内都处于执行状态。从而，通过这种连续快照的时间间隔累加成为估算的方法执行时间。时间估算方法如下图所示：\n在上图中，d0-d10 代表 10 次连续的内存栈快照，实际方法执行时间在 d3-d4 区间，结束时间在 d8-d9 之间。性能剖析无法告诉你方法的准确执行时间，但是他会估算出方法执行时间为 d4-d8 的 4 个快照采集间隔时间之和，这已经是非常的精确的时间估算了。\n而这个过程因为不涉及代码埋点，所以自然性能消耗是稳定和可控的，也无需担心是否被埋点，是否是 JDK 方法等问题。同时，由于上层已经在分布式追踪之下，性能剖析方法可以明确地确定分析开始和结束时间，减少不必要的性能开销。\n性能剖析可以很好的对线程的堆栈信息进行监控，主要有以下几点优势：\n精确的问题定位，直接到代码方法和代码行； 无需反复的增删埋点，大大减少了人力开发成本； 不用承担过多埋点对目标系统和监控系统的压力和性能风险； 按需使用，平时对系统无消耗，使用时的消耗稳定可能。 SkyWalking 实践实例 我们首先在 Apache SkyWalking APM 中实现此技术方法，下面我们就以一个真实的例子来说明此方法的执行效果。\nfinal CountDownLatchcountDownLatch= new CountDownLatch(2); threadPool.submit(new Task1(countDownLatch)); threadPool.submit(new Task2(countDownLatch)); try { countDownLatch.await(500, TimeUnit.MILLISECONDS); } catch (InterruptedExceptione) { } 这是我们故意加入的问题代码，我们使用 CountDownLanth 设置了两个任务完成后方法执行结束，Task1 和 Task2 是两个执行时间不稳定的任务，所以主任务也会执行速度不稳定。但对于运维和监控团队来说，很难定位到这个方法片段。\n针对于这种情况，我们看看性能剖析会怎样直接定位此问题。\n上图所示的就是我们在进行链路追踪时所看到的真实执行情况，其中我们可以看到在 service/processWithThreadPool 执行速度缓慢，这正是我们植入问题代码的方法。此时在这个调用中没有后续链路了，所以并没有更细致的原因，我们也不打算去 review 代码，从而增加新埋点。这时，我们可以对 HelloService 进行性能剖析，并执行只剖析响应速度大于 500 毫秒的请求。\n注意，指定特定响应时间的剖析是保证剖析有效性的重要特性，如果方法在平均响应时间上已经出现问题，往往通过分布式链路可以快速定位，因为此时链路总时间长，新埋点带来的性能影响相对可控。但是方法性能抖动是不容易用新增埋点来解决的，而且往往只发生在生产环境。\n上图就是我们进行性能剖析后的真实结果图。从左到右分别表示：栈帧名称、该栈帧总计耗时（包含其下面所有自栈帧）、当前栈帧自身耗时和监控次数。我们可以在最后一行看到，线程卡在了 sun.misc.Unsafe.park 中了。如果你熟悉 Java 就可以知道此时进行了锁等待，我们继续按照树的结构向上推，便可以看到线程真正是卡在了 CountDownLatch.await 方法中。\n方法局限性 当然任何的方法都不是万能的，性能剖析也有一些局限性。\n第一， 对于高频反复执行的方法，如循环调用，可能会误报为缓慢方法。但这并不是大问题，因为如果反复执行的耗时较长，必然是系统需要关注的性能瓶颈。\n第二， 由于性能栈快照有一定的性能消耗，所以采集周期不宜过密，如 SkyWalking 实践中，不支持小于 10ms 的采集间隔。所以如果问题方法执行时间过小（比如在 10 毫秒内波动），此方法并不适用。我们也再此强调，方法论和工具的强大，始终不能代替程序员。\n","excerpt":"\u003cul\u003e\n\u003cli\u003e作者：\u003ca href=\"https://github.com/wu-sheng\"\u003e吴晟\u003c/a\u003e，\u003ca href=\"https://github.com/mrproliu\"\u003e刘晗\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://www.infoq.cn/article/CWUOl1JA0EyXw0CxQ4Zm\"\u003e原文地址\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e在本文中，我们详细介绍了代码级的性能剖析方法，以及我们在 Apache SkyWalking 中的实践。希望能够帮助大家在线定位系统性能短板，缓解系统压力。\u003c/p\u003e\n\u003ch2 id=\"分布式链路追踪的局限性\"\u003e分布式链路 …\u003c/h2\u003e","ref":"https://skywalking.apache.org/zh/2020-03-23-using-profiling-to-fix-the-blind-spot-of-distributed-tracing/","title":"在线代码级性能剖析，补全分布式追踪的最后一块“短板”"},{"body":"SkyWalking CLI 0.2.0 is released. Go to downloads page to find release tars.\nSupport visualization of heat map Support top N entities, swctl metrics top 5 --name service_sla Support thermodynamic metrics, swctl metrics thermodynamic --name all_heatmap Support multiple linear metrics, swctl --display=graph --debug metrics multiple-linear --name all_percentile ","excerpt":"\u003cp\u003eSkyWalking CLI 0.2.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eSupport visualization …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cli-0-2-0/","title":"Release Apache SkyWalking CLI 0.2.0"},{"body":"SkyWalking Chart 1.1.0 is released. Go to downloads page to find release tars.\nSupport SkyWalking 6.6.0 Support deploy Elasticsearch 7 The official helm repo was changed to the official Elasticsearch repo (https://helm.elastic.co/) ","excerpt":"\u003cp\u003eSkyWalking Chart 1.1.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eSupport SkyWalking …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-chart-1-1-0-for-skywalking-6-6-0/","title":"Release Apache SkyWalking Chart 1.1.0 for SkyWalking 6.6.0"},{"body":"Support tracing and collect metrics from Nginx server. Require SkyWalking APM 7.0+.\n","excerpt":"\u003cp\u003eSupport tracing and collect metrics from Nginx server. Require SkyWalking APM 7.0+.\u003c/p\u003e","ref":"https://skywalking.apache.org/events/skywalking-nginx-lua-0-1-0-release/","title":"SkyWalking Nginx LUA 0.1.0 release"},{"body":"Based on his continuous contributions, Ming Wen (a.k.a moonming) has been voted as a new committer.\n","excerpt":"\u003cp\u003eBased on his continuous contributions, Ming Wen (a.k.a \u003ca href=\"https://github.com/moonming\"\u003emoonming\u003c/a\u003e) has been voted as a new committer.\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-ming-wen-as-new-committer/","title":"Welcome Ming Wen as new committer"},{"body":"Based on his continuous contributions, Haochao Zhuang (a.k.a dmsolr) has been invited to join the PMC. Welcome aboard.\n","excerpt":"\u003cp\u003eBased on his continuous contributions, Haochao Zhuang (a.k.a \u003ca href=\"https://github.com/dmsolr\"\u003edmsolr\u003c/a\u003e) has been invited to join the …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-haochao-zhuang-to-join-the-pmc/","title":"Welcome Haochao Zhuang to join the PMC"},{"body":"Based on his continuous contributions, Zhusheng Xu (a.k.a aderm) has been voted as a new committer.\n","excerpt":"\u003cp\u003eBased on his continuous contributions, Zhusheng Xu (a.k.a \u003ca href=\"https://github.com/aderm\"\u003eaderm\u003c/a\u003e) has been voted as a new committer.\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-zhusheng-xu-as-new-committer/","title":"Welcome Zhusheng Xu as new committer"},{"body":"Based on his continuous contributions, Han Liu (a.k.a mrproliu) has been voted as a new committer.\n","excerpt":"\u003cp\u003eBased on his continuous contributions, Han Liu (a.k.a \u003ca href=\"https://github.com/mrproliu\"\u003emrproliu\u003c/a\u003e) has been voted as a new committer.\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-han-liu-as-new-committer/","title":"Welcome Han Liu as new committer"},{"body":" Author: Wu Sheng, tetrate.io, SkyWalking original creator, SkyWalking V.P. GitHub, Twitter, Linkedin The SkyWalking project provides distributed tracing, topology map analysis, service mesh telemetry analysis, metrics analysis and a super cool visualization targeting distributed systems in k8s or traditional VM deployments.\nThe project is widely used in Alibaba, Huawei, Tencent, DiDi, xiaomi, Pingan, China’s top 3 telecom companies (China Mobile, China telecom, China Unicom), airlines, banks and more. It has over 140 company users listed on our powered by page.\nToday, we welcome and celebrate reaching 200 code contributors on our main repo. We hereby mark this milestone as official today, : Jan. 20th 2020.\nAt this great moment, I would like to share SkyWalking’s 4-year open source journey.\nI wrote the first line on Nov. 1st, 2015, guiding people to understand a distributed system just as micro-services and distributed architecture were becoming popular. In the first 2 years, I never thought it would become such a big and active community. I didn’t even expect it would be an open source project. Initially, the goal was primarily to teach others about distributed tracing and analysis.\nIt was a typical open source project in obscurity in its first two years. But people still showed up, asked questions, and tried to improve the project. I got several invitations to share the project at local meetups.All these made me realize people really needed a good open source APM project.\nIn 2017, I decided to dedicate myself as much as possible to make the project successful, and it became my day job. To be honest, I had no clue about how to do that; at that time in China, it was rare to have this kind of job. So, I began to ask friends around me, “Do you want to collaborate on the open source APM with me?” Most people were busy and gave a clear NO, but two of them agreed to help: Xin Zhang and Yongsheng Peng. We built SkyWalking 3.x and shared the 3.2 release at GOPS Shanghai, China.\nIt became the first adoption version used in production\nCompared to today\u0026rsquo;s SkyWalking, it was a toy prototype, but it had the same tracing design, protocol and analysis method.\nThat year the contributor team was 15-20, and the project had obvious potential to expand. I began to consider bringing the project into a worldwide, top-level open source foundation. Thanks to our initial incubator mentors, Michael Semb Wever, William Jiang, and Luke Han, this really worked. At the end of 2017, SkyWalking joined the Apache Incubator, and kept following the Apache Way to build community. More contributors joined the community.\nWith more people spending time on the project collaborations, including codes, tests, blogs, conference talks, books and uses of the project, a chemical reaction happens. New developers begin to provide bug fixes, new feature requirements and new proposals. At the moment of graduation in spring 2019, the project had 100 contributors. Now, only 9 months later, it’s surged to 200 super quickly. They enhance the project and extend it to frontiers we never imaged: 5 popular language agents, service mesh adoption, CLI tool, super cool visualization. We are even moving on thread profiling, browser performance and Nginx tracing NOW.\nOver the whole 4+ years open source journey, we have had supports from leaders in the tracing open source community around the world, including Adrian Cole, William Jiang, Luke Han, Michael Semb Wever, Ben Sigelman, and Jonah Kowall. And we’ve had critical foundations\u0026rsquo; help, especially Apache Software Foundation and the Cloud Native Computing Foundation.\nOur contributors also have their support from their employers, including, to the best of my knowledge, Alibaba, Huawei, China Mobile, ke.com, DaoCloud, Lizhi.fm, Yonghui Supermarket, and dangdang.com. I also have support from my employers, tetrate.io, Huawei, and OneAPM.\nThanks to our 200+ contributors and the companies behind them. You make this magic happen.\n","excerpt":"\u003cul\u003e\n\u003cli\u003eAuthor: Wu Sheng, tetrate.io, SkyWalking original creator, SkyWalking V.P.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/wu-sheng\"\u003eGitHub\u003c/a\u003e, \u003ca href=\"https://twitter.com/wusheng1108\"\u003eTwitter\u003c/a\u003e, …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/blog/2020-01-20-celebrate-200th-contributor/","title":"SkyWalking hits 200 contributors mark"},{"body":"Based on his continuous contributions, Hongwei Zhai (a.k.a innerpeacez) has been invited to join the PMC. Welcome aboard.\n","excerpt":"\u003cp\u003eBased on his continuous contributions, Hongwei Zhai (a.k.a \u003ca href=\"https://github.com/innerpeacez\"\u003einnerpeacez\u003c/a\u003e) has been invited to join the …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-hongwei-zhai-to-join-the-pmc/","title":"Welcome Hongwei Zhai to join the PMC"},{"body":"Apache APM 6.6.0 release. Go to downloads page to find release tars.\nService Instance dependency detection are available. Support ElasticSearch 7 as a storage option. Reduce the register load. ","excerpt":"\u003cp\u003eApache APM 6.6.0 release. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eService Instance dependency …\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-6-6-0/","title":"Release Apache SkyWalking APM 6.6.0"},{"body":"SkyWalking Chart 1.0.0 is released. Go to downloads page to find release tars.\nDeploy SkyWalking 6.5.0 by Chart. Elasticsearch deploy optional. ","excerpt":"\u003cp\u003eSkyWalking Chart 1.0.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eDeploy SkyWalking …\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-chart-1-0-0-for-skywalking-6-5-0/","title":"Release Apache SkyWalking Chart 1.0.0 for SkyWalking 6.5.0"},{"body":"SkyWalking CLI 0.1.0 is released. Go to downloads page to find release tars.\nAdd command swctl service to list services Add command swctl instance and swctl search to list and search instances of service. Add command swctl endpoint to list endpoints of service. Add command swctl linear-metrics to query linear metrics and plot the metrics in Ascii Graph mode. Add command swctl single-metrics to query single-value metrics. ","excerpt":"\u003cp\u003eSkyWalking CLI 0.1.0 is released. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eAdd command \u003ccode\u003eswctl …\u003c/code\u003e\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-cli-0-1-0/","title":"Release Apache SkyWalking CLI 0.1.0"},{"body":"Based on his continuous contributions, Weiyi Liu (a.k.a wayilau) has been voted as a new committer.\n","excerpt":"\u003cp\u003eBased on his continuous contributions, Weiyi Liu (a.k.a \u003ca href=\"https://github.com/wayilau\"\u003ewayilau\u003c/a\u003e) has been voted as a new committer.\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-weiyi-liu-as-new-committer/","title":"Welcome Weiyi Liu as new committer"},{"body":"Based on his contributions to the project, he has been accepted as SkyWalking committer. Welcome aboard.\n","excerpt":"\u003cp\u003eBased on his contributions to the project, he has been accepted as SkyWalking committer. Welcome …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-lang-li-as-a-new-committer/","title":"Welcome Lang Li as a new committer"},{"body":"Based on her continuous contributions, Qiuxia Fan (a.k.a Fine0830) has been voted as a new committer.\n","excerpt":"\u003cp\u003eBased on her continuous contributions, Qiuxia Fan (a.k.a \u003ca href=\"https://github.com/Fine0830\"\u003eFine0830\u003c/a\u003e) has been voted as a new …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-qiuxia-fan-as-new-committer/","title":"Welcome Qiuxia Fan as new committer"},{"body":"6.5.0 release. Go to downloads page to find release tars.\nNew metrics comparison view in UI. Dynamic Alert setting supported. JDK9-12 supported in backend. ","excerpt":"\u003cp\u003e6.5.0 release. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eNew metrics comparison view in UI. …\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-6-5-0/","title":"Release Apache SkyWalking APM 6.5.0"},{"body":"Based on his continuous contributions, Wei Zhang (a.k.a arugal) has been voted as a new committer.\n","excerpt":"\u003cp\u003eBased on his continuous contributions, Wei Zhang (a.k.a \u003ca href=\"https://github.com/arugal\"\u003earugal\u003c/a\u003e) has been voted as a new committer.\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-wei-zhang-as-new-committer/","title":"Welcome Wei Zhang as new committer"},{"body":"PS：本文仅仅是在我的测试环境实验过，如果有问题，请自行优化调整\n前记：记得skywlking还是6.0版本的时候我就在试用，当时是skywalking基本在两三天左右就会监控数据完全查不出来，elasticsearch日志报错，由于当时也算是初用es，主要用来日志收集，并且时间有限，没有继续深入研究，最近空闲，更新到最新的6.5.0(开发版本)还是会出现同样的问题，下定决心解决下，于是有了本文的浅知拙见\n本次调优环境 skywalking: 6.5.0 elasticsearch:6.3.2(下文用es代替)\n调优过程 当然是百度了，百度后其实翻来翻去就找到一个相关的文章https://my.oschina.net/keking/blog/3025303 ，参考之。\n调整skywalking的这两个参数试试 bulkActions: 4000 # Execute the bulk every 2000 requests bulkSize: 60 # flush the bulk every 20mb 然后es还是继续挂，继续频繁的重启\n继续看这个文章，发现了另外一篇https://www.easyice.cn/archives/207 ，继续参考之\n这篇文章发现每一个字我都认识，看起来也能懂，但是对于es小白的我来说，着实不知道怎么调整这些参数，姑且先加到es的配置文件里边试试看吧，于是就加了，然后重启es的时候说发现index参数配置，自从5.0之后就不支持这样配置了，还给调了个es的接口去设置，但是设置失败（真够不错的），朝着这个思路去百度，百度到快放弃，后来就寻思，再试试看吧，（百度的结果是知道了index有静态参数和动态参数，动态的参数是可以随时设置，静态的只能创建或者关闭状态的索引才可以设置） 然鹅并不知道怎么关闭索引，继续百度，（怎么全特么百度，好吧不百度了，直接来干货）\n关闭索引（我的skywalking索引命名空间是dry_trace） curl -XPOST \u0026quot;http://localhost:9200/dry_trace*/_close\u0026quot; 设置参数 curl -XPUT \u0026#39;http://localhost:9200/dry_trace*/_settings?preserve_existing=true\u0026#39; -H \u0026#39;Content-type:application/json\u0026#39; -d \u0026#39;{ \u0026#34;index.refresh_interval\u0026#34; : \u0026#34;10s\u0026#34;, \u0026#34;index.translog.durability\u0026#34; : \u0026#34;async\u0026#34;, \u0026#34;index.translog.flush_threshold_size\u0026#34; : \u0026#34;1024mb\u0026#34;, \u0026#34;index.translog.sync_interval\u0026#34; : \u0026#34;120s\u0026#34; }\u0026#39; 打开索引 curl -XPOST \u0026quot;http://localhost:9200/dry_trace*/_open\u0026quot; 还有一点，第四步的方式只适用于现有的索引设置，那么新的索引设置呢，总不能每天重复下第四步吧。当然不需要，来干货 首先登陆kinaba控制台找到开发工具 贴入以下代码\nPUT /_template/dry_trace_tmp { \u0026#34;index_patterns\u0026#34;: \u0026#34;dry_trace*\u0026#34;, \u0026#34;order\u0026#34;: 1, \u0026#34;settings\u0026#34;: { \u0026#34;index\u0026#34;: { \u0026#34;refresh_interval\u0026#34;: \u0026#34;30s\u0026#34;, \u0026#34;translog\u0026#34;: { \u0026#34;flush_threshold_size\u0026#34;: \u0026#34;1GB\u0026#34;, \u0026#34;sync_interval\u0026#34;: \u0026#34;60s\u0026#34;, \u0026#34;durability\u0026#34;: \u0026#34;async\u0026#34; } } } } 截止目前为止运行一周，还未发现挂掉，一切看起来正常 完结\u0026mdash; 于 2019年11月\n","excerpt":"\u003cp\u003ePS：本文仅仅是在我的测试环境实验过，如果有问题，请自行优化调整\u003c/p\u003e\n\u003cp\u003e前记：记得skywlking还是6.0版本的时候我就在试用，当时是skywalking基本在两三天左右就会监控数据完全查不出来 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2019-11-07-skywalking-elasticsearch-storage-optimization/","title":"SkyWalking 使用 ElasticSearch 存储的优化"},{"body":"Based on his continuous contributions, Haochao Zhuang (a.k.a dmsolr) has been voted as a new committer.\n","excerpt":"\u003cp\u003eBased on his continuous contributions, Haochao Zhuang (a.k.a \u003ca href=\"https://github.com/dmsolr\"\u003edmsolr\u003c/a\u003e) has been voted as a new …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-haochao-zhuang-as-new-committer/","title":"Welcome Haochao Zhuang as new committer"},{"body":" 作者：innerpeacez 原文地址 本文主要讲述的是如何使用 Helm Charts 将 SkyWalking 部署到 Kubernetes 集群中，相关文档可以参考skywalking-kubernetes 和 backend-k8s 文档 。\n目前推荐的四种方式：\n使用 helm 2 提供的 helm serve 启动本地 helm repo 使用本地 chart 文件部署 使用 harbor 提供的 repo 功能 直接从官方 repo 进行部署 注意：目前 skywalking 的 chart 还没有提交到官方仓库，请先参照前三种方式进行部署\nHelm 2 提供的 helm serve 打包对应版本的 skywalking chart 1.配置 helm 环境，参考 Helm 环境配置 ，如果你要部署 helm2 相关 chart 可以直接配置 helm2 的相关环境\n2.克隆/下载ZIP skywalking-kubernetes 这个仓库，仓库关于chart的目录结构如下\nhelm-chart\nhelm2 6.0.0-GA 6.1.0 helm3 6.3.0 6.4.0 克隆/下载ZIP 完成后进入指定目录打包对应版本的chart\ncd skywalking-kubernetes/helm-chart/\u0026lt;helm-version\u0026gt;/\u0026lt;skywalking-version\u0026gt; 注意：helm-version 为对应的 helm 版本目录，skywalking-version 为对应的 skywalking 版本目录，下面以helm3 和 skywalking 6.3.0 为例\ncd skywalking-kubernetes/helm-chart/helm3/6.3.0 3.由于skywalking 依赖 elasticsearch 作为存储库，执行以下命令更新依赖，默认会从官方repo进行拉取\nhelm dep up skywalking Hang tight while we grab the latest from your chart repositories\u0026hellip; \u0026hellip;Successfully got an update from the \u0026ldquo;stable\u0026rdquo; chart repository Update Complete. ⎈Happy Helming!⎈ Saving 1 charts Downloading elasticsearch from repo https://kubernetes-charts.storage.googleapis.com/ Deleting outdated charts\n如果官方 repo 不存在，请先添加官方仓库\nhelm repo add stable https://kubernetes-charts.storage.googleapis.com \u0026ldquo;stable\u0026rdquo; has been added to your repositories\n4.打包 skywalking , 执行以下命令\nhelm package skywalking/ Successfully packaged chart and saved it to: C:\\code\\innerpeacez_github\\skywalking-kubernetes\\helm-chart\\helm3\\6.3.0\\skywalking-0.1.0.tgz\n打包完成后会在当前目录的同级目录生成 .tgz 文件\nls skywalking/ skywalking-0.1.0.tgz\n启动 helm serve 由于上文配置的 helm 为 helm3 ,但是 helm 3中移除了 helm serve 的相关命令，所以需要另外一个环境配置helm2 的相关环境，下载 helm 2.14.3 的二进制文件，配置基本上没有大的差别，不在赘述\n初始化 helm\nhelm init 将上文生成的 skywalking-0.1.0.tgz 文件复制到 helm 相关目录 /root/.helm/repository/local,启动 serve\nhelm serve --address \u0026lt;ip\u0026gt;:8879 --repo-path /root/.helm/repository/local 注意： ip 为要能够被上文配置 helm 3 环境的机器访问到\n可以访问一下看看服务 serve 是否启动成功\ncurl ip:8879 部署 skywalking 1.在helm3 环境中添加启动的本地 repo\nhelm repo add local http://\u0026lt;ip\u0026gt;:8879 2.查看 skywalking chart 是否存在于本地仓库中\nhelm search skywalking NAME CHART VERSION\tAPP VERSION\tDESCRIPTION local/skywalking 0.1.0 6.3.0 Apache SkyWalking APM System\n3.部署\nhelm -n test install skywalking local/skywalking 这样 skywalking 就部署到了 k8s 集群中的 test 命名空间了，至此本地安装skywalking 就完成了。\n本地文件部署 如果你不想存储到 chart 到仓库中也可以直接使用本地文件部署 skywalking,按照上面的步骤将skywalking chart 打包完成之后，直接使用以下命令进行部署\nhelm -n test install skywalking skywalking-0.1.0.tgz harbor 作为 repo 存储 charts harbor 目前已经提供了，charts repo 的能力，这样就可以将 docker 镜像和 chart 存储在一个仓库中了，方便维护，具体harbor 的部署方法参考 Harbor 作为存储仓库存储 chart\n官方 repo 部署 目前没有发布到官方 repo 中，后续发布完成后，只需要执行下面命令即可\nhelm install -n test stable/skywalking 总结 四种方式都可以进行部署，如果你想要自定义 chart ,需要使用上述两种本地方法及 harbor 存储的方式，以便你修改好 chart 之后进行部署.\n","excerpt":"\u003cul\u003e\n\u003cli\u003e作者：innerpeacez\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://ipzgo.top/2019-10-08-%E5%A6%82%E4%BD%95%E4%BD%BF%E7%94%A8-helm-chart-%E9%83%A8%E7%BD%B2-skywalking/\"\u003e原文地址\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e本文主要讲述的是如何使用 Helm Charts  将 SkyWalking 部署到 Kubernetes 集群中，相关文档可以参考 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2019-10-08-how-to-use-sw-chart/","title":"使用 chart 部署 SkyWalking"},{"body":" Author: Wei Qiang GitHub Background SkyWalking backend provides the alarm function, we can define some Alarm rules, call webhook after the rule is triggered. I share my implementation\nDemonstration SkyWalking alarm UI\ndingtalk message body\nIntroduction install go get -u github.com/weiqiang333/infra-skywalking-webhook cd $GOPATH/src/github.com/weiqiang333/infra-skywalking-webhook/ bash build/build.sh ./bin/infra-skywalking-webhook help Configuration main configs file: configs/production.yml dingtalk: p3: token... Example ./bin/infra-skywalking-webhook --config configs/production.yml --address 0.0.0.0:8000 SkyWalking backend alarm settings webhooks: - http://127.0.0.1:8000/dingtalk Collaboration Hope that we can improve together webhook\nSkyWalking alarm rules may add more metric names (eg priority name), we can send different channels by locating different levels of alerts (dingtalk / SMS / phone)\nThanks.\n","excerpt":"\u003cul\u003e\n\u003cli\u003eAuthor: Wei Qiang\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/weiqiang333\"\u003eGitHub\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"background\"\u003eBackground\u003c/h2\u003e\n\u003cp\u003eSkyWalking backend provides the alarm function, we can define …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2019-09-25-alarm-webhook-share/","title":"SkyWalking alarm webhook sharing"},{"body":"作者： SkyWalking committer，Kdump\n本文介绍申请Apache SkyWalking Committer流程, 流程包括以下步骤\n与PMC成员表达想成为committer的意愿(主动/被动) PMC内部投票 PMC正式邮件邀请 填写Apache iCLA申请表 设置ApacheID和邮箱 设置GitHub加入Apache组织 GitHub其它一些不重要设置 前期过程 与PMC成员表达想成为committer的意愿(主动/被动) PMC内部投票 当你对项目的贡献活跃度足够高或足够多时, Skywalking项目的PMC(项目管理委员会)会找到你并询问你是否有意愿成为项目的Committer, 或者也可以主动联系项目的PMC表达自己的意向, 在此之后PMC们会进行内部讨论和投票并告知你是否可以进入下一个环节.这个过程可能需要一周. 如果PMC主动邀请你进行非正式的意愿咨询, 你可以选择接受或拒绝.\nPS:PMC会向你索要你的个人邮箱, 建议提供Gmail, 因为后期绑定Apache邮箱需要用到, 其它邮箱我不确定是否能绑定.\nPS:从Apache官方的流程来讲, 现有的PMC会在没有通知候选人的情况下先进行候选人投票, 但是Skywalking项目的PMC有可能更倾向于先得到候选人的意愿再进行投票.\n正式阶段 PMC正式邮件邀请\n当你收到PMC正式的邀请邮件时, 恭喜你, 你已经通过了PMC的内部投票, 你需要用英文回答接受邀请或者拒绝邀请, 记住回复的时候一定要选择全部回复. 填写Apache iCLA申请表\n在你收到的PMC邮件中, 有几个ASF官方链接需要你去浏览, 重点的内容是查看CLAs, 并填写Individual Contributor License Agreement, 你可以将icla.pdf文件下载到本地, 使用PDF工具填写里面所需的信息, 并打印出来签名(一定要手写签名, 否则会被要求重新签名), 再扫描(或手机拍照)成电子文档(需要回复PDF格式, 文件名建议重命名为你的名字-icla.pdf), 使用gpg对电子文档进行签名(参考[HOW-TO: SUBMITTING LICENSE AGREEMENTS AND GRANTS\n](http://www.apache.org/licenses/contributor-agreements.html#submitting)), Window可以使用GnuPG或者Gpg4win.\n完成gpg签名后, 请将你签名用的公钥上送到pool.sks-keyservers.net服务器, 并在这个页面中验证你的公钥是否可以被搜索到, 搜索关键词可以是你秘钥中填写的名字或者邮箱地址.\ngpg签名后, 会生成.pdf.asc的文件, 需要将你的你的名字-icla.pdf和你的名字-icla.pdf.asc以附件的方式一起发送到secretary@apache.org, 并抄送给private@skywalking.apache.org.\n设置ApacheID和邮箱\n大概5个工作日内, 你会收到一封来至于root@apache.org的邮件, 主题为Welcome to the Apache Software Foundation (ASF)!, 恭喜你, 你已经获得了ApacheID, 这时候你需要根据邮件内容的提示去设置你的ApacheID密码, 密码设置完成后, 需要在Apache Account Utility页面中重点设置Forwarding email address和Your GitHub Username两个信息.保存信息的时候需要你填写当前的ApacheID的密码. 现在进入Gmail, 选择右上角的齿轮-\u0026gt;设置-\u0026gt;账号和导入-\u0026gt;添加其他电子邮件地址-\u0026gt;参考Sending email from your apache.org email address给出的信息根据向导填写Apache邮箱. 设置GitHub加入Apache组织\n进入Welcome to the GitBox Account Linking Utility!, 按照顺序将Apache Account和GitHub Account点绿, 想点绿MFA Status, 需要去GitHub开启2FA, 请参考配置双重身份验证完成2FA的功能. 等待1~2小时后登陆自己的GitHub的dashboard界面, 你应该会看到一条Apache组织邀请你加入的通知, 这个时候接受即可享有Skywalking相关GitHub项目权限了. 其它提示 GitHub其它一些不重要设置 在GitHub首页展示Apache组织的logo: 进入Apache GitHub组织-\u0026gt;People-\u0026gt;搜索自己的GitHubID-\u0026gt;将Private改成Public ","excerpt":"\u003cp\u003e作者： SkyWalking committer，\u003ca href=\"https://github.com/x22x22\"\u003eKdump\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e本文介绍申请Apache SkyWalking Committer流程, 流程包括以下步骤\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e与PMC成员表达想成为committer的意 …\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/zh/2019-09-12-apache-skywalking-committer-apply-process/","title":"Apache SkyWalking Committer申请流程"},{"body":"Based on his contributions to the skywalking ui project, Weijie Zou (a.k.a Kdump) has been accepted as a new committer.\n","excerpt":"\u003cp\u003eBased on his contributions to the skywalking ui project, Weijie Zou (a.k.a \u003ca href=\"https://github.com/x22x22\"\u003eKdump\u003c/a\u003e) has been accepted …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-weijie-zou-as-a-new-committer/","title":"Welcome Weijie Zou as a new committer"},{"body":"6.4.0 release. Go to downloads page to find release tars.\nHighly recommend to upgrade due to Pxx metrics calculation bug. Make agent working in JDK9+ Module system. Read changelog for the details.\n","excerpt":"\u003cp\u003e6.4.0 release. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eHighly recommend to upgrade due to Pxx …\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-6-4-0/","title":"Release Apache SkyWalking APM 6.4.0"},{"body":" 作者：innerpeacez 原文地址 如果你还不知道 Skywalking agent 是什么，请点击这里查看 Probe 或者这里查看快速了解agent,由于我这边大部分都是 JAVA 服务，所以下文以 Java 中使用 agent 为例，提供了以下三种方式供你选择\n三种方式： 使用官方提供的基础镜像 将 agent 包构建到已经存在的基础镜像中 sidecar 模式挂载 agent 1.使用官方提供的基础镜像 查看官方 docker hub 提供的基础镜像，只需要在你构建服务镜像是 From 这个镜像即可，直接集成到 Jenkins 中可以更加方便\n2.将 agent 包构建到已经存在的基础镜像中 提供这种方式的原因是：官方的镜像属于精简镜像，并且是 openjdk ，可能很多命令没有，需要自己二次安装，以下是我构建的过程\n下载 oracle jdk\n这个现在 oracle 有点恶心了，wget 各种不行，然后我放弃了，直接从官网下载了\n下载 skywalking 官方发行包，并解压（以6.3.0为例）\nwget https://www.apache.org/dyn/closer.cgi/skywalking/6.3.0/apache-skywalking-apm-6.3.0.tar.gz \u0026amp;\u0026amp; tar -zxvf apache-skywalking-apm-6.3.0.tar.gz 通过以下 dockerfile 构建基础镜像\nFROM alpine:3.8 ENV LANG=C.UTF-8 RUN set -eux \u0026amp;\u0026amp; \\ apk update \u0026amp;\u0026amp; apk upgrade \u0026amp;\u0026amp; \\ wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub \u0026amp;\u0026amp;\\ wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.30-r0/glibc-2.30-r0.apk \u0026amp;\u0026amp;\\ apk --no-cache add unzip vim curl git bash ca-certificates glibc-2.30-r0.apk file \u0026amp;\u0026amp; \\ rm -rf /var/lib/apk/* \u0026amp;\u0026amp;\\ mkdir -p /usr/skywalking/agent/ # A streamlined jre ADD jdk1.8.0_221/ /usr/java/jdk1.8.0_221/ ADD apache-skywalking-apm-bin/agent/ /usr/skywalking/agent/ # set env ENV JAVA_HOME /usr/java/jdk1.8.0_221 ENV PATH ${PATH}:${JAVA_HOME}/bin # run container with base path:/ WORKDIR / CMD bash 这里由于 alpine 是基于mini lib 的，但是 java 需要 glibc ,所以加入了 glibc 相关的东西，最后构建出的镜像大小在 490M 左右，因为加了挺多命令还是有点大，仅供参考，同样构建出的镜像也可以直接配置到 jenkins 中。\n3.sidecar 模式挂载 agent 如果你们的服务是部署在 Kubernetes 中，你还可以使用这种方式来使用 Skywalking Agent ,这种方式的好处在与不需要修改原来的基础镜像，也不用重新构建新的服务镜像，而是以sidecar 模式，通过共享volume的方式将agent 所需的相关文件挂载到已经存在的服务镜像中\n构建 skywalking agent sidecar 镜像的方法\n下载skywalking 官方发行包，并解压\nwget https://www.apache.org/dyn/closer.cgi/skywalking/6.3.0/apache-skywalking-apm-6.3.0.tar.gz \u0026amp;\u0026amp; tar -zxvf apache-skywalking-apm-6.3.0.tar.gz 通过以下 dockerfile 进行构建\nFROM busybox:latest ENV LANG=C.UTF-8 RUN set -eux \u0026amp;\u0026amp; mkdir -p /usr/skywalking/agent/ ADD apache-skywalking-apm-bin/agent/ /usr/skywalking/agent/ WORKDIR / 注意：这里我没有在dockerfile中下载skywalking 发行包是因为保证构建出的 sidecar 镜像保持最小，bosybox 只有700 k左右，加上 agent 最后大小小于20M\n如何使用 sidecar 呢？\napiVersion: apps/v1 kind: Deployment metadata: labels: name: demo-sw name: demo-sw spec: replicas: 1 selector: matchLabels: name: demo-sw template: metadata: labels: name: demo-sw spec: initContainers: - image: innerpeacez/sw-agent-sidecar:latest name: sw-agent-sidecar imagePullPolicy: IfNotPresent command: [\u0026#39;sh\u0026#39;] args: [\u0026#39;-c\u0026#39;,\u0026#39;mkdir -p /skywalking/agent \u0026amp;\u0026amp; cp -r /usr/skywalking/agent/* /skywalking/agent\u0026#39;] volumeMounts: - mountPath: /skywalking/agent name: sw-agent containers: - image: nginx:1.7.9 name: nginx volumeMounts: - mountPath: /usr/skywalking/agent name: sw-agent ports: - containerPort: 80 volumes: - name: sw-agent emptyDir: {} 以上是挂载 sidecar 的 deployment.yaml 文件，以nginx 作为服务为例，主要是通过共享 volume 的方式挂载 agent，首先 initContainers 通过 sw-agent 卷挂载了 sw-agent-sidecar 中的 /skywalking/agent ，并且将上面构建好的镜像中的 agent 目录 cp 到了 /skywalking/agent 目录，完成之后 nginx 启动时也挂载了 sw-agent 卷，并将其挂载到了容器的 /usr/skywalking/agent 目录，这样就完成了共享过程。\n总结 这样除去 ServiceMesh 以外，我能想到的方式就介绍完了，希望可以帮助到你。最后给 Skywalking 一个 Star 吧，国人的骄傲。\n","excerpt":"\u003cblockquote\u003e\n\u003cul\u003e\n\u003cli\u003e作者：innerpeacez\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://ipzgo.top/2019-08-30-%E5%A6%82%E4%BD%95%E4%BD%BF%E7%94%A8-Skywalking-Agent/\"\u003e原文地址\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/blockquote\u003e\n\u003cp\u003e如果你还不知道 Skywalking agent 是什么，\u003ca href=\"https://github.com/apache/skywalking/blob/master/docs/en/concepts-and-designs/README.md\"\u003e请点击这里查看 Probe\u003c/a\u003e 或者\u003ca href=\"https://github.com/apache/skywalking/blob/master/docs/en/setup/service-agent/java-agent/README.md\"\u003e这里查看快速了解agent\u003c/a\u003e,由于我这边大部分都是 JAVA 服 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2019-08-30-how-to-use-skywalking-agent/","title":"如何使用 SkyWalking Agent ？"},{"body":"Based on his continuous contributions, Yuguang Zhao (a.k.a zhaoyuguang) has been invited to join the PMC. Welcome aboard.\n","excerpt":"\u003cp\u003eBased on his continuous contributions, Yuguang Zhao (a.k.a \u003ca href=\"https://github.com/zhaoyuguang\"\u003ezhaoyuguang\u003c/a\u003e) has been invited to join the …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-yuguang-zhao-to-join-the-pmc/","title":"Welcome Yuguang Zhao to join the PMC"},{"body":"Based on his continuous contributions, Zhenxu Ke (a.k.a kezhenxu94) has been invited to join the PMC. Welcome aboard.\n","excerpt":"\u003cp\u003eBased on his continuous contributions, Zhenxu Ke (a.k.a \u003ca href=\"https://github.com/kezhenxu94\"\u003ekezhenxu94\u003c/a\u003e) has been invited to join the …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-zhenxu-ke-to-join-the-pmc/","title":"Welcome Zhenxu Ke to join the PMC"},{"body":"Based on his contributions to the skywalking PHP project, Yanlong He (a.k.a heyanlong has been accepted as a new committer.\n","excerpt":"\u003cp\u003eBased on his contributions to the skywalking PHP project, Yanlong He (a.k.a \u003ca href=\"https://github.com/heyanlong\"\u003eheyanlong\u003c/a\u003e has been …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-yanlong-he-as-a-new-committer/","title":"Welcome Yanlong He as a new committer"},{"body":"6.3.0 release. Go to downloads page to find release tars.\nImprove ElasticSearch storage implementation performance again. OAP backend re-install w/o agent reboot required. Read changelog for the details.\n","excerpt":"\u003cp\u003e6.3.0 release. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eImprove ElasticSearch storage …\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-6-3-0/","title":"Release Apache SkyWalking APM 6.3.0"},{"body":"6.2.0 release. Go to downloads page to find release tars. ElasticSearch storage implementation changed, high reduce payload to ElasticSearch cluster.\nRead changelog for the details.\n","excerpt":"\u003cp\u003e6.2.0 release. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nElasticSearch storage implementation …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-6-2-0/","title":"Release Apache SkyWalking APM 6.2.0"},{"body":"Based on his continuous contributions, Zhenxu Ke (a.k.a kezhenxu94) has been voted as a new committer.\n","excerpt":"\u003cp\u003eBased on his continuous contributions, Zhenxu Ke (a.k.a \u003ca href=\"https://github.com/kezhenxu94\"\u003ekezhenxu94\u003c/a\u003e) has been voted as a new …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-zhenxu-ke-as-a-new-committer/","title":"Welcome Zhenxu Ke as a new committer"},{"body":"6.1.0 release. Go to downloads page to find release tars. This is the first top level project version.\nKey updates\nRocketBot UI OAP performance improvement ","excerpt":"\u003cp\u003e6.1.0 release. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nThis is the first top level project …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-6-1-0/","title":"Release Apache SkyWalking APM 6.1.0"},{"body":"Apache SkyWalking PMC accept the RocketBot UI contributions. After IP clearance, it will be released in SkyWalking 6.1 soon.\n","excerpt":"\u003cp\u003eApache SkyWalking PMC accept the RocketBot UI contributions. After IP clearance, it will be released …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/rocketbot-ui-has-been-accepted-as-skywalking-primary-ui/","title":"RocketBot UI has been accepted as SkyWalking primary UI"},{"body":"Apache board approved SkyWalking graduated as TLP at April 17th 2019.\n","excerpt":"\u003cp\u003eApache board approved SkyWalking graduated as TLP at April 17th 2019.\u003c/p\u003e","ref":"https://skywalking.apache.org/events/skywalking-graduated-as-apache-top-level-project/","title":"SkyWalking graduated as Apache Top Level Project"},{"body":"Based on his continuous contributions, he has been accepted as a new committer.\n","excerpt":"\u003cp\u003eBased on his continuous contributions, he has been accepted as a new committer.\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-yuguang-zhao-as-a-new-committer/","title":"Welcome Yuguang Zhao as a new committer"},{"body":"APM和调用链跟踪 随着企业经营规模的扩大，以及对内快速诊断效率和对外SLA（服务品质协议，service-level agreement)的追求，对于业务系统的掌控度的要求越来越高，主要体现在：\n对于第三方依赖的监控，实时/准实时了解第三方的健康状况/服务品质，降低第三方依赖对于自身系统的扰动（服务降级、故障转移） 对于容器的监控，实时/准实时的了解应用部署环境（CPU、内存、进程、线程、网络、带宽）情况，以便快速扩容/缩容、流量控制、业务迁移 业务方对于自己的调用情况，方便作容量规划，同时对于突发的请求也能进行异常告警和应急准备 自己业务的健康、性能监控，实时/准实时的了解自身的业务运行情况，排查业务瓶颈，快速诊断和定位异常，增加对自己业务的掌控力 同时，对于企业来说，能够更精确的了解资源的使用情况，对于成本核算和控制也有非常大的裨益。\n在这种情况下，一般都会引入APM（Application Performance Management \u0026amp; Monitoring）系统，通过各种探针采集数据，收集关键指标，同时搭配数据呈现和监控告警，能够解决上述的大部分问题。\n然而随着RPC框架、微服务、云计算、大数据的发展，同时业务的规模和深度相比过往也都增加了很多，一次业务可能横跨多个模块/服务/容器，依赖的中间件也越来越多，其中任何一个节点出现异常，都可能导致业务出现波动或者异常，这就导致服务质量监控和异常诊断/定位变得异常复杂，于是催生了新的业务监控模式：调用链跟踪\n能够分布式的抓取多个节点的业务记录，并且通过统一的业务id（traceId，messageId，requestId等）将一次业务在各个节点的记录串联起来，方便排查业务的瓶颈或者异常点 产品对比 APM和调用链跟踪均不是新诞生事务，很多公司已经有了大量的实践，不过开源的并且能够开箱即用的产品并不多，这里主要选取了Pinpoint，Skywalking，CAT来进行对比（当然也有其他的例如Zipkin，Jaeger等产品，不过总体来说不如前面选取的3个完成度高），了解一下APM和调用链跟踪在开源方面的发展状态。\nPinpoint Pinpoint是一个比较早并且成熟度也非常高的APM+调用链监控的项目，在全世界范围内均有用户使用，支持Java和PHP的探针，数据容器为HBase，其界面参考：\nSkywalking Skywalking是一个新晋的项目，最近一两年发展非常迅猛，本身支持OpenTracing规范，优秀的设计提供了良好的扩展性，支持Java、PHP、.Net、NodeJs探针，数据容器为ElasticSearch，其界面参考：\nCAT CAT是由美团开源的一个APM项目，也历经了多年的迭代升级，拥有大量的企业级用户，对于监控和报警整合比较紧密，支持Java、C/C++、.Net、Python、Go、NodeJs，不过CAT目前主要通过侵入性的方式接入，数据容器包括HDFS（存储原始数据）和mysql（二次统计），其界面参考：\n横向对比 上面只是做了一个简介，那这三个项目各自有什么特色或者优势/劣势呢（三者的主要产品均针对Java，这里也主要针对Java的特性）？\nPinpoint 优势 大企业/长时间验证，稳定性和完成度高 探针收集的数据粒度比较细 HBase的数据密度较大，支持PB级别下的数据查询 代码设计考虑的扩展性较弱，二次开发难度较大（探针为插件式，开发比较简单） 拥有完整的APM和调用链跟踪功能 劣势 代码针对性强，扩展较难 容器为HBase，查询功能较弱（主要为时间维度） 探针的额外消耗较多（探针采集粒度细，大概10%~20%） 项目趋于成熟，而扩展难度较大，目前社区活跃度偏低，基本只进行探针的增加或者升级 缺少自定义指标的设计 Skywalking 优势 数据容器为ES，查询支持的维度较多并且扩展潜力大 项目设计采用微内核+插件，易读性和扩展性都比较强 主要的研发人员为华人并且均比较活跃，能够进行更加直接的沟通 拥有完整的APM和调用链跟踪功能 劣势 项目发展非常快，稳定性有待验证 ES数据密度较小，在PB级别可能会有性能压力 缺少自定义指标的设计 CAT 优势 大企业/长时间验证，稳定性和完成度高 采用手动数据埋点而不是探针，数据采集的灵活性更强 支持自定义指标 代码设计考虑的扩展性较弱，并且数据结构复杂，二次开发难度较大 拥有完善的监控告警机制 劣势 代码针对性强，扩展较难 需要手动接入埋点，代码侵入性强 APM功能完善，但是不支持调用链跟踪 基本组件 如果分别去看Pinpoint/Skywalking/CAT的整体设计，我们会发现三者更像是一个规范的三种实现，虽然各自有不同的机制和特性，但是从模块划分和功能基本是一致的：\n当然也有一些微小的区别：\nPinpoint基本没有aggregator，同时query和alarm集成在了web中，只有agent，collector和web Skywalking则是把collector、aggregator、alarm集成为OAP（Observability Analysis Platform），并且可以通过集群部署，不同的实例可以分别承担collector或者aggregator+alarm的角色 CAT则和Skywalking类似，把collector、aggregator、alarm集成为cat-consumer，而由于CAT有比较复杂的配置管理，所以query和配置一起集成为cat-home 当然最大的区别是Pinpoint和Skywalking均是通过javaagent做字节码的扩展，通过切面编程采集数据，类似于探针，而CAT的agent则更像是一个工具集，用于手动埋点 Skywalking 前戏这么多，终于开始进入主题，介绍今天的主角：Skywalking，不过通过之前的铺垫，我们基本都知道了Skywalking期望解决的问题以及总体的结构，下面我们则从细节来看Skywalking是怎么一步一步实现的。\n模块构成 首先，Skywalking进行了精准的领域模型划分：\n整个系统分为三部分：\nagent：采集tracing（调用链数据）和metric（指标）信息并上报 OAP：收集tracing和metric信息通过analysis core模块将数据放入持久化容器中（ES，H2（内存数据库），mysql等等），并进行二次统计和监控告警 webapp：前后端分离，前端负责呈现，并将查询请求封装为graphQL提交给后端，后端通过ribbon做负载均衡转发给OAP集群，再将查询结果渲染展示 而整个Skywalking（包括agent和OAP，而webapp后端业务非常简单主要就是认证和请求转发）均通过微内核+插件式的模式进行编码，代码结构和扩展性均非常强，具体设计可以参考： 从Skywalking看如何设计一个微核+插件式扩展的高扩展框架 ，Spring Cloud Gateway的GatewayFilterFactory的扩展也是通过这种plugin define的方式来实现的。\nSkywalking也提供了其他的一些特性：\n配置重载：支持通过jvm参数覆写默认配置，支持动态配置管理 集群管理：这个主要体现在OAP，通过集群部署分担数据上报的流量压力和二次计算的计算压力，同时集群也可以通过配置切换角色，分别面向数据采集（collector）和计算（aggregator，alarm），需要注意的是agent目前不支持多collector负载均衡，而是随机从集群中选择一个实例进行数据上报 支持k8s和mesh 支持数据容器的扩展，例如官方主推是ES，通过扩展接口，也可以实现插件去支持其他的数据容器 支持数据上报receiver的扩展，例如目前主要是支持gRPC接受agent的上报，但是也可以实现插件支持其他类型的数据上报（官方默认实现了对Zipkin，telemetry和envoy的支持） 支持客户端采样和服务端采样，不过服务端采样最有意义 官方制定了一个数据查询脚本规范：OAL（Observability Analysis Language），语法类似Linq，以简化数据查询扩展的工作量 支持监控预警，通过OAL获取数据指标和阈值进行对比来触发告警，支持webhook扩展告警方式，支持统计周期的自定义，以及告警静默防止重复告警 数据容器 由于Skywalking并没有自己定制的数据容器或者使用多种数据容器增加复杂度，而是主要使用ElasticSearch（当然开源的基本上都是这样来保持简洁，例如Pinpoint也只使用了HBase），所以数据容器的特性以及自己数据结构基本上就限制了业务的上限，以ES为例：\nES查询功能异常强大，在数据筛选方面碾压其他所有容器，在数据筛选潜力巨大（Skywalking默认的查询维度就比使用HBase的Pinpoint强很多） 支持sharding分片和replicas数据备份，在高可用/高性能/大数据支持都非常好 支持批量插入，高并发下的插入性能大大增强 数据密度低，源于ES会提前构建大量的索引来优化搜索查询，这是查询功能强大和性能好的代价，但是链路跟踪往往有非常多的上下文需要记录，所以Skywalking把这些上下文二进制化然后通过Base64编码放入data_binary字段并且将字段标记为not_analyzed来避免进行预处理建立查询索引 总体来说，Skywalking尽量使用ES在大数据和查询方面的优势，同时尽量减少ES数据密度低的劣势带来的影响，从目前来看，ES在调用链跟踪方面是不二的数据容器，而在数据指标方面，ES也能中规中矩的完成业务，虽然和时序数据库相比要弱一些，但在PB级以下的数据支持也不会有太大问题。\n数据结构 如果说数据容器决定了上限，那么数据结构则决定了实际到达的高度。Skywalking的数据结构主要为：\n数据维度（ES索引为skywalking_*_inventory) service：服务 instance：实例 endpoint：接口 network_adress：外部依赖 数据内容 原始数据 调用链跟踪数据（调用链的trace信息，ES索引为skywalking_segment，Skywalking主要的数据消耗都在这里） 指标（主要是jvm或者envoy的运行时指标，例如ES索引skywalking_instance_jvm_cpu） 二次统计指标 指标（按维度/时间二次统计出来的例如pxx、sla等指标，例如ES索引skywalking_database_access_p75_month） 数据库慢查询记录（数据库索引：skywalking_top_n_database_statement） 关联关系（维度/指标之间的关联关系，ES索引为skywalking_*_relation_*) 特别记录 告警信息（ES索引为skywalking_alarm_record） 并发控制（ES索引为skywalking_register_lock） 其中数量占比最大的就是调用链跟踪数据和各种指标，而这些数据均可以通过OAP设置过期时间，以降低历史数据的对磁盘占用和查询效率的影响。\n调用链跟踪数据 作为Skywalking的核心数据，调用链跟踪数据（skywalking_segment）基本上奠定了整个系统的基础，而如果要详细的了解调用链跟踪的话，就不得不提到openTracing。\nopenTracing基本上是目前开源调用链跟踪系统的一个事实标准，它制定了调用链跟踪的基本流程和基本的数据结构，同时也提供了各个语言的实现。如果用一张图来表现openTracing，则是如下：\n其中：\nSpanContext：一个类似于MDC（Slfj)或者ThreadLocal的组件，负责整个调用链数据采集过程中的上下文保持和传递 Trace：一次调用的完整记录 Span：一次调用中的某个节点/步骤，类似于一层堆栈信息，Trace是由多个Span组成，Span和Span之间也有父子或者并列的关系来标志这个节点/步骤在整个调用中的位置 Tag：节点/步骤中的关键信息 Log：节点/步骤中的详细记录，例如异常时的异常堆栈 Baggage：和SpanContext一样并不属于数据结构而是一种机制，主要用于跨Span或者跨实例的上下文传递，Baggage的数据更多是用于运行时，而不会进行持久化 以一个Trace为例：\n首先是外部请求调用A，然后A依次同步调用了B和C，而B被调用时会去同步调用D，C被调用的时候会依次同步调用E和F，F被调用的时候会通过异步调用G，G则会异步调用H，最终完成一次调用。\n上图是通过Span之间的依赖关系来表现一个Trace，而在时间线上，则可以有如下的表达：\n当然，如果是同步调用的话，父Span的时间占用是包括子Span的时间消耗的。\n而落地到Skywalking中，我们以一条skywalking_segment的记录为例：\n{ \u0026#34;trace_id\u0026#34;: \u0026#34;52.70.15530767312125341\u0026#34;, \u0026#34;endpoint_name\u0026#34;: \u0026#34;Mysql/JDBI/Connection/commit\u0026#34;, \u0026#34;latency\u0026#34;: 0, \u0026#34;end_time\u0026#34;: 1553076731212, \u0026#34;endpoint_id\u0026#34;: 96142, \u0026#34;service_instance_id\u0026#34;: 52, \u0026#34;version\u0026#34;: 2, \u0026#34;start_time\u0026#34;: 1553076731212, \u0026#34;data_binary\u0026#34;: \u0026#34;CgwKCjRGnPvp5eikyxsSXhD///////////8BGMz62NSZLSDM+tjUmS0wju8FQChQAVgBYCF6DgoHZGIudHlwZRIDc3FsehcKC2RiLmluc3RhbmNlEghyaXNrZGF0YXoOCgxkYi5zdGF0ZW1lbnQYAiA0\u0026#34;, \u0026#34;service_id\u0026#34;: 2, \u0026#34;time_bucket\u0026#34;: 20190320181211, \u0026#34;is_error\u0026#34;: 0, \u0026#34;segment_id\u0026#34;: \u0026#34;52.70.15530767312125340\u0026#34; } 其中：\ntrace_id：本次调用的唯一id，通过snowflake模式生成 endpoint_name：被调用的接口 latency：耗时 end_time：结束时间戳 endpoint_id：被调用的接口的唯一id service_instance_id：被调用的实例的唯一id version：本数据结构的版本号 start_time：开始时间戳 data_binary：里面保存了本次调用的所有Span的数据，序列化并用Base64编码，不会进行分析和用于查询 service_id：服务的唯一id time_bucket：调用所处的时段 is_error：是否失败 segment_id：数据本身的唯一id，类似于主键，通过snowflake模式生成 这里可以看到，目前Skywalking虽然相较于Pinpoint来说查询的维度要多一些，但是也很有限，而且除了endPoint，并没有和业务有关联的字段，只能通过时间/服务/实例/接口/成功标志/耗时来进行非业务相关的查询，如果后续要增强业务相关的搜索查询的话，应该还需要增加一些用于保存动态内容（如messageId，orderId等业务关键字）的字段用于快速定位。\n指标 指标数据相对于Tracing则要简单得多了，一般来说就是指标标志、时间戳、指标值，而Skywalking中的指标有两种：一种是采集的原始指标值，例如jvm的各种运行时指标（例如cpu消耗、内存结构、GC信息等）；一种是各种二次统计指标（例如tp性能指标、SLA等，当然也有为了便于查询的更高时间维度的指标，例如基于分钟、小时、天、周、月）\n例如以下是索引skywalking_endpoint_cpm_hour中的一条记录，用于标志一个小时内某个接口的cpm指标：\n{ \u0026#34;total\u0026#34;: 8900, \u0026#34;service_id\u0026#34;: 5, \u0026#34;time_bucket\u0026#34;: 2019031816, \u0026#34;service_instance_id\u0026#34;: 5, \u0026#34;entity_id\u0026#34;: \u0026#34;7\u0026#34;, \u0026#34;value\u0026#34;: 148 } 各个字段的释义如下：\ntotal：一分钟内的调用总量 service_id：所属服务的唯一id time_bucket：统计的时段 service_instance_id：所属实例的唯一id entity_id：接口（endpoint）的唯一id value：cpm的指标值（cpm=call per minute，即total/60） 工程实现 Skywalking的工程实现堪比Dubbo，框架设计和代码质量都达到非常高的水准，以dubbo为例，即使2012年发布的老版本放到当今，其设计和编码看起来也依然赏心悦目，设计简洁但是覆盖了所有的核心需求，同时又具备非常强的扩展性，二次开发非常简单，然而却又不会像Spring那样过度封装（当然Spring作为一个更加高度通用的框架，更高的封装也是有必要的）导致代码阅读异常困难。\nagent agent（apm-sniffer）是Skywalking的Java探针实现，主要负责：\n采集应用实例的jvm指标 通过切向编程进行数据埋点，采集调用链数据 通过RPC将采集的数据上报 当然，agent还实现了客户端采样，不过在APM监控系统里进行客户端数据采样都是没有灵魂的，所以这里就不再赘述了。\n首先，agent通过 org.apache.skywalking.apm.agent.core.boot.BootService 实现了整体的插件化，agent启动会加载所有的BootService实现，并通过 ServiceManager 来管理这些插件的生命周期，采集jvm指标、gRPC连接管理、调用链数据维护、数据上报OAP这些服务均是通过这种方式扩展。\n然后，agent还通过bytebuddy以javaagent的模式，通过字节码增强的机制来构造AOP环境，再提供PluginDefine的规范方便探针的开发，最终实现非侵入性的数据埋点，采集调用链数据。\n最终落地到代码上则异常清晰：\n//通过bytebuddy的AgentBuilder构造javaagent增强classLoader new AgentBuilder.Default(byteBuddy) .ignore( //忽略这些包的内容，不进行增强 nameStartsWith(\u0026#34;net.bytebuddy.\u0026#34;) .or(nameStartsWith(\u0026#34;org.slf4j.\u0026#34;)) .or(nameStartsWith(\u0026#34;org.apache.logging.\u0026#34;)) .or(nameStartsWith(\u0026#34;org.groovy.\u0026#34;)) .or(nameContains(\u0026#34;javassist\u0026#34;)) .or(nameContains(\u0026#34;.asm.\u0026#34;)) .or(nameStartsWith(\u0026#34;sun.reflect\u0026#34;)) .or(allSkyWalkingAgentExcludeToolkit()) .or(ElementMatchers.\u0026lt;TypeDescription\u0026gt;isSynthetic())) //通过pluginFinder加载所有的探针扩展，并获取所有可以增强的class .type(pluginFinder.buildMatch()) //按照pluginFinder的实现，去改变字节码增强类 .transform(new Transformer(pluginFinder)) //通过listener订阅增强的操作记录，方便调试 .with(new Listener()) .installOn(instrumentation); try { //加载所有的service实现并启动 ServiceManager.INSTANCE.boot(); } catch (Exception e) { logger.error(e, \u0026#34;Skywalking agent boot failure.\u0026#34;); } agent也提供了非常简单的扩展实现机制，以增强一个普通类的方法为例，首先你需要定义一个切向点：\npublic interface InstanceMethodsInterceptPoint { //定义切向方法的适配器，符合适配器的class将被增强 ElementMatcher\u0026lt;MethodDescription\u0026gt; getMethodsMatcher(); //增强的具体实现类，classReference String getMethodsInterceptor(); //是否重写参数 boolean isOverrideArgs(); } 然后你还需要一个增强的实现类：\npublic interface InstanceMethodsAroundInterceptor { //方法真正执行前执行 void beforeMethod(EnhancedInstance objInst, Method method, Object[] allArguments, Class\u0026lt;?\u0026gt;[] argumentsTypes, MethodInterceptResult result) throws Throwable; //方法真正执行后执行 Object afterMethod(EnhancedInstance objInst, Method method, Object[] allArguments, Class\u0026lt;?\u0026gt;[] argumentsTypes, Object ret) throws Throwable; //当异常发生时执行 void handleMethodException(EnhancedInstance objInst, Method method, Object[] allArguments, Class\u0026lt;?\u0026gt;[] argumentsTypes, Throwable t); } 一般在执行前和执行后进行数据埋点，就可以采集到想要的数据，当然实际编程要稍微复杂一点，不过官方也实现了对应的abstract类和数据埋点工具类，所以探针的二次开发在Skywalking这个级别确实是非常简单，只需要处理好资源占用和并发问题即可。真正的难点是要对需要增强的对象非常了解，熟悉其运作机制，才能找准切向点，既要所有的流程都需要经过这个点，又可以抓取到期望抓取的上下文信息。同时，多版本的适配和测试也是非常大的工作量，官方虽然提供witness的机制（通过验证某个class是否存在来验证版本），但是作为影响全局的探针，开发和测试都是需要慎之又慎的。\nOAP 同agent类似，OAP作为Skywalking最核心的模块，也实现了自己的扩展机制，不过在这里叫做Module，具体可以参考library-module，在module的机制下，Skywalking实现了自己必须核心组件：\ncore：整个OAP核心业务（remoting、cluster、storage、analysis、query、alarm）的规范和接口 cluster：集群管理的具体实现 storage：数据容器的具体实现 query：为前端提供的查询接口的具体实现 receiver：接收探针上报数据的接收器的具体实现 alarm：监控告警的具体实现 以及一个可选组件：\ntelemetry：用于监控OAP自身的健康状况 而前面提到的OAP的高扩展性则体现在核心业务的规范均定义在了core中，如果有需要自己扩展的，只需要自己单独做自己的实现，而不需要做侵入式的改动，最典型的示例则是官方支持的storage，不仅支持单机demo的内存数据库H2和经典的ES，连目前开源的Tidb都可以接入。\n初步实践 对于Skywalking的实践我们经历了三个阶段\n线下测试 第一次生产环境小规模测试 第二次生产环境小规模测试+全量接入 线下测试 环境 由于是线下测试，所以我们直接使用物理机（E5-2680v2 x2, 128G)虚拟了一个集群（实际性能相比云服务器应该偏好一些）：\nES：单机实例，v6.5，4C8G，jvm内存分配为4G OAP：单机实例，v6.1.0-SNAPSHOT，4C8G，jvm内存分配为4G 应用：基于SpringCloud的4个测试实例,调用关系为A-\u0026gt;B-\u0026gt;C-\u0026gt;D，QPS为200 测试结果 拓扑图：\nOAP机器监控：\nES机器监控：\n服务监控面板：\n其中一个调用链记录：\n可以看出，Skywalking非常依赖CPU（不论是OAP还是ES），同时对于网络IO也有一定的要求，至于ES的文件IO在可接受范围内，毕竟确实有大量内容需要持久化。测试结果也基本达到预期要求，调用链和各个指标的监控都工作良好。\n第一次生产环境测试 在线下测试之后，我们再进行了一次基于实际业务针对探针的测试，测试没有发现探针的异常问题，也没有影响业务的正常运作，同时对于jvm实例影响也不是很大，CPU大概提高了5%左右，并不很明显。在这个基础上我们选择了线上的一台服务器，进行了我们第一次生产环境的测试。\n环境 ES：基于现有的一个ES集群，node x 3，v6.0 OAP：2C4G x 2，v6.1.0-SNAPSHOT，jvm内存分配为2G 应用：两个jvm实例 测试时间：03.11-03.16\n测试结果 业务机器负载情况：\n从最敏感的CPU指标上来看，增加agent并没有导致可见的CPU使用率的变化，而其他的内存、网络IO、连接数也基本没有变化。\nOAP负载情况：\n可以看到机器的CPU和网络均有较大的波动，但是也都没有真正打爆服务器，但是我们的实例却经常出现两种日志：\nOne trace segment has been abandoned, cause by buffer is full.\nCollector traceSegment service doesn\u0026rsquo;t response in xxx seconds.\n通过阅读源码发现：\nagent和OAP只会使用一个长连接阻塞式的交换数据，如果某次数据交换没有得到响应，则会阻塞后续的上报流程（一般长连接的RPC请求会在数据传输期间互相阻塞，但是不会在等待期间互相阻塞，当然这也是源于agent并没有并发上报的机制），所以一旦OAP在接收数据的过程中发生阻塞，就会导致agent本地的缓冲区满，最终只能将监控数据直接丢弃防止内存泄漏 而导致OAP没有及时响应的一方面是OAP本身性能不够（OAP需要承担大量的二次统计工作，通过Jstack统计，长期有超过几十个线程处于RUNNABLE状态，据吴晟描述目前OAP都是高性能模式，后续将会提供配置来支持低性能模式），另一方面可能是ES批量插入效率不够，因此我们修改了OAP的批量插入参数来增加插入频率，降低单次插入数量：\nbulkActions: ${SW_STORAGE_ES_BULK_ACTIONS:2000 -\u0026gt; 20} # Execute the bulk every 2000 requests bulkSize: ${SW_STORAGE_ES_BULK_SIZE:20 -\u0026gt; 2} # flush the bulk every 20mb flushInterval: ${SW_STORAGE_ES_FLUSH_INTERVAL:10 -\u0026gt; 2} # flush the bulk every 10 seconds whatever the number of requests 虽然 service doesn\u0026rsquo;t response 出现的频率明显降低，但是依然还是会偶尔出现，而每一次出现都会伴随大量的 trace segment has been abandoned ，推测OAP和ES可能都存在性能瓶颈（应该进行更进一步的诊断确定问题，不过当时直接和吴晟沟通，确认确实OAP非常消耗CPU资源，考虑到当时部署只是2C，并且还部署有其他业务，就没有进一步的测试）。\n同时，在频繁的数据丢弃过程中，也偶发了一个bug：当agent上报数据超时并且大量丢弃数据之后，即使后续恢复正常也能通过日志看到数据正常上报，在查询界面查询的时候，会查不到这个实例上报的数据，不过在重启OAP和agent之后，之前上报的数据又能查询到，这个也和吴晟沟通过，没有其他的案例，后续想重现却也一直没有成功。\n而同时还发现两个更加严重的问题：\n我们使用的是线上已经部署好的ES集群，其版本只有6.0，而新的Skywalking使用了6.3的查询特性，导致很多查询执行报错，只能使用最简单的查询 我们的kafka集群版本也非常古老，不支持v1或者更高版本的header，而kafka的探针强依赖header来传输上下文信息，导致kafka客户端直接报错影响业务，所以也立即移除了kafka的探针 在这一次测试中，我们基本确认了agent对于应用的影响，同时也发现了一些我们和Skywalking的一些问题，留待后续测试确认。\n第二次生产环境测试 为了排除性能和ES版本的影响，测试Skywalking本身的可用性，参考吴晟的建议（这也是在最初技术选型的时候没有选择Pinpoint和CAT的部分原因：一方面Skywalking的功能符合我们的要求，更重要的是有更加直接和效率的和项目维护者直接沟通的渠道），所以这一次我们新申请了ES集群和OAP机器。\n环境 ES：腾讯云托管ES集群，4C16G x 3 SSD，v6.4 OAP：16C32G，standalone，jvm分配24G 应用：2~8个jvm实例 测试时间：03.18-至今\n测试结果 OAP负载情况：\nES集群负载：\n测试过程中，我们先接入了一台机器上的两个实例，完全没有遇到一测中的延迟或者数据丢弃的问题，三天后我们又接入了另外两台机器的4个实例，这之后两天我们又接入了另外两台机器的2个实例。依然没有遇到一测中的延迟或者数据丢弃的问题。\n而ES负载的监控也基本验证了一测延迟的问题，Skywalking由于较高的并发插入，对于ES的性能压力很大（批量插入时需要针对每条数据分析并且构建查询索引），大概率是ES批量插入性能不够导致延迟，考虑到我们仅仅接入了8个实例，日均segment插入量大概5000万条（即日均5000万次独立调用），如果想支持更大规模的监控，对于ES容量规划势必要留够足够的冗余。同时OAP和ES集群的网络开销也不容忽视，在支撑大规模的监控时，需要集群并且receiver和aggregattor分离部署来分担网络IO的压力。\n而在磁盘容量占用上，我们设置的原始数据7天过期，目前刚刚开始滚动过期，目前segment索引已经累计了314757240条记录总计158G数据，当然我们目前异常记录较少，如果异常记录较多的话，其磁盘开销将会急剧增加（span中会记录异常堆栈信息）。而由于选择的SSD，磁盘的写入和查询性能都很高，即使只有3个节点，也完全没有任何压力。\n而在新版本的ES集群下，Skywalking的所有查询功能都变得可用，和我们之前自己的单独编写的异常指标监控都能完美对照。当然我们也遇到一个问题：Skywalking仅采集了调用记录，但是对于调用过程中的过程数据，除了异常堆栈其他均没有采集，导致真的出现异常也缺少充足的上下文信息还原现场，于是我们扩展了Skywalking的两个探针（我们项目目前重度依赖的组件）：OkHttp（增加对requestBody和responseBody的采集）和SpringMVC（增加了对requestBody的采集），目前工作正常，如果进一步的增加其他的探针，采集到足够的数据，那么我们基本可以脱离ELK了。\n而OAP方面，CPU和内存的消耗远远低于预期的估计，CPU占用率一直较低，而分配的24G内存也仅使用了10+G，完全可以支持更大规模的接入量，不过在网络IO方面可能存在一定的风险，推测应该8C16G的容器就足以支持十万CPM级别的数据接入。\n当然我们在查询也遇到了一些瓶颈，最大的问题就是无法精确的命中某一条调用记录，就如前面的分析，因为segment的数据结构问题，无法进行面向业务的查询（例如messageId、requestId、orderId等），所以如果想精确匹配某一次调用请求，需要通过各个维度的条件约束慢慢缩小范围最后定位。\nSkywalking展望 通过上述对Skywalking的剖析和实践，Skywalking确实是一个优秀的APM+调用链跟踪监控系统，能够覆盖大部分使用场景，让研发和运维能够更加实时/准实时的了解线上服务的运行情况。当然Skywailking也不是尽善尽美，例如下面就是个人觉得目前可见的不满足我们期望的：\n数据准实时通过gRPC上报，本地缓存的瓶颈（当然官方主要是为了简化模型，减少依赖，否则Skywalking还依赖ELK就玩得有点大了） 缓存队列的长度，过长占据内存，过短容易buffer满丢弃数据 优雅停机同时又不丢失缓存 数据上报需要在起点上报，链路回传的时候需要携带SPAN及子SPAN的信息，当链路较长或者SPAN保存的信息较多时，会额外消耗一定的带宽 skywalking更多是一个APM系统而不是分布式调用链跟踪系统 在整个链路的探针上均缺少输入输出的抓取 在调用链的筛查上并没用进行增强，并且体现在数据结构的设计，例如TAG信息均保存在SPAN信息中，而SPAN信息均被BASE64编码作为数据保存，无法检索，最终trace的筛查只能通过时间/traceId/service/endPoint/state进行非业务相关的搜索 skywalking缺少对三方接口依赖的指标，这个对于系统稳定往往非常重要 而作为一个初级的使用者，个人觉得我们可以使用有限的人力在以下方向进行扩展：\n增加receiver：整合ELK，通过日志采集采集数据，降低异构系统的采集开发成本 优化数据结构，提供基于业务关键数据的查询接口 优化探针，采集更多的业务数据，争取代替传统的ELK日志简单查询，绝大部分异常诊断和定位均可以通过Skywalking即可完成 增加业务指标监控的模式，能够自定义业务指标（目前官方已经在实现 Metric Exporter ） ","excerpt":"\u003ch2 id=\"apm和调用链跟踪\"\u003eAPM和调用链跟踪\u003c/h2\u003e\n\u003cp\u003e随着企业经营规模的扩大，以及对内快速诊断效率和对外SLA（服务品质协议，service-level agreement)的追求，对于业务系统的掌控度的要求越来越高，主要体现在： …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2019-03-29-introduction-of-skywalking-and-simple-practice/","title":"SkyWalking调研与初步实践"},{"body":"前言 首先描述下问题的背景，博主有个习惯，每天上下班的时候看下skywalking的trace页面的error情况。但是某天突然发现生产环境skywalking页面没有任何数据了，页面也没有显示任何的异常，有点慌，我们线上虽然没有全面铺开对接skywalking，但是也有十多个应用。看了应用agent端日志后，其实也不用太担心，对应用毫无影响。大概情况就是这样，但是问题还是要解决，下面就开始排查skywalking不可用的问题。\n使用到的工具arthas Arthas是阿里巴巴开源的一款在线诊断java应用程序的工具，是greys工具的升级版本，深受开发者喜爱。当你遇到以下类似问题而束手无策时，Arthas可以帮助你解决：\n这个类从哪个 jar 包加载的？为什么会报各种类相关的 Exception？ 我改的代码为什么没有执行到？难道是我没 commit？分支搞错了？ 遇到问题无法在线上 debug，难道只能通过加日志再重新发布吗？ 线上遇到某个用户的数据处理有问题，但线上同样无法 debug，线下无法重现！ 是否有一个全局视角来查看系统的运行状况？ 有什么办法可以监控到JVM的实时运行状态？ Arthas采用命令行交互模式，同时提供丰富的 Tab 自动补全功能，进一步方便进行问题的定位和诊断。 项目地址：https://github.com/alibaba/arthas\n先定位问题一 查看skywalking-oap-server.log的日志，发现会有一条异常疯狂的在输出，异常详情如下：\n2019-03-01 09:12:11,578 - org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker -3264081149 [DataCarrier.IndicatorPersistentWorker.endpoint_inventory.Consumser.0.Thread] ERROR [] - Validation Failed: 1: id is too long, must be no longer than 512 bytes but was: 684; org.elasticsearch.action.ActionRequestValidationException: Validation Failed: 1: id is too long, must be no longer than 512 bytes but was: 684; at org.elasticsearch.action.ValidateActions.addValidationError(ValidateActions.java:26) ~[elasticsearch-6.3.2.jar:6.3.2] at org.elasticsearch.action.index.IndexRequest.validate(IndexRequest.java:183) ~[elasticsearch-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:515) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:348) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2] at org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.forceInsert(ElasticSearchClient.java:141) ~[library-client-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.RegisterEsDAO.forceInsert(RegisterEsDAO.java:66) ~[storage-elasticsearch-plugin-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.lambda$onWork$0(RegisterPersistentWorker.java:83) ~[server-core-6.0.0-alpha.jar:6.0.0-alpha] at java.util.HashMap$Values.forEach(HashMap.java:981) [?:1.8.0_201] at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.onWork(RegisterPersistentWorker.java:74) [server-core-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.access$100(RegisterPersistentWorker.java:35) [server-core-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker$PersistentConsumer.consume(RegisterPersistentWorker.java:120) [server-core-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.consume(ConsumerThread.java:101) [apm-datacarrier-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.run(ConsumerThread.java:68) [apm-datacarrier-6.0.0-alpha.jar:6.0.0-alpha] 2019-03-01 09:12:11,627 - org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker -3264081198 [DataCarrier.IndicatorPersistentWorker.endpoint_inventory.Consumser.0.Thread] ERROR [] - Validation Failed: 1: id is too long, must be no longer than 512 bytes but was: 684; org.elasticsearch.action.ActionRequestValidationException: Validation Failed: 1: id is too long, must be no longer than 512 bytes but was: 684; at org.elasticsearch.action.ValidateActions.addValidationError(ValidateActions.java:26) ~[elasticsearch-6.3.2.jar:6.3.2] at org.elasticsearch.action.index.IndexRequest.validate(IndexRequest.java:183) ~[elasticsearch-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:515) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:348) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2] at org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.forceInsert(ElasticSearchClient.java:141) ~[library-client-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.RegisterEsDAO.forceInsert(RegisterEsDAO.java:66) ~[storage-elasticsearch-plugin-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.lambda$onWork$0(RegisterPersistentWorker.java:83) ~[server-core-6.0.0-alpha.jar:6.0.0-alpha] at java.util.HashMap$Values.forEach(HashMap.java:981) [?:1.8.0_201] at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.onWork(RegisterPersistentWorker.java:74) [server-core-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.access$100(RegisterPersistentWorker.java:35) [server-core-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker$PersistentConsumer.consume(RegisterPersistentWorker.java:120) [server-core-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.consume(ConsumerThread.java:101) [apm-datacarrier-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.run(ConsumerThread.java:68) [apm-datacarrier-6.0.0-alpha.jar:6.0.0-alpha] 可以看到，上面的异常输出的时间节点，以这种频率在疯狂的刷新。通过异常message，得知到是因为skywalking在写elasticsearch时，索引的id太长了。下面是elasticsearch的源码：\nif (id != null \u0026amp;\u0026amp; id.getBytes(StandardCharsets.UTF_8).length \u0026gt; 512) { validationException = addValidationError(\u0026#34;id is too long, must be no longer than 512 bytes but was: \u0026#34; + id.getBytes(StandardCharsets.UTF_8).length, validationException); } 具体可见：elasticsearch/action/index/IndexRequest.java#L240\n问题一： 通过日志，初步定位是哪个系统的url太长，skywalking在注册url数据时触发elasticsearch针对索引id校验的异常，而skywalking注册失败后会不断的重试，所以才有了上面日志不断刷的现象。\n问题解决： elasticsearch client在写es前通过硬编码的方式写死了索引id的长度不能超过512字节大小。也就是我们不能通过从ES侧找解决方案了。回到异常的message，只能看到提示id太长，并没有写明id具体是什么，这个异常提示其实是不合格的，博主觉得应该把id的具体内容抛出来，问题就简单了。因为异常没有明确提示，系统又比较多，不能十多个系统依次关闭重启来验证到底是哪个系统的哪个url有问题。这个时候Arthas就派上用场了，在不重启应用不开启debug模式下，查看实例中的属性对象。下面通过Arthas找到具体的url。\n从异常中得知，org.elasticsearch.action.index.IndexRequest这个类的validate方法触发的，这个方法是没有入参的，校验的id属性其实是对象本身的属性，那么我们使用Arthas的watch指令来看下这个实例id属性。先介绍下watch的用法：\n功能说明 让你能方便的观察到指定方法的调用情况。能观察到的范围为：返回值、抛出异常、入参，通过编写 \bOGNL 表达式进行对应变量的查看。\n参数说明 watch 的参数比较多，主要是因为它能在 4 个不同的场景观察对象\n参数名称 参数说明 class-pattern 类名表达式匹配 method-pattern 方法名表达式匹配 express 观察表达式 condition-express 条件表达式 [b] 在方法调用之前观察 [e] 在方法异常之后观察 [s] 在方法返回之后观察 [f] 在方法结束之后(正常返回和异常返回)观察 [E] 开启正则表达式匹配，默认为通配符匹配 [x:] 指定输出结果的属性遍历深度，默认为 1 从上面的用法说明结合异常信息，我们得到了如下的指令脚本：\nwatch org.elasticsearch.action.index.IndexRequest validate \u0026ldquo;target\u0026rdquo;\n执行后，就看到了我们希望了解到的内容，如：\n索引id的具体内容看到后，就好办了。我们暂时把定位到的这个应用启动脚本中的的skywalking agent移除后（计划后面重新设计下接口）重启了下系统验证下。果然疯狂输出的日志停住了，但是问题并没完全解决，skywalking页面上的数据还是没有恢复。\n定位问题二 skywalking数据存储使用了elasticsearch，页面没有数据，很有可能是elasticsearch出问题了。查看elasticsearch日志后，发现elasticsearch正在疯狂的GC，日志如：\n: 139939K-\u0026gt;3479K(153344K), 0.0285655 secs] 473293K-\u0026gt;336991K(5225856K), 0.0286918 secs] [Times: user=0.05 sys=0.00, real=0.03 secs] 2019-02-28T20:05:38.276+0800: 3216940.387: Total time for which application threads were stopped: 0.0301495 seconds, Stopping threads took: 0.0001549 seconds 2019-02-28T20:05:38.535+0800: 3216940.646: [GC (Allocation Failure) 2019-02-28T20:05:38.535+0800: 3216940.646: [ParNew Desired survivor size 8716288 bytes, new threshold 6 (max 6) - age 1: 1220136 bytes, 1220136 total - age 2: 158496 bytes, 1378632 total - age 3: 88200 bytes, 1466832 total - age 4: 46240 bytes, 1513072 total - age 5: 126584 bytes, 1639656 total - age 6: 159224 bytes, 1798880 total : 139799K-\u0026gt;3295K(153344K), 0.0261667 secs] 473311K-\u0026gt;336837K(5225856K), 0.0263158 secs] [Times: user=0.06 sys=0.00, real=0.03 secs] 2019-02-28T20:05:38.562+0800: 3216940.673: Total time for which application threads were stopped: 0.0276971 seconds, Stopping threads took: 0.0001030 seconds 2019-02-28T20:05:38.901+0800: 3216941.012: [GC (Allocation Failure) 2019-02-28T20:05:38.901+0800: 3216941.012: [ParNew Desired survivor size 8716288 bytes, new threshold 6 (max 6) 问题二： 查询后得知，elasticsearch的内存配置偏大了，GC时间太长，导致elasticsearch脱离服务了。elasticsearch所在主机的内存是8G的实际内存7.6G,刚开始配置了5G的堆内存大小，可能Full GC的时候耗时太久了。查询elasticsearch官方文档后，得到如下的jvm优化建议：\n将最小堆大小（Xms）和最大堆大小（Xmx）设置为彼此相等。 Elasticsearch可用的堆越多，它可用于缓存的内存就越多。但请注意，过多的堆可能会使您陷入长时间的垃圾收集暂停。 设置Xmx为不超过物理RAM的50％，以确保有足够的物理RAM用于内核文件系统缓存。 不要设置Xmx为JVM用于压缩对象指针（压缩oops）的截止值之上; 确切的截止值变化但接近32 GB。 详情见：https://www.elastic.co/guide/en/elasticsearch/reference/6.5/heap-size.html\n问题解决： 根据Xmx不超过物理RAM的50％上面的jvm优化建议。后面将Xms和Xmx都设置成了3G。然后先停掉skywalking（由于skywalking中会缓存部分数据，如果直接先停ES，会报索引找不到的类似异常，这个大部分skywalking用户应该有遇到过），清空skywalking缓存目录下的内容，如：\n在重启elasticsearch，接着启动skywalking后页面终于恢复了\n结语 整个问题排查到解决大概花了半天时间，幸好一点也不影响线上应用的使用，这个要得益于skywalking的设计，不然就是大灾难了。然后要感谢下Arthas的技术团队，写了这么好用的一款产品并且开源了，如果没有Arthas，这个问题真的不好定位，甚至一度想到了换掉elasticsearch，采用mysql来解决索引id过长的问题。Arthas真的是线上找问题的利器，博主在Arthas刚面世的时候就关注了，并且一直在公司推广使用，在这里在硬推一波。\n作者简介： 陈凯玲，2016年5月加入凯京科技。曾任职高级研发和项目经理，现任凯京科技研发中心架构\u0026amp;运维部负责人。pmp项目管理认证，阿里云MVP。热爱开源，先后开源过多个热门项目。热爱分享技术点滴，独立博客KL博客（http://www.kailing.pub）博主。\n","excerpt":"\u003ch1 id=\"前言\"\u003e前言\u003c/h1\u003e\n\u003cp\u003e首先描述下问题的背景，博主有个习惯，每天上下班的时候看下skywalking的trace页面的error情况。但是某天突然发现生产环境skywalking页面没有任何数据了，页面也没有显示任何的 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2019-03-01-skywalking-troubleshoot/","title":"SkyWalking线上问题排查定位"},{"body":" 作者：王振飞, 写于：2019-02-24 说明：此文是个人所写，版本归属作者，代表个人观点，仅供参考，不代表skywalking官方观点。 说明：本次对比基于skywalking-6.0.0-GA和Pinpoint-1.8.2（截止2019-02-19最新版本）。另外，我们这次技术选型直接否定了Zipkin，其最大原因是它对代码有侵入性，CAT也是一样。这是我们所完全无法接受的。\n这应该是目前最优秀的两款开源APM产品了，而且两款产品都通过字节码注入的方式，实现了对代码完全无任何侵入，他们的对比信息如下：\nOAP说明: skywalking6.x才有OAP这个概念，skywalking5.x叫collector。\n接下来，对每个PK项进行深入分析和对比。更多精彩和首发内容请关注公众号：【阿飞的博客】。\n社区比较\n这一点上面skywalking肯定完胜。一方面，skywalking已经进入apache孵化，社区相当活跃。而且项目发起人是中国人，我们能够进入官方群（Apache SkyWalking交流群：392443393）和项目发起人吴晟零距离沟通，很多问题能第一时间得到大家的帮助（玩过开源的都知道，这个价值有多大）。 而Pinpoint是韩国人开发的，免不了有沟通障碍。至于github上最近一年的commit频率，skywalking和Pinpoint旗鼓相当，都是接近20的水平: 所以，社区方面，skywalking更胜一筹。\n支持语言比较 Pinpoint只支持Java和PHP，而skywalking支持5种语言：Java, C#, PHP, Node.js, Go。如果公司的服务涉及到多个开发语言，那么skywalking会是你更好的选择。并且，如果你要实现自己的探针（比如python语言），skywalking的二次开发成本也比Pinpoint更低。\n说明：Github上有开发者为Pinpoint贡献了对Node.js的支持，请戳链接：https://github.com/peaksnail/pinpoint-node-agent。但是已经停止维护，几年没更新了！\n所以，支持语言方面，skywalking更胜一筹。\n协议比较 SkyWalking支持gRPC和http，不过建议使用gRPC，skywalking6.x版本已经不提供http方式（但是还会保留接收5.x的数据），以后会考虑删除。 而Pinpoint使用的是thrift协议。 协议本身没有谁好谁坏。\n存储比较(重要) 笔者认为，存储是skywalking和Pinpoint最大的差异所在，因为底层存储决定了上层功能。\nPinpoint只支持HBase，且扩展代价较大。这就意味着，如果选择Pinpoint，还要有能力hold住一套HBase集群（daocloud从Pinpoint切换到skywalking就是因为HBase的维护代价有点大）。在这方面，skywalking支持的存储就多很多，这样的话，技术选型时可以根据团队技术特点选择合适的存储，而且还可以自行扩展（不过生产环境上应该大部分是以es存储为主）。\nPinpoint只支持HBase的另一个缺陷就是，HBase本身查询能力有限（HBase只能支持三种方式查询：RowKey精确查找，SCAN范围查找，全表扫描）限制了Pinpoint的查询能力，所以其支持的查询一定是在时间的基础上（Pinpoint通过鼠标圈定一个时间范围后查看这个范围内的Trace信息）。而skywalking可以多个维度任意组合查询，例如：时间范围，服务名，Trace状态，请求路径，TraceId等。\n另外，Pinpoint和skywalking都支持TTL，即历史数据保留策略。skywalking是在OAP模块的application.yml中配置从而指定保留时间。而Pinpoint是通过HBase的ttl功能实现，通过Pinpoint提供的hbase脚本https://github.com/naver/pinpoint/blob/master/hbase/scripts/hbase-create.hbase可以看到：ApplicationTraceIndex配置了TTL =\u0026gt; 5184000，SqlMetaData_Ver2配合了TTL =\u0026gt; 15552000，单位是秒。\n说明：es并不是完全碾压HBase，es和HBase没有绝对的好和坏。es强在检索能力，存储能力偏弱(千亿以下，es还是完全有能力hold的住的)。HBase强在存储能力，检索能力偏弱。如果搜集的日志量非常庞大，那么es存储就比较吃力。当然，没有蹩脚的中间件，只有蹩脚的程序员，无论是es还是HBase，调优才是最关键的。同样的，如果对检索能力有一定的要求，那么HBase肯定满足不了你。所以，又到了根据你的业务和需求决定的时刻了，trade-off真是无所不在。\nUI比较 Pinpoint的UI确实比skywalking稍微好些，尤其是服务的拓扑图展示。不过daocloud根据Pinpoint的风格为skywalking定制了一款UI。请戳链接：https://github.com/TinyAllen/rocketbot，项目介绍是：rocketbot: A UI for Skywalking。截图如下所示； 所以，只比较原生UI的话，Pinpoint更胜一筹。\n扩展性比较 Pinpoint好像设计之初就没有过多考虑扩展性，无论是底层的存储，还是自定义探针实现等。而skywalking核心设计目标之一就是Pluggable，即可插拔。\n以存储为例，pinpoint完全没有考虑扩展性，而skywalking如果要自定义实现一套存储，只需要定义一个类实现接口org.apache.skywalking.oap.server.library.module.ModuleProvider，然后实现一些DAO即可。至于Pinpoint则完全没有考虑过扩展底层存储。\n再以实现一个自己的探针为例（比如我要实现python语言的探针），Pinpoint选择thrift作为数据传输协议标准，而且为了节省数据传输大小，在传递常量的时候也尽量使用数据参考字典，传递一个数字而不是直接传递字符串等等。这些优化也增加了系统的复杂度：包括使用 Thrift 接口的难度、UDP 数据传输的问题、以及数据常量字典的注册问题等等。Pinpoint发展这么年才支持Java和PHP，可见一斑。而skywalking的数据接口就标准很多，并且支持OpenTracing协议，除了官方支持Java以外，C#、PHP和Node.js的支持都是由社区开发并维护。\n还有后面会提到的告警，skywalking的可扩展性也要远好于Pinpoint。\n最后，Pinpoint和skywalking都支持插件开发，Pinpoint插件开发参考：http://naver.github.io/pinpoint/1.8.2/plugindevguide.html。skywalking插件开发参考：https://github.com/apache/incubator-skywalking/blob/master/docs/en/guides/Java-Plugin-Development-Guide.md。\n所以，扩展性方面skywalking更胜一筹。\n告警比较 Pinpoint和skywalking都支持自定义告警规则。\n但是恼人的是，Pinpoint如果要配置告警规则，还需要安装MySQL(配置告警时的用户，用户组信息以及告警规则都持久化保存在MySQL中)，这就导致Pinpoint的维护成本又高了一些，既要维护HBase又要维护MySQL。\nPinpoint支持的告警规则有：SLOW COUNT|RATE, ERROR COUNT|RATE, TOTAL COUNT, SLOW COUNT|RATE TO CALLEE, ERROR COUNT|RATE TO CALLEE, ERROR RATE TO CALLEE, HEAP USAGE RATE, JVM CPU USAGE RATE, DATASOURCE CONNECTION USAGE RATE。\nPinpoint每3分钟周期性检查过去5分钟的数据，如果有符合规则的告警，就会发送sms/email给用户组下的所有用户。需要说明的是，实现发送sms/email的逻辑需要自己实现，Pinpoint只提供了接口com.navercorp.pinpoint.web.alarm.AlarmMessageSender。并且Pinpoint发现告警持续时，会递增发送sms/email的时间间隔 3min -\u0026gt; 6min -\u0026gt; 12min -\u0026gt; 24min，防止sms/email狂刷。\nPinpoint告警参考：http://naver.github.io/pinpoint/1.8.2/alarm.html\nskywalking配置告警不需要引入任何其他存储。skywalking在config/alarm-settings.xml中可以配置告警规则，告警规则支持自定义。\nskywalking支持的告警规则（配置项中的名称是indicator-name）有：service_resp_time, service_sla, service_cpm, service_p99, service_p95, service_p90, service_p75, service_p50, service_instance_sla, service_instance_resp_time, service_instance_cpm, endpoint_cpm, endpoint_avg, endpoint_sla, endpoint_p99, endpoint_p95, endpoint_p90, endpoint_p75, endpoint_p50。\nSkywalking通过HttpClient的方式远程调用在配置项webhooks中定义的告警通知服务地址。skywalking也支持silence-period配置，假设在TN这个时间点触发了告警，那么TN -\u0026gt; TN+period 这段时间内不会再重复发送该告警。\nskywalking告警参考：https://github.com/apache/incubator-skywalking/blob/master/docs/en/setup/backend/backend-alarm.md。目前只支持official_analysis.oal脚本中Service, Service Instance, Endpoint scope的metric，其他scope的metric需要等待后续扩展。\nPinpoint和skywalking都支持常用的告警规则配置，但是skywalking采用webhooks的方式就灵活很多：短信通知，邮件通知，微信通知都是可以支持的。而Pinpoint只能sms/email通知，并且还需要引入MySQL存储，增加了整个系统复杂度。所以，告警方面，skywalking更胜一筹。\nJVM监控 skywalking支持监控：Heap, Non-Heap, GC(YGC和FGC)。 Pinpoint能够监控的指标主要有：Heap, Non-Heap, FGC, DirectBufferMemory, MappedBufferMemory，但是没有YGC。另外，Pinpoint还支持多个指标同一时间点查看的功能。如下图所示：\n所以，对JVM的监控方面，Pinpoint更胜一筹。\n服务监控 包括操作系统，和部署的服务实例的监控。 Pinpoint支持的维度有：CPU使用率，Open File Descriptor，数据源，活动线程数，RT，TPS。 skywalking支持的维度有：CPU使用率，SLA，RT，CPM（Call Per Minutes）。 所以，这方面两者旗鼓相当，没有明显的差距。\n跟踪粒度比较 Pinpoint在这方面做的非常好，跟踪粒度非常细。如下图所示，是Pinpoint对某个接口的trace信息： 而同一个接口skywalking的trace信息如下图所示： 备注: 此截图是skywalking加载了插件apm-spring-annotation-plugin-6.0.0-GA.jar（这个插件允许跟踪加了@Bean, @Service, @Component and @Repository注解的spring context中的bean的方法）。\n通过对比发现，在跟踪粒度方面，Pinpoint更胜一筹。\n过滤追踪 Pinpoint和skywalking都可以实现，而且配置的表达式都是基于ant风格。 Pinpoint在Web UI上配置 filter wizard 即可自定义过滤追踪。 skywalking通过加载apm-trace-ignore-plugin插件就能自定义过滤跟踪，skywalking这种方式更灵活，比如一台高配服务器上有若干个服务，在共用的agent配置文件apm-trace-ignore-plugin.config中可以配置通用的过滤规则，然后通过-D的方式为每个服务配置个性化过滤。\n所以，在过滤追踪方面，skywalking更胜一筹。\n性能损耗 由于Pinpoint采集信息太过详细，所以，它对性能的损耗最大。而skywalking默认策略比较保守，对性能损耗很小。 有网友做过压力测试，对比如下：\n图片来源于：https://juejin.im/post/5a7a9e0af265da4e914b46f1\n所以，在性能损耗方面，skywalking更胜一筹。\n发布包比较 skywalking与时俱进，全系标配jar包，部署只需要执行start.sh脚本即可。而Pinpoint的collector和web还是war包，部署时依赖web容器（比如Tomcat）。拜托，都9012年了。\n所以，在发布包方面，skywalking更胜一筹。\n支持组件比较 skywalking和Pinpoint支持的中间件对比说明：\nWEB容器说明：Pinpoint支持几乎所有的WEB容器，包括开源和商业的。而wkywalking只支持开源的WEB容器，对2款大名鼎鼎的商业WEB容器Weblogic和Wevsphere都不支持。 RPC框架说明：对RPC框架的支持，skywalking简直秒杀Pinpoint。连小众的motan和sofarpc都支持。 MQ说明：skywalking比Pinpoint多支持一个国产的MQ中间件RocketMQ，毕竟RocketMQ在国内名气大，而在国外就一般了。加之skywalking也是国产的。 RDBMS/NoSQL说明：Pinpoint对RDBMS和NoSQL的支持都要略好于skywalking，RDBMS方面，skywalking不支持MSSQL和MariaDB。而NoSQL方面，skywalking不支持Cassandra和HBase。至于Pinpoint不支持的H2，完全不是问题，毕竟生产环境是肯定不会使用H2作为底层存储的。 Redis客户端说明：虽然skywalking和Pinpoint都支持Redis，但是skywalking支持三种流行的Redis客户端：Jedis，Redisson，Lettuce。而Pinpoint只支持Jedis和Lettuce，再一次，韩国人开发的Pinpoint无视了目前中国人开发的GitHub上star最多的Redis Client \u0026ndash; Redisson。 日志框架说明：Pinpoint居然不支持log4j2？但是已经有人开发了相关功能，详情请戳链接：log4j plugin support log4j2 or not? https://github.com/naver/pinpoint/issues/3055 通过对skywalking和Pinpoint支持中间件的对比我们发现，skywalking对国产软件的支持真的是全方位秒杀Pinpoint，比如小众化的RPC框架：motan（微博出品），sofarpc，阿里的RocketMQ，Redis客户端Redisson，以及分布式任务调度框架elastic-job等。当然也从另一方面反应国产开源软件在世界上的影响力还很小。\n这方面没有谁好谁坏，毕竟每个公司使用的技术栈不一样。如果你对RocketMQ有强需求，那么skywalking是你的最佳选择。如果你对es有强需求，那么skywalking也是你的最佳选择。如果HBase是你的强需求，那么Pinpoint就是你的最佳选择。如果MSSQL是你的强需求，那么Pinpoint也是你的最佳选择。总之，这里完全取决你的项目了。\n总结 经过前面对skywalking和Pinpoint全方位对比后我们发现，对于两款非常优秀的APM软件，有一种既生瑜何生亮的感觉。Pinpoint的优势在于：追踪数据粒度非常细、功能强大的用户界面，以及使用HBase作为存储带来的海量存储能力。而skywalking的优势在于：非常活跃的中文社区，支持多种语言的探针，对国产开源软件非常全面的支持，以及使用es作为底层存储带来的强大的检索能力，并且skywalking的扩展性以及定制化要更优于Pinpoint：\n如果你有海量的日志存储需求，推荐Pinpoint。 如果你更看重二次开发的便捷性，推荐skywalking。 最后，参考上面的对比，结合你的需求，哪些不能妥协，哪些可以舍弃，从而更好的选择一款最适合你的APM软件。\n参考链接 参考[1]. https://github.com/apache/incubator-skywalking/blob/master/docs/en/setup/service-agent/java-agent/Supported-list.md 参考[2]. http://naver.github.io/pinpoint/1.8.2/main.html#supported-modules 参考[3]. https://juejin.im/post/5a7a9e0af265da4e914b46f1 如果觉得本文不错，请关注作者公众号：【阿飞的博客】，多谢！\n","excerpt":"\u003cblockquote\u003e\n\u003cp\u003e作者：王振飞, 写于：2019-02-24\n\u003cstrong\u003e说明\u003c/strong\u003e：此文是个人所写，版本归属作者，代表个人观点，仅供参考，不代表skywalking官方观点。\n\u003cstrong\u003e说明\u003c/strong\u003e：本次对比基于skywalking-6.0.0-GA …\u003c/p\u003e\u003c/blockquote\u003e","ref":"https://skywalking.apache.org/zh/2019-02-24-skywalking-pk-pinpoint/","title":"APM巅峰对决：SkyWalking P.K. Pinpoint"},{"body":"According to Apache Software Foundation branding policy all docker images of Apache Skywalking should be transferred from skywalking to apache with a prefix skywalking-. The transfer details are as follows\nskywalking/base -\u0026gt; apache/skywalking-base skywalking/oap -\u0026gt; apache/skywalking-oap-server skywalking/ui -\u0026gt; apache/skywalking-ui All of repositories in skywalking will be removed after one week.\n","excerpt":"\u003cp\u003eAccording to Apache Software Foundation branding policy all docker images of Apache Skywalking …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/transfer-docker-images-to-apache-official-repository/","title":"Transfer Docker Images to Apache Official Repository"},{"body":"6.0.0-GA release. Go to downloads page to find release tars. This is an important milestone version, we recommend all users upgrade to this version.\nKey updates\nBug fixed Register bug fix, refactor and performance improvement New trace UI ","excerpt":"\u003cp\u003e6.0.0-GA release. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\nThis is an important milestone version, …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-6-0-0-ga/","title":"Release Apache SkyWalking APM 6.0.0-GA"},{"body":"Based on his contributions to the project, he has been accepted as SkyWalking PPMC. Welcome aboard.\n","excerpt":"\u003cp\u003eBased on his contributions to the project, he has been accepted as SkyWalking PPMC. Welcome aboard.\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-jian-tan-as-a-new-ppmc/","title":"Welcome Jian Tan as a new PPMC"},{"body":" Author: Hongtao Gao, Apache SkyWalking \u0026amp; ShardingShpere PMC GitHub, Twitter, Linkedin Service mesh receiver was first introduced in Apache SkyWalking 6.0.0-beta. It is designed to provide a common entrance for receiving telemetry data from service mesh framework, for instance, Istio, Linkerd, Envoy etc. What’s the service mesh? According to Istio’s explain:\nThe term service mesh is used to describe the network of microservices that make up such applications and the interactions between them.\nAs a PMC member of Apache SkyWalking, I tested trace receiver and well understood the performance of collectors in trace scenario. I also would like to figure out the performance of service mesh receiver.\nDifferent between trace and service mesh Following chart presents a typical trace map:\nYou could find a variety of elements in it just like web service, local method, database, cache, MQ and so on. But service mesh only collect service network telemetry data that contains the entrance and exit data of a service for now(more elements will be imported soon, just like Database). A smaller quantity of data is sent to the service mesh receiver than the trace.\nBut using sidecar is a little different.The client requesting “A” that will send a segment to service mesh receiver from “A”’s sidecar. If “A” depends on “B”, another segment will be sent from “A”’s sidecar. But for a trace system, only one segment is received by the collector. The sidecar model splits one segment into small segments, that will increase service mesh receiver network overhead.\nDeployment Architecture In this test, I will pick two different backend deployment. One is called mini unit, consist of one collector and one elasticsearch instance. Another is a standard production cluster, contains three collectors and three elasticsearch instances.\nMini unit is a suitable architecture for dev or test environment. It saves your time and VM resources, speeds up depolyment process.\nThe standard cluster provides good performance and HA for a production scenario. Though you will pay more money and take care of the cluster carefully, the reliability of the cluster will be a good reward to you.\nI pick 8 CPU and 16GB VM to set up the test environment. This test targets the performance of normal usage scenarios, so that choice is reasonable. The cluster is built on Google Kubernetes Engine(GKE), and every node links each other with a VPC network. For running collector is a CPU intensive task, the resource request of collector deployment should be 8 CPU, which means every collector instance occupy a VM node.\nTesting Process Receiving mesh fragments per second(MPS) depends on the following variables.\nIngress query per second(QPS) The topology of a microservice cluster Service mesh mode(proxy or sidecar) In this test, I use Bookinfo app as a demo cluster.\nSo every request will touch max 4 nodes. Plus picking the sidecar mode(every request will send two telemetry data), the MPS will be QPS * 4 *2.\nThere are also some important metrics that should be explained\nClient Query Latency: GraphQL API query response time heatmap. Client Mesh Sender: Send mesh segments per second. The total line represents total send amount and the error line is the total number of failed send. Mesh telemetry latency: service mesh receiver handling data heatmap. Mesh telemetry received: received mesh telemetry data per second. Mini Unit You could find collector can process up to 25k data per second. The CPU usage is about 4 cores. Most of the query latency is less than 50ms. After login the VM on which collector instance running, I know that system load is reaching the limit(max is 8).\nAccording to the previous formula, a single collector instance could process 3k QPS of Bookinfo traffic.\nStandard Cluster Compare to the mini-unit, cluster’s throughput increases linearly. Three instances provide total 80k per second processing power. Query latency increases slightly, but it’s also very small(less than 500ms). I also checked every collector instance system load that all reached the limit. 10k QPS of BookInfo telemetry data could be processed by the cluster.\nConclusion Let’s wrap them up. There are some important things you could get from this test.\nQPS varies by the there variables. The test results in this blog are not important. The user should pick property value according to his system. Collector cluster’s processing power could scale out. The collector is CPU intensive application. So you should provide sufficient CPU resource to it. This blog gives people a common method to evaluate the throughput of Service Mesh Receiver. Users could use this to design their Apache Skywalking backend deployment architecture.\n","excerpt":"\u003cul\u003e\n\u003cli\u003eAuthor: Hongtao Gao, Apache SkyWalking \u0026amp; ShardingShpere PMC\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/hanahmily\"\u003eGitHub\u003c/a\u003e, \u003ca href=\"https://twitter.com/hanahmily\"\u003eTwitter\u003c/a\u003e, \u003ca href=\"https://www.linkedin.com/in/gao-hongtao-47b835168/\"\u003eLinkedin\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eService …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2019-01-25-mesh-loadtest/","title":"SkyWalking performance in Service Mesh scenario"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/development/","title":"Development"},{"body":"ps:本文仅写给菜鸟，以及不知道如何远程调试的程序员，并且仅仅适用skywalking的远程调试\n概述 远程调试的目的是为了解决代码或者说程序包部署在服务器上运行，只能通过log来查看问题，以及不能跟在本地IDE运行debug那样查找问题，观看程序运行流程\u0026hellip; 想想当你的程序运行在服务器上，你在本地的IDE随时debug，是不是很爽的感觉。\n好了不废话，切入正题。\n环境篇 IDE：推荐 IntelliJ IDEA\n开发语言: 本文仅限于java，其他语言请自行询问google爸爸或者baidu娘娘\n源代码：自行从github下载，并且确保你运行的skywalking包也源代码的一致，（也就是说你自己从源代码编译打包运行，虽然不一样也可以调试，但是你想想你在本地开发，更改完代码，没有重新运行，debug出现的诡异情况）\n场景篇 假定有如下三台机器\nIP 用途 备注 10.193.78.1 oap-server skywalking 的oap服务（或者说collector所在的服务器） 10.193.78.2 agent skywalking agent运行所在的服务器 10.193.78.0 IDE 你自己装IDE也就是IntelliJ IDEA的机器 以上环境，场景请自行安装好，并确认正常运行。本文不在赘述\n废话终于说完了\n操作篇 首要条件，下载源码后，先用maven 打包编译。然后使用Idea打开源码的父目录，整体结构大致如下图 1 :agent调试 1)Idea 配置部分 点击Edit Configurations 在弹出窗口中依次找到（红色线框的部分）并点击 打开的界面如下 修改Name值，自己随意，好记即可 然后Host输入10.193.78.2 Port默认或者其他的，重要的是这个端口在10.193.78.2上没有被占用\n然后找到Use module classpath 选择 apm-agent 最终的结果如下： 注意选择目标agent运行的jdk版本，很重要\n然后点击Apply，并找到如下内容，并且复制待用 2）agent配置部分 找到agent配置的脚本，并打开，找到配置agent的地方， 就这个地方，在这个后边加上刚才复制的内容 最终的结果如下 提供一个我配置的weblogic的配置（仅供参考） 然后重启应用（agent）\n3）调试 回到Idea中找到这个地方，并点击debug按钮，你没看错，就是红色圈住的地方 然后控制台如果出现以下字样： 那么恭喜你，可以愉快的加断点调试了。 ps:需要注意的是agent的、 service instance的注册可能不能那么愉快的调试。因为这个注册比较快，而且是在agent启动的时候就发生的， 而远程调试也需要agent打开后才可以调试，所以，如果你手快当我没说这句话。\n2 :oap-server的调试（也就是collector的调试） 具体过程不在赘述，和上一步的agent调试大同小异，不同的是 Use module classpath需要选择oap-server\n","excerpt":"\u003cp\u003eps:本文仅写给菜鸟，以及不知道如何远程调试的程序员，并且仅仅适用skywalking的远程调试\u003c/p\u003e\n\u003ch2 id=\"概述\"\u003e概述\u003c/h2\u003e\n\u003cp\u003e远程调试的目的是为了解决代码或者说程序包部署在服务器上运行，只能通过log来查看问题，以及不能跟 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2019-01-24-skywalking-remote-debug/","title":"SkyWalking的远程调试"},{"body":"引言 《SkyWalking Java 插件贡献实践》：本文将基于SkyWalking 6.0.0-GA-SNAPSHOT版本，以编写Redis客户端Lettuce的SkyWalking Java Agent 插件为例，与大家分享我贡献PR的过程，希望对大家了解SkyWalking Java Agent插件有所帮助。\n基础概念 OpenTracing和SkyWalking链路模块几个很重要的语义概念。\nSpan:可理解为一次方法调用，一个程序块的调用，或一次RPC/数据库访问。只要是一个具有完整时间周期的程序访问，都可以被认为是一个span。SkyWalking Span对象中的重要属性\n属性 名称 备注 component 组件 插件的组件名称，如：Lettuce，详见:ComponentsDefine.Class。 tag 标签 k-v结构，关键标签，key详见：Tags.Class。 peer 对端资源 用于拓扑图，若DB组件，需记录集群信息。 operationName 操作名称 若span=0，operationName将会搜索的下拉列表。 layer 显示 在链路页显示，详见SpanLayer.Class。 Trace:调用链，通过归属于其的Span来隐性的定义。一条Trace可被认为是一个由多个Span组成的有向无环图（DAG图），在SkyWalking链路模块你可以看到，Trace又由多个归属于其的trace segment组成。\nTrace segment:Segment是SkyWalking中的一个概念，它应该包括单个OS进程中每个请求的所有范围，通常是基于语言的单线程。由多个归属于本线程操作的Span组成。\n核心API 跨进程ContextCarrier核心API 为了实现分布式跟踪，需要绑定跨进程的跟踪，并且应该传播上下文 整个过程。 这就是ContextCarrier的职责。 以下是实现有关跨进程传播的步骤： 在客户端，创建一个新的空的ContextCarrier，将ContextCarrier所有信息放到HTTP heads、Dubbo attachments 或者Kafka messages。 通过服务调用，将ContextCarrier传递到服务端。 在服务端，在对应组件的heads、attachments或messages获取ContextCarrier所有消息。将服务端和客户端的链路信息绑定。 跨线程ContextSnapshot核心API 除了跨进程，跨线程也是需要支持的，例如异步线程（内存中的消息队列）和批处理在Java中很常见，跨进程和跨线程十分相似，因为都是需要传播 上下文。 唯一的区别是，不需要跨线程序列化。 以下是实现有关跨线程传播的步骤： 使用ContextManager＃capture获取ContextSnapshot对象。 让子线程以任何方式，通过方法参数或由现有参数携带来访问ContextSnapshot。 在子线程中使用ContextManager#continued。 详尽的核心API相关知识，可点击阅读 《插件开发指南-中文版本》\n插件实践 Lettuce操作redis代码 @PostMapping(\u0026#34;/ping\u0026#34;) public String ping(HttpServletRequest request) throws ExecutionException, InterruptedException { RedisClient redisClient = RedisClient.create(\u0026#34;redis://\u0026#34; + \u0026#34;127.0.0.1\u0026#34; + \u0026#34;:6379\u0026#34;); StatefulRedisConnection\u0026lt;String, String\u0026gt; connection0 = redisClient.connect(); RedisAsyncCommands\u0026lt;String, String\u0026gt; asyncCommands0 = connection0.async(); AsyncCommand\u0026lt;String, String, String\u0026gt; future = (AsyncCommand\u0026lt;String, String, String\u0026gt;)asyncCommands0.set(\u0026#34;key_a\u0026#34;, \u0026#34;value_a\u0026#34;); future.onComplete(s -\u0026gt; OkHttpClient.call(\u0026#34;http://skywalking.apache.org\u0026#34;)); future.get(); connection0.close(); redisClient.shutdown(); return \u0026#34;pong\u0026#34;; } 插件源码架构 Lettuce对Redis封装与Redisson Redisson 类似，目的均是实现简单易用，且无学习曲线的Java的Redis客户端。所以要是先对Redis操作的拦截，需要学习对应客户端的源码。\n设计插件 理解插件实现过程，找到最佳InterceptPoint位置是实现插件融入SkyWalking的核心所在。\n代码实现 PR的url：Support lettuce plugin\n实践中遇到的问题 多线程编程使用debug断点会将链路变成同步，建议使用run模式增加log，或者远程debug来解决。 多线程编程，需要使用跨线程ContextSnapshot核心API，否则链路会断裂。 CompleteableCommand.onComplete方法有时会同步执行，这个和内部机制有关，有时候不分离线程。 插件编译版本若为1.7+，需要将插件放到可选插件中。因为sniffer支持的版本是1.6。 插件兼容 为了插件得到插件最终的兼容兼容版本，我们需要使用docker对所有插件版本的测试，具体步骤如下：\n编写测试用例：关于如何编写测试用例，请按照如何编写文档来实现。 提供自动测试用例。 如：Redisson插件testcase 确保本地几个流行的插件版本，在本地运行起来是和自己的预期是一致的。 在提供自动测试用例并在CI中递交测试后，插件提交者会批准您的插件。 最终得到完整的插件测试报告。 Pull Request 提交PR 提交PR的时候，需要简述自己对插件的设计，这样有助于与社区的贡献者讨论完成codereview。\n申请自动化测试 测试用例编写完成后，可以申请自动化测试，在自己的PR中会生成插件兼容版本的报告。\n插件文档 插件文档需要更新：Supported-list.md相关插件信息的支持。\n插件如果为可选插件需要在agent-optional-plugins可选插件文档中增加对应的描述。\n注释 Lettuce是一个完全无阻塞的Redis客户端，使用netty构建，提供反应，异步和同步数据访问。了解细节可点击阅读 lettuce.io;\nOpenTracing是一个跨编程语言的标准，了解细节可点击阅读 《OpenTracing语义标准》;\nspan:org.apache.skywalking.apm.agent.core.context.trace.AbstractSpan接口定义了所有Span实现需要完成的方法;\nRedisson是一个非常易用Java的Redis客户端， 它没有学习曲线，无需知道任何Redis命令即可开始使用它。了解细节可点击阅读 redisson.org;\n","excerpt":"\u003ch2 id=\"引言\"\u003e引言\u003c/h2\u003e\n\u003cp\u003e《SkyWalking Java 插件贡献实践》：本文将基于SkyWalking 6.0.0-GA-SNAPSHOT版本，以编写Redis客户端\u003ca href=\"#Lettuce\"\u003e\u003ccode\u003eLettuce\u003c/code\u003e\u003c/a\u003e的SkyWalking Java …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2019-01-21-agent-plugin-practice/","title":"SkyWalking Java 插件贡献实践"},{"body":"Jinlin Fu has contributed 4 new plugins, including gson, activemq, rabbitmq and canal, which made SkyWalking supporting all mainstream OSS MQ. Also provide several documents and bug fixes. The SkyWalking PPMC based on these, promote him as new committer. Welcome on board.\n","excerpt":"\u003cp\u003eJinlin Fu has contributed 4 new plugins, including gson, activemq, rabbitmq and canal, which made …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-jinlin-fu-as-new-committer/","title":"Welcome Jinlin Fu as new committer"},{"body":" 作者：赵瑞栋 原文地址 引言 微服务框架落地后，分布式部署架构带来的问题就会迅速凸显出来。服务之间的相互调用过程中，如果业务出现错误或者异常，如何快速定位问题？如何跟踪业务调用链路？如何分析解决业务瓶颈？\u0026hellip;本文我们来看看如何解决以上问题。\n一、SkyWalking初探 Skywalking 简介 Skywalking是一款国内开源的应用性能监控工具，支持对分布式系统的监控、跟踪和诊断。\n它提供了如下的主要功能特性： Skywalking 技术架构 SW总体可以分为四部分：\n1.Skywalking Agent：使用Javaagent做字节码植入，无侵入式的收集，并通过HTTP或者gRPC方式发送数据到Skywalking Collector。\nSkywalking Collector ：链路数据收集器，对agent传过来的数据进行整合分析处理并落入相关的数据存储中。 Storage：Skywalking的存储，时间更迭，sw已经开发迭代到了6.x版本，在6.x版本中支持以ElasticSearch、Mysql、TiDB、H2、作为存储介质进行数据存储。 UI ：Web可视化平台，用来展示落地的数据。 Skywalking Agent配置 通过了解配置，可以对一个组件功能有一个大致的了解。让我们一起看一下skywalking的相关配置。\n解压开skywalking的压缩包，在agent/config文件夹中可以看到agent的配置文件。\n从skywalking支持环境变量配置加载，在启动的时候优先读取环境变量中的相关配置。\nagent.namespace: 跨进程链路中的header，不同的namespace会导致跨进程的链路中断 agent.service_name:一个服务（项目）的唯一标识，这个字段决定了在sw的UI上的关于service的展示名称 agent.sample_n_per_3_secs: 客户端采样率，默认是-1代表全采样 agent.authentication: 与collector进行通信的安全认证，需要同collector中配置相同 agent.ignore_suffix: 忽略特定请求后缀的trace collecttor.backend_service: agent需要同collector进行数据传输的IP和端口 logging.level: agent记录日志级别 skywalking agent使用javaagent无侵入式的配合collector实现对分布式系统的追踪和相关数据的上下文传递。\nSkywalking Collector关键配置 Collector支持集群部署，zookeeper、kubernetes（如果你的应用是部署在容器中的）、consul（GO语言开发的服务发现工具）是sw可选的集群管理工具，结合大家具体的部署方式进行选择。详细配置大家可以去Skywalking官网下载介质包进行了解。\nCollector端口设置\ndownsampling: 采样汇总统计维度，会分别按照分钟、【小时、天、月】（可选）来统计各项指标数据。 通过设置TTL相关配置项可以对数据进行自动清理。 Skywalking 在6.X中简化了配置。collector提供了gRPC和HTTP两种通信方式。\nUI使用rest http通信，agent在大多数场景下使用grpc方式通信，在语言不支持的情况下会使用http通信。\n关于绑定IP和端口需要注意的一点是，通过绑定IP，agent和collector必须配置对应ip才可以正常通信。\nCollector存储配置\n在application.yml中配置的storage模块配置中选择要使用的数据库类型，并填写相关的配置信息。\nCollector Receiver\nReceiver是Skywalking在6.x提出的新的概念，负责从被监控的系统中接受指标数据。用户完全可以参照OpenTracing规范来上传自定义的监控数据。Skywalking官方提供了service-mesh、istio、zipkin的相关能力。\n现在Skywalking支持服务端采样，配置项为sampleRate，比例采样，如果配置为5000则采样率就是50%。\n关于采样设置的一点注意事项\n关于服务采样配置的一点建议，如果Collector以集群方式部署，比如：Acollector和Bcollector，建议Acollector.sampleRate = Bcollector.sampleRate。如果采样率设置不相同可能会出现数据丢失问题。\n假设Agent端将所有数据发送到后端Collector处，A采样率设置为30%，B采样率为50%。\n假设有30%的数据，发送到A上，这些数据被全部正确接受并存储，极端情况（与期望的采样数据量相同）下，如果剩下20%待采样的数据发送到了B，这个时候一切都是正常的，如果这20%中有一部分数据被送到了A那么，这些数据将是被忽略的，由此就会造成数据丢失。\n二、业务调用链路监控 Service Topology监控 调用链路监控可以从两个角度去看待。我们先从整体上来认识一下我们所监控的系统。\n通过给服务添加探针并产生实际的调用之后，我们可以通过Skywalking的前端UI查看服务之间的调用关系。\n我们简单模拟一次服务之间的调用。新建两个服务，service-provider以及service-consumer，服务之间简单的通过Feign Client 来模拟远程调用。\n从图中可以看到:\n有两个服务节点：provider \u0026amp; consumer 有一个数据库节点：localhost【mysql】 一个注册中心节点 consumer消费了provider提供出来的接口。\n一个系统的拓扑图让我们清晰的认识到系统之间的应用的依赖关系以及当前状态下的业务流转流程。细心的可能发现图示节点consumer上有一部分是红色的，红色是什么意思呢？\n红色代表当前流经consumer节点的请求有一断时间内是响应异常的。当节点全部变红的时候证明服务现阶段内就彻底不可用了。运维人员可以通过Topology迅速发现某一个服务潜在的问题，并进行下一步的排查并做到预防。\nSkywalking Trace监控 Skywalking通过业务调用监控进行依赖分析，提供给我们了服务之间的服务调用拓扑关系、以及针对每个endpoint的trace记录。\n我们在之前看到consumer节点服务中发生了错误，让我们一起来定位下错误是发生在了什么地方又是什么原因呢？\n在每一条trace的信息中都可以看到当前请求的时间、GloableId、以及请求被调用的时间。我们分别看一看正确的调用和异常的调用。\nTrace调用链路监控 图示展示的是一次正常的响应，这条响应总耗时19ms，它有4个span：\nspan1 /getStore = 19ms 响应的总流转时间 span2 /demo2/stores = 14ms feign client 开始调用远程服务后的响应的总时间 span3 /stores = 14ms 接口服务响应总时间 span4 Mysql = 1ms 服务提供端查询数据库的时间 这里span2和span3的时间表现相同，其实是不同的，因为这里时间取了整。\n在每个Span中可以查看当前Span的相关属性。\n组件类型: SpringMVC、Feign Span状态: false HttpMethod: GET Url: http://192.168.16.125:10002/demo2/stores 这是一次正常的请求调用Trace日志，可能我们并不关心正常的时候，毕竟一切正常不就是我们期待的么！\n我们再来看下，异常状态下我们的Trace以及Span又是什么样的呢。\n发生错误的调用链中Span中的is error标识变为true，并且在名为Logs的TAB中可以看到错误发生的具体原因。根据异常情况我们就可以轻松定位到影响业务的具体原因，从而快速定位问题，解决问题。\n通过Log我们看到连接被拒，那么可能是我们的网络出现了问题（可能性小，因为实际情况如果网络出现问题我们连这个trace都看不到了），也有可能是服务端配置问题无法正确建立连接。通过异常日志，我们迅速就找到了问题的关键。\n实际情况是，我把服务方停掉了，做了一次简单的模拟。可见，通过拓扑图示我们可以清晰的看到众多服务中哪个服务是出现了问题的，通过trace日志我们可以很快就定位到问题所在，在最短的时间内解决问题。\n三、服务性能指标监控 Skywalking还可以查看具体Service的性能指标，根据相关的性能指标可以分析系统的瓶颈所在并提出优化方案。\nSkywalking 性能监控 在服务调用拓扑图上点击相应的节点我们可以看到该服务的\nSLA: 服务可用性（主要是通过请求成功与失败次数来计算） CPM: 每分钟调用次数 Avg Response Time: 平均响应时间 从应用整体外部来看我们可以监测到应用在一定时间段内的\n服务可用性指标SLA 每分钟平均响应数 平均响应时间 服务进程PID 服务所在物理机的IP、HostName、Operation System Service JVM信息监控 还可以监控到Service运行时的CPU、堆内存、非堆内存使用率、以及GC情况。这些信息来源于JVM。注意这里的数据可不是机器本身的数据。\n四、服务告警 前文我们提到了通过查看拓扑图以及调用链路可以定位问题，可是运维人员又不可能一直盯着这些数据，那么我们就需要告警能力，在异常达到一定阈值的时候主动的提示我们去查看系统状态。\n在Sywalking 6.x版本中新增了对服务状态的告警能力。它通过webhook的方式让我们可以自定义我们告警信息的通知方式。诸如:邮件通知、微信通知、短信通知等。\nSkywalking 服务告警 先来看一下告警的规则配置。在alarm-settings.xml中可以配置告警规则，告警规则支持自定义。\n一份告警配置由以下几部分组成：\nservice_resp_time_rule：告警规则名称 ***_rule （规则名称可以自定义但是必须以’_rule’结尾 indicator-name：指标数据名称： 定义参见http://t.cn/EGhfbmd op: 操作符： \u0026gt; , \u0026lt; , = 【当然你可以自己扩展开发其他的操作符】 threshold：目标值：指标数据的目标数据 如sample中的1000就是服务响应时间，配合上操作符就是大于1000ms的服务响应 period: 告警检查周期：多久检查一次当前的指标数据是否符合告警规则 counts: 达到告警阈值的次数 silence-period：忽略相同告警信息的周期 message：告警信息 webhooks：服务告警通知服务地址 Skywalking通过HttpClient的方式远程调用在配置项webhooks中定义的告警通知服务地址。\n了解了SW所传送的数据格式我们就可以对告警信息进行接收处理，实现我们需要的告警通知服务啦！\n我们将一个服务停掉，并将另外一个服务的某个对外暴露的接口让他休眠一定的时间。然后调用一定的次数观察服务的状态信息以及告警情况。\n总结 本文简单的通过skwaylking的配置来对skywlaking的功能进行一次初步的了解，对skwaylking新提出的概念以及新功能进行简单的诠释，方便大家了解和使用。通过使用APM工具，可以让我们方便的查看微服务架构中系统瓶颈以及性能问题等。\n精选提问 问1：想问问选型的时候用pinpoint还是SK好？\n答：选型问题\n要结合具体的业务场景， 比如你的代码运行环境 是java、php、net还是什么。 pinpoint在安装部署上要比skywalking略微复杂 pinpoint和sw支持的组件列表是不同的。 https://github.com/apache/incubator-skywalking/blob/master/docs/en/setup/service-agent/java-agent/Supported-list.md你可以参照这里的支持列表对比下pinpoint的支持对象做一个简单对比。 sw经过测试在并发量较高的情况下比pinpoint的吞吐量更好一些。 问2：有没有指标统计，比如某个url 的top10 请求、响应最慢的10个请求？某个服务在整个链条中的耗时占比？\n答：1.sw自带有响应最慢的请求top10统计针对所有的endpoint的统计。 2.针对每个url的top10统计，sw本身没有做统计，数据都是现成的通过简单的检索就可以搜到你想要的结果。 3.没有具体的耗时占比，但是有具体总链路时间统计以及某个服务的耗时统计，至于占比自己算吧，可以看ppt中的调用链路监控的span时间解释。\n问3：能不能具体说一下在你们系统中的应用？\n答：EOS8LA版本中，我们整合sw对应用提供拓扑、调用链路、性能指标的监控、并在sw数据的基础上增加系统的维度。 当服务数很庞大的时候，整体的拓扑其实就是一张密密麻麻的蜘蛛网。我们可以通过系统来选择具体某个系统下的应用。 8LA中SW是5.0.0alpha版本，受限于sw功能，我们并没有提供告警能力，这在之后会是我们的考虑目标。\n问4：业务访问日志大概每天100G，kubernetes 环境中部署，使用稳定吗？\n答：监控数据没有长时间的存储必要，除非你有特定的需求。它有一定的时效性，你可以设置ttl自动清除过时信息。100g，es集群还是能轻松支撑的。\n问5：和pinpoint相比有什么优势吗？\n答：\n部署方式、使用方式简单 功能特性支持的更多 高并发性能会更好一些 问6：skywalking的侵入式追踪功能方便进行单服务链的服务追踪。但是跨多台服务器多项目的整体服务链追踪是否有整体设计考虑？\n答：sw本身特性就是对分布式系统的追踪，他是无侵入式的。无关你的应用部署在多少台服务器上。\n问7：应用在加上代理之后性能会下降。请问您有什么解决方法吗？\n答：性能下降是在所难免的，但是据我了解，以及官方的测试，他的性能影响是很低的。这是sw的测试数据供你参考。 https://skywalkingtest.github.io/Agent-Benchmarks/README_zh.html。\n问8：有异构系统需求的话可以用sw吗？\n答：只要skywalking的探针支持的应该都是可以的。\n问9：sw对于商用的web中间件，如bes、tongweb、websphere、weblogic的支持如何？\n答：商业组件支持的比较少，因为涉及到相关license的问题，sw项目组需要获得他们的支持来进行数据上报，据我了解，支持不是很好。\n","excerpt":"\u003cul\u003e\n\u003cli\u003e作者：赵瑞栋\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://mp.weixin.qq.com/s/0XXUpnxR8xiExE4iwu90xg\"\u003e原文地址\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"引言\"\u003e引言\u003c/h2\u003e\n\u003cp\u003e微服务框架落地后，分布式部署架构带来的问题就会迅速凸显出来。服务之间的相互调用过程中，如果业务出现错误或者异常，如何快速定位问题？如何跟踪业务调用链路？如何分析解决业 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2019-01-03-monitor-microservice/","title":"SkyWalking 微服务监控分析"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/elasticsearch/","title":"ElasticSearch"},{"body":"SkyWalking 依赖 elasticsearch 集群，如果 elasticsearch 安装有 x-pack 插件的话，那么就会存在一个 Basic 认证，导致 skywalking 无法调用 elasticsearch, 解决方法是使用 nginx 做代理，让 nginx 来做这个 Basic 认证，那么这个问题就自然解决了。\n方法如下:\n安装 nginx yum install -y nginx\n配置 nginx server { listen 9200 default_server; server_name _; location / { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_pass http://localhost:9200; #Basic字符串就是使用你的用户名(admin),密码(12345)编码后的值 #注意:在进行Basic加密的时候要使用如下格式如:admin:123456 注意中间有个冒号 proxy_set_header Authorization \u0026#34;Basic YWRtaW4gMTIzNDU2\u0026#34;; } } 验证 curl localhost:9200\n{ \u0026#34;name\u0026#34; : \u0026#34;Yd0rCp9\u0026#34;, \u0026#34;cluster_name\u0026#34; : \u0026#34;es-cn-4590xv9md0009doky\u0026#34;, \u0026#34;cluster_uuid\u0026#34; : \u0026#34;jAPLrqY5R6KWWgHnGCWOAA\u0026#34;, \u0026#34;version\u0026#34; : { \u0026#34;number\u0026#34; : \u0026#34;6.3.2\u0026#34;, \u0026#34;build_flavor\u0026#34; : \u0026#34;default\u0026#34;, \u0026#34;build_type\u0026#34; : \u0026#34;tar\u0026#34;, \u0026#34;build_hash\u0026#34; : \u0026#34;053779d\u0026#34;, \u0026#34;build_date\u0026#34; : \u0026#34;2018-07-20T05:20:23.451332Z\u0026#34;, \u0026#34;build_snapshot\u0026#34; : false, \u0026#34;lucene_version\u0026#34; : \u0026#34;7.3.1\u0026#34;, \u0026#34;minimum_wire_compatibility_version\u0026#34; : \u0026#34;5.6.0\u0026#34;, \u0026#34;minimum_index_compatibility_version\u0026#34; : \u0026#34;5.0.0\u0026#34; }, \u0026#34;tagline\u0026#34; : \u0026#34;You Know, for Search\u0026#34; } 看到如上结果那么恭喜你成功了。\n","excerpt":"\u003cp\u003eSkyWalking 依赖 elasticsearch 集群，如果 elasticsearch 安装有 x-pack 插件的话，那么就会存在一个 Basic 认证，导致 skywalking 无法调用 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2019-01-02-skywalking-elasticsearch-basic/","title":"关于 ElasticSearch 因 basic 认证导致 SkyWalking 无法正常调用接口问题"},{"body":" 作者: Wu Sheng, tetrate, SkyWalking original creator GitHub, Twitter, Linkedin 翻译: jjlu521016 背景 在当前的微服务架构中分布式链路追踪是很有必要的一部分，但是对于一些用户来说如何去理解和使用分布式链路追踪的相关数据是不清楚的。 这个博客概述了典型的分布式跟踪用例，以及Skywalking的V6版本中新的可视化功能。我们希望新的用户通过这些示例来更好的理解。\n指标和拓扑图 跟踪数据支持两个众所周知的分析特性：指标和拓扑图\n指标: 每个service, service instance, endpoint的指标都是从跟踪中的入口span派生的。指标代表响应时间的性能。所以可以有一个平均响应时间，99%的响应时间，成功率等。它们按service, service instance, endpoint进行分解。\n拓扑图: 拓扑表示服务之间的链接，是分布式跟踪最有吸引力的特性。拓扑结构允许所有用户理解分布式服务关系和依赖关系，即使它们是不同的或复杂的。这一点很重要，因为它为所有相关方提供了一个单一的视图，无论他们是开发人员、设计者还是操作者。\n这里有一个拓扑图的例子包含了4个项目，包括kafka和两个外部依赖。\n-在skywalking的可选择UI0RocketBot的拓扑图-\nTrace 在分布式链路追踪系统中，我们花费大量资源（CPU、内存、磁盘和网络）来生成、传输和持久跟踪数据。让我们试着回答为什么要这样做？我们可以用跟踪数据回答哪些典型的诊断和系统性能问题？\nSkywalking v6包含两种追踪视图:\nTreeMode: 第一次提供,帮助您更容易识别问题。 ListMode: 常规的时间线视图，通常也出现在其他跟踪系统中，如Zipkin。 发生错误 在trace视图，最简单的部分是定位错误，可能是由代码异常或网络故障引起的。通过span详情提供的细节，ListMode和TreeMode都能够找到错误 -ListMode 错误span-\n-TreeMode 错误span-\n慢span 一个高优先级的特性是识别跟踪中最慢的span。这将使用应用程序代理捕获的执行持续时间。在旧的ListMode跟踪视图中，由于嵌套，父span几乎总是包括子span的持续时间。换句话说，一个缓慢的span通常会导致它的父节点也变慢，在Skywalking 6中，我们提供了 最慢的前5个span 过滤器来帮助你您直接定位span。\n-最慢的前5个span-\n太多子span 在某些情况下，个别持续时间很快，但跟踪速度仍然很慢，如： -没有慢span的追踪-\n如果要了解根问题是否与太多操作相关，请使用子范围号的Top 5 of children span number,筛选器显示每个span的子级数量，突出显示前5个。 -13个数据库访问相关的span-\n在这个截图中，有一个包含13个子项的span，这些子项都是数据库访问。另外，当您看到跟踪的概述时，这个2000ms跟踪的数据库花费了1380ms。 -1380ms花费在数据库访问-\n在本例中，根本原因是数据库访问太多。这在其他场景中也很常见，比如太多的RPC或缓存访问。\n链路深度 跟踪深度也与延迟有关。像太多子span的场景一样，每个span延迟看起来不错，但整个链路追踪的过程很慢。 -链路深度-\n上图所示,最慢的span小鱼500ms,对于2000毫秒的跟踪来说，速度并不太慢。当您看到第一行时，有四种不同的颜色表示这个分布式跟踪中涉及的四个services。每一个都需要100~400ms，这四个都需要近2000ms，从这里我们知道这个缓慢的跟踪是由一个序列中的3个RPC造成的。\n结束语 分布式链路追踪和APM 工具帮助我们确定造成问题的根源，允许开发和操作团队进行相应的优化。我们希望您喜欢这一点，并且喜欢Apache Skywalking和我们的新链路追踪可视化界面。如果你喜欢的话，在github上面给我们加start来鼓励我们\nSkywakling 6计划在2019年的1月底完成release。您可以通过以下渠道联系项目团队成员\n关注 skywalking推特 订阅邮件:dev@skywalking.apache.org。发送邮件到 dev-subscribe@kywalking.apache.org 来订阅. 加入Gitter聊天室 ","excerpt":"\u003cul\u003e\n\u003cli\u003e作者: Wu Sheng, tetrate, SkyWalking original creator\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/wu-sheng\"\u003eGitHub\u003c/a\u003e, \u003ca href=\"https://twitter.com/wusheng1108\"\u003eTwitter\u003c/a\u003e, \u003ca href=\"https://www.linkedin.com/in/wusheng1108\"\u003eLinkedin\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e翻译: jjlu521016\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch1 id=\"背景\"\u003e背景\u003c/h1\u003e\n\u003cp\u003e在当前 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2019-01-02-understand-trace-trans2cn/","title":"更容易理解将要到来的分布式链路追踪 6.0GA (翻译)"},{"body":"Background Distributed tracing is a necessary part of modern microservices architecture, but how to understand or use distributed tracing data is unclear to some end users. This blog overviews typical distributed tracing use cases with new visualization features in SkyWalking v6. We hope new users will understand more through these examples.\nMetric and topology Trace data underpins in two well known analysis features: metric and topology\nMetric of each service, service instance, endpoint are derived from entry spans in trace. Metrics represent response time performance. So, you could have average response time, 99% response time, success rate, etc. These are broken down by service, service instance, endpoint.\nTopology represents links between services and is distributed tracing\u0026rsquo;s most attractive feature. Topologies allows all users to understand distributed service relationships and dependencies even when they are varied or complex. This is important as it brings a single view to all interested parties, regardless of if they are a developer, designer or operator.\nHere\u0026rsquo;s an example topology of 4 projects, including Kafka and two outside dependencies.\nTopology in SkyWalking optional UI, RocketBot\nTrace In a distributed tracing system, we spend a lot of resources(CPU, Memory, Disk and Network) to generate, transport and persistent trace data. Let\u0026rsquo;s try to answer why we do this? What are the typical diagnosis and system performance questions we can answer with trace data?\nSkyWalking v6 includes two trace views:\nTreeMode: The first time provided. Help you easier to identify issues. ListMode: Traditional view in time line, also usually seen in other tracing system, such as Zipkin. Error occurred In the trace view, the easiest part is locating the error, possibly caused by a code exception or network fault. Both ListMode and TreeMode can identify errors, while the span detail screen provides details.\nListMode error span\nTreeMode error span\nSlow span A high priority feature is identifying the slowest spans in a trace. This uses execution duration captured by application agents. In the old ListMode trace view, parent span almost always includes the child span\u0026rsquo;s duration, due to nesting. In other words, a slow span usually causes its parent to also become slow. In SkyWalking 6, we provide Top 5 of slow span filter to help you locate the spans directly.\nTop 5 slow span\nThe above screenshot highlights the top 5 slow spans, excluding child span duration. Also, this shows all spans\u0026rsquo; execution time, which helps identify the slowest ones.\nToo many child spans In some cases, individual durations are quick, but the trace is still slow, like this one:\nTrace with no slow span\nTo understand if the root problem is related to too many operations, use Top 5 of children span number. This filter shows the amount of children each span has, highlighting the top 5.\n13 database accesses of a span\nIn this screenshot, there is a span with 13 children, which are all Database accesses. Also, when you see overview of trace, database cost 1380ms of this 2000ms trace.\n1380ms database accesses\nIn this example, the root cause is too many database accesses. This is also typical in other scenarios like too many RPCs or cache accesses.\nTrace depth Trace depth is also related latency. Like the too many child spans scenario, each span latency looks good, but the whole trace is slow.\nTrace depth\nHere, the slowest spans are less than 500ms, which are not too slow for a 2000ms trace. When you see the first line, there are four different colors representing four services involved in this distributed trace. Every one of them costs 100~400ms. For all four, there nearly 2000ms. From here, we know this slow trace is caused by 3 RPCs in a serial sequence.\nAt the end Distributed tracing and APM tools help users identify root causes, allowing development and operation teams to optimize accordingly. We hope you enjoyed this, and love Apache SkyWalking and our new trace visualization. If so, give us a star on GitHub to encourage us.\nSkyWalking 6 is scheduled to release at the end of January 2019. You can contact the project team through the following channels:\nFollow SkyWalking twitter Subscribe mailing list: dev@skywalking.apache.org . Send to dev-subscribe@kywalking.apache.org to subscribe the mail list. Join Gitter room. ","excerpt":"\u003ch2 id=\"background\"\u003eBackground\u003c/h2\u003e\n\u003cp\u003eDistributed tracing is a necessary part of modern microservices architecture, but how to …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2019-01-01-understand-trace/","title":"Understand distributed trace easier in the incoming 6-GA"},{"body":"6.0.0-beta release. Go to downloads page to find release tars.\nKey updates\nBugs fixed, closed to GA New protocols provided, old still compatible. Spring 5 supported MySQL and TiDB as optional storage ","excerpt":"\u003cp\u003e6.0.0-beta release. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e\n\u003cp\u003eKey updates\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eBugs fixed, closed to GA …\u003c/li\u003e\u003c/ol\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-6-0-0-beta/","title":"Release Apache SkyWalking APM 6.0.0-beta"},{"body":"Based on his contributions. Including created RocketBot as our secondary UI, new website and very cool trace view page in next release. he has been accepted as SkyWalking PPMC. Welcome aboard.\n","excerpt":"\u003cp\u003eBased on his contributions. Including created \u003ca href=\"https://github.com/TinyAllen/rocketbot\"\u003eRocketBot\u003c/a\u003e as our secondary UI, new \u003ca href=\"http://skywalking.apache.org/\"\u003ewebsite\u003c/a\u003e and very …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-yao-wang-as-a-new-ppmc/","title":"Welcome Yao Wang as a new PPMC"},{"body":"导读 SkyWalking 中 Java 探针是使用 JavaAgent 的两大字节码操作工具之一的 Byte Buddy（另外是 Javassist）实现的。项目还包含.Net core 和 Nodejs 自动探针，以及 Service Mesh Istio 的监控。总体上，SkyWalking 是一个多语言，多场景的适配，特别为微服务、云原生和基于容器架构设计的可观测性分析平台（Observability Analysis Platform）。 本文基于 SkyWalking 5.0.0-RC2 和 Byte Buddy 1.7.9 版本，会从以下几个章节，让大家掌握 SkyWalking Java 探针的使用，进而让 SkyWalking 在自己公司中的二次开发变得触手可及。 Byte Buddy 实现 JavaAgent 项目 迭代 JavaAgent 项目的方法论 SkyWalking agent 项目如何 Debug SkyWalking 插件开发实践 文章底部有 SkyWalking 和 Byte Buddy 相应的学习资源。 Byte Buddy 实现 首先如果你对 JavaAgent 还不是很了解可以先百度一下，或在公众号内看下《JavaAgent 原理与实践》简单入门下。 SpringMVC 分发请求的关键方法相信已经不用我在赘述了，那我们来编写 Byte Buddy JavaAgent 代码吧。 public class AgentMain { public static void premain(String agentOps, Instrumentation instrumentation) { new AgentBuilder.Default() .type(ElementMatchers.named(\u0026#34;org.springframework.web.servlet.DispatcherServlet\u0026#34;)) .transform((builder, type, classLoader, module) -\u0026gt; builder.method(ElementMatchers.named(\u0026#34;doDispatch\u0026#34;)) .intercept(MethodDelegation.to(DoDispatchInterceptor.class))) .installOn(instrumentation); } } 编写 DispatcherServlet doDispatch 拦截器代码（是不是跟 AOP 如出一辙） public class DoDispatchInterceptor { @RuntimeType public static Object intercept(@Argument(0) HttpServletRequest request, @SuperCall Callable\u0026lt;?\u0026gt; callable) { final StringBuilder in = new StringBuilder(); if (request.getParameterMap() != null \u0026amp;\u0026amp; request.getParameterMap().size() \u0026gt; 0) { request.getParameterMap().keySet().forEach(key -\u0026gt; in.append(\u0026#34;key=\u0026#34; + key + \u0026#34;_value=\u0026#34; + request.getParameter(key) + \u0026#34;,\u0026#34;)); } long agentStart = System.currentTimeMillis(); try { return callable.call(); } catch (Exception e) { System.out.println(\u0026#34;Exception :\u0026#34; + e.getMessage()); return null; } finally { System.out.println(\u0026#34;path:\u0026#34; + request.getRequestURI() + \u0026#34; 入参:\u0026#34; + in + \u0026#34; 耗时:\u0026#34; + (System.currentTimeMillis() - agentStart)); } } } resources/META-INF/MANIFEST.MF Manifest-Version: 1.0 Premain-Class: com.z.test.agent.AgentMain Can-Redefine-Classes: true pom.xml 文件 dependencies +net.bytebuddy.byte-buddy +javax.servlet.javax.servlet-api *scope=provided plugins +maven-jar-plugin *manifestFile=src/main/resources/META-INF/MANIFEST.MF +maven-shade-plugin *include:net.bytebuddy:byte-buddy:jar: +maven-compiler-plugin 小结：没几十行代码就完成了，通过 Byte Buddy 实现应用组件 SpringMVC 记录请求路径、入参、执行时间 JavaAgent 项目，是不是觉得自己很优秀。 持续迭代 JavaAgent 本章节主要介绍 JavaAgent 如何 Debug，以及持续集成的方法论。 首先我的 JavaAgent 项目目录结构如图所示: 应用项目是用几行代码实现的 SpringBootWeb 项目: @SpringBootApplication(scanBasePackages = {\u0026#34;com\u0026#34;}) public class TestBootWeb { public static void main(String[] args) { SpringApplication.run(TestBootWeb.class, args); } @RestController public class ApiController { @PostMapping(\u0026#34;/ping\u0026#34;) public String ping(HttpServletRequest request) { return \u0026#34;pong\u0026#34;; } } } 下面是关键 JavaAgent 项目如何持续迭代与集成: VM options增加:-JavaAgent:{$HOME}/Code/github/z_my_test/test-agent/target/test-agent-1.0-SNAPSHOT.jar=args Before launch 在Build之前增加： Working directory:{$HOME}/Code/github/incubator-skywalking Command line:-T 1C -pl test-agent -am clean package -Denforcer.skip=true -Dmaven.test.skip=true -Dmaven.compile.fork=true 小结：看到这里的将 JavaAgent 持续迭代集成方法，是不是瞬间觉得自己手心已经发痒起来，很想编写一个自己的 agent 项目了呢，等等还有一个好消息:test-demo 这 10 几行的代码实现的 Web 服务，居然有 5k 左右的类可以使用 agent 增强。 注意 mvn 编译加速的命令是 maven3 + 版本以上才支持的哈。 SkyWalking Debug 峰回路转，到了文章的主题《SkyWalking 之高级用法》的正文啦。首先，JavaAgent 项目想 Debug，还需要将 agent 代码与接入 agent 项目至少在同一个工作空间内，网上方法有很多，这里我推荐大家一个最简单的方法。File-\u0026gt;New-\u0026gt;Module from Exisiting Sources… 引入 skywalking-agent 源码即可 详细的 idea 编辑器配置： 优化 SkyWalking agent 编译时间，我的集成时间优化到 30 秒左右： VM options增加:-JavaAgent:-JavaAgent:{$HOME}/Code/github/incubator-skywalking/skywalking-agent/skywalking-agent.jar：不要用dist里面的skywalking-agent.jar，具体原因大家可以看看源码：apm-sniffer/apm-agent/pom.xml中的maven插件的使用。 Before launch 在Build之前增加： Working directory:{$HOME}/Code/github/incubator-skywalking Command line:-T 1C -pl apm-sniffer/apm-sdk-plugin -amd clean package -Denforcer.skip=true -Dmaven.test.skip=true -Dmaven.compile.fork=true： 这里我针对插件包，因为紧接着下文要开发插件 另外根pom注释maven-checkstyle-plugin也可加速编译 kob 之 SkyWalking 插件编写 kob（贝壳分布式作业调度框架）是贝壳找房项目微服务集群中的基础组件，通过编写贝壳分布式作业调度框架的 SkyWalking 插件，可以实时收集作业调度任务的执行链路信息，从而及时得到基础组件的稳定性，了解细节可点击阅读《贝壳分布式调度框架简介》。想详细了解 SkyWalking 插件编写可在文章底部参考链接中，跳转至对应的官方资源，好话不多说，代码一把唆起来。 apm-sdk-plugin pom.xml 增加自己的插件 model \u0026lt;artifactId\u0026gt;apm-sdk-plugin\u0026lt;/artifactId\u0026gt; \u0026lt;modules\u0026gt; \u0026lt;module\u0026gt;kob-plugin\u0026lt;/module\u0026gt; ... \u0026lt;modules\u0026gt; resources.skywalking-plugin.def 增加自己的描述 kob=org.apache.skywalking.apm.plugin.kob.KobInstrumentation 在 SkyWalking 的项目中，通过继承 ClassInstanceMethodsEnhancePluginDefine 可以定义需要拦截的类和增强的方法，编写作业调度方法的 instrumentation public class KobInstrumentation extends ClassInstanceMethodsEnhancePluginDefine { private static final String ENHANCE_CLASS = \u0026#34;com.ke.kob.client.spring.core.TaskDispatcher\u0026#34;; private static final String INTERCEPT_CLASS = \u0026#34;org.apache.skywalking.apm.plugin.kob.KobInterceptor\u0026#34;; @Override protected ClassMatch enhanceClass() { return NameMatch.byName(ENHANCE_CLASS); } @Override protected ConstructorInterceptPoint[] getConstructorsInterceptPoints() { return null; } @Override protected InstanceMethodsInterceptPoint[] getInstanceMethodsInterceptPoints() { return new InstanceMethodsInterceptPoint[] { new InstanceMethodsInterceptPoint() { @Override public ElementMatcher\u0026lt;MethodDescription\u0026gt; getMethodsMatcher() { return named(\u0026#34;dispatcher1\u0026#34;); } @Override public String getMethodsInterceptor() { return INTERCEPT_CLASS; } @Override public boolean isOverrideArgs() { return false; } } }; } } 通过实现 InstanceMethodsAroundInterceptor 后，定义 beforeMethod、afterMethod 和 handleMethodException 的实现方法，可以环绕增强指定目标方法，下面自定义 interceptor 实现 span 的跟踪（这里需要注意 SkyWalking 中 span 的生命周期，在 afterMethod 方法中结束 span） public class KobInterceptor implements InstanceMethodsAroundInterceptor { @Override public void beforeMethod(EnhancedInstance objInst, Method method, Object[] allArguments, Class\u0026lt;?\u0026gt;[] argumentsTypes, MethodInterceptResult result) throws Throwable { final ContextCarrier contextCarrier = new ContextCarrier(); com.ke.kob.client.spring.model.TaskContext context = (TaskContext) allArguments[0]; CarrierItem next = contextCarrier.items(); while (next.hasNext()) { next = next.next(); next.setHeadValue(JSON.toJSONString(context.getUserParam())); } AbstractSpan span = ContextManager.createEntrySpan(\u0026#34;client:\u0026#34;+allArguments[1]+\u0026#34;,task:\u0026#34;+context.getTaskKey(), contextCarrier); span.setComponent(ComponentsDefine.TRANSPORT_CLIENT); SpanLayer.asRPCFramework(span); } @Override public Object afterMethod(EnhancedInstance objInst, Method method, Object[] allArguments, Class\u0026lt;?\u0026gt;[] argumentsTypes, Object ret) throws Throwable { ContextManager.stopSpan(); return ret; } @Override public void handleMethodException(EnhancedInstance objInst, Method method, Object[] allArguments, Class\u0026lt;?\u0026gt;[] argumentsTypes, Throwable t) { } } 实现效果，将操作名改成任务执行节点 + 任务执行方法，实现 kob 的 SkyWalking 的插件编写，加上报警体系，可以进一步增加公司基础组件的稳定性。 参考链接 Apache SkyWalking Byte Buddy（runtime code generation for the Java virtual machine） ","excerpt":"\u003ch2 id=\"导读\"\u003e导读\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003eSkyWalking 中 Java 探针是使用 JavaAgent 的两大字节码操作工具之一的 Byte Buddy（另外是 Javassist）实现的。项目还包含.Net core 和 …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/zh/2018-12-21-skywalking-apm-sniffer-beginning/","title":"SkyWalking apm-sniffer 原理学习与插件编写"},{"body":"搭建调试环境 阅读 SkyWalking 源码，从配置调试环境开始。\n一定一定一定不要干读代码，而是通过调试的方式。\n01 通过 Skywalking-5.x 版本的源码构建并运行 👉：哔哩哔哩 | 腾讯视频 02 通过 Skywalking-6.x 版本的源码构建并运行 👉：哔哩哔哩 | 腾讯视频 03 Java 应用（探针）接入 Skywalking[6.x] 👉：哔哩哔哩 | 腾讯视频 SkyWalking 3.X 源码解析合集 虽然是基于 3.X 版本的源码解析，但是对于阅读 SkyWalking Java Agent 和插件部分，同样适用。\n对于 SkyWalking Collector 部分，可以作为一定的参考。\n《SkyWalking 源码分析 —— 调试环境搭建》 《SkyWalking 源码分析 —— Agent 初始化》 《SkyWalking 源码分析 —— Agent 插件体系》 《SkyWalking 源码分析 —— Collector 初始化》 《SkyWalking 源码分析 —— Collector Cluster 集群管理》 《SkyWalking 源码分析 —— Collector Client Component 客户端组件》 《SkyWalking 源码分析 —— Collector Server Component 服务器组件》 《SkyWalking 源码分析 —— Collector Jetty Server Manager》 《SkyWalking 源码分析 —— Collector gRPC Server Manager》 《SkyWalking 源码分析 —— Collector Naming Server 命名服务》 《SkyWalking 源码分析 —— Collector Queue 队列组件》 《SkyWalking 源码分析 —— Collector Storage 存储组件》 《SkyWalking 源码分析 —— Collector Streaming Computing 流式处理（一）》 《SkyWalking 源码分析 —— Collector Streaming Computing 流式处理（二）》 《SkyWalking 源码分析 —— Collector Cache 缓存组件》 《SkyWalking 源码分析 —— Collector Remote 远程通信服务》 《SkyWalking 源码分析 —— DataCarrier 异步处理库》 《SkyWalking 源码分析 —— Agent Remote 远程通信服务》 《SkyWalking 源码分析 —— 应用于应用实例的注册》 《SkyWalking 源码分析 —— Agent DictionaryManager 字典管理》 《SkyWalking 源码分析 —— Agent 收集 Trace 数据》 《SkyWalking 源码分析 —— Agent 发送 Trace 数据》 《SkyWalking 源码分析 —— Collector 接收 Trace 数据》 《SkyWalking 源码分析 —— Collector 存储 Trace 数据》 《SkyWalking 源码分析 —— JVM 指标的收集与存储》 《SkyWalking 源码分析 —— 运维界面（一）之应用视角》 《SkyWalking 源码分析 —— 运维界面（二）之应用实例视角》 《SkyWalking 源码分析 —— 运维界面（三）之链路追踪视角》 《SkyWalking 源码分析 —— 运维界面（四）之操作视角》 《SkyWalking 源码分析 —— @Trace 注解想要追踪的任何方法》 《SkyWalking 源码分析 —— traceId 集成到日志组件》 《SkyWalking 源码分析 —— Agent 插件（一）之 Tomcat》 《SkyWalking 源码分析 —— Agent 插件（二）之 Dubbo》 《SkyWalking 源码分析 —— Agent 插件（三）之 SpringMVC》 《SkyWalking 源码分析 —— Agent 插件（四）之 MongoDB》 SkyWalking 6.X 源码解析合集 《SkyWalking 6.x 源码分析 —— 调试环境搭建》 ","excerpt":"\u003ch2 id=\"搭建调试环境\"\u003e搭建调试环境\u003c/h2\u003e\n\u003cp\u003e阅读 SkyWalking 源码，从配置调试环境开始。\u003c/p\u003e\n\u003cp\u003e一定一定一定不要干读代码，而是通过调试的方式。\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/JaredTan95/skywalking-tutorials/blob/master/01-%E9%80%9A%E8%BF%87Skywalking-5.x%E7%89%88%E6%9C%AC%E7%9A%84%E6%BA%90%E7%A0%81%E6%9E%84%E5%BB%BA%E5%B9%B6%E8%BF%90%E8%A1%8C/Note.md\"\u003e01 通过 Skywalking-5.x 版本的源码构建并运行\u003c/a\u003e 👉：\u003ca href=\"https://www.bilibili.com/video/av35806851/\"\u003e哔哩哔哩\u003c/a\u003e | …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/zh/2018-12-21-skywalking-source-code-read/","title":"SkyWalking 源码解析合集"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/source-code/","title":"Source Code"},{"body":"版本选择 我们采用的是 5.0.0-RC2 的版本，SkyWalking 的版本信息可以参考 https://github.com/apache/incubator-skywalking/blob/5.x/CHANGES.md\n那么为什么我们没有采用 5.1.0 版本呢，这是因为我们公司内部需要支持 es x-pack，但是在官方发布里面，没有支持 xpack 的版本。\n在 Apache SkyWalking 官方文档 https://github.com/CharlesMaster/incubator-skywalking/tree/master/docs/others/cn 中有提到，SkyWalking 5.x 仍受社区支持。\n对于用户计划从 5.x 升级到 6.x，您应该知道关于有一些概念的定义的变更。最重要的两个改变了的概念是：\nApplication（在 5.x 中）更改为 Service（在 6.x 中），Application Instance 也更改为 Service Instance。 Service（在 5.x 中）更改为 Endpoint（在 6.x 中）。 图文详解 Apache SkyWalking 的监控界面由 Monitor 和 Trace 两者构成，Monitor 菜单又包括 Dashbord、Topology、Application、Service、Alarm 五个子菜单构成。本文就是围绕这些菜单分别逐一进行介绍。\nMonitor 当用户通过 SkyWalking 登陆界面使用用户名、密码登陆以后，就会默认进入到 SkyWalking 的 Monitor 下的 Dashboard 界面\nDashboard 下图就是用户登陆之后都会看到的关键 Dashboard 页面，在这个页面的下方的关键指标，图中都做了详细的解释。\n上图中 app 需要强调的是，52 个 app 并不代表 52 个应用，比如 paycenter 有两台 paycenter1 和 paycenter2 就算了 2 个 app，当然还有一些应用是 3 个以上的。在我们公司，paycenter1、paycenter2 这些运维都和我们跳板机管理平台上的名称设置的一样，约定大于配置，开发人员可以更加便捷的排查问题。\n再次修正一下，关于 dashboard 页面的 app 数，语言类探针，是探针的 app_code 来决定的。比如我们公司的线上配置就是 agent.application_code=auth-center-1\n上图中需要解释两个概念：\ncpm 代表每分钟请求次数 SLA=(TRANSACTION_CALLS- TRANSACTION_ERROR_CALLS ) * 10000 ) / TRANSACTION_CALLS 该页面主要支持四个跳转：\n一、在上图中，App 板块上的帮助选项是可以直接跳转到 Application 监控页面的。 二、 Service 板块上的帮助选项是可以直接跳转到 Service 监控页面的。\n三、 Slow Service 列表中的每一个慢服务点击以后都会进入到其专项的 Service 监控页面。\n四、 Application Throughput 列表中的每一个 Application 点击以后也都是可以进入到其专项的 Application 监控页面。\n关于 Application 和 Service 的详细介绍我们后续会展开\n在 Dashboard 的页面上部分，还有一个选择功能模块： 左侧部分可以定期 refresh Dashboard 的数据，右侧则可以调整整体的查询区间。\nTopology 点击 Monitor 菜单下的 Topology 你会看到下面这张拓扑图\n当然这张图太过于夸张了，如果接入 SkyWalking 的应用并不是很多，会如下图所示： 左侧的三个小按钮可以调整你的视图，支持拖拽。右侧可以输入你所关心的应用名。比如我们输入一个支付和订单两个应用，左侧的拓扑图会变得更加清晰：\n另外，上图中的绿色圆圈都是可以点击的，如果你点击以后，还会出现节点信息： Application 点击 Monitor 菜单下的 Application 你会看到下面这张图，这张图里你可以看到的东西都做了注解。\n这张图里有一个惊喜，就是如果你点开 More Server Details，你可以看到更多的信息\n是的，除了 Host、IPv4、Pid、OS 以外，你还可以看到 CPU、Heap、Non-Heap、GC（Young GC、Old GC）等详细监控信息。\nService 点击 Monitor 菜单下的 Service 你会看到下面这张图，这张图里你可以看到的同样都做了注解。 关于 Dependency Map 这张图我们再补充一下，鼠标悬停可以看到每个阶段的执行时间，这是 Service 下的功能 我们点开图中该图中 Top 20 Slow Traces 下面的被我马赛克掉的 trace 的按钮框，可以看到如下更加详细的信息：\n这些信息可以帮助我们知道每一个方法在哪个阶段那个具体实现耗时了多久。\n如上图所示，每一行基本都是可以打开的，每一行都包含了 Tags、Logs 等监控内容\nAlarm 点击 Monitor 菜单下的 Alarm 你会看到告警菜单。目前 5.X 版本的还没有接入邮件、短信等告警方式，后续 6 支持 webhook，用户可以自己去接短信和邮件。\n告警内容中你可以看到 Applicaion、Server 和 Service 三个层面的告警内容\nTrace Trace 是一个非常实用的功能，用户可以根据精确的 TraceId 去查找\n也可以设定时间段去查找\n我在写使用手册时候，非常巧的是，看到了上图三起异常，于是我们往下拉列表看到了具体的数据\n点击进去，我们可以看到具体的失败原因 当然用户也可以直接将 Trace State 调整为 Error 级别进行查询\n再回顾一遍 一、首先我们进入首页：\n二、点击一下首页的 Slow Service 的 projectC，可以看到如下信息：\n三、如果点击首页的 Appliation Throughput 中的 projectD，可以看到如下信息：\n四、继续点进去右下角的这个 slow service 里的 Consumer，我们可以看到下图：\n参考资料 https://twitter.com/AsfSkyWalking/status/1013616673218179072 https://twitter.com/AsfSkyWalking/status/1013617100143800320 ","excerpt":"\u003ch2 id=\"版本选择\"\u003e版本选择\u003c/h2\u003e\n\u003cp\u003e我们采用的是 5.0.0-RC2 的版本，SkyWalking 的版本信息可以参考 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2018-12-18-apache-skywalking-5-0-userguide/","title":"Apache SkyWalking 5.0 中文版图文详解使用手册"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/web-ui/","title":"Web UI"},{"body":"Based on his contributions to the project, he has been accepted as SkyWalking committer. Welcome aboard.\n","excerpt":"\u003cp\u003eBased on his contributions to the project, he has been accepted as SkyWalking committer. Welcome …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-yixiong-cao-as-a-new-committer/","title":"Welcome Yixiong Cao as a new committer"},{"body":"Original link, Tetrate.io blog\nContext The integration of SkyWalking and Istio Service Mesh yields an essential open-source tool for resolving the chaos created by the proliferation of siloed, cloud-based services.\nApache SkyWalking is an open, modern performance management tool for distributed services, designed especially for microservices, cloud native and container-based (Docker, K8s, Mesos) architectures. We at Tetrate believe it is going to be an important project for understanding the performance of microservices. The recently released v6 integrates with Istio Service Mesh and focuses on metrics and tracing. It natively understands the most common language runtimes (Java, .Net, and NodeJS). With its new core code, SkyWalking v6 also supports Istrio telemetry data formats, providing consistent analysis, persistence, and visualization.\nSkyWalking has evolved into an Observability Analysis Platform that enables observation and monitoring of hundreds of services all at once. It promises solutions for some of the trickiest problems faced by system administrators using complex arrays of abundant services: Identifying why and where a request is slow, distinguishing normal from deviant system performance, comparing apples-to-apples metrics across apps regardless of programming language, and attaining a complete and meaningful view of performance.\nSkyWalking History Launched in China by Wu Sheng in 2015, SkyWalking started as just a distributed tracing system, like Zipkin, but with auto instrumentation from a Java agent. This enabled JVM users to see distributed traces without any change to their source code. In the last two years, it has been used for research and production by more than 50 companies. With its expanded capabilities, we expect to see it adopted more globally.\nWhat\u0026rsquo;s new Service Mesh Integration Istio has picked up a lot of steam as the framework of choice for distributed services. Based on all the interest in the Istio project, and community feedback, some SkyWalking (P)PMC members decided to integrate with Istio Service Mesh to move SkyWalking to a higher level.\nSo now you can use Skywalking to get metrics and understand the topology of your applications. This works not just for Java, .NET and Node using our language agents, but also for microservices running under the Istio service mesh. You can get a full topology of both kinds of applications.\nObservability analysis platform With its roots in tracing, SkyWalking is now transitioning into an open-standards based Observability Analysis Platform, which means the following:\nIt can accept different kinds and formats of telemetry data from mesh like Istio telemetry. Its agents support various popular software technologies and frameworks like Tomcat, Spring, Kafka. The whole supported framework list is here. It can accept data from other compliant sources like Zipkin-formatted traces reported from Zipkin, Jaeger, or OpenCensus clients. SkyWalking is logically split into four parts: Probes, Platform Backend, Storage and UI:\nThere are two kinds of probes:\nLanguage agents or SDKs following SkyWalking across-thread propagation formats and trace formats, run in the user’s application process. The Istio mixer adaptor, which collects telemetry from the Service Mesh. The platform backend provides gRPC and RESTful HTTP endpoints for all SkyWalking-supported trace and metric telemetry data. For example, you can stream these metrics into an analysis system.\nStorage supports multiple implementations such as ElasticSearch, H2 (alpha), MySQL, and Apache ShardingSphere for MySQL Cluster. TiDB will be supported in next release.\nSkyWalking’s built-in UI with a GraphQL endpoint for data allows intuitive, customizable integration.\nSome examples of SkyWalking’s UI:\nObserve a Spring app using the SkyWalking JVM-agent Observe on Istio without any agent, no matter what langugage the service is written in See fine-grained metrics like request/Call per Minute, P99/95/90/75/50 latency, avg response time, heatmap Service dependencies and metrics Service Focused At Tetrate, we are focused on discovery, reliability, and security of your running services. This is why we are embracing Skywalking, which makes service performance observable.\nBehind this admittedly cool UI, the aggregation logic is very easy to understand, making it easy to customize SkyWalking in its Observability Analysis Language (OAL) script.\nWe’ll post more about OAL for developers looking to customize SkyWalking, and you can read the official OAL introduction document.\nScripts are based on three core concepts:\nService represents a group of workloads that provide the same behaviours for incoming requests. You can define the service name whether you are using instrument agents or SDKs. Otherwise, SkyWalking uses the name you defined in the underlying platform, such as Istio.\nService Instance Each workload in the Service group is called an instance. Like Pods in Kubernetes, it doesn\u0026rsquo;t need to be a single OS process. If you are using an instrument agent, an instance does map to one OS process.\nEndpoint is a path in a certain service that handles incoming requests, such as HTTP paths or a gRPC service + method. Mesh telemetry and trace data are formatted as source objects (aka scope). These are the input for the aggregation, with the script describing how to aggregate, including input, conditions, and the resulting metric name.\nCore Features The other core features in SkyWalking v6 are:\nService, service instance, endpoint metrics analysis. Consistent visualization in Service Mesh and no mesh. Topology discovery, Service dependency analysis. Distributed tracing. Slow services and endpoints detected. Alarms. Of course, SkyWalking has some more upgrades from v5, such as:\nElasticSearch 6 as storage is supported. H2 storage implementor is back. Kubernetes cluster management is provided. You don’t need Zookeeper to keep the backend running in cluster mode. Totally new alarm core. Easier configuration. More cloud native style. MySQL will be supported in the next release. Please: Test and Provide Feedback! We would love everyone to try to test our new version. You can find everything you need in our Apache repository,read the document for further details. You can contact the project team through the following channels:\nSubmit an issue on GitHub repository Mailing list: dev@skywalking.apache.org . Send to dev-subscribe@kywalking.apache.org to subscribe the mail list. Gitter Project twitter Oh, and one last thing! If you like our project, don\u0026rsquo;t forget to give us a star on GitHub.\n","excerpt":"\u003cp\u003eOriginal link, \u003ca href=\"https://www.tetrate.io/blog/apache-skywalking-v6/\"\u003eTetrate.io blog\u003c/a\u003e\u003c/p\u003e\n\u003ch1 id=\"context\"\u003eContext\u003c/h1\u003e\n\u003cp\u003eThe integration of SkyWalking and Istio Service Mesh yields …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2018-12-12-skywalking-service-mesh-ready/","title":"SkyWalking v6 is Service Mesh ready"},{"body":"Based on his contributions to the project, he has been accepted as SkyWalking committer. Welcome aboard.\n","excerpt":"\u003cp\u003eBased on his contributions to the project, he has been accepted as SkyWalking committer. Welcome …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/welcome-jian-tan-as-a-new-committer/","title":"Welcome Jian Tan as a new committer"},{"body":"APM consistently compatible in language agent(Java, .Net, NodeJS), 3rd party format(Zipkin) and service mesh telemetry(Istio). Go to downloads page to find release tars.\n","excerpt":"\u003cp\u003eAPM consistently compatible in language agent(Java, .Net, NodeJS), 3rd party format(Zipkin) and …\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-6-0-0-alpha/","title":"Release Apache SkyWalking 6.0.0-alpha"},{"body":"A stable version of 5.x release. Go to downloads page to find release tars.\n","excerpt":"\u003cp\u003eA stable version of 5.x release. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-5-0-0-ga/","title":"Release Apache SkyWalking 5.0.0-GA"},{"body":"5.0.0-RC2 release. Go to downloads page to find release tars.\n","excerpt":"\u003cp\u003e5.0.0-RC2 release. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-5-0-0-rc2/","title":"Release Apache SkyWalking 5.0.0-RC2"},{"body":"5.0.0-beta2 release. Go to downloads page to find release tars.\n","excerpt":"\u003cp\u003e5.0.0-beta2 release. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-5-0-0-beta2/","title":"Release Apache SkyWalking 5.0.0-beta2"},{"body":"Translated by Sheng Wu.\nIn many big systems, distributed and especially microservice architectures become more and more popular. With the increase of modules and services, one incoming request could cross dozens of service. How to pinpoint the issues of the online system, and the bottleneck of the whole distributed system? This became a very important problem, which must be resolved.\nTo resolve the problems in distributed system, Google published the paper “Dapper, a Large-Scale Distributed Systems Tracing Infrastructure”, which mentioned the designs and ideas of building a distributed system. Many projects are inspired by it, created in the last 10 years. At 2015, Apache SkyWalking was created by Wu Sheng as a simple distributed system at first and open source. Through almost 3 years developments, at 2018, according to its 5.0.0-alpha/beta releases, it had already became a cool open source APM system for cloud native, container based system.\nAt the early of this year, I was trying to build the Butterfly open source APM in .NET Core, and that is when I met the Apache SkyWalking team and its creator. I decided to join them, and cooperate with them, to provide .NET Core agent native compatible with SkyWalking. At April, I released the first version .NET core agent 0.1.0. After several weeks interation, we released 0.2.0, for increasing the stability and adding HttpClient, Database driver supports.\nBefore we used .NET Core agent, we need to deploy SkyWalking collector, UI and ElasticSearch 5.x. You can download the release versions at here: http://skywalking.apache.org/downloads/ and follow the docs (Deploy-backend-in-standalone-mode, Deploy-backend-in-cluster-mode) to setup the backend.\nAt here, I are giving a quick start to represent, how to monitor a demo distributed .NET Core applications. I can say, that is easy.\ngit clone https://github.com/OpenSkywalking/skywalking-netcore.git\ncd skywalking-netcore\ndotnet restore\ndotnet run -p sample/SkyWalking.Sample.Backend dotnet run -p sample/SkyWalking.Sample.Frontend\nNow you can open http://localhost:5001/api/values to access the demo application. Then you can open SkyWalking WebUI http://localhost:8080\nOverview of the whole distributed system Topology of distributed system Application view Trace query Span’s tags, logs and related traces GitHub Website: http://skywalking.apache.org/ SkyWalking Github Repo: https://github.com/apache/incubator-skywalking SkyWalking-NetCore Github Repo: https://github.com/OpenSkywalking/skywalking-netcore ","excerpt":"\u003cp\u003eTranslated by Sheng Wu.\u003c/p\u003e\n\u003cp\u003eIn many big systems, distributed and especially microservice architectures …\u003c/p\u003e","ref":"https://skywalking.apache.org/blog/2018-05-24-skywalking-net/","title":"Apache SkyWalking provides open source APM and distributed tracing in .NET Core field"},{"body":"在大型网站系统设计中，随着分布式架构，特别是微服务架构的流行，我们将系统解耦成更小的单元，通过不断的添加新的、小的模块或者重用已经有的模块来构建复杂的系统。随着模块的不断增多，一次请求可能会涉及到十几个甚至几十个服务的协同处理，那么如何准确快速的定位到线上故障和性能瓶颈，便成为我们不得不面对的棘手问题。\n为解决分布式架构中复杂的服务定位和性能问题，Google 在论文《Dapper, a Large-Scale Distributed Systems Tracing Infrastructure》中提出了分布式跟踪系统的设计和构建思路。在这样的背景下，Apache SkyWalking 创建于 2015 年，参考 Dapper 论文实现分布式追踪功能，并逐渐进化为一个完整功能的 Application Performance Management 系统，用于追踪、监控和诊断大型分布式系统，尤其是容器和云原生下的微服务系统。\n今年初我在尝试使用.NET Core 构建分布式追踪系统 Butterfly 时接触到 SkyWalking 团队，开始和 SkyWalking 团队合作探索 SkyWalking 对.NET Core 的支持，并于 4 月发布 SkyWalking .NET Core 探针的 第一个版本，同时我也有幸加入 SkyWalking 团队共同进行 SkyWalking 在多语言生态的推动。在.NET Core 探针 v0.1 版本发布之后，得到了一些同学的尝鲜使用，也得到诸多改进的建议。经过几周的迭代，SkyWalking .NET Core 探针于今天发布 v0.2 release，在 v0.1 的基础上增加了\b稳定性和 HttpClient 及数据库驱动的追踪支持。\n在使用 SkyWalking 对.NET Core 应用追踪之前，我们需要先部署 SkyWalking Collector 收集分析 Trace 和 Elasticsearch 作为 Trace 数据存储。SkyWalking 支持 5.x 的 ES，所以我们需要下载安装对应版本的 ES，并配置 ES 的 cluster.name 为 CollectorDBCluster。然后部署 SkyWalking 5.0 beta 或更高版本 (下载地址:http://skywalking.apache.org/downloads/)。更详细的 Collector 部署文档，请参考 Deploy-backend-in-standalone-mode 和 Deploy-backend-in-cluster-mode。\n最后我们使用示例项目来演示在.NET Core 应用中使用 SkyWalking 进行追踪和监控，克隆 SkyWalking-NetCore 项目到本地：\ngit clone https://github.com/OpenSkywalking/skywalking-netcore.git 进入 skywalking-netcore 目录：\ncd skywalking-netcore 还原 nuget package：\ndotnet restore 启动示例项目：\ndotnet run -p sample/SkyWalking.Sample.Backend dotnet run -p sample/SkyWalking.Sample.Frontend 访问示例应用：\n打开 SkyWalking WebUI 即可看到我们的应用监控面板 http://localhost:8080\nDashboard 视图\nTopologyMap 视图\nApplication 视图\nTrace 视图\nTraceDetails 视图\nGitHub SkyWalking Github Repo：https://github.com/apache/incubator-skywalking SkyWalking-NetCore Github Repo：https://github.com/OpenSkywalking/skywalking-netcore ","excerpt":"\u003cp\u003e在大型网站系统设计中，随着分布式架构，特别是微服务架构的流行，我们将系统解耦成更小的单元，通过不断的添加新的、小的模块或者重用已经有的模块来构建复杂的系统。随着模块的不断增多，一次请求可能会涉及到十几 …\u003c/p\u003e","ref":"https://skywalking.apache.org/zh/2018-05-24-skywalking-net/","title":"Apache SkyWalking 为.NET Core带来开箱即用的分布式追踪和应用性能监控"},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/zh_tags/dotnetcore/","title":"DotNetCore"},{"body":"5.0.0-beta release. Go to downloads page to find release tars.\n","excerpt":"\u003cp\u003e5.0.0-beta release. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-5-0-0-beta/","title":"Release Apache SkyWalking 5.0.0-beta"},{"body":"5.0.0-alpha release. Go to downloads page to find release tars.\n","excerpt":"\u003cp\u003e5.0.0-alpha release. Go to \u003ca href=\"/downloads\"\u003edownloads\u003c/a\u003e page to find release tars.\u003c/p\u003e","ref":"https://skywalking.apache.org/events/release-apache-skywalking-apm-5-0-0-alpha/","title":"Release Apache SkyWalking APM 5.0.0-alpha"},{"body":" \u003c!doctype html\u003e BDB Native Index Spec — Overview and contracts BDB NATIVE INDEX implementation specification REVIEW BDB-NIDX-SPEC-001\nRevision 0.2 · 2026-08-25 CHAPTER 01/05 · OVERVIEW 00 Document status 01 Contract and scope 02 Data model 03 Product requirements 04 Architecture and APIs 05 Fields and identity 06 Analyzers and terms 07 Write semantics 08 ICE v3 format 09 Snapshot v3 format 10 Query execution 11 Sort and projection 12 Durability and recovery 13 Merge, delete, and GC 14 External and admin 15 Integrity and limits 16 Compatibility 17 Verification program 18 Implementation tickets 19 Operations 20 Review decisions 21 Acceptance criteria 22 Traceability Animated ICE walkthrough Visual research report All Normative Open only ◐ Print BDB-NIDX-SPEC-001 · review draft Native inverted index implementation spec. A complete review contract for replacing Bluge and ICE with a minimal BanyanDB-owned engine while preserving ICE v3 segments, snapshot v3 manifests, live-query behavior, crash recovery, and same-file rollback.\nRevision 0.2 Date 2026-08-25 Compatibility target ICE v3 + snapshot v3 Implementation Direct immutable builder Source baseline BanyanDB 8f1c6f5f + pinned forks Disposition Review; not approved for production REVIEW DRAFT STABLE Source-observed contracts suitable for implementation after fixture confirmation.\nFIXTURE Normative intent is clear, but exact historical bytes still require golden corpus proof.\nDECIDE Maintainer choice is required before the corresponding behavior becomes normative.\nDO NOT No mechanical source copy and no production-directory rewrite on open.\nReview completion 11 of 11 decisions dispositioned\nRead the specification by chapter. The former monolith is now five focused review surfaces. Requirement IDs and section numbers remain stable.\nCHAPTER 01 Overview and contracts Scope, product roles, the native data model, and the owned architecture boundary. CHAPTER 02 Write path and disk grammar Fields, analyzers, batch transformation, ICE v3 sections, and snapshot v3 publication. CHAPTER 03 Query and lifecycle Filter execution, projection, ordering, durability, merge, garbage collection, and external receive. CHAPTER 04 Safety and verification Bounds, malformed-input behavior, compatibility matrices, fixtures, fuzzing, and crash evidence. CHAPTER 05 Delivery and review Five bounded cutovers, operational signals, accepted decisions, release gates, and source traceability. 01 Stable contract Defines what the replacement is—and is not.\nContract and scope The replacement is a BanyanDB filter, projection, and sort engine—not a general-purpose relevance search product.\nRequirement language Marker Review meaning MUST Required for conformance and release acceptance. SHOULD Expected unless a reviewed exception records the reason, impact, and replacement control. MAY Optional behavior that must not weaken a MUST or the compatibility boundary. fixture gate The stated rule guides the prototype, but production enablement waits for independent byte evidence. decision gate The corresponding Section 20 disposition must be approved and folded into the next revision. Objective SCOPE-001 MUST Implement the index behavior required by Series, Stream, Property, index-mode Measure, migration, repair, dump, backup, and verification.\nSCOPE-002 MUST Read existing supported BanyanDB index directories and, during the rollback window, emit ICE v3 segments and snapshot v3 manifests readable by the pinned legacy implementation.\nSCOPE-003 MUST Own query semantics, analysis, term encoding, snapshots, durability, merge, external receive, administration, and lifecycle in BanyanDB packages.\nSCOPE-004 SHOULD Retain only low-level codecs whose serialized bytes are already part of the compatibility boundary: Roaring, Vellum, and verified compression implementations.\nNon-goals Excluded capability Reason Compatibility handling BM25 scoring and relevance ranking No production caller consumes scores. Reader tolerates legacy detail; native query results are unscored. Phrase/fuzzy/geo queries No required caller identified. Locations may be preserved during compatible merge without exposing APIs. Highlights, facets, aggregations, explanations Outside BanyanDB index contracts. No native implementation. General plugin or remote-directory API Would recreate Bluge abstraction surface. Private segment and directory interfaces only. New disk version during rollout Would break same-file downgrade. Consider only after legacy fallback retirement. Legacy-to-native implementation comparison The native index intentionally implements the BanyanDB contract, not the entire Bluge/ICE feature surface. This table distinguishes required compatibility work from capabilities that must remain absent. “Not implemented” is a deliberate boundary, not deferred work.\nnative BanyanDB-owned behavior compatibility only retained format or codec boundary not implemented intentionally absent from the native index deferred prohibited during the rollback window Legacy capability or format area Native index behavior State Compatibility consequence Document identity, batch mutation, replacement, and deletion masks Implement with BanyanDB-owned writer, snapshot, and identity semantics. native Series, Stream, Property, and index-mode Measure retain their required write contracts. Exact, range, prefix, wildcard, MATCH, existence, and boolean filters Implement the caller-required filter algebra without scoring. native Pure-negative and field-universe behavior follows the accepted DEC-003 corrections. Stored projection, field dictionaries, doc-value sort, and search-after Implement projection and stable explicit ordering required by online and offline consumers. native Sort is explicit; relevance ordering is never implied. Snapshots, publication, recovery, merge, expiry, repair, migration, and administration Implement the BanyanDB-required lifecycle with private owned APIs. native Includes external segment reception and read-only administrative access. ICE v3 segments and snapshot v3 manifests Read and write the required byte layout during the rollback window. native Existing supported directories and native-produced files remain cross-readable. Vellum dictionaries, Roaring postings, and proven compression codecs Retain only these low-level codecs at the serialized-byte boundary. compatibility only Native code owns index behavior; codec versions remain pinned by fixture evidence. Legacy frequency, norm, and location detail Read or preserve it only where fixture-proven compatibility and compatible merge require it. compatibility only The native query API does not expose scoring or positional-search behavior. Segment-footer and snapshot CRC32 fields Preserve each four-byte field in its historical position, but do not calculate, validate, or use its value. processing not implemented CRC32 is a layout-compatibility field only; structural, bounds, and codec validation remain required. BM25 scoring, boosts, and relevance ranking Return unscored results only. not implemented No production caller may depend on scores or relevance ordering. Phrase and other positional-search APIs Do not expose a phrase-query or public term-vector surface. not implemented Location detail may be preserved only when needed for compatible merge. Fuzzy and edit-distance queries Do not build fuzzy expansion or distance-scoring machinery. not implemented Callers retain exact, prefix, wildcard, and MATCH behavior only. Geospatial indexing and geo queries Do not implement geo shapes, spatial terms, or geo-distance search. not implemented No production BanyanDB index contract requires this surface. Highlights, facets, aggregations, and explanations Expose none of these search-product result features. not implemented Aggregation remains outside the native inverted-index contract. General plugin and remote-directory abstractions Use private BanyanDB segment and directory interfaces. not implemented The native implementation must not recreate the Bluge abstraction surface. New disk version during the rollback window Do not introduce one. deferred Same-file downgrade remains possible until legacy fallback retirement. ! Provenance rule: existing Bluge/ICE source may establish behavior, but production code must be implemented from this specification, independent fixtures, and black-box tests—not by mechanical translation.\n02 Stable model Defines the immutable objects and three access directions.\nData model and terminology A logical field may be materialized more than once because filtering, sorting, and returning values travel in opposite directions.\nVIEW A · INVERTED term → local documents Vellum FST dictionaries lead to Roaring postings and optional frequency/location detail. Used for exact, range, prefix, wildcard, and boolean filtering.\nVIEW B · DOC VALUES local document → terms Per-field compressed chunks return analyzed, comparable term bytes. Used for explicit sort, reconstruction, and merge.\nVIEW C · STORED local document → source bytes Compressed document records preserve original stored values. Used for projection, repair, migration, and administration.\nINPUT\nlatency = 120 INDEXED Numeric terms point to a posting containing local document 0.\nSORTABLE Document 0 points to the shift-zero comparable numeric term.\nSTORED Document 0 points to the original value bytes used for return.\n≠ Not a duplicate doc value: stored chunks contain original source values; doc-value chunks contain analyzed, escaped term bytes. When one logical field is both stored and sortable, ICE deliberately materializes both representations because they serve different access directions.\nNormative terms Term Definition Document identity The unique _id term used to find the current live version of a document. Local document number A dense zero-based number meaningful only within one immutable segment. Global hit number A snapshot-relative number obtained by adding a segment base offset to a local document number. Term An analyzer/codec-produced byte string stored as a field dictionary key. Posting A term's local document set and optional frequency, norm, and location detail. Stored field Original field bytes explicitly retained for document materialization. Doc value Analyzed term bytes indexed by document number for sort, merge, or reconstruction; not the original source value. Segment An immutable ICE v3 byte sequence containing documents and indexes but no logical deletion state. Snapshot An immutable generation selecting segments and per-segment deletion bitmaps. Live document A segment document not present in its snapshot deletion bitmap. 03 Stable contract Maps every feature to a BanyanDB consumer.\nProduct requirements Every production feature must map to one of these consumers or to disk compatibility.\nConsumer Identity/write semantics Required reads Lifecycle Series index One live serialized entity identity; field-set-aware ensure/upsert. Exact/prefix/wildcard identity, filters, projection, timestamp/version, dictionary scan, explicit sort. Per-storage-segment persistence and migration. Stream element index Numeric document ID plus mandatory series identity. Exact/range/MATCH/existence, document and timestamp postings, explicit sort. Batch write, rebuild, external receive. Property Replacement document with stored source, delete time, hash, and identity. Boolean/range filters, stable sorted repair scan, stored fields. Durable callbacks, merge-time expiry, repair, backup. Index-mode Measure Series entity plus stored tags, timestamp, and version. Time/range filter, full projection, sort/doc values. Migration reconstruction and verification. Admin/offline No mutation for read-only open. Count, latest generation, walk, verify, search-after, stored/doc-value visitation. Snapshot, dump, rebuild, repair, migration. PROD-001 MUST The native backend must satisfy the existing pkg/index online contracts and replace every direct production Bluge bypass with an owned admin interface.\nPROD-002 MUST External segment reception must remain available for Stream, Measure, and Trace migration workflows.\nPROD-003 MUST Property expiration must be evaluated during merge using a private deletion set and must not mutate a published reader snapshot.\n04 Recommended Direct immutable builder with isolated codecs.\nArchitecture and APIs BanyanDB concepts form the public boundary. No Bluge, ICE, Vellum, Roaring, or compression types escape the internal codec layer.\npkg/index stable product contracts → Query IR + codecs owned semantics → Native store write and execute → Segment boundary immutable refs Snapshot manager live masks + MVCC → ICE v3 codec bounded read/write → Directory publish + recover → Admin/external verify + receive Package ownership pkg/index/queryir backend-neutral query tree pkg/index/analyzer keyword, simple, standard, URL tokenization pkg/index/termcodec byte, numeric, and timestamp term encodings pkg/index/native/store admission, identity evolution, public adapters pkg/index/native/snapshot immutable generations and deletion masks pkg/index/native/execute postings algebra, ranges, wildcard, projection, sort pkg/index/native/segment private immutable segment interfaces pkg/index/native/icev3 bounded ICE segment codec pkg/index/native/persist ordered durable publication and callbacks pkg/index/native/directory naming, locking, fsync, recovery, and GC pkg/index/native/external raw segment validation and introduction pkg/index/native/admin count, inspect, verify, walk, rebuild, backup Required private boundaries type Segment interface { NumDocs() uint64 Fields() FieldIterator Dictionary(fieldID uint16) Dictionary Postings(fieldID uint16, term []byte, except LiveMask) Postings VisitStored(docNum uint64, visit StoredVisitor) error VisitDocValues(docNum uint64, fieldIDs []uint16, visit TermVisitor) error TimeBounds() (min, max int64) } type Snapshot interface { Acquire() SnapshotRef Segments() []SegmentRef Deletions(segmentID uint64) LiveMask Generation() uint64 } ARCH-001 MUST Use the direct immutable segment builder as the target architecture. A permanent mutable document arena is not part of the design.\nARCH-002 MUST Serialize mutations per index directory while permitting concurrent ref-counted immutable readers.\nARCH-003 MUST Keep third-party serialized codec types behind native/icev3; other packages operate on owned interfaces and byte slices.\nNext chapter →Write path and disk grammar No section in this chapter matches the search/filter. ","excerpt":"\u003c!--\n  Licensed to Apache Software Foundation (ASF) under one or more contributor\n  license …","ref":"https://skywalking.apache.org/docs/skywalking-banyandb/next/design/archive/0.12.0/native-inverted-index/","title":""},{"body":"","excerpt":"","ref":"https://skywalking.apache.org/index.json","title":""},{"body":" ActiveMQ The ACTIVEMQ layer monitors Apache ActiveMQ message brokers. SkyWalking collects ActiveMQ metrics over OpenTelemetry and rolls them up into cluster-, broker-, and destination-scope metrics, so operators can watch queue depth, throughput, connection counts, and broker JVM health alongside the rest of their estate. See the upstream ActiveMQ monitoring setup for how to wire the telemetry pipeline.\nIn Horizon\u0026rsquo;s sidebar this layer is named ActiveMQ. Its services are listed as ActiveMQ clusters, its instances as Brokers, and its endpoints as Destinations (the queues and topics a broker serves). The ACTIVEMQ layer enables the Service, Instance, and Endpoint sub-tabs; it has no topology, traces, or logs view.\nThis page is the operator reference for the bundled ACTIVEMQ dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled ACTIVEMQ template; if an operator has published a customized ACTIVEMQ template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every ActiveMQ cluster with four sortable columns, sorted by Enqueue/s by default:\nEnqueue/s — messages enqueued per second across the cluster (meter_activemq_cluster_enqueue_rate).\nDequeue/s — messages dequeued per second across the cluster (meter_activemq_cluster_dequeue_rate).\nSystem Load — the cluster\u0026rsquo;s average system load (meter_activemq_cluster_system_load_average/10000).\nThreads — the cluster\u0026rsquo;s average thread count (meter_activemq_cluster_thread_count).\nService dashboard The primary drill-down for one selected ActiveMQ cluster, mixing throughput, message timing, and broker JVM health.\nSystem Load Average — the cluster\u0026rsquo;s system load over time (meter_activemq_cluster_system_load_average/10000).\nThread Count — live JVM threads across the cluster (meter_activemq_cluster_thread_count).\nHeap Used (MB) — JVM heap memory in use, in MB (meter_activemq_cluster_heap_memory_usage_used/1024/1024).\nHeap Max (MB) — the configured maximum heap across the cluster, summed and shown as the latest value in MB (latest(aggregate_labels(meter_activemq_cluster_heap_memory_usage_max,sum))/1024/1024).\nEnqueue / Dequeue / Dispatch /s — the three core message rates on one chart: messages enqueued, dequeued, and dispatched per second (meter_activemq_cluster_enqueue_rate, meter_activemq_cluster_dequeue_rate, meter_activemq_cluster_dispatch_rate).\nExpired /s — messages that expired before delivery, per second (meter_activemq_cluster_expired_rate).\nEnqueue Time — average and maximum time a message spends being enqueued, in seconds (meter_activemq_cluster_average_enqueue_time/1000, meter_activemq_cluster_max_enqueue_time/1000).\nGC Counts (G1+Parallel) — old- and young-generation garbage-collection counts, each combining the G1 and Parallel collectors so the chart reads correctly regardless of which collector the broker JVM uses (view_as_seq(meter_activemq_cluster_gc_g1_old_collection_count, meter_activemq_cluster_gc_parallel_old_collection_count), and the matching young-collection counters).\nGC Time (ms) — old- and young-generation GC time in ms, again combining the G1 and Parallel collectors (view_as_seq(meter_activemq_cluster_gc_g1_old_collection_time, meter_activemq_cluster_gc_parallel_old_collection_time), and the matching young-collection timers).\nInstance dashboard For one selected broker. A row of single-value cards summarizes the broker\u0026rsquo;s current state, followed by trend charts.\nSummary cards\nConnections — current TCP/JMS connections to this broker (latest(meter_activemq_broker_current_connections)).\nProducer Count — active producer sessions on this broker, summed across destinations (latest(aggregate_labels(meter_activemq_broker_current_producer_count,sum))).\nConsumer Count — active consumer sessions on this broker, summed across destinations (latest(aggregate_labels(meter_activemq_broker_current_consumer_count,sum))).\nUptime — broker uptime in hours since its last restart (latest(meter_activemq_broker_uptime)/1000/60/60).\nTrends\nConnections (trend) — the broker\u0026rsquo;s connection count over time (meter_activemq_broker_current_connections).\nEnqueue / Dequeue Count — per-minute enqueue and dequeue totals summed across the broker\u0026rsquo;s destinations (aggregate_labels(meter_activemq_broker_enqueue_count,sum), aggregate_labels(meter_activemq_broker_dequeue_count,sum)).\nProducer / Consumer Increase — new producer and consumer sessions opened per minute (aggregate_labels(meter_activemq_broker_producer_count,sum), aggregate_labels(meter_activemq_broker_consumer_count,sum)).\nMemory Usage — aggregate memory usage across destinations, in MB (aggregate_labels(meter_activemq_broker_memory_usage,sum)/1024/1024).\nMemory Limit — the configured memory ceiling across destinations, in GB (aggregate_labels(meter_activemq_broker_memory_limit,sum)/1024/1024/1024).\nAvg Message Size — average message size across destinations, in bytes (aggregate_labels(meter_activemq_broker_average_message_size,avg)).\nEndpoint dashboard For one selected destination (a queue or topic).\nProducer Count — producers attached to this destination (meter_activemq_destination_producer_count).\nConsumer Count — consumers attached to this destination (meter_activemq_destination_consumer_count).\nQueue Size — messages currently held in the destination (meter_activemq_destination_queue_size).\nMemory Usage (MB) — memory the destination is consuming, in MB (meter_activemq_destination_memory_usage/1024/1024).\nMessage Counts — the destination\u0026rsquo;s message lifecycle on one chart: enqueued, dequeued, dispatched, expired, and in-flight counts (meter_activemq_destination_enqueue_count, meter_activemq_destination_dequeue_count, meter_activemq_destination_dispatch_count, meter_activemq_destination_expired_count, meter_activemq_destination_inflight_count).\nEnqueue Time (s) — average and maximum enqueue time for the destination, in seconds (meter_activemq_destination_average_enqueue_time/1000, meter_activemq_destination_max_enqueue_time/1000).\nRequirements The ACTIVEMQ dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the ActiveMQ meter families, produced from broker telemetry collected over OpenTelemetry:\nCluster metrics — the meter_activemq_cluster_* family (enqueue / dequeue / dispatch / expired rates, enqueue time, system load, thread count, heap usage, and the G1 / Parallel GC counters) for the service list and the cluster dashboard.\nBroker metrics — the meter_activemq_broker_* family (current connections, producer / consumer counts, uptime, enqueue / dequeue counts, memory usage and limit, average message size) for the broker dashboard.\nDestination metrics — the meter_activemq_destination_* family (producer / consumer count, queue size, memory usage, the enqueue / dequeue / dispatch / expired / in-flight counts, and enqueue time) for the destination dashboard.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a broker- or destination-scope metric is empty until that level of data is reported. See the upstream ActiveMQ monitoring setup for the collection pipeline that produces these families.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/activemq/","title":"\u003c!--"},{"body":" Airflow The AIRFLOW layer monitors Apache Airflow workflow schedulers. OAP collects Airflow\u0026rsquo;s OpenTelemetry metrics and presents each monitored Airflow deployment as a cluster, so this layer is where you watch scheduler health, DAG parsing, executor and pool capacity, and triggerer activity.\nIn Horizon\u0026rsquo;s sidebar this layer is named Airflow, grouped under Workflow Scheduler. Its services are listed as Airflow clusters and the components that report into each cluster (the scheduler and triggerer processes) are listed as Components. The AIRFLOW layer enables only the Service and Instance scopes — there is no endpoint scope, no topology, and no traces or logs tab, because Airflow\u0026rsquo;s telemetry is scheduler-level meter data rather than request traffic.\nThis page is the operator reference for the bundled AIRFLOW dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled AIRFLOW template; if an operator has published a customized AIRFLOW template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every Airflow cluster with four sortable columns, sorted by DAG Bag Size by default:\nDAG Bag Size — number of DAGs found in the last scheduler scan (meter_airflow_dagbag_size).\nDAG Total Parse Time — seconds spent scanning and importing the queued DAG files (meter_airflow_dag_total_parse_time).\nExecutor Open Slots — free executor slots available to run tasks (meter_airflow_executor_open_slots).\nScheduled Slots — pool slots scheduled but not yet running, summed across pools (aggregate_labels(meter_airflow_pool_scheduled_slots, sum)).\nService dashboard The primary drill-down for one selected Airflow cluster. It opens with four single-value cards reporting the current scheduler and executor state, then a set of time-series charts tracking executor capacity and DAG-processing health.\nCards (current value)\nTasks Executable — tasks ready for execution across the cluster (latest(meter_airflow_scheduler_tasks_executable)).\nRunning Tasks — tasks currently running on the executor (latest(meter_airflow_executor_running_tasks)).\nScheduled Slots — pool slots scheduled but not yet running, aggregated across pools (latest(aggregate_labels(meter_airflow_pool_scheduled_slots, sum))).\nQueued Tasks — tasks waiting on the executor (latest(meter_airflow_executor_queued_tasks)).\nCharts (over time)\nExecutor Open Slots — free executor slots over the window (meter_airflow_executor_open_slots).\nDAG File Queue Size — DAG files pending a scan (meter_airflow_dag_file_queue_size).\nDAG Import Errors — DAG files that failed to parse (meter_airflow_dag_import_errors).\nDAG Bag Size — DAGs found in the last scheduler scan (meter_airflow_dagbag_size).\nDAG Total Parse Time — seconds to scan and import the queued DAG files (meter_airflow_dag_total_parse_time).\nDAG File Refresh Errors — DAG file load failures per minute (meter_airflow_dag_file_refresh_error).\nAsset Updates — asset update events per minute (meter_airflow_asset_updates).\nInstance dashboard For one selected Component of the cluster. Airflow reports different meters from different processes — the scheduler emits pool, executor, and asset metrics, while the triggerer emits trigger metrics — so each widget appears only when that component actually reports the metric behind it. A scheduler component therefore shows the pool / executor / heartbeat widgets, a triggerer component shows the triggerer widgets, and neither is shown empty for a component that does not emit it.\nScheduler component\nPool Open / Deferred / Running Slots — pool capacity on the scheduler, plotted as three series: open, deferred, and running slots (meter_airflow_instance_pool_open_slots, meter_airflow_instance_pool_deferred_slots, meter_airflow_instance_pool_running_slots).\nRunning Tasks / Scheduled Slots — executor running-task count against pool slots waiting to run (meter_airflow_instance_executor_running_tasks, meter_airflow_instance_pool_scheduled_slots).\nScheduler Heartbeat — scheduler heartbeats per minute (meter_airflow_instance_scheduler_heartbeat).\nExecutor Open / Queued Slots — executor capacity and queue depth on the scheduler (meter_airflow_instance_executor_open_slots, meter_airflow_instance_executor_queued_tasks).\nAsset Updates — asset update events on the scheduler, per minute (meter_airflow_instance_asset_updates).\nAsset Triggered DagRuns — DagRuns triggered by asset events on the scheduler, per minute (meter_airflow_instance_asset_triggered_dagruns).\nTriggerer component\nTriggerer Heartbeat — triggerer process heartbeats per minute (meter_airflow_instance_triggerer_heartbeat).\nTriggers Running / Capacity Left — live deferrable-trigger load on the triggerer: triggers running against capacity left (meter_airflow_instance_triggers_running, meter_airflow_instance_triggerer_capacity_left).\nTriggers Blocked / Failed / Succeeded — deferred-trigger outcomes on the triggerer host, per minute (meter_airflow_instance_triggers_blocked_main_thread, meter_airflow_instance_triggers_failed, meter_airflow_instance_triggers_succeeded).\nRequirements The AIRFLOW dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nCluster (service-scope) Airflow meters — the meter_airflow_* family (DAG bag size and parse time, DAG-processing queue and import / refresh errors, executor open / queued slots and running / executable tasks, pool scheduled slots, asset updates), aggregated per Airflow cluster.\nComponent (instance-scope) Airflow meters — the meter_airflow_instance_* family (per-component pool, executor, scheduler-heartbeat, asset, and triggerer metrics), reported by each scheduler or triggerer process.\nThese meters are produced by OAP from Airflow\u0026rsquo;s OpenTelemetry metric export — see Airflow monitoring for how to wire Airflow up to OAP. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so an instance-scope component metric is empty until that component reports it. When a component does not emit a family — a triggerer that reports no scheduler pool metrics, or a scheduler that reports no triggerer metrics — those widgets are hidden rather than shown empty.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/airflow/","title":"\u003c!--"},{"body":" Alipay Mini Program The ALIPAY_MINI_PROGRAM layer holds front-end real-user monitoring data reported from Alipay (支付宝) mini-programs. The mini-program monitoring agent feeds OAP launch, render, request, and error metrics from inside the Alipay container, and those land here.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Mobile. Its services are listed as Mini-programs, instances as Versions (each version of a published mini-program), and endpoints as Pages. The ALIPAY_MINI_PROGRAM layer enables the Service, Instance (Version), and Endpoint (Page) dashboards along with the Traces and Logs sub-tabs. It does not ship a topology / service-map view — mini-program RUM data has no call graph to draw.\nThis page is the operator reference for the bundled ALIPAY_MINI_PROGRAM dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled ALIPAY_MINI_PROGRAM template; if an operator has published a customized template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a mini-program, the layer landing page lists every Mini-program with four sortable columns, sorted by traffic (Request RPM) by default:\nRequest RPM — requests per minute (meter_alipay_mp_request_cpm).\nLaunch — average app-launch duration in ms (meter_alipay_mp_app_launch_duration).\nFirst Render — average first-render duration in ms (meter_alipay_mp_first_render_duration).\nErrors — error count over the window (meter_alipay_mp_error_count).\nService dashboard The primary drill-down for one selected mini-program.\nApp Launch Duration — the approximate cold-launch duration measured from the Alipay container, in ms (meter_alipay_mp_app_launch_duration).\nFirst Render Duration — time to first render, in ms (meter_alipay_mp_first_render_duration).\nError Count — number of reported front-end errors (meter_alipay_mp_error_count).\nRequest Load — requests per minute for the mini-program (meter_alipay_mp_request_cpm).\nRequest Duration Percentile — p50 / p75 / p90 / p95 / p99 of request duration, the tail of the request-time distribution, in ms (meter_alipay_mp_request_duration_percentile).\nInstance dashboard For one selected Version of the mini-program.\nLaunch Duration — app-launch duration for this version, in ms (meter_alipay_mp_instance_app_launch_duration).\nFirst Render Duration — first-render duration for this version, in ms (meter_alipay_mp_instance_first_render_duration).\nRequest Load — requests per minute for this version (meter_alipay_mp_instance_request_cpm).\nRequest Duration Percentile — p50 / p75 / p90 / p95 / p99 of request duration for this version, in ms (meter_alipay_mp_instance_request_duration_percentile).\nEndpoint dashboard For one selected Page.\nLaunch Duration on Page — app-launch duration attributed to this page, in ms (meter_alipay_mp_endpoint_app_launch_duration).\nFirst Render Duration — first-render duration for this page, in ms (meter_alipay_mp_endpoint_first_render_duration).\nRequest Load — requests per minute for this page (meter_alipay_mp_endpoint_request_cpm).\nRequest Duration Percentile — p50 / p75 / p90 / p95 / p99 of request duration for this page, in ms (meter_alipay_mp_endpoint_request_duration_percentile).\nRequirements The ALIPAY_MINI_PROGRAM dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the Alipay mini-program meter families, reported by the Alipay mini-program monitoring agent:\nMini-program (service) metrics — the meter_alipay_mp_* family at service scope: request load (meter_alipay_mp_request_cpm), launch and first-render duration (meter_alipay_mp_app_launch_duration, meter_alipay_mp_first_render_duration), error count (meter_alipay_mp_error_count), and the request-duration percentiles (meter_alipay_mp_request_duration_percentile).\nVersion (instance) metrics — the meter_alipay_mp_instance_* family for the per-version widgets (launch, first render, request load, request percentile).\nPage (endpoint) metrics — the meter_alipay_mp_endpoint_* family for the per-page widgets (launch, first render, request load, request percentile).\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a version- or page-scope metric is empty until that level of data is reported. For how to enable the upstream collector, see the Alipay Mini-Program monitoring setup docs.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/alipay_mini_program/","title":"\u003c!--"},{"body":" APISIX The APISIX layer monitors Apache APISIX API gateways. APISIX exposes its built-in Prometheus metrics, OpenTelemetry collects and forwards them to OAP, and OAP aggregates them into the meter_apisix_* families this dashboard renders. The layer key is APISIX, and in Horizon\u0026rsquo;s sidebar it is grouped under Gateways.\nIn the sidebar this layer\u0026rsquo;s services are listed as APISIX services, its instances as Nodes (the individual APISIX data-plane nodes), and its endpoints as Routes (the matched APISIX routes). The APISIX layer enables three scopes — Service, Instance (Node), and Endpoint (Route). It does not ship a topology, traces, or logs tab, so this dashboard is purely the metric drill-down across those three scopes.\nThis page is the operator reference for the bundled APISIX dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled APISIX template; if an operator has published a customized APISIX template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a service, the layer landing page lists every APISIX service, sorted by request rate (RPS) by default, with four columns:\nRPS — total HTTP requests per second across the service\u0026rsquo;s nodes (meter_apisix_sv_http_requests).\n200/s — 200-status responses per second (meter_apisix_sv_http_status_matched{code='200'}).\n404/s — 404-status responses per second (meter_apisix_sv_http_status_matched{code='404'}).\n503/s — 503-status responses per second (meter_apisix_sv_http_status_matched{code='503'}).\nThe three status columns give an at-a-glance health read across the fleet — a service with a climbing 503/s next to its 200/s is shedding load.\nService dashboard The primary drill-down for one selected APISIX service. All eight widgets aggregate across the service\u0026rsquo;s nodes.\nHTTP Request Trend — total requests per second for the service (meter_apisix_sv_http_requests).\nHTTP Status Trend — requests per second broken down by HTTP status code (meter_apisix_sv_http_status_matched, one line per code).\nHTTP Latency — request latency in ms, split by latency type and percentile (meter_apisix_sv_http_latency_matched).\nHTTP Bandwidth — ingress / egress bandwidth in KB, by type (meter_apisix_sv_bandwidth_matched, divided to KB).\nHTTP Connections — active connections by state — active, reading, writing, waiting (meter_apisix_sv_http_connections, one line per state).\nNon-matched Status Trend — requests per second by status code for traffic that hit no matching APISIX route (meter_apisix_sv_http_status_unmatched). Unmatched traffic is usually a misconfigured client or a probe; a rising line here is worth investigating.\nNon-matched Latency — latency in ms for the same no-matching-route traffic (meter_apisix_sv_http_latency_unmatched).\nNon-matched Bandwidth — bandwidth in KB for the same no-matching-route traffic (meter_apisix_sv_bandwidth_unmatched, divided to KB).\nInstance dashboard For one selected node. These widgets are reported per node, so they show how one APISIX data-plane process is behaving.\nHTTP Request Trend — requests per second for the node (meter_apisix_instance_http_requests).\nHTTP Status Trend — requests per second by status code for the node (meter_apisix_instance_http_status_matched).\nHTTP Latency — request latency in ms for the node (meter_apisix_instance_http_latency_matched).\nHTTP Bandwidth — bandwidth in KB for the node (meter_apisix_instance_bandwidth_matched, divided to KB).\nHTTP Connections — connections by state for the node (meter_apisix_instance_http_connections).\nShared Dict — the node\u0026rsquo;s shared-memory dictionary capacity vs. free space in MB (meter_apisix_instance_shared_dict_capacity_bytes and meter_apisix_instance_shared_dict_free_space_bytes, divided to MB, labelled capacity / free). When free space approaches zero the node can no longer cache new entries.\netcd — the node\u0026rsquo;s view of the control-plane etcd: the latest known etcd index and whether etcd is reachable (meter_apisix_instance_etcd_indexes and latest(meter_apisix_instance_etcd_reachable), labelled indexes / reachable). A node that can\u0026rsquo;t reach etcd is no longer receiving config updates.\nNon-matched Traffic — a combined view of no-matching-route activity for the node: status, latency in ms, and bandwidth in KB on one chart (meter_apisix_instance_http_status_unmatched, meter_apisix_instance_http_latency_unmatched, and meter_apisix_instance_bandwidth_unmatched divided to KB).\nEndpoint dashboard For one selected route. APISIX reports a tighter metric set at route scope — status, latency, and bandwidth.\nHTTP Status Trend — requests per second by status code for the route (meter_apisix_endpoint_http_status, one line per code).\nHTTP Latency — request latency in ms for the route, by type and percentile (meter_apisix_endpoint_http_latency).\nHTTP Bandwidth — bandwidth in KB for the route, by type (meter_apisix_endpoint_bandwidth, divided to KB).\nRequirements The APISIX dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs APISIX metrics flowing in through the OpenTelemetry receiver:\nService metrics — the meter_apisix_sv_* family (requests, status, latency, bandwidth, connections, and the unmatched-route counterparts), aggregated by OAP across a service\u0026rsquo;s nodes.\nInstance (node) metrics — the meter_apisix_instance_* family, including the node-only meter_apisix_instance_shared_dict_* and meter_apisix_instance_etcd_* health metrics.\nEndpoint (route) metrics — the meter_apisix_endpoint_* family for the per-route status, latency, and bandwidth widgets.\nThese come from APISIX\u0026rsquo;s Prometheus plugin scraped by an OpenTelemetry Collector and forwarded to OAP, which converts them into the meter_apisix_* metrics through its meter-analysis rules. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a route-scope or node-scope metric is empty until that level of data is reported. For the end-to-end collector and OAP setup, see the APISIX monitoring backend guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/apisix/","title":"\u003c!--"},{"body":" AWS DynamoDB The AWS_DYNAMODB layer monitors Amazon DynamoDB through CloudWatch metrics that OAP pulls in and aggregates. It is an agentless layer — there is no SkyWalking agent inside DynamoDB — so the dashboard is a read-only view of the throttling, error, capacity, and latency metrics CloudWatch exposes for your DynamoDB usage.\nIn Horizon\u0026rsquo;s sidebar this layer is named AWS DynamoDB. A service here represents one DynamoDB account, so services are listed as DynamoDB accounts; each account\u0026rsquo;s endpoints are its Tables. The layer enables only two scopes — the account-level Service dashboard and the per-table Endpoint dashboard. It has no instance scope, no topology / map, and no traces or logs tabs.\nThis page is the operator reference for the bundled AWS_DYNAMODB dashboard: what you see at the account level and per table, and what each widget means.\nThe widgets and metrics below are read from the bundled AWS_DYNAMODB template; if an operator has published a customized AWS_DYNAMODB template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening an account, the layer landing page lists every DynamoDB account with four sortable columns, sorted by Read Throttled by default. All four are window sums (aggregation: \u0026quot;sum\u0026quot;), so they surface the accounts taking the most throttling and system-error pressure over the selected range:\nRead Throttled — throttled read requests across the account (aws_dynamodb_read_throttled_requests).\nWrite Throttled — throttled write requests across the account (aws_dynamodb_write_throttled_requests).\nRead Sys Err — read requests that failed with a DynamoDB system error (aws_dynamodb_read_system_errors).\nWrite Sys Err — write requests that failed with a DynamoDB system error (aws_dynamodb_write_system_errors).\nService dashboard The account-level drill-down for one selected DynamoDB account. All widgets are time-series lines over the selected window.\nThrottled Requests — throttled read vs write requests for the account (aws_dynamodb_read_throttled_requests, aws_dynamodb_write_throttled_requests).\nThrottle Events — throttle events on read vs write, counted independently of throttled request volume (aws_dynamodb_read_throttle_events, aws_dynamodb_write_throttle_events).\nSystem Errors — read vs write requests that hit a DynamoDB-side system error (aws_dynamodb_read_system_errors, aws_dynamodb_write_system_errors).\nUser Errors — requests rejected for a client-side / user error such as a bad request (aws_dynamodb_user_errors).\nConditional Check Failed — write requests rejected because a conditional expression evaluated to false (aws_dynamodb_conditional_check_failed_requests).\nTransaction Conflict — transactional requests rejected due to a conflict with another in-flight transaction (aws_dynamodb_transaction_conflict).\nRead Capacity (unit/s) — provisioned read capacity vs consumed write capacity for the account (as the bundled template plots them), in capacity units per second (aws_dynamodb_provisioned_read_capacity_units, aws_dynamodb_consumed_write_capacity_units).\nWrite Capacity (unit/s) — provisioned vs consumed write capacity for the account, in capacity units per second (aws_dynamodb_provisioned_write_capacity_units, aws_dynamodb_consumed_write_capacity_units).\nSuccessful Request Latency (ms) — latency of successful operations, broken out by operation type — get / put / query / scan — in ms (aws_dynamodb_get_successful_request_latency, aws_dynamodb_put_successful_request_latency, aws_dynamodb_query_successful_request_latency, aws_dynamodb_scan_successful_request_latency).\nTTL Deleted Items — items removed by DynamoDB\u0026rsquo;s time-to-live expiry process (aws_dynamodb_time_to_live_deleted_item_count).\nScan Returned Items — items returned by Scan operations (aws_dynamodb_scan_returned_item_count).\nQuery Returned Items — items returned by Query operations (aws_dynamodb_query_returned_item_count).\nAccount Max Reads / Writes — the account-level capacity ceilings CloudWatch reports: table-level read / write maxima and account-wide read / write maxima (aws_dynamodb_account_max_table_level_reads, aws_dynamodb_account_max_table_level_writes, aws_dynamodb_account_max_reads, aws_dynamodb_account_max_writes).\nAccount Capacity Utilization — provisioned read vs write capacity utilization for the account, in percent (aws_dynamodb_account_provisioned_read_capacity_utilization, aws_dynamodb_account_provisioned_write_capacity_utilization).\nEndpoint dashboard For one selected Table under the account. These are the per-table counterparts of the account-level widgets, evaluated at endpoint scope (aws_dynamodb_endpoint_*), so you can see which individual table is responsible for the account\u0026rsquo;s throttling, errors, or capacity draw.\nThrottled Requests — throttled read vs write requests against the table (aws_dynamodb_endpoint_read_throttled_requests, aws_dynamodb_endpoint_write_throttled_requests).\nThrottle Events — read vs write throttle events on the table (aws_dynamodb_endpoint_read_throttle_events, aws_dynamodb_endpoint_write_throttle_events).\nSystem Errors — read vs write DynamoDB system errors on the table (aws_dynamodb_endpoint_read_system_errors, aws_dynamodb_endpoint_write_system_errors).\nConditional Check Failed — write requests on the table rejected by a failed conditional expression (aws_dynamodb_endpoint_conditional_check_failed_requests).\nTransaction Conflict — transactional requests on the table rejected due to a conflict (aws_dynamodb_endpoint_transaction_conflict).\nTTL Deleted Items — items removed from the table by time-to-live expiry (aws_dynamodb_endpoint_time_to_live_deleted_item_count).\nRead Capacity — provisioned vs consumed read capacity for the table (aws_dynamodb_endpoint_provisioned_read_capacity_units, aws_dynamodb_endpoint_consumed_read_capacity_units).\nWrite Capacity — provisioned vs consumed write capacity for the table (aws_dynamodb_endpoint_provisioned_write_capacity_units, aws_dynamodb_endpoint_consumed_write_capacity_units).\nSuccessful Request Latency (ms) — latency of successful operations on the table by operation type — get / put / query / scan — in ms (aws_dynamodb_endpoint_get_successful_request_latency, aws_dynamodb_endpoint_put_successful_request_latency, aws_dynamodb_endpoint_query_successful_request_latency, aws_dynamodb_endpoint_scan_successful_request_latency).\nScan Returned Items — items returned by Scan operations on the table (aws_dynamodb_endpoint_scan_returned_item_count).\nQuery Returned Items — items returned by Query operations on the table (aws_dynamodb_endpoint_query_returned_item_count).\nRequirements The AWS_DYNAMODB dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the DynamoDB metric families collected from CloudWatch and aggregated under this layer:\nAccount (service) metrics — the aws_dynamodb_* family: throttled requests and throttle events, read / write system errors, user errors, conditional-check failures, transaction conflicts, provisioned vs consumed read / write capacity, per-operation successful request latency (get / put / query / scan), TTL-deleted items, scan / query returned items, the account-level max read / write ceilings, and provisioned capacity utilization.\nTable (endpoint) metrics — the matching aws_dynamodb_endpoint_* family for the same throttling, error, capacity, latency, and returned-item metrics resolved per table.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the table-scope aws_dynamodb_endpoint_* metrics are empty until per-table data is reported, independently of the account-scope metrics. For how to collect these metrics into OAP, see the DynamoDB monitoring setup in the SkyWalking backend docs.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/aws_dynamodb/","title":"\u003c!--"},{"body":" AWS EKS The AWS_EKS layer monitors Amazon Elastic Kubernetes Service (EKS) clusters. SkyWalking ingests EKS observability data through OpenTelemetry — Container Insights / CloudWatch metrics scraped into OAP — and reshapes it into cluster, node, and pod metrics. It groups under AWS in the sidebar.\nIn Horizon\u0026rsquo;s sidebar this layer\u0026rsquo;s entities are renamed to fit the EKS model: services are listed as Clusters, instances as Nodes, and endpoints as EKS services (the Kubernetes services running inside the cluster). The AWS_EKS layer enables the Service (Cluster), Instance (Node), and Endpoint (EKS service) scopes; it does not enable a topology, traces, or logs tab — EKS reports metric data only.\nThis page is the operator reference for the bundled AWS_EKS dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled AWS_EKS template; if an operator has published a customized AWS_EKS template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nCluster list Before opening a cluster, the layer landing page lists every EKS cluster with four sortable columns, sorted by Nodes by default. Each column shows the latest reading averaged across the window:\nNodes — number of nodes in the cluster (latest(eks_cluster_node_count)).\nFailed Nodes — nodes currently in a failed state (latest(eks_cluster_failed_node_count)).\nNamespaces — Kubernetes namespaces in the cluster (latest(eks_cluster_namespace_count)).\nServices — Kubernetes services in the cluster (latest(eks_cluster_service_count)).\nCluster dashboard The primary drill-down for one selected cluster (a Service in OAP terms).\nNode Count — number of nodes over time (eks_cluster_node_count).\nFailed Nodes — nodes in a failed state over time (eks_cluster_failed_node_count).\nNamespace Count — Kubernetes namespaces in the cluster (eks_cluster_namespace_count).\nEKS Service Count — Kubernetes services in the cluster (eks_cluster_service_count).\nCluster Network Errors — cluster-wide receive and transmit error counts, plotted as two series, rx (eks_cluster_net_rx_error) and tx (eks_cluster_net_tx_error).\nCluster Network Drops — cluster-wide dropped packets on receive and transmit, rx (eks_cluster_net_rx_dropped) and tx (eks_cluster_net_tx_dropped).\nNode dashboard For one selected node (an Instance in OAP terms).\nPod Count — pods scheduled on the node (eks_cluster_node_pod_number).\nCPU Utilization (%) — node CPU utilization (eks_cluster_node_cpu_utilization).\nMemory Utilization (%) — node memory utilization (eks_cluster_node_memory_utilization).\nFS Utilization (%) — node filesystem utilization (eks_cluster_node_fs_utilization).\nNetwork RX (KB/s) — node receive throughput in KB/s (eks_cluster_node_net_rx_bytes/1024) on the left axis, with receive errors (eks_cluster_node_net_rx_error) on a second axis so the error count doesn\u0026rsquo;t get lost against the byte scale.\nNetwork TX (KB/s) — node transmit throughput in KB/s (eks_cluster_node_net_tx_bytes/1024) on the left axis, with transmit errors (eks_cluster_node_net_tx_error) on a second axis.\nDisk IO (B/s) — node disk read and write throughput in bytes/s, plotted as read (eks_cluster_node_disk_io_read) and write (eks_cluster_node_disk_io_write).\nPod CPU on Node — aggregate CPU utilization of the pods running on this node (eks_cluster_node_pod_cpu_utilization).\nPod Memory on Node — aggregate memory utilization of the pods running on this node (eks_cluster_node_pod_memory_utilization).\nEKS service dashboard For one selected EKS service (an Endpoint in OAP terms) — a Kubernetes service running inside the cluster, with its pod-level resource and network metrics.\nPod CPU Utilization (%) — CPU utilization across the service\u0026rsquo;s pods (eks_cluster_service_pod_cpu_utilization).\nPod Memory Utilization (%) — memory utilization across the service\u0026rsquo;s pods (eks_cluster_service_pod_memory_utilization).\nPod Network RX (KB/s) — pod receive throughput in KB/s (eks_cluster_service_pod_net_rx_bytes/1024).\nPod RX Errors / s — pod receive error rate (eks_cluster_service_pod_net_rx_error).\nPod Network TX (KB/s) — pod transmit throughput in KB/s (eks_cluster_service_pod_net_tx_bytes/1024).\nPod TX Errors / s — pod transmit error rate (eks_cluster_service_pod_net_tx_error).\nRequirements The AWS_EKS dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the EKS observability metric families, fed in through the OpenTelemetry receiver from Amazon CloudWatch / Container Insights:\nCluster metrics — the eks_cluster_* family at cluster scope: node / failed-node / namespace / service counts and cluster-wide network error and drop counters.\nNode metrics — the eks_cluster_node_* family at node scope: pod count, CPU / memory / filesystem utilization, network receive / transmit bytes and errors, disk read / write IO, and the per-node aggregate pod CPU / memory utilization.\nEKS service metrics — the eks_cluster_service_pod_* family at EKS-service scope: per-service pod CPU / memory utilization and pod network receive / transmit bytes and errors.\nEach metric is queried at its own OAP scope (Cluster / Node / EKS service); OAP does not roll a metric up across scopes, so a node- or service-scope metric stays empty until that level of data is reported. For how to stand up the EKS-to-OAP pipeline, see the layer\u0026rsquo;s setup documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/aws_eks/","title":"\u003c!--"},{"body":" AWS API Gateway The AWS_GATEWAY layer monitors Amazon API Gateway. OAP pulls per-gateway and per-route metrics from AWS CloudWatch — request counts, latency, error rates, cache behavior, and data volume — and presents each gateway as a service in SkyWalking.\nIn Horizon\u0026rsquo;s sidebar this layer is named AWS API Gateway. Its services are listed as AWS Gateways and its endpoints as Routes (each route is a method-plus-resource path on a gateway). The layer enables only the Service and Endpoint sub-tabs — there is no instance scope, no topology, and no traces or logs tab, because CloudWatch reports gateway- and route-level aggregates rather than per-instance, per-request, or relationship data.\nThis page is the operator reference for the bundled AWS_GATEWAY dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled AWS_GATEWAY template; if an operator has published a customized AWS_GATEWAY template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a gateway, the layer landing page lists every AWS Gateway with four sortable columns, sorted by request volume (Requests) by default:\nRequests — total request count over the window (aws_gateway_service_count).\nLatency — average request latency in ms (aws_gateway_service_latency).\n4xx — count of client-error (4xx) responses (aws_gateway_service_4xx).\n5xx — count of server-error (5xx) responses (aws_gateway_service_5xx).\nService dashboard The primary drill-down for one selected gateway.\nRequest Count — total requests handled by the gateway (aws_gateway_service_count).\n4xx Count — client-error responses (aws_gateway_service_4xx).\n5xx Count — server-error responses (aws_gateway_service_5xx).\nRequest Avg Latency — average end-to-end request latency in ms (aws_gateway_service_latency).\nIntegration Avg Latency — average latency between the gateway and its backend integration in ms, isolating backend time from gateway overhead (aws_gateway_service_integration_latency).\nData Processed (HTTP API only) — bytes processed by the gateway, shown in KB (aws_gateway_service_data_processed/1024). Populated only for HTTP API gateways.\nCache Hit Rate (REST API only) — percent of requests served from the gateway cache (aws_gateway_service_cache_hit_rate). Populated only for REST API gateways with caching enabled.\nCache Miss Rate (REST API only) — percent of requests that missed the gateway cache (aws_gateway_service_cache_miss_rate). Populated only for REST API gateways with caching enabled.\nEndpoint dashboard For one selected route (an endpoint under a gateway).\nRequest Count — total requests to the route (aws_gateway_endpoint_count).\n4xx Count — client-error responses on the route (aws_gateway_endpoint_4xx).\n5xx Count — server-error responses on the route (aws_gateway_endpoint_5xx).\nRequest Avg Latency — average request latency in ms (aws_gateway_endpoint_latency).\nIntegration Avg Latency — average gateway-to-backend integration latency in ms (aws_gateway_endpoint_integration_latency).\nData Processed — bytes processed for the route, shown in KB (aws_gateway_endpoint_DataProcessed/1024).\nCache Hit Rate — percent of requests served from cache (aws_gateway_endpoint_cache_hit_rate).\nCache Miss Rate — percent of requests that missed the cache (aws_gateway_endpoint_cache_miss_rate).\nRequirements The AWS_GATEWAY dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the AWS API Gateway receiver enabled and pulling CloudWatch metrics, which produces:\nGateway (service-scope) metrics — the aws_gateway_service_* family: request count, latency, integration latency, 4xx / 5xx counts, data processed, and cache hit / miss rates.\nRoute (endpoint-scope) metrics — the aws_gateway_endpoint_* family: the same measures at route granularity.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a route-scope metric is empty until route-level data is reported. Cache-rate and data-processed widgets stay empty for gateways whose API type (REST vs HTTP API) or configuration does not emit that CloudWatch metric.\nFor setting up the receiver, see the AWS API Gateway monitoring setup guide in the SkyWalking backend docs.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/aws_gateway/","title":"\u003c!--"},{"body":" AWS S3 The AWS_S3 layer monitors Amazon S3 storage by reading CloudWatch request metrics for your buckets, so each S3 bucket appears in SkyWalking as a service with its own request, error, latency, and transfer dashboard.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under AWS and its services are listed as S3 buckets — one entry per monitored bucket. The AWS_S3 layer is a metrics-only layer: it enables the Service scope alone. There is no instance, endpoint, topology, trace, or log sub-tab, because S3 monitoring is CloudWatch metric data rather than agent-instrumented traffic.\nThis page is the operator reference for the bundled AWS_S3 dashboard: what you see for each S3 bucket and what each widget means.\nThe widgets and metrics below are read from the bundled AWS_S3 template; if an operator has published a customized AWS_S3 template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a bucket, the layer landing page lists every S3 bucket with four sortable columns, sorted by request volume (Requests) by default:\nRequests — total request count for the bucket (aws_s3_all_requests).\nAvg Latency — average request latency in ms (aws_s3_request_latency).\n4xx — count of 4xx (client-error) responses (aws_s3_4xx).\n5xx — count of 5xx (server-error) responses (aws_s3_5xx).\nService dashboard The drill-down for one selected S3 bucket.\nAll Request Count — total requests against the bucket over the window (aws_s3_all_requests).\nGET Request Count — GET (read / download) requests (aws_s3_get_requests).\nPUT Request Count — PUT (write / upload) requests (aws_s3_put_requests).\nDELETE Request Count — DELETE requests (aws_s3_delete_requests).\n4xx Count — client-error responses, the 4xx family (aws_s3_4xx).\n5xx Count — server-error responses, the 5xx family (aws_s3_5xx).\nRequest Avg Latency — average total request latency in ms (aws_s3_request_latency).\nFirst Byte Avg Latency — average time to first byte in ms, the latency before any payload starts streaming back (aws_s3_first_latency_bytes).\nDownloaded (KB) — bytes downloaded from the bucket, in KB (aws_s3_downloaded_bytes).\nUploaded (KB) — bytes uploaded to the bucket, in KB (aws_s3_uploaded_bytes).\nRequirements The AWS_S3 dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the AWS S3 monitoring receiver enabled, pulling the bucket\u0026rsquo;s CloudWatch request metrics, which OAP turns into the aws_s3_* service-scope metric family: request counts (aws_s3_all_requests, aws_s3_get_requests, aws_s3_put_requests, aws_s3_delete_requests), error counts (aws_s3_4xx, aws_s3_5xx), latency (aws_s3_request_latency, aws_s3_first_latency_bytes), and transfer volume (aws_s3_downloaded_bytes, aws_s3_uploaded_bytes).\nEvery metric in this dashboard is queried at the Service scope — the S3 bucket — so each bucket you have configured CloudWatch monitoring for appears as one entry in the S3 buckets list. For the OAP-side setup (CloudWatch credentials, the buckets to watch, and the collection interval), follow the AWS S3 monitoring setup guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/aws_s3/","title":"\u003c!--"},{"body":" BanyanDB The BANYANDB layer is the self-observability dashboard for Apache SkyWalking BanyanDB, the native storage that backs an OAP cluster. It surfaces the health of the storage tier itself — write and query throughput, the liaison front door, the data nodes that hold the shards, the lifecycle sidecar that migrates data between tiers, and the per-group load across the measure / stream / trace / property data models.\nIn Horizon\u0026rsquo;s sidebar this layer sits under the Self-Observability group and is named BanyanDB. It maps BanyanDB\u0026rsquo;s own topology onto the standard entity slots: a BanyanDB cluster is a Cluster (the service slot), each running container — a liaison, data, or lifecycle process — is a Container (the instance slot, badged with its container_name), and each storage group is a Group (the endpoint slot). The layer enables the Cluster, Container, and Group dashboards plus a layer-specific Deployment tab; it ships no service topology, no API-dependency view, and no traces or logs tabs.\nThis page is the operator reference for the bundled BANYANDB dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled BANYANDB template; if an operator has published a customized BANYANDB template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nCluster list Before opening a cluster, the layer landing page lists every BanyanDB cluster with three sortable columns, sorted by write rate by default:\nWrite/s — cluster-wide writes per second (meter_banyandb_cluster_write_rate).\nQuery/s — cluster-wide query calls per second (meter_banyandb_cluster_query_rate).\nErrors — cluster-wide errors per minute (meter_banyandb_cluster_error_rate).\nCluster dashboard The primary drill-down for one selected cluster, summarizing the whole storage tier.\nWrite Rate / Query Rate / Error Rate — three headline cards: cluster-wide writes per second across the measure + stream + trace data models (meter_banyandb_cluster_write_rate), gRPC query calls per second seen at the liaison front door (meter_banyandb_cluster_query_rate), and errors per minute summed across the cluster (meter_banyandb_cluster_error_rate). On a healthy cluster the error card reads 0.\nCPU Cores / Memory Used / Disk Used — capacity cards rolled up across the cluster\u0026rsquo;s containers: total CPU cores visible (meter_banyandb_total_cpu_cores), total memory used in GB (meter_banyandb_total_memory_used), and total on-disk bytes used across the data paths in GB (meter_banyandb_total_disk_used).\nCluster Throughput — write rate versus query rate over time on one chart (meter_banyandb_cluster_write_rate, meter_banyandb_cluster_query_rate).\nCluster Errors / min — cluster-wide errors per minute over time (meter_banyandb_cluster_error_rate).\nContainers by Role — a table of the live container count per role (data / liaison), derived from the system uptime gauge (meter_banyandb_reporting_instances). The lifecycle sidecar runs no system collector, so it does not appear here.\nContainer dashboard For one selected container. Because the three BanyanDB roles report different metrics, most widgets are role-specific and appear only on the container they apply to — a liaison container shows the front-door widgets, a data container the storage-engine widgets, and the lifecycle sidecar its migration widgets. A handful of common runtime widgets render on every container, and a few host widgets appear only when that container\u0026rsquo;s system collector reports them.\nCommon runtime (every container)\nCPU Usage — process CPU consumption in cores (meter_banyandb_instance_cpu_usage).\nResident Memory — process resident memory in MB (meter_banyandb_instance_rss_memory).\nGoroutines — live goroutine count (meter_banyandb_instance_goroutines).\nGC Pause (avg) — average Go GC pause per cycle in ms (meter_banyandb_instance_gc_pause_avg).\nGo Heap — Go heap in-use versus next-GC threshold in MB (meter_banyandb_instance_heap_inuse, meter_banyandb_instance_heap_next_gc).\nGo Alloc Rate — Go allocation rate in MB/s (meter_banyandb_instance_alloc_rate).\nHost (when the system collector reports it)\nUptime — days since the node started (meter_banyandb_instance_node_uptime). Absent on the lifecycle sidecar, which runs no system collector.\nSystem Memory Used — host memory used as a percentage (meter_banyandb_instance_system_memory_percent).\nDisk Usage — used / total across the node\u0026rsquo;s data paths as a percentage (meter_banyandb_instance_disk_usage_percent).\nDisk Used / Total — used per data path against total filesystem capacity in GB (meter_banyandb_instance_disk_used_by_path, meter_banyandb_instance_disk_total_by_path). Paths that share one filesystem report identical figures.\nNetwork I/O — per-interface receive / send throughput in KB/s (meter_banyandb_instance_network_recv, meter_banyandb_instance_network_sent).\nLifecycle sidecar (container_name = lifecycle)\nTime Since Last Sync — how long ago the last migration cycle started, shown as a duration (meter_banyandb_instance_lifecycle_last_run). Appears once the first migration cycle has run.\nLast Sync — whether the last migration cycle succeeded (OK) or failed (meter_banyandb_instance_lifecycle_last_run_success).\nMigration Cycles — cumulative tier-migration cycles run by the sidecar (meter_banyandb_instance_lifecycle_migration_cycles).\nLiaison front door (container_name = liaison)\nQuery Rate by Service — gRPC query calls per second, split by data-model service (measure / stream / trace / property) (meter_banyandb_instance_liaison_query_rate).\ngRPC Errors / min — gRPC errors per minute, summed across total + registry + stream-msg (meter_banyandb_instance_liaison_grpc_error_rate). Lazily registered, so it reads 0 on a healthy liaison.\nRegistry Ops / s — schema-registry operations per second at the front door (meter_banyandb_instance_liaison_registry_op_rate).\nWrite Rate — writes per second at the front door across the three data models (meter_banyandb_instance_liaison_write_rate).\nPublish Throughput — the tier-2 publish pipeline (liaison → data) broken out by operation (meter_banyandb_instance_liaison_publish_throughput).\nPublish p99 Latency — p99 send latency of the publish pipeline, per operation (meter_banyandb_instance_liaison_publish_latency_p99).\nPart-sync Bytes — bytes per second streamed to data nodes on the part-sync (file-sync) path in KB/s (meter_banyandb_instance_liaison_publish_bytes). Only chunked file-sync increments this counter; regular write / query publishes are not counted.\nWrite Queue Pending — liaison write-buffer depth: records buffered at the front door before publish (meter_banyandb_instance_liaison_wqueue_pending).\nPublish Batch Throughput — batches published per second by operation (meter_banyandb_instance_liaison_publish_batch_throughput). Hidden until the cluster emits batch metrics.\nPublish Batch p99 — p99 latency of batch publishes in ms (meter_banyandb_instance_liaison_publish_batch_latency_p99).\nData node (container_name = data)\nStored Data Elements — total file elements stored across measure + stream + trace (meter_banyandb_instance_data_total_data).\nWrite Queue (wqueue) — the data-node write queue: pending records, on-disk file parts, and in-memory parts (meter_banyandb_instance_data_wqueue_pending, meter_banyandb_instance_data_wqueue_file_parts, meter_banyandb_instance_data_wqueue_mem_part).\nMerge Loop Rate — file merge-loop iterations per second (meter_banyandb_instance_data_merge_file_rate).\nMerge File Latency — average on-disk file-merge latency per merge loop in ms (meter_banyandb_instance_data_merge_file_latency).\nMerge Parts / Loop — average parts merged per on-disk merge loop (meter_banyandb_instance_data_merge_file_partitions).\nInverted Index Rate — series-index updates and term searches per second across measure + stream storage + stream tst (meter_banyandb_instance_data_series_write_rate, meter_banyandb_instance_data_series_term_search_rate, meter_banyandb_instance_data_stream_tst_write_rate, meter_banyandb_instance_data_stream_tst_term_search_rate).\nIndex Documents — total inverted-index documents, used as a series proxy (meter_banyandb_instance_data_total_series, meter_banyandb_instance_data_stream_tst_total_docs).\nSubscribe Throughput — subscribe-side queue throughput by operation (query / file-sync / batch-write / control) (meter_banyandb_instance_data_queue_sub_throughput).\nSubscribe p99 Latency — p99 latency of subscribe-side queue processing in ms (meter_banyandb_instance_data_queue_sub_latency_p99).\nRetention Disk Usage — per data-model retention disk-usage percentage (meter_banyandb_instance_data_retention_measure_disk_usage_percent, meter_banyandb_instance_data_retention_stream_disk_usage_percent, meter_banyandb_instance_data_retention_trace_disk_usage_percent).\nSubscribe Message Throughput — per-record processing rate the subscriber unpacks from batches in msgs/s (meter_banyandb_instance_data_queue_sub_message_throughput).\nGroup dashboard For one selected group — a BanyanDB storage group, mapped to the endpoint slot. The widgets are organized by data model (measure, stream, trace, property); each model\u0026rsquo;s widgets render only when that model\u0026rsquo;s group reports data, and a final set of queue widgets is common to every group.\nMeasure\nMeasure Write / s — writes per second for this group (meter_banyandb_endpoint_measure_write_rate).\nMeasure Query Latency — mean liaison query latency for this group in ms (meter_banyandb_endpoint_measure_query_latency).\nMeasure Total Data — current stored data elements for this group (meter_banyandb_endpoint_measure_total_data).\nMeasure Merge Rate — file merge-loop iterations per minute (meter_banyandb_endpoint_measure_merge_file_rate).\nMeasure Merge Latency — mean file-merge latency per merge loop in ms (meter_banyandb_endpoint_measure_merge_file_latency).\nMeasure Merge Partitions — average parts merged per file merge loop (meter_banyandb_endpoint_measure_merge_file_partitions).\nMeasure Series Write / s — inverted-index updates per second, used as a series-write proxy (meter_banyandb_endpoint_measure_series_write_rate).\nMeasure Term Search / s — inverted-index term-search invocations per second, the index read-pressure signal (meter_banyandb_endpoint_measure_series_term_search_rate).\nMeasure Total Series — total inverted-index documents for this group, used as a series proxy (meter_banyandb_endpoint_measure_total_series).\nStream\nStream Write / s — writes per second for this group (meter_banyandb_endpoint_stream_write_rate).\nStream Query Latency — mean liaison query latency for this group in ms (meter_banyandb_endpoint_stream_query_latency).\nStream Total Data — current stored data elements for this group (meter_banyandb_endpoint_stream_total_data).\nStream Merge Rate — file merge-loop iterations per minute (meter_banyandb_endpoint_stream_merge_file_rate).\nStream Merge Latency — mean file-merge latency per merge loop in ms (meter_banyandb_endpoint_stream_merge_file_latency).\nStream Merge Partitions — average parts merged per file merge loop (meter_banyandb_endpoint_stream_merge_file_partitions).\nStream Series Write / s — inverted-index updates per second, used as a series-write proxy (meter_banyandb_endpoint_stream_series_write_rate).\nStream TST Index Write / s — stream tst-scope inverted-index updates per second, distinct from the storage-scope series index (meter_banyandb_endpoint_stream_tst_index_write_rate).\nStream Term Search / s — inverted-index term-search invocations per second (meter_banyandb_endpoint_stream_series_term_search_rate).\nStream Total Series — total inverted-index documents for this group (meter_banyandb_endpoint_stream_total_series).\nStream TST Total Series — the stream tst-scope index document total, distinct from the storage-scope series index (meter_banyandb_endpoint_stream_tst_total_series).\nTrace\nTrace Write / s — writes per second for this group (meter_banyandb_endpoint_trace_write_rate).\nTrace Query Latency — mean liaison query latency for this group in ms (meter_banyandb_endpoint_trace_query_latency).\nTrace Total Data — current stored data elements for this group (meter_banyandb_endpoint_trace_total_data).\nTrace Merge Rate — file merge-loop iterations per minute (meter_banyandb_endpoint_trace_merge_file_rate).\nTrace Merge Latency — mean file-merge latency per merge loop in ms (meter_banyandb_endpoint_trace_merge_file_latency).\nTrace Merge Partitions — average parts merged per file merge loop (meter_banyandb_endpoint_trace_merge_file_partitions).\nTrace Series Write / s — inverted-index updates per second, used as a series-write proxy (meter_banyandb_endpoint_trace_series_write_rate).\nTrace Term Search / s — inverted-index term-search invocations per second (meter_banyandb_endpoint_trace_series_term_search_rate).\nTrace Total Series — total inverted-index documents for this group (meter_banyandb_endpoint_trace_total_series).\nProperty\nProperty Index Write / s — property-registry inverted-index updates per second, the property model\u0026rsquo;s write signal (meter_banyandb_endpoint_property_index_write_rate).\nProperty Index Merge Rate — property inverted-index segment merges per minute; property has no tst merge loop (meter_banyandb_endpoint_property_index_merge_rate).\nProperty Index Merge Latency — mean property inverted-index merge latency in ms (meter_banyandb_endpoint_property_index_merge_latency).\nProperty Term Search / s — property term-search invocations per second; property is read via the registry / term-search path rather than the liaison query method, so this is its read-load signal (meter_banyandb_endpoint_property_series_term_search_rate).\nProperty Total Series — total property inverted-index documents for this group (meter_banyandb_endpoint_property_total_series).\nQueue (every group)\nSubscribe Throughput — subscribe-side queue messages per second for this group, by operation (meter_banyandb_endpoint_queue_throughput).\nPublish p99 — publish-side queue p99 latency for this group in ms (meter_banyandb_endpoint_queue_latency_p99).\nBatch Throughput — per-group write-batch rate, by operation (meter_banyandb_endpoint_queue_batch_throughput).\nMessage Throughput — per-group per-record rate, by operation (meter_banyandb_endpoint_queue_message_throughput).\nPart-sync Bytes / s — part-sync (file-sync) bytes per second for this group in KB/s (meter_banyandb_endpoint_publish_bytes). Only the chunked part-streaming path increments this; regular write / query publishes are not counted.\nDeployment The BANYANDB layer enables the layer-specific Deployment tab — the deployment topology of one cluster\u0026rsquo;s own containers and the intra-cluster calls between them. Pick a cluster from the header and the tab draws its containers as health-ring nodes laid out left → right along the calls between them, with animated edge flow, a per-edge metric panel, and a node popover that opens the container dashboard. For how the Deployment tab is read and navigated in general, see the Deployment section of Layer Dashboard Templates.\nContainers are grouped into three roles by their node_role / node_type attributes:\nLiaison — the front door. Its node center shows Query/s (meter_banyandb_instance_liaison_query_rate) and its health ring tracks gRPC err/min (meter_banyandb_instance_liaison_grpc_error_rate).\nData — the storage nodes. Center shows Ingest/s (meter_banyandb_instance_data_queue_sub_throughput) and the ring tracks Disk % (meter_banyandb_instance_disk_usage_percent).\nLifecycle — the tier-migration sidecar. Center shows cumulative Cycles (meter_banyandb_instance_lifecycle_migration_cycles) and the ring tracks Last OK (meter_banyandb_instance_lifecycle_last_run_success).\nBecause role-pair edges are configured, the Deployment map gains a Flows sub-tab listing every edge grouped by role-pair. Each edge type carries its own client-side (publish) and server-side (subscribe) metrics, so a liaison → data call surfaces a different metric set than a liaison → liaison forward or a lifecycle → data migration:\nliaison → data — the main write / query path. Per-operation Write/s, Query/s, and Part-sync/s throughput; Write p99 and Query p99 latency; Part-sync B/s bytes; and Errors/s. Each is paired across the publish side (meter_banyandb_instance_relation_publish_*, filtered by operation) and the subscribe side (meter_banyandb_instance_relation_queue_sub_*).\nliaison → liaison — node-to-node forwarding. Forward/s and Forward p99 for the batch-write forward, Control/s for the control channel, and Errors/s (meter_banyandb_instance_relation_publish_throughput{operation='batch-write'} and the matching subscribe / control / error counters).\nlifecycle → data — the tier-migration path. Migrate/s throughput, Migrate p99 latency, Migrate B/s bytes, and Errors/s (meter_banyandb_instance_relation_migration_* on the publish side, meter_banyandb_instance_relation_queue_sub_* on the subscribe side).\nany other pair — a generic fallback showing aggregated Msg/s and p99 (aggregate_labels(meter_banyandb_instance_relation_publish_throughput,sum) and the matching latency / subscribe counters), so an edge that matches no specific role-pair still reports something.\nRequirements The BANYANDB dashboard is a pure consumer of what OAP reports about its BanyanDB storage tier — it invents no data, and a widget with no backing data simply reads no data (or 0 for the lazily-registered error counters). To populate it, OAP needs BanyanDB self-observability enabled so that BanyanDB exposes its metrics and OAP ingests them into the meter_banyandb_* families:\nCluster metrics — meter_banyandb_cluster_* and the meter_banyandb_total_* capacity rollups for the Cluster list and Cluster dashboard.\nContainer metrics — meter_banyandb_instance_* for the per-container runtime, host, liaison, data, and lifecycle widgets. A container only shows the families its role emits, and the host widgets need a running system collector (absent on the lifecycle sidecar).\nGroup metrics — the per-data-model meter_banyandb_endpoint_* families (measure / stream / trace / property, plus the shared queue counters) for the Group dashboard.\nRelation metrics — meter_banyandb_instance_relation_* (publish / subscribe / migration throughput, latency, bytes, and error counters) for the Deployment tab\u0026rsquo;s edges.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a container- or group-scope metric is empty until that level of data is reported, and an entire data model\u0026rsquo;s group widgets stay hidden until that model\u0026rsquo;s group reports.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/banyandb/","title":"\u003c!--"},{"body":" BookKeeper The BOOKKEEPER layer monitors Apache BookKeeper, the distributed write-ahead log storage that backs systems such as Apache Pulsar. OAP gathers BookKeeper\u0026rsquo;s metrics through the OpenTelemetry receiver and aggregates them per bookie node and per cluster.\nIn Horizon\u0026rsquo;s sidebar this layer is named BookKeeper. Its services are listed as BookKeeper clusters and its instances as Bookies — each bookie is one storage node in the cluster. This layer enables the Service and Instance scopes only: there is no endpoint scope, no topology, and no traces or logs tab, because BookKeeper reports node-level meters rather than request traffic.\nThis page is the operator reference for the bundled BOOKKEEPER dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled BOOKKEEPER template; if an operator has published a customized BOOKKEEPER template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every BookKeeper cluster with four sortable columns, sorted by Ledgers by default:\nLedgers — total ledgers held across the cluster\u0026rsquo;s bookies (meter_bookkeeper_bookie_ledgers_count, summed).\nEntries — total entries stored across the cluster\u0026rsquo;s bookies (meter_bookkeeper_bookie_entries_count, summed).\nWritable Dirs — number of ledger directories currently accepting writes (meter_bookkeeper_bookie_ledger_writable_dirs, summed).\nDir Usage — the ledger data directory\u0026rsquo;s fill level (meter_bookkeeper_bookie_ledger_dir_data_bookkeeper_ledgers_usage).\nService dashboard The primary drill-down for one selected BookKeeper cluster. Every widget aggregates the cluster\u0026rsquo;s bookies with aggregate_labels(..., sum).\nBookie Ledgers — ledgers held across the cluster over time (meter_bookkeeper_bookie_ledgers_count).\nBookie Entries — entries stored across the cluster over time (meter_bookkeeper_bookie_entries_count).\nWritable Ledger Dirs — ledger directories currently accepting writes (meter_bookkeeper_bookie_ledger_writable_dirs).\nWrite Cache — the bookie write cache on a dual axis: cache size in MB on the left (meter_bookkeeper_bookie_write_cache_size, divided to MB) and cached entry count on the right (meter_bookkeeper_bookie_write_cache_count).\nRead Cache — the bookie read cache on a dual axis: cache size in MB on the left (meter_bookkeeper_bookie_read_cache_size, divided to MB) and cached entry count on the right (meter_bookkeeper_bookie_read_cache_count).\nRead / Write Rate (B/s) — bytes per second served and ingested, plotted together: read (meter_bookkeeper_bookie_read_rate) and write (meter_bookkeeper_bookie_write_rate).\nLedger Dir Usage — fill level of the ledger data directory over time (meter_bookkeeper_bookie_ledger_dir_data_bookkeeper_ledgers_usage).\nInstance dashboard For one selected bookie. These widgets cover the bookie\u0026rsquo;s JVM runtime and its internal thread pools.\nJVM Memory Pool (MB) — used memory per JVM memory pool in MB (meter_bookkeeper_node_jvm_memory_pool_used).\nJVM Memory (MB) — JVM memory in MB: used, committed, and init (meter_bookkeeper_node_jvm_memory_used, meter_bookkeeper_node_jvm_memory_committed, meter_bookkeeper_node_jvm_memory_init).\nJVM Threads — thread counts: current, daemon, peak, and deadlocked (meter_bookkeeper_node_jvm_threads_current, meter_bookkeeper_node_jvm_threads_daemon, meter_bookkeeper_node_jvm_threads_peak, meter_bookkeeper_node_jvm_threads_deadlocked).\nGC — garbage-collection activity on a dual axis: cumulative GC seconds on the left (meter_bookkeeper_node_jvm_gc_collection_seconds_sum) and GC count on the right (meter_bookkeeper_node_jvm_gc_collection_seconds_count).\nThread Executor — the bookie\u0026rsquo;s task executor: completed, tasks completed, rejected, and failed (meter_bookkeeper_node_thread_executor_completed, meter_bookkeeper_node_thread_executor_tasks_completed, meter_bookkeeper_node_thread_executor_tasks_rejected, meter_bookkeeper_node_thread_executor_tasks_failed).\nPooled Threads — thread counts for the high-priority and read pools (meter_bookkeeper_node_high_priority_threads, meter_bookkeeper_node_read_thread_pool_threads).\nPool Max Queue Size — the maximum queue size of the high-priority and read thread pools (meter_bookkeeper_node_high_priority_thread_max_queue_size, meter_bookkeeper_node_read_thread_pool_max_queue_size).\nRequirements The BOOKKEEPER dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nBookie metrics — the meter_bookkeeper_bookie_* family (ledgers, entries, writable directories, directory usage, read/write caches, and read/write rates), aggregated at the BookKeeper cluster (Service) scope.\nBookie node metrics — the meter_bookkeeper_node_* family (JVM memory, threads, and GC, plus the bookie\u0026rsquo;s thread executor and thread pools), reported at the bookie (ServiceInstance) scope.\nThese metrics come from BookKeeper\u0026rsquo;s own OpenTelemetry export, gathered by OAP\u0026rsquo;s OpenTelemetry receiver — see the BookKeeper monitoring setup. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the instance-scope widgets stay empty until per-bookie data is reported.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/bookkeeper/","title":"\u003c!--"},{"body":" Browser The BROWSER layer is where SkyWalking\u0026rsquo;s browser agent (the client-side JavaScript SDK) reports. It is real-user monitoring: page views, front-end errors, page-load timing, and Core Web Vitals collected from the visitor\u0026rsquo;s browser rather than from a server-side agent.\nIn Horizon\u0026rsquo;s sidebar this layer is named Browser. Its top-level entities are web applications, listed as Apps; each app reports under one or more Versions (the instance slot), and each app serves a set of Pages (the endpoint slot). So where the GENERAL layer reads \u0026ldquo;Service / Instance / Endpoint\u0026rdquo;, BROWSER reads \u0026ldquo;App / Version / Page\u0026rdquo;. The layer enables the App, Version, and Page dashboards, plus the Traces tab and a Browser Logs tab — the per-page front-end error stream, which can de-obfuscate a minified JavaScript stack against a source map you upload. BROWSER has no service topology; there is no map view.\nThis page is the operator reference for the bundled BROWSER dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled BROWSER template; if an operator has published a customized BROWSER template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nApp list Before opening an app, the layer landing page lists every BROWSER app with three columns, sorted by traffic (Page Views) by default:\nPage Views — page views per minute (browser_app_pv).\nError Rate — percent of page views that recorded an error (browser_app_error_rate/100).\nErrors — total front-end errors in the window (browser_app_error_sum).\nApp dashboard The primary drill-down for one selected app.\nApp Load (PV) — page views per minute for the app (browser_app_pv).\nApp Error Rate — percent of page views that recorded an error (browser_app_error_rate/100).\nApp Error Count — total front-end errors per minute (browser_app_error_sum).\nTop Hot Pages — the app\u0026rsquo;s busiest pages, with tabs to re-rank by PV (browser_app_page_pv, /min), Errors (browser_app_page_error_sum), and Error Rate (browser_app_page_error_rate, %), worst-first. Click a row to jump into that page.\nTop Versions — the app\u0026rsquo;s versions broken down the same three ways: PV (browser_app_single_version_pv, /min), Errors (browser_app_single_version_error_sum), and Error Rate (browser_app_single_version_error_rate, %).\nVersion dashboard For one selected app Version — the per-release view of the same load and error signals.\nVersion PV — page views per minute for this version (browser_app_single_version_pv).\nVersion Error Rate — percent of this version\u0026rsquo;s page views that recorded an error (browser_app_single_version_error_rate/100).\nVersion Error Count — total front-end errors per minute for this version (browser_app_single_version_error_sum).\nPage dashboard For one selected Page — the deepest scope, where browser timing and Web Vitals live. This is the page-performance view: most of these metrics exist only at page scope.\nFirst Meaningful Paint Percentile — p50 / p75 / p90 / p95 / p99 of FMP latency for the page, in ms (browser_app_page_fmp_percentile). Below 1s at p75 is a common target.\nPage Load Percentile — p50 / p75 / p90 / p95 / p99 of full page-load time, in ms (browser_app_page_load_page_percentile).\nTime-to-Live Percentile — p50 / p75 / p90 / p95 / p99 of the page\u0026rsquo;s time-to-live, in ms (browser_app_page_ttl_percentile).\nFirst Pack Latency Percentile — p50 / p75 / p90 / p95 / p99 of first-pack latency, in ms (browser_app_page_first_pack_percentile).\nPage Performance Breakdown — average time spent in each phase of the page load, in ms, on one chart: DNS, redirect, TCP, TTFB, transfer, DOM analysis, DOM ready, FPT, load, and resource (browser_app_page_dns_avg, browser_app_page_redirect_avg, browser_app_page_tcp_avg, browser_app_page_ttfb_avg, browser_app_page_trans_avg, browser_app_page_dom_analysis_avg, browser_app_page_dom_ready_avg, browser_app_page_fpt_avg, browser_app_page_load_page_avg, browser_app_page_res_avg).\nPage Errors by Type — front-end error counters per minute split by source: resource, JS, AJAX, and unknown (browser_app_page_resource_error_sum, browser_app_page_js_error_sum, browser_app_page_ajax_error_sum, browser_app_page_unknown_error_sum).\nWeb Vitals — Core Web Vitals as averages per minute: FMP (ms), LCP (ms), and CLS (browser_app_web_vitals_fmp_avg, browser_app_web_vitals_lcp_avg, browser_app_web_vitals_cls_avg / 1000 — CLS is scaled down to its typical 0 – 1 score range).\nInteraction to Next Paint Percentile — p50 / p75 / p90 / p95 / p99 of INP, in ms (browser_app_web_interaction_inp_percentile). INP is the responsiveness metric that replaces FID.\nRequirements The BROWSER dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, your front-end must run the SkyWalking browser agent (the client-side JavaScript SDK) reporting to OAP, which produces:\nApp metrics — the browser_app_* family (page views, error rate, error sum), produced by OAP from browser-agent reports, for the App list and App dashboard.\nVersion metrics — browser_app_single_version_* (PV, error rate, error sum) for the Top Versions widget and the Version dashboard.\nPage metrics — browser_app_page_* (PV, errors, the timing percentiles, and the per-phase performance averages) and the browser_app_web_vitals_* / browser_app_web_interaction_* families for the Page dashboard.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a version- or page-scope metric is empty until that level of data is reported. BROWSER carries no relation metrics, so it has no topology or map view.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/browser/","title":"\u003c!--"},{"body":" Cilium Service The CILIUM_SERVICE layer monitors Kubernetes services observed through Cilium\u0026rsquo;s eBPF data plane. SkyWalking collects L4 (TCP) packet activity and L7 protocol telemetry (HTTP, DNS, Kafka) that Cilium reports for each service, giving you network-level and protocol-level visibility into the mesh without instrumenting the application. It belongs to the Kubernetes layer group.\nIn Horizon\u0026rsquo;s sidebar this layer is named Cilium Service. Its services are listed as Services, instances as Pods, and endpoints as Endpoints — service names are grouped and displayed by their Kubernetes namespace. The CILIUM_SERVICE layer enables the Service, Pod, Endpoint, and Topology sub-tabs. It does not enable Traces or Logs.\nThis page is the operator reference for the bundled CILIUM_SERVICE dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled CILIUM_SERVICE template; if an operator has published a customized CILIUM_SERVICE template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a service, the layer landing page lists every CILIUM_SERVICE service with four sortable columns, sorted by traffic (RPM) by default:\nRPM — protocol calls per minute (cilium_service_protocol_cpm).\nLatency — average protocol call duration in ms (cilium_service_protocol_call_duration/cilium_service_protocol_cpm/1000000).\nSuccess Rate — percent of successful protocol calls (cilium_service_protocol_call_success_count/cilium_service_protocol_cpm*100).\nTCP Drop — dropped read + write packets per minute at L4 (cilium_service_l4_read_pkg_drop_cpm + cilium_service_l4_write_pkg_drop_cpm).\nService dashboard The primary drill-down for one selected service. It splits into an L4 (TCP) row and per-protocol (HTTP, DNS, Kafka) groups.\nL4 (TCP)\nL4 Read Packages/min — inbound packets per minute (cilium_service_l4_read_pkg_cpm).\nL4 Write Packages/min — outbound packets per minute (cilium_service_l4_write_pkg_cpm).\nTCP Drop / min — dropped read and write packets per minute, plotted as two series (cilium_service_l4_read_pkg_drop_cpm, cilium_service_l4_write_pkg_drop_cpm).\nTCP Drop by Reason — dropped-packet count broken out by Cilium\u0026rsquo;s drop-reason label (cilium_service_l4_drop_reason_count).\nHTTP\nHTTP Load — HTTP calls per minute alongside the successful-call count (cilium_service_protocol_http_call_cpm, cilium_service_protocol_http_call_success_count).\nHTTP Duration — average HTTP call duration in ms (cilium_service_protocol_http_call_duration/cilium_service_protocol_http_call_cpm/1000000).\nHTTP Non-2xx Status — HTTP responses per minute by status class: 1xx / 3xx / 4xx / 5xx (cilium_service_protocol_http_status_1xx_cpm, cilium_service_protocol_http_status_3xx_cpm, cilium_service_protocol_http_status_4xx_cpm, cilium_service_protocol_http_status_5xx_cpm).\nDNS\nDNS Load — DNS calls per minute alongside the successful-call count (cilium_service_protocol_dns_call_cpm, cilium_service_protocol_dns_call_success_count).\nDNS Duration — average DNS call duration in ms (cilium_service_protocol_dns_call_duration/cilium_service_protocol_dns_call_cpm/1000000).\nDNS Errors / min — DNS error count per minute (cilium_service_protocol_dns_error_count).\nKafka\nKafka Load — Kafka calls per minute alongside the successful-call count (cilium_service_protocol_kafka_call_cpm, cilium_service_protocol_kafka_call_success_count).\nKafka Duration — average Kafka call duration in ms (cilium_service_protocol_kafka_call_duration/cilium_service_protocol_kafka_call_cpm/1000000).\nKafka Errors / min — Kafka error count per minute (cilium_service_protocol_kafka_call_error_count).\nPod dashboard For one selected pod (a service instance). The widgets mirror the service view at instance scope, covering L4 plus the HTTP, DNS, and Kafka protocols.\nL4 Read Packages/min — inbound packets per minute for the pod (cilium_service_instance_l4_read_pkg_cpm).\nL4 Write Packages/min — outbound packets per minute for the pod (cilium_service_instance_l4_write_pkg_cpm).\nHTTP Load — HTTP calls per minute alongside the successful-call count (cilium_service_instance_protocol_http_call_cpm, cilium_service_instance_protocol_http_call_success_count).\nHTTP Non-2xx Status — HTTP responses per minute by status class: 1xx / 3xx / 4xx / 5xx (cilium_service_instance_protocol_http_status_1xx_cpm, cilium_service_instance_protocol_http_status_3xx_cpm, cilium_service_instance_protocol_http_status_4xx_cpm, cilium_service_instance_protocol_http_status_5xx_cpm).\nDNS Load — DNS calls per minute alongside the successful-call count (cilium_service_instance_protocol_dns_call_cpm, cilium_service_instance_protocol_dns_call_success_count).\nDNS Errors — DNS error count (cilium_service_instance_protocol_dns_error_count).\nKafka Load — Kafka calls per minute alongside the successful-call count (cilium_service_instance_protocol_kafka_call_cpm, cilium_service_instance_protocol_kafka_call_success_count).\nEndpoint dashboard For one selected endpoint. Cilium endpoints carry L7 protocol traffic, so this scope is HTTP- and DNS-focused.\nHTTP Load — HTTP calls per minute alongside the successful-call count (cilium_endpoint_protocol_http_call_cpm, cilium_endpoint_protocol_http_call_success_count).\nHTTP Duration — average HTTP call duration in ms (cilium_endpoint_protocol_http_call_duration/cilium_endpoint_protocol_http_call_cpm/1000000).\nHTTP Non-2xx Status — HTTP responses per minute by status class: 1xx / 3xx / 4xx / 5xx (cilium_endpoint_protocol_http_status_1xx_cpm, cilium_endpoint_protocol_http_status_3xx_cpm, cilium_endpoint_protocol_http_status_4xx_cpm, cilium_endpoint_protocol_http_status_5xx_cpm).\nDNS Load — DNS calls per minute alongside the successful-call count (cilium_endpoint_protocol_dns_call_cpm, cilium_endpoint_protocol_dns_call_success_count).\nDNS Duration — average DNS call duration in ms (cilium_endpoint_protocol_dns_call_duration/cilium_endpoint_protocol_dns_call_cpm/1000000).\nDNS Errors — DNS error count (cilium_endpoint_protocol_dns_error_count).\nTopology and maps The CILIUM_SERVICE layer ships a service topology with an instance-level drill-down.\nTopology (service map) — each service node is decorated with RPM (cilium_service_protocol_cpm), a Success % health ring (cilium_service_protocol_call_success_count/cilium_service_protocol_cpm*100, colored green / amber / red — higher is better, so the ring turns red as the success rate drops), and Latency (cilium_service_protocol_call_duration/cilium_service_protocol_cpm/1000000). Each call edge carries server-side HTTP RPM (cilium_service_relation_server_protocol_http_call_cpm) and Avg Latency (cilium_service_relation_server_protocol_http_call_duration/cilium_service_relation_server_protocol_http_call_cpm/1000000).\nInstance map — from a call between two services on the service map, drill into the pod-to-pod calls between them. Each instance node shows HTTP RPM (cilium_service_instance_protocol_http_call_cpm) and Avg Latency (cilium_service_instance_protocol_http_call_duration/cilium_service_instance_protocol_http_call_cpm/1000000); each edge carries server-side HTTP RPM (cilium_service_instance_relation_server_protocol_http_call_cpm) and Avg Latency (cilium_service_instance_relation_server_protocol_http_call_duration/cilium_service_instance_relation_server_protocol_http_call_cpm/1000000).\nFor how these maps are read and navigated, see the 3D Infrastructure Map for the cross-layer view, and the topology section of Layer Dashboard Templates for how the node and edge metrics are configured.\nRequirements The CILIUM_SERVICE dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Cilium monitoring enabled, with the Cilium-Hubble fetcher feeding SkyWalking. Specifically:\nL4 metrics — the cilium_service_l4_* family (read / write packet rates, packet drops, and drop-reason breakdown) for the service-scope TCP widgets.\nProtocol metrics — the cilium_service_protocol_*, cilium_service_instance_protocol_*, and cilium_endpoint_protocol_* families covering HTTP, DNS, and Kafka call counts, durations, success counts, status classes, and errors, at their respective service / instance / endpoint scopes.\nRelation metrics — cilium_service_relation_server_protocol_* and cilium_service_instance_relation_server_protocol_* for the service map and instance map edges.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so an instance- or endpoint-scope metric is empty until that level of data is reported. A pod or endpoint that carries only one protocol shows no data for the others. For setup, see the Cilium monitoring guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/cilium_service/","title":"\u003c!--"},{"body":" ClickHouse The CLICKHOUSE layer monitors ClickHouse database clusters. SkyWalking collects ClickHouse\u0026rsquo;s internal metrics — queries, query latency, merges and mutations, data parts, replication, ZooKeeper / Keeper coordination, and per-node host stats — through OpenTelemetry, and Horizon renders them as a cluster-level and a node-level dashboard. This is a metrics-only layer: there are no traces, logs, endpoints, or a topology map for ClickHouse.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Databases and named ClickHouse. Its services are listed as ClickHouse clusters and its instances as Nodes. Because the layer carries no endpoint, topology, trace, or log data, it enables only two sub-tabs: a Service (cluster) dashboard and an Instance (node) dashboard.\nThis page is the operator reference for the bundled CLICKHOUSE dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled CLICKHOUSE template; if an operator has published a customized CLICKHOUSE template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every ClickHouse cluster with four sortable columns, sorted by select rate (Select / s) by default:\nSelect / s — SELECT queries per second across the cluster (aggregate_labels(meter_clickhouse_query_select_rate,sum)).\nInsert / s — INSERT queries per second across the cluster (aggregate_labels(meter_clickhouse_query_insert_rate,sum)).\nSlow Reads — slow file reads across the cluster (aggregate_labels(meter_clickhouse_query_slow,sum)).\nOpen Files — the latest count of open files across the cluster (latest(aggregate_labels(meter_clickhouse_file_open,sum))).\nService dashboard The cluster-level drill-down for one selected ClickHouse cluster. Every widget aggregates across the cluster\u0026rsquo;s nodes with aggregate_labels(...,sum).\nFiles Open — the latest number of open files in the cluster, as a single card (latest(aggregate_labels(meter_clickhouse_file_open,sum))).\nQPS — query rate per second, plotted as two series: select (aggregate_labels(meter_clickhouse_query_select_rate,sum)) and insert (aggregate_labels(meter_clickhouse_query_insert_rate,sum)).\nQueries — query counts split into total, select, and insert (aggregate_labels(meter_clickhouse_query,sum), aggregate_labels(meter_clickhouse_query_select,sum), aggregate_labels(meter_clickhouse_query_insert,sum)).\nQuery Time (ms) — average time per query in ms, as avg, select, and insert series, each computed as total query microseconds divided by query count and converted to ms (aggregate_labels(meter_clickhouse_querytime_microseconds,sum)/aggregate_labels(meter_clickhouse_query,sum)/1000 and the matching _select_ / _insert_ pair).\nConnections — open client connections by protocol: TCP (aggregate_labels(meter_clickhouse_tcp_connections,sum)) and HTTP (aggregate_labels(meter_clickhouse_http_connections,sum)).\nSlow Reads — slow file reads across the cluster (aggregate_labels(meter_clickhouse_query_slow,sum)).\nMerge / Mutations — background merge operations (aggregate_labels(meter_clickhouse_background_merge,sum)) and mutations (aggregate_labels(meter_clickhouse_mutations,sum)).\nInsert Throughput — insert volume on a dual axis: bytes/s on the left (aggregate_labels(meter_clickhouse_inserted_bytes,sum)) and rows/s on the right (aggregate_labels(meter_clickhouse_inserted_rows,sum)).\nDelayed Inserts (s) — inserts that were throttled / delayed (aggregate_labels(meter_clickhouse_delayed_inserts,sum)).\nActive Data Parts — the number of active MergeTree data parts in the cluster (aggregate_labels(meter_clickhouse_parts_active,sum)).\nReplicated Fetch / Send — replication traffic between replicas: fetch (aggregate_labels(meter_clickhouse_replicated_fetch,sum)) and send (aggregate_labels(meter_clickhouse_replicated_send,sum)).\nZookeeper Activity — the coordination layer\u0026rsquo;s health, with the latest sessions and watches (latest(aggregate_labels(meter_clickhouse_zookeeper_session,sum)), latest(aggregate_labels(meter_clickhouse_zookeeper_watch,sum))) plus bytes sent and bytes recv over time (aggregate_labels(meter_clickhouse_zookeeper_bytes_sent,sum), aggregate_labels(meter_clickhouse_zookeeper_bytes_received,sum)).\nKeeper Alive Conns — the latest count of alive ClickHouse Keeper connections, as a single card (latest(aggregate_labels(meter_clickhouse_keeper_connections_alive,sum))).\nKeeper Outstanding Requests — the latest count of outstanding ClickHouse Keeper requests, as a single card (latest(aggregate_labels(meter_clickhouse_keeper_outstanding_requests,sum))).\nInstance dashboard The node-level drill-down for one selected ClickHouse node. These widgets read the per-node meter_clickhouse_instance_* family directly, with no cross-node aggregation.\nUptime (days) — how long the node has been running, as a card, converted from seconds (latest(meter_clickhouse_instance_uptime)/3600/24).\nVersion — the node\u0026rsquo;s ClickHouse version, as a card (latest(meter_clickhouse_instance_version)).\nCPU (cores) — CPU consumption expressed in cores (meter_clickhouse_instance_cpu_usage/1000000).\nMemory (%) — used vs available memory percentage (meter_clickhouse_instance_memory_usage, meter_clickhouse_instance_memory_available).\nNetwork (B) — bytes receive vs send on the node (meter_clickhouse_instance_network_receive_bytes, meter_clickhouse_instance_network_send_bytes).\nConnections — open client connections by protocol: TCP (meter_clickhouse_instance_tcp_connections) and HTTP (meter_clickhouse_instance_http_connections).\nQueries — query counts split into total, select, and insert (meter_clickhouse_instance_query, meter_clickhouse_instance_query_select, meter_clickhouse_instance_query_insert).\nQPS — query rate per second, as select and insert series (meter_clickhouse_instance_query_select_rate, meter_clickhouse_instance_query_insert_rate).\nQuery Time (ms) — average time per query in ms, as avg, select, and insert series, each total query microseconds divided by query count and converted to ms (meter_clickhouse_instance_querytime_microseconds/meter_clickhouse_instance_query/1000 and the matching _select_ / _insert_ pair).\nFile Slow Read — slow file reads on the node (meter_clickhouse_instance_query_slow).\nBackground Merge — background merge operations on the node (meter_clickhouse_instance_background_merge).\nMutations — mutation operations on the node (meter_clickhouse_instance_mutations).\nFiles Open — the latest number of open files on the node, as a card (latest(meter_clickhouse_instance_file_open)).\nInsert Throughput — insert volume on a dual axis: bytes/s on the left (meter_clickhouse_instance_inserted_bytes) and rows/s on the right (meter_clickhouse_instance_inserted_rows).\nDelayed Inserts (s) — inserts that were throttled / delayed on the node (meter_clickhouse_instance_delayed_inserts).\nRequirements The CLICKHOUSE dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs ClickHouse metrics delivered through the OpenTelemetry receiver, which OAP aggregates into the meter_clickhouse_* families:\nCluster (service-scope) metrics — the meter_clickhouse_* family (queries, query rate, query time, connections, slow reads, merges, mutations, data parts, insert throughput, delayed inserts, replication, ZooKeeper, and Keeper), which the cluster dashboard and the Service list aggregate across nodes.\nNode (instance-scope) metrics — the meter_clickhouse_instance_* family (uptime, version, CPU, memory, network, connections, queries, query rate, query time, slow reads, merges, mutations, open files, and insert throughput) for the node dashboard.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the node-scope widgets stay empty until per-node data is reported. Setting up the ClickHouse OpenTelemetry collection is described in the upstream ClickHouse monitoring guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/clickhouse/","title":"\u003c!--"},{"body":" Elasticsearch The ELASTICSEARCH layer monitors Elasticsearch clusters that OAP scrapes through its Elasticsearch monitoring receiver. It groups under Databases in the sidebar and gives an operator the cluster-health, node-runtime, and per-index view that an Elasticsearch admin expects.\nIn Horizon\u0026rsquo;s sidebar this layer carries the display name Elasticsearch. An Elasticsearch cluster maps onto SkyWalking\u0026rsquo;s entity scopes, and the layer renames each slot to match: services are listed as ES clusters, instances as Nodes, and endpoints as Indices. The layer enables three drill-down tabs — Service (the cluster dashboard), Instance (a node), and Endpoint (an index). It ships no topology, traces, or logs tabs.\nThis page is the operator reference for the bundled ELASTICSEARCH dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled ELASTICSEARCH template; if an operator has published a customized ELASTICSEARCH template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nES clusters list Before opening a cluster, the layer landing page lists every Elasticsearch cluster with four sortable columns, sorted by Shards by default:\nHealth — the cluster health status (meter_elasticsearch_cluster_health_status), averaged over the window.\nShards — total active shards across the cluster (meter_elasticsearch_cluster_shards_total, latest value).\nNodes — number of nodes in the cluster (meter_elasticsearch_cluster_nodes, latest value).\nUnassigned — shards the cluster has not yet placed on a node (meter_elasticsearch_cluster_unassigned_shards_total, latest value) — a non-zero value is the usual first sign of a cluster under stress.\nService dashboard (cluster) The primary drill-down for one selected cluster — the cluster-wide health and capacity picture.\nCluster Health — a table of the current health status and value (meter_elasticsearch_cluster_health_status, latest), the green / yellow / red rollup Elasticsearch reports for the cluster.\nNodes — number of nodes currently in the cluster (meter_elasticsearch_cluster_nodes, latest).\nPending Tasks — average count of cluster-level tasks queued for the master (meter_elasticsearch_cluster_pending_tasks_total) — a rising queue points at master-node pressure.\nPrimary Shards — total primary shards (meter_elasticsearch_cluster_primary_shards_total, latest).\nActive Shards — total active shards (meter_elasticsearch_cluster_shards_total, latest).\nInitializing — shards currently initializing (meter_elasticsearch_cluster_initializing_shards_total, latest).\nRelocating — shards being moved between nodes (meter_elasticsearch_cluster_relocating_shards_total, latest).\nUnassigned — shards not assigned to any node (meter_elasticsearch_cluster_unassigned_shards_total, latest).\nDelayed Unassigned — unassigned shards whose reassignment is being delayed (meter_elasticsearch_cluster_delayed_unassigned_shards_total, latest).\nTripped Breakers — count of tripped circuit breakers across the cluster (meter_elasticsearch_cluster_breakers_tripped, latest), a memory-protection signal.\nCluster CPU Avg — average CPU usage across the cluster\u0026rsquo;s nodes in percent (meter_elasticsearch_cluster_cpu_usage_avg).\nJVM Memory Used Avg — average JVM heap memory used across the cluster (meter_elasticsearch_cluster_jvm_memory_used_avg).\nOpen Files Avg — average open file-descriptor count across the cluster (meter_elasticsearch_cluster_open_file_count).\nInstance dashboard (node) For one selected node — the per-node OS, JVM, and storage detail.\nProcess CPU (%) — CPU consumed by the Elasticsearch process (meter_elasticsearch_node_process_cpu_percent).\nOS CPU (%) — host CPU usage on the node (meter_elasticsearch_node_os_cpu_percent).\nLoad Average — the node\u0026rsquo;s 1-minute, 5-minute, and 15-minute OS load averages (meter_elasticsearch_node_os_load1, meter_elasticsearch_node_os_load5, meter_elasticsearch_node_os_load15).\nJVM Memory (MB) — heap used, heap max, and non-heap used in MB (meter_elasticsearch_node_jvm_memory_heap_used, meter_elasticsearch_node_jvm_memory_heap_max, meter_elasticsearch_node_jvm_memory_nonheap_used).\nGC — garbage-collection activity on a dual axis: GC count on the left, GC time in ms/min on the right (meter_elasticsearch_node_jvm_gc_count, meter_elasticsearch_node_jvm_gc_time).\nTranslog — transaction-log operations and translog size in MB on a dual axis (meter_elasticsearch_node_indices_translog_operations, meter_elasticsearch_node_indices_translog_size).\nBreakers — tripped circuit breakers and the estimated breaker size in MB on this node (meter_elasticsearch_node_breakers_tripped, meter_elasticsearch_node_breakers_estimated_size).\nSegments — Lucene segment count and segment memory in MB on a dual axis (meter_elasticsearch_node_segment_count, meter_elasticsearch_node_segment_memory).\nDisk Usage — disk used in GB and disk-used percent on a dual axis (meter_elasticsearch_node_disk_usage, meter_elasticsearch_node_disk_usage_percent).\nNetwork — bytes sent and received on the node (meter_elasticsearch_node_network_send_bytes, meter_elasticsearch_node_network_receive_bytes).\nOpen Files — average open file-descriptor count on the node (meter_elasticsearch_node_open_file_count).\nEndpoint dashboard (index) For one selected index — indexing throughput, search throughput, size, and document counts.\nIndexing Rate — indexing requests vs. processed operations (meter_elasticsearch_index_stats_indexing_index_total_req_rate, meter_elasticsearch_index_stats_indexing_index_total_proc_rate).\nSearch Rate — search-query requests vs. processed operations (meter_elasticsearch_index_stats_search_query_total_req_rate, meter_elasticsearch_index_stats_search_query_total_proc_rate).\nIndex Size (all shards) — total store size of the index across all shards in GB (meter_elasticsearch_index_indices_store_size_bytes_total, latest).\nIndex Size (primary) — store size of the index\u0026rsquo;s primary shards in GB (meter_elasticsearch_index_indices_store_size_bytes_primary, latest).\nDocuments — document counts: all, primary, and deleted (meter_elasticsearch_index_indices_docs_total, meter_elasticsearch_index_indices_docs_primary, meter_elasticsearch_index_indices_deleted_docs_primary).\nAvg Search Time / Req (s) — average per-request time in seconds for each search phase: fetch, query, scroll, and suggest (meter_elasticsearch_index_search_fetch_avg_time, meter_elasticsearch_index_search_query_avg_time, meter_elasticsearch_index_search_scroll_avg_time, meter_elasticsearch_index_search_suggest_avg_time).\nRequirements The ELASTICSEARCH dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Elasticsearch monitoring enabled (see the upstream Elasticsearch monitoring setup), which feeds the three metric families this dashboard reads:\nCluster metrics — the meter_elasticsearch_cluster_* family (health, node count, shard states, pending tasks, tripped breakers, average CPU / JVM memory / open files) for the cluster list and the Service dashboard.\nNode metrics — the meter_elasticsearch_node_* family (process / OS CPU, load averages, JVM memory and GC, translog, breakers, segments, disk, network, open files) for the Instance dashboard.\nIndex metrics — the meter_elasticsearch_index_* family (indexing and search rates, store size, document counts, per-phase search times) for the Endpoint dashboard.\nEach metric is queried at its own OAP scope — cluster metrics at service scope, node metrics at instance scope, index metrics at endpoint scope. OAP does not roll a metric up across scopes, so a node- or index-scope widget stays empty until that level of data is reported.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/elasticsearch/","title":"\u003c!--"},{"body":" Envoy AI Gateway The ENVOY_AI_GATEWAY layer monitors Envoy AI Gateway deployments — the Envoy-based gateway that fronts LLM providers and models, routing chat / completion traffic to OpenAI, Anthropic, and other backends. SkyWalking turns the gateway\u0026rsquo;s OpenTelemetry GenAI signals into request, latency, token, and streaming-quality metrics, broken down by provider and model, and lands them here.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Gateways. Its services are listed as AI Gateways and its instances as Nodes. The ENVOY_AI_GATEWAY layer enables the Service (AI Gateway) and Instance (Node) dashboards plus the Logs sub-tab. It does not ship an Endpoint dashboard, a topology / service-map view, or a Traces tab — the gateway is monitored entirely through its GenAI meter families, which carry no per-endpoint scope or call graph.\nThis page is the operator reference for the bundled ENVOY_AI_GATEWAY dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled ENVOY_AI_GATEWAY template; if an operator has published a customized template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a gateway, the layer landing page lists every AI Gateway with four sortable columns, sorted by traffic (RPM) by default:\nRPM — requests per minute across the gateway (meter_envoy_ai_gw_request_cpm).\nAvg Latency — average request latency in ms (meter_envoy_ai_gw_request_latency_avg).\nInput Tokens/min — input (prompt) token throughput per minute (meter_envoy_ai_gw_input_token_rate).\nOutput Tokens/min — output (completion) token throughput per minute (meter_envoy_ai_gw_output_token_rate).\nService dashboard The primary drill-down for one selected AI Gateway. Beyond the headline request and token widgets, it breaks traffic down by GenAI provider and model and exposes streaming-quality timings (TTFT / TPOT). The Model Context Protocol (MCP) widgets only appear when the gateway actually serves MCP traffic.\nRequests, latency, and tokens\nRequest RPM — requests per minute for the gateway (meter_envoy_ai_gw_request_cpm).\nRequest Latency Avg — average request latency in ms (meter_envoy_ai_gw_request_latency_avg).\nInput Token Rate — input (prompt) tokens per minute (meter_envoy_ai_gw_input_token_rate).\nOutput Token Rate — output (completion) tokens per minute (meter_envoy_ai_gw_output_token_rate).\nRequest Latency Percentile — p50 / p75 / p90 / p95 / p99 request latency, the tail of the latency distribution, in ms (meter_envoy_ai_gw_request_latency_percentile).\nStreaming quality\nTTFT (Time to First Token) — time to first token for streaming responses, shown as both the average and the percentile distribution in ms (meter_envoy_ai_gw_ttft_avg, meter_envoy_ai_gw_ttft_percentile).\nTPOT (Time Per Output Token) — time per output token (inter-token latency) for streaming responses, shown as both the average and the percentile distribution in ms (meter_envoy_ai_gw_tpot_avg, meter_envoy_ai_gw_tpot_percentile).\nBy provider — each widget is split per gen_ai_provider_name, so every upstream LLM provider the gateway routes to gets its own series:\nRPM by Provider — requests per minute per provider (meter_envoy_ai_gw_provider_request_cpm).\nTokens by Provider — token throughput per provider (meter_envoy_ai_gw_provider_token_rate).\nLatency Avg by Provider — average latency per provider, in ms (meter_envoy_ai_gw_provider_latency_avg).\nBy model — each widget is split per gen_ai_response_model, so every model the gateway answered with gets its own series:\nRPM by Model — requests per minute per model (meter_envoy_ai_gw_model_request_cpm).\nTokens by Model — token throughput per model (meter_envoy_ai_gw_model_token_rate).\nLatency Avg by Model — average latency per model, in ms (meter_envoy_ai_gw_model_latency_avg).\nTTFT by Model — average time to first token per model, in ms (meter_envoy_ai_gw_model_ttft_avg).\nTPOT by Model — average time per output token per model, in ms (meter_envoy_ai_gw_model_tpot_avg).\nMCP (Model Context Protocol) — these widgets render only when the gateway serves MCP traffic; on a gateway that never sees MCP requests they stay hidden rather than showing empty:\nMCP RPM — MCP requests per minute (meter_envoy_ai_gw_mcp_request_cpm).\nMCP Avg Latency — average MCP request latency in ms (meter_envoy_ai_gw_mcp_request_latency_avg).\nMCP Error RPM — MCP errors per minute (meter_envoy_ai_gw_mcp_error_cpm).\nMCP by Method — MCP requests per minute split per mcp_method_name (meter_envoy_ai_gw_mcp_method_cpm).\nMCP by Backend — MCP requests per minute split per mcp_backend (meter_envoy_ai_gw_mcp_backend_request_cpm).\nInstance dashboard For one selected Node of the gateway. The same request, latency, token, and streaming-quality timings as the service view, scoped to a single gateway node.\nRequest RPM — requests per minute for this node (meter_envoy_ai_gw_instance_request_cpm).\nRequest Latency Avg — average request latency for this node, in ms (meter_envoy_ai_gw_instance_request_latency_avg).\nInput Token Rate — input (prompt) tokens per minute for this node (meter_envoy_ai_gw_instance_input_token_rate).\nOutput Token Rate — output (completion) tokens per minute for this node (meter_envoy_ai_gw_instance_output_token_rate).\nRequest Latency Percentile — p50 / p75 / p90 / p95 / p99 request latency for this node, in ms (meter_envoy_ai_gw_instance_request_latency_percentile).\nTTFT — time to first token for this node, shown as both the average and the percentile distribution in ms (meter_envoy_ai_gw_instance_ttft_avg, meter_envoy_ai_gw_instance_ttft_percentile).\nTPOT — time per output token for this node, shown as both the average and the percentile distribution in ms (meter_envoy_ai_gw_instance_tpot_avg, meter_envoy_ai_gw_instance_tpot_percentile).\nRequirements The ENVOY_AI_GATEWAY dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the Envoy AI Gateway GenAI meter families, derived from the gateway\u0026rsquo;s OpenTelemetry GenAI signals:\nGateway (service) metrics — the meter_envoy_ai_gw_* family at service scope: request load (meter_envoy_ai_gw_request_cpm), latency average and percentile (meter_envoy_ai_gw_request_latency_avg, meter_envoy_ai_gw_request_latency_percentile), input / output token rates (meter_envoy_ai_gw_input_token_rate, meter_envoy_ai_gw_output_token_rate), and the streaming-quality timings (meter_envoy_ai_gw_ttft_*, meter_envoy_ai_gw_tpot_*).\nPer-provider and per-model metrics — the meter_envoy_ai_gw_provider_* and meter_envoy_ai_gw_model_* families, labelled by gen_ai_provider_name and gen_ai_response_model, for the provider and model breakdown widgets.\nMCP metrics — the meter_envoy_ai_gw_mcp_* family (request, latency, error, per-method, per-backend), reported only when the gateway serves Model Context Protocol traffic; the MCP widgets stay hidden until these arrive.\nNode (instance) metrics — the meter_envoy_ai_gw_instance_* family for the per-node widgets (request load, latency, tokens, percentile, TTFT, TPOT).\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a node-scope metric is empty until that level of data is reported. For how to enable the upstream collector, see the Envoy AI Gateway monitoring setup docs.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/envoy_ai_gateway/","title":"\u003c!--"},{"body":" Flink The FLINK layer monitors an Apache Flink stream-processing cluster: the JobManager that coordinates the cluster, the TaskManagers that run the work, and the Flink jobs themselves. It is sourced from Flink\u0026rsquo;s metric reporter via OpenTelemetry, so the dashboard reads the same JVM, slot, network, and checkpoint metrics Flink already exposes.\nIn Horizon\u0026rsquo;s sidebar this layer is named Flink. Its three scopes are aliased to Flink\u0026rsquo;s own vocabulary: services are listed as Flink JobManagers, instances as TaskManagers, and endpoints as Jobs. The FLINK layer enables the Service, Instance, and Endpoint sub-tabs only — it ships no topology, no traces, and no logs.\nThis page is the operator reference for the bundled FLINK dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled FLINK template; if an operator has published a customized FLINK template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a JobManager, the layer landing page lists every Flink JobManager with four sortable columns, sorted by Running Jobs by default. Each column is the latest reported value:\nRunning Jobs — jobs currently running on this JobManager (meter_flink_jobManager_running_job_number).\nTaskManagers — TaskManagers registered with this JobManager (meter_flink_jobManager_taskManagers_registered_number).\nSlots Available — free task slots across the registered TaskManagers (meter_flink_jobManager_taskManagers_slots_available).\nSlots Total — total task slots across the registered TaskManagers (meter_flink_jobManager_taskManagers_slots_total).\nService dashboard The primary drill-down for one selected JobManager. The top row is four single-value cards, followed by the JobManager\u0026rsquo;s JVM health, GC behavior, and a running-jobs ranking.\nRunning Jobs — jobs currently running (meter_flink_jobManager_running_job_number).\nTaskManagers — registered TaskManagers (meter_flink_jobManager_taskManagers_registered_number).\nSlots Total — total task slots (meter_flink_jobManager_taskManagers_slots_total).\nSlots Available — free task slots (meter_flink_jobManager_taskManagers_slots_available).\nJM JVM CPU Load (%) — JobManager JVM CPU load (meter_flink_jobManager_jvm_cpu_load).\nJM JVM Thread Count — live JVM threads in the JobManager (meter_flink_jobManager_jvm_thread_count).\nJM CPU Time (ms) — JobManager JVM CPU time in ms (meter_flink_jobManager_jvm_cpu_time).\nJM Heap (MB) — JobManager heap memory, used vs available in MB (meter_flink_jobManager_jvm_memory_heap_used, meter_flink_jobManager_jvm_memory_heap_available).\nJM NonHeap (MB) — JobManager non-heap memory, used vs available in MB (meter_flink_jobManager_jvm_memory_nonHeap_used, meter_flink_jobManager_jvm_memory_nonHeap_available).\nJM Metaspace (MB) — JobManager metaspace, used vs available in MB (meter_flink_jobManager_jvm_memory_metaspace_used, meter_flink_jobManager_jvm_memory_metaspace_available).\nG1 Young GC — G1 young-generation collections on a dual axis: count on the left, time in ms on the right (meter_flink_jobManager_jvm_g1_young_generation_count, meter_flink_jobManager_jvm_g1_young_generation_time).\nG1 Old GC — G1 old-generation collections, count and time in ms on a dual axis (meter_flink_jobManager_jvm_g1_old_generation_count, meter_flink_jobManager_jvm_g1_old_generation_time).\nAll GC — all garbage collectors combined, count and time in ms on a dual axis (meter_flink_jobManager_jvm_all_garbageCollector_count, meter_flink_jobManager_jvm_all_garbageCollector_time).\nTop 10 Running Jobs — the ten jobs with the longest running time, ranked descending (meter_flink_job_runningTime).\nInstance dashboard For one selected TaskManager — the JVM health and network/back-pressure detail of a single worker.\nJVM CPU Load (%) — TaskManager JVM CPU load (meter_flink_taskManager_jvm_cpu_load).\nJVM Thread Count — live JVM threads in the TaskManager (meter_flink_taskManager_jvm_thread_count).\nCPU Time (ms) — TaskManager JVM CPU time in ms (meter_flink_taskManager_jvm_cpu_time).\nBack Pressured — whether the TaskManager is currently back-pressured (meter_flink_taskManager_isBackPressured).\nHeap (MB) — TaskManager heap memory, used vs available in MB (meter_flink_taskManager_jvm_memory_heap_used, meter_flink_taskManager_jvm_memory_heap_available).\nNonHeap (MB) — TaskManager non-heap memory, used vs available in MB (meter_flink_taskManager_jvm_memory_nonHeap_used, meter_flink_taskManager_jvm_memory_nonHeap_available).\nMetaspace (MB) — TaskManager metaspace, used vs available in MB (meter_flink_taskManager_jvm_memory_metaspace_used, meter_flink_taskManager_jvm_memory_metaspace_available).\nRecords In / Out — records read in and written out by the TaskManager (meter_flink_taskManager_numRecordsIn, meter_flink_taskManager_numRecordsOut).\nBytes In / Out / s — bytes per second read in and written out (meter_flink_taskManager_numBytesInPerSecond, meter_flink_taskManager_numBytesOutPerSecond).\nNetty Memory (MB) — Netty network-shuffle memory, used vs available in MB (meter_flink_taskManager_netty_usedMemory, meter_flink_taskManager_netty_availableMemory).\nPool Usage (%) — input vs output buffer-pool usage (meter_flink_taskManager_inPoolUsage, meter_flink_taskManager_outPoolUsage).\nBack-Pressure Time (ms/s) — per-second time the TaskManager spent in each state: soft back-pressure, hard back-pressure, idle, and busy (meter_flink_taskManager_softBackPressuredTimeMsPerSecond, meter_flink_taskManager_hardBackPressuredTimeMsPerSecond, meter_flink_taskManager_idleTimeMsPerSecond, meter_flink_taskManager_busyTimeMsPerSecond).\nEndpoint dashboard For one selected Job — its lifecycle timing, checkpoint behavior, and throughput. The top row is four single-value cards.\nJob Running Time (min) — how long the job has been running, in minutes (meter_flink_job_runningTime).\nJob Restarting Time (min) — time the job has spent restarting, in minutes (meter_flink_job_restartingTime).\nJob Cancelling Time (min) — time the job has spent cancelling, in minutes (meter_flink_job_cancellingTime).\nJob Restarts — number of job restarts (meter_flink_job_restart_number).\nCheckpoints — checkpoint counts over the window: total, completed, failed, and in-progress (meter_flink_job_checkpoints_total, meter_flink_job_checkpoints_completed, meter_flink_job_checkpoints_failed, meter_flink_job_checkpoints_inProgress).\nLast Checkpoint — the most recent checkpoint on a dual axis: size in bytes on the left, duration in ms on the right (meter_flink_job_lastCheckpointSize, meter_flink_job_lastCheckpointDuration).\nCurrent Emit Event Time Lag (ms) — lag between event time and emit time, in ms (meter_flink_job_currentEmitEventTimeLag).\nRecords In / Out — records read in and written out by the job (meter_flink_job_numRecordsIn, meter_flink_job_numRecordsOut).\nBytes In / Out / s — bytes per second read in and written out (meter_flink_job_numBytesInPerSecond, meter_flink_job_numBytesOutPerSecond).\nRequirements The FLINK dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the Flink meter families produced from Flink\u0026rsquo;s OpenTelemetry metric export:\nJobManager metrics — the meter_flink_jobManager_* family (running jobs, registered TaskManagers, slot totals, and JVM CPU / thread / memory / GC detail), driving the Service list and Service dashboard.\nTaskManager metrics — the meter_flink_taskManager_* family (JVM detail, record / byte throughput, Netty and buffer-pool usage, and back-pressure timing), driving the Instance dashboard.\nJob metrics — the meter_flink_job_* family (running / restarting / cancelling time, restarts, checkpoints, emit-time lag, and throughput), driving the Endpoint dashboard and the Top 10 Running Jobs ranking.\nEach metric is queried at its own OAP scope, and OAP does not roll a metric up across scopes — a JobManager-, TaskManager-, or Job-scope metric is empty until that level of data is reported. To set up the Flink metric reporter and the OAP receiver, follow the Flink monitoring setup guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/flink/","title":"\u003c!--"},{"body":" General Service The GENERAL layer is where SkyWalking\u0026rsquo;s language agents report. Any service instrumented by a SkyWalking native agent — Java, .NET (CLR), Go, Python, Ruby, Node.js, PHP, and the Spring Boot / Spring Sleuth meter integrations — lands here, so it is the most-used layer and the reference dashboard every other layer\u0026rsquo;s dashboard is modelled on.\nIn Horizon\u0026rsquo;s sidebar this layer is named General Service. Its services are listed as Services, instances as Instances (each badged with the agent language), and endpoints as API — the endpoint-to-endpoint view is called API dependency. The GENERAL layer enables the full set of sub-tabs: Service, Instance, Endpoint, API dependency, Topology, Traces, Logs, and the profiling tabs (trace, eBPF, async, pprof).\nThis page is the operator reference for the bundled GENERAL dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled GENERAL template; if an operator has published a customized GENERAL template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a service, the layer landing page lists every GENERAL service with three sortable columns, sorted by traffic (RPM) by default:\nRPM — calls per minute (service_cpm). Apdex — user-satisfaction score on a 0 – 1 scale (service_apdex/10000). Error Rate — percent of failed calls (100 - service_sla/100). Service dashboard The primary drill-down for one selected service.\nTop 20 APIs — the busiest endpoints under this service, with tabs to re-rank by Traffic (endpoint_cpm, rpm), Slow (endpoint_resp_time, ms), and Successful Rate (endpoint_sla, %, worst-first). Click a row to jump into that endpoint. Traffic — calls per minute for the service (service_cpm). Error Rate — percent of failed calls (100 - service_sla/100). Apdex — user-satisfaction score on a 0 – 1 scale (service_apdex/10000). Response Time Percentile — p50 / p75 / p90 / p95 / p99 latency, the tail of the response-time distribution (service_percentile). Avg Response Time — mean latency in ms (service_resp_time). MQ Consume rate + latency — message-queue consume count and latency on a dual axis: count on the left, latency on the right (service_mq_consume_count, service_mq_consume_latency). Top 10 instances by load — this service\u0026rsquo;s instances ranked by traffic (service_instance_cpm, rpm). Top 10 slowest instances — instances ranked by average response time (service_instance_resp_time, ms). Top 10 instances by success rate — instances ranked worst-first by success rate (service_instance_sla, %). Slow Database Statements — the 20 slowest sampled database statements captured against this service (top_n_service_database_statement, ms). Each row carries the statement text and, when the sample has one, a jump-to-trace link. Shows no data when OAP captured no statements in the window. The latency and error widgets ship with the metric-to-trace drill enabled: click a data point on Avg Response Time or Response Time Percentile to open the slowest traces at that moment, or on Error Rate or Apdex to open the error traces. The same drill rides the instance latency / success-rate widgets and the endpoint latency, percentile, success-rate, and MQ-latency widgets below. See Dashboard Widgets → Metric-to-trace drill.\nInstance dashboard For one selected service instance. The first three widgets always render; the rest are runtime-specific and appear only when the instance actually reports those metrics — so a Java instance shows the JVM family, a Go instance the Golang family, and so on, without manual configuration.\nAlways shown\nService Instance Load — calls per minute against the instance (service_instance_cpm). Service Instance Latency — average response time in ms (service_instance_resp_time). Service Instance Success Rate — percent of successful calls (service_instance_sla/100). JVM (Java instances)\nJVM CPU — JVM CPU as reported by the agent (instance_jvm_cpu). JVM Memory — heap and non-heap used / max in MB (instance_jvm_memory_heap, instance_jvm_memory_heap_max, instance_jvm_memory_noheap, instance_jvm_memory_noheap_max). JVM Memory Detail — per-pool used memory in MB: code cache, newgen, oldgen, survivor, permgen, metaspace, plus the newer JVM pools (zheap, compressed class space, and the segmented codeheaps). Pools a JVM doesn\u0026rsquo;t expose stay at 0. JVM Thread Count — live / daemon / peak threads. JVM Thread State Count — threads by state: runnable / blocked / waiting / timed-waiting. JVM GC Time — young / old / normal GC time in ms. JVM GC Count — young / old / normal GC counts. JVM Class Count — loaded / total-loaded / total-unloaded classes. CLR (.NET instances)\nCLR CPU — process CPU percentage (instance_clr_cpu). CLR Thread — worker-available, completion-port-available, and completion-port-max threads. CLR Heap Memory — managed heap in MB. CLR GC — gen 0 / gen 1 / gen 2 collection counts. Spring (Spring Boot Actuator / Spring Sleuth meters)\nSpring HTTP Request Count and Spring HTTP Request Duration — http.server.requests count and latency. Spring Instance CPU Usage / Spring OS CPU Usage / Spring OS System Load — process CPU, OS CPU, and 1-minute load average. Spring OS Process Files — open vs max file descriptors. Spring JVM GC Pause Duration, Spring JVM Memory (used / max), Spring JVM Threads (live / daemon / peak), Spring JVM Classes (loaded / unloaded). Spring Database Connection Pool (HikariCP / datasource), Spring Thread Pool, Spring JDBC Connections (active / idle / max), Spring Tomcat Sessions (active / max / rejected). Golang (Go instances)\nGolang Goroutines / OS Threads, Golang GC Pause Time, Golang GC Count, Golang Heap Alloc, Golang Goroutine Schedule Time, Golang GC Free, Golang Alloc Size, Golang Free Size, Golang Heap Objects, Golang Heap, Golang Metadata Mspan, Golang Metadata Mcache, Golang GC Goal Size, and Golang CGO Calls — the Go runtime\u0026rsquo;s goroutine, scheduler, GC, and heap detail. Python (PVM instances)\nPython CPU Utilization and Python Memory Utilization — host vs process. Python Thread Count, Python GC Count (gen 0 / 1 / 2), and Python GC Time. Ruby instances\nRuby CPU Usage, Ruby Memory (RSS), Ruby Memory Usage, Ruby Thread Status (active / running), Ruby GC Count (total / minor / major), Ruby GC Time, Ruby Heap Usage, and Ruby Heap Slots (live / available). Node.js instances\nProcess CPU — process CPU percentage (meter_instance_nodejs_process_cpu). V8 Heap Used / V8 Heap Total / V8 Heap Limit — the V8 heap in MB: currently used, currently allocated, and the maximum the heap may grow to (meter_instance_nodejs_heap_used / _heap_total / _heap_limit). Process RSS — resident set size in MB (meter_instance_nodejs_rss). External Memory — memory held outside the V8 heap (buffers and native objects) in MB (meter_instance_nodejs_external_memory). Array Buffers — ArrayBuffer / SharedArrayBuffer memory in MB (meter_instance_nodejs_array_buffers). Process Uptime — days since the process started (meter_instance_nodejs_uptime/86400). Peak Malloced Memory / Malloced Memory — peak and current V8 malloced memory in MB (meter_instance_nodejs_peak_malloced_memory / _malloced_memory). Old Space Used / New Space Used — V8 old / new generation heap used in MB (meter_instance_nodejs_old_space_used / _new_space_used). PHP (PHM) instances\nPHP CPU Utilization — process CPU percentage (meter_instance_php_process_cpu_utilization). PHP Memory Used and PHP Memory Peak — current and peak process memory in MB (meter_instance_php_memory_used_mb, meter_instance_php_memory_peak_mb). PHP Virtual Memory — virtual memory size in MB (meter_instance_php_virtual_memory_mb). PHP Thread Count — live threads (meter_instance_php_thread_count). PHP Open FDs — open file descriptors (meter_instance_php_open_fd_count). Endpoint dashboard For one selected endpoint (an API).\nTraffic — calls per minute for the endpoint (endpoint_cpm). Response Time — average latency in ms (endpoint_resp_time). Success Rate — percent of successful calls (endpoint_sla/100). Response Time Percentile — p50 / p75 / p90 / p95 / p99 latency for the endpoint (endpoint_percentile). MQ Avg Consuming Latency — consume latency in ms, shown only for endpoints that serve message-queue traffic (endpoint_mq_consume_latency). Topology and maps The GENERAL layer ships a full set of maps.\nTopology (service map) — every service node is decorated with RPM (service_cpm), an SLA health ring (service_sla/100, colored green / amber / red — higher is better, so the ring turns red as success rate drops), and Latency (service_resp_time). Each call edge carries server-side and client-side RPM, Avg response time, p95, and SLA (service_relation_server_* and service_relation_client_*), shown aligned in the edge panel.\nInstance map — from a call between two services on the service map, drill into the instance-to-instance calls between them. The same node / edge metric set is evaluated at instance scope (service_instance_* and service_instance_relation_server/client_*).\nAPI dependency (endpoint map) — the endpoint-to-endpoint dependency view. Each endpoint node shows RPM (endpoint_cpm), an SLA ring (endpoint_sla/100), and Latency (endpoint_resp_time); each edge shows RPM, Avg response time, p95, and SLA (endpoint_relation_*). Endpoint relations are server-side only.\nFor how these maps are read and navigated, see the 3D Infrastructure Map for the cross-layer view, and the topology section of Layer Dashboard Templates for how the node and edge metrics are configured.\nRequirements The GENERAL dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nService / instance / endpoint metrics — the service_*, service_instance_*, and endpoint_* families (traffic, response time, SLA, apdex, percentiles), produced by OAP from agent-reported traces or meters. Relation metrics — service_relation_*, service_instance_relation_*, and endpoint_relation_* for the service map, instance map, and API-dependency views. Runtime metrics, for the runtime-specific instance widgets to appear: JVM (instance_jvm_*), CLR (instance_clr_*), the Spring meter family (meter_*), and the Golang / Python / Ruby / Node.js / PHP agent meter families. An instance only shows the families its agent emits. Sampled records — top_n_service_database_statement for the Slow Database Statements list, captured by OAP when slow-statement sampling is enabled. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so an instance- or endpoint-scope metric is empty until that level of data is reported. When a whole family is missing (for example a non-JVM runtime), its widgets are hidden rather than shown empty.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/general/","title":"\u003c!--"},{"body":" iOS The IOS layer is where SkyWalking\u0026rsquo;s iOS client-side monitoring reports. An iOS app instrumented with the SkyWalking iOS SDK surfaces Apple MetricKit diagnostics — app launch time, hang time, abnormal exits, OOM kills, peak memory, scroll responsiveness, and network transfer — alongside the latency and success rate of the HTTP calls the app makes out to your backends. It sits in the Mobile group of layers.\nIn Horizon\u0026rsquo;s sidebar this layer is named iOS. Its services are listed as Apps, instances as App Sessions, and endpoints as Outbound APIs — these are the names you see on the picker and column headers. The IOS layer enables the Service, Instance, and Endpoint sub-tabs plus Logs. It has no Topology, Traces, or endpoint-dependency view — iOS reports client-side device telemetry and outbound calls, not a server-side call graph.\nThis page is the operator reference for the bundled IOS dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled IOS template; if an operator has published a customized IOS template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nApp list Before opening an app, the layer landing page lists every IOS app with four sortable columns, sorted by Launch (P95) by default:\nLaunch (P95) — 95th-percentile app launch time in ms (meter_ios_app_launch_time_percentile{p='95'}), the tail of how long the app takes to become usable.\nHang Time — total time the main thread spent hung in the window, in ms (meter_ios_hang_time_sum).\nCrashes — abnormal exits, foreground and background summed (meter_ios_foreground_abnormal_exit_count + meter_ios_background_abnormal_exit_count).\nOutbound RPM — calls per minute the app makes to backends (service_cpm).\nApp (service) dashboard The primary drill-down for one selected app.\nApp Launch Time Percentile — p50 / p75 / p90 / p95 / p99 launch time in ms, the distribution of how long the app takes to start (meter_ios_app_launch_time_percentile).\nHang Time Percentile — p50 / p75 / p90 / p95 / p99 main-thread hang time in ms (meter_ios_hang_time_percentile).\nHang Time (sum) — total hang time over the window, in ms (meter_ios_hang_time_sum).\nAbnormal Exits (Crashes) — abnormal exits split into foreground and background series; MetricKit reports the two separately (meter_ios_foreground_abnormal_exit_count, meter_ios_background_abnormal_exit_count).\nOOM Kill Count — background out-of-memory kills, the iOS system reaping the app under memory pressure (meter_ios_background_oom_kill_count).\nPeak Memory — peak resident memory in bytes (meter_ios_peak_memory).\nScroll Hitch Ratio — the fraction of scroll frames classified as hitched; higher means a laggier scrolling UI (meter_ios_scroll_hitch_ratio).\nNetwork Transfer — bytes transferred over wifi vs cellular, download and upload, as four series (meter_ios_wifi_download, meter_ios_wifi_upload, meter_ios_cellular_download, meter_ios_cellular_upload).\nOutbound HTTP — the calls this app makes to backends, on a dual axis: RPM and Avg latency (ms) on the left axis, Success Rate (%) on the right (service_cpm, service_resp_time, service_sla/100).\nApp Session (instance) dashboard For one selected app session — the same MetricKit and outbound-HTTP families evaluated at session scope.\nLaunch Time Percentile — p50 / p75 / p90 / p95 / p99 launch time in ms for the session (meter_ios_instance_app_launch_time_percentile).\nHang Time Percentile — p50 / p75 / p90 / p95 / p99 hang time in ms for the session (meter_ios_instance_hang_time_percentile).\nAbnormal Exits — foreground vs background abnormal exits for the session (meter_ios_instance_foreground_abnormal_exit_count, meter_ios_instance_background_abnormal_exit_count).\nOOM Kill Count — background OOM kills for the session (meter_ios_instance_background_oom_kill_count).\nPeak Memory — peak resident memory in bytes for the session (meter_ios_instance_peak_memory).\nOutbound HTTP — the session\u0026rsquo;s outbound calls on a dual axis: RPM and Avg latency (ms) on the left axis, Success Rate (%) on the right (service_instance_cpm, service_instance_resp_time, service_instance_sla/100).\nOutbound API (endpoint) dashboard For one selected Outbound API — a backend endpoint the app calls.\nOutbound Load — calls per minute to the endpoint (endpoint_cpm).\nOutbound Avg Latency — average call latency in ms (endpoint_resp_time).\nOutbound Success Rate — percent of successful calls (endpoint_sla/100).\nOutbound Latency Percentile — p50 / p75 / p90 / p95 / p99 latency for the endpoint (endpoint_percentile).\nRequirements The IOS dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\niOS MetricKit metrics — the meter_ios_* family at service scope and the meter_ios_instance_* family at session scope: launch-time and hang-time percentiles, hang-time sum, foreground / background abnormal exits, background OOM kills, peak memory, scroll hitch ratio, and wifi / cellular network transfer. These are produced by OAP from the SkyWalking iOS SDK\u0026rsquo;s MetricKit reports.\nOutbound HTTP metrics — the service_*, service_instance_*, and endpoint_* families (traffic, response time, SLA, percentiles), produced by OAP from the calls the app makes to instrumented backends.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a session- or endpoint-scope metric is empty until that level of data is reported. When a family is missing, its widgets read no data rather than being shown with fabricated values.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/ios/","title":"\u003c!--"},{"body":" Kubernetes The K8S layer monitors Kubernetes clusters and the nodes inside them. SkyWalking builds this layer from cluster-state and node-resource telemetry collected through OpenTelemetry (kube-state-metrics and the node / cAdvisor metric pipelines scraped into OAP) and reshapes it into cluster-wide and per-node metrics. In the sidebar it groups under Kubernetes.\nIn Horizon\u0026rsquo;s sidebar this layer\u0026rsquo;s entities are renamed to fit the Kubernetes model: services are listed as Clusters and instances as Nodes. The K8S layer enables the Service (Cluster) and Instance (Node) scopes only — there is no endpoint scope, no topology, and no traces or logs tab, because this layer reports cluster-state and node-resource metrics rather than request traffic.\nThis page is the operator reference for the bundled K8S dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled K8S template; if an operator has published a customized K8S template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nCluster list Before opening a cluster, the layer landing page lists every Kubernetes cluster with four sortable columns, sorted by Pods by default. Each column shows the latest reading summed across the cluster:\nPods — total pods in the cluster (k8s_cluster_pod_total).\nNodes — total nodes in the cluster (k8s_cluster_node_total).\nNamespaces — total namespaces in the cluster (k8s_cluster_namespace_total).\nDeployments — total deployments in the cluster (k8s_cluster_deployment_total).\nCluster dashboard The primary drill-down for one selected cluster (a Service in OAP terms). It opens with a row of count cards summarizing the cluster\u0026rsquo;s object inventory, then resource trends, then status tables that break the cluster down per node, deployment, service, and pod.\nInventory cards — each is a single latest count:\nNode Total — nodes in the cluster (latest(k8s_cluster_node_total)).\nNamespace Total — namespaces in the cluster (latest(k8s_cluster_namespace_total)).\nDeployment Total — deployments in the cluster (latest(k8s_cluster_deployment_total)).\nStatefulSet Total — statefulsets in the cluster (latest(k8s_cluster_statefulset_total)).\nDaemonSet Total — daemonsets in the cluster (latest(k8s_cluster_daemonset_total)).\nService Total — Kubernetes services in the cluster (latest(k8s_cluster_service_total)).\nPod Total — pods in the cluster (latest(k8s_cluster_pod_total)).\nContainer Total — containers in the cluster (latest(k8s_cluster_container_total)).\nResource trends — cluster-wide capacity vs. demand over time:\nCPU Resources — cluster CPU capacity against requests, limits, and allocatable, in millicores (k8s_cluster_cpu_cores, k8s_cluster_cpu_cores_requests, k8s_cluster_cpu_cores_limits, k8s_cluster_cpu_cores_allocatable).\nMemory Resources — cluster memory requests, allocatable, limits, and total, in GiB (k8s_cluster_memory_requests, k8s_cluster_memory_allocatable, k8s_cluster_memory_limits, k8s_cluster_memory_total).\nStorage Resources — cluster ephemeral-storage total against allocatable, in GiB (k8s_cluster_storage_total, k8s_cluster_storage_allocatable).\nStatus tables — each lists the entities currently matching the condition; they read no data when nothing matches:\nNode Status — per-node Kubernetes conditions currently true or unknown — Ready, the various Pressure conditions, and so on (latest(k8s_cluster_node_status)).\nDeployment Status — deployments reporting the Available condition (latest(k8s_cluster_deployment_status)).\nDeployment Spec Replicas — desired replica count per deployment (latest(k8s_cluster_deployment_spec_replicas)).\nService Status — pods backing each Kubernetes service, grouped by pod phase — Running / Pending / Failed and so on (latest(k8s_cluster_service_pod_status)).\nPod Status Not Running — pods in any non-Running phase (latest(k8s_cluster_pod_status_not_running)).\nPod Status Waiting — containers in a waiting state, grouped by the waiting reason (latest(k8s_cluster_pod_status_waiting)).\nNode dashboard For one selected node (an Instance in OAP terms) — its scheduling state and CPU / memory / network / storage resources.\nNode Status — the node\u0026rsquo;s current status as a single latest reading (latest(k8s_node_node_status)).\nPods on Node — pods scheduled on the node over time (k8s_node_pod_total).\nPod Total — the current count of pods scheduled on the node, as a single latest reading (latest(k8s_node_pod_total)).\nNode CPU Usage — node CPU usage in millicores (k8s_node_cpu_usage).\nNode CPU Resources — node CPU total against allocatable, requests, and limits, in millicores (k8s_node_cpu_cores, k8s_node_cpu_cores_allocatable, k8s_node_cpu_cores_requests, k8s_node_cpu_cores_limits).\nNode Memory Usage — node memory usage in GiB (k8s_node_memory_usage).\nNode Memory Resources — node memory total against allocatable, requests, and limits, in GiB (k8s_node_memory_total, k8s_node_memory_allocatable, k8s_node_memory_requests, k8s_node_memory_limits).\nNode Network I/O — node receive and transmit throughput in KB/s (k8s_node_network_receive, k8s_node_network_transmit).\nNode Storage Resources — node storage total against allocatable, in GiB (k8s_node_storage_total, k8s_node_storage_allocatable).\nRequirements The K8S dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the Kubernetes monitoring metric families, fed in through the OpenTelemetry receiver from kube-state-metrics and the node / cAdvisor pipelines:\nCluster metrics — the k8s_cluster_* family at cluster scope: object-inventory totals (node / namespace / deployment / statefulset / daemonset / service / pod / container), CPU / memory / storage capacity-and-demand series, and the per-node, per-deployment, per-service, and per-pod status breakdowns.\nNode metrics — the k8s_node_* family at node scope: node status, pod count, and CPU / memory / network / storage usage and resource series.\nEach metric is queried at its own OAP scope (Cluster / Node); OAP does not roll a metric up across scopes, so a node-scope metric stays empty until that level of data is reported. For how to stand up the Kubernetes-to-OAP pipeline, see the layer\u0026rsquo;s setup documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/k8s/","title":"\u003c!--"},{"body":" Kubernetes Services The K8S_SERVICE layer monitors the network behavior of Kubernetes services, observed at the kernel level by SkyWalking Rover\u0026rsquo;s eBPF probes. It captures the HTTP and TCP traffic flowing in and out of each service\u0026rsquo;s pods — call rate, latency, status codes, header / body sizes, packet counts, and connection activity — without instrumenting the application. It belongs to the Kubernetes layer group.\nIn Horizon\u0026rsquo;s sidebar this layer is named Kubernetes Services. Its services are listed as K8s services, instances as Pods, and endpoints as Endpoints — service names are grouped and displayed by their Kubernetes namespace. The K8S_SERVICE layer enables the Service, Pod, Endpoint, Topology, eBPF Profiling, Network Profiling, and Pod Logs sub-tabs. It does not enable an endpoint-dependency map, Traces, or Logs.\nThis page is the operator reference for the bundled K8S_SERVICE dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled K8S_SERVICE template; if an operator has published a customized K8S_SERVICE template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a service, the layer landing page lists every K8S_SERVICE service with four sortable columns, sorted by HTTP traffic (HTTP RPM) by default:\nPods — number of pods backing the service, summed from the latest reading (latest(k8s_service_pod_total)).\nHTTP RPM — HTTP calls per minute, summed across the service (kubernetes_service_http_call_cpm).\nLatency — average HTTP response time in ms (kubernetes_service_http_call_time).\nSuccess Rate — percent of successful HTTP calls (kubernetes_service_http_call_successful_rate/100).\nService dashboard The primary drill-down for one selected service. It mixes pod-lifecycle and resource widgets with the HTTP and TCP traffic the service\u0026rsquo;s pods carry.\nPods and resources\nService Pods — pod count over time (k8s_service_pod_total).\nPods Waiting — a table of containers currently in the Waiting state, keyed by container · pod · reason (latest(k8s_service_pod_status_waiting)).\nPod Restarts — a table of pods by cumulative restart count (latest(k8s_service_pod_status_restarts_total)).\nCPU Resources — requested vs. limited CPU in millicores, as two series (k8s_service_cpu_cores_requests, k8s_service_cpu_cores_limits).\nMemory Resources — requested vs. limited memory in MiB (k8s_service_memory_requests, k8s_service_memory_limits).\nPod CPU Usage — actual CPU consumed by the pods in millicores (k8s_service_pod_cpu_usage).\nPod Memory Usage — actual memory consumed by the pods in MiB (k8s_service_pod_memory_usage).\nHTTP traffic\nHTTP Request RPM — HTTP calls per minute for the service (kubernetes_service_http_call_cpm).\nHTTP Response Time — average HTTP response time in ms (kubernetes_service_http_call_time).\nHTTP Status Code RPM — calls per minute broken out by status class: 1xx / 2xx / 3xx / 4xx / 5xx (kubernetes_service_http_status_1xx_cpm … kubernetes_service_http_status_5xx_cpm).\nHTTP Request / Response Size — average request and response header and body sizes in KB, as four series (kubernetes_service_http_avg_req_header_size, kubernetes_service_http_avg_req_body_size, kubernetes_service_http_avg_resp_header_size, kubernetes_service_http_avg_resp_body_size).\nTCP traffic\nTCP Connect — client-side connect attempts and successes per minute, as two series (kubernetes_service_connect_cpm, kubernetes_service_connect_success_cpm).\nTCP Connect Duration — average connect time in ns (kubernetes_service_connect_time).\nTCP Accept — server-side accept events per minute (kubernetes_service_accept_cpm).\nTCP Packets — read, write, and write-retransmit packet counts per minute, as three series (kubernetes_service_read_package_cpm, kubernetes_service_write_package_cpm, kubernetes_service_write_retrains_package_cpm).\nTCP Bytes — read vs. write payload bytes per minute in KB (kubernetes_service_read_package_size, kubernetes_service_write_package_size).\nPod dashboard For one selected pod (a service instance). The widgets mirror the service view at instance scope, covering the pod\u0026rsquo;s HTTP and TCP traffic.\nPod HTTP RPM — HTTP calls per minute for the pod (kubernetes_service_instance_http_call_cpm).\nPod HTTP Response Time — average HTTP response time in ms (kubernetes_service_instance_http_call_time).\nPod HTTP Status Code RPM — calls per minute by status class: 1xx / 2xx / 3xx / 4xx / 5xx (kubernetes_service_instance_http_status_1xx_cpm … kubernetes_service_instance_http_status_5xx_cpm).\nPod HTTP Sizes — average request and response header and body sizes in KB, as four series (kubernetes_service_instance_http_avg_req_header_size, kubernetes_service_instance_http_avg_req_body_size, kubernetes_service_instance_http_avg_resp_header_size, kubernetes_service_instance_http_avg_resp_body_size).\nPod TCP Connect — client-side connect attempts and successes per minute (kubernetes_service_instance_connect_cpm, kubernetes_service_instance_connect_success_cpm).\nPod TCP Packets — read, write, and write-retransmit packet counts per minute (kubernetes_service_instance_read_package_cpm, kubernetes_service_instance_write_package_cpm, kubernetes_service_instance_write_retrains_package_cpm).\nPod TCP Bytes — read vs. write payload bytes per minute in KB (kubernetes_service_instance_read_package_size, kubernetes_service_instance_write_package_size).\nEndpoint dashboard For one selected endpoint. K8S_SERVICE endpoints carry HTTP traffic, so this scope is HTTP-focused.\nEndpoint HTTP RPM — HTTP calls per minute for the endpoint (kubernetes_service_endpoint_http_call_cpm).\nEndpoint HTTP Response Time — average HTTP response time in ms (kubernetes_service_endpoint_http_call_time).\nEndpoint Status Code RPM — calls per minute by status class: 1xx / 2xx / 3xx / 4xx / 5xx (kubernetes_service_endpoint_http_status_1xx_cpm … kubernetes_service_endpoint_http_status_5xx_cpm).\nEndpoint Request Sizes — average request header and body sizes in KB (kubernetes_service_endpoint_http_avg_req_header_size, kubernetes_service_endpoint_http_avg_req_body_size).\nEndpoint Response Sizes — average response header and body sizes in KB (kubernetes_service_endpoint_http_avg_resp_header_size, kubernetes_service_endpoint_http_avg_resp_body_size).\nTopology and maps The K8S_SERVICE layer ships a service topology with an instance-level drill-down.\nTopology (service map) — each service node is decorated with RPM (kubernetes_service_http_call_cpm), a Success Rate health ring (kubernetes_service_http_call_successful_rate/100, colored green / amber / red — higher is better, so the ring turns red as the success rate drops), and Latency (kubernetes_service_http_call_time). Each call edge carries server-side and client-side RPM (kubernetes_service_relation_server_http_call_cpm, kubernetes_service_relation_client_http_call_cpm) and Avg response time (kubernetes_service_relation_server_http_call_time, kubernetes_service_relation_client_http_call_time).\nInstance map — from a call between two services on the service map, drill into the pod-to-pod calls between them. Each instance node shows RPM (kubernetes_service_instance_http_call_cpm), a Success Rate ring (kubernetes_service_instance_http_call_successful_rate/100), and Latency (kubernetes_service_instance_http_call_time); each edge carries server-side and client-side RPM (kubernetes_service_instance_relation_server_http_call_cpm, kubernetes_service_instance_relation_client_http_call_cpm) and Avg response time (kubernetes_service_instance_relation_server_http_call_time, kubernetes_service_instance_relation_client_http_call_time).\nFor how these maps are read and navigated, see the 3D Infrastructure Map for the cross-layer view, and the topology section of Layer Dashboard Templates for how the node and edge metrics are configured.\neBPF Profiling, Network Profiling, and Pod Logs Because K8S_SERVICE data comes from eBPF probes, this layer also enables three investigation tabs alongside the dashboards:\neBPF Profiling — on-CPU / off-CPU profiling tasks targeted at a selected service, with the flame-graph and span-attached results SkyWalking Rover reports.\nNetwork Profiling — the process-to-process network conversations within the service, rendered as a process-level topology, captured by SkyWalking Rover on a selected pod.\nPod Logs — the container logs collected from the service\u0026rsquo;s pods, with the same filtering and search the logs surface provides elsewhere.\nThese tabs query their own data on demand and are independent of the metric widgets above.\nRequirements The K8S_SERVICE dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Kubernetes network monitoring enabled, with SkyWalking Rover\u0026rsquo;s eBPF probes feeding traffic telemetry and a Kubernetes metrics source feeding pod / resource state. Specifically:\nPod and resource metrics — the k8s_service_* family (pod totals, waiting / restart status, CPU and memory requests / limits, and actual pod CPU / memory usage) for the service-scope lifecycle and resource widgets.\nHTTP metrics — the kubernetes_service_http_*, kubernetes_service_instance_http_*, and kubernetes_service_endpoint_http_* families covering call counts, response time, success rate, status classes, and header / body sizes, at their respective service / instance / endpoint scopes.\nTCP metrics — the kubernetes_service_* and kubernetes_service_instance_* connect, accept, packet, and byte families for the L4 widgets.\nRelation metrics — kubernetes_service_relation_server/client_http_* and kubernetes_service_instance_relation_server/client_http_* for the service map and instance map edges.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so an instance- or endpoint-scope metric is empty until that level of data is reported. For setup, see the Kubernetes network monitoring guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/k8s_service/","title":"\u003c!--"},{"body":" Kafka The KAFKA layer monitors Apache Kafka clusters. SkyWalking reads Kafka\u0026rsquo;s JMX metrics (via OpenTelemetry\u0026rsquo;s Kafka receiver or an equivalent collector) and turns them into per-cluster and per-broker meters, so this dashboard is a JMX-derived view of cluster health, partition / replication state, and broker throughput rather than agent-traced request data.\nIn Horizon\u0026rsquo;s sidebar this layer is named Kafka, grouped under MQ. Its services are listed as Kafka clusters and its instances as Brokers. The KAFKA layer enables only the Service (cluster) and Instance (broker) sub-tabs — it ships no endpoint scope, no topology, and no traces or logs, because Kafka\u0026rsquo;s JMX feed is metrics-only.\nThis page is the operator reference for the bundled KAFKA dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled KAFKA template; if an operator has published a customized KAFKA template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nCluster list Before opening a cluster, the layer landing page lists every Kafka cluster with four sortable columns, sorted by Partitions by default:\nPartitions — average partition count across the cluster (meter_kafka_partition_count).\nOffline Partitions — total partitions with no active leader, summed across the cluster (meter_kafka_offline_partitions_count). A non-zero value means data on those partitions is currently unavailable.\nMax Lag — the worst replica lag observed (meter_kafka_max_lag), the maximum of how far any follower trails its leader.\nLeaders — total partition leaders hosted across the cluster (meter_kafka_leader_count).\nCluster dashboard The primary drill-down for one selected Kafka cluster — the cluster-wide controller and partition health view.\nPartition Count — total partitions in the cluster (meter_kafka_partition_count).\nLeader Count — partition leaders in the cluster (meter_kafka_leader_count).\nActive Controllers — the count of active controllers (meter_kafka_active_controller_count). A healthy cluster has exactly one; zero or more than one signals a controller problem.\nMax Lag — the worst replica lag across the cluster (meter_kafka_max_lag).\nUnder-Replicated Partitions — partitions that have fewer in-sync replicas than configured (meter_kafka_under_replicated_partitions). Sustained non-zero values indicate replication is falling behind.\nOffline Partitions — partitions with no active leader (meter_kafka_offline_partitions_count).\nLeader Election Rate — partition-leader elections per second, split into two series: normal elections (meter_kafka_leader_election_rate) and unclean elections (meter_kafka_unclean_leader_elections_per_second). Unclean elections promote an out-of-sync replica and can lose data, so they should stay at zero.\nBroker dashboard For one selected broker. The widgets cover the broker\u0026rsquo;s CPU and memory, message and byte throughput, request handling, queue timings, replication, and partition/ISR state.\nCPU Usage — broker CPU percentage (meter_kafka_broker_cpu_time_total).\nIncoming Messages / s — messages produced into the broker per second (meter_kafka_broker_messages_per_second).\nBytes In / s — inbound throughput in bytes per second (meter_kafka_broker_bytes_in_per_second).\nBytes Out / s — outbound throughput in bytes per second (meter_kafka_broker_bytes_out_per_second).\nRequests / s — total requests handled per second (meter_kafka_broker_requests_per_second).\nPurgatory Size — requests parked in the broker\u0026rsquo;s request purgatory awaiting completion (meter_kafka_broker_purgatory_size).\nISR Shrinks/s — the latest rate at which in-sync-replica sets are shrinking (latest(meter_kafka_broker_isr_shrinks_per_second)), shown as a single number. Frequent shrinks mean replicas are repeatedly dropping out of sync.\nISR Expands/s — the latest rate at which in-sync-replica sets are re-expanding (latest(meter_kafka_broker_isr_expands_per_second)), shown as a single number.\nMemory Usage (%) — broker memory utilization (meter_kafka_broker_memory_usage_percentage).\nUnder-Replicated Partitions — the latest count of under-replicated partitions on this broker (latest(meter_kafka_broker_under_replicated_partitions)), shown as a single number.\nUnder Min-ISR Partitions — the latest count of partitions below their minimum in-sync-replica threshold on this broker (latest(meter_kafka_broker_under_min_isr_partition_count)), shown as a single number. These partitions reject produces under the default acks setting.\nPartitions + Leaders — partitions hosted on the broker (meter_kafka_broker_partition_count) overlaid with the partitions it currently leads (meter_kafka_broker_leader_count).\nQueue / Send Times — broker request latency breakdown in ms across four stages: request q (meter_kafka_broker_request_queue_time_ms), response q (meter_kafka_broker_response_queue_time_ms), response send (meter_kafka_broker_response_send_time_ms), and remote (meter_kafka_broker_remote_time_ms).\nTopic Rates — per-broker topic activity: produce req/s (meter_kafka_broker_topic_produce_requests_per_second), fetch req/s (meter_kafka_broker_topic_fetch_requests_per_second), and bytes-in/s (meter_kafka_broker_topic_bytesin_per_second).\nReplication — replication traffic in bytes per second between brokers, bytes in (meter_kafka_broker_replication_bytes_in_per_second) and bytes out (meter_kafka_broker_replication_bytes_out_per_second).\nGC Count — garbage-collection count for the broker JVM (meter_kafka_broker_garbage_collector_count).\nMax Lag (broker) — the broker\u0026rsquo;s total replica lag (sum(meter_kafka_broker_max_lag)), shown as a single number — the sum of how far this broker\u0026rsquo;s followers trail their leaders.\nRequirements The KAFKA dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Kafka\u0026rsquo;s JMX metrics ingested and aggregated into the two meter families this layer renders:\nCluster meters — the meter_kafka_* family at Service scope (partition, leader, controller, lag, under-replicated / offline partition, and leader-election metrics) for the cluster list and cluster dashboard.\nBroker meters — the meter_kafka_broker_* family at ServiceInstance scope (CPU, memory, message / byte / request throughput, purgatory, ISR, queue and send times, topic rates, replication, GC, and per-broker partition / leader / lag metrics) for the broker dashboard.\nThese meters are produced by SkyWalking\u0026rsquo;s Kafka monitoring, which reads Kafka\u0026rsquo;s JMX through OpenTelemetry\u0026rsquo;s Kafka receiver. See the Kafka monitoring setup for how to wire the collector to OAP. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a broker-scope meter is empty until per-broker data is reported.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/kafka/","title":"\u003c!--"},{"body":" Kong The KONG layer monitors Kong API gateways. Kong exposes its built-in Prometheus metrics, OpenTelemetry collects and forwards them to OAP, and OAP aggregates them into the meter_kong_* families this dashboard renders. The layer key is KONG, and in Horizon\u0026rsquo;s sidebar it is grouped under Gateways.\nIn the sidebar this layer\u0026rsquo;s services are listed as Kong services, its instances as Nodes (the individual Kong data-plane nodes), and its endpoints as Routes (the matched Kong routes). The KONG layer enables three scopes — Service, Instance (Node), and Endpoint (Route). It does not ship a topology, traces, or logs tab, so this dashboard is purely the metric drill-down across those three scopes.\nThis page is the operator reference for the bundled KONG dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled KONG template; if an operator has published a customized KONG template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a service, the layer landing page lists every Kong service, sorted by request rate (RPS) by default, with four columns:\nRPS — total HTTP requests per second across the service\u0026rsquo;s nodes (aggregate_labels(meter_kong_service_http_requests,sum)).\n200/s — 200-status responses per second (aggregate_labels(meter_kong_service_http_status{code='200'}, sum)).\n404/s — 404-status responses per second (aggregate_labels(meter_kong_service_http_status{code='404'}, sum)).\n500/s — 500-status responses per second (aggregate_labels(meter_kong_service_http_status{code='500'}, sum)).\nThe three status columns give an at-a-glance health read across the fleet — a service whose 500/s is climbing next to its 200/s is failing requests upstream.\nService dashboard The primary drill-down for one selected Kong service. Every widget aggregates across the service\u0026rsquo;s nodes.\nHTTP Request Trend — total requests per second for the service (aggregate_labels(meter_kong_service_http_requests,sum)).\nHTTP Status Trend — requests per second broken down by HTTP status code (aggregate_labels(meter_kong_service_http_status,sum(code)), one line per code).\nHTTP Bandwidth — ingress / egress bandwidth in KB/s, by direction (aggregate_labels(meter_kong_service_http_bandwidth,sum(direction)), divided to KB).\nKong Latency — the time spent inside Kong itself (plugins and routing), in ms, averaged across percentiles (aggregate_labels(meter_kong_service_kong_latency,avg(p))).\nRequest Latency — total request latency in ms — the time as seen by the client, averaged across percentiles (aggregate_labels(meter_kong_service_request_latency,avg(p))).\nUpstream Latency — the time spent waiting on the upstream service Kong proxies to, in ms, averaged across percentiles (aggregate_labels(meter_kong_service_upstream_latency,avg(p))). Comparing Kong Latency, Request Latency, and Upstream Latency tells you whether added latency is coming from the gateway or from the backend behind it.\nNginx Connections — Kong\u0026rsquo;s underlying Nginx connections by state (aggregate_labels(meter_kong_service_nginx_connections_total,sum(state)), one line per state).\nNginx Timers — Nginx timers by state — running vs pending (aggregate_labels(meter_kong_service_nginx_timers,sum(state)), one line per state).\nDatastore Reachable — a per-instance table of whether each node can reach Kong\u0026rsquo;s datastore (latest(aggregate_labels(meter_kong_service_datastore_reachable,sum(service_instance_id)))), with Instance and Reachable columns. A node that can\u0026rsquo;t reach the datastore is no longer receiving config updates.\nNginx Metric Errors — the latest count of errors Kong hit while exporting its own Nginx metrics (latest(aggregate_labels(meter_kong_service_nginx_metric_errors_total,sum))), shown as a single number — a non-zero value means metric collection on that service is degraded.\nInstance dashboard For one selected node. These widgets are reported per node, so they show how one Kong data-plane process is behaving.\nHTTP Request Trend — requests per second for the node (meter_kong_instance_http_requests).\nHTTP Status Trend — requests per second by status code for the node (meter_kong_instance_http_status).\nHTTP Bandwidth — bandwidth in KB/s for the node (meter_kong_instance_http_bandwidth, divided to KB).\nKong Latency — time spent inside Kong for the node, in ms (meter_kong_instance_kong_latency).\nRequest Latency — total request latency for the node, in ms (meter_kong_instance_request_latency).\nUpstream Latency — upstream wait time for the node, in ms (meter_kong_instance_upstream_latency).\nDatastore Reachable — the latest datastore-reachability reading for the node, shown as a single number (latest(meter_kong_instance_datastore_reachable)).\nNginx Connections — Nginx connections by state for the node (meter_kong_instance_nginx_connections_total).\nNginx Timers — Nginx timers by state for the node (meter_kong_instance_nginx_timers).\nShared Memory Usage — how full the node\u0026rsquo;s Nginx shared-memory dictionaries are, as a percentage of total (meter_kong_instance_shared_dict_bytes over meter_kong_instance_shared_dict_total_bytes). When this approaches 100% the node can no longer cache new entries.\nWorker Lua VM Usage — memory used by the worker processes\u0026rsquo; Lua VMs, in MB (meter_kong_instance_memory_workers_lua_vms_bytes, divided to MB).\nEndpoint dashboard For one selected route. Kong reports a tighter metric set at route scope — status, bandwidth, and the three latency views.\nHTTP Status Trend — requests per second by status code for the route (meter_kong_endpoint_http_status).\nTotal Bandwidth — ingress / egress bandwidth in KB/s for the route (meter_kong_endpoint_http_bandwidth, divided to KB).\nKong Latency — time spent inside Kong for the route, in ms (meter_kong_endpoint_kong_latency).\nRequest Latency — total request latency for the route, in ms (meter_kong_endpoint_request_latency).\nUpstream Latency — upstream wait time for the route, in ms (meter_kong_endpoint_upstream_latency).\nRequirements The KONG dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Kong metrics flowing in through the OpenTelemetry receiver:\nService metrics — the meter_kong_service_* family (requests, status, bandwidth, the Kong / request / upstream latency trio, Nginx connections and timers, datastore reachability, and the Nginx metric-error counter), aggregated by OAP across a service\u0026rsquo;s nodes.\nInstance (node) metrics — the meter_kong_instance_* family, including the node-only meter_kong_instance_shared_dict_* and meter_kong_instance_memory_workers_lua_vms_bytes health metrics.\nEndpoint (route) metrics — the meter_kong_endpoint_* family for the per-route status, bandwidth, and latency widgets.\nThese come from Kong\u0026rsquo;s Prometheus plugin scraped by an OpenTelemetry Collector and forwarded to OAP, which converts them into the meter_kong_* metrics through its meter-analysis rules. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a route-scope or node-scope metric is empty until that level of data is reported. For the end-to-end collector and OAP setup, see the Kong monitoring backend guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/kong/","title":"\u003c!--"},{"body":" Istio Managed Services The MESH layer is where SkyWalking observes services running inside an Istio service mesh. Telemetry comes from the Envoy sidecars via Envoy\u0026rsquo;s Access Log Service (ALS), so a service does not need a language agent to appear here — Envoy reports the traffic, latency, and Envoy-runtime metrics on its behalf. This makes MESH the natural home for any workload managed by Istio, instrumented or not.\nIn Horizon\u0026rsquo;s sidebar this layer is named Istio Managed Services. Its services are listed as Services, instances as Sidecars (one per Envoy proxy), and endpoints as Endpoints. Service names follow the Istio service.namespace convention, so the namespace is surfaced as a grouping value alongside the service name. The MESH layer enables the Service, Instance (Sidecar), Endpoint, Topology, Traces, and Logs sub-tabs, plus eBPF profiling, network profiling, and pod logs. There is no endpoint-to-endpoint dependency map for this layer.\nThis page is the operator reference for the bundled MESH dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled MESH template; if an operator has published a customized MESH template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a service, the layer landing page lists every MESH service with four sortable columns, sorted by traffic (Traffic) by default:\nTraffic — calls per minute observed by the sidecars (service_cpm).\nApdex — user-satisfaction score on a 0 – 1 scale (service_apdex/10000).\nLatency — average response time in ms (service_resp_time).\nError Rate — percent of failed calls (100 - service_sla/100).\nService dashboard The primary drill-down for one selected service.\nTop 20 APIs — the busiest endpoints under this service, with tabs to re-rank by Traffic (endpoint_cpm, rpm), Slow (endpoint_resp_time, ms), and Successful Rate (endpoint_sla, %, worst-first). Click a row to jump into that endpoint.\nTraffic — mesh-wide requests per minute observed by the Envoy sidecars (service_cpm).\nError Rate — percent of failed calls (100 - service_sla/100).\nApdex — user-satisfaction score on a 0 – 1 scale (service_apdex/10000).\nAvg Response Time — mean latency in ms (service_resp_time).\nResponse Time Percentile — p50 / p75 / p90 / p95 / p99 latency, the tail of the response-time distribution (service_percentile).\nService Throughput — bytes per minute through the sidecar, received and sent on the same chart (service_throughput_received, service_throughput_sent).\nSidecar Internal Latency — Envoy-internal request and response latency in nanoseconds. Useful when you suspect the sidecar itself is adding overhead (service_sidecar_internal_req_latency_nanos, service_sidecar_internal_resp_latency_nanos).\nTop 10 sidecars — this service\u0026rsquo;s sidecar instances ranked across three tabs: Traffic (service_instance_cpm, rpm), Slow (service_instance_resp_time, ms), and Successful Rate (service_instance_sla, %, worst-first).\nInstance dashboard For one selected sidecar instance. The first five widgets always render; the Envoy-runtime widgets that follow appear only when the sidecar actually reports those metrics, so a non-Envoy or partially-instrumented sidecar simply shows fewer panels.\nAlways shown\nSidecar Load — calls per minute against the selected sidecar instance (service_instance_cpm).\nSidecar Latency — average response time in ms (service_instance_resp_time).\nSidecar Success Rate — percent of successful calls (service_instance_sla/100).\nSidecar Throughput — bytes through the sidecar, received and sent (service_instance_throughput_received, service_instance_throughput_sent).\nSidecar Internal Latency — Envoy-internal request and response latency in nanoseconds (service_instance_sidecar_internal_req_latency_nanos, service_instance_sidecar_internal_resp_latency_nanos).\nEnvoy runtime (shown when the sidecar reports it)\nEnvoy Upstream Request Active — in-flight upstream requests per cluster (envoy_cluster_up_rq_active).\nEnvoy Upstream Request Increase — upstream requests added per minute (envoy_cluster_up_rq_incr).\nEnvoy Upstream Pending Active — pending upstream requests, a sign of connection-pool back-pressure (envoy_cluster_up_rq_pending_active).\nEnvoy Upstream Connection Active — active upstream connections per cluster (envoy_cluster_up_cx_active).\nEnvoy Upstream Connection Increase — upstream connections added per minute (envoy_cluster_up_cx_incr).\nEnvoy Cluster Healthy Membership — healthy upstream members per cluster; a non-trivial drop signals upstream churn (envoy_cluster_membership_healthy).\nEnvoy Total Connections — total vs parent connections in use (envoy_total_connections_used, envoy_parent_connections_used).\nEnvoy Heap Memory — Envoy memory in MB: heap used / max, allocated used / max, and physical size / max (envoy_heap_memory_used, envoy_heap_memory_max_used, envoy_memory_allocated, envoy_memory_allocated_max, envoy_memory_physical_size, envoy_memory_physical_size_max).\nEnvoy Worker Threads — live vs max worker threads (envoy_worker_threads, envoy_worker_threads_max).\nEnvoy Bug Failures — Envoy\u0026rsquo;s own assertion / bug counter; expected to be zero in healthy clusters (envoy_bug_failures).\nEndpoint dashboard For one selected endpoint.\nEndpoint Traffic — calls per minute for the endpoint (endpoint_cpm).\nEndpoint Avg Response Time — average latency in ms (endpoint_resp_time).\nEndpoint Success Rate — percent of successful calls (endpoint_sla/100).\nEndpoint Response Time Percentile — p50 / p75 / p90 / p95 / p99 latency for the endpoint (endpoint_percentile).\nSidecar Internal Latency (endpoint scope) — Envoy-internal request and response latency on this endpoint, in nanoseconds (endpoint_sidecar_internal_req_latency_nanos, endpoint_sidecar_internal_resp_latency_nanos).\nTopology and maps The MESH layer ships the service map and the instance (sidecar) map. There is no endpoint-dependency map for this layer.\nTopology (service map) — every service node is decorated with RPM (service_cpm), an SLA health ring (service_sla/100, colored green / amber / red — higher is better, so the ring turns red as success rate drops), and Latency (service_resp_time). Each call edge carries server-side and client-side RPM, Avg response time, p95, and SLA (service_relation_server_* and service_relation_client_*), shown aligned in the edge panel.\nInstance map (sidecars) — from a call between two services on the service map, drill into the sidecar-to-sidecar calls between them. The same node / edge metric set is evaluated at instance scope: node RPM (service_instance_cpm), SLA ring (service_instance_sla/100), and Latency (service_instance_resp_time); each edge shows server-side and client-side RPM, Avg response time, p95, and SLA (service_instance_relation_server/client_*).\nFor how these maps are read and navigated, see the 3D Infrastructure Map for the cross-layer view, and the topology section of Layer Dashboard Templates for how the node and edge metrics are configured.\nTraces MESH traces are served from Zipkin. Open the Traces tab to query the sidecar-reported spans; the workflow and filters are the same as any other layer\u0026rsquo;s trace view — see Traces.\nRequirements The MESH dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nEnvoy ALS enabled — the sidecars must stream access logs to OAP via the Access Log Service so it can derive the service / instance / endpoint traffic, latency, and SLA metrics. See the Envoy ALS setup guide.\nService / instance / endpoint metrics — the service_*, service_instance_*, and endpoint_* families (traffic, response time, SLA, apdex, percentiles), including the mesh-specific *_throughput_* and *_sidecar_internal_*_latency_nanos metrics produced from the ALS stream.\nRelation metrics — service_relation_* and service_instance_relation_* for the service map and the sidecar map.\nEnvoy-runtime metrics — the envoy_cluster_*, envoy_*_connections_used, envoy_*_memory_*, envoy_worker_threads*, and envoy_bug_failures families, for the Envoy-runtime instance widgets to appear. A sidecar only shows the families its Envoy build emits.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so an instance- or endpoint-scope metric is empty until that level of data is reported. When a whole family is missing (for example a sidecar that does not export Envoy-runtime metrics), its widgets are hidden rather than shown empty.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/mesh/","title":"\u003c!--"},{"body":" Istio Control Plane The MESH_CP layer monitors the Istio control plane — the istiod / Pilot process that distributes configuration to the data-plane proxies. SkyWalking scrapes the control-plane\u0026rsquo;s Prometheus metrics over OpenTelemetry and rolls them into per-control-plane meters, so operators can watch xDS push health, proxy convergence, configuration validation, and the Go runtime of istiod itself. See the upstream Istio monitoring setup for how to wire the telemetry pipeline.\nIn Horizon\u0026rsquo;s sidebar this layer is named Istio Control Plane (grouped under Istio). Its services are listed as Control Planes — each control plane is named service.namespace, with the namespace shown as its grouping alias. The MESH_CP layer enables only the Service sub-tab; it has no instance, endpoint, topology, traces, or logs view.\nThis page is the operator reference for the bundled MESH_CP dashboard: what you see on the service scope and what each widget means.\nThe widgets and metrics below are read from the bundled MESH_CP template; if an operator has published a customized MESH_CP template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a control plane, the layer landing page lists every Istio control plane with four sortable columns, sorted by CPU by default:\nCPU — average CPU usage of the control-plane process (meter_istio_cpu).\nGoroutines — total goroutines running in istiod (meter_istio_go_goroutines).\nPilot xDS — total xDS connections Pilot is serving (meter_istio_pilot_xds).\nServices — total services Pilot knows about (meter_istio_pilot_services).\nService dashboard The primary drill-down for one selected control plane, covering its Go runtime, xDS push pipeline, configuration validation, and proxy conflicts.\nCPU — CPU usage of the control-plane process over time (meter_istio_cpu).\nGoroutines — goroutines running in istiod (meter_istio_go_goroutines).\nIstio Versions — the reported Pilot / Istio version build info, so a version roll-out is visible on the timeline (meter_istio_pilot_version).\nMemory (MB) — the Go runtime\u0026rsquo;s memory footprint on one chart, in MB: allocated, heap in-use, stack in-use, virtual, and resident (meter_istio_go_alloc, meter_istio_go_heap_inuse, meter_istio_go_stack_inuse, meter_istio_virtual_memory, meter_istio_resident_memory, each /1024/1024).\nPilot Errors — xDS rejections and push timeouts that indicate the control plane could not deliver config: CDS / EDS / RDS / LDS rejects plus write timeouts (meter_istio_pilot_xds_cds_reject, meter_istio_pilot_xds_eds_reject, meter_istio_pilot_xds_rds_reject, meter_istio_pilot_xds_lds_reject, meter_istio_pilot_xds_write_timeout).\nProxy Push Time (percentile) — how long it takes to push config to proxies, as a latency percentile distribution in ms (meter_istio_pilot_proxy_push_percentile).\nPilot Pushes — the rate of xDS pushes Pilot sends to proxies (meter_istio_pilot_xds_pushes).\nSidecar Injection Success — successful sidecar-injection webhook calls (meter_istio_sidecar_injection_success_total).\nADS Monitoring — the aggregated discovery surface on one chart: xDS connections, known services, and virtual services (meter_istio_pilot_xds, meter_istio_pilot_services, meter_istio_pilot_virt_services).\nConfiguration Validation — Galley configuration-validation outcomes, passed vs failed (meter_istio_galley_validation_passed, meter_istio_galley_validation_failed).\nPilot Conflicts — listener conflicts Pilot detected while generating config, broken out by type: outbound TCP/TCP, inbound, outbound TCP/HTTP, and outbound HTTP/TCP (meter_istio_pilot_conflict_ol_tcp_tcp, meter_istio_pilot_conflict_il, meter_istio_pilot_conflict_ol_tcp_http, meter_istio_pilot_conflict_ol_http_tcp).\nRequirements The MESH_CP dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the Istio control-plane meter family, produced from istiod\u0026rsquo;s Prometheus metrics collected over OpenTelemetry:\nControl-plane metrics — the meter_istio_* family for the service list and the service dashboard: the Go runtime (meter_istio_cpu, meter_istio_go_goroutines, the meter_istio_go_* memory gauges, meter_istio_virtual_memory, meter_istio_resident_memory), the Pilot / xDS pipeline (meter_istio_pilot_xds, meter_istio_pilot_xds_pushes, the meter_istio_pilot_xds_*_reject rejection counters, meter_istio_pilot_xds_write_timeout, meter_istio_pilot_proxy_push_percentile, meter_istio_pilot_services, meter_istio_pilot_virt_services, meter_istio_pilot_version, the meter_istio_pilot_conflict_* counters), sidecar injection (meter_istio_sidecar_injection_success_total), and Galley validation (meter_istio_galley_validation_passed, meter_istio_galley_validation_failed). All MESH_CP metrics are reported at the control-plane Service scope; OAP does not roll a metric up across scopes, so the dashboard stays empty until the control plane\u0026rsquo;s telemetry is reported. See the upstream Istio monitoring setup for the collection pipeline that produces this family.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/mesh_cp/","title":"\u003c!--"},{"body":" Istio Data Plane The MESH_DP layer monitors the Istio data plane — the Envoy sidecar proxies that carry the mesh\u0026rsquo;s traffic. Where the service-mesh control-plane and request telemetry live in the MESH layer, MESH_DP is the proxy\u0026rsquo;s own view: the runtime health of each Envoy process and its upstream clusters, fed by Envoy\u0026rsquo;s metrics-service output. It is grouped under Istio in the sidebar.\nIn Horizon\u0026rsquo;s sidebar this layer\u0026rsquo;s entities are the Envoy sidecars themselves. Its top-level entities are listed as Sidecar services, and each sidecar process is a Sidecars instance — there is no separate per-application service or endpoint slot, so MESH_DP reads \u0026ldquo;Sidecar service / Sidecar\u0026rdquo; rather than the GENERAL layer\u0026rsquo;s \u0026ldquo;Service / Instance / Endpoint\u0026rdquo;. Sidecar names follow the Istio name.namespace convention, so the namespace is surfaced as the displayed grouping.\nMESH_DP enables the Sidecar (instance) dashboard plus the Logs and eBPF profiling tabs; pod logs are available for the sidecar. It does not ship a service dashboard, an endpoint dashboard, a topology / map view, or a traces tab — the layer is scoped to per-sidecar runtime metrics, so those sections are absent.\nThis page is the operator reference for the bundled MESH_DP dashboard: what you see on the sidecar scope and what each widget means.\nThe widgets and metrics below are read from the bundled MESH_DP template; if an operator has published a customized MESH_DP template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nSidecar service list The layer landing page lists every sidecar service. This layer defines no metric columns on the list, so the landing view is a plain, namespace-grouped roster of sidecar services — pick one to open its sidecars, then drill into a single sidecar\u0026rsquo;s dashboard.\nSidecar dashboard For one selected sidecar (an Envoy instance). The dashboard opens with four single-value status cards, then a set of time-series trends.\nStatus cards\nBug Failures — Envoy\u0026rsquo;s internal bug-failure counter; a non-zero value means an assertion or debug check tripped inside the proxy (envoy_bug_failures).\nMembership Healthy — the count of healthy endpoints across all of this Envoy\u0026rsquo;s upstream clusters (envoy_cluster_membership_healthy).\nWorker Threads — concurrent worker threads currently in use (envoy_worker_threads).\nUpstream Request Active — total active upstream requests across this Envoy\u0026rsquo;s clusters (envoy_cluster_up_rq_active).\nConnections and requests\nUpstream Connection Active — active upstream connections over time (envoy_cluster_up_cx_active).\nUpstream Request Pending — requests waiting in upstream queues (envoy_cluster_up_rq_pending_active).\nConnections Used — server-side connections in use, plotted as total and parent (envoy_total_connections_used, envoy_parent_connections_used).\nUpstream Connection Increase — new upstream connections opened per minute (envoy_cluster_up_cx_incr).\nUpstream Request Increase — new upstream requests per minute (envoy_cluster_up_rq_incr).\nThreads and memory\nWorker Threads (current vs max) — concurrent worker threads in use plotted against the window maximum, as current and max (envoy_worker_threads, envoy_worker_threads_max).\nServer Memory — the proxy\u0026rsquo;s memory footprint in bytes, each line paired with its window maximum: heap / heap max (envoy_heap_memory_used, envoy_heap_memory_max_used), allocated / allocated max (envoy_memory_allocated, envoy_memory_allocated_max), and physical / physical max (envoy_memory_physical_size, envoy_memory_physical_size_max).\nLogs and profiling Beyond the dashboard, the sidecar\u0026rsquo;s triage tabs are:\nLogs — the log stream is scoped to the sidecar (instance), so logs are read against the selected Envoy proxy rather than a higher-level service.\neBPF profiling — on-CPU / network profiling of the sidecar process via the eBPF profiling workflow.\nPod logs are also available for the sidecar, surfacing the underlying pod\u0026rsquo;s container output alongside the proxy\u0026rsquo;s own log stream.\nRequirements The MESH_DP dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the Envoy metrics-service receiver enabled and the sidecars configured to push their metrics to it. Every widget on the sidecar dashboard reads the envoy_* metric family at instance scope:\nProxy health — envoy_bug_failures, envoy_cluster_membership_healthy.\nWorker threads — envoy_worker_threads, envoy_worker_threads_max.\nUpstream clusters — envoy_cluster_up_rq_active, envoy_cluster_up_cx_active, envoy_cluster_up_rq_pending_active, envoy_cluster_up_cx_incr, envoy_cluster_up_rq_incr.\nServer connections — envoy_total_connections_used, envoy_parent_connections_used.\nServer memory — envoy_heap_memory_used, envoy_heap_memory_max_used, envoy_memory_allocated, envoy_memory_allocated_max, envoy_memory_physical_size, envoy_memory_physical_size_max.\nThese are instance-scope metrics; OAP does not roll a metric up across scopes, so the dashboard is empty until each sidecar\u0026rsquo;s Envoy is actually reporting to the metrics-service receiver. For how to point Envoy at OAP, see Envoy\u0026rsquo;s metrics service setting.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/mesh_dp/","title":"\u003c!--"},{"body":" MongoDB The MONGODB layer monitors MongoDB database clusters. SkyWalking collects MongoDB\u0026rsquo;s internal metrics — document and operation throughput, connections, cursors, replication lag and buffer, per-database data and index size, and per-node host stats — and Horizon renders them as a cluster-level and a node-level dashboard. This is a metrics-only layer: there are no traces, logs, endpoints, or a topology map for MongoDB.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Databases and named MongoDB. Its services are listed as MongoDB clusters and its instances as Nodes. Because the layer carries no endpoint, topology, trace, or log data, it enables only two sub-tabs: a Service (cluster) dashboard and an Instance (node) dashboard.\nThis page is the operator reference for the bundled MONGODB dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled MONGODB template; if an operator has published a customized MONGODB template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every MongoDB cluster with four sortable columns, sorted by document throughput (Doc QPS) by default:\nDoc QPS — document operations per second across the cluster, summed over the document-operation types (aggregate_labels(meter_mongodb_cluster_document_avg_qps, sum(doc_op_type))).\nOp QPS — database operations per second across the cluster, summed over the operation types (aggregate_labels(meter_mongodb_cluster_operation_avg_qps, sum(legacy_op_type))).\nConns — total open connections across the cluster (aggregate_labels(meter_mongodb_cluster_connections,sum)).\nRepl Lag — replication lag in ms (meter_mongodb_cluster_repl_lag).\nService dashboard The cluster-level drill-down for one selected MongoDB cluster. Most widgets aggregate across the cluster\u0026rsquo;s nodes with aggregate_labels(...).\nCluster Uptime (days) — how long the cluster has been running, as a single card, taking the max uptime across nodes and converting from seconds (latest(aggregate_labels(meter_mongodb_cluster_uptime,max))/3600/24).\nData Size (GB) — total stored data across the cluster, as a card, summed across nodes and converted from bytes (latest(aggregate_labels(meter_mongodb_cluster_data_size,sum))/1024/1024/1024).\nCollection Count — total number of collections across the cluster, as a card (latest(aggregate_labels(meter_mongodb_cluster_collection_count,sum))).\nObject Count — total number of objects (documents) across the cluster, as a card (latest(aggregate_labels(meter_mongodb_cluster_object_count,sum))).\nDocument QPS — document operations per second over time, summed over the document-operation types (aggregate_labels(meter_mongodb_cluster_document_avg_qps, sum(doc_op_type))).\nOperation QPS — database operations per second over time, summed over the operation types (aggregate_labels(meter_mongodb_cluster_operation_avg_qps, sum(legacy_op_type))).\nTotal Connections — open connections across the cluster over time (aggregate_labels(meter_mongodb_cluster_connections,sum)).\nCursor Total — open cursors across the cluster, summed over the cursor types (aggregate_labels(meter_mongodb_cluster_cursor_avg, sum(csr_type))).\nReplication Lag (ms) — replication lag in ms (meter_mongodb_cluster_repl_lag).\nDB Total Data (GB) — a per-database table of stored data size in GB, summed per database and converted from bytes, with columns Database and Data (GB) (latest(aggregate_labels(meter_mongodb_cluster_db_data_size, sum(database)))/1024/1024/1024).\nDB Total Index (GB) — a per-database table of index size in GB, summed per database and converted from bytes, with columns Database and Index (GB) (latest(aggregate_labels(meter_mongodb_cluster_db_index_size, sum(database)))/1024/1024/1024).\nInstance dashboard The node-level drill-down for one selected MongoDB node. These widgets read the per-node meter_mongodb_node_* family directly, with no cross-node aggregation.\nUptime (days) — how long the node has been running, as a card, converted from seconds (latest(meter_mongodb_node_uptime)/3600/24).\nQPS — total query throughput on the node (meter_mongodb_node_qps).\nReplSet State — a table of the node\u0026rsquo;s replica-set state, with columns Node and ReplSet state (latest(meter_mongodb_node_rs_state)).\nConnections — open connections on the node (meter_mongodb_node_connections).\nCPU Usage (%) — total CPU usage percentage on the node (meter_mongodb_node_cpu_total_percentage).\nMemory Usage — memory used by the node (meter_mongodb_node_memory_usage).\nMemory Free (GB) — free memory in GB as two series, mem and swap, each converted from KB (meter_mongodb_node_memory_free_kb/1024/1024, meter_mongodb_node_swap_memory_free_kb/1024/1024).\nDisk (GB) — filesystem used vs total in GB, converted from bytes (meter_mongodb_node_fs_used_size/1024/1024/1024, meter_mongodb_node_fs_total_size/1024/1024/1024).\nNetwork (KB/s) — network throughput in KB/s as in vs out, converted from bytes (meter_mongodb_node_network_bytes_in/1024, meter_mongodb_node_network_bytes_out/1024).\nActive Clients — active client connections as total, writers, and readers (meter_mongodb_node_active_total_num, meter_mongodb_node_active_writer_num, meter_mongodb_node_active_reader_num).\nDocument QPS — document operations per second on the node (meter_mongodb_node_document_qps).\nOperation QPS — database operations per second on the node (meter_mongodb_node_operation_qps).\nOp Latency (µs) — average operation latency in microseconds, computed as total latency divided by operation count, each summed over the operation types (aggregate_labels(meter_mongodb_node_latency_rate,sum(op_type))/aggregate_labels(meter_mongodb_node_op_rate,sum(op_type))).\nTransactions — active vs inactive transactions on the node (meter_mongodb_node_transactions_active, meter_mongodb_node_transactions_inactive).\nRepl Buffer — replication buffer count and size (MB), the size converted from bytes (meter_mongodb_node_repl_buffer_count, meter_mongodb_node_repl_buffer_size/1024/1024).\nQueued Operations — operations queued on the node (meter_mongodb_node_queued_operation).\nRequirements The MONGODB dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs MongoDB metrics, which OAP aggregates into the meter_mongodb_* families:\nCluster (service-scope) metrics — the meter_mongodb_cluster_* family (uptime, data and index size, collection and object counts, document and operation QPS, connections, cursors, and replication lag), which the cluster dashboard and the Service list aggregate across nodes.\nNode (instance-scope) metrics — the meter_mongodb_node_* family (uptime, QPS, replica-set state, connections, CPU, memory, disk, network, active clients, document and operation QPS, operation latency, transactions, replication buffer, and queued operations) for the node dashboard.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the node-scope widgets stay empty until per-node data is reported. Setting up MongoDB collection is described in the upstream MongoDB monitoring guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/mongodb/","title":"\u003c!--"},{"body":" MySQL / MariaDB The MYSQL layer monitors MySQL and MariaDB servers. It is populated by OAP\u0026rsquo;s MySQL/MariaDB monitoring, which scrapes a Prometheus-style mysqld-exporter and turns the result into SkyWalking meters — there is no language agent here, the data comes from the exporter.\nIn Horizon\u0026rsquo;s sidebar this layer lives under the Databases group and is named MySQL / MariaDB. A monitored cluster is listed as a MySQL cluster and each member server as a Node. This is a metrics-only layer: it enables the Service and Instance scopes and nothing else — there is no endpoint scope, no topology, and no traces or logs tabs.\nThis page is the operator reference for the bundled MySQL / MariaDB dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled MYSQL template; if an operator has published a customized MYSQL template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every MySQL cluster with four sortable columns, sorted by QPS by default. Each column aggregates the per-node meters across the whole cluster:\nQPS — queries per second across the cluster (aggregate_labels(meter_mysql_qps,sum)).\nTPS — transactions per second across the cluster (aggregate_labels(meter_mysql_tps,sum)).\nSlow QPS — slow queries per second across the cluster (aggregate_labels(meter_mysql_slow_queries_rate,sum)).\nConn Errors — connection-error rate, internal rejects plus max-connection rejects summed (aggregate_labels(meter_mysql_connection_errors_internal,sum) + aggregate_labels(meter_mysql_connection_errors_max_connections,sum)).\nService dashboard The primary drill-down for one selected cluster. Every widget here aggregates across the cluster\u0026rsquo;s nodes with aggregate_labels(..., sum).\nQPS — cluster-wide queries per second (aggregate_labels(meter_mysql_qps,sum)).\nTPS — cluster-wide transactions per second (aggregate_labels(meter_mysql_tps,sum)).\nSlow Queries / s — cluster-wide slow-query rate (aggregate_labels(meter_mysql_slow_queries_rate,sum)).\nConnection Errors — two series, internal rejects vs max-connection rejects (aggregate_labels(meter_mysql_connection_errors_internal,sum) and aggregate_labels(meter_mysql_connection_errors_max_connections,sum)).\nCommands Trend — rows-affected rate per command type: select / insert / update / delete (aggregate_labels(meter_mysql_commands_select_rate,sum), meter_mysql_commands_insert_rate, meter_mysql_commands_update_rate, meter_mysql_commands_delete_rate).\nThreads — thread counters: connected / running / cached / created (aggregate_labels(meter_mysql_threads_connected,sum), meter_mysql_threads_running, meter_mysql_threads_cached, meter_mysql_threads_created).\nSlow Statements — the top 20 slowest captured statements across the cluster, in ms, worst-first (top_n(top_n_database_statement, 20, des)). Each row carries the sampled statement text. Shows no data when OAP captured no statements in the window.\nInstance dashboard For one selected Node (a single MySQL/MariaDB server in the cluster). The four top cards are point-in-time configuration / status readings; the rest are per-node time series.\nStatus cards\nUptime — how long the server has been up, in days (latest(meter_mysql_instance_uptime)/3600/24).\nMax Connections — the server\u0026rsquo;s configured max_connections ceiling (latest(meter_mysql_instance_max_connections)).\nInnoDB Buffer Pool — the InnoDB buffer-pool size in MB (latest(meter_mysql_instance_innodb_buffer_pool_size)/1024/1024).\nThread Cache Size — the configured thread-cache size (latest(meter_mysql_instance_thread_cache_size)).\nTime series\nQPS / TPS — this node\u0026rsquo;s queries per second and transactions per second on one chart (meter_mysql_instance_qps, meter_mysql_instance_tps).\nSlow Queries / s — this node\u0026rsquo;s slow-query rate (meter_mysql_instance_slow_queries_rate).\nCommands Trend — rows-affected rate per command type for this node: select / insert / update / delete (meter_mysql_instance_commands_select_rate, meter_mysql_instance_commands_insert_rate, meter_mysql_instance_commands_update_rate, meter_mysql_instance_commands_delete_rate).\nThreads — this node\u0026rsquo;s thread counters: connected / running / cached / created (meter_mysql_instance_threads_connected, meter_mysql_instance_threads_running, meter_mysql_instance_threads_cached, meter_mysql_instance_threads_created).\nConnects — available vs aborted connection rate (meter_mysql_instance_connects_available, meter_mysql_instance_connects_aborted).\nConnection Errors — internal rejects vs max-connection rejects for this node (meter_mysql_instance_connection_errors_internal, meter_mysql_instance_connection_errors_max_connections).\nRequirements The MySQL / MariaDB dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs MySQL/MariaDB monitoring enabled so it scrapes a mysqld-exporter and produces:\nCluster (service-scope) meters — the meter_mysql_* family: QPS / TPS, slow-query rate, connection errors, the per-command rates, and the thread counters. These back the Service list and the Service dashboard.\nNode (instance-scope) meters — the meter_mysql_instance_* family: uptime, max-connections, InnoDB buffer-pool size, thread-cache size, and the per-node QPS/TPS, slow-query, command, thread, connect, and connection-error series. These back the Instance dashboard.\nSampled statements — top_n_database_statement for the Slow Statements list, captured by OAP when slow-statement sampling is configured.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the node-scope meter_mysql_instance_* series are empty until per-node data is reported. For the upstream setup steps — exporter configuration and which OAP rules to enable — see the MySQL/MariaDB monitoring documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/mysql/","title":"\u003c!--"},{"body":" Nginx The NGINX layer monitors Nginx servers and reverse proxies. Nginx, with the SkyWalking Lua module, reports request, latency, bandwidth, connection, status, and error-log telemetry, which OAP aggregates into the meter_nginx_* families this dashboard renders. The layer key is NGINX, and in Horizon\u0026rsquo;s sidebar it is grouped under Gateways.\nIn the sidebar this layer\u0026rsquo;s services are listed as Nginx services, its instances as Nodes (the individual Nginx server nodes), and its endpoints as Routes (the matched Nginx routes). The NGINX layer enables three metric scopes — Service, Instance (Node), and Endpoint (Route) — plus a Logs tab. It does not ship a topology or a traces tab, so apart from logs this dashboard is the metric drill-down across those three scopes.\nThis page is the operator reference for the bundled NGINX dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled NGINX template; if an operator has published a customized NGINX template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a service, the layer landing page lists every NGINX service, sorted by request rate (RPS) by default, with three columns:\nRPS — total HTTP requests per second across the service\u0026rsquo;s nodes (aggregate_labels(meter_nginx_service_http_requests, sum)).\n5xx % — percent of requests that returned a 5xx status, as a share of all requests over the window (aggregate_labels(meter_nginx_service_http_5xx_requests_increment, sum)/aggregate_labels(meter_nginx_service_http_requests_increment, sum)*100).\n4xx % — percent of requests that returned a 4xx status, as a share of all requests over the window (aggregate_labels(meter_nginx_service_http_4xx_requests_increment, sum)/aggregate_labels(meter_nginx_service_http_requests_increment, sum)*100).\nThe two error columns give an at-a-glance health read across the fleet — a service with a climbing 5xx % is failing requests at the proxy, a climbing 4xx % is rejecting client requests.\nService dashboard The primary drill-down for one selected Nginx service. All eight widgets aggregate across the service\u0026rsquo;s nodes.\nHTTP Request Trend — total requests per second for the service (aggregate_labels(meter_nginx_service_http_requests, sum)).\nHTTP Latency — request latency in ms, averaged across the reported percentiles (aggregate_labels(meter_nginx_service_http_latency, avg(p))).\nHTTP Bandwidth — bandwidth in KB/s, summed across the bandwidth types (aggregate_labels(meter_nginx_service_http_bandwidth, sum(type)), divided to KB/s).\nHTTP Connections — connections summed by state (aggregate_labels(meter_nginx_service_http_connections, sum(state)), one line per state).\nHTTP Status Trend — requests summed by HTTP status (aggregate_labels(meter_nginx_service_http_status, sum(status)), one line per status).\n4xx % / min — percent of requests returning a 4xx status per minute (aggregate_labels(meter_nginx_service_http_4xx_requests_increment, sum)/aggregate_labels(meter_nginx_service_http_requests_increment, sum)*100).\n5xx % / min — percent of requests returning a 5xx status per minute (aggregate_labels(meter_nginx_service_http_5xx_requests_increment, sum)/aggregate_labels(meter_nginx_service_http_requests_increment, sum)*100).\nError Log Count — count of error-log entries summed by log level (aggregate_labels(meter_nginx_service_error_log_count, sum(level)), one line per level).\nInstance dashboard For one selected node. These widgets are reported per node, so they show how one Nginx server process is behaving.\nHTTP Request Trend — requests per second for the node (meter_nginx_instance_http_requests).\nHTTP Latency — request latency in ms for the node (meter_nginx_instance_http_latency).\nHTTP Bandwidth — bandwidth in KB/s for the node (meter_nginx_instance_http_bandwidth, divided to KB/s).\nHTTP Connections — connections by state for the node (meter_nginx_instance_http_connections).\nHTTP Status Trend — requests by HTTP status for the node (meter_nginx_instance_http_status).\n4xx % / min — percent of requests returning a 4xx status per minute for the node ((meter_nginx_instance_http_4xx_requests_increment/meter_nginx_instance_http_requests_increment)*100).\n5xx % / min — percent of requests returning a 5xx status per minute for the node ((meter_nginx_instance_http_5xx_requests_increment/meter_nginx_instance_http_requests_increment)*100).\nError Log Count — count of error-log entries for the node (meter_nginx_instance_error_log_count).\nEndpoint dashboard For one selected route. Nginx reports a tighter metric set at route scope — requests, latency, bandwidth, status, and the per-route error rates.\nHTTP Request Trend — requests per second for the route (meter_nginx_endpoint_http_requests).\nHTTP Latency — request latency in ms for the route (meter_nginx_endpoint_http_latency).\nHTTP Bandwidth — bandwidth in KB for the route (meter_nginx_endpoint_http_bandwidth, divided to KB).\nHTTP Status Trend — requests by HTTP status for the route (meter_nginx_endpoint_http_status).\n4xx % / min — percent of requests returning a 4xx status per minute for the route ((meter_nginx_endpoint_http_4xx_requests_increment/meter_nginx_endpoint_http_requests_increment)*100).\n5xx % / min — percent of requests returning a 5xx status per minute for the route ((meter_nginx_endpoint_http_5xx_requests_increment/meter_nginx_endpoint_http_requests_increment)*100).\nLogs The NGINX layer enables the Logs tab. Nginx access and error logs forwarded to OAP are searchable here, scoped to the selected Nginx service, with the standard log filters and time range. This is the same logs experience as other log-enabled layers — see the layer logs view for how to filter, page, and inspect entries.\nRequirements The NGINX dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Nginx telemetry flowing in:\nService metrics — the meter_nginx_service_* family (requests, latency, bandwidth, connections, status, the 4xx / 5xx increment counters, and error-log count), aggregated by OAP across a service\u0026rsquo;s nodes.\nInstance (node) metrics — the meter_nginx_instance_* family for the per-node request, latency, bandwidth, connection, status, error-rate, and error-log widgets.\nEndpoint (route) metrics — the meter_nginx_endpoint_* family for the per-route request, latency, bandwidth, status, and error-rate widgets.\nLogs — Nginx access / error logs shipped to OAP, for the Logs tab.\nThese come from the SkyWalking Nginx Lua module emitting Nginx telemetry to OAP, which converts it into the meter_nginx_* metrics through its meter-analysis rules. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a route-scope or node-scope metric is empty until that level of data is reported. For the end-to-end setup, see the Nginx monitoring backend guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/nginx/","title":"\u003c!--"},{"body":" Linux The OS_LINUX layer monitors Linux hosts. It is populated by OAP\u0026rsquo;s VM monitoring, which scrapes a Prometheus node-exporter and turns the result into SkyWalking meters — there is no language agent here, the data comes from the exporter.\nIn Horizon\u0026rsquo;s sidebar this layer lives under the OS group and is named Linux. Each monitored host is listed as a Host. This is a metrics-only, single-scope layer: it enables the Service scope and nothing else — there is no instance scope, no endpoint scope, no topology, and no traces or logs tabs.\nThis page is the operator reference for the bundled Linux dashboard: what you see on the host scope and what each widget means.\nThe widgets and metrics below are read from the bundled OS_LINUX template; if an operator has published a customized OS_LINUX template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nHost list Before opening a host, the layer landing page lists every Linux host with four sortable columns, sorted by CPU % by default:\nCPU % — average CPU utilization across all cores (meter_vm_cpu_total_percentage).\nMemory MB — memory in use, in MB (meter_vm_memory_used/1024/1024).\nLoad 1m — the 1-minute load average (meter_vm_cpu_load1/100).\nFS % — filesystem space used, as a percent (meter_vm_filesystem_percentage).\nHost dashboard The primary drill-down for one selected host.\nCPU Average Used (%) — average CPU utilization across cores, as a percent (meter_vm_cpu_average_used).\nCPU Load — the load average at three windows: 1m / 5m / 15m (meter_vm_cpu_load1/100, meter_vm_cpu_load5/100, meter_vm_cpu_load15/100).\nFile FD Allocated — the number of allocated file descriptors (meter_vm_filefd_allocated).\nMemory RAM (MB) — four series in MB: used / total / available / buff/cache (meter_vm_memory_used/1024/1024, meter_vm_memory_total/1024/1024, meter_vm_memory_available/1024/1024, meter_vm_memory_buff_cache/1024/1024).\nMemory Swap (MB) — swap free vs swap total, in MB (meter_vm_memory_swap_free/1024/1024, meter_vm_memory_swap_total/1024/1024).\nNetwork Bandwidth (KB/s) — receive vs transmit throughput, in KB/s (meter_vm_network_receive/1024, meter_vm_network_transmit/1024).\nDisk R/W (KB/s) — disk read vs written throughput, in KB/s (meter_vm_disk_read/1024, meter_vm_disk_written/1024).\nFilesystem Usage (%) — filesystem space used, as a percent (meter_vm_filesystem_percentage).\nNetwork Status — five socket / TCP counters: established TCP connections, TCP time-wait, TCP alloc, sockets used, and UDP in-use (meter_vm_tcp_curr_estab, meter_vm_tcp_tw, meter_vm_tcp_alloc, meter_vm_sockets_used, meter_vm_udp_inuse).\nRequirements The Linux dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs VM monitoring enabled so it scrapes a node-exporter and produces:\nHost (service-scope) meters — the meter_vm_* family: CPU utilization and load average, memory (used / total / available / buff-cache / swap), file-descriptor allocation, network receive/transmit, disk read/written, filesystem usage, and the TCP / socket / UDP counters. These back the Host list and the Host dashboard. Each metric is queried at its own OAP scope; because this layer is service-scope only, every widget reads the host-level meter_vm_* series and there is no instance- or endpoint-level rollup. For the upstream setup steps — node-exporter configuration and which OAP rules to enable — see the VM monitoring documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/os_linux/","title":"\u003c!--"},{"body":" Windows The OS_WINDOWS layer monitors Windows hosts. It is populated by OAP\u0026rsquo;s Windows monitoring, which receives host telemetry (CPU, memory, network, disk) and turns it into SkyWalking meters — there is no language agent here, the data comes from the host telemetry.\nIn Horizon\u0026rsquo;s sidebar this layer lives under the OS group and is named Windows. Each monitored Windows machine is listed as a Host. This is a metrics-only, single-scope layer: it enables the Service scope and nothing else — there is no instance or endpoint scope, no topology, and no traces or logs tabs.\nThis page is the operator reference for the bundled Windows dashboard: what you see and what each widget means.\nThe widgets and metrics below are read from the bundled OS_WINDOWS template; if an operator has published a customized OS_WINDOWS template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nHost list Before opening a host, the layer landing page lists every Windows host with three sortable columns, sorted by CPU % by default:\nCPU % — average CPU utilization across the host (meter_win_cpu_total_percentage).\nMemory MB — physical memory used, in MB (meter_win_memory_used/1024/1024).\nVMem % — virtual-memory utilization percentage (avg(meter_win_memory_virtual_memory_percentage)).\nHost dashboard The primary drill-down for one selected Windows host.\nCPU Average Used (%) — average CPU utilization over the window (meter_win_cpu_average_used).\nMemory RAM (MB) — physical memory in MB, three series: used / total / available (meter_win_memory_used/1024/1024, meter_win_memory_total/1024/1024, meter_win_memory_available/1024/1024).\nVirtual Memory (MB) — virtual (page-file backed) memory in MB, free vs total (meter_win_memory_virtual_memory_free/1024/1024, meter_win_memory_virtual_memory_total/1024/1024).\nNetwork Bandwidth (KB/s) — network throughput in KB/s, receive vs transmit (meter_win_network_receive/1024, meter_win_network_transmit/1024).\nDisk R/W (KB/s) — disk throughput in KB/s, read vs written (meter_win_disk_read/1024, meter_win_disk_written/1024).\nRequirements The Windows dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Windows monitoring enabled so it ingests host telemetry and produces the host (service-scope) meter_win_* family:\nCPU — meter_win_cpu_total_percentage and meter_win_cpu_average_used back the CPU column and the CPU widget.\nMemory — meter_win_memory_used, meter_win_memory_total, meter_win_memory_available, and the meter_win_memory_virtual_memory_* series back the memory columns and the RAM / virtual-memory widgets.\nNetwork and disk — meter_win_network_receive / meter_win_network_transmit and meter_win_disk_read / meter_win_disk_written back the network and disk throughput widgets.\nEvery metric is queried at the Service (host) scope; OAP does not roll a metric up across scopes, so the dashboard stays empty until per-host data is reported. For the upstream setup steps — host-telemetry collection and which OAP rules to enable — see the Windows monitoring documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/os_windows/","title":"\u003c!--"},{"body":" Mesh Dashboard The Mesh Dashboard is the cross-layer overview for an Istio service mesh. Where the Services Dashboard centers on language-agent traffic, this one centers on the data plane: it pulls onto one screen the services routed through the Istio data plane, the Istio control-plane (pilot / xDS) push activity that keeps them configured, and — because a mesh always runs on Kubernetes — the same cluster capacity strip. It draws from the MESH, MESH_CP, and K8S layers.\nLike every overview, it sits at the top of the sidebar above the per-layer entries and appears only while at least one of its layers is reporting; a layer\u0026rsquo;s tile auto-hides when that layer has nothing reporting (refreshed on the same ~60-second cadence as the menu).\nThe widgets and metrics below are read from the bundled overview template; if an administrator has published a customized copy to OAP, the live page reflects that copy instead. These are editable defaults — reshape them in the Overview Templates admin page on a bundled-default → local-draft → Check diff \u0026amp; push flow. See Overview Templates for the editor and the stored format, and Overview Widgets for the widget vocabulary.\nMesh services row Istio-managed services (MESH) — a KPI tile with the mesh service count plus RPM (calls per minute, service_cpm), P95 (95th-percentile latency in ms, service_percentile{p='95'}), and SLA (percent successful, service_sla/100). This is the data-plane equivalent of the General-services tile. Istio pilot (MESH_CP) — a composite summarizing control-plane activity: xDS pushes (config pushes Pilot sent, meter_istio_pilot_xds_pushes), xDS connections (proxies currently connected to Pilot, meter_istio_pilot_xds), Services (the layer\u0026rsquo;s service count), and Pilot errors (rejected pushes + write timeouts across CDS / EDS / LDS / RDS, summed: meter_istio_pilot_xds_cds_reject+meter_istio_pilot_xds_eds_reject+meter_istio_pilot_xds_lds_reject+meter_istio_pilot_xds_rds_reject+meter_istio_pilot_xds_write_timeout). A climbing Pilot-errors number means the control plane is struggling to push valid config — a mesh-specific failure the data-plane tiles won\u0026rsquo;t surface. Topology \u0026amp; active alarms Mesh service topology — a live service map of the MESH layer, the bulk of the row. Same renderer as the per-layer Topology tab. Active alarms — the right-hand rail of alarms currently firing on mesh-reported services, up to 12. Read-only — alarm recovery is backend-automatic. Kubernetes Cluster capacity \u0026amp; utilisation — the same full-width K8S composite as the Services Dashboard: cluster inventory counts (Nodes, Namespaces, Deployments, StatefulSets, DaemonSets, Services, Containers) on the left, and CPU / Memory / Storage commitment bars on the right (same k8s_cluster_* metrics and 0 – 100 % scale). Mesh deployments always ride on Kubernetes, so the capacity block lives directly under the mesh health. Requirements An overview is a pure consumer of what OAP reports — it invents no data, and a tile or panel with no backing metric simply hides or reads no data. To populate the Mesh Dashboard, OAP needs:\nService-scope metrics on the MESH layer — the service_* family (traffic, response time, percentile, SLA), produced by OAP from the mesh-reported telemetry. Queried at its own OAP scope; OAP does not roll a metric up across scopes. Istio control-plane meters — the meter_istio_pilot_* family on the MESH_CP layer, for the Istio pilot composite. Relation metrics for the embedded service map — service_relation_* at the MESH layer. Alarm data — firing alarms scoped to the MESH layer, for the Active-alarms rail. Kubernetes cluster metrics — the k8s_cluster_* family on the K8S layer, for the capacity composite. When a whole layer is missing — no mesh, no Kubernetes monitoring — its tile or block is hidden rather than shown empty, and the overview drops out of the sidebar once none of its layers are reporting.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/overview-mesh/","title":"\u003c!--"},{"body":" Services Dashboard The Services Dashboard is Horizon\u0026rsquo;s default cross-layer overview — the \u0026ldquo;is everything OK?\u0026rdquo; war-room pane for your traced application services. It pulls several layers onto one screen at once: a row of count + health tiles for application and virtual-backend services, a live service map, the alarms firing right now, and the Kubernetes capacity underneath them all. It answers \u0026ldquo;how many services are up, how hard are they working, is anything on fire, and is the cluster running out of room\u0026rdquo; without you clicking into any one service.\nIt folds in the GENERAL, VIRTUAL_DATABASE, VIRTUAL_CACHE, VIRTUAL_MQ, VIRTUAL_GENAI, and K8S layers; any of those that isn\u0026rsquo;t reporting drops its tile automatically. Overviews are listed at the top of the sidebar, above the per-layer entries, and each appears only while at least one of its layers is reporting (refreshed on the same ~60-second cadence as the menu). For the service-mesh counterpart, see the Mesh Dashboard.\nThe widgets and metrics below are read from the bundled overview template; if an administrator has published a customized copy to OAP, the live page reflects that copy instead. These are editable defaults — reshape them (add / remove / resize widgets, swap MQE) in the Overview Templates admin page on a bundled-default → local-draft → Check diff \u0026amp; push flow. See Overview Templates for the editor and the stored format, and Overview Widgets for the widget vocabulary.\nServices row Five KPI tiles, one per service-class layer, each showing that layer\u0026rsquo;s reporting service count plus three headline numbers:\nGeneral services (GENERAL) — traced application services. RPM (total calls per minute, service_cpm), Latency (average response time in ms, service_resp_time), SLA (percent successful, service_sla/100). Virtual databases (VIRTUAL_DATABASE) — backend databases observed via client-side spans. RPM (database_access_cpm), Latency (database_access_resp_time), SLA (database_access_sla/100). Virtual caches (VIRTUAL_CACHE) — Redis / Memcached / … observed via client-side spans. RPM (cache_access_cpm), Latency (cache_access_resp_time), SLA (cache_access_sla/100). Virtual MQs (VIRTUAL_MQ) — message queues observed via consume + produce spans. Consume (consume rate per minute, mq_service_consume_cpm), Produce (produce rate per minute, mq_service_produce_cpm), Consume latency (ms, mq_service_consume_latency). Virtual GenAI (VIRTUAL_GENAI) — GenAI backends observed via instrumented client spans. RPM (gen_ai_provider_cpm), Latency (gen_ai_provider_resp_time), SLA (gen_ai_provider_sla/100). The RPM / consume / produce numbers are summed across the layer; latency and SLA are averaged. A layer with nothing reporting (no GenAI backends in this deployment, say) simply leaves its tile off the row.\nTopology \u0026amp; active alarms General service topology — a live service map of the GENERAL layer, taking up most of the row. Same map you see on the per-layer Topology tab, embedded here for the war-room at-a-glance view. Active alarms — a rail down the right side listing the alarms currently firing on agent-reported (GENERAL) services, up to 12 at a time. Read-only — alarm recovery is backend-automatic. Kubernetes Cluster capacity \u0026amp; utilisation — a full-width composite summarizing the K8S layer. On the left, the cluster inventory as latest counts: Nodes (k8s_cluster_node_total), Namespaces (k8s_cluster_namespace_total), Deployments (k8s_cluster_deployment_total), StatefulSets (k8s_cluster_statefulset_total), DaemonSets (k8s_cluster_daemonset_total), Services (k8s_cluster_service_total), and Containers (k8s_cluster_container_total). On the right, three utilisation bars showing how much of the cluster is already committed — CPU (requested cores over capacity, k8s_cluster_cpu_cores_requests/k8s_cluster_cpu_cores*100), Memory (requested over total, k8s_cluster_memory_requests/k8s_cluster_memory_total*100), and Storage (allocated over total, (k8s_cluster_storage_total-k8s_cluster_storage_allocatable)/k8s_cluster_storage_total*100), each on a 0 – 100 % scale. This block is the \u0026ldquo;are we about to run out of room\u0026rdquo; check that the service tiles above can\u0026rsquo;t tell you. Requirements An overview is a pure consumer of what OAP reports — it invents no data, and a tile or panel with no backing metric simply hides (the layer count drops to zero and the tile is omitted) or reads no data. To populate the Services Dashboard, OAP needs:\nService-scope metrics for each service-class layer — the service_* family for GENERAL (traffic, response time, SLA), and the virtual-backend families database_access_*, cache_access_*, mq_service_*, and gen_ai_provider_* for the virtual layers. Each is queried at its own OAP scope; OAP does not roll a metric up across scopes. Relation metrics for the embedded service map — service_relation_* at the GENERAL layer. Alarm data — firing alarms scoped to the layer, for the Active-alarms rail. Kubernetes cluster metrics — the k8s_cluster_* family (inventory totals plus CPU / memory / storage capacity), reported by the OAP Kubernetes monitoring on the K8S layer, for the capacity composite. When a whole layer is missing — no virtual MQs, no Kubernetes monitoring — its tile or block is hidden rather than shown empty, and the overview itself drops out of the sidebar once none of its layers are reporting.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/overview-services/","title":"\u003c!--"},{"body":" PostgreSQL The POSTGRESQL layer monitors PostgreSQL servers. It is populated by OAP\u0026rsquo;s PostgreSQL monitoring, which scrapes a Prometheus-style postgres-exporter and turns the result into SkyWalking meters — there is no language agent here, the data comes from the exporter.\nIn Horizon\u0026rsquo;s sidebar this layer lives under the Databases group and is named PostgreSQL. A monitored cluster is listed as a PostgreSQL cluster and each member server as a Node. This is a metrics-only layer: it enables the Service and Instance scopes and nothing else — there is no endpoint scope, no topology, and no traces or logs tabs.\nThis page is the operator reference for the bundled PostgreSQL dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled POSTGRESQL template; if an operator has published a customized POSTGRESQL template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every PostgreSQL cluster with four sortable columns, sorted by Fetched / s by default. Each column aggregates the per-node meters across the whole cluster:\nFetched / s — rows fetched per second across the cluster (aggregate_labels(meter_pg_fetched_rows_rate,sum)).\nInserted / s — rows inserted per second across the cluster (aggregate_labels(meter_pg_inserted_rows_rate,sum)).\nCache Hit — buffer-cache hit ratio across the cluster, in percent (aggregate_labels(meter_pg_cache_hit_rate,avg)).\nDeadlocks — deadlocks per second across the cluster (aggregate_labels(meter_pg_deadlocks_rate,sum)).\nService dashboard The primary drill-down for one selected cluster. Most widgets here aggregate across the cluster\u0026rsquo;s nodes with aggregate_labels(...).\nFetched Rows / s — cluster-wide rows fetched per second (aggregate_labels(meter_pg_fetched_rows_rate,sum)).\nInserted Rows / s — cluster-wide rows inserted per second (aggregate_labels(meter_pg_inserted_rows_rate,sum)).\nUpdated Rows / s — cluster-wide rows updated per second (aggregate_labels(meter_pg_updated_rows_rate,sum)).\nDeleted Rows / s — cluster-wide rows deleted per second (aggregate_labels(meter_pg_deleted_rows_rate,sum)).\nReturned Rows / s — cluster-wide rows returned per second (aggregate_labels(meter_pg_returned_rows_rate,sum)).\nTemporary Files / s — temporary files created per second across the cluster, a sign of queries spilling to disk (aggregate_labels(meter_pg_temporary_files_rate,sum)).\nCache Hit Rate — cluster-wide buffer-cache hit ratio in percent (aggregate_labels(meter_pg_cache_hit_rate,avg)).\nTransactions / s — committed vs rolled-back transactions per second (aggregate_labels(meter_pg_committed_transactions_rate,sum) and aggregate_labels(meter_pg_rolled_back_transactions_rate,sum)).\nConflicts + Deadlocks / s — two series, recovery conflicts vs deadlocks per second (aggregate_labels(meter_pg_conflicts_rate,sum) and aggregate_labels(meter_pg_deadlocks_rate,sum)).\nSessions — active vs idle sessions and the lock count across the cluster (aggregate_labels(meter_pg_active_sessions,sum), aggregate_labels(meter_pg_idle_sessions,sum), aggregate_labels(meter_pg_locks_count,sum)).\nBuffers / s — background-writer and checkpoint buffer activity: checkpoint / clean / backend fsync / alloc / backend (aggregate_labels(meter_pg_buffers_checkpoint,sum), aggregate_labels(meter_pg_buffers_clean,sum), aggregate_labels(meter_pg_buffers_backend_fsync,sum), aggregate_labels(meter_pg_buffers_alloc,sum), aggregate_labels(meter_pg_buffers_backend,sum)).\nCheckpoint Stats / s — checkpoint counters: timed / requested / write time / sync time (aggregate_labels(meter_pg_checkpoints_timed_rate,sum), aggregate_labels(meter_pg_checkpoint_req_rate,sum), aggregate_labels(meter_pg_checkpoint_write_time_rate,sum), aggregate_labels(meter_pg_checkpoint_sync_time_rate,sum)).\nSlow Statements — the top 20 slowest captured statements across the cluster, in ms, worst-first (top_n(top_n_database_statement, 20, des)). Each row carries the sampled statement text. Shows no data when OAP captured no statements in the window.\nInstance dashboard For one selected Node (a single PostgreSQL server in the cluster). The four top cards are point-in-time configuration readings; the rest are per-node time series.\nStatus cards\nShared Buffers — the node\u0026rsquo;s configured shared_buffers size in MB (latest(meter_pg_instance_shared_buffers)/1024/1024).\nEffective Cache — the node\u0026rsquo;s configured effective_cache_size in GB (latest(meter_pg_instance_effective_cache)/1024/1024/1024).\nWork Mem — the node\u0026rsquo;s configured work_mem in MB (latest(meter_pg_instance_work_mem)/1024/1024).\nMax WAL Size — the node\u0026rsquo;s configured max_wal_size in GB (latest(meter_pg_instance_max_wal_size)/1024/1024/1024).\nTime series\nFetched Rows / s — this node\u0026rsquo;s rows fetched per second (meter_pg_instance_fetched_rows_rate).\nInserted Rows / s — this node\u0026rsquo;s rows inserted per second (meter_pg_instance_inserted_rows_rate).\nCache Hit Rate — this node\u0026rsquo;s buffer-cache hit ratio in percent (meter_pg_instance_cache_hit_rate).\nSessions — this node\u0026rsquo;s active vs idle sessions and lock count (meter_pg_instance_active_sessions, meter_pg_instance_idle_sessions, meter_pg_instance_locks_count).\nConflicts + Deadlocks / s — recovery conflicts vs deadlocks per second for this node (meter_pg_instance_conflicts_rate, meter_pg_instance_deadlocks_rate).\nRequirements The PostgreSQL dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs PostgreSQL monitoring enabled so it scrapes a postgres-exporter and produces:\nCluster (service-scope) meters — the meter_pg_* family: the per-operation row rates (fetched / inserted / updated / deleted / returned), temporary-file rate, cache-hit ratio, committed and rolled-back transaction rates, conflicts and deadlocks, active / idle sessions and locks, and the background-writer buffer and checkpoint counters. These back the Service list and the Service dashboard.\nNode (instance-scope) meters — the meter_pg_instance_* family: the configured shared_buffers, effective_cache_size, work_mem, and max_wal_size readings, plus the per-node fetched / inserted row rates, cache-hit ratio, sessions and locks, and conflict / deadlock series. These back the Instance dashboard.\nSampled statements — top_n_database_statement for the Slow Statements list, captured by OAP when slow-statement sampling is configured.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the node-scope meter_pg_instance_* series are empty until per-node data is reported. For the upstream setup steps — exporter configuration and which OAP rules to enable — see the PostgreSQL monitoring documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/postgresql/","title":"\u003c!--"},{"body":" Pulsar The PULSAR layer monitors Apache Pulsar message brokers. SkyWalking collects Pulsar\u0026rsquo;s broker metrics through OpenTelemetry and renders each Pulsar cluster as a service, with its brokers as instances — so a cluster\u0026rsquo;s topic, subscription, and message-flow health sits beside the broker-level connection and JVM detail in one place.\nIn Horizon\u0026rsquo;s sidebar this layer is named Pulsar, grouped under MQ. Its services are listed as Pulsar clusters and its instances as Brokers. The PULSAR layer enables the Service and Instance sub-tabs only — there is no endpoint scope, no topology, and no traces or logs tab for this layer.\nThis page is the operator reference for the bundled PULSAR dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled PULSAR template; if an operator has published a customized PULSAR template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nPulsar cluster list Before opening a cluster, the layer landing page lists every Pulsar cluster with four sortable columns, sorted by Topics by default. Each column sums the per-label series across the cluster:\nTopics — total topics on the cluster (meter_pulsar_total_topics).\nSubscriptions — total subscriptions on the cluster (meter_pulsar_total_subscriptions).\nMsg In — incoming message rate (meter_pulsar_message_rate_in).\nMsg Out — outgoing message rate (meter_pulsar_message_rate_out).\nService dashboard The primary drill-down for one selected Pulsar cluster. Every widget aggregates the cluster\u0026rsquo;s per-label series with aggregate_labels(..., sum), giving cluster-wide totals.\nTotal Topics — number of topics on the cluster (meter_pulsar_total_topics).\nSubscriptions — number of subscriptions on the cluster (meter_pulsar_total_subscriptions).\nProducers — number of connected producers (meter_pulsar_total_producers).\nConsumers — number of connected consumers (meter_pulsar_total_consumers).\nMessage Rate — incoming vs outgoing message rate on one chart, plotted as in (meter_pulsar_message_rate_in) and out (meter_pulsar_message_rate_out).\nThroughput — incoming vs outgoing byte throughput on one chart, plotted as in (meter_pulsar_throughput_in) and out (meter_pulsar_throughput_out).\nStorage Read/Write Rate — bookkeeper storage read vs write rate, plotted as read (meter_pulsar_storage_read_rate) and write (meter_pulsar_storage_write_rate).\nStorage Size (MB) — physical vs logical storage size in MB, plotted as physical (meter_pulsar_storage_size) and logical (meter_pulsar_storage_logical_size); both are reported in bytes and divided by 1024 / 1024 for display.\nInstance dashboard For one selected broker. These widgets read the broker-scope meter_pulsar_broker_* family directly.\nActive Connections — connections currently open on the broker (meter_pulsar_broker_active_connections).\nTotal Connections — connections handled by the broker (meter_pulsar_broker_total_connections).\nConn Create Fail — failed connection-create attempts (meter_pulsar_broker_connection_create_fail_count).\nConn Create Success — successful connection-create attempts (meter_pulsar_broker_connection_create_success_count).\nConnection Closed — total connections closed (meter_pulsar_broker_connection_closed_total_count).\nJVM Buffer Pool (MB) — JVM buffer-pool bytes used by the broker, in MB (meter_pulsar_broker_jvm_buffer_pool_used_bytes, divided by 1024 / 1024).\nJVM Memory Pool Used (MB) — JVM memory-pool bytes used, in MB (meter_pulsar_broker_jvm_memory_pool_used, divided by 1024 / 1024).\nJVM Memory (MB) — JVM memory in MB plotted as used, committed, and init (meter_pulsar_broker_jvm_memory_used, meter_pulsar_broker_jvm_memory_committed, meter_pulsar_broker_jvm_memory_init, each divided by 1024 / 1024).\nJVM Threads — thread counts plotted as current, daemon, peak, and deadlocked (meter_pulsar_broker_jvm_threads_current, meter_pulsar_broker_jvm_threads_daemon, meter_pulsar_broker_jvm_threads_peak, meter_pulsar_broker_jvm_threads_deadlocked).\nGC — garbage-collection time vs count on a dual axis, with cumulative seconds on the left axis (meter_pulsar_broker_jvm_gc_collection_seconds_sum) and count on the right axis (meter_pulsar_broker_jvm_gc_collection_seconds_count).\nRequirements The PULSAR dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Pulsar monitoring enabled, so that the broker\u0026rsquo;s OpenTelemetry metrics reach OAP and are aggregated into the Pulsar meter families:\nCluster (service) metrics — the meter_pulsar_* family (topics, subscriptions, producers, consumers, message rate, throughput, and bookkeeper storage), which back the cluster list and the Service dashboard.\nBroker (instance) metrics — the meter_pulsar_broker_* family (connections and the broker JVM buffer / memory / thread / GC detail), which back the Instance dashboard.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a broker-scope metric is empty until that broker reports it. See Pulsar monitoring for how to wire a Pulsar deployment into OAP.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/pulsar/","title":"\u003c!--"},{"body":" RabbitMQ The RABBITMQ layer monitors RabbitMQ message brokers. OAP collects the metrics from RabbitMQ\u0026rsquo;s Prometheus / OpenMetrics endpoint, so each broker cluster and each broker node surfaces as a SkyWalking entity in this layer.\nIn Horizon\u0026rsquo;s sidebar this layer is named RabbitMQ. Its services are listed as RabbitMQ clusters and its instances as Nodes — a service is one RabbitMQ cluster, and each instance is one broker node inside it. The layer enables two scopes only: the Service (cluster) dashboard and the Instance (node) dashboard. There is no endpoint scope, no topology / map, and no traces or logs tab for this layer.\nThis page is the operator reference for the bundled RABBITMQ dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled RABBITMQ template; if an operator has published a customized RABBITMQ template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every RabbitMQ cluster with four sortable columns, sorted by Queues by default:\nQueues — total queues across the cluster (aggregate_labels(meter_rabbitmq_queues,sum)). Channels — total open channels across the cluster (aggregate_labels(meter_rabbitmq_channels,sum)). Connections — total open connections across the cluster (aggregate_labels(meter_rabbitmq_connections,sum)). Unconfirmed — publisher messages awaiting confirmation across the cluster (aggregate_labels(meter_rabbitmq_messages_unconfirmed,sum)). Service dashboard The cluster-level view for one selected RabbitMQ cluster.\nMemory Available Before Block (MB) — headroom in MB before the broker hits its memory high-watermark and starts blocking publishers (meter_rabbitmq_memory_available_before_publisher_blocked). Disk Available Before Block (GB) — headroom in GB before the broker hits its disk free-space limit and starts blocking publishers (meter_rabbitmq_disk_space_available_before_publisher_blocked). File Descriptors + Sockets — available file descriptors (fds) and available TCP sockets (sockets), the two resource pools that gate how many connections the broker can still accept (meter_rabbitmq_file_descriptors_available, meter_rabbitmq_tcp_socket_available). Ready Messages — messages ready to be delivered to consumers (meter_rabbitmq_message_ready_delivered_consumers). Pending Ack — messages delivered to consumers but not yet acknowledged (meter_rabbitmq_message_unacknowledged_delivered_consumers). Publish Pipeline — the publish path across four series: published, confirmed, routed, and unconfirmed (meter_rabbitmq_messages_published, meter_rabbitmq_messages_confirmed, meter_rabbitmq_messages_routed, meter_rabbitmq_messages_unconfirmed). A growing gap between published and confirmed/routed flags a routing or confirmation problem. Unroutable Messages — messages with no matching binding, split into dropped and returned (meter_rabbitmq_messages_unroutable_dropped, meter_rabbitmq_messages_unroutable_returned). Queues — queue lifecycle across the cluster: total currently present, plus the declared, created, and deleted running totals (meter_rabbitmq_queues, meter_rabbitmq_queues_declared_total, meter_rabbitmq_queues_created_total, meter_rabbitmq_queues_deleted_total). Channels — channel lifecycle: total currently open, plus the opened and closed running totals (meter_rabbitmq_channels, meter_rabbitmq_channels_opened_total, meter_rabbitmq_channels_closed_total). Connections — connection lifecycle: total currently open, plus the opened and closed running totals (meter_rabbitmq_connections, meter_rabbitmq_connections_opened_total, meter_rabbitmq_connections_closed_total). Instance dashboard The node-level view for one selected broker node. The cards across the top are single-value (latest) readings; the remaining widgets are time-series.\nReady Messages — messages ready for delivery on this node, latest value (latest(meter_rabbitmq_node_queue_messages_ready)). Incoming Messages — incoming message rate on this node, latest value (latest(meter_rabbitmq_node_incoming_messages)). Outgoing Messages — outgoing message total on this node, latest value (latest(meter_rabbitmq_node_outgoing_messages_total)). Unacknowledged Messages — delivered-but-unacknowledged messages on this node, latest value (latest(meter_rabbitmq_node_unacknowledged_messages)). Connections / Publishers / Consumers — the node\u0026rsquo;s connections, publishers, and consumers counts (latest(meter_rabbitmq_node_connections_total), latest(meter_rabbitmq_node_publisher_total), latest(meter_rabbitmq_node_consumer_total)). Channels + Queues — the node\u0026rsquo;s channels and queues counts (latest(meter_rabbitmq_node_channel_total), latest(meter_rabbitmq_node_queue_total)). Allocated Used % — percentage of the node\u0026rsquo;s allocated memory that is in use, latest value (latest(meter_rabbitmq_node_allocated_used_percent)). Memory (MB) — the node\u0026rsquo;s memory breakdown in MB: used, unused, resident, and total allocated (meter_rabbitmq_node_allocated_used_bytes, meter_rabbitmq_node_allocated_unused_bytes, meter_rabbitmq_node_process_resident_memory_bytes, meter_rabbitmq_node_allocated_total_bytes). Allocated By Type (MB) — allocated memory broken down by allocator type in MB, one series per type (meter_rabbitmq_node_allocated_by_type). Multi/Single-block Memory (MB) — allocator block usage in MB across multi used, multi unused, single used, and single unused (meter_rabbitmq_node_allocated_multiblock_used, meter_rabbitmq_node_allocated_multiblock_unused, meter_rabbitmq_node_allocated_singleblock_used, meter_rabbitmq_node_allocated_singleblock_unused). Requirements The RABBITMQ dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nCluster (service-scope) metrics — the meter_rabbitmq_* family (memory and disk headroom, file descriptors and sockets, ready / pending / unroutable messages, the publish pipeline, and queue / channel / connection lifecycle counters) that drives the service list and Service dashboard. Node (instance-scope) metrics — the meter_rabbitmq_node_* family (message counters, connections / publishers / consumers, channels / queues, and the allocator memory breakdown) that drives the Instance dashboard. These metrics come from OAP\u0026rsquo;s RabbitMQ monitoring, which scrapes the broker\u0026rsquo;s Prometheus / OpenMetrics endpoint. See the RabbitMQ monitoring setup in the SkyWalking backend documentation for how to enable it. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the node-scope widgets stay empty until per-node data is reported.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/rabbitmq/","title":"\u003c!--"},{"body":" Redis The REDIS layer monitors Redis deployments scraped through OpenTelemetry\u0026rsquo;s Redis receiver and forwarded to OAP as meters. It groups under Databases in the sidebar and is a metrics-only layer: each Redis cluster is a service, and the individual Redis processes under it are instances.\nIn Horizon\u0026rsquo;s sidebar this layer\u0026rsquo;s services are listed as Redis clusters, and the processes under a cluster as Nodes. The REDIS layer enables only the Service and Instance scopes — it has no endpoint dashboard, no topology or maps, and no Traces or Logs tabs.\nThis page is the operator reference for the bundled REDIS dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled REDIS template; if an operator has published a customized REDIS template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every Redis cluster with four sortable columns, sorted by command throughput (Commands/s) by default:\nCommands/s — total commands per second across the cluster, summed over command types (aggregate_labels(meter_redis_total_commands_rate,sum(cmd))).\nHit Rate — keyspace hit rate as a percentage, averaged across the cluster (aggregate_labels(meter_redis_hit_rate,avg)).\nMemory % — used memory as a percentage of max memory across the cluster (latest(aggregate_labels(meter_redis_memory_used_bytes,sum)/aggregate_labels(meter_redis_memory_max_bytes,sum))*100).\nClients — connected clients across the cluster (latest(aggregate_labels(meter_redis_connected_clients,sum))).\nService dashboard The primary drill-down for one selected Redis cluster. All cluster-scope widgets aggregate over the nodes that make up the cluster.\nStatus cards\nUptime (days) — cluster uptime in days, taken from the longest-running node (latest(aggregate_labels(meter_redis_uptime,max))/3600/24).\nConnected Clients — total connected clients across the cluster (latest(aggregate_labels(meter_redis_connected_clients,sum))).\nBlocked Clients — total clients blocked on a blocking call across the cluster (latest(aggregate_labels(meter_redis_blocked_clients,sum))).\nMemory Usage — used memory as a percentage of max memory across the cluster (latest(aggregate_labels(meter_redis_memory_used_bytes,sum)/aggregate_labels(meter_redis_memory_max_bytes,sum))*100).\nCharts\nTotal Commands / s — commands per second across the cluster, summed over command types (aggregate_labels(meter_redis_total_commands_rate,sum(cmd))).\nHit Rate — keyspace hit rate as a percentage (aggregate_labels(meter_redis_hit_rate,avg)).\nAvg Command Time / s — mean per-command duration, total command duration divided by total command count, summed over command types (aggregate_labels(meter_redis_commands_duration,sum(cmd))/aggregate_labels(meter_redis_commands_total,sum(cmd))).\nNet I/O (KB) — network throughput in KB, split into in and out (aggregate_labels(meter_redis_net_input_bytes_total,sum)/1024, aggregate_labels(meter_redis_net_output_bytes_total,sum)/1024).\nKeys — keyspace size over time, split into total keys, evicted keys, and expired keys (aggregate_labels(meter_redis_db_keys,sum), aggregate_labels(meter_redis_evicted_keys_total,sum), aggregate_labels(meter_redis_expired_keys_total,sum)).\nSlow Commands — the top 10 slowest captured commands in ms, sampled by the SkyWalking agent at the call site (top_n(top_n_database_statement,10,des)). Each row carries the command text. Shows no data when OAP captured no slow commands in the window.\nInstance dashboard For one selected node (a single Redis process under the cluster).\nStatus cards\nUptime (days) — node uptime in days (latest(meter_redis_instance_uptime)/3600/24).\nConnected Clients — clients connected to this node (latest(meter_redis_instance_connected_clients)).\nBlocked Clients — clients blocked on a blocking call on this node (latest(meter_redis_instance_redis_blocked_clients)).\nMemory Max (MB) — configured max memory for this node in MB (latest(meter_redis_instance_memory_max_bytes)/1000/1000).\nCharts\nMemory Usage (%) — used memory as a percentage of max for this node (meter_redis_instance_memory_usage).\nCommands / s — commands per second on this node (meter_redis_instance_total_commands_rate).\nHit Rate — keyspace hit rate as a percentage for this node (meter_redis_instance_hit_rate).\nNet I/O (KB) — network throughput in KB for this node, split into in and out (meter_redis_instance_net_input_bytes_total/1024, meter_redis_instance_net_output_bytes_total/1024).\nKeys — keyspace size over time for this node, split into total, evicted, and expired keys (meter_redis_instance_db_keys, meter_redis_instance_evicted_keys_total, meter_redis_instance_expired_keys_total).\nTotal Command Time (s) — total time spent on commands per second for this node (meter_redis_instance_commands_duration_seconds_total_rate).\nAvg Command Time — mean time spent per command on this node (meter_redis_instance_average_time_spent_by_command).\nRequirements The REDIS dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nCluster meters — the meter_redis_* family (commands rate, hit rate, used / max memory, connected and blocked clients, uptime, network bytes, keyspace counts, command duration and count), aggregated by command label where the metric is per-command. These back the service list and the cluster dashboard.\nNode meters — the meter_redis_instance_* family (the same measures at single-process scope), which back the node dashboard.\nSampled records — top_n_database_statement for the Slow Commands list, captured by the SkyWalking agent at the call site when slow-command sampling is enabled.\nThese meters come from the OpenTelemetry Redis receiver; see SkyWalking\u0026rsquo;s Redis monitoring setup for how to wire the collector to OAP. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a node-scope metric is empty until that level of data is reported.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/redis/","title":"\u003c!--"},{"body":" RocketMQ The ROCKETMQ layer monitors Apache RocketMQ message-queue clusters. SkyWalking collects RocketMQ metrics over OpenTelemetry and rolls them up into cluster-, broker-, and topic-scope metrics, so operators can watch produce / consume throughput, message size, consumer latency and backlog, and broker disk and thread-pool pressure alongside the rest of their estate. See the upstream RocketMQ monitoring setup for how to wire the telemetry pipeline.\nIn Horizon\u0026rsquo;s sidebar this layer is named RocketMQ. Its services are listed as RocketMQ clusters, its instances as Brokers, and its endpoints as Topics. The ROCKETMQ layer enables the Service, Instance, and Endpoint sub-tabs; it has no topology, traces, or logs view.\nThis page is the operator reference for the bundled ROCKETMQ dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled ROCKETMQ template; if an operator has published a customized ROCKETMQ template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every RocketMQ cluster with four sortable columns, sorted by Produce TPS by default:\nProduce TPS — messages produced per second across the cluster (meter_rocketmq_cluster_total_producer_tps).\nConsume TPS — messages consumed per second across the cluster (meter_rocketmq_cluster_total_consumer_tps).\nTopics — the latest topic count in the cluster (latest(meter_rocketmq_cluster_topic_count)).\nBrokers — the latest broker count in the cluster (latest(meter_rocketmq_cluster_broker_count)).\nService dashboard The primary drill-down for one selected RocketMQ cluster, mixing daily message volume, live throughput, disk and thread-pool health, and the cluster\u0026rsquo;s topic / broker totals.\nProduced Today — messages produced since the start of today (latest(meter_rocketmq_cluster_messages_produced_today)).\nConsumed Today — messages consumed since the start of today (latest(meter_rocketmq_cluster_messages_consumed_today)).\nProduced Yesterday — messages produced over the previous full day (latest(meter_rocketmq_cluster_messages_produced_until_yesterday)).\nConsumed Yesterday — messages consumed over the previous full day (latest(meter_rocketmq_cluster_messages_consumed_until_yesterday)).\nProducer / Consumer TPS — produce and consume throughput per second on one chart (meter_rocketmq_cluster_total_producer_tps, meter_rocketmq_cluster_total_consumer_tps).\nProducer / Consumer Message Size (MB) — produced and consumed message size, in MB (meter_rocketmq_cluster_producer_message_size/1024/1024, meter_rocketmq_cluster_consumer_message_size/1024/1024).\nMax Consumer Latency — the highest consumer latency seen across the cluster (latest(meter_rocketmq_cluster_max_consumer_latency)).\nCommitLog Disk Ratio (%) — how full the CommitLog disk is, in percent: the current ratio over time plus the latest maximum across brokers (meter_rocketmq_cluster_commitLog_disk_ratio, latest(meter_rocketmq_cluster_max_commitLog_disk_ratio)).\nThreadPool Queue Head Wait (ms) — how long the head request has waited in the pull and send broker thread-pool queues, in ms — a rising value signals broker back-pressure (meter_rocketmq_cluster_pull_threadPool_queue_head_wait_time, meter_rocketmq_cluster_send_threadPool_queue_head_wait_time).\nTopics — the latest topic count in the cluster (latest(meter_rocketmq_cluster_topic_count)).\nBrokers — the latest broker count in the cluster (latest(meter_rocketmq_cluster_broker_count)).\nInstance dashboard For one selected broker, focused on the broker\u0026rsquo;s produce / consume throughput and message size.\nProduce TPS — messages produced per second by this broker (meter_rocketmq_broker_produce_tps).\nConsume QPS — consume requests per second served by this broker (meter_rocketmq_broker_consume_qps).\nProducer Msg Size (MB) — produced message size on this broker, in MB (meter_rocketmq_broker_producer_message_size/1024/1024).\nConsumer Msg Size (MB) — consumed message size on this broker, in MB (meter_rocketmq_broker_consumer_message_size/1024/1024).\nEndpoint dashboard For one selected topic, covering producer / consumer-group throughput, message size, consumer latency, offsets, and lag.\nProducer / Consumer Group TPS — produce throughput and consumer-group consume throughput per second on one chart (meter_rocketmq_topic_producer_tps, meter_rocketmq_topic_consumer_group_tps).\nMessage Size (MB) — produced and consumed message size for the topic, in MB (meter_rocketmq_topic_producer_message_size/1024/1024, meter_rocketmq_topic_consumer_message_size/1024/1024).\nMax Message Size (MB) — the latest maximum produced and consumed message size for the topic, in MB (latest(meter_rocketmq_topic_max_producer_message_size)/1024/1024, latest(meter_rocketmq_topic_max_consumer_message_size)/1024/1024).\nConsumer Latency (s) — consumer latency for the topic, in seconds (meter_rocketmq_topic_consumer_latency/1000).\nProducer / Consumer Offsets — the topic\u0026rsquo;s producer offset and consumer-group offset over time (meter_rocketmq_topic_producer_offset, meter_rocketmq_topic_consumer_group_offset).\nBacklogged Messages — the topic lag: producer offset minus consumer-group offset, the count of produced messages a consumer group has not yet consumed (meter_rocketmq_topic_producer_offset-meter_rocketmq_topic_consumer_group_offset).\nConsumer Group Count — the latest number of consumer groups on the topic (latest(meter_rocketmq_topic_consumer_group_count)).\nBroker Count — the latest number of brokers serving the topic (latest(meter_rocketmq_topic_broker_count)).\nRequirements The ROCKETMQ dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the RocketMQ meter families, produced from cluster telemetry collected over OpenTelemetry:\nCluster metrics — the meter_rocketmq_cluster_* family (total producer / consumer TPS, messages produced / consumed today and yesterday, producer / consumer message size, max consumer latency, CommitLog disk ratio, the pull / send thread-pool queue head-wait timers, and the topic / broker counts) for the service list and the cluster dashboard.\nBroker metrics — the meter_rocketmq_broker_* family (produce TPS, consume QPS, producer / consumer message size) for the broker dashboard.\nTopic metrics — the meter_rocketmq_topic_* family (producer TPS and consumer-group TPS, producer / consumer message size and their maxima, consumer latency, producer and consumer-group offsets, and the consumer-group / broker counts) for the topic dashboard.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a broker- or topic-scope metric is empty until that level of data is reported. See the upstream RocketMQ monitoring setup for the collection pipeline that produces these families.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/rocketmq/","title":"\u003c!--"},{"body":" Go Agent (Self-Observability) The SO11Y_GO_AGENT layer is the self-observability view of the SkyWalking Go agent itself. It does not measure the application the agent instruments — it measures the agent\u0026rsquo;s own tracing machinery: how many tracing contexts it creates and finishes, how many it ignores, where contexts may have leaked, and how long the agent spends building them. Use it to confirm a Go agent is healthy and not accumulating leaked contexts or interceptor errors.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Self-Observability and named Go Agent. Its services are listed as Agent services and its instances as Agents — each Agent is one running Go process reporting these meters. This is an instance-only layer: it enables the Instance sub-tab and nothing else. There is no Service dashboard, no Endpoint dashboard, no Topology, and no Traces or Logs tabs — the agent reports a flat set of self-observability meters per process, with no service-, endpoint-, or relation-scoped data behind them.\nThis page is the operator reference for the bundled SO11Y_GO_AGENT dashboard: what you see on the Agent (instance) scope and what each widget means.\nThe widgets and metrics below are read from the bundled SO11Y_GO_AGENT template; if an operator has published a customized SO11Y_GO_AGENT template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nAgent list Selecting the layer lists the Agent services, and under one Agent service its Agents (instances) — one row per reporting Go process. This layer adds no extra landing columns, so the list is the plain name list; pick an Agent to open its dashboard.\nInstance dashboard For one selected Agent (instance). Every widget on this dashboard is a Go-agent self-observability meter, charted over the selected time window.\nTracing Context Creation / min — tracing contexts the agent created per minute (meter_sw_go_created_tracing_context_count). This is the agent\u0026rsquo;s working rate — how many trace contexts it is spinning up to follow requests.\nTracing Created + Finished / min — created vs finished tracing contexts per minute on one chart: the created series (aggregate_labels(meter_sw_go_created_tracing_context_count,sum)) against the finished series (meter_sw_go_finished_tracing_context_count). In a healthy agent the two lines track each other; a persistent gap (created running ahead of finished) is the signal that contexts are not being closed.\nIgnored Context Creation / min — contexts the agent created but deliberately ignored per minute (meter_sw_go_created_ignored_context_count), e.g. traffic filtered out of tracing.\nIgnored Created + Finished / min — the same created-vs-finished comparison for ignored contexts: created (aggregate_labels(meter_sw_go_created_ignored_context_count,sum)) against finished (meter_sw_go_finished_ignored_context_count).\nPossible Leaked Context / min — contexts the agent flags as possibly leaked per minute (meter_sw_go_possible_leaked_context_count). A non-zero, sustained line here points at instrumentation that opens a context without closing it — the key health signal on this dashboard.\nInterceptor Error Count / min — errors raised inside the agent\u0026rsquo;s interceptors per minute (meter_sw_go_interceptor_error_count). Rising values indicate the agent is failing while wrapping calls, which can mean lost or incomplete traces.\nTracing Context Execution Time (ms) — the time the agent spends building a tracing context, as a p50 / p75 / p90 / p95 / p99 latency distribution in milliseconds (relabels(meter_sw_go_tracing_context_execution_time_percentile,p='50,75,90,95,99',p='50,75,90,95,99')/1000000). The agent reports this percentile in nanoseconds, so the dashboard divides by 1,000,000 to display milliseconds. Watch the tail (p95 / p99) for instrumentation overhead.\nRequirements The SO11Y_GO_AGENT dashboard is a pure consumer of what the Go agent reports through OAP — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the Go agent\u0026rsquo;s self-observability meter family at instance scope:\nAgent self-observability meters — the meter_sw_go_* family: created / finished / ignored / leaked tracing-context counts, interceptor error count, and the tracing-context execution-time percentile. These are emitted by the SkyWalking Go agent\u0026rsquo;s own self-observability reporting, not derived from the traced application. Every metric here is queried at the ServiceInstance (Agent) scope; OAP does not roll a metric up across scopes, so the dashboard stays empty until a Go agent is actively reporting these meters. When the meter family is missing entirely — for example a Go agent build with self-observability disabled — the widgets render no data rather than failing.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/so11y_go_agent/","title":"\u003c!--"},{"body":" Java Agent (Self-Observability) The SO11Y_JAVA_AGENT layer is the self-observability view of the SkyWalking Java agent itself — not the services it instruments, but the health of the agent running inside each Java process. It surfaces the agent\u0026rsquo;s own internal counters: how many tracing contexts it creates and finishes, how many it ignores, how many may have leaked, how often its interceptors error, and how long its tracing context bookkeeping takes. Use it to confirm an agent is healthy and to catch agent-side problems (context leaks, interceptor failures) that would otherwise be invisible from the application\u0026rsquo;s own metrics.\nIn Horizon\u0026rsquo;s sidebar this layer lives under the Self-Observability group and is named Java Agent. It has no service-level page: the layer reports per-agent, so its services are listed as Agent services and its instances as Agents, and the only drill-down it enables is the Instance (per-agent) dashboard. There is no Service, Endpoint, Topology, Traces, Logs, or profiling tab in this layer — agent self-observability is purely instance-scoped runtime telemetry.\nThis page is the operator reference for the bundled SO11Y_JAVA_AGENT dashboard: what you see on the agent dashboard and what each widget means.\nThe widgets and metrics below are read from the bundled SO11Y_JAVA_AGENT template; if an operator has published a customized copy to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nAgent list The layer landing page lists every reporting agent (Agents). This layer defines no extra landing columns, so the list is the agent roster on its own — pick an agent to open its dashboard.\nAgent dashboard The per-agent drill-down. Every widget is a time series of the agent\u0026rsquo;s own internal counters; the counts are per-minute rates and the one timing widget is in milliseconds.\nTracing Context Creation / min — how many tracing contexts the agent created per minute (meter_java_agent_created_tracing_context_count). This is the agent\u0026rsquo;s working rate: each context corresponds to a traced execution it started tracking. Tracing Created + Finished / min — created vs. finished tracing contexts on one chart, so you can see the two lines track each other (aggregate_labels(meter_java_agent_created_tracing_context_count,sum) as created, meter_java_agent_finished_tracing_context_count as finished). A persistent gap where created outruns finished points at contexts that never closed. Ignored Context Creation / min — contexts the agent deliberately skipped tracing per minute (meter_java_agent_created_ignored_context_count), for example traffic matched by the agent\u0026rsquo;s ignore/exclusion rules. Ignored Created + Finished / min — the created vs. finished pair for ignored contexts (aggregate_labels(meter_java_agent_created_ignored_context_count,sum) as created, meter_java_agent_finished_ignored_context_count as finished), the same balance check applied to the ignored path. Possible Leaked Context / min — contexts the agent suspects were leaked per minute (meter_java_agent_possible_leaked_context_count). A sustained non-zero line here is the headline agent-health warning: it usually means trace contexts are not being cleaned up correctly in the instrumented application. Interceptor Error Count / min — errors raised inside the agent\u0026rsquo;s bytecode interceptors per minute (meter_java_agent_interceptor_error_count). Non-zero values flag a misbehaving or incompatible plugin and warrant a look at the agent log. Tracing Context Execution Time (ms) — the p50 / p75 / p90 / p95 / p99 distribution of how long the agent\u0026rsquo;s tracing-context handling takes, in milliseconds (relabels(meter_java_agent_tracing_context_execution_time_percentile,p='50,75,90,95,99',p='50,75,90,95,99')/1000000). This is the agent\u0026rsquo;s own overhead tail; the percentile values are converted from nanoseconds to milliseconds for display. Requirements The SO11Y_JAVA_AGENT dashboard is a pure consumer of what the Java agent reports about itself — it invents no data, and a widget with no backing data simply reads no data. To populate it, the Java agent must have its self-observability (so11y) meters enabled so OAP receives the meter_java_agent_* family:\nContext counters — meter_java_agent_created_tracing_context_count, meter_java_agent_finished_tracing_context_count, meter_java_agent_created_ignored_context_count, meter_java_agent_finished_ignored_context_count, and meter_java_agent_possible_leaked_context_count for the creation, created-vs-finished, ignored, and leaked widgets. Interceptor errors — meter_java_agent_interceptor_error_count for the interceptor error widget. Execution-time percentiles — meter_java_agent_tracing_context_execution_time_percentile for the execution-time tail. All of these are reported at the ServiceInstance scope (one agent = one instance), which is why this layer has only the agent dashboard and no service, endpoint, or topology view. An agent that does not emit the self-observability meter family will appear in the list but render no data on every widget.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/so11y_java_agent/","title":"\u003c!--"},{"body":" OAP (Self-Observability) The SO11Y_OAP layer is SkyWalking\u0026rsquo;s own self-observability — the OAP backend reporting metrics about itself. It answers \u0026ldquo;is the backend healthy?\u0026rdquo;: each OAP node\u0026rsquo;s JVM, the analysis pipelines it runs (trace / mesh / OTEL / K8s ALS), the GraphQL query surface the UI itself hits, and the storage backend it persists to. This is the layer you watch to tell whether OAP — not the services it monitors — is the bottleneck.\nIn Horizon\u0026rsquo;s sidebar this layer is named OAP, grouped under Self-Observability. Its services are listed as OAP services and its instances as OAP nodes — one node per running OAP backend in the cluster.\nUnlike the application layers, SO11Y_OAP is a node-only layer: it ships a single instance (OAP node) dashboard and no service, endpoint, topology, traces, or logs tabs. There is no per-node landing table — pick an OAP node and you land directly on its dashboard.\nThis page is the operator reference for the bundled SO11Y_OAP dashboard: what you see on the OAP-node dashboard and what each widget means.\nThe widgets and metrics below are read from the bundled SO11Y_OAP template; if an operator has published a customized SO11Y_OAP template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nOAP node dashboard For one selected OAP node. Every widget on this dashboard is OAP-node-scoped, fed by the meter_oap_* self-observability meter family.\nJVM health The runtime the OAP node runs on.\nCPU (%) — process CPU utilization for the OAP node (meter_oap_instance_cpu_percentage).\nJVM Memory (MB) — JVM heap memory used (meter_oap_instance_jvm_memory_bytes_used, converted to MB).\nGC Count / min — garbage-collection count per minute (meter_oap_instance_jvm_gc_count).\nGC Time (ms / min) — time spent in garbage collection per minute (meter_oap_instance_jvm_gc_time).\nBuffer Pool (MB) — JVM buffer-pool memory used (meter_oap_instance_jvm_buffer_pool_bytes_used, converted to MB).\nThread Count — JVM threads broken out as live, peak, and daemon (meter_oap_jvm_thread_live_count, meter_oap_jvm_thread_peak_count, meter_oap_jvm_thread_daemon_count).\nThread States — threads by state: runnable, timed-waiting, blocked, waiting (meter_oap_jvm_thread_runnable_count, meter_oap_jvm_thread_timed_waiting_count, meter_oap_jvm_thread_blocked_count, meter_oap_jvm_thread_waiting_count).\nClass Count — loaded, unloaded total, and loaded total classes (meter_oap_jvm_class_loaded_count, meter_oap_jvm_class_total_unloaded_count, meter_oap_jvm_class_total_loaded_count).\nMetrics aggregation and persistence How much work the analysis-and-write pipeline is doing on this node.\nAggregation / min — metrics aggregated per minute (meter_oap_instance_metrics_aggregation).\nPersistence Counts / min — persistence operations per minute, split into prepare and execute (meter_oap_instance_persistence_prepare_count, meter_oap_instance_persistence_execute_count).\nPersistent Cache / min — persistent-cache activity per minute (meter_oap_instance_metrics_persistent_cache).\nPersistence Prepare Latency (ms) — p50 / p75 / p90 / p95 / p99 latency of the persistence prepare phase (meter_oap_instance_persistence_prepare_percentile).\nPersistence Execute Latency (ms) — p50 / p75 / p90 / p95 / p99 latency of the persistence execute phase (meter_oap_instance_persistence_execute_percentile).\nAggregation Queue Usage (%) — fill level of the L1 and L2 metrics-aggregation queues, top-10 worst series each (meter_oap_instance_metrics_aggregation_queue_used_per_ten_thousand, level 1 and level 2). A queue trending toward 100% is back-pressure — OAP is ingesting faster than it can aggregate.\nQuery surface (GraphQL) The query API that Horizon (and any GraphQL client) hits.\nGraphQL Latency (ms) — p50 / p75 / p90 / p95 / p99 latency of GraphQL queries served by this node (meter_oap_graphql_query_latency_percentile).\nGraphQL Query Count — GraphQL queries per minute, split into total queries and errors (meter_oap_instance_graphql_query_count, meter_oap_instance_graphql_query_error_count).\nIngestion and analysis pipelines The receivers and analyzers turning raw telemetry into metrics.\nTrace Analysis / min — traces analyzed per minute, total vs errors (meter_oap_instance_trace_count, meter_oap_instance_trace_analysis_error_count).\nTrace Analysis Latency (ms) — p50 / p75 / p90 / p95 / p99 latency of trace analysis (meter_oap_instance_trace_latency_percentile).\nMesh Analysis / min — service-mesh telemetry analyzed per minute, total vs errors (meter_oap_instance_mesh_count, meter_oap_instance_mesh_analysis_error_count).\nMesh Analysis Latency (ms) — p50 / p75 / p90 / p95 / p99 latency of mesh analysis (meter_oap_instance_mesh_latency_percentile).\nOTEL Received / s — OpenTelemetry records received per second, broken out as metrics, logs, and spans (meter_oap_otel_metrics_received, meter_oap_otel_logs_received, meter_oap_otel_spans_received).\nK8S ALS — Kubernetes Access Log Service throughput: count, dropped, streams, and err streams (meter_oap_instance_k8s_als_count, meter_oap_instance_k8s_als_drop, meter_oap_instance_k8s_als_streams, meter_oap_instance_k8s_als_error_streams).\nWatermark Circuit Breaker — cumulative break and recover counters per listener; when OAP sheds load under memory pressure, breaks climb (meter_oap_instance_watermark_circuit_breaker_break_count, meter_oap_instance_watermark_circuit_breaker_recover_count).\nZipkin Spans Dropped — Zipkin spans dropped by this node, for deployments running the Zipkin receiver (meter_oap_instance_spans_dropped_count).\nStorage backend Write latency against whichever storage backend this OAP is configured with. These two widgets are storage-specific and only render when the matching backend is in use — a BanyanDB deployment shows the BanyanDB widget, an Elasticsearch deployment shows the Elasticsearch widget.\nBanyanDB Write Latency (ms) — write latency by catalog and operation: measure bulk, stream bulk, trace bulk, stream single, and property (meter_oap_banyandb_write_latency_percentile). Shown only when BanyanDB write metrics are present.\nElasticsearch Write Latency (ms) — write latency split into single (single write / update / delete) and bulk (meter_oap_elasticsearch_write_latency_percentile). Shown only when Elasticsearch write metrics are present.\nRequirements The SO11Y_OAP dashboard is a pure consumer of what OAP reports about itself — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs its self-observability telemetry enabled, which produces the meter_oap_* meter family:\nJVM and process metrics — meter_oap_instance_cpu_percentage, meter_oap_instance_jvm_*, and the meter_oap_jvm_thread_* / meter_oap_jvm_class_* families behind the JVM-health widgets.\nPipeline and persistence metrics — meter_oap_instance_metrics_aggregation, meter_oap_instance_persistence_*, meter_oap_instance_metrics_persistent_cache, and meter_oap_instance_metrics_aggregation_queue_used_per_ten_thousand for the aggregation / persistence widgets.\nQuery metrics — meter_oap_graphql_query_latency_percentile and meter_oap_instance_graphql_query_count / _error_count for the GraphQL surface.\nIngestion metrics — the trace, mesh, OTEL, K8s ALS, watermark, and Zipkin families (meter_oap_instance_trace_*, meter_oap_instance_mesh_*, meter_oap_otel_*, meter_oap_instance_k8s_als_*, meter_oap_instance_watermark_circuit_breaker_*, meter_oap_instance_spans_dropped_count). A pipeline that isn\u0026rsquo;t running on a given node simply reports nothing, and its widget reads no data.\nStorage metrics — meter_oap_banyandb_write_latency_percentile or meter_oap_elasticsearch_write_latency_percentile, depending on the configured storage backend; only the matching widget renders.\nEach metric is queried at the OAP-node (instance) scope; OAP does not roll a metric up across scopes, so the dashboard is empty until self-observability telemetry is reported by the OAP nodes themselves. See the OAP backend setup docs for enabling the self-observability telemetry source.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/so11y_oap/","title":"\u003c!--"},{"body":" Satellite (Self-Observability) The SO11Y_SATELLITE layer is SkyWalking\u0026rsquo;s self-observability view of Apache SkyWalking Satellite — the lightweight telemetry collector that sits in front of OAP, buffering and forwarding agent traffic. When a Satellite instance reports its own runtime metrics to OAP (via the OpenTelemetry receiver), each collector shows up here as a service so you can watch the collection tier the same way you watch instrumented applications.\nIn Horizon\u0026rsquo;s sidebar this layer is named Satellite, and it is grouped under Self-Observability alongside the other components SkyWalking monitors about itself. Its services are listed as Satellite services. This layer is intentionally focused: it enables only the Service scope — there are no instance, endpoint, topology, traces, or logs sub-tabs. Everything Satellite exposes is read at the service level.\nThis page is the operator reference for the bundled Satellite dashboard: what you see on the service scope and what each widget means.\nThe widgets and metrics below are read from the bundled SO11Y_SATELLITE template; if an operator has published a customized Satellite template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list The layer landing page lists every Satellite service that has reported. This layer defines no custom landing-page metric columns, so services are listed by name only — pick one to open its dashboard.\nService dashboard The dashboard for one selected Satellite collector. Every widget is a time-series line over the selected window, covering the collector\u0026rsquo;s connection load, host CPU, internal queue, and the four stages of its event pipeline. The queue and event widgets break their series out per Satellite pipeline (tracingpipe, jvmpipe, logpipe, meterpipe, …), so you can see which collection pipeline is driving the rate; Connection Count and CPU are single series.\nConnection Count — the number of gRPC connections the collector currently holds, i.e. how many upstream agents and downstream OAP links are attached (satellite_service_grpc_connect_count).\nCPU (%) — host CPU utilization of the process running the Satellite gRPC server, as a percentage (satellite_service_server_cpu_utilization).\nQueue Used — how much of the internal buffering queue is currently occupied. Watch this against the collector\u0026rsquo;s queue capacity — a queue that stays near full means Satellite is backing up and is at risk of dropping events (satellite_service_queue_used_count).\nReceive Events — events received from upstream agents per minute, the inbound rate into the collector (satellite_service_receive_event_count).\nFetch Events — events fetched into the pipeline per minute, the rate at which buffered data is pulled forward for processing (satellite_service_fetch_event_count).\nQueue Input / Output — two series on one chart that show whether the queue is keeping pace: input is events written into the queue per minute (satellite_service_queue_input_count) and output is events sent on to OAP per minute (satellite_service_send_event_count). When output tracks input the collector is draining as fast as it fills; a persistent gap is the same backlog signal as a full Queue Used.\nRequirements The Satellite dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs each Satellite instance to push its self-observability metrics to OAP\u0026rsquo;s OpenTelemetry receiver, where they are aggregated into the satellite_service_* family at Service scope:\nConnection and host metrics — satellite_service_grpc_connect_count (gRPC connections) and satellite_service_server_cpu_utilization (server-process CPU).\nQueue metrics — satellite_service_queue_used_count for current queue occupancy, plus satellite_service_queue_input_count for the inbound queue rate.\nEvent-pipeline metrics — satellite_service_receive_event_count, satellite_service_fetch_event_count, and satellite_service_send_event_count for the receive → fetch → send stages of the collection pipeline.\nEach metric is queried at its own OAP scope; this layer reports only at Service scope, so the dashboard stays empty until a Satellite instance is configured to export its runtime metrics and they reach OAP. For how to wire that export and the underlying metric rules, see the SkyWalking Satellite self-observability setup documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/so11y_satellite/","title":"\u003c!--"},{"body":" Virtual Cache The VIRTUAL_CACHE layer monitors the cache systems your services talk to — Redis, Memcached, and the like — as virtual targets. There is no agent inside the cache itself; the data is synthesized from the cache calls that instrumented services make, so each cache appears as a service whose traffic, latency, and success rate are reconstructed from the client side.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Virtual targets and named Virtual Cache. Its services are listed as Caches. The layer is single-scope: it ships only the Service (cache) dashboard — there are no instance, endpoint, topology, trace, or log tabs for virtual caches, so this page documents the Cache list and the Cache dashboard only.\nThis page is the operator reference for the bundled VIRTUAL_CACHE dashboard: what you see on the cache landing list and what each widget on the Cache dashboard means.\nThe widgets and metrics below are read from the bundled VIRTUAL_CACHE template; if an operator has published a customized VIRTUAL_CACHE template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nCache list Before opening a cache, the layer landing page lists every virtual cache with four sortable columns, sorted by access traffic (Access RPM) by default:\nAccess RPM — total cache accesses per minute (cache_access_cpm).\nLatency — average access latency in ms (cache_access_resp_time).\np95 — 95th-percentile access latency in ms (cache_access_percentile{p='95'}).\nError Rate — percent of failed accesses (100 - cache_access_sla/100).\nCache dashboard The drill-down for one selected cache. The dashboard splits into three views of the same traffic: the combined access (all operations), then read and write broken out separately, and finally the slowest captured commands.\nAccess (all operations)\nAccess Traffic — total cache accesses per minute (cache_access_cpm).\nAvg Access Latency — mean access latency in ms (cache_access_resp_time).\nAccess Success Rate — percent of successful accesses (cache_access_sla/100).\nAccess Latency Percentile — p50 / p75 / p90 / p95 / p99 access latency, the tail of the access-latency distribution (cache_access_percentile).\nRead\nRead Traffic — cache read operations per minute (cache_read_cpm).\nRead Avg Latency — mean read latency in ms (cache_read_resp_time).\nRead Success Rate — percent of successful reads (cache_read_sla/100).\nRead Latency Percentile — p50 / p75 / p90 / p95 / p99 read latency (cache_read_percentile).\nWrite\nWrite Traffic — cache write operations per minute (cache_write_cpm).\nWrite Avg Latency — mean write latency in ms (cache_write_resp_time).\nWrite Success Rate — percent of successful writes (cache_write_sla/100).\nWrite Latency Percentile — p50 / p75 / p90 / p95 / p99 write latency (cache_write_percentile).\nSlow commands\nSlow Read Commands — the 10 slowest captured read commands against this cache (top_n(top_n_cache_read_command, 10, des), ms). Each row is a single execution — click it to copy the command, or use the trace icon at the row head to open its originating trace (shown only when the sample carries one). Shows no data when OAP captured no slow read commands in the window.\nSlow Write Commands — the 10 slowest captured write commands against this cache (top_n(top_n_cache_write_command, 10, des), ms). Same row behavior as Slow Read Commands — click to copy, or open the originating trace when the sample has one. Shows no data when none were captured.\nRequirements The VIRTUAL_CACHE dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nCache-access metrics — the cache_access_* family (traffic, response time, SLA, percentiles), produced by OAP from the cache calls that instrumented services make.\nRead / write metrics — the cache_read_* and cache_write_* families, the same measures split by operation, for the Read and Write widgets.\nSampled records — top_n_cache_read_command and top_n_cache_write_command for the Slow Read / Write Commands lists, captured by OAP when slow-command sampling is enabled.\nEach metric is queried at the cache\u0026rsquo;s Service scope; OAP does not roll a metric up across scopes, so a widget stays empty until that measure is reported for the cache. Virtual-cache data only appears when the services calling the cache are instrumented and OAP\u0026rsquo;s virtual-cache analysis is enabled — see the Virtual Cache setup guide for how OAP derives these targets.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/virtual_cache/","title":"\u003c!--"},{"body":" Virtual Database The VIRTUAL_DATABASE layer is the conjugate view of database traffic: instead of monitoring the database server itself, it shows each database as a peer that your instrumented services talk to. SkyWalking\u0026rsquo;s language agents detect outbound database calls in their traces and synthesize a virtual database node from the connection\u0026rsquo;s peer address — so a database appears here whether or not it is independently monitored, reconstructed entirely from the caller\u0026rsquo;s perspective.\nIn Horizon\u0026rsquo;s sidebar this layer lives under the Virtual targets group and is named Virtual Database. Each synthesized database is listed as a Database. This is a virtual-target layer with a single scope: it enables only the Service scope — there is no instance or endpoint scope, no topology, and no traces or logs tabs. Everything you see is derived from the access traffic the calling agents reported, so the figures describe the database as seen by its clients, not by the database engine.\nThis page is the operator reference for the bundled Virtual Database dashboard: what you see on the scope and what each widget means.\nThe widgets and metrics below are read from the bundled VIRTUAL_DATABASE template; if an operator has published a customized VIRTUAL_DATABASE template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a database, the layer landing page lists every virtual database with four sortable columns, sorted by Access RPM by default:\nAccess RPM — accesses per minute against the database (database_access_cpm).\nLatency — average access latency in ms (database_access_resp_time).\np95 — 95th-percentile access latency in ms (database_access_percentile{p='95'}).\nError Rate — percent of accesses that threw (100 - database_access_sla/100).\nService dashboard The primary drill-down for one selected database.\nAccess Traffic — accesses per minute against the virtual database (database_access_cpm).\nAvg Response Time — mean access latency in ms (database_access_resp_time).\nSuccess Rate — percent of accesses that returned without throwing (database_access_sla/100).\nResponse Time Percentile — p50 / p75 / p90 / p95 / p99 access latency, the tail of the access-time distribution (database_access_percentile).\nSlow Statements — the top 20 slowest captured statements against this database, in ms, worst-first (top_n(top_n_database_statement, 20, des)). Each row is a single statement execution; click a row to copy the statement text, or use the trace icon at the row head to open its originating trace — shown only when the sample carries one. Reads no data when OAP captured no statements in the window.\nRequirements The Virtual Database dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs your services instrumented by SkyWalking language agents that capture database calls, which produces:\nDatabase access metrics — the database_access_* family: database_access_cpm (traffic), database_access_resp_time (latency), database_access_sla (success rate), and database_access_percentile (the latency tail). These back both the Service list and the Service dashboard.\nSampled statements — top_n_database_statement for the Slow Statements list, captured by OAP when slow-statement sampling is configured on the calling services.\nEach metric is queried at its own OAP scope; the whole layer lives at the service (database) scope, so the dashboard is empty until at least one instrumented service reports database access traffic. For the upstream setup — how virtual databases are detected and configured — see the virtual database documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/virtual_database/","title":"\u003c!--"},{"body":" Virtual GenAI The VIRTUAL_GENAI layer monitors the GenAI / LLM providers your services talk to — OpenAI, Anthropic, and other model backends — as virtual targets. There is no agent inside the provider; the data is synthesized from the GenAI calls that instrumented services make, so each provider appears as a service whose request load, latency, success rate, token throughput, and estimated cost are reconstructed from the client side, then broken down per model.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Virtual targets and named Virtual GenAI. Its services are listed as GenAI Providers and its instances as Models. The VIRTUAL_GENAI layer enables the Service (GenAI Provider) and Instance (Model) dashboards only — it does not ship an Endpoint dashboard, a topology / service-map view, or Traces / Logs tabs, because the providers are monitored entirely through their GenAI meter families, which carry no per-endpoint scope or call graph.\nThis page is the operator reference for the bundled VIRTUAL_GENAI dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled VIRTUAL_GENAI template; if an operator has published a customized VIRTUAL_GENAI template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a provider, the layer landing page lists every GenAI Provider with four sortable columns, sorted by traffic (RPM) by default:\nRPM — calls per minute to the provider (gen_ai_provider_cpm).\nLatency — average response time in ms (gen_ai_provider_resp_time).\nSuccess Rate — percent of successful calls (gen_ai_provider_sla/100).\nOutput Tokens — total output (completion) tokens produced over the window (latest(gen_ai_provider_output_tokens_sum)).\nService dashboard The primary drill-down for one selected GenAI Provider. The dashboard covers the request golden signals, the latency tail, token throughput split into input and output, and an estimated spend.\nCalls / min — calls per minute to the provider (gen_ai_provider_cpm).\nAvg Response Time — mean response time in ms (gen_ai_provider_resp_time).\nSuccess Rate — percent of successful calls (gen_ai_provider_sla/100).\nLatency Percentile — p50 / p75 / p90 / p95 / p99 response time, the tail of the latency distribution, in ms (gen_ai_provider_latency_percentile).\nInput Tokens — input (prompt) token throughput, shown as the running total over the window and the per-call average (latest(gen_ai_provider_input_tokens_sum), gen_ai_provider_input_tokens_avg).\nOutput Tokens — output (completion) token throughput, shown as the running total over the window and the per-call average (latest(gen_ai_provider_output_tokens_sum), gen_ai_provider_output_tokens_avg).\nEstimated Cost — estimated spend against the provider, shown as the total over the window and the per-call average (latest(gen_ai_provider_total_estimated_cost)/1000000, gen_ai_provider_avg_estimated_cost/1000000). OAP carries the cost in micro-units, so each series is divided by 1000000 to land in whole currency units.\nInstance dashboard For one selected Model of the provider. The same golden signals as the Service view, scoped to a single model, plus a streaming time-to-first-token timing.\nCalls / min — calls per minute to this model (gen_ai_model_call_cpm).\nAvg Latency — mean latency for this model, in ms (gen_ai_model_latency_avg).\nSuccess Rate — percent of successful calls to this model (gen_ai_model_sla/100).\nLatency Percentile — p50 / p75 / p90 / p95 / p99 latency for this model, in ms (gen_ai_model_latency_percentile).\nTTFT (Time to First Token) — time to first token for streaming responses, shown as both the average and the percentile distribution, in ms (gen_ai_model_ttft_avg, gen_ai_model_ttft_percentile).\nInput Tokens — input (prompt) token throughput for this model, shown as the running total over the window and the per-call average (latest(gen_ai_model_input_tokens_sum), gen_ai_model_input_tokens_avg).\nOutput Tokens — output (completion) token throughput for this model, shown as the running total over the window and the per-call average (latest(gen_ai_model_output_tokens_sum), gen_ai_model_output_tokens_avg).\nEstimated Cost — estimated spend against this model, shown as the total over the window and the per-call average (latest(gen_ai_model_total_estimated_cost)/1000000, gen_ai_model_avg_estimated_cost/1000000). As on the Service view, the micro-unit cost is divided by 1000000 to land in whole currency units.\nRequirements The VIRTUAL_GENAI dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nProvider (service) metrics — the gen_ai_provider_* family at Service scope: call load (gen_ai_provider_cpm), response time (gen_ai_provider_resp_time), SLA (gen_ai_provider_sla), latency percentile (gen_ai_provider_latency_percentile), input / output token sums and averages (gen_ai_provider_input_tokens_*, gen_ai_provider_output_tokens_*), and the estimated-cost totals and averages (gen_ai_provider_total_estimated_cost, gen_ai_provider_avg_estimated_cost).\nModel (instance) metrics — the gen_ai_model_* family at ServiceInstance scope: call load (gen_ai_model_call_cpm), latency average and percentile (gen_ai_model_latency_avg, gen_ai_model_latency_percentile), SLA (gen_ai_model_sla), the streaming time-to-first-token timings (gen_ai_model_ttft_avg, gen_ai_model_ttft_percentile), input / output token sums and averages (gen_ai_model_input_tokens_*, gen_ai_model_output_tokens_*), and the estimated-cost totals and averages (gen_ai_model_total_estimated_cost, gen_ai_model_avg_estimated_cost).\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a model-scope (instance) metric is empty until that level of data is reported. Virtual-GenAI data only appears when the services calling the provider are instrumented and OAP\u0026rsquo;s virtual-GenAI analysis is enabled — see the Virtual GenAI setup guide for how OAP derives these targets.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/virtual_genai/","title":"\u003c!--"},{"body":" Virtual MQ The VIRTUAL_MQ layer monitors the message-queue systems your services publish to and consume from — Kafka, RocketMQ, RabbitMQ, Pulsar, and the like — as virtual targets. There is no agent inside the broker itself; the data is synthesized from the produce and consume calls that instrumented services make, so each message-queue cluster appears as a service whose throughput, latency, and success rate are reconstructed from the client side.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Virtual targets and named Virtual MQ. Its services are listed as MQ clusters, and its endpoints — the queues / topics a cluster carries — are listed as Topics. The layer enables two scopes: the Service (MQ cluster) dashboard and the Endpoint (Topic) dashboard. There are no instance, topology, trace, or log tabs for virtual MQ, so this page documents the cluster list, the MQ cluster dashboard, and the Topic dashboard only.\nThis page is the operator reference for the bundled VIRTUAL_MQ dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled VIRTUAL_MQ template; if an operator has published a customized VIRTUAL_MQ template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nMQ cluster list Before opening a cluster, the layer landing page lists every MQ cluster with four sortable columns, sorted by consume throughput (Consume RPM) by default:\nConsume RPM — messages consumed per minute across the cluster (mq_service_consume_cpm).\nProduce RPM — messages produced per minute across the cluster (mq_service_produce_cpm).\nConsume Latency — average time between message production and consumption, in ms (mq_service_consume_latency).\nConsume Error Rate — percent of failed consume operations (100 - mq_service_consume_sla/100).\nMQ cluster dashboard The drill-down for one selected MQ cluster. The dashboard pairs the produce and consume sides of the cluster\u0026rsquo;s traffic — throughput, success rate, and the consume-latency profile.\nConsume Traffic — messages consumed per minute (mq_service_consume_cpm).\nProduce Traffic — messages produced per minute (mq_service_produce_cpm).\nConsume Avg Latency — average time between message production and consumption, in ms (mq_service_consume_latency).\nConsume Success Rate — percent of successful consume operations (mq_service_consume_sla/100).\nProduce Success Rate — percent of successful produce operations (mq_service_produce_sla/100).\nConsume Latency Percentile — p50 / p75 / p90 / p95 / p99 of consume latency, the tail of the consume-latency distribution (mq_service_consume_percentile).\nTopic dashboard For one selected Topic — a queue / topic under the cluster, on the Endpoint scope. It mirrors the cluster widgets at the per-topic level.\nTopic Consume Traffic — messages consumed per minute on the topic (mq_endpoint_consume_cpm).\nTopic Produce Traffic — messages produced per minute on the topic (mq_endpoint_produce_cpm).\nTopic Consume Avg Latency — average consume latency for the topic, in ms (mq_endpoint_consume_latency).\nTopic Consume Success Rate — percent of successful consume operations on the topic (mq_endpoint_consume_sla/100).\nTopic Produce Success Rate — percent of successful produce operations on the topic (mq_endpoint_produce_sla/100).\nTopic Consume Latency Percentile — p50 / p75 / p90 / p95 / p99 of the topic\u0026rsquo;s consume latency (mq_endpoint_consume_percentile).\nRequirements The VIRTUAL_MQ dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nMQ cluster metrics — the mq_service_* family (consume / produce throughput, consume latency, consume / produce SLA, consume percentiles), produced by OAP from the produce and consume calls that instrumented services make.\nTopic metrics — the mq_endpoint_* family, the same measures evaluated at the Endpoint (Topic) scope, for the Topic dashboard.\nEach metric is queried at its own OAP scope — the mq_service_* family at the MQ cluster\u0026rsquo;s Service scope and the mq_endpoint_* family at the Topic\u0026rsquo;s Endpoint scope. OAP does not roll a metric up across scopes, so a Topic widget stays empty until that measure is reported at the Topic level, independent of the cluster-scope data. Virtual-MQ data only appears when the services producing to and consuming from the broker are instrumented and OAP\u0026rsquo;s virtual-MQ analysis is enabled — see the Virtual MQ setup guide for how OAP derives these targets.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/virtual_mq/","title":"\u003c!--"},{"body":" WeChat Mini Program The WECHAT_MINI_PROGRAM layer holds WeChat (微信) Mini Programs monitored by the SkyWalking mini-program agent. The agent runs inside the mini-program runtime and reports client-side performance — app launch, first render, package load, page routing, script execution, and outbound request timing — so each mini-program lands here rather than in a server-side layer.\nIn Horizon\u0026rsquo;s sidebar this layer is named WeChat Mini Program (under the Mobile group). Its services are listed as Mini-programs, instances as Versions (one per released mini-program version), and endpoints as Pages (one per mini-program page). The layer enables the Service, Version, Page, Traces, and Logs sub-tabs. It has no service map, instance map, or page-dependency view — mini-program telemetry is client-side timing, with no inter-service call topology to draw.\nThis page is the operator reference for the bundled WECHAT_MINI_PROGRAM dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled WECHAT_MINI_PROGRAM template; if an operator has published a customized template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nMini-program list Before opening a mini-program, the layer landing page lists every WECHAT_MINI_PROGRAM service with four sortable columns, sorted by request traffic (Request RPM) by default:\nRequest RPM — outbound requests per minute (meter_wechat_mp_request_cpm).\nLaunch — average app-launch duration in ms (meter_wechat_mp_app_launch_duration).\nFirst Render — average first-render duration in ms (meter_wechat_mp_first_render_duration).\nErrors — count of reported errors (meter_wechat_mp_error_count).\nService dashboard The primary drill-down for one selected mini-program.\nApp Launch Duration — time to launch the mini-program, in ms (meter_wechat_mp_app_launch_duration).\nFirst Render Duration — time to the first render, in ms (meter_wechat_mp_first_render_duration).\nPackage Load Duration — time to download and parse the mini-program package bundle, in ms (meter_wechat_mp_package_load_duration).\nError Count — number of errors reported by the mini-program (meter_wechat_mp_error_count).\nRoute Duration — time spent in page-route transitions, in ms (meter_wechat_mp_route_duration).\nScript Duration — script-execution time, in ms (meter_wechat_mp_script_duration).\nRequest Load — outbound requests per minute (meter_wechat_mp_request_cpm).\nRequest Duration Percentile — p50 / p75 / p90 / p95 / p99 of outbound request duration, in ms — the tail of the request-timing distribution (meter_wechat_mp_request_duration_percentile).\nVersion dashboard For one selected released Version of the mini-program. The same timing families as the service dashboard, evaluated at version (instance) scope so you can compare one release against another.\nLaunch Duration — app-launch duration for this version, in ms (meter_wechat_mp_instance_app_launch_duration).\nFirst Render Duration — first-render duration for this version, in ms (meter_wechat_mp_instance_first_render_duration).\nPackage Load Duration — package download-and-parse time for this version, in ms (meter_wechat_mp_instance_package_load_duration).\nRequest Load — outbound requests per minute for this version (meter_wechat_mp_instance_request_cpm).\nRoute Duration — page-route transition time for this version, in ms (meter_wechat_mp_instance_route_duration).\nScript Duration — script-execution time for this version, in ms (meter_wechat_mp_instance_script_duration).\nRequest Duration Percentile — p50 / p75 / p90 / p95 / p99 of outbound request duration for this version, in ms (meter_wechat_mp_instance_request_duration_percentile).\nPage dashboard For one selected Page (endpoint) of the mini-program.\nLaunch Duration on Page — app-launch duration attributed to this page, in ms (meter_wechat_mp_endpoint_app_launch_duration).\nFirst Render Duration — first-render duration for this page, in ms (meter_wechat_mp_endpoint_first_render_duration).\nRequest Load — outbound requests per minute originating from this page (meter_wechat_mp_endpoint_request_cpm).\nRequest Duration Percentile — p50 / p75 / p90 / p95 / p99 of outbound request duration for this page, in ms (meter_wechat_mp_endpoint_request_duration_percentile).\nRequirements The WECHAT_MINI_PROGRAM dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nMini-program (service) metrics — the meter_wechat_mp_* family at service scope: app launch, first render, package load, route, script, error count, request load, and request-duration percentile.\nVersion (instance) metrics — the meter_wechat_mp_instance_* family, the same timings reported per released version.\nPage (endpoint) metrics — the meter_wechat_mp_endpoint_* family, the launch / first-render / request-load / request-percentile timings reported per page.\nThese metrics come from the WeChat Mini Program agent reporting client-side timing to OAP, where the mini-program meter rules aggregate them. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the Version and Page dashboards stay empty until that level of data is reported. See the WeChat Mini Program monitoring setup for enabling the receiver and meter rules on OAP.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/dashboards/wechat_mini_program/","title":"\u003c!--"},{"body":" Alarms Path: /alarms. The page is read-only and needs no special permission to view.\nThe Alarms page is the triage surface for everything OAP\u0026rsquo;s alerting engine is firing right now, across every layer. It pulls the alarms OAP recorded over a recent window, groups the repeat firings of a rule on the same entity into a single incident, lays them out on a per-layer timeline, and shows the trigger expression and the captured metric snapshot for whichever alarm you select.\nAlarms are read-only here by design. OAP recovers an alarm automatically once the condition clears — there is no acknowledge, close, or silence action in the UI, and there is nothing to dismiss. A firing alarm stops firing when the underlying metric stops crossing the threshold; the page reflects that state, it does not drive it.\nThe time window The window picker offers three presets — 20m, 2h, 4h — plus a custom range capped at 4 hours.\nAlarms are second-precision events, and a long window pulls thousands of rows that some storage backends struggle to return; the 4-hour ceiling is enforced both in the picker and on the server, so a custom range wider than 4 hours is rejected. When the window genuinely holds more alarms than were fetched, the timeline header says so — narrow the window to see a complete slice. A window that exactly fills the fetch is complete and carries no notice.\nThe window\u0026rsquo;s starting preset can be set per deployment — see Alert page setup below.\nActive count and per-layer breakdown The KPI strip at the top counts what is actively firing, not the raw event count.\nActive — the total number of incidents that are currently firing. A fully recovered incident contributes nothing here, so this number answers \u0026ldquo;what is on fire right now?\u0026rdquo; rather than \u0026ldquo;what happened recently?\u0026rdquo;. Per-layer tiles — one tile per pinned layer (for example General, Mesh), each showing that layer\u0026rsquo;s active count. Pinned layers always render, even at zero, so the strip is stable across refreshes. Other — a read-only aggregate of active alarms in layers you did not pin, plus any alarm OAP could not attribute to a known layer. The arithmetic Active = (sum of pinned tiles) + Other always holds, so nothing hides off-screen. Overflow chips — below the tiles, the non-pinned layers that actually have an active alarm appear as small pills, sorted by count, as a filter shortcut. Clicking a tile, a chip, or a list tab narrows the timeline and the list to that layer; the selection is reflected in the URL, so a refresh or a shared link preserves it. Click the active tile again (or the Active tile) to clear the filter.\nFiltering Above the timeline is a filter row. What it offers depends on the connected OAP version:\nOn a current OAP, you get a cascading Layer → Service → Instance → Endpoint picker plus a free-text Keyword match on the alarm message. These filters are applied at the source, so the page only fetches the alarms that match. On an older OAP that does not support entity-scoped alarm queries, the row collapses to Keyword only, with a note inviting an upgrade for the full layer and entity filters. The filter is a draft until you press apply — nothing refires while you are composing it. clear resets every field.\nTimeline The timeline plots each alarm as a flag on a per-layer lane, so you can see at a glance when a burst happened and which layers it touched. It keeps every individual firing and recovery — not the merged incident — so a fire-then-recover pattern stays visible.\nTwo interactions:\nClick a flag to select that alarm and load its detail on the right. Brush a region to slice the list (and the counts) to that sub-window. The brushed rectangle is the only marker for the selection; the timeline itself still shows the full window so you can see other peaks to re-brush onto. reset clears the brushed range and the selected alarm.\nIncidents and the list OAP emits one alarm record per firing, so a rule that re-fires after its silence period produces several records. The list collapses the repeat firings of one rule on one entity into a single incident row, tagged with how many times it triggered. Each row carries a state:\nfiring — currently firing, and it never recovered within the window. unstable — currently firing, but it recovered at least once earlier in the window and fired again (a flapping rule). The badge shows how many of its firings are currently active versus recovered. Unstable still counts as active. recovered — the latest firing has cleared. Recovered incidents stay in the list as recent history but drop out of the Active count and the per-layer tiles — recovered is \u0026ldquo;no alarm\u0026rdquo;. For an incident that triggered more than once, the chevron at the end of the row expands a per-firing history: every individual firing and recovery on that entity and rule, in time order. Clicking a sub-entry loads that specific event into the detail panel. The list pages ten incidents at a time.\nAlarm detail Selecting an alarm — from a timeline flag, a list row, or an expanded history entry — opens the detail panel on the right:\nStatus — a firing or recovered pill, plus when the alarm started and (if cleared) when it recovered, and its layer. Message — the human-readable alarm text OAP formatted from the rule. Tags — any tags OAP attached to the alarm. Trigger expression — the MQE expression the rule evaluated, exactly as it fired. Rule — when the OAP admin port is reachable, the matched rule\u0026rsquo;s body: period, silence, recovery-obs, notification hooks, and the metrics it references. A \u0026ldquo;view in catalog\u0026rdquo; link jumps to the same rule on the Alerting rules page. When the admin port is unreachable, this section is omitted. Snapshot — one small chart per metric, plotting the values OAP captured at the firing moment so you can see what actually crossed the threshold. The trigger minute is marked, and the rule\u0026rsquo;s evaluation window is shaded when the rule body is available. An alarm recorded without an MQE snapshot (older OAP, or snapshot capture disabled in the rule) shows a note instead of charts. Admin: setup, pinned layers, and default window Which layers get their own KPI tile, and which window preset the page opens on, are configured on the Alert page setup admin page (/admin/alert-page-setup, verb alarm-setup:read), reachable from the page\u0026rsquo;s intro text.\nAlerting rules: the running context Path: /operate/alerting-rules. Verb: alarm-rule:read.\nThe Alerting rules page is a read-only catalog of every alarm rule loaded into the OAP cluster. Rules themselves are authored in OAP\u0026rsquo;s alarm-settings.yml and reloaded by OAP\u0026rsquo;s watcher — there is no add, edit, or delete here.\nEach rule lists its expression, window settings (period, silence, recovery-obs, and any additional period), the metrics it references, hooks, tags, entity include/exclude filters, and a per-node load state (loaded a/b) — because in a cluster each OAP instance loads the rule independently, and a partial count flags a node that has not picked it up.\nPer-entity running state Each OAP instance only evaluates a rule over the slice of entities it holds, so a rule\u0026rsquo;s Currently watching list is the union of evaluated entities across all nodes, with each entity tagged by the node watching it. Click an entity to open its live running context. Because the entity may be evaluated on only one node, the popup answers per node: the node actually evaluating it returns a populated body; the others read as \u0026ldquo;Not evaluated on this instance.\u0026rdquo;\nFor the evaluating node, the popup shows the rule\u0026rsquo;s current evaluation window (its size, the silence countdown, the recovery-observation countdown, and the window\u0026rsquo;s end time), the last alarm time and message, and a snapshot sparkline of the metric values in the window — each point annotated with its value and bucket time.\nThe headline of each node block is the rule\u0026rsquo;s current state for that entity. The states an operator will see:\nState Meaning FIRING The rule\u0026rsquo;s condition is currently met for this entity and the alarm is active. This is what surfaces as a firing alarm on the Alarms page. SILENCED_FIRING The condition is still met, but the alarm is inside its silence period after a recent firing, so OAP is holding off re-notifying. It is firing but quiet — no fresh notification goes out until the silence window elapses. OBSERVING_RECOVERY The condition has stopped being met and OAP is watching to confirm the recovery holds for the rule\u0026rsquo;s recovery-observation period before fully clearing the alarm. A flap back into breach during this window keeps the alarm active. These states are the live evaluation context behind the alarms you see on the Alarms page — they let you confirm that a rule is watching the entity you expect, see exactly where it is in the fire / silence / recover cycle, and read the very metric values it is acting on. The running context comes straight off OAP\u0026rsquo;s admin port; when that port is unreachable, the catalog surfaces a banner and the per-entity context is unavailable.\nRelated Runtime Rules (DSL) — runtime-editable MAL / LAL analysis rules that produce the metrics alarm rules evaluate. Metrics Inspect — browse the metric catalog and find which entities report a given metric. ","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/operate/alarms/","title":"\u003c!--"},{"body":" Events Events are the lifecycle records OAP has collected for a service — agent restarts, Kubernetes events, and other point-in-time facts reported by SkyWalking agents, the SkyWalking CLI, and the Kubernetes Event Exporter. Each event has a name, a type (Normal / Error), a message, and any reporter-supplied parameters. Events are distinct from alarms: an event records that something happened, not that a threshold was breached — for alerting, see Alarms.\nOpening the events popout Events are scoped to a single service and shown in a popout, so you review them without leaving the layer you\u0026rsquo;re on. On any layer drill-down, pick a service in the service banner at the top, then click the Events button next to the banner\u0026rsquo;s Share control. A modal opens for that service. The button appears only for users with the events:read permission (the built-in viewer, maintainer, and operator roles all have it).\nThe swimlane — instance × time The service is fixed (it\u0026rsquo;s in the popout title), so the view has two axes: each service instance is a row, and time runs left to right.\nAn event with a duration is a bar spanning its start to its end. An event with no end time is an instant marker (a small diamond). Each instance row is a distinct color, so the rows read apart at a glance. Error events carry a red ring so they stand out. If one instance reports overlapping events, they stack into sub-rows so nothing is hidden. A service that reports events without an instance shows a single row for the service. A rolling restart of a large service therefore shows as many bars at the same moment — one per instance — rather than a single summarised line. When a service runs many instances, use the search box at the top of the popout to filter the rows to the instances whose name matches.\nTime window and scrolling The popout owns its own window — 6h, 1d, 2d presets, plus a custom range — queried at second precision so the most recent events are never rounded out. The custom range takes an absolute start and end (entered in your browser\u0026rsquo;s local time) spanning up to 7 days; an invalid range — end before start, or a span past the 7-day cap — is rejected with the reason before anything is queried. A preset window is anchored to the moment you pick it, while a custom range is pinned exactly where you set it. Events are stored under OAP\u0026rsquo;s record retention; a window reaching past it simply returns fewer rows.\nScrolling stays inside the popout: the time-axis header stays pinned at the top and the instance column stays pinned at the left. A long range (a multi-day window) gets a wider, horizontally-scrollable canvas so bars keep a legible spacing instead of collapsing together, and the view opens scrolled to the newest events — scroll left for history. The time axis marks the date at day boundaries, so a range that crosses midnight is unambiguous.\nHow many events are shown The popout fetches the newest events up to a cap (200 by default; configurable under the server\u0026rsquo;s page-size limits). It tells you which case you\u0026rsquo;re in:\n\u0026ldquo;N events · all in range shown\u0026rdquo; — everything in the window is on screen. \u0026ldquo;Showing newest N — more available, narrow the range\u0026rdquo; — the window holds more than the cap; tighten the time range to reach older events. Event detail Click a bar to open the detail panel:\nHeader — the event type (Normal / Error) and name. Scope — the service, the instance (or \u0026ldquo;service-scoped\u0026rdquo;), the endpoint if present, and the layer. Started / Ended / Duration — for an event with a duration; a single Time for an instantaneous event. Message — the human-readable text the reporter attached. Parameters — the key/value details carried with the event (for example a Java agent\u0026rsquo;s startup options). Service names, instance names, messages, and parameter values are shown exactly as OAP reported them.\nRelated Alarms — threshold breaches from OAP\u0026rsquo;s alerting engine, a separate read-only triage surface. Traces and Logs — the other per-entity triage surfaces. ","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/operate/events/","title":"\u003c!--"},{"body":" 3D Infrastructure Map A single WebGL view of your whole deployment, stacked in 3D. Every SkyWalking layer\u0026rsquo;s services become cubes, grouped onto horizontal tiers, with live traffic, alarms, and call relationships drawn between them. It is the \u0026ldquo;stand back and look at everything at once\u0026rdquo; companion to the per-layer dashboards.\nOpen it from the 3D Infra pill in the topbar, or go directly to /3d/map. The map runs as a standalone full-screen view — no sidebar, no topbar, no global time picker — so the scene gets the whole viewport. The SkyWalking mark sits at the bottom-left; the × at the top-right returns you to the rest of Horizon.\nTiers A tier is a horizontal plane in the stack that groups related SkyWalking layers by their role in the system. Tiers are the spine of the map: they read top-to-bottom the way a request flows, from the apps a user touches down to the platform everything runs on.\nHorizon ships four bundled tiers:\nTier What lives here Examples Apps (top) The application surfaces and their direct dependencies as the app sees them General (agent) services, Browser/RUM, iOS, mini-programs, and the Virtual* targets (database / cache / MQ / gateway / GenAI) Middleware The data and messaging services, gateways, and self-observability MySQL, PostgreSQL, Redis, MongoDB, Elasticsearch, Kafka, RocketMQ, RabbitMQ, Pulsar, APISIX, Nginx, Kong, Flink, the SkyWalking SO11Y components, and cloud-managed data services Service Mesh The mesh that fronts the apps Istio managed services, Istio data plane (Envoy sidecars), Istio control plane, Cilium, Envoy AI Gateway Infra (bottom) The platform the rest runs on Kubernetes cluster + service, Linux/Windows hosts, virtual machines, EKS Every layer OAP reports is placed onto exactly one tier. A layer that Horizon hasn\u0026rsquo;t classified yet (for example a brand-new OAP layer) lands on the Middleware tier with an \u0026ldquo;unclassified\u0026rdquo; mark so an operator notices it and can re-assign it.\nThe tier list on the right-hand panel mirrors this stack. Click a tier row to fly the camera to it; use the eye toggle to show or hide every layer in that tier at once. The row also shows how many of the tier\u0026rsquo;s services are currently visible.\nReading the map Cubes Each cube is one service. Cubes are grouped into their layer\u0026rsquo;s zone on the tier, and each zone is colored with the layer\u0026rsquo;s brand color and stamped with the project\u0026rsquo;s logo (Istio\u0026rsquo;s sail, the Kubernetes helm wheel, a database cylinder, a queue, and so on) so you can identify a zone at a glance from any camera angle.\nLayers that ship a topology (General, Service Mesh, Kubernetes Service, Cilium) lay their cubes out by call dependency — upstream callers on one side, downstream services on the other — like the 2D service map. Layers without a topology pack their cubes into a tidy grid.\nTraffic A small pill under a cube shows that service\u0026rsquo;s live traffic — requests per minute for app and mesh services, queries or operations per second for data services, and so on, each with its own unit. The number is the service\u0026rsquo;s headline throughput metric for the current window.\nTraffic pills appear on cubes that are close enough to read; zoom out far enough and they fade away to keep the scene clean, then return as you zoom back in. A selected cube always shows its number.\nAlarms When a service has an alarm in the last 20 minutes, a small red beacon pulses on the top corner of its cube. The cube keeps its layer color — the beacon is the alert signal, so you can still tell which layer a troubled service belongs to. The alarm feed refreshes on its own while the map is open.\nConnections The map draws three kinds of lines:\nIn-layer calls — light cyan tubes between two services in the same layer, with animated packets flowing along them. This is each layer\u0026rsquo;s internal call graph. Cross-layer calls — soft orange arrows between services in different layers on the same tier (for example Browser → Frontend, or Frontend → Virtual Database). The arrow points from caller to callee. Hierarchy links — thicker gray tubes that connect the different views of the same logical service across tiers (for example a service seen by its agent, by the mesh, and as a Kubernetes service). These represent identity, not traffic, so they only appear when you select a cube, and show just that cube\u0026rsquo;s relatives — then disappear when you deselect. Interacting Camera — drag to rotate, scroll to zoom, and the on-screen toolbar (top-left) gives the same gestures as buttons. Arrow keys or WASD pan the view; hold Shift for a bigger step. Select a service — click a cube. It highlights, a detail card appears beside it (service name, layer, and an Open dashboard button that jumps to that service\u0026rsquo;s layer dashboard in a new tab), and its cross-tier hierarchy links light up. Click empty space, click another cube, or press Esc to deselect. Hover — hovering a cube shows a quick tooltip with the service\u0026rsquo;s name and layer next to it. Loading timeline Because a full deployment is too much to fetch in one request, the map loads in stages, and a slim timeline strip at the bottom shows the progress live:\nServices — the service roster and which layers they belong to. Templates — which layers carry a topology. Topologies — each topology-bearing layer\u0026rsquo;s call graph. Hierarchy — the cross-tier identity links between the different views of the same service. Only services that are new since the last run are fetched; the rest are reused, so a steady deployment costs nothing here on refresh. Layout — placing the cubes. Metrics — the per-service traffic numbers, fetched in batches so the cubes light up progressively. Each step shows its status as the map builds; click a step to open a drawer with its detail (services added/removed since last run, per-layer topology results, metric progress, and so on). A refresh button on the strip re-runs the whole sequence.\nConfiguration What the map shows is driven by a single configuration that an administrator edits in the UI at /admin/3d-map (linked under Dashboard setup in the sidebar). It is a structured editor — you work with tiers, layers, colors, and metrics through form controls, not raw JSON. Horizon ships a bundled default, seeded into OAP at first boot so the map is useful out of the box; your edits are kept as a local draft in your browser, and Check diff \u0026amp; push publishes them to OAP — the copy the map renders. In the default live template mode that OAP copy is the only source: if the template store cannot be read, the map reports that instead of rendering the bundled default. See Configuration File → Template source mode.\nFrom the editor you can:\nFilter layers — one global layer filter, written as a regex. A layer it excludes is dropped from the map entirely. This is the only filter; everything it admits is then placed on a tier. Arrange tiers — rename tiers, reorder them top-to-bottom, and pin each layer to a tier. A layer you don\u0026rsquo;t pin lands on the failover tier you nominate, so nothing silently falls off the map. Group layers — cluster several related layers (for example the SkyWalking self-observability components) into one labelled block on a tier, while each member keeps its own cube color. Color layers — pick each layer\u0026rsquo;s brand color (used for the cube, zone, and stamp). Choose a traffic metric — for each layer, set the single throughput metric its cubes display: the MQE expression, a display label, and a unit. The bundled defaults are seeded from each layer\u0026rsquo;s dashboard template, so most layers show a sensible number out of the box. A read-only Service-map layers list shows which layers lay their cubes out as a call graph — that comes from each layer\u0026rsquo;s template (its service-map capability), not from this page.\nPushed changes take effect the next time the map is opened. A Reset action reloads either the shipped bundled default or OAP\u0026rsquo;s current version, so you can start over before saving.\nExport downloads the map\u0026rsquo;s in-use configuration — the version live on OAP, or the bundled default when OAP has none — as a JSON file, for backup, sharing, or moving it to another OAP. Import reads a configuration JSON file and loads it as a local draft; preview it, then Check diff \u0026amp; push to publish. Import never writes OAP directly, and a file that isn\u0026rsquo;t a valid 3D-map configuration is rejected with a message.\nTuning the metric fan-out The map\u0026rsquo;s loading stages run in batches, several requests at once. How aggressively they do this is governed by the performance.bulk.infra3d block in horizon.yaml — an operator setting, not part of the map configuration, so it is not in the structured editor and does not travel with an exported / imported map. Edit horizon.yaml; the change is hot-reloaded and takes effect the next time the map is opened:\nmetricConcurrency — how many metric batches load at the same time. Default 4, range 1–8. Raise it to fill the cubes faster on a large deployment when OAP has headroom; lower it (toward 1) if a busy OAP rejects or slows the burst of metric requests during the Metrics step. metricBulkSize — how many services share one metric request. Default 6, range 1–12. Larger means fewer requests, but OAP rejects an oversized request, so this is capped — leave it at the default unless you have a reason to change it. topologyConcurrency — how many layer call-graphs load at once during the Topologies step. Default 4, range 1–16. templateConcurrency — how many layer templates load at once during the Templates step. Default 8, range 1–32. The defaults are tuned for a typical deployment; only revisit these if the loading timeline stalls on the Metrics, Topologies, or Templates step, or if OAP returns errors under the load.\nViewing the map needs read access (infra-3d:read, held by the built-in viewer role and above). A role without it does not get the topbar entry to the map at all. Editing and publishing the configuration needs overview:write (operators and admins by default). See Roles and Permissions.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/operate/infra-3d-map/","title":"\u003c!--"},{"body":" Live Debugger Path: /operate/live-debug.\nThe Live Debugger captures, step by step, how a single analysis rule processes real data inside the connected OAP — so you can see why a metric comes out the way it does (or why it comes out empty) without reading the backend logs. You pick one rule, start a capture session, and OAP records each pipeline stage (input → filter → function → output) for a bounded number of executions. The capture runs on every reachable OAP node at once, so a rule that behaves differently on one node in a cluster is visible side by side. When you are done, you stop the session; captures are also saved locally so you can re-open them later.\nThis is a diagnostic surface for the same DSL families you edit on the Runtime Rules (DSL) page — it does not change any rule. Starting a session never alters collection; it attaches a recorder to the rule for the length of the session and detaches when you stop it or the retention window lapses.\nThe three DSL tabs The page is split into three tabs, one per DSL family. Each tab runs its own independent session, so you can have a MAL, a LAL, and an OAL capture going at the same time.\nTab DSL family What it debugs MAL Meter Analysis Language otel-rules, log-mal-rules, telegraf-rules, meter-analyzer-config — the meter pipeline for OTEL, log-derived, Telegraf, and agent-reported metrics. LAL Log Analysis Language lal — log parsing and extraction, capturable at block or statement granularity. OAL Observability Analysis Language the connected OAP\u0026rsquo;s OAL clauses — input source columns through aggregation and output. OAL rules are not runtime-editable (they are compiled into the OAP build), but they are still debuggable here — this is the one place you can watch an OAL clause execute against live source data.\nRunning a capture Pick a rule. For MAL, choose a rule file and then a specific metric inside it; for LAL choose the log rule (and block / statement granularity); for OAL choose the source and clause. Set the bounds. recordCap limits how many executions are captured (default and maximum 100). retention (min) is how long the session stays alive on OAP before it is reaped (default 5 minutes, maximum 60). Start. OAP installs the recorder across the cluster and begins collecting. The state pill moves through starting → capturing → captured. While capturing, the view refreshes about once a second; it stops polling on its own once every reachable node has finished (captured). Stop at any time to detach the recorder early. You do not have to wait for the retention window. Starting a new session for a rule that already has one running automatically replaces the prior session — the coverage strip notes how many prior sessions were stopped.\nCluster coverage strip Above the captured records, a per-node strip shows, for each OAP node, an install result (whether the recorder was accepted on that node) and a collect status (whether data came back). A rollup line summarizes how many nodes the session is live on (e.g. live on 2 of 3 nodes). Use it to spot a node that rejected the install or was unreachable — a missing node there explains a partial capture.\nReading the captured stages Each captured execution is shown as a chain of stages. Every stage reports an in → out count, so a stage that drops everything (a filter that matched nothing) is obvious at a glance. Clicking a stage highlights the matching fragment of the rule\u0026rsquo;s source text above the chain, tying the captured step back to the line of DSL that produced it.\nDiff-default label grouping When a stage emits many samples that share a metric name, they are grouped under a one-line summary rather than listed in full. Expanding a multi-sample group lands in diff mode by default: the labels that are identical across every sample collapse into a shared context shown once, and each sample row shows only the labels that differ. This makes \u0026ldquo;what distinguishes these series\u0026rdquo; the thing you see first. A toggle switches to the full per-sample label list when you want every label on every row. The same diff-first treatment applies to a run of output entities that share a metric — only the entity fields that vary are shown per row.\nVery large groups render a capped number of detail rows with a \u0026ldquo;+ N more\u0026rdquo; note; the summary count is always exact.\nThe LAL pipeline matrix A LAL capture renders as a grid — one column per captured record, one row per pipeline step (input, the per-statement or per-block function steps, output). The first column names each step and stays pinned as you scroll sideways through the records; each cell holds that record\u0026rsquo;s data at that step.\nIt reads any log format. A cell shows whatever fields OAP serialized for the record — a plain LogData input shows service / endpoint / tags / body, while an Envoy access-log (ALS) record shows its built snapshot (service, endpoint, response data, and the access-log content as JSON). When OAP cannot serialize a record\u0026rsquo;s raw input, the cell shows the reason (for example jsonformat-failed …) instead of rendering blank, and a small label names each cell\u0026rsquo;s payload class.\nFilter a row to the records that have data. A step row that has gaps carries a filter; turning it on narrows the grid to just the records that produced data for that step — for example the output row to only the records that emitted output (an abnormal-only rule aborts most records, so only a few reach output). The row count shows how many of all captured records reached that step.\nInspect and diff a cell. Each cell has a button — VIEW on the input row, DIFF on the builder rows — that opens the cell\u0026rsquo;s complete payload in a JSON viewer with the log content shown as formatted JSON. For the built-log snapshots you can compare stages: a picker presents the captured rule with each per-statement step on its line and the extractor / sink blocks as selectable ranges, and choosing one shows a side-by-side diff of the two snapshots — the quickest way to see which statement or stage added, changed, or dropped a field.\nEach OAP node renders its own matrix; filtering or selecting in one node\u0026rsquo;s grid does not affect another\u0026rsquo;s.\nCapture history Every session you run is saved to capture history, browse it at /operate/live-debug/history (or the history link on each tab). History is stored locally in your browser — it is not shared between users or machines and survives reloads, with the most recent captures kept per DSL family.\nFrom history you can:\nReplay a finished capture — re-open the recorded stages exactly as they were captured, without re-running anything on OAP. A banner marks that you are viewing a saved capture, with a back to live control to return. Resume a capture whose retention window has not yet lapsed — re-attach to the still-live OAP session and continue polling it. A capture that was archived before its first poll returned data shows as having no records; run a longer-lived capture to give the pipeline time to fire.\nRequirements The OAP dsl-debugging module must be loaded. This is the module that powers start / poll / stop across MAL / LAL / OAL; the page shows a warning banner when it is missing. See Required OAP Modules. The receiver-runtime-rule module must also be loaded — it backs the rule picker (the catalog of rules you choose from). It is a separate module from dsl-debugging: a deployment can have one without the other, in which case either the picker or the capture itself will be unavailable. OAP admin port reachable from Horizon. Access control Permission Grants live-debug:read View the Live Debugger, the active-session list, cluster status, and capture history. Nothing else is required to watch a capture. live-debug:write Start and stop capture sessions. Nothing else is required to run one — no rule:* grant takes part, and holding every rule verb without live-debug:* gets you nothing here. In the bundled roles, both are held by operator (and admin). A read-only viewer can be granted live-debug:read on its own to inspect existing sessions and history without being able to start new captures. See Roles and Permissions.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/operate/live-debugger/","title":"\u003c!--"},{"body":" Log Inspect Log Inspect (/operate/log-inspect) is the cross-layer log query tool in the sidebar. The per-layer Logs tab starts from a layer and the service picked in its header; this page starts from nothing. Open it, name any service — or no service at all — and query across everything the log store holds. It unifies three log sources on one page: the stored log stream, browser JavaScript errors, and on-demand Kubernetes pod tails.\nLike the other triage pages, it owns its own time range — the global topbar time picker does not apply — and (for the stored sources) nothing is fetched until you press Run query. Conditions are staged; switching source clears the previous result so streams never mix.\nSources The Source toggle at the top picks what kind of logs you are after:\nRaw — the logs SkyWalking has collected and stored: the same store as the per-layer Logs tab, queried across every layer. Browser — the JavaScript errors reported by the browser agent, with inline source-map management and per-row stack de-obfuscation. Kubernetes Pod logs — a live tail of one pod\u0026rsquo;s container output, pulled through OAP on demand and never persisted. Target — pick it, type it, or leave it blank For the Raw and Browser sources the Target is optional: blank queries every service in the window. Two modes scope it:\nPick — choose a Layer, then a Service from its catalog, then optionally an Instance and/or Endpoint. On the Browser source these last two are labelled Version and Page, because that is what a browser app\u0026rsquo;s instances and endpoints are. Type — enter a Service name directly, with a Real checkbox (off for a virtual/peer service), plus optional instance/endpoint (version/page) names. Typing needs no layer. The → edit as text link converts the current Pick selection into the Type form. Raw and Browser share one target, so switching between them keeps your pick; only crossing into or out of the pods source resets it.\nRaw — stored logs across layers Conditions for the stored stream:\nCondition What it does Tags Comma-separated key=value pairs, AND-joined, with autocomplete: type to see known keys, type = for that key\u0026rsquo;s known values, Enter commits the pair and primes a comma for the next. Filter by level with a level=… tag. Trace ID Show only the lines correlated with one trace. Time A rolling preset (15m, 30m, 1h, 3h, 6h, 12h, 24h) or Custom… with an absolute start/end pair. Second-precision, like the per-layer tab. Limit Result cap: 20, 50 (default), 100, or 200. The server additionally caps a single batch at its configured page-size limit (100 by default), so 200 only takes effect when that limit has been raised — see the query-limits section of the horizon.yaml reference. Run query fetches one batch of the newest matching lines. Rows render exactly as on the per-layer tab — timestamp, level, service, an ↗ trace link when trace-correlated, a format chip, and a one-line preview — and clicking a row opens the same full-payload popout: format-aware pretty-printing, Copy, the service/instance/endpoint/trace context, and the tag table. The ↗ trace links open the related trace\u0026rsquo;s waterfall in an overlay without leaving the page. Escape or the backdrop closes the popout, and a re-run that no longer contains the open row closes it too.\nUnlike the per-layer Logs tab there is no density histogram, no Levels strip, and no pager — this page returns a single batch capped by Limit. It trades the browsing chrome for reach: any service, any layer, or all of them at once.\nBrowser — JS errors with source-map resolve The Browser source queries the errors browser agents report, across every browser app at once if you leave the target blank. Its conditions are Category (All, or one of AJAX, RESOURCE, VUE, PROMISE, JS, UNKNOWN) plus the shared Time and Limit.\nBelow the conditions sits the source-map manager — the same map store the per-layer Browser Logs tab uses, managed inline so you never have to leave the page to make a stack readable. It lists the maps currently available (statically mounted ones and temporary uploads), shows the memory-usage bar, and offers Upload .map and per-upload remove. Uploaded maps live in server memory only; mounted maps cannot be removed here. If de-obfuscation is disabled on the server, the manager says so instead.\nResults render as a dense error list: time, category (color-keyed), page, app version, and the message. Click a row to open the browser-error popout — the error\u0026rsquo;s metadata and raw stack on one side, and the de-obfuscation control on the other: pick a hosted map (the first one is pre-selected), press Resolve, and read the original file/line/symbol frames with source snippets. Which map matches which build is your call — see Browser Logs \u0026amp; Source Maps for the matching rules and the resolvable categories.\nKubernetes Pod logs — live tails without entering a layer The pods source is the cross-layer twin of the per-layer Pod Logs tab: it tails one pod\u0026rsquo;s container output straight from the cluster through OAP. Nothing is persisted — each poll pulls the trailing window and discards it — so the pod must be currently running.\nUnlike the other two sources, the target here is required: a specific pod and container.\nPick a Layer and a Service (with exactly one Kubernetes-aware layer in your menu, the layer is pre-selected) — or switch the service field to Type and enter the service name directly, no layer needed. Pick the Pod — the service instance. A single-pod service is auto-selected. Pick the Container — the pod\u0026rsquo;s containers are listed and the first is auto-selected. Choose the trailing Window (Last 30s to Last 30m) and the poll Interval (2s–30s). Press Start to tail live, Pause to stop, or Refresh for a one-shot fetch (which also pauses a running tail). Include / Exclude chip fields narrow the lines: type a full-line regular expression (for example .*error.*) and press Enter to add it; the × removes a chip. Includes keep matching lines, excludes drop them, and changing them mid-tail re-runs with the new filters. Re-targeting the pod, container, or service stops the tail so a stale loop never bleeds across pods.\nOn-demand pod logs are disabled by default on OAP; when the feature is off or the pod no longer exists, the reason appears in a banner — see the pod-logs troubleshooting on the Logs page, which applies here unchanged.\nResolved query For the Raw and Browser sources, a Resolved query toggle appears after each run: it names the source and expands to the exact condition that was sent — resolved service ids, computed window, filled-in defaults. When a query returns something unexpected, read it first. Pod tails are live fetches rather than stored-store queries, so they have no resolved-query panel.\nPermissions The page and the raw/browser queries require the inspect:read permission. The tag autocomplete, the container list, and the pod tail additionally use logs:read; the source-map list and stack resolve use browser-errors:read (uploading or removing maps needs source-map:write); and the Pick-mode layer/service dropdowns use metrics:read. The bundled roles that grant inspect:read include the read verbs. See Roles and Permissions.\nRelated Logs — the per-layer stored-log stream and Pod Logs tab, with the full condition and troubleshooting reference. Browser Logs \u0026amp; Source Maps — source-map matching rules, static provisioning, and which error categories resolve. Trace Inspect — the cross-layer sibling for traces. Metrics Inspect — the cross-layer view of OAP\u0026rsquo;s metric catalog. ","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/operate/log-inspect/","title":"\u003c!--"},{"body":" Logs Horizon surfaces logs through two distinct tabs, each backed by a different OAP source.\nThe Logs tab queries the logs SkyWalking has collected and stored — application and service log records, indexed and filterable, correlated with traces. The Pod Logs tab does something different: it live-tails a Kubernetes pod\u0026rsquo;s container logs on demand, pulled straight from the Kubernetes API through OAP and never persisted. They appear as separate tabs because they answer different questions — \u0026ldquo;what did this service log over the last half hour?\u0026rdquo; versus \u0026ldquo;what is this pod printing to stdout right now?\u0026rdquo;.\nWhich tabs a layer shows depends on the layer template. The Logs tab appears on layers whose template enables it (for example GENERAL, MESH, MESH_DP, NGINX, ENVOY_AI_GATEWAY, the mini-program and mobile layers). The Pod Logs tab appears only on the Kubernetes-aware layers K8S_SERVICE, MESH, and MESH_DP.\nFor browser JavaScript errors reported by the browser agent — a separate stream with its own source-map de-obfuscation — see Browser Logs \u0026amp; Source Maps. That is not the same as the collected service logs described here.\nFor cross-layer digs — querying any service\u0026rsquo;s stored logs by name (or all services at once), browser errors, or a pod tail without entering a layer — see Log Inspect.\nStored logs Open a layer that has a Logs tab and pick a service in the header. The stored log stream loads for that service over the page\u0026rsquo;s own time range, newest first.\nScoping and filtering The conditions bar narrows the stream. Every filter is optional; together they are AND-joined.\nInstance — restrict to one service instance. The default is All. On a sidecar layer this picker is labelled Sidecar.\nEndpoint — restrict to one endpoint. Type to search the endpoint list, then click a result to pin it; the × clears it back to All.\nTrace ID — paste a trace id to show only the log lines correlated with that trace. Copy the id from a trace\u0026rsquo;s span detail and paste it here; there is no one-click jump from a trace to its logs.\nContent — words the log line must contain, space-separated for AND (timeout db matches only lines carrying both). This field appears only when your storage backend can search log content — ElasticSearch can, BanyanDB and the others cannot, and Horizon asks the connected OAP which it is. On a backend that cannot, the field is absent rather than present-and-ignored, because OAP accepts the condition there and returns the unfiltered stream — which reads as \u0026ldquo;everything matched\u0026rdquo;.\nTags — a single key=value field with autocomplete. Start typing a key to see suggested keys; type = to switch the suggestions to known values for that key; press Enter to commit the tag. Committed tags show as removable chips under the bar and ride along on the query as additional filters.\nLevel — the Levels strip above the stream doubles as a filter. Click error, warn, info, or debug to show only that level; click again to clear. The level filter is sent to OAP as a level tag, so pagination and counts reflect the filtered set. The other chip (lines whose level tag is missing or unrecognized) is informational only — it has no server-side value to filter on, so it is not clickable.\nThe stream queries on demand, not on every keystroke. Editing a condition stages it; nothing is fetched until you press Run query, which runs the query and resets to the first page. A freshly opened tab shows a Pick your conditions, then click Run query prompt rather than auto-loading, and switching service resets to that prompt — clearing the level and tag filters — so the previous service\u0026rsquo;s logs never linger under the new one. Paging and the page-size picker fetch immediately once you have run a query.\nTime range The Logs tab owns its own time range — the global topbar time picker is paused while you are here, so auto-refresh won\u0026rsquo;t shift the window mid-investigation. Pick a rolling preset (Last 15 min through Last 24 hours, default Last 30 min) or choose Custom… to pin an absolute start/end with two date-time inputs.\nLog queries use second-precision time windows. Logs are record-style data anchored at second granularity, so the window is not rounded to the minute — the most recent (and usually most interesting) lines are never chopped off. The window is capped at 7 days. A custom range longer than that is refused on the page, with the reason under the control, rather than being quietly shortened — a query made directly against the API is trimmed to the most recent week instead, so it still answers with the part that matters.\nReading the stream A density histogram sits above the stream: time on the x-axis, log count on the y-axis, each bar stacked by level (error / warn / info / debug / other) with the same colour as the legend. Hover a bar to see that bucket\u0026rsquo;s time range and per-level counts. The histogram is built from the currently loaded page, so it shows the shape of what is on screen, not the whole window.\nThe Levels strip carries a count per level next to each chip. Those counts come from a window-scoped sample (a few hundred of the most recent rows in the window, larger than one page), so they reflect the window\u0026rsquo;s level distribution rather than only the visible page. The strip notes the sample size it used, and says when the window held more rows than the sample counted — narrow the window if you need the counts to cover all of it.\nEach row shows the timestamp, the level, the service (with any group prefix decoded), an ↗ trace link when the line is trace-correlated, a format chip (JSON / YAML / TEXT), and a one-line preview of the content. Rows are colour-keyed by level.\nHorizon renders the payload according to its content. OAP labels payloads as JSON or plain text; on top of that, Horizon sniffs for JSON and YAML structure so an unlabelled-but-structured body still gets the right treatment. JSON is compacted to a single line in the preview and pretty-printed in the detail view; YAML keeps its keys; plain text is whitespace-collapsed.\nClick a row to open the full payload in a popout: the complete content, format-aware pretty-printing, a Copy button, the service / instance / endpoint / trace context, and a table of all tags on the line. If the line is trace-correlated, an ↗ trace button there (and the ↗ trace link on the row) opens the related trace\u0026rsquo;s waterfall in an overlay without leaving the log stream — the row\u0026rsquo;s timestamp is passed along so the trace is found even when it sits in a colder storage tier. Press Escape or click the backdrop to close.\nThe pager at the foot shows the current page and the row count on it; Prev / Next walk the pages, and the page size (20, 50, or 100) is set on the conditions bar. There is no \u0026ldquo;N of M\u0026rdquo; total, because the log query does not report one — Next is offered only when there really is another page with rows on it, so a full last page ends the walk instead of stepping onto an empty screen. Changing the page size restarts at page 1.\nTroubleshooting stored logs No rows returned. Confirm the service actually ships logs to OAP, that the storage backend has the logs module enabled, and that the time range covers when the logs were produced. Narrow filters (a tag, a level, an endpoint) can also empty the result — clear them and widen the window.\nA filter empties the stream. Tag and level filters are exact-match on indexed dimensions. A level value or tag value that doesn\u0026rsquo;t exist in the stored data returns nothing; check the value against what the Levels counts and the tag autocomplete actually offer.\nRun query is greyed out. The tab does not yet know which service to read, and says which case it is: Resolving service… while the picked service is being looked up, or a note that the selected service is not in this layer (it aged out of OAP, was renamed, or the link points elsewhere) — pick another one. The stream is always read for one service, so the tab waits instead of querying the whole layer.\nPod logs The Pod Logs tab tails a Kubernetes pod\u0026rsquo;s container logs live. There is no stored history to page through — each refresh pulls the trailing window straight from the Kubernetes API through OAP, shows it, and discards it. Nothing is persisted.\nStarting a tail Pick a service in the header, then pick a Pod (a service instance) — the page is pinned to one pod at a time. Pick a Container. Horizon lists the pod\u0026rsquo;s containers and auto-selects the first; switch if the pod runs more than one. Choose the look-back Window (Last 30s, 1m, 5m, 15m, or 30m) — how far back each poll reaches. Choose the poll Interval (2s, 5s, 10s, or 30s) — how often the window is re-fetched while live. Press Start. The trailing window streams into a read-only viewer and re-polls on the interval until you press Pause. The header shows a live indicator, the line count, and how long ago the view last updated. Changing the container, window, interval, or filters while tailing re-runs the query with the new settings. The viewer is read-only and keeps the newest line in view as fresh logs arrive.\nInclude and exclude filters Two filter rows narrow what the tail shows. Include keeps only lines that match; Exclude drops lines that match. Type an expression and press Enter to add it as a chip; the × on a chip removes it. Both are evaluated by OAP as full-line regular expressions (for example .*error.*), so they match against the whole log line, not a substring. Multiple expressions in a row stack as additional conditions.\nTime precision Pod-log windows are second-precision — this is a live tail, anchored at the current second. OAP caps a single tail window at 30 minutes; the longest selectable window is Last 30m.\nTroubleshooting pod logs On-demand pod logs are disabled by default on OAP because container logs can leak secrets. When the feature is off, or when the pod can\u0026rsquo;t be resolved, OAP returns a reason instead of data and Horizon shows it in a banner rather than an empty pane. Two common cases:\n\u0026ldquo;Logs unavailable\u0026rdquo; with a reason. If the reason indicates the feature is off, enable on-demand pod logs on the OAP side. If it indicates the pod wasn\u0026rsquo;t found, the instance you picked points at a pod that no longer exists (a finished rollout or a scaled-down replica) — pick a currently-running pod.\nThe tail stops on its own. A pod that vanishes mid-tail (a rollout or scale-down) makes the next poll fail; Horizon stops the loop and surfaces the reason rather than spinning on errors. Re-pick a live pod and Start again.\nPermissions Both tabs — stored log queries, tag autocomplete, the container list, and the on-demand tail — require the logs:read permission. See Roles and Permissions.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/operate/logs/","title":"\u003c!--"},{"body":" Profiling Profiling drills past metrics and traces into the call stacks, kernel events, and process-to-process conversations of a running service. Horizon surfaces SkyWalking\u0026rsquo;s profiling capabilities as a set of per-layer tabs on the service you have selected: Trace Profiling, eBPF Profiling, Async Profiling, Network Profiling, and pprof. Each profiling tab only appears on a layer when OAP reports that the service supports that kind of profiling, so the tabs you see depend on the agent and platform behind the service.\nEvery profiling tab follows the same shape: a task list on the left, a New Task control to start a profiling run, and a result panel on the right that renders the captured data once OAP has fanned the task out to the relevant instances or processes. Results are shown as an indented stack tree or a flame graph, with a toggle between the two where both apply.\nTask creation is consistent across every tab. The New Task control opens once you have selected a service; for Network Profiling, the target instance is picked inside the dialog. Inside the dialog, a target that cannot be profiled at all — no profilable processes for eBPF, or no instances on the service — disables Create with the reason shown next to it, rather than a silently greyed-out control; advisory checks (such as Network Profiling\u0026rsquo;s process list) warn without blocking. You always see why a task cannot be started.\nAccess control Profiling is gated by two distinct permissions:\nprofile:enable — required to start a profiling task (the New Task control). It is held by the operator role and above.\nprofile:read — required to view profiling results. It is part of the read-only data catalog held by viewer, maintainer, and operator.\nA viewer can therefore open a profiling tab and inspect existing results, but cannot create new tasks. See Roles and Permissions for the full permission catalog.\nTrace Profiling Trace Profiling samples the call stacks of slow trace segments. You start a task scoped to a service (and optionally a single endpoint), and the agent dumps CPU stacks from segments that exceed the task\u0026rsquo;s threshold while the task is running.\nTo start a task, open the New Task dialog and set:\nEndpoint name — restrict sampling to one endpoint, or leave it as (any) to profile all endpoints on the service.\nStart when — begin immediately (now) or at a scheduled time.\nDuration — how long the task runs, in minutes.\nMin threshold (ms) — only segments slower than this are sampled.\nDump period (ms) — how often a stack snapshot is taken while a sampled request runs.\nMax sampling count — the cap on how many segments the task collects.\nOnce the task has collected sampled traces, pick a trace from the Sampled traces list to load its spans. Select a profiled span and press Analyze to build its call tree. The result renders as either a Tree (indented stack table) or a Flame graph. A Data mode toggle switches between Include children (the whole span\u0026rsquo;s time) and Exclude children (only the time spent in the span itself, with child-span windows subtracted). The eye icon on a task opens a detail panel with the task\u0026rsquo;s parameters and the per-instance operation log.\neBPF Profiling eBPF Profiling samples kernel-level stacks from a process without an in-process agent, driven by SkyWalking Rover. It supports two capture targets:\nON_CPU — where the process spends CPU time.\nOFF_CPU — where the process is blocked off CPU (waiting on locks, I/O, scheduling).\nA task targets a service and, optionally, a set of process labels (leave the labels empty to profile all processes). You choose the target, a start time, and a duration in minutes. Open the New Task dialog from the selected service; if OAP reports no profilable processes for it, the dialog says so and Create stays disabled.\nWhen you select a task, the result auto-analyzes. The filter bar lets you narrow the view:\nLabels — restrict the aggregation to the chosen process labels.\nAggregate — Count (number of stack samples) or Duration. Duration is only available on OFF_CPU tasks, since off-CPU samples carry a blocked-time duration that on-CPU samples do not.\nProcesses — pin specific processes from the capture; pinning re-runs the analysis immediately.\nThe result is shown as a Flame graph or a Tree, with a banner stating the wall-clock window the capture covers and how many schedules contributed.\nAsync Profiling Async Profiling runs the async-profiler against a live Java service, capturing JVM-level stacks without restarting the process. A task targets one or more service instances and one or more event types. The supported events are:\nCPU ALLOC LOCK WALL CTIMER ITIMER You can select multiple instances and multiple events in a single task, with a duration from 30 seconds up to 15 minutes. After the task runs, choose which instances to include and which event type\u0026rsquo;s tree to render, then press Analyze. Because a single task can collect several event types, the result panel has an Event type selector — switching it re-draws the flame graph for the selected JVM event (for example EXECUTION_SAMPLE for CPU/Wall/Timer events, LOCK for lock contention, or one of the object-allocation event types for ALLOC).\npprof pprof profiles a live Go service through the standard Go runtime profiler. Unlike Async Profiling, a pprof task captures exactly one event type, chosen from:\nCPU HEAP BLOCK GOROUTINE MUTEX ALLOCS THREADCREATE The dialog adapts to the event you pick:\nCPU, BLOCK, and MUTEX are time-bounded captures and require a Duration (up to 15 minutes).\nBLOCK and MUTEX additionally take a Dump period sampling rate — for BLOCK it is a blocked-nanoseconds rate, for MUTEX a contention-occurrences rate; a value of 1 samples every event. Because lower means more samples, an invalid value is rejected with the reason rather than silently replaced with a default.\nHEAP, GOROUTINE, ALLOCS, and THREADCREATE are one-shot snapshots — they take no duration and no sampling rate, capturing the current state at the moment the task fires.\nA task can target multiple Go service instances. After it runs, select the instances to include and press Analyze to render the single result tree as a flame graph.\nNetwork Profiling Network Profiling captures the network conversations between processes of a service instance and renders them as a process-level topology. It mounts on a specific instance, which you pick inside the New Task dialog. The dialog lists the rover-monitored processes that recently reported on that instance — as advice, not a gate: an instance with no recently-reported process shows a warning that the task may collect nothing, but you can still create it and let OAP decide. Once an instance is chosen, the task defines which traffic to sample.\nEach sampling rule scopes the capture — by URI pattern, by HTTP 4xx / 5xx responses, or by a minimum duration — and controls how much of each request and response body is collected. OAP runs every network task for a fixed ten minutes and the create request carries no duration, so the New Task dialog defines the sampling rules rather than a run length.\nThe result is a honeycomb topology: each cell is a process, and the edges between them are the observed inter-process calls. Selecting an edge opens a detail panel with that process-to-process relation\u0026rsquo;s metrics (call rate, latency, and bytes transferred) charted over the task\u0026rsquo;s run window. The topology that drives this layout is the same process-relation data that powers the 3D Infrastructure Map.\nContinuous Profiling Everything above starts a profiling task on demand — you pick a target and start it. Continuous profiling is the opposite: you arm a policy once, and the profiling task starts by itself whenever a process crosses a threshold, with nobody present. It is how you catch a problem that only appears at 3 a.m.\nContinuous profiling is eBPF profiling only, and it requires Rover. A policy can trigger ON_CPU, OFF_CPU or NETWORK — the same three flavours as the eBPF and Network Profiling tabs above — and the Rover agent both evaluates the thresholds and runs the resulting task. There is no continuous trace, async-profiler or pprof profiling; those stay on demand. So a service with no Rover agent can hold a saved policy, but nothing will fire until one is deployed.\nPolicies are edited on the layer\u0026rsquo;s Continuous Profiling tab, beside the eBPF and Network Profiling tabs whose tasks they trigger. The tab has its own Target service picker: each service is labelled with the targets it already has armed, and the picker can be filtered by that — including no policy, which is the set you want when arming services that are not set up yet. Opening the tab selects the first service that already has a policy, or the first service in the layer if none does. Once a service is selected, the tab shows its policy plus the instances OAP is currently evaluating it against.\nNothing here is gated on the agent already being present, because arming a policy before deploying the agent is a valid order of work: the policy is backend configuration, and it simply starts firing once an eBPF agent begins reporting. If no process of the selected service has reported eBPF-profiling support recently, the tab says so as a warning and still lets you save.\nThe tab appears on a layer whose template enables the Continuous Profiling component (Layer Setup). It ships enabled on MESH, matching where the previous SkyWalking UI placed it. Rover registers its processes into MESH, MESH_DP and K8S_SERVICE by default (which layer is configurable per discovery analyzer), so those are the layers where enabling it is likely to be useful — turn it on there if your Rover deployment reports into them.\nA policy is a set of targets — ON_CPU, OFF_CPU, or NETWORK — and each target carries one or more conditions. A condition is:\na measurement (labelled that way on screen; OAP\u0026rsquo;s own name for it is ContinuousProfilingMonitorType) — PROCESS_CPU, PROCESS_THREAD_COUNT, SYSTEM_LOAD, HTTP_ERROR_RATE, or HTTP_AVG_RESPONSE_TIME; a threshold, whose unit follows the measurement — a percentage for CPU and error rate, a thread count, a load average, milliseconds for response time. Every threshold is a whole number: OAP parses all five as integers and rejects anything else, so 0.5% or 4.5 will not save. CPU percent and HTTP error rate must be 1–100; the rest must be greater than 0. The count cannot exceed the period, and one target cannot carry two conditions of the same measurement. a period, the number of seconds of metrics to evaluate; a count, how many matching evaluations must occur before profiling is triggered. The two HTTP monitors can additionally be scoped to specific traffic. Choose All traffic, URI list or URI regex — one or the other, never both. Nothing on the backend rejects a rule carrying both, but the agent applies the list and silently ignores the regex, so the form makes the choice explicit; switching away from a filter you have filled asks before erasing it.\nTwo things are worth knowing before you save:\nSaving replaces the service\u0026rsquo;s whole policy. OAP stores one policy per service, and the page sends everything you see. A target you delete is deleted; keep every rule you want to survive.\nA policy only evaluates processes an eBPF agent reports. Inside each target sits a paged Where it runs panel: the instances and processes OAP evaluates for that target, with how often each has actually triggered profiling recently, searchable by instance or process name, each row expanding to that instance\u0026rsquo;s processes. That trigger count is the thing to read: it is the difference between a policy that is stored and one that is working, and it is the only per-target signal here (the process list itself is the same for every target). An empty panel means nothing is reporting for the service at all.\nThe panel is not a Rover presence check — it lists a process whether or not that process can be eBPF-profiled. If the panel has rows and the warning above says no process reported eBPF-profiling support, the reading is \u0026ldquo;processes are there, but none are profilable\u0026rdquo;, which points at Rover\u0026rsquo;s configuration rather than its absence.\nTasks a policy starts appear in the eBPF Profiling and Network Profiling tabs alongside the ones you start by hand, so a fired policy is read the same way as an on-demand task.\nReading policies needs profile:read; saving one needs profile:enable, the same permission as starting a task by hand — because that is what a policy eventually does.\nTroubleshooting A continuous-profiling policy never fires — first check that it is actually applied: each target shows Applied or Not applied, and rules that have only been typed are not running. Then check the Where it runs panel. If it is empty, nothing is reporting for that service and the thresholds are irrelevant; deploy Rover for the service. If processes are listed but the trigger count stays at zero, the threshold is not being crossed — lower it, lengthen the period, or reduce the required count.\nNo profiling tabs on a layer — OAP did not report profiling support for that service. Each tab requires the corresponding capability (trace, eBPF, async-profiler, network, or pprof), which depends on the agent or Rover deployment behind the service.\nNew Task is unavailable — you have not selected a service, or you lack profile:enable.\nCreate is disabled inside the New Task dialog — the chosen target cannot be profiled, and the reason is shown next to the button: for eBPF, OAP reports no profilable processes for the service; for Async Profiling, pprof, and Network Profiling, the service has no instances. On Network Profiling, an instance whose processes have not reported recently is a warning, not a block — the task can still be created.\nTask list is empty after creating a task — the task is created, but results only appear once OAP has dispatched it to the instances or processes and they report back. The view polls for the new task briefly; use the refresh control if it does not appear.\nAnalyze returns no data — the task ran but collected no samples in the selected window or scope. For Trace Profiling, confirm the threshold was low enough to sample real traffic; for eBPF and pprof, confirm the chosen processes or instances were live during the capture.\nRelated Roles and Permissions — profile:enable and profile:read.\n3D Infrastructure Map — the process and instance topology that the network view draws on.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/operate/profiling/","title":"\u003c!--"},{"body":" Service Map \u0026amp; Topology The per-layer Topology tab draws a layer\u0026rsquo;s services as an interactive, directed call graph: who calls whom, how hard each call lane is running, and how healthy each service is. It is the per-layer companion to the deployment-wide 3D Infrastructure Map — same call relationships, but flat, focused on one layer, and clickable down to the individual instance.\nAround the service map sit three related views that share the same canvas and reading conventions: the instance map drill-down (instance-to-instance traffic across a service-pair), the Deployment tab (instance-to-instance traffic inside one service), and the API dependency tab (the same graph drawn at the endpoint level). All four are driven by each layer\u0026rsquo;s dashboard template, so what they measure varies by layer — but how you read and narrow them is identical.\nWhich tabs you see These views are layer capabilities, not global pages — a layer shows only the tabs its template enables:\nTopology appears for any layer whose template declares a service map.\nDeployment appears only for layers that configure an intra-service instance graph (for example a clustered store whose nodes call each other).\nDependency (API dependency) appears only for layers that configure endpoint-level dependencies.\nA layer with none of these declared shows no topology tabs at all. The map always opens on the layer\u0026rsquo;s own service map; the instance map is reached by drilling into an edge, not from the sidebar.\nReading the service map The graph flows left to right. Entry traffic (the synthetic User node, and other callers with no upstream service) anchors the left edge; each downstream hop sits one column to the right. Within a column, the busiest services are stacked toward the top so the heavy lanes line up across columns.\nNodes Each circle is one service. Three visual channels carry its numbers, and all three come from the layer\u0026rsquo;s template — nothing is hardcoded, so the exact metric and unit differ per layer:\nThe number inside the circle is the service\u0026rsquo;s headline throughput (requests per minute for app-style layers, queries or operations per second for data layers, and so on), shown with its configured unit.\nThe colored ring around the circle is the health band. It maps a health metric (SLA, success rate, Apdex, error rate, …) onto a green → yellow → orange → red ramp. The legend under the map names the metric, prints the four break points, and states the reading direction — higher = better for SLA / success-rate / Apdex style metrics, lower = better for error-rate style metrics.\nA technology badge floats above the circle, picked from the service\u0026rsquo;s detected component (the database, cache, queue, gateway, or framework SkyWalking identified). A service whose component SkyWalking could not resolve shows a neutral badge.\nTwo node shapes are not real services: the User entry node, and conjectured peers — external or unresolved callees (an address like localhost:-1 or rcmd:80 that SkyWalking observed traffic to but has no agent on). Conjectured peers are drawn as a cloud-with-? and carry no metrics of their own; they exist on the map only to complete a call lane. Selecting one shows a virtual tag in its detail panel.\nEdges A line is a call relationship. Its thickness tracks the call rate on that lane — heavier line, more traffic. The flow animation along the line shows direction (caller → callee). Edges are not colored by health; the ring on the nodes carries that signal.\nClick a line to open its detail panel. Each line metric is shown twice — Client (as the caller measured it) and Server (as the callee measured it) — side by side, each with a sparkline over the window so you can see the trend, not just the latest number. A lane may report only one side: a call into a conjectured peer has no server-side numbers, a call out from the User node has no client-side numbers, and the panel labels those client only / server only rather than showing a blank.\nNode detail Selecting a node opens a panel with its template metrics, its Upstream list (services it calls) and Downstream list (services calling it), and two jumps: Open service (its layer dashboard) and API map → (its endpoint dependency graph). The node and edge panels are independent — you can keep both open at once.\nCross-layer hierarchy (Smartscape) One logical service is often observed by several layers at once — the same workload seen by its in-process agent (GENERAL), by its sidecar (MESH / MESH_DP), and as a Kubernetes service (K8S_SERVICE). When the service you have selected has such cross-layer counterparts, a small chip appears on the selected node\u0026rsquo;s edge; clicking it opens the hierarchy overlay.\nThe map dims but stays visible for spatial context, the selected service lights up in place with a FOCUS tag, and its counterparts in other layers fan out around it — one labeled, layer-colored lane per layer, request-near layers above the focus and infrastructure-near layers below, with counterparts in the same lane spread side by side. Each counterpart is named the way its own layer\u0026rsquo;s map would name it, and one that SkyWalking knows only from observed traffic carries a virtual tag. Auto-refresh is paused while the overlay is open, so nothing shifts under you.\nNavigation is deliberately two-step so scanning never jumps you away: click a counterpart once to select it, then click the Open in \u0026lt;layer\u0026gt; chip beside it to open that layer\u0026rsquo;s drill-down in a new browser tab with the service pre-selected. A counterpart whose layer has no active layer template in Horizon is dimmed and cannot be opened — the service exists on OAP, but there is no page to land on. Close the overlay with the ×, the Esc key, or a click on the dimmed background.\nThe chip only appears when OAP reports cross-layer counterparts for the selected service, and it is not offered in the embedded overview-widget map — open the full Topology tab.\nFocusing and narrowing the map By default the map seeds from every service in the layer — the full layer overview. That is the right starting point for a small layer and the wrong one for a large estate. Two controls narrow it:\nFocus — open the service picker (top-right of the Topology toolbar) and select one or more services. The map then redraws around just those services and their neighbors. The picker supports search and selecting a whole service group at once.\nDepth — once at least one service is focused, a depth control appears: 1 hop, 2 hops, or 3 hops. Depth is how many call hops out from the focused service the map walks. Depth has no effect on the full-layer overview (it already includes everything), so the control is hidden until you focus a service.\nAdditional controls on the canvas:\nFilter (top-left) hides nodes by layer, or hides the User node, so a busy graph reading from several layers can be thinned to the layers you care about. The filter stores what is hidden, so a service that only appears after a depth or time change starts out visible. Reset clears it.\nZoom / Fit (top-right) and drag-to-pan move the camera; double-click the canvas to fit the whole graph. Drag a node to reposition it; the layout holds your placement.\nThe map honors the topbar time picker — change the window and every node and edge metric re-reads for that range.\nInstance map (drill-down) When a service-to-service edge is selected, the edge panel offers Instance map →. This opens the instance-to-instance graph for that one service pair: the caller\u0026rsquo;s instances in the left column, the callee\u0026rsquo;s instances in the right, and the instance-level call relationships between them. It is the view for answering \u0026ldquo;which instance is the slow one\u0026rdquo; once the service map has pointed at the lane.\nThe instance map keeps two service pickers at the top so you can swap either side to an adjacent service without returning to the service map, a Service map back link, and the same client | server line-metric panel as the edge detail. A picker is shown only when there is a real choice — if a side has a single counterpart, its name is simply printed.\nDeployment (intra-service topology) The Deployment tab draws the instance-to-instance call graph within a single service — the nodes of a clustered service talking to each other (for example a distributed store\u0026rsquo;s members). It shows the full container inventory for the service, grouped by cluster or by role, with per-node metrics; call relationships are drawn as edges where SkyWalking reports them. A container that exists but has no intra-service call in the window (an idle sidecar, say) still appears on the map as an inventory node rather than being hidden.\nGrouping (by cluster, by node role / node type) comes from the layer template. When a layer reports no intra-service relations, the tab is a grouped inventory of the service\u0026rsquo;s containers with their metrics — no edges — which is the expected, by-design state for those layers, not an error.\nAPI dependency (endpoint graph) The Dependency tab is the service map drawn one level down, at the endpoint (API) level. Pick a service in the header, search its endpoints, and select one — the map then shows that endpoint\u0026rsquo;s upstream and downstream endpoint dependencies as a directed graph, with the same node metrics, edge sparklines, and pan / zoom / focus conventions as the service map.\nOne difference is inherent to the data: endpoint-relation metrics are recorded by the callee only. Edges therefore carry server-side numbers, and an endpoint with no resolvable metric values in the window is dropped from the graph rather than drawn empty.\nTwo safeguards to know The maps protect you from two failure modes that would otherwise read as \u0026ldquo;the data\u0026rdquo;.\n\u0026ldquo;Topology too large to render\u0026rdquo; A graph that grows past 5,000 services or 15,000 calls cannot be drawn legibly and risks overwhelming the browser, so the map declines to draw a partial picture. Instead it shows a notice with the actual counts and the remedy:\nTopology too large to render — N services · M calls. Pick a specific service above, or lower the depth, to see a complete map.\nThis is almost always the full-layer overview of a large estate. Focus one or a few services, and/or lower the depth, and the map renders. (Inside the embedded overview-widget snapshot, the same notice points you to open the full Topology tab to narrow the scope.)\nPartial metrics Node and edge metrics are fetched from OAP in batches. When some of those batches fail (an OAP hiccup, a backend limit), the map still draws the graph but flags that the gaps are unknown, not zero:\nSome metrics could not be loaded (X of Y batches failed) — blank values may be unavailable, not zero.\nThis matters operationally: a blank ring or an empty traffic number under this banner means \u0026ldquo;we could not read it this time\u0026rdquo;, and you should re-run before concluding a service is idle or down. On the API dependency map the same banner is phrased for its data shape — some endpoints or links may be missing, because an endpoint whose metrics failed to load is dropped rather than drawn empty. Refresh to retry.\nAccess Viewing any of these maps — service map, instance map, deployment, API dependency — requires the topology:read permission, which the built-in viewer role and above hold. See Roles and Permissions.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/operate/service-map/","title":"\u003c!--"},{"body":" Trace Inspect Trace Inspect (/operate/trace-inspect) is the cross-layer trace query tool in the sidebar. The per-layer Traces tab starts from a layer and the service picked in its header; this page starts from nothing. Open it, name any service — or no service at all — and run one query across everything the trace store holds. It is built for the deep-dive that does not begin inside a dashboard: a trace id pasted from a log line or an alarm, a service you only know by name, or a \u0026ldquo;show me every error trace in the last hour, anywhere\u0026rdquo; sweep.\nLike the other triage pages, it owns its own time range — the global topbar time picker does not apply — and nothing is fetched until you press Run query. Every condition is staged: edit as much as you like, then run.\nSources The Source toggle at the top switches between the two trace stores:\nNative — SkyWalking\u0026rsquo;s own trace store, the same store the per-layer Traces tab queries. Zipkin — the Zipkin store behind OAP, with Zipkin\u0026rsquo;s own service universe and query conditions. On the per-layer tab, which store you see is decided by the layer template; here both are always one click away and you choose per query. Switching source clears the previous result so the two never mix.\nTarget — pick it, type it, or leave it blank For native traces the Target is optional: leaving it blank queries every service in the window. When you do want to scope it, there are two modes:\nPick — choose a Layer, then a Service from that layer\u0026rsquo;s catalog, then optionally narrow to one Instance and/or Endpoint. This is the discovery path: the dropdowns show you what exists. Type — enter a Service name directly, with a Real checkbox (leave it on for a normal instrumented service; turn it off for a virtual/peer service such as a database or remote endpoint that only exists as a conjectured node). Instance and Endpoint names are optional free text. Typing needs no layer at all — the name plus the Real flag identify the service. The → edit as text link converts the current Pick selection into the Type form — pick to discover, then tweak the name or flag by hand.\nThe Zipkin target is different because Zipkin has its own service universe (no layers, no SkyWalking ids): a Service field (blank means all services), plus Remote service and Span name narrowing fields whose suggestions load once a service is picked.\nConditions Native conditions:\nCondition What it does Trace ID Paste a known trace id for a direct lookup. Status ALL, SUCCESS, or ERROR. Order Newest (by start time) or Slowest (by duration). Duration (ms) Min–max trace duration bounds, in milliseconds. Tags Comma-separated key=value pairs, AND-joined, with autocomplete (below). Time A rolling preset (15m, 30m, 1h, 3h, 6h, 12h, 24h) or Custom…, which swaps in an absolute start/end pair. The × returns to presets. Limit Result cap: 20, 30 (default), 50, or 100. Zipkin conditions are the store\u0026rsquo;s own: Duration (ms) bounds, an Annotation query (error or key=value terms, AND-joined), plus the shared Time and Limit. There is no Trace ID field on the Zipkin side.\nThe Tags field autocompletes from the tags actually stored in the window: start typing to see known keys, type = to switch the suggestions to that key\u0026rsquo;s known values, and press Enter to commit the pair — the field then primes a comma so you can keep typing the next one. Time windows are evaluated at second precision, same as the per-layer tab, so a trace that just finished still falls inside the window.\nRun query and the resolved query Run query executes the staged conditions and replaces the result area. Next to it, a Resolved query toggle appears after each run: it names the source (and, for native, which trace query API answered) and expands to the exact condition that was sent — the service ids resolved from your picks or typed names, the computed window, and every filled-in default. When a query returns something unexpected, read this panel first: it shows what was actually asked, not what you meant.\nDistribution chart Beside the conditions, a Distribution chart plots one dot per result — start time on the X axis, colored by success/error, with the duration surfaced on hover. Click a dot to open that trace, or drag a rectangle to brush a subset: the list below narrows to the brushed traces and shows an N / total count with a clear control. Brushing filters what is already loaded; it does not re-query.\nResults — segments or whole traces For native traces, a banner above the results states which trace query API this OAP serves: on backends with whole-trace support (Trace Query v2) full traces come back inline; on any other backend (Trace Query v1) each row is a trace segment and clicking one fetches its full trace. This is a property of the storage backend, not a setting — see Traces for the full explanation.\nClicking a row opens the same trace detail the per-layer tab uses: the span waterfall with its Default / Tree / Statistics layouts, per-span detail (meta, tags, logs, cross-trace refs, attached events), and the id / url copy buttons — a copied shareable URL reopens the trace in an overlay for whoever you send it to, on either store. While a trace is open, the result list folds into a collapsible rail on the left so you can step through traces without losing the query. Escape closes the span panel first, then the trace, in that order. Zipkin results render with the Zipkin waterfall and keep their Zipkin span shape.\nHow it differs from the per-layer Traces tab No layer, no header picker. The target is part of the query form, optional, and can be a typed name — including services in layers you never open, or all services at once. Both stores on one page. Native vs. Zipkin is a per-query toggle here; on the layer tab it is fixed by the layer template. Built for id-first triage. Paste a trace id with no service at all and run — the common \u0026ldquo;a log/alarm gave me an id\u0026rdquo; entry point. The waterfall, the distribution chart, the staged Run-query flow, and the v1/v2 behavior are identical to the per-layer tab — this page changes how you scope the query, not how results render.\nPermissions The page and its queries require the inspect:read permission. Opening a trace\u0026rsquo;s waterfall and the tag / Zipkin suggestion lists additionally use traces:read, and the Pick-mode layer/service dropdowns use metrics:read — the bundled roles that grant inspect:read include these. See Roles and Permissions.\nRelated Traces — the per-layer trace explorer, with the full waterfall and condition reference. Log Inspect — the cross-layer sibling for logs, browser errors, and pod tails. Metrics Inspect — the cross-layer view of OAP\u0026rsquo;s metric catalog. ","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/operate/trace-inspect/","title":"\u003c!--"},{"body":" Traces The Traces tab is the distributed-trace explorer inside a layer. You pick a service, set conditions (status, sort, duration, tags, time window), run the query, then click a result to read its span timeline. It surfaces two trace stores — SkyWalking-native traces and Zipkin traces — depending on what the layer is configured for.\nTraces are triage data, so this tab owns its own time range and conditions. It is not driven by the global topbar time picker, and it does not auto-refresh: you set your conditions and press Run query. Nothing is fetched until you do — until then the list shows a \u0026ldquo;Pick your conditions, then click Run query.\u0026rdquo; prompt.\nWhich trace store appears A layer template carries a traces.source setting that decides which trace store the tab queries:\nnative (the default when a layer has no traces block) — only the SkyWalking-native trace explorer. zipkin — only the Zipkin trace explorer. both — two separate sidebar tabs, Trace (native) and Zipkin Trace. Native and Zipkin spans have different shapes and different query conditions, so they are kept as distinct tabs rather than one tab with a toggle. Mesh and Kubernetes-flavored layers commonly land on Zipkin; instrumented-agent layers land on native.\nNative traces The native explorer queries SkyWalking\u0026rsquo;s own trace store. The service is taken from the layer\u0026rsquo;s Service header picker at the top of the page; the in-tab conditions narrow within that service.\nConditions All conditions are staged in the toolbar and only take effect on Run query — editing a field does not refetch on its own.\nCondition What it does Instance Restrict to one service instance. Defaults to All. Resets when you switch service. Endpoint Restrict to one endpoint. Defaults to All. A dropdown of the service\u0026rsquo;s endpoints (capped at 50). Status ALL, SUCCESS, or ERROR — the trace state. Order BY_START_TIME (Newest) or BY_DURATION (Slowest). Limit Cap on result rows: 30 by default. The server caps a single page at 200. The list says when the window held more traces than the limit returned — there is no total on the wire, only \u0026ldquo;there is more\u0026rdquo;. Time range A rolling preset (Last 15 min through Last 24 hours) or a Custom… absolute start/end pair. Trace ID Paste a known trace id to look it up directly. Duration range (ms) Min–max trace duration, in milliseconds. Tag Free-form span tags as key=value (for example http.status_code=500). Press Enter to add; each committed tag shows as an Active-tag chip. Multiple tags are AND-joined. The time window is evaluated at second precision so a trace that just finished still falls inside it — minute rounding would drop the most recent (and usually most interesting) traces during triage.\nRicher vs. universal results, by storage backend What a result row represents depends on the storage backend behind OAP, and Horizon detects this automatically — you do not configure it:\nOn backends that support it, the explorer fetches whole traces with their spans inline. The list shows complete traces, and selecting one renders its waterfall immediately with no second round-trip. A banner reads \u0026ldquo;This OAP serves traces via Trace Query v2 API\u0026rdquo; and \u0026ldquo;Full traces are returned inline.\u0026rdquo; On any other backend, the explorer falls back to the universal basic query, which returns trace segments. Each row is one segment; the full trace is fetched on click. The banner reads \u0026ldquo;Trace Query v1 API\u0026rdquo; and \u0026ldquo;Each row is a trace segment — click one to fetch its full trace.\u0026rdquo; The banner stays visible across both the browse list and the open-trace view, so it is always clear what a row represents. The richer inline view is a property of the storage backend, not a setting — if your rows are segments, the backend does not support whole-trace queries.\nDuration distribution Beside the conditions, a Distribution chart plots one dot per result: the X axis is the trace\u0026rsquo;s start time, and the dot\u0026rsquo;s duration (the Y value) is surfaced on hover. Error traces are drawn in the error color, successful ones in the accent color.\nThe chart is an in-page filter. Click a dot — or drag a rectangle across several — to pick a subset; the result list then narrows to just the picked traces and the header switches to an \u0026ldquo;N picked\u0026rdquo; count with a Reset button. This filters what is already loaded; it does not issue a new query.\nResult list and the trace waterfall Each row in the result list shows the trace\u0026rsquo;s root endpoint, an OK/ERR status flag, the duration, and a bar sized relative to the slowest trace in the set. Click a row to open it.\nSelecting a trace opens the detail view, which offers three layouts:\nDefault — the span waterfall: an indented timeline, one row per span. Each row carries a service-colored bar positioned and sized by the span\u0026rsquo;s start offset and duration, a span-kind glyph, a component icon, the endpoint or peer name, and the span\u0026rsquo;s own duration. Errored spans are highlighted. A flag badge marks spans that carry attached events. Tree — the same spans drawn as a zoomable node graph. Statistics — spans rolled up by name, with count and total / average / maximum duration, sortable per column. Span kinds are grouped into entry (server), exit (client), local, producer, and consumer families, each with its own glyph and color. The waterfall stitches spans across segments using their parent references, so a single trace that spans multiple services renders as one connected timeline.\nClick any span row to open its detail panel:\nMeta — service, instance, endpoint, kind, component, peer, layer, start time, duration, and error flag. Cross-trace refs — when a span references a parent in a different trace, those references are listed with the parent trace id, parent segment, parent span, and ref type. The trace id is a link that opens that other trace. Tags — the span\u0026rsquo;s key/value tags. Logs — per-span log entries with their timestamps. Attached Events — named events on the span with their start/end times and summary key/values. The detail view\u0026rsquo;s header KPIs report the trace\u0026rsquo;s start time, total duration, span count, and the number of distinct services it touched. You can copy the trace id or a shareable URL from there; opening a shared ?traceId= link lands directly on the trace in an overlay.\nZipkin traces When a layer enables Zipkin, the Zipkin tab queries an upstream Zipkin store through OAP. Zipkin organizes data by its own service universe (the localEndpoint.serviceName reported on each span), which can drift from SkyWalking\u0026rsquo;s service list, so this tab carries its own service controls rather than binding to the shell\u0026rsquo;s Service picker.\nConditions Condition What it does Service Free-text service name (with suggestions). Empty means every service. Remote service Narrow to spans calling a given remote service. Requires a service to be picked first. Span name Narrow to one span/operation name. Requires a service to be picked first. Min duration (ms) / Max duration (ms) Duration bounds, entered in milliseconds. Annotations Zipkin annotation query — error or key=value terms, AND-joined. Open trace ID Paste a trace id to open it directly. Limit Result cap: 10, 30, 50, 100, or 200. The list says when the window held more traces than the limit returned. Time range A lookback preset (Last 15 min through Last 24 hours) or a Custom range… absolute window. As with the native tab, conditions are staged and only applied on Run query.\nEach Zipkin result shows its duration and error state, with a duration bar colored fast-to-slow (errored traces are forced to the error color). Selecting a trace renders the Zipkin span waterfall, and a span detail panel exposes the span\u0026rsquo;s duration, kind, and Zipkin tags. Because the two stores have different span formats, there is no field mapping between native and Zipkin results — Zipkin spans keep their Zipkin shape.\nTroubleshooting \u0026ldquo;No traces in window.\u0026rdquo; — the query ran but matched nothing. Widen the time range, relax the Status / Duration / Tag conditions, or confirm the service is actually reporting traces. An unreachable chip on the list — the trace store did not answer, and the reason is printed in a banner above the results. For native traces this points at OAP or its storage backend; for Zipkin it points at the configured Zipkin endpoint. The two stores fail independently — one being down does not blank the other. Run query is greyed out — the tab does not yet know which service to read. It says which: Resolving service… while the picked service is being looked up, or a note that the selected service is not in this layer (it aged out of OAP, was renamed, or the link points elsewhere) — pick another one. Traces are always read for one service, so the tab waits instead of querying the whole layer. Rows are segments, not whole traces — that is expected on storage backends without whole-trace support; the banner says so. Click a segment to fetch its full trace. A pasted trace id from a log row won\u0026rsquo;t resolve — older traces can sit outside the default lookup window or in a cold storage tier. Open the trace from the log row (which carries its timestamp) rather than pasting the id cold, so the lookup is widened around the right time. No data even with a valid service — double-check the time range first; this tab does not follow the global topbar, so the window is whatever the tab\u0026rsquo;s own Time range control says. Related Trace Inspect — the cross-layer trace query tool: look up a trace by id or query any service (picked, typed by name, or all of them) without entering a layer. 3D Infrastructure Map — topology-level view of the same services these traces flow through. Metrics Inspect — confirm which metrics a service is reporting when traces look incomplete. Layer Dashboard Templates — where a layer\u0026rsquo;s traces.source is configured. ","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/operate/traces/","title":"\u003c!--"},{"body":" ActiveMQ The ACTIVEMQ layer monitors Apache ActiveMQ message brokers. SkyWalking collects ActiveMQ metrics over OpenTelemetry and rolls them up into cluster-, broker-, and destination-scope metrics, so operators can watch queue depth, throughput, connection counts, and broker JVM health alongside the rest of their estate. See the upstream ActiveMQ monitoring setup for how to wire the telemetry pipeline.\nIn Horizon\u0026rsquo;s sidebar this layer is named ActiveMQ. Its services are listed as ActiveMQ clusters, its instances as Brokers, and its endpoints as Destinations (the queues and topics a broker serves). The ACTIVEMQ layer enables the Service, Instance, and Endpoint sub-tabs; it has no topology, traces, or logs view.\nThis page is the operator reference for the bundled ACTIVEMQ dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled ACTIVEMQ template; if an operator has published a customized ACTIVEMQ template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every ActiveMQ cluster with four sortable columns, sorted by Enqueue/s by default:\nEnqueue/s — messages enqueued per second across the cluster (meter_activemq_cluster_enqueue_rate).\nDequeue/s — messages dequeued per second across the cluster (meter_activemq_cluster_dequeue_rate).\nSystem Load — the cluster\u0026rsquo;s average system load (meter_activemq_cluster_system_load_average/10000).\nThreads — the cluster\u0026rsquo;s average thread count (meter_activemq_cluster_thread_count).\nService dashboard The primary drill-down for one selected ActiveMQ cluster, mixing throughput, message timing, and broker JVM health.\nSystem Load Average — the cluster\u0026rsquo;s system load over time (meter_activemq_cluster_system_load_average/10000).\nThread Count — live JVM threads across the cluster (meter_activemq_cluster_thread_count).\nHeap Used (MB) — JVM heap memory in use, in MB (meter_activemq_cluster_heap_memory_usage_used/1024/1024).\nHeap Max (MB) — the configured maximum heap across the cluster, summed and shown as the latest value in MB (latest(aggregate_labels(meter_activemq_cluster_heap_memory_usage_max,sum))/1024/1024).\nEnqueue / Dequeue / Dispatch /s — the three core message rates on one chart: messages enqueued, dequeued, and dispatched per second (meter_activemq_cluster_enqueue_rate, meter_activemq_cluster_dequeue_rate, meter_activemq_cluster_dispatch_rate).\nExpired /s — messages that expired before delivery, per second (meter_activemq_cluster_expired_rate).\nEnqueue Time — average and maximum time a message spends being enqueued, in seconds (meter_activemq_cluster_average_enqueue_time/1000, meter_activemq_cluster_max_enqueue_time/1000).\nGC Counts (G1+Parallel) — old- and young-generation garbage-collection counts, each combining the G1 and Parallel collectors so the chart reads correctly regardless of which collector the broker JVM uses (view_as_seq(meter_activemq_cluster_gc_g1_old_collection_count, meter_activemq_cluster_gc_parallel_old_collection_count), and the matching young-collection counters).\nGC Time (ms) — old- and young-generation GC time in ms, again combining the G1 and Parallel collectors (view_as_seq(meter_activemq_cluster_gc_g1_old_collection_time, meter_activemq_cluster_gc_parallel_old_collection_time), and the matching young-collection timers).\nInstance dashboard For one selected broker. A row of single-value cards summarizes the broker\u0026rsquo;s current state, followed by trend charts.\nSummary cards\nConnections — current TCP/JMS connections to this broker (latest(meter_activemq_broker_current_connections)).\nProducer Count — active producer sessions on this broker, summed across destinations (latest(aggregate_labels(meter_activemq_broker_current_producer_count,sum))).\nConsumer Count — active consumer sessions on this broker, summed across destinations (latest(aggregate_labels(meter_activemq_broker_current_consumer_count,sum))).\nUptime — broker uptime in hours since its last restart (latest(meter_activemq_broker_uptime)/1000/60/60).\nTrends\nConnections (trend) — the broker\u0026rsquo;s connection count over time (meter_activemq_broker_current_connections).\nEnqueue / Dequeue Count — per-minute enqueue and dequeue totals summed across the broker\u0026rsquo;s destinations (aggregate_labels(meter_activemq_broker_enqueue_count,sum), aggregate_labels(meter_activemq_broker_dequeue_count,sum)).\nProducer / Consumer Increase — new producer and consumer sessions opened per minute (aggregate_labels(meter_activemq_broker_producer_count,sum), aggregate_labels(meter_activemq_broker_consumer_count,sum)).\nMemory Usage — aggregate memory usage across destinations, in MB (aggregate_labels(meter_activemq_broker_memory_usage,sum)/1024/1024).\nMemory Limit — the configured memory ceiling across destinations, in GB (aggregate_labels(meter_activemq_broker_memory_limit,sum)/1024/1024/1024).\nAvg Message Size — average message size across destinations, in bytes (aggregate_labels(meter_activemq_broker_average_message_size,avg)).\nEndpoint dashboard For one selected destination (a queue or topic).\nProducer Count — producers attached to this destination (meter_activemq_destination_producer_count).\nConsumer Count — consumers attached to this destination (meter_activemq_destination_consumer_count).\nQueue Size — messages currently held in the destination (meter_activemq_destination_queue_size).\nMemory Usage (MB) — memory the destination is consuming, in MB (meter_activemq_destination_memory_usage/1024/1024).\nMessage Counts — the destination\u0026rsquo;s message lifecycle on one chart: enqueued, dequeued, dispatched, expired, and in-flight counts (meter_activemq_destination_enqueue_count, meter_activemq_destination_dequeue_count, meter_activemq_destination_dispatch_count, meter_activemq_destination_expired_count, meter_activemq_destination_inflight_count).\nEnqueue Time (s) — average and maximum enqueue time for the destination, in seconds (meter_activemq_destination_average_enqueue_time/1000, meter_activemq_destination_max_enqueue_time/1000).\nRequirements The ACTIVEMQ dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the ActiveMQ meter families, produced from broker telemetry collected over OpenTelemetry:\nCluster metrics — the meter_activemq_cluster_* family (enqueue / dequeue / dispatch / expired rates, enqueue time, system load, thread count, heap usage, and the G1 / Parallel GC counters) for the service list and the cluster dashboard.\nBroker metrics — the meter_activemq_broker_* family (current connections, producer / consumer counts, uptime, enqueue / dequeue counts, memory usage and limit, average message size) for the broker dashboard.\nDestination metrics — the meter_activemq_destination_* family (producer / consumer count, queue size, memory usage, the enqueue / dequeue / dispatch / expired / in-flight counts, and enqueue time) for the destination dashboard.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a broker- or destination-scope metric is empty until that level of data is reported. See the upstream ActiveMQ monitoring setup for the collection pipeline that produces these families.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/activemq/","title":"\u003c!--"},{"body":" AI Agents The AI_AGENT layer is where the SkyWalking AI Sessionizer lands the conversations of long-lived AI agents. The Sessionizer collects an agent runtime\u0026rsquo;s transcripts on the machine where the agent runs, assembles them into conversations, and pushes them to OAP; OAP stores them under this layer and answers Horizon\u0026rsquo;s reads. Nothing in Horizon talks to the agent runtime or to the Sessionizer directly.\nIn Horizon\u0026rsquo;s sidebar this layer is named AI Agents. Its top-level entities are Agent runtimes (the service slot): one per kind of agent, Claude Code for the Claude Code adapter, or whatever service name the Sessionizer was configured with. Each runtime reports through one or more Senders (the instance slot): one Sessionizer process on one machine, named user@host by default, or the mailbox or machine name its operator set.\nThe layer has no metric dashboards yet, so it has no Service, Instance or Endpoint page; its one tab is Conversations. See AI Agent Conversations for what that tab shows and how to read it.\nThe layer appears only when OAP reports it, which needs OAP 11.1.0 or later with at least one conversation pushed. The OAP side — receiving, verifying and storing the files, and its retention — is documented with the other OAP backend setup pages in the SkyWalking repository.\nBundled template The bundled AI_AGENT template enables the aiConversations component and nothing else, and names the two entity slots as above. Like every layer template it can be edited under Dashboard setup → Layer dashboards — for example to rename the slots for your organisation — and published to OAP; see Layer Dashboard Templates.\nWhen the Sessionizer starts exporting conversation metrics, this template is where their dashboards will be added.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/ai_agent/","title":"\u003c!--"},{"body":" Airflow The AIRFLOW layer monitors Apache Airflow workflow schedulers. OAP collects Airflow\u0026rsquo;s OpenTelemetry metrics and presents each monitored Airflow deployment as a cluster, so this layer is where you watch scheduler health, DAG parsing, executor and pool capacity, and triggerer activity.\nIn Horizon\u0026rsquo;s sidebar this layer is named Airflow, grouped under Workflow Scheduler. Its services are listed as Airflow clusters and the components that report into each cluster (the scheduler and triggerer processes) are listed as Components. The AIRFLOW layer enables only the Service and Instance scopes — there is no endpoint scope, no topology, and no traces or logs tab, because Airflow\u0026rsquo;s telemetry is scheduler-level meter data rather than request traffic.\nThis page is the operator reference for the bundled AIRFLOW dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled AIRFLOW template; if an operator has published a customized AIRFLOW template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every Airflow cluster with four sortable columns, sorted by DAG Bag Size by default:\nDAG Bag Size — number of DAGs found in the last scheduler scan (meter_airflow_dagbag_size).\nDAG Total Parse Time — seconds spent scanning and importing the queued DAG files (meter_airflow_dag_total_parse_time).\nExecutor Open Slots — free executor slots available to run tasks (meter_airflow_executor_open_slots).\nScheduled Slots — pool slots scheduled but not yet running, summed across pools (aggregate_labels(meter_airflow_pool_scheduled_slots, sum)).\nService dashboard The primary drill-down for one selected Airflow cluster. It opens with four single-value cards reporting the current scheduler and executor state, then a set of time-series charts tracking executor capacity and DAG-processing health.\nCards (current value)\nTasks Executable — tasks ready for execution across the cluster (latest(meter_airflow_scheduler_tasks_executable)).\nRunning Tasks — tasks currently running on the executor (latest(meter_airflow_executor_running_tasks)).\nScheduled Slots — pool slots scheduled but not yet running, aggregated across pools (latest(aggregate_labels(meter_airflow_pool_scheduled_slots, sum))).\nQueued Tasks — tasks waiting on the executor (latest(meter_airflow_executor_queued_tasks)).\nCharts (over time)\nExecutor Open Slots — free executor slots over the window (meter_airflow_executor_open_slots).\nDAG File Queue Size — DAG files pending a scan (meter_airflow_dag_file_queue_size).\nDAG Import Errors — DAG files that failed to parse (meter_airflow_dag_import_errors).\nDAG Bag Size — DAGs found in the last scheduler scan (meter_airflow_dagbag_size).\nDAG Total Parse Time — seconds to scan and import the queued DAG files (meter_airflow_dag_total_parse_time).\nDAG File Refresh Errors — DAG file load failures per minute (meter_airflow_dag_file_refresh_error).\nAsset Updates — asset update events per minute (meter_airflow_asset_updates).\nInstance dashboard For one selected Component of the cluster. Airflow reports different meters from different processes — the scheduler emits pool, executor, and asset metrics, while the triggerer emits trigger metrics — so each widget appears only when that component actually reports the metric behind it. A scheduler component therefore shows the pool / executor / heartbeat widgets, a triggerer component shows the triggerer widgets, and neither is shown empty for a component that does not emit it.\nScheduler component\nPool Open / Deferred / Running Slots — pool capacity on the scheduler, plotted as three series: open, deferred, and running slots (meter_airflow_instance_pool_open_slots, meter_airflow_instance_pool_deferred_slots, meter_airflow_instance_pool_running_slots).\nRunning Tasks / Scheduled Slots — executor running-task count against pool slots waiting to run (meter_airflow_instance_executor_running_tasks, meter_airflow_instance_pool_scheduled_slots).\nScheduler Heartbeat — scheduler heartbeats per minute (meter_airflow_instance_scheduler_heartbeat).\nExecutor Open / Queued Slots — executor capacity and queue depth on the scheduler (meter_airflow_instance_executor_open_slots, meter_airflow_instance_executor_queued_tasks).\nAsset Updates — asset update events on the scheduler, per minute (meter_airflow_instance_asset_updates).\nAsset Triggered DagRuns — DagRuns triggered by asset events on the scheduler, per minute (meter_airflow_instance_asset_triggered_dagruns).\nTriggerer component\nTriggerer Heartbeat — triggerer process heartbeats per minute (meter_airflow_instance_triggerer_heartbeat).\nTriggers Running / Capacity Left — live deferrable-trigger load on the triggerer: triggers running against capacity left (meter_airflow_instance_triggers_running, meter_airflow_instance_triggerer_capacity_left).\nTriggers Blocked / Failed / Succeeded — deferred-trigger outcomes on the triggerer host, per minute (meter_airflow_instance_triggers_blocked_main_thread, meter_airflow_instance_triggers_failed, meter_airflow_instance_triggers_succeeded).\nRequirements The AIRFLOW dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nCluster (service-scope) Airflow meters — the meter_airflow_* family (DAG bag size and parse time, DAG-processing queue and import / refresh errors, executor open / queued slots and running / executable tasks, pool scheduled slots, asset updates), aggregated per Airflow cluster.\nComponent (instance-scope) Airflow meters — the meter_airflow_instance_* family (per-component pool, executor, scheduler-heartbeat, asset, and triggerer metrics), reported by each scheduler or triggerer process.\nThese meters are produced by OAP from Airflow\u0026rsquo;s OpenTelemetry metric export — see Airflow monitoring for how to wire Airflow up to OAP. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so an instance-scope component metric is empty until that component reports it. When a component does not emit a family — a triggerer that reports no scheduler pool metrics, or a scheduler that reports no triggerer metrics — those widgets are hidden rather than shown empty.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/airflow/","title":"\u003c!--"},{"body":" Alipay Mini Program The ALIPAY_MINI_PROGRAM layer holds front-end real-user monitoring data reported from Alipay (支付宝) mini-programs. The mini-program monitoring agent feeds OAP launch, render, request, and error metrics from inside the Alipay container, and those land here.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Mobile. Its services are listed as Mini-programs, instances as Versions (each version of a published mini-program), and endpoints as Pages. The ALIPAY_MINI_PROGRAM layer enables the Service, Instance (Version), and Endpoint (Page) dashboards along with the Traces and Logs sub-tabs. It does not ship a topology / service-map view — mini-program RUM data has no call graph to draw.\nThis page is the operator reference for the bundled ALIPAY_MINI_PROGRAM dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled ALIPAY_MINI_PROGRAM template; if an operator has published a customized template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a mini-program, the layer landing page lists every Mini-program with four sortable columns, sorted by traffic (Request RPM) by default:\nRequest RPM — requests per minute (meter_alipay_mp_request_cpm).\nLaunch — average app-launch duration in ms (meter_alipay_mp_app_launch_duration).\nFirst Render — average first-render duration in ms (meter_alipay_mp_first_render_duration).\nErrors — error count over the window (meter_alipay_mp_error_count).\nService dashboard The primary drill-down for one selected mini-program.\nApp Launch Duration — the approximate cold-launch duration measured from the Alipay container, in ms (meter_alipay_mp_app_launch_duration).\nFirst Render Duration — time to first render, in ms (meter_alipay_mp_first_render_duration).\nError Count — number of reported front-end errors (meter_alipay_mp_error_count).\nRequest Load — requests per minute for the mini-program (meter_alipay_mp_request_cpm).\nRequest Duration Percentile — p50 / p75 / p90 / p95 / p99 of request duration, the tail of the request-time distribution, in ms (meter_alipay_mp_request_duration_percentile).\nInstance dashboard For one selected Version of the mini-program.\nLaunch Duration — app-launch duration for this version, in ms (meter_alipay_mp_instance_app_launch_duration).\nFirst Render Duration — first-render duration for this version, in ms (meter_alipay_mp_instance_first_render_duration).\nRequest Load — requests per minute for this version (meter_alipay_mp_instance_request_cpm).\nRequest Duration Percentile — p50 / p75 / p90 / p95 / p99 of request duration for this version, in ms (meter_alipay_mp_instance_request_duration_percentile).\nEndpoint dashboard For one selected Page.\nLaunch Duration on Page — app-launch duration attributed to this page, in ms (meter_alipay_mp_endpoint_app_launch_duration).\nFirst Render Duration — first-render duration for this page, in ms (meter_alipay_mp_endpoint_first_render_duration).\nRequest Load — requests per minute for this page (meter_alipay_mp_endpoint_request_cpm).\nRequest Duration Percentile — p50 / p75 / p90 / p95 / p99 of request duration for this page, in ms (meter_alipay_mp_endpoint_request_duration_percentile).\nRequirements The ALIPAY_MINI_PROGRAM dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the Alipay mini-program meter families, reported by the Alipay mini-program monitoring agent:\nMini-program (service) metrics — the meter_alipay_mp_* family at service scope: request load (meter_alipay_mp_request_cpm), launch and first-render duration (meter_alipay_mp_app_launch_duration, meter_alipay_mp_first_render_duration), error count (meter_alipay_mp_error_count), and the request-duration percentiles (meter_alipay_mp_request_duration_percentile).\nVersion (instance) metrics — the meter_alipay_mp_instance_* family for the per-version widgets (launch, first render, request load, request percentile).\nPage (endpoint) metrics — the meter_alipay_mp_endpoint_* family for the per-page widgets (launch, first render, request load, request percentile).\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a version- or page-scope metric is empty until that level of data is reported. For how to enable the upstream collector, see the Alipay Mini-Program monitoring setup docs.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/alipay_mini_program/","title":"\u003c!--"},{"body":" APISIX The APISIX layer monitors Apache APISIX API gateways. APISIX exposes its built-in Prometheus metrics, OpenTelemetry collects and forwards them to OAP, and OAP aggregates them into the meter_apisix_* families this dashboard renders. The layer key is APISIX, and in Horizon\u0026rsquo;s sidebar it is grouped under Gateways.\nIn the sidebar this layer\u0026rsquo;s services are listed as APISIX services, its instances as Nodes (the individual APISIX data-plane nodes), and its endpoints as Routes (the matched APISIX routes). The APISIX layer enables three scopes — Service, Instance (Node), and Endpoint (Route). It does not ship a topology, traces, or logs tab, so this dashboard is purely the metric drill-down across those three scopes.\nThis page is the operator reference for the bundled APISIX dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled APISIX template; if an operator has published a customized APISIX template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a service, the layer landing page lists every APISIX service, sorted by request rate (RPS) by default, with four columns:\nRPS — total HTTP requests per second across the service\u0026rsquo;s nodes (meter_apisix_sv_http_requests).\n200/s — 200-status responses per second (meter_apisix_sv_http_status_matched{code='200'}).\n404/s — 404-status responses per second (meter_apisix_sv_http_status_matched{code='404'}).\n503/s — 503-status responses per second (meter_apisix_sv_http_status_matched{code='503'}).\nThe three status columns give an at-a-glance health read across the fleet — a service with a climbing 503/s next to its 200/s is shedding load.\nService dashboard The primary drill-down for one selected APISIX service. All eight widgets aggregate across the service\u0026rsquo;s nodes.\nHTTP Request Trend — total requests per second for the service (meter_apisix_sv_http_requests).\nHTTP Status Trend — requests per second broken down by HTTP status code (meter_apisix_sv_http_status_matched, one line per code).\nHTTP Latency — request latency in ms, split by latency type and percentile (meter_apisix_sv_http_latency_matched).\nHTTP Bandwidth — ingress / egress bandwidth in KB, by type (meter_apisix_sv_bandwidth_matched, divided to KB).\nHTTP Connections — active connections by state — active, reading, writing, waiting (meter_apisix_sv_http_connections, one line per state).\nNon-matched Status Trend — requests per second by status code for traffic that hit no matching APISIX route (meter_apisix_sv_http_status_unmatched). Unmatched traffic is usually a misconfigured client or a probe; a rising line here is worth investigating.\nNon-matched Latency — latency in ms for the same no-matching-route traffic (meter_apisix_sv_http_latency_unmatched).\nNon-matched Bandwidth — bandwidth in KB for the same no-matching-route traffic (meter_apisix_sv_bandwidth_unmatched, divided to KB).\nInstance dashboard For one selected node. These widgets are reported per node, so they show how one APISIX data-plane process is behaving.\nHTTP Request Trend — requests per second for the node (meter_apisix_instance_http_requests).\nHTTP Status Trend — requests per second by status code for the node (meter_apisix_instance_http_status_matched).\nHTTP Latency — request latency in ms for the node (meter_apisix_instance_http_latency_matched).\nHTTP Bandwidth — bandwidth in KB for the node (meter_apisix_instance_bandwidth_matched, divided to KB).\nHTTP Connections — connections by state for the node (meter_apisix_instance_http_connections).\nShared Dict — the node\u0026rsquo;s shared-memory dictionary capacity vs. free space in MB (meter_apisix_instance_shared_dict_capacity_bytes and meter_apisix_instance_shared_dict_free_space_bytes, divided to MB, labelled capacity / free). When free space approaches zero the node can no longer cache new entries.\netcd — the node\u0026rsquo;s view of the control-plane etcd: the latest known etcd index and whether etcd is reachable (meter_apisix_instance_etcd_indexes and latest(meter_apisix_instance_etcd_reachable), labelled indexes / reachable). A node that can\u0026rsquo;t reach etcd is no longer receiving config updates.\nNon-matched Traffic — a combined view of no-matching-route activity for the node: status, latency in ms, and bandwidth in KB on one chart (meter_apisix_instance_http_status_unmatched, meter_apisix_instance_http_latency_unmatched, and meter_apisix_instance_bandwidth_unmatched divided to KB).\nEndpoint dashboard For one selected route. APISIX reports a tighter metric set at route scope — status, latency, and bandwidth.\nHTTP Status Trend — requests per second by status code for the route (meter_apisix_endpoint_http_status, one line per code).\nHTTP Latency — request latency in ms for the route, by type and percentile (meter_apisix_endpoint_http_latency).\nHTTP Bandwidth — bandwidth in KB for the route, by type (meter_apisix_endpoint_bandwidth, divided to KB).\nRequirements The APISIX dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs APISIX metrics flowing in through the OpenTelemetry receiver:\nService metrics — the meter_apisix_sv_* family (requests, status, latency, bandwidth, connections, and the unmatched-route counterparts), aggregated by OAP across a service\u0026rsquo;s nodes.\nInstance (node) metrics — the meter_apisix_instance_* family, including the node-only meter_apisix_instance_shared_dict_* and meter_apisix_instance_etcd_* health metrics.\nEndpoint (route) metrics — the meter_apisix_endpoint_* family for the per-route status, latency, and bandwidth widgets.\nThese come from APISIX\u0026rsquo;s Prometheus plugin scraped by an OpenTelemetry Collector and forwarded to OAP, which converts them into the meter_apisix_* metrics through its meter-analysis rules. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a route-scope or node-scope metric is empty until that level of data is reported. For the end-to-end collector and OAP setup, see the APISIX monitoring backend guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/apisix/","title":"\u003c!--"},{"body":" AWS DynamoDB The AWS_DYNAMODB layer monitors Amazon DynamoDB through CloudWatch metrics that OAP pulls in and aggregates. It is an agentless layer — there is no SkyWalking agent inside DynamoDB — so the dashboard is a read-only view of the throttling, error, capacity, and latency metrics CloudWatch exposes for your DynamoDB usage.\nIn Horizon\u0026rsquo;s sidebar this layer is named AWS DynamoDB. A service here represents one DynamoDB account, so services are listed as DynamoDB accounts; each account\u0026rsquo;s endpoints are its Tables. The layer enables only two scopes — the account-level Service dashboard and the per-table Endpoint dashboard. It has no instance scope, no topology / map, and no traces or logs tabs.\nThis page is the operator reference for the bundled AWS_DYNAMODB dashboard: what you see at the account level and per table, and what each widget means.\nThe widgets and metrics below are read from the bundled AWS_DYNAMODB template; if an operator has published a customized AWS_DYNAMODB template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening an account, the layer landing page lists every DynamoDB account with four sortable columns, sorted by Read Throttled by default. All four are window sums (aggregation: \u0026quot;sum\u0026quot;), so they surface the accounts taking the most throttling and system-error pressure over the selected range:\nRead Throttled — throttled read requests across the account (aws_dynamodb_read_throttled_requests).\nWrite Throttled — throttled write requests across the account (aws_dynamodb_write_throttled_requests).\nRead Sys Err — read requests that failed with a DynamoDB system error (aws_dynamodb_read_system_errors).\nWrite Sys Err — write requests that failed with a DynamoDB system error (aws_dynamodb_write_system_errors).\nService dashboard The account-level drill-down for one selected DynamoDB account. All widgets are time-series lines over the selected window.\nThrottled Requests — throttled read vs write requests for the account (aws_dynamodb_read_throttled_requests, aws_dynamodb_write_throttled_requests).\nThrottle Events — throttle events on read vs write, counted independently of throttled request volume (aws_dynamodb_read_throttle_events, aws_dynamodb_write_throttle_events).\nSystem Errors — read vs write requests that hit a DynamoDB-side system error (aws_dynamodb_read_system_errors, aws_dynamodb_write_system_errors).\nUser Errors — requests rejected for a client-side / user error such as a bad request (aws_dynamodb_user_errors).\nConditional Check Failed — write requests rejected because a conditional expression evaluated to false (aws_dynamodb_conditional_check_failed_requests).\nTransaction Conflict — transactional requests rejected due to a conflict with another in-flight transaction (aws_dynamodb_transaction_conflict).\nRead Capacity (unit/s) — provisioned read capacity vs consumed write capacity for the account (as the bundled template plots them), in capacity units per second (aws_dynamodb_provisioned_read_capacity_units, aws_dynamodb_consumed_write_capacity_units).\nWrite Capacity (unit/s) — provisioned vs consumed write capacity for the account, in capacity units per second (aws_dynamodb_provisioned_write_capacity_units, aws_dynamodb_consumed_write_capacity_units).\nSuccessful Request Latency (ms) — latency of successful operations, broken out by operation type — get / put / query / scan — in ms (aws_dynamodb_get_successful_request_latency, aws_dynamodb_put_successful_request_latency, aws_dynamodb_query_successful_request_latency, aws_dynamodb_scan_successful_request_latency).\nTTL Deleted Items — items removed by DynamoDB\u0026rsquo;s time-to-live expiry process (aws_dynamodb_time_to_live_deleted_item_count).\nScan Returned Items — items returned by Scan operations (aws_dynamodb_scan_returned_item_count).\nQuery Returned Items — items returned by Query operations (aws_dynamodb_query_returned_item_count).\nAccount Max Reads / Writes — the account-level capacity ceilings CloudWatch reports: table-level read / write maxima and account-wide read / write maxima (aws_dynamodb_account_max_table_level_reads, aws_dynamodb_account_max_table_level_writes, aws_dynamodb_account_max_reads, aws_dynamodb_account_max_writes).\nAccount Capacity Utilization — provisioned read vs write capacity utilization for the account, in percent (aws_dynamodb_account_provisioned_read_capacity_utilization, aws_dynamodb_account_provisioned_write_capacity_utilization).\nEndpoint dashboard For one selected Table under the account. These are the per-table counterparts of the account-level widgets, evaluated at endpoint scope (aws_dynamodb_endpoint_*), so you can see which individual table is responsible for the account\u0026rsquo;s throttling, errors, or capacity draw.\nThrottled Requests — throttled read vs write requests against the table (aws_dynamodb_endpoint_read_throttled_requests, aws_dynamodb_endpoint_write_throttled_requests).\nThrottle Events — read vs write throttle events on the table (aws_dynamodb_endpoint_read_throttle_events, aws_dynamodb_endpoint_write_throttle_events).\nSystem Errors — read vs write DynamoDB system errors on the table (aws_dynamodb_endpoint_read_system_errors, aws_dynamodb_endpoint_write_system_errors).\nConditional Check Failed — write requests on the table rejected by a failed conditional expression (aws_dynamodb_endpoint_conditional_check_failed_requests).\nTransaction Conflict — transactional requests on the table rejected due to a conflict (aws_dynamodb_endpoint_transaction_conflict).\nTTL Deleted Items — items removed from the table by time-to-live expiry (aws_dynamodb_endpoint_time_to_live_deleted_item_count).\nRead Capacity — provisioned vs consumed read capacity for the table (aws_dynamodb_endpoint_provisioned_read_capacity_units, aws_dynamodb_endpoint_consumed_read_capacity_units).\nWrite Capacity — provisioned vs consumed write capacity for the table (aws_dynamodb_endpoint_provisioned_write_capacity_units, aws_dynamodb_endpoint_consumed_write_capacity_units).\nSuccessful Request Latency (ms) — latency of successful operations on the table by operation type — get / put / query / scan — in ms (aws_dynamodb_endpoint_get_successful_request_latency, aws_dynamodb_endpoint_put_successful_request_latency, aws_dynamodb_endpoint_query_successful_request_latency, aws_dynamodb_endpoint_scan_successful_request_latency).\nScan Returned Items — items returned by Scan operations on the table (aws_dynamodb_endpoint_scan_returned_item_count).\nQuery Returned Items — items returned by Query operations on the table (aws_dynamodb_endpoint_query_returned_item_count).\nRequirements The AWS_DYNAMODB dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the DynamoDB metric families collected from CloudWatch and aggregated under this layer:\nAccount (service) metrics — the aws_dynamodb_* family: throttled requests and throttle events, read / write system errors, user errors, conditional-check failures, transaction conflicts, provisioned vs consumed read / write capacity, per-operation successful request latency (get / put / query / scan), TTL-deleted items, scan / query returned items, the account-level max read / write ceilings, and provisioned capacity utilization.\nTable (endpoint) metrics — the matching aws_dynamodb_endpoint_* family for the same throttling, error, capacity, latency, and returned-item metrics resolved per table.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the table-scope aws_dynamodb_endpoint_* metrics are empty until per-table data is reported, independently of the account-scope metrics. For how to collect these metrics into OAP, see the DynamoDB monitoring setup in the SkyWalking backend docs.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/aws_dynamodb/","title":"\u003c!--"},{"body":" AWS EKS The AWS_EKS layer monitors Amazon Elastic Kubernetes Service (EKS) clusters. SkyWalking ingests EKS observability data through OpenTelemetry — Container Insights / CloudWatch metrics scraped into OAP — and reshapes it into cluster, node, and pod metrics. It groups under AWS in the sidebar.\nIn Horizon\u0026rsquo;s sidebar this layer\u0026rsquo;s entities are renamed to fit the EKS model: services are listed as Clusters, instances as Nodes, and endpoints as EKS services (the Kubernetes services running inside the cluster). The AWS_EKS layer enables the Service (Cluster), Instance (Node), and Endpoint (EKS service) scopes; it does not enable a topology, traces, or logs tab — EKS reports metric data only.\nThis page is the operator reference for the bundled AWS_EKS dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled AWS_EKS template; if an operator has published a customized AWS_EKS template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nCluster list Before opening a cluster, the layer landing page lists every EKS cluster with four sortable columns, sorted by Nodes by default. Each column shows the latest reading averaged across the window:\nNodes — number of nodes in the cluster (latest(eks_cluster_node_count)).\nFailed Nodes — nodes currently in a failed state (latest(eks_cluster_failed_node_count)).\nNamespaces — Kubernetes namespaces in the cluster (latest(eks_cluster_namespace_count)).\nServices — Kubernetes services in the cluster (latest(eks_cluster_service_count)).\nCluster dashboard The primary drill-down for one selected cluster (a Service in OAP terms).\nNode Count — number of nodes over time (eks_cluster_node_count).\nFailed Nodes — nodes in a failed state over time (eks_cluster_failed_node_count).\nNamespace Count — Kubernetes namespaces in the cluster (eks_cluster_namespace_count).\nEKS Service Count — Kubernetes services in the cluster (eks_cluster_service_count).\nCluster Network Errors — cluster-wide receive and transmit error counts, plotted as two series, rx (eks_cluster_net_rx_error) and tx (eks_cluster_net_tx_error).\nCluster Network Drops — cluster-wide dropped packets on receive and transmit, rx (eks_cluster_net_rx_dropped) and tx (eks_cluster_net_tx_dropped).\nNode dashboard For one selected node (an Instance in OAP terms).\nPod Count — pods scheduled on the node (eks_cluster_node_pod_number).\nCPU Utilization (%) — node CPU utilization (eks_cluster_node_cpu_utilization).\nMemory Utilization (%) — node memory utilization (eks_cluster_node_memory_utilization).\nFS Utilization (%) — node filesystem utilization (eks_cluster_node_fs_utilization).\nNetwork RX (KB/s) — node receive throughput in KB/s (eks_cluster_node_net_rx_bytes/1024) on the left axis, with receive errors (eks_cluster_node_net_rx_error) on a second axis so the error count doesn\u0026rsquo;t get lost against the byte scale.\nNetwork TX (KB/s) — node transmit throughput in KB/s (eks_cluster_node_net_tx_bytes/1024) on the left axis, with transmit errors (eks_cluster_node_net_tx_error) on a second axis.\nDisk IO (B/s) — node disk read and write throughput in bytes/s, plotted as read (eks_cluster_node_disk_io_read) and write (eks_cluster_node_disk_io_write).\nPod CPU on Node — aggregate CPU utilization of the pods running on this node (eks_cluster_node_pod_cpu_utilization).\nPod Memory on Node — aggregate memory utilization of the pods running on this node (eks_cluster_node_pod_memory_utilization).\nEKS service dashboard For one selected EKS service (an Endpoint in OAP terms) — a Kubernetes service running inside the cluster, with its pod-level resource and network metrics.\nPod CPU Utilization (%) — CPU utilization across the service\u0026rsquo;s pods (eks_cluster_service_pod_cpu_utilization).\nPod Memory Utilization (%) — memory utilization across the service\u0026rsquo;s pods (eks_cluster_service_pod_memory_utilization).\nPod Network RX (KB/s) — pod receive throughput in KB/s (eks_cluster_service_pod_net_rx_bytes/1024).\nPod RX Errors / s — pod receive error rate (eks_cluster_service_pod_net_rx_error).\nPod Network TX (KB/s) — pod transmit throughput in KB/s (eks_cluster_service_pod_net_tx_bytes/1024).\nPod TX Errors / s — pod transmit error rate (eks_cluster_service_pod_net_tx_error).\nRequirements The AWS_EKS dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the EKS observability metric families, fed in through the OpenTelemetry receiver from Amazon CloudWatch / Container Insights:\nCluster metrics — the eks_cluster_* family at cluster scope: node / failed-node / namespace / service counts and cluster-wide network error and drop counters.\nNode metrics — the eks_cluster_node_* family at node scope: pod count, CPU / memory / filesystem utilization, network receive / transmit bytes and errors, disk read / write IO, and the per-node aggregate pod CPU / memory utilization.\nEKS service metrics — the eks_cluster_service_pod_* family at EKS-service scope: per-service pod CPU / memory utilization and pod network receive / transmit bytes and errors.\nEach metric is queried at its own OAP scope (Cluster / Node / EKS service); OAP does not roll a metric up across scopes, so a node- or service-scope metric stays empty until that level of data is reported. For how to stand up the EKS-to-OAP pipeline, see the layer\u0026rsquo;s setup documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/aws_eks/","title":"\u003c!--"},{"body":" AWS API Gateway The AWS_GATEWAY layer monitors Amazon API Gateway. OAP pulls per-gateway and per-route metrics from AWS CloudWatch — request counts, latency, error rates, cache behavior, and data volume — and presents each gateway as a service in SkyWalking.\nIn Horizon\u0026rsquo;s sidebar this layer is named AWS API Gateway. Its services are listed as AWS Gateways and its endpoints as Routes (each route is a method-plus-resource path on a gateway). The layer enables only the Service and Endpoint sub-tabs — there is no instance scope, no topology, and no traces or logs tab, because CloudWatch reports gateway- and route-level aggregates rather than per-instance, per-request, or relationship data.\nThis page is the operator reference for the bundled AWS_GATEWAY dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled AWS_GATEWAY template; if an operator has published a customized AWS_GATEWAY template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a gateway, the layer landing page lists every AWS Gateway with four sortable columns, sorted by request volume (Requests) by default:\nRequests — total request count over the window (aws_gateway_service_count).\nLatency — average request latency in ms (aws_gateway_service_latency).\n4xx — count of client-error (4xx) responses (aws_gateway_service_4xx).\n5xx — count of server-error (5xx) responses (aws_gateway_service_5xx).\nService dashboard The primary drill-down for one selected gateway.\nRequest Count — total requests handled by the gateway (aws_gateway_service_count).\n4xx Count — client-error responses (aws_gateway_service_4xx).\n5xx Count — server-error responses (aws_gateway_service_5xx).\nRequest Avg Latency — average end-to-end request latency in ms (aws_gateway_service_latency).\nIntegration Avg Latency — average latency between the gateway and its backend integration in ms, isolating backend time from gateway overhead (aws_gateway_service_integration_latency).\nData Processed (HTTP API only) — bytes processed by the gateway, shown in KB (aws_gateway_service_data_processed/1024). Populated only for HTTP API gateways.\nCache Hit Rate (REST API only) — percent of requests served from the gateway cache (aws_gateway_service_cache_hit_rate). Populated only for REST API gateways with caching enabled.\nCache Miss Rate (REST API only) — percent of requests that missed the gateway cache (aws_gateway_service_cache_miss_rate). Populated only for REST API gateways with caching enabled.\nEndpoint dashboard For one selected route (an endpoint under a gateway).\nRequest Count — total requests to the route (aws_gateway_endpoint_count).\n4xx Count — client-error responses on the route (aws_gateway_endpoint_4xx).\n5xx Count — server-error responses on the route (aws_gateway_endpoint_5xx).\nRequest Avg Latency — average request latency in ms (aws_gateway_endpoint_latency).\nIntegration Avg Latency — average gateway-to-backend integration latency in ms (aws_gateway_endpoint_integration_latency).\nData Processed — bytes processed for the route, shown in KB (aws_gateway_endpoint_DataProcessed/1024).\nCache Hit Rate — percent of requests served from cache (aws_gateway_endpoint_cache_hit_rate).\nCache Miss Rate — percent of requests that missed the cache (aws_gateway_endpoint_cache_miss_rate).\nRequirements The AWS_GATEWAY dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the AWS API Gateway receiver enabled and pulling CloudWatch metrics, which produces:\nGateway (service-scope) metrics — the aws_gateway_service_* family: request count, latency, integration latency, 4xx / 5xx counts, data processed, and cache hit / miss rates.\nRoute (endpoint-scope) metrics — the aws_gateway_endpoint_* family: the same measures at route granularity.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a route-scope metric is empty until route-level data is reported. Cache-rate and data-processed widgets stay empty for gateways whose API type (REST vs HTTP API) or configuration does not emit that CloudWatch metric.\nFor setting up the receiver, see the AWS API Gateway monitoring setup guide in the SkyWalking backend docs.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/aws_gateway/","title":"\u003c!--"},{"body":" AWS S3 The AWS_S3 layer monitors Amazon S3 storage by reading CloudWatch request metrics for your buckets, so each S3 bucket appears in SkyWalking as a service with its own request, error, latency, and transfer dashboard.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under AWS and its services are listed as S3 buckets — one entry per monitored bucket. The AWS_S3 layer is a metrics-only layer: it enables the Service scope alone. There is no instance, endpoint, topology, trace, or log sub-tab, because S3 monitoring is CloudWatch metric data rather than agent-instrumented traffic.\nThis page is the operator reference for the bundled AWS_S3 dashboard: what you see for each S3 bucket and what each widget means.\nThe widgets and metrics below are read from the bundled AWS_S3 template; if an operator has published a customized AWS_S3 template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a bucket, the layer landing page lists every S3 bucket with four sortable columns, sorted by request volume (Requests) by default:\nRequests — total request count for the bucket (aws_s3_all_requests).\nAvg Latency — average request latency in ms (aws_s3_request_latency).\n4xx — count of 4xx (client-error) responses (aws_s3_4xx).\n5xx — count of 5xx (server-error) responses (aws_s3_5xx).\nService dashboard The drill-down for one selected S3 bucket.\nAll Request Count — total requests against the bucket over the window (aws_s3_all_requests).\nGET Request Count — GET (read / download) requests (aws_s3_get_requests).\nPUT Request Count — PUT (write / upload) requests (aws_s3_put_requests).\nDELETE Request Count — DELETE requests (aws_s3_delete_requests).\n4xx Count — client-error responses, the 4xx family (aws_s3_4xx).\n5xx Count — server-error responses, the 5xx family (aws_s3_5xx).\nRequest Avg Latency — average total request latency in ms (aws_s3_request_latency).\nFirst Byte Avg Latency — average time to first byte in ms, the latency before any payload starts streaming back (aws_s3_first_latency_bytes).\nDownloaded (KB) — bytes downloaded from the bucket, in KB (aws_s3_downloaded_bytes).\nUploaded (KB) — bytes uploaded to the bucket, in KB (aws_s3_uploaded_bytes).\nRequirements The AWS_S3 dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the AWS S3 monitoring receiver enabled, pulling the bucket\u0026rsquo;s CloudWatch request metrics, which OAP turns into the aws_s3_* service-scope metric family: request counts (aws_s3_all_requests, aws_s3_get_requests, aws_s3_put_requests, aws_s3_delete_requests), error counts (aws_s3_4xx, aws_s3_5xx), latency (aws_s3_request_latency, aws_s3_first_latency_bytes), and transfer volume (aws_s3_downloaded_bytes, aws_s3_uploaded_bytes).\nEvery metric in this dashboard is queried at the Service scope — the S3 bucket — so each bucket you have configured CloudWatch monitoring for appears as one entry in the S3 buckets list. For the OAP-side setup (CloudWatch credentials, the buckets to watch, and the collection interval), follow the AWS S3 monitoring setup guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/aws_s3/","title":"\u003c!--"},{"body":" BanyanDB The BANYANDB layer is the self-observability dashboard for Apache SkyWalking BanyanDB, the native storage that backs an OAP cluster. It surfaces the health of the storage tier itself — write and query throughput, the liaison front door, the data nodes that hold the shards, the lifecycle sidecar that migrates data between tiers, and the per-group load across the measure / stream / trace / property data models.\nIn Horizon\u0026rsquo;s sidebar this layer sits under the Self-Observability group and is named BanyanDB. It maps BanyanDB\u0026rsquo;s own topology onto the standard entity slots: a BanyanDB cluster is a Cluster (the service slot), each running container — a liaison, data, or lifecycle process — is a Container (the instance slot, badged with its container_name), and each storage group is a Group (the endpoint slot). The layer enables the Cluster, Container, and Group dashboards, an extension page under Cluster for Trace Sampling, and a layer-specific Deployment tab; it ships no service topology, no API-dependency view, and no traces or logs tabs.\nThis page is the operator reference for the bundled BANYANDB dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled BANYANDB template; if an operator has published a customized BANYANDB template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nCluster list Before opening a cluster, the layer landing page lists every BanyanDB cluster with three sortable columns, sorted by write rate by default:\nWrite/s — cluster-wide writes per second (meter_banyandb_cluster_write_rate).\nQuery/s — cluster-wide query calls per second (meter_banyandb_cluster_query_rate).\nErrors — cluster-wide errors per minute (meter_banyandb_cluster_error_rate).\nCluster dashboard The primary drill-down for one selected cluster, summarizing the whole storage tier.\nWrite Rate / Query Rate / Error Rate — three headline cards: cluster-wide writes per second across the measure + stream + trace data models (meter_banyandb_cluster_write_rate), gRPC query calls per second seen at the liaison front door (meter_banyandb_cluster_query_rate), and errors per minute summed across the cluster (meter_banyandb_cluster_error_rate). On a healthy cluster the error card reads 0.\nCPU Cores / Memory Used / Disk Used — capacity cards rolled up across the cluster\u0026rsquo;s containers: total CPU cores visible (meter_banyandb_total_cpu_cores), total memory used in GB (meter_banyandb_total_memory_used), and total on-disk bytes used across the data paths in GB (meter_banyandb_total_disk_used).\nCluster Throughput — write rate versus query rate over time on one chart (meter_banyandb_cluster_write_rate, meter_banyandb_cluster_query_rate).\nCluster Errors / min — cluster-wide errors per minute over time (meter_banyandb_cluster_error_rate).\nContainers by Role — a table of the live container count per role (data / liaison), derived from the system uptime gauge (meter_banyandb_reporting_instances). The lifecycle sidecar runs no system collector, so it does not appear here.\nContainer dashboard For one selected container. Because the three BanyanDB roles report different metrics, most widgets are role-specific and appear only on the container they apply to — a liaison container shows the front-door widgets, a data container the storage-engine widgets, and the lifecycle sidecar its migration widgets. A handful of common runtime widgets render on every container, and a few host widgets appear only when that container\u0026rsquo;s system collector reports them.\nCommon runtime (every container)\nCPU Usage — process CPU consumption in cores (meter_banyandb_instance_cpu_usage).\nResident Memory — process resident memory in MB (meter_banyandb_instance_rss_memory).\nGoroutines — live goroutine count (meter_banyandb_instance_goroutines).\nGC Pause (avg) — average Go GC pause per cycle in ms (meter_banyandb_instance_gc_pause_avg).\nGo Heap — Go heap in-use versus next-GC threshold in MB (meter_banyandb_instance_heap_inuse, meter_banyandb_instance_heap_next_gc).\nGo Alloc Rate — Go allocation rate in MB/s (meter_banyandb_instance_alloc_rate).\nHost (when the system collector reports it)\nUptime — days since the node started (meter_banyandb_instance_node_uptime). Absent on the lifecycle sidecar, which runs no system collector.\nSystem Memory Used — host memory used as a percentage (meter_banyandb_instance_system_memory_percent).\nDisk Usage — used / total across the node\u0026rsquo;s data paths as a percentage (meter_banyandb_instance_disk_usage_percent).\nDisk Used / Total — used per data path against total filesystem capacity in GB (meter_banyandb_instance_disk_used_by_path, meter_banyandb_instance_disk_total_by_path). Paths that share one filesystem report identical figures.\nNetwork I/O — per-interface receive / send throughput in KB/s (meter_banyandb_instance_network_recv, meter_banyandb_instance_network_sent).\nLifecycle sidecar (container_name = lifecycle)\nTime Since Last Sync — how long ago the last migration cycle started, shown as a duration (meter_banyandb_instance_lifecycle_last_run). Appears once the first migration cycle has run.\nLast Sync — whether the last migration cycle succeeded (OK) or failed (meter_banyandb_instance_lifecycle_last_run_success).\nMigration Cycles — cumulative tier-migration cycles run by the sidecar (meter_banyandb_instance_lifecycle_migration_cycles).\nLiaison front door (container_name = liaison)\nQuery Rate by Service — gRPC query calls per second, split by data-model service (measure / stream / trace / property) (meter_banyandb_instance_liaison_query_rate).\ngRPC Errors / min — gRPC errors per minute, summed across total + registry + stream-msg (meter_banyandb_instance_liaison_grpc_error_rate). Lazily registered, so it reads 0 on a healthy liaison.\nRegistry Ops / s — schema-registry operations per second at the front door (meter_banyandb_instance_liaison_registry_op_rate).\nWrite Rate — writes per second at the front door across the three data models (meter_banyandb_instance_liaison_write_rate).\nPublish Throughput — the tier-2 publish pipeline (liaison → data) broken out by operation (meter_banyandb_instance_liaison_publish_throughput).\nPublish p99 Latency — p99 send latency of the publish pipeline, per operation (meter_banyandb_instance_liaison_publish_latency_p99).\nPart-sync Bytes — bytes per second streamed to data nodes on the part-sync (file-sync) path in KB/s (meter_banyandb_instance_liaison_publish_bytes). Only chunked file-sync increments this counter; regular write / query publishes are not counted.\nWrite Queue Pending — liaison write-buffer depth: records buffered at the front door before publish (meter_banyandb_instance_liaison_wqueue_pending).\nPublish Batch Throughput — batches published per second by operation (meter_banyandb_instance_liaison_publish_batch_throughput). Hidden until the cluster emits batch metrics.\nPublish Batch p99 — p99 latency of batch publishes in ms (meter_banyandb_instance_liaison_publish_batch_latency_p99).\nData node (container_name = data)\nStored Data Elements — total file elements stored across measure + stream + trace (meter_banyandb_instance_data_total_data).\nWrite Queue (wqueue) — the data-node write queue: pending records, on-disk file parts, and in-memory parts (meter_banyandb_instance_data_wqueue_pending, meter_banyandb_instance_data_wqueue_file_parts, meter_banyandb_instance_data_wqueue_mem_part).\nMerge Loop Rate — file merge-loop iterations per second (meter_banyandb_instance_data_merge_file_rate).\nMerge File Latency — average on-disk file-merge latency per merge loop in ms (meter_banyandb_instance_data_merge_file_latency).\nMerge Parts / Loop — average parts merged per on-disk merge loop (meter_banyandb_instance_data_merge_file_partitions).\nInverted Index Rate — series-index updates and term searches per second across measure + stream storage + stream tst (meter_banyandb_instance_data_series_write_rate, meter_banyandb_instance_data_series_term_search_rate, meter_banyandb_instance_data_stream_tst_write_rate, meter_banyandb_instance_data_stream_tst_term_search_rate).\nIndex Documents — total inverted-index documents, used as a series proxy (meter_banyandb_instance_data_total_series, meter_banyandb_instance_data_stream_tst_total_docs).\nSubscribe Throughput — subscribe-side queue throughput by operation (query / file-sync / batch-write / control) (meter_banyandb_instance_data_queue_sub_throughput).\nSubscribe p99 Latency — p99 latency of subscribe-side queue processing in ms (meter_banyandb_instance_data_queue_sub_latency_p99).\nRetention Disk Usage — per data-model retention disk-usage percentage (meter_banyandb_instance_data_retention_measure_disk_usage_percent, meter_banyandb_instance_data_retention_stream_disk_usage_percent, meter_banyandb_instance_data_retention_trace_disk_usage_percent).\nSubscribe Message Throughput — per-record processing rate the subscriber unpacks from batches in msgs/s (meter_banyandb_instance_data_queue_sub_message_throughput).\nGroup dashboard For one selected group — a BanyanDB storage group, mapped to the endpoint slot. The widgets are organized by data model (measure, stream, trace, property); each model\u0026rsquo;s widgets render only when that model\u0026rsquo;s group reports data, and a final set of queue widgets is common to every group.\nMeasure\nMeasure Write / s — writes per second for this group (meter_banyandb_endpoint_measure_write_rate).\nMeasure Query Latency — mean liaison query latency for this group in ms (meter_banyandb_endpoint_measure_query_latency).\nMeasure Total Data — current stored data elements for this group (meter_banyandb_endpoint_measure_total_data).\nMeasure Merge Rate — file merge-loop iterations per minute (meter_banyandb_endpoint_measure_merge_file_rate).\nMeasure Merge Latency — mean file-merge latency per merge loop in ms (meter_banyandb_endpoint_measure_merge_file_latency).\nMeasure Merge Partitions — average parts merged per file merge loop (meter_banyandb_endpoint_measure_merge_file_partitions).\nMeasure Series Write / s — inverted-index updates per second, used as a series-write proxy (meter_banyandb_endpoint_measure_series_write_rate).\nMeasure Term Search / s — inverted-index term-search invocations per second, the index read-pressure signal (meter_banyandb_endpoint_measure_series_term_search_rate).\nMeasure Total Series — total inverted-index documents for this group, used as a series proxy (meter_banyandb_endpoint_measure_total_series).\nStream\nStream Write / s — writes per second for this group (meter_banyandb_endpoint_stream_write_rate).\nStream Query Latency — mean liaison query latency for this group in ms (meter_banyandb_endpoint_stream_query_latency).\nStream Total Data — current stored data elements for this group (meter_banyandb_endpoint_stream_total_data).\nStream Merge Rate — file merge-loop iterations per minute (meter_banyandb_endpoint_stream_merge_file_rate).\nStream Merge Latency — mean file-merge latency per merge loop in ms (meter_banyandb_endpoint_stream_merge_file_latency).\nStream Merge Partitions — average parts merged per file merge loop (meter_banyandb_endpoint_stream_merge_file_partitions).\nStream Series Write / s — inverted-index updates per second, used as a series-write proxy (meter_banyandb_endpoint_stream_series_write_rate).\nStream TST Index Write / s — stream tst-scope inverted-index updates per second, distinct from the storage-scope series index (meter_banyandb_endpoint_stream_tst_index_write_rate).\nStream Term Search / s — inverted-index term-search invocations per second (meter_banyandb_endpoint_stream_series_term_search_rate).\nStream Total Series — total inverted-index documents for this group (meter_banyandb_endpoint_stream_total_series).\nStream TST Total Series — the stream tst-scope index document total, distinct from the storage-scope series index (meter_banyandb_endpoint_stream_tst_total_series).\nTrace\nTrace Write / s — writes per second for this group (meter_banyandb_endpoint_trace_write_rate).\nTrace Query Latency — mean liaison query latency for this group in ms (meter_banyandb_endpoint_trace_query_latency).\nTrace Total Data — current stored data elements for this group (meter_banyandb_endpoint_trace_total_data).\nTrace Merge Rate — file merge-loop iterations per minute (meter_banyandb_endpoint_trace_merge_file_rate).\nTrace Merge Latency — mean file-merge latency per merge loop in ms (meter_banyandb_endpoint_trace_merge_file_latency).\nTrace Merge Partitions — average parts merged per file merge loop (meter_banyandb_endpoint_trace_merge_file_partitions).\nTrace Series Write / s — inverted-index updates per second, used as a series-write proxy (meter_banyandb_endpoint_trace_series_write_rate).\nTrace Term Search / s — inverted-index term-search invocations per second (meter_banyandb_endpoint_trace_series_term_search_rate).\nTrace Total Series — total inverted-index documents for this group (meter_banyandb_endpoint_trace_total_series).\nProperty\nProperty Index Write / s — property-registry inverted-index updates per second, the property model\u0026rsquo;s write signal (meter_banyandb_endpoint_property_index_write_rate).\nProperty Index Merge Rate — property inverted-index segment merges per minute; property has no tst merge loop (meter_banyandb_endpoint_property_index_merge_rate).\nProperty Index Merge Latency — mean property inverted-index merge latency in ms (meter_banyandb_endpoint_property_index_merge_latency).\nProperty Term Search / s — property term-search invocations per second; property is read via the registry / term-search path rather than the liaison query method, so this is its read-load signal (meter_banyandb_endpoint_property_series_term_search_rate).\nProperty Total Series — total property inverted-index documents for this group (meter_banyandb_endpoint_property_total_series).\nQueue (every group)\nSubscribe Throughput — subscribe-side queue messages per second for this group, by operation (meter_banyandb_endpoint_queue_throughput).\nPublish p99 — publish-side queue p99 latency for this group in ms (meter_banyandb_endpoint_queue_latency_p99).\nBatch Throughput — per-group write-batch rate, by operation (meter_banyandb_endpoint_queue_batch_throughput).\nMessage Throughput — per-group per-record rate, by operation (meter_banyandb_endpoint_queue_message_throughput).\nPart-sync Bytes / s — part-sync (file-sync) bytes per second for this group in KB/s (meter_banyandb_endpoint_publish_bytes). Only the chunked part-streaming path increments this; regular write / query publishes are not counted.\nTrace Sampling An extension page under Cluster, reached from its own sidebar row directly below the Cluster row (route /layer/BANYANDB/service/trace-sampling). It covers BanyanDB trace tail sampling: from 0.11 a data node can run an ordered chain of sampler plugins during trace merge and finalization, dropping whole traces after their fragments have already landed. See Trace Tail Sampling for how a trace is judged and how the chain is configured in OAP\u0026rsquo;s bydb.yml.\nThe plugin chain is optional, and this page is honest about that. Only the Active Samplers table is always shown; every other panel is gated on one shared probe, so on a cluster with no sampler configured the page collapses to that single table reading no data — rather than two dozen empty charts — and OAP runs one gate query instead of the whole page\u0026rsquo;s MQE.\nEverything is per storage group. Every sampling metric is keyed on the group, so each panel charts one series per group and none of them sum across groups. There are deliberately no headline cards: a card collapses its series with an unweighted mean of the per-step values, which reads correctly for a rate but distorts a ratio, and summarising a metric a panel already plots earns nothing. Trace Outcomes leads instead, at full width. That is also why this page lives under Cluster rather than under Group — one page covers every group at once.\nThe recurring distinction to hold on to while reading it: the sampler plugins propose drops, and storage commits them. A trace can be evaluated across several merge or finalization rounds, and BanyanDB\u0026rsquo;s safety guards can retain a trace a plugin voted to drop — so the two never have to match, and the gap between them is what most of these panels exist to explain.\nA panel showing nothing is usually good news rather than a fault. The families behind the failure and edge-case metrics are registered on first use, so a counter that never fired is never exposed; on a healthy cluster that leaves the load-failure, chain-error, link-bypass, drop-set-ceiling, finalization and telemetry-safety panels empty.\nOutcomes Trace Outcomes — Retained + dropped account for the evaluated traces. Immature traces were never eligible — their fragments are still inside the maturity boundary, so they are not a sampling failure. (meter_banyandb_trace_sampling_traces_dropped / _traces_evaluated / _traces_immature / _traces_retained)\nActive Samplers — Sampler plugins registered per storage group. A group absent from this list has no pipeline — its traces are never evaluated, so nothing is dropped for it. (meter_banyandb_trace_sampling_active_samplers) A table, one row per group.\nDrop Ratio by Group — Committed drop ratio per storage group, per step — dropped over evaluated traces after BanyanDB\u0026rsquo;s safety guards. Deliberately per group rather than one cluster headline: a cluster-wide card would be an unweighted mean of the per-step ratios, so a quiet minute would count as much as a busy one. A group below the others is usually explained by the safety events further down rather than by its rules. (meter_banyandb_trace_sampling_drop_ratio)\nPlugin Load Failures — Plugin load failures per minute, by group, failing plugin and reason. Reconciliation fails open, so any value here means the chain is running WITHOUT a plugin the config asked for — sampling is not doing what it says. The plugin name and reason are only shown here. (meter_banyandb_trace_sampling_plugin_load_failures)\nPlugin execution Decide Rate by Result — Decide() calls per second, per plugin and result. Alert on any panic, length_mismatch or late — they mean broken plugin behaviour or a call that outlived the merge timeout. decide_error may be transient. (meter_banyandb_trace_sampling_plugin_execution_rate)\nDecide Latency — Successful Decide() wall time per plugin — p99 takes the worst group, mean averages them. Derive a per-plugin SLO from a healthy baseline; an intentionally expensive plugin is not a fault. (meter_banyandb_trace_sampling_plugin_decide_latency / _plugin_decide_latency_p99)\nPlugin Time per Trace — Whole-chain plugin wall time per evaluated trace, per group — the sampling overhead paid on merge, over time. The upstream Grafana board splits this per plugin with PromQL\u0026rsquo;s group_left; MAL has no equivalent, so the metric is the chain total and the per-plugin split lives in Decide Latency. Keep it comfortably below the data node\u0026rsquo;s \u0026ndash;trace-pipeline-decide-timeout (default 5s). (meter_banyandb_trace_sampling_plugin_time_per_trace)\nChain Batches — Chain batches per second by result. circuit_open means sampling is currently BYPASSED for that group; timeout is chain-level because the host can abandon the chain before a link returns. The traces each batch carried is the Batch Size p99 panel. (meter_banyandb_trace_sampling_chain_batch_rate)\nBatch Size p99 — p99 traces presented to the plugin chain per successful batch, per group. Read with Chain Batches: rate tells you how often the chain runs, this tells you how much it sees each time. (meter_banyandb_trace_sampling_batch_size_p99)\nSampler decisions Published by the first-party sw-trace-sampler / zipkin-trace-sampler plugins — these are proposed drops, so read them against Trace Outcomes.\nSampler Decisions by Rule — What the first-party samplers PROPOSED, attributed to the first matching rule — not what storage committed. A trace can be evaluated over several merge or finalization rounds, and the guards can retain a trace the plugin voted to drop, so read drop verdicts against Trace Outcomes. (meter_banyandb_trace_sampling_sampler_decisions)\nSampler Rows — Span / segment rows carried by evaluated traces, against the rows the samplers proposed dropping, broken out by the rule responsible. Same unit, so the two are directly comparable; the resulting share is the Dropped Row Ratio panel. (meter_banyandb_trace_sampling_sampler_rows / _sampler_rows_dropped)\nDropped Row Ratio — Share of rows the samplers proposed dropping, per group. Proposed, not committed — the safety guards can retain a trace a plugin voted to drop. Treat it as incomplete whenever Row Count Unavailable is non-zero, since those traces are missing from the denominator. (meter_banyandb_trace_sampling_sampler_dropped_row_ratio)\nRow Count Unavailable — Trace evaluations whose metadata-only projection exposed no row count, per group. These traces are omitted from the row totals, so any non-zero value here means the Dropped Row Ratio is computed over an incomplete denominator. (meter_banyandb_trace_sampling_sampler_row_count_unavailable)\nSafety and fail-open Every panel in this block explains a drop ratio that came in under target — each one retains data a sampler proposed dropping.\nSafety Events — Every series here RETAINS data a sampler proposed dropping. A drop ratio under target is explained here before it is explained by plugin logic. (meter_banyandb_trace_sampling_ambiguous / _guard_budget_exhausted / _guard_bypassed / _guard_deferred / _guard_lossless_retry / _guard_publication_rejected / _oversized_traces_bypassed)\nChain Errors by Reason — Chain-level errors that forced fail-open retention of the whole batch. (meter_banyandb_trace_sampling_plugin_errors)\nLink Bypasses — Individual links bypassed after decide_error, length_mismatch or panic. A chain batch can report success while one link was bypassed and its input retained — which is why this is separate from chain errors. (meter_banyandb_trace_sampling_link_bypasses)\nGuard \u0026amp; Index Work — Bloom-filter probes are the fragment guard checking for trace fragments outside the parts being merged; pruned entries are the secondary-index work a confirmed drop causes. (meter_banyandb_trace_sampling_guard_bloom_probes / _sidx_pruned)\nDrop-set capacity Drop-set Cap Impact — The per-merge dropped-trace-ID set is memory-bounded. When it hits the ceiling the merge retains the rest — this is what makes a drop ratio plateau under load rather than tracking the rules. (meter_banyandb_trace_sampling_capped_merges / _traces_retained_by_ceiling)\nDrop-set Size p99 — p99 dropped trace IDs held per merge lane. Read it against the Drop-set Budget panel: when the set outgrows the budget the merge stops accepting drops and retains the rest, which is what shows up as Drop-set Cap Impact. (meter_banyandb_trace_sampling_drop_set_entries_p99)\nDrop-set Budget — Resolved per-merge memory budget for confirmed dropped trace IDs, per group — the ceiling the drop-set size runs into. Resolved per group, so groups can carry different budgets; a group with a smaller one reaches its ceiling first. Flat unless the data node is retuned. (meter_banyandb_trace_sampling_drop_set_budget_bytes)\nLifecycle and host limits Registrations \u0026amp; Updates — Pipeline registration, update and removal rate. A rejected result means OAP pushed a pipeline config the data node would not accept. (meter_banyandb_trace_sampling_register_rate / _remove_rate / _update_rate)\nFinalization Rounds — Highest finalization round count observed across each group\u0026rsquo;s cooled shards. Whether a shard has run out of rounds is the Finalization Terminal panel. (meter_banyandb_trace_sampling_finalize_rounds)\nFinalization Terminal — 1 when any cooled shard in the group is terminal and cannot run another finalization round — no more tail sampling will happen for that data, so whatever is stored is final. 0 otherwise. Charted over time so the transition is visible. (meter_banyandb_trace_sampling_finalize_terminal)\nPlugin Telemetry Host Safety — The host caps what a plugin may publish (100 label-value series per instrument) and rate-limits its logs. Non-zero means a plugin\u0026rsquo;s OWN telemetry is truncated — the sampler decision metrics above are then incomplete, though sampling itself is unaffected. (meter_banyandb_trace_sampling_plugin_log_dropped / _plugin_telemetry_panic / _plugin_telemetry_series_rejected)\nDeployment The BANYANDB layer enables the layer-specific Deployment tab — the deployment topology of one cluster\u0026rsquo;s own containers and the intra-cluster calls between them. Pick a cluster from the header and the tab draws its containers as health-ring nodes laid out left → right along the calls between them, with animated edge flow, a per-edge metric panel, and a node popover that opens the container dashboard. For how the Deployment tab is read and navigated in general, see the Deployment section of Layer Dashboard Templates.\nContainers are grouped into three roles by their node_role / node_type attributes:\nLiaison — the front door. Its node center shows Query/s (meter_banyandb_instance_liaison_query_rate) and its health ring tracks gRPC err/min (meter_banyandb_instance_liaison_grpc_error_rate).\nData — the storage nodes. Center shows Ingest/s (meter_banyandb_instance_data_queue_sub_throughput) and the ring tracks Disk % (meter_banyandb_instance_disk_usage_percent).\nLifecycle — the tier-migration sidecar. Center shows cumulative Cycles (meter_banyandb_instance_lifecycle_migration_cycles) and the ring tracks Last OK (meter_banyandb_instance_lifecycle_last_run_success).\nBecause role-pair edges are configured, the Deployment map gains a Flows sub-tab listing every edge grouped by role-pair. Each edge type carries its own client-side (publish) and server-side (subscribe) metrics, so a liaison → data call surfaces a different metric set than a liaison → liaison forward or a lifecycle → data migration:\nliaison → data — the main write / query path. Per-operation Write/s, Query/s, and Part-sync/s throughput; Write p99 and Query p99 latency; Part-sync B/s bytes; and Errors/s. Each is paired across the publish side (meter_banyandb_instance_relation_publish_*, filtered by operation) and the subscribe side (meter_banyandb_instance_relation_queue_sub_*).\nliaison → liaison — node-to-node forwarding. Forward/s and Forward p99 for the batch-write forward, Control/s for the control channel, and Errors/s (meter_banyandb_instance_relation_publish_throughput{operation='batch-write'} and the matching subscribe / control / error counters).\nlifecycle → data — the tier-migration path. Migrate/s throughput, Migrate p99 latency, Migrate B/s bytes, and Errors/s (meter_banyandb_instance_relation_migration_* on the publish side, meter_banyandb_instance_relation_queue_sub_* on the subscribe side).\nany other pair — a generic fallback showing aggregated Msg/s and p99 (aggregate_labels(meter_banyandb_instance_relation_publish_throughput,sum) and the matching latency / subscribe counters), so an edge that matches no specific role-pair still reports something.\nRequirements The BANYANDB dashboard is a pure consumer of what OAP reports about its BanyanDB storage tier — it invents no data, and a widget with no backing data simply reads no data (or 0 for the lazily-registered error counters). To populate it, OAP needs BanyanDB self-observability enabled so that BanyanDB exposes its metrics and OAP ingests them into the meter_banyandb_* families:\nCluster metrics — meter_banyandb_cluster_* and the meter_banyandb_total_* capacity rollups for the Cluster list and Cluster dashboard.\nContainer metrics — meter_banyandb_instance_* for the per-container runtime, host, liaison, data, and lifecycle widgets. A container only shows the families its role emits, and the host widgets need a running system collector (absent on the lifecycle sidecar).\nGroup metrics — the per-data-model meter_banyandb_endpoint_* families (measure / stream / trace / property, plus the shared queue counters) for the Group dashboard.\nRelation metrics — meter_banyandb_instance_relation_* (publish / subscribe / migration throughput, latency, bytes, and error counters) for the Deployment tab\u0026rsquo;s edges.\nTrace sampling metrics — meter_banyandb_trace_sampling_* for the Trace Sampling page. These exist only while a sampler plugin chain is configured on the cluster, so the page is gated on them and stays collapsed to its Active Samplers card until one is enabled.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a container- or group-scope metric is empty until that level of data is reported, and an entire data model\u0026rsquo;s group widgets stay hidden until that model\u0026rsquo;s group reports.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/banyandb/","title":"\u003c!--"},{"body":" BookKeeper The BOOKKEEPER layer monitors Apache BookKeeper, the distributed write-ahead log storage that backs systems such as Apache Pulsar. OAP gathers BookKeeper\u0026rsquo;s metrics through the OpenTelemetry receiver and aggregates them per bookie node and per cluster.\nIn Horizon\u0026rsquo;s sidebar this layer is named BookKeeper. Its services are listed as BookKeeper clusters and its instances as Bookies — each bookie is one storage node in the cluster. This layer enables the Service and Instance scopes only: there is no endpoint scope, no topology, and no traces or logs tab, because BookKeeper reports node-level meters rather than request traffic.\nThis page is the operator reference for the bundled BOOKKEEPER dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled BOOKKEEPER template; if an operator has published a customized BOOKKEEPER template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every BookKeeper cluster with four sortable columns, sorted by Ledgers by default:\nLedgers — total ledgers held across the cluster\u0026rsquo;s bookies (meter_bookkeeper_bookie_ledgers_count, summed).\nEntries — total entries stored across the cluster\u0026rsquo;s bookies (meter_bookkeeper_bookie_entries_count, summed).\nWritable Dirs — number of ledger directories currently accepting writes (meter_bookkeeper_bookie_ledger_writable_dirs, summed).\nDir Usage — the ledger data directory\u0026rsquo;s fill level (meter_bookkeeper_bookie_ledger_dir_data_bookkeeper_ledgers_usage).\nService dashboard The primary drill-down for one selected BookKeeper cluster. Every widget aggregates the cluster\u0026rsquo;s bookies with aggregate_labels(..., sum).\nBookie Ledgers — ledgers held across the cluster over time (meter_bookkeeper_bookie_ledgers_count).\nBookie Entries — entries stored across the cluster over time (meter_bookkeeper_bookie_entries_count).\nWritable Ledger Dirs — ledger directories currently accepting writes (meter_bookkeeper_bookie_ledger_writable_dirs).\nWrite Cache — the bookie write cache on a dual axis: cache size in MB on the left (meter_bookkeeper_bookie_write_cache_size, divided to MB) and cached entry count on the right (meter_bookkeeper_bookie_write_cache_count).\nRead Cache — the bookie read cache on a dual axis: cache size in MB on the left (meter_bookkeeper_bookie_read_cache_size, divided to MB) and cached entry count on the right (meter_bookkeeper_bookie_read_cache_count).\nRead / Write Rate (B/s) — bytes per second served and ingested, plotted together: read (meter_bookkeeper_bookie_read_rate) and write (meter_bookkeeper_bookie_write_rate).\nLedger Dir Usage — fill level of the ledger data directory over time (meter_bookkeeper_bookie_ledger_dir_data_bookkeeper_ledgers_usage).\nInstance dashboard For one selected bookie. These widgets cover the bookie\u0026rsquo;s JVM runtime and its internal thread pools.\nJVM Memory Pool (MB) — used memory per JVM memory pool in MB (meter_bookkeeper_node_jvm_memory_pool_used).\nJVM Memory (MB) — JVM memory in MB: used, committed, and init (meter_bookkeeper_node_jvm_memory_used, meter_bookkeeper_node_jvm_memory_committed, meter_bookkeeper_node_jvm_memory_init).\nJVM Threads — thread counts: current, daemon, peak, and deadlocked (meter_bookkeeper_node_jvm_threads_current, meter_bookkeeper_node_jvm_threads_daemon, meter_bookkeeper_node_jvm_threads_peak, meter_bookkeeper_node_jvm_threads_deadlocked).\nGC — garbage-collection activity on a dual axis: cumulative GC seconds on the left (meter_bookkeeper_node_jvm_gc_collection_seconds_sum) and GC count on the right (meter_bookkeeper_node_jvm_gc_collection_seconds_count).\nThread Executor — the bookie\u0026rsquo;s task executor: completed, tasks completed, rejected, and failed (meter_bookkeeper_node_thread_executor_completed, meter_bookkeeper_node_thread_executor_tasks_completed, meter_bookkeeper_node_thread_executor_tasks_rejected, meter_bookkeeper_node_thread_executor_tasks_failed).\nPooled Threads — thread counts for the high-priority and read pools (meter_bookkeeper_node_high_priority_threads, meter_bookkeeper_node_read_thread_pool_threads).\nPool Max Queue Size — the maximum queue size of the high-priority and read thread pools (meter_bookkeeper_node_high_priority_thread_max_queue_size, meter_bookkeeper_node_read_thread_pool_max_queue_size).\nRequirements The BOOKKEEPER dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nBookie metrics — the meter_bookkeeper_bookie_* family (ledgers, entries, writable directories, directory usage, read/write caches, and read/write rates), aggregated at the BookKeeper cluster (Service) scope.\nBookie node metrics — the meter_bookkeeper_node_* family (JVM memory, threads, and GC, plus the bookie\u0026rsquo;s thread executor and thread pools), reported at the bookie (ServiceInstance) scope.\nThese metrics come from BookKeeper\u0026rsquo;s own OpenTelemetry export, gathered by OAP\u0026rsquo;s OpenTelemetry receiver — see the BookKeeper monitoring setup. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the instance-scope widgets stay empty until per-bookie data is reported.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/bookkeeper/","title":"\u003c!--"},{"body":" Browser The BROWSER layer is where SkyWalking\u0026rsquo;s browser agent (the client-side JavaScript SDK) reports. It is real-user monitoring: page views, front-end errors, page-load timing, and Core Web Vitals collected from the visitor\u0026rsquo;s browser rather than from a server-side agent.\nIn Horizon\u0026rsquo;s sidebar this layer is named Browser. Its top-level entities are web applications, listed as Apps; each app reports under one or more Versions (the instance slot), and each app serves a set of Pages (the endpoint slot). So where the GENERAL layer reads \u0026ldquo;Service / Instance / Endpoint\u0026rdquo;, BROWSER reads \u0026ldquo;App / Version / Page\u0026rdquo;. The layer enables the App, Version, and Page dashboards, plus the Traces tab and a Browser Logs tab — the per-page front-end error stream, which can de-obfuscate a minified JavaScript stack against a source map you upload. BROWSER has no service topology; there is no map view.\nThis page is the operator reference for the bundled BROWSER dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled BROWSER template; if an operator has published a customized BROWSER template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nApp list Before opening an app, the layer landing page lists every BROWSER app with three columns, sorted by traffic (Page Views) by default:\nPage Views — page views per minute (browser_app_pv).\nError Rate — percent of page views that recorded an error (browser_app_error_rate/100).\nErrors — total front-end errors in the window (browser_app_error_sum).\nApp dashboard The primary drill-down for one selected app.\nApp Load (PV) — page views per minute for the app (browser_app_pv).\nApp Error Rate — percent of page views that recorded an error (browser_app_error_rate/100).\nApp Error Count — total front-end errors per minute (browser_app_error_sum).\nTop Hot Pages — the app\u0026rsquo;s busiest pages, with tabs to re-rank by PV (browser_app_page_pv, /min), Errors (browser_app_page_error_sum), and Error Rate (browser_app_page_error_rate, %), worst-first. Click a row to jump into that page.\nTop Versions — the app\u0026rsquo;s versions broken down the same three ways: PV (browser_app_single_version_pv, /min), Errors (browser_app_single_version_error_sum), and Error Rate (browser_app_single_version_error_rate, %).\nVersion dashboard For one selected app Version — the per-release view of the same load and error signals.\nVersion PV — page views per minute for this version (browser_app_single_version_pv).\nVersion Error Rate — percent of this version\u0026rsquo;s page views that recorded an error (browser_app_single_version_error_rate/100).\nVersion Error Count — total front-end errors per minute for this version (browser_app_single_version_error_sum).\nPage dashboard For one selected Page — the deepest scope, where browser timing and Web Vitals live. This is the page-performance view: most of these metrics exist only at page scope.\nFirst Meaningful Paint Percentile — p50 / p75 / p90 / p95 / p99 of FMP latency for the page, in ms (browser_app_page_fmp_percentile). Below 1s at p75 is a common target.\nPage Load Percentile — p50 / p75 / p90 / p95 / p99 of full page-load time, in ms (browser_app_page_load_page_percentile).\nTime-to-Live Percentile — p50 / p75 / p90 / p95 / p99 of the page\u0026rsquo;s time-to-live, in ms (browser_app_page_ttl_percentile).\nFirst Pack Latency Percentile — p50 / p75 / p90 / p95 / p99 of first-pack latency, in ms (browser_app_page_first_pack_percentile).\nPage Performance Breakdown — average time spent in each phase of the page load, in ms, on one chart: DNS, redirect, TCP, TTFB, transfer, DOM analysis, DOM ready, FPT, load, and resource (browser_app_page_dns_avg, browser_app_page_redirect_avg, browser_app_page_tcp_avg, browser_app_page_ttfb_avg, browser_app_page_trans_avg, browser_app_page_dom_analysis_avg, browser_app_page_dom_ready_avg, browser_app_page_fpt_avg, browser_app_page_load_page_avg, browser_app_page_res_avg).\nPage Errors by Type — front-end error counters per minute split by source: resource, JS, AJAX, and unknown (browser_app_page_resource_error_sum, browser_app_page_js_error_sum, browser_app_page_ajax_error_sum, browser_app_page_unknown_error_sum).\nWeb Vitals — Core Web Vitals as averages per minute: FMP (ms), LCP (ms), and CLS (browser_app_web_vitals_fmp_avg, browser_app_web_vitals_lcp_avg, browser_app_web_vitals_cls_avg / 1000 — CLS is scaled down to its typical 0 – 1 score range).\nInteraction to Next Paint Percentile — p50 / p75 / p90 / p95 / p99 of INP, in ms (browser_app_web_interaction_inp_percentile). INP is the responsiveness metric that replaces FID.\nRequirements The BROWSER dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, your front-end must run the SkyWalking browser agent (the client-side JavaScript SDK) reporting to OAP, which produces:\nApp metrics — the browser_app_* family (page views, error rate, error sum), produced by OAP from browser-agent reports, for the App list and App dashboard.\nVersion metrics — browser_app_single_version_* (PV, error rate, error sum) for the Top Versions widget and the Version dashboard.\nPage metrics — browser_app_page_* (PV, errors, the timing percentiles, and the per-phase performance averages) and the browser_app_web_vitals_* / browser_app_web_interaction_* families for the Page dashboard.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a version- or page-scope metric is empty until that level of data is reported. BROWSER carries no relation metrics, so it has no topology or map view.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/browser/","title":"\u003c!--"},{"body":" Cilium Service The CILIUM_SERVICE layer monitors Kubernetes services observed through Cilium\u0026rsquo;s eBPF data plane. SkyWalking collects L4 (TCP) packet activity and L7 protocol telemetry (HTTP, DNS, Kafka) that Cilium reports for each service, giving you network-level and protocol-level visibility into the mesh without instrumenting the application. It belongs to the Kubernetes layer group.\nIn Horizon\u0026rsquo;s sidebar this layer is named Cilium Service. Its services are listed as Services, instances as Pods, and endpoints as Endpoints — service names are grouped and displayed by their Kubernetes namespace. The CILIUM_SERVICE layer enables the Service, Pod, Endpoint, and Topology sub-tabs. It does not enable Traces or Logs.\nThis page is the operator reference for the bundled CILIUM_SERVICE dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled CILIUM_SERVICE template; if an operator has published a customized CILIUM_SERVICE template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a service, the layer landing page lists every CILIUM_SERVICE service with four sortable columns, sorted by traffic (RPM) by default:\nRPM — protocol calls per minute (cilium_service_protocol_cpm).\nLatency — average protocol call duration in ms (cilium_service_protocol_call_duration/cilium_service_protocol_cpm/1000000).\nSuccess Rate — percent of successful protocol calls (cilium_service_protocol_call_success_count/cilium_service_protocol_cpm*100).\nTCP Drop — dropped read + write packets per minute at L4 (cilium_service_l4_read_pkg_drop_cpm + cilium_service_l4_write_pkg_drop_cpm).\nService dashboard The primary drill-down for one selected service. It splits into an L4 (TCP) row and per-protocol (HTTP, DNS, Kafka) groups.\nL4 (TCP)\nL4 Read Packages/min — inbound packets per minute (cilium_service_l4_read_pkg_cpm).\nL4 Write Packages/min — outbound packets per minute (cilium_service_l4_write_pkg_cpm).\nTCP Drop / min — dropped read and write packets per minute, plotted as two series (cilium_service_l4_read_pkg_drop_cpm, cilium_service_l4_write_pkg_drop_cpm).\nTCP Drop by Reason — dropped-packet count broken out by Cilium\u0026rsquo;s drop-reason label (cilium_service_l4_drop_reason_count).\nHTTP\nHTTP Load — HTTP calls per minute alongside the successful-call count (cilium_service_protocol_http_call_cpm, cilium_service_protocol_http_call_success_count).\nHTTP Duration — average HTTP call duration in ms (cilium_service_protocol_http_call_duration/cilium_service_protocol_http_call_cpm/1000000).\nHTTP Non-2xx Status — HTTP responses per minute by status class: 1xx / 3xx / 4xx / 5xx (cilium_service_protocol_http_status_1xx_cpm, cilium_service_protocol_http_status_3xx_cpm, cilium_service_protocol_http_status_4xx_cpm, cilium_service_protocol_http_status_5xx_cpm).\nDNS\nDNS Load — DNS calls per minute alongside the successful-call count (cilium_service_protocol_dns_call_cpm, cilium_service_protocol_dns_call_success_count).\nDNS Duration — average DNS call duration in ms (cilium_service_protocol_dns_call_duration/cilium_service_protocol_dns_call_cpm/1000000).\nDNS Errors / min — DNS error count per minute (cilium_service_protocol_dns_error_count).\nKafka\nKafka Load — Kafka calls per minute alongside the successful-call count (cilium_service_protocol_kafka_call_cpm, cilium_service_protocol_kafka_call_success_count).\nKafka Duration — average Kafka call duration in ms (cilium_service_protocol_kafka_call_duration/cilium_service_protocol_kafka_call_cpm/1000000).\nKafka Errors / min — Kafka error count per minute (cilium_service_protocol_kafka_call_error_count).\nPod dashboard For one selected pod (a service instance). The widgets mirror the service view at instance scope, covering L4 plus the HTTP, DNS, and Kafka protocols.\nL4 Read Packages/min — inbound packets per minute for the pod (cilium_service_instance_l4_read_pkg_cpm).\nL4 Write Packages/min — outbound packets per minute for the pod (cilium_service_instance_l4_write_pkg_cpm).\nHTTP Load — HTTP calls per minute alongside the successful-call count (cilium_service_instance_protocol_http_call_cpm, cilium_service_instance_protocol_http_call_success_count).\nHTTP Non-2xx Status — HTTP responses per minute by status class: 1xx / 3xx / 4xx / 5xx (cilium_service_instance_protocol_http_status_1xx_cpm, cilium_service_instance_protocol_http_status_3xx_cpm, cilium_service_instance_protocol_http_status_4xx_cpm, cilium_service_instance_protocol_http_status_5xx_cpm).\nDNS Load — DNS calls per minute alongside the successful-call count (cilium_service_instance_protocol_dns_call_cpm, cilium_service_instance_protocol_dns_call_success_count).\nDNS Errors — DNS error count (cilium_service_instance_protocol_dns_error_count).\nKafka Load — Kafka calls per minute alongside the successful-call count (cilium_service_instance_protocol_kafka_call_cpm, cilium_service_instance_protocol_kafka_call_success_count).\nEndpoint dashboard For one selected endpoint. Cilium endpoints carry L7 protocol traffic, so this scope is HTTP- and DNS-focused.\nHTTP Load — HTTP calls per minute alongside the successful-call count (cilium_endpoint_protocol_http_call_cpm, cilium_endpoint_protocol_http_call_success_count).\nHTTP Duration — average HTTP call duration in ms (cilium_endpoint_protocol_http_call_duration/cilium_endpoint_protocol_http_call_cpm/1000000).\nHTTP Non-2xx Status — HTTP responses per minute by status class: 1xx / 3xx / 4xx / 5xx (cilium_endpoint_protocol_http_status_1xx_cpm, cilium_endpoint_protocol_http_status_3xx_cpm, cilium_endpoint_protocol_http_status_4xx_cpm, cilium_endpoint_protocol_http_status_5xx_cpm).\nDNS Load — DNS calls per minute alongside the successful-call count (cilium_endpoint_protocol_dns_call_cpm, cilium_endpoint_protocol_dns_call_success_count).\nDNS Duration — average DNS call duration in ms (cilium_endpoint_protocol_dns_call_duration/cilium_endpoint_protocol_dns_call_cpm/1000000).\nDNS Errors — DNS error count (cilium_endpoint_protocol_dns_error_count).\nTopology and maps The CILIUM_SERVICE layer ships a service topology with an instance-level drill-down.\nTopology (service map) — each service node is decorated with RPM (cilium_service_protocol_cpm), a Success % health ring (cilium_service_protocol_call_success_count/cilium_service_protocol_cpm*100, colored green / amber / red — higher is better, so the ring turns red as the success rate drops), and Latency (cilium_service_protocol_call_duration/cilium_service_protocol_cpm/1000000). Each call edge carries server-side HTTP RPM (cilium_service_relation_server_protocol_http_call_cpm) and Avg Latency (cilium_service_relation_server_protocol_http_call_duration/cilium_service_relation_server_protocol_http_call_cpm/1000000).\nInstance map — from a call between two services on the service map, drill into the pod-to-pod calls between them. Each instance node shows HTTP RPM (cilium_service_instance_protocol_http_call_cpm) and Avg Latency (cilium_service_instance_protocol_http_call_duration/cilium_service_instance_protocol_http_call_cpm/1000000); each edge carries server-side HTTP RPM (cilium_service_instance_relation_server_protocol_http_call_cpm) and Avg Latency (cilium_service_instance_relation_server_protocol_http_call_duration/cilium_service_instance_relation_server_protocol_http_call_cpm/1000000).\nFor how these maps are read and navigated, see the 3D Infrastructure Map for the cross-layer view, and the topology section of Layer Dashboard Templates for how the node and edge metrics are configured.\nRequirements The CILIUM_SERVICE dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Cilium monitoring enabled, with the Cilium-Hubble fetcher feeding SkyWalking. Specifically:\nL4 metrics — the cilium_service_l4_* family (read / write packet rates, packet drops, and drop-reason breakdown) for the service-scope TCP widgets.\nProtocol metrics — the cilium_service_protocol_*, cilium_service_instance_protocol_*, and cilium_endpoint_protocol_* families covering HTTP, DNS, and Kafka call counts, durations, success counts, status classes, and errors, at their respective service / instance / endpoint scopes.\nRelation metrics — cilium_service_relation_server_protocol_* and cilium_service_instance_relation_server_protocol_* for the service map and instance map edges.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so an instance- or endpoint-scope metric is empty until that level of data is reported. A pod or endpoint that carries only one protocol shows no data for the others. For setup, see the Cilium monitoring guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/cilium_service/","title":"\u003c!--"},{"body":" ClickHouse The CLICKHOUSE layer monitors ClickHouse database clusters. SkyWalking collects ClickHouse\u0026rsquo;s internal metrics — queries, query latency, merges and mutations, data parts, replication, ZooKeeper / Keeper coordination, and per-node host stats — through OpenTelemetry, and Horizon renders them as a cluster-level and a node-level dashboard. This is a metrics-only layer: there are no traces, logs, endpoints, or a topology map for ClickHouse.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Databases and named ClickHouse. Its services are listed as ClickHouse clusters and its instances as Nodes. Because the layer carries no endpoint, topology, trace, or log data, it enables only two sub-tabs: a Service (cluster) dashboard and an Instance (node) dashboard.\nThis page is the operator reference for the bundled CLICKHOUSE dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled CLICKHOUSE template; if an operator has published a customized CLICKHOUSE template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every ClickHouse cluster with four sortable columns, sorted by select rate (Select / s) by default:\nSelect / s — SELECT queries per second across the cluster (aggregate_labels(meter_clickhouse_query_select_rate,sum)).\nInsert / s — INSERT queries per second across the cluster (aggregate_labels(meter_clickhouse_query_insert_rate,sum)).\nSlow Reads — slow file reads across the cluster (aggregate_labels(meter_clickhouse_query_slow,sum)).\nOpen Files — the latest count of open files across the cluster (latest(aggregate_labels(meter_clickhouse_file_open,sum))).\nService dashboard The cluster-level drill-down for one selected ClickHouse cluster. Every widget aggregates across the cluster\u0026rsquo;s nodes with aggregate_labels(...,sum).\nFiles Open — the latest number of open files in the cluster, as a single card (latest(aggregate_labels(meter_clickhouse_file_open,sum))).\nQPS — query rate per second, plotted as two series: select (aggregate_labels(meter_clickhouse_query_select_rate,sum)) and insert (aggregate_labels(meter_clickhouse_query_insert_rate,sum)).\nQueries — query counts split into total, select, and insert (aggregate_labels(meter_clickhouse_query,sum), aggregate_labels(meter_clickhouse_query_select,sum), aggregate_labels(meter_clickhouse_query_insert,sum)).\nQuery Time (ms) — average time per query in ms, as avg, select, and insert series, each computed as total query microseconds divided by query count and converted to ms (aggregate_labels(meter_clickhouse_querytime_microseconds,sum)/aggregate_labels(meter_clickhouse_query,sum)/1000 and the matching _select_ / _insert_ pair).\nConnections — open client connections by protocol: TCP (aggregate_labels(meter_clickhouse_tcp_connections,sum)) and HTTP (aggregate_labels(meter_clickhouse_http_connections,sum)).\nSlow Reads — slow file reads across the cluster (aggregate_labels(meter_clickhouse_query_slow,sum)).\nMerge / Mutations — background merge operations (aggregate_labels(meter_clickhouse_background_merge,sum)) and mutations (aggregate_labels(meter_clickhouse_mutations,sum)).\nInsert Throughput — insert volume on a dual axis: bytes/s on the left (aggregate_labels(meter_clickhouse_inserted_bytes,sum)) and rows/s on the right (aggregate_labels(meter_clickhouse_inserted_rows,sum)).\nDelayed Inserts (s) — inserts that were throttled / delayed (aggregate_labels(meter_clickhouse_delayed_inserts,sum)).\nActive Data Parts — the number of active MergeTree data parts in the cluster (aggregate_labels(meter_clickhouse_parts_active,sum)).\nReplicated Fetch / Send — replication traffic between replicas: fetch (aggregate_labels(meter_clickhouse_replicated_fetch,sum)) and send (aggregate_labels(meter_clickhouse_replicated_send,sum)).\nZookeeper Activity — the coordination layer\u0026rsquo;s health, with the latest sessions and watches (latest(aggregate_labels(meter_clickhouse_zookeeper_session,sum)), latest(aggregate_labels(meter_clickhouse_zookeeper_watch,sum))) plus bytes sent and bytes recv over time (aggregate_labels(meter_clickhouse_zookeeper_bytes_sent,sum), aggregate_labels(meter_clickhouse_zookeeper_bytes_received,sum)).\nKeeper Alive Conns — the latest count of alive ClickHouse Keeper connections, as a single card (latest(aggregate_labels(meter_clickhouse_keeper_connections_alive,sum))).\nKeeper Outstanding Requests — the latest count of outstanding ClickHouse Keeper requests, as a single card (latest(aggregate_labels(meter_clickhouse_keeper_outstanding_requests,sum))).\nInstance dashboard The node-level drill-down for one selected ClickHouse node. These widgets read the per-node meter_clickhouse_instance_* family directly, with no cross-node aggregation.\nUptime (days) — how long the node has been running, as a card, converted from seconds (latest(meter_clickhouse_instance_uptime)/3600/24).\nVersion — the node\u0026rsquo;s ClickHouse version, as a card (latest(meter_clickhouse_instance_version)).\nCPU (cores) — CPU consumption expressed in cores (meter_clickhouse_instance_cpu_usage/1000000).\nMemory (%) — used vs available memory percentage (meter_clickhouse_instance_memory_usage, meter_clickhouse_instance_memory_available).\nNetwork (B) — bytes receive vs send on the node (meter_clickhouse_instance_network_receive_bytes, meter_clickhouse_instance_network_send_bytes).\nConnections — open client connections by protocol: TCP (meter_clickhouse_instance_tcp_connections) and HTTP (meter_clickhouse_instance_http_connections).\nQueries — query counts split into total, select, and insert (meter_clickhouse_instance_query, meter_clickhouse_instance_query_select, meter_clickhouse_instance_query_insert).\nQPS — query rate per second, as select and insert series (meter_clickhouse_instance_query_select_rate, meter_clickhouse_instance_query_insert_rate).\nQuery Time (ms) — average time per query in ms, as avg, select, and insert series, each total query microseconds divided by query count and converted to ms (meter_clickhouse_instance_querytime_microseconds/meter_clickhouse_instance_query/1000 and the matching _select_ / _insert_ pair).\nFile Slow Read — slow file reads on the node (meter_clickhouse_instance_query_slow).\nBackground Merge — background merge operations on the node (meter_clickhouse_instance_background_merge).\nMutations — mutation operations on the node (meter_clickhouse_instance_mutations).\nFiles Open — the latest number of open files on the node, as a card (latest(meter_clickhouse_instance_file_open)).\nInsert Throughput — insert volume on a dual axis: bytes/s on the left (meter_clickhouse_instance_inserted_bytes) and rows/s on the right (meter_clickhouse_instance_inserted_rows).\nDelayed Inserts (s) — inserts that were throttled / delayed on the node (meter_clickhouse_instance_delayed_inserts).\nRequirements The CLICKHOUSE dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs ClickHouse metrics delivered through the OpenTelemetry receiver, which OAP aggregates into the meter_clickhouse_* families:\nCluster (service-scope) metrics — the meter_clickhouse_* family (queries, query rate, query time, connections, slow reads, merges, mutations, data parts, insert throughput, delayed inserts, replication, ZooKeeper, and Keeper), which the cluster dashboard and the Service list aggregate across nodes.\nNode (instance-scope) metrics — the meter_clickhouse_instance_* family (uptime, version, CPU, memory, network, connections, queries, query rate, query time, slow reads, merges, mutations, open files, and insert throughput) for the node dashboard.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the node-scope widgets stay empty until per-node data is reported. Setting up the ClickHouse OpenTelemetry collection is described in the upstream ClickHouse monitoring guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/clickhouse/","title":"\u003c!--"},{"body":" Elasticsearch The ELASTICSEARCH layer monitors Elasticsearch clusters that OAP scrapes through its Elasticsearch monitoring receiver. It groups under Databases in the sidebar and gives an operator the cluster-health, node-runtime, and per-index view that an Elasticsearch admin expects.\nIn Horizon\u0026rsquo;s sidebar this layer carries the display name Elasticsearch. An Elasticsearch cluster maps onto SkyWalking\u0026rsquo;s entity scopes, and the layer renames each slot to match: services are listed as ES clusters, instances as Nodes, and endpoints as Indices. The layer enables three drill-down tabs — Service (the cluster dashboard), Instance (a node), and Endpoint (an index). It ships no topology, traces, or logs tabs.\nThis page is the operator reference for the bundled ELASTICSEARCH dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled ELASTICSEARCH template; if an operator has published a customized ELASTICSEARCH template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nES clusters list Before opening a cluster, the layer landing page lists every Elasticsearch cluster with four sortable columns, sorted by Shards by default:\nHealth — the cluster health status (meter_elasticsearch_cluster_health_status), averaged over the window.\nShards — total active shards across the cluster (meter_elasticsearch_cluster_shards_total, latest value).\nNodes — number of nodes in the cluster (meter_elasticsearch_cluster_nodes, latest value).\nUnassigned — shards the cluster has not yet placed on a node (meter_elasticsearch_cluster_unassigned_shards_total, latest value) — a non-zero value is the usual first sign of a cluster under stress.\nService dashboard (cluster) The primary drill-down for one selected cluster — the cluster-wide health and capacity picture.\nCluster Health — a table of the current health status and value (meter_elasticsearch_cluster_health_status, latest), the green / yellow / red rollup Elasticsearch reports for the cluster.\nNodes — number of nodes currently in the cluster (meter_elasticsearch_cluster_nodes, latest).\nPending Tasks — average count of cluster-level tasks queued for the master (meter_elasticsearch_cluster_pending_tasks_total) — a rising queue points at master-node pressure.\nPrimary Shards — total primary shards (meter_elasticsearch_cluster_primary_shards_total, latest).\nActive Shards — total active shards (meter_elasticsearch_cluster_shards_total, latest).\nInitializing — shards currently initializing (meter_elasticsearch_cluster_initializing_shards_total, latest).\nRelocating — shards being moved between nodes (meter_elasticsearch_cluster_relocating_shards_total, latest).\nUnassigned — shards not assigned to any node (meter_elasticsearch_cluster_unassigned_shards_total, latest).\nDelayed Unassigned — unassigned shards whose reassignment is being delayed (meter_elasticsearch_cluster_delayed_unassigned_shards_total, latest).\nTripped Breakers — count of tripped circuit breakers across the cluster (meter_elasticsearch_cluster_breakers_tripped, latest), a memory-protection signal.\nCluster CPU Avg — average CPU usage across the cluster\u0026rsquo;s nodes in percent (meter_elasticsearch_cluster_cpu_usage_avg).\nJVM Memory Used Avg — average JVM heap memory used across the cluster (meter_elasticsearch_cluster_jvm_memory_used_avg).\nOpen Files Avg — average open file-descriptor count across the cluster (meter_elasticsearch_cluster_open_file_count).\nInstance dashboard (node) For one selected node — the per-node OS, JVM, and storage detail.\nProcess CPU (%) — CPU consumed by the Elasticsearch process (meter_elasticsearch_node_process_cpu_percent).\nOS CPU (%) — host CPU usage on the node (meter_elasticsearch_node_os_cpu_percent).\nLoad Average — the node\u0026rsquo;s 1-minute, 5-minute, and 15-minute OS load averages (meter_elasticsearch_node_os_load1, meter_elasticsearch_node_os_load5, meter_elasticsearch_node_os_load15).\nJVM Memory (MB) — heap used, heap max, and non-heap used in MB (meter_elasticsearch_node_jvm_memory_heap_used, meter_elasticsearch_node_jvm_memory_heap_max, meter_elasticsearch_node_jvm_memory_nonheap_used).\nGC — garbage-collection activity on a dual axis: GC count on the left, GC time in ms/min on the right (meter_elasticsearch_node_jvm_gc_count, meter_elasticsearch_node_jvm_gc_time).\nTranslog — transaction-log operations and translog size in MB on a dual axis (meter_elasticsearch_node_indices_translog_operations, meter_elasticsearch_node_indices_translog_size).\nBreakers — tripped circuit breakers and the estimated breaker size in MB on this node (meter_elasticsearch_node_breakers_tripped, meter_elasticsearch_node_breakers_estimated_size).\nSegments — Lucene segment count and segment memory in MB on a dual axis (meter_elasticsearch_node_segment_count, meter_elasticsearch_node_segment_memory).\nDisk Usage — disk used in GB and disk-used percent on a dual axis (meter_elasticsearch_node_disk_usage, meter_elasticsearch_node_disk_usage_percent).\nNetwork — bytes sent and received on the node (meter_elasticsearch_node_network_send_bytes, meter_elasticsearch_node_network_receive_bytes).\nOpen Files — average open file-descriptor count on the node (meter_elasticsearch_node_open_file_count).\nEndpoint dashboard (index) For one selected index — indexing throughput, search throughput, size, and document counts.\nIndexing Rate — indexing requests vs. processed operations (meter_elasticsearch_index_stats_indexing_index_total_req_rate, meter_elasticsearch_index_stats_indexing_index_total_proc_rate).\nSearch Rate — search-query requests vs. processed operations (meter_elasticsearch_index_stats_search_query_total_req_rate, meter_elasticsearch_index_stats_search_query_total_proc_rate).\nIndex Size (all shards) — total store size of the index across all shards in GB (meter_elasticsearch_index_indices_store_size_bytes_total, latest).\nIndex Size (primary) — store size of the index\u0026rsquo;s primary shards in GB (meter_elasticsearch_index_indices_store_size_bytes_primary, latest).\nDocuments — document counts: all, primary, and deleted (meter_elasticsearch_index_indices_docs_total, meter_elasticsearch_index_indices_docs_primary, meter_elasticsearch_index_indices_deleted_docs_primary).\nAvg Search Time / Req (s) — average per-request time in seconds for each search phase: fetch, query, scroll, and suggest (meter_elasticsearch_index_search_fetch_avg_time, meter_elasticsearch_index_search_query_avg_time, meter_elasticsearch_index_search_scroll_avg_time, meter_elasticsearch_index_search_suggest_avg_time).\nRequirements The ELASTICSEARCH dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Elasticsearch monitoring enabled (see the upstream Elasticsearch monitoring setup), which feeds the three metric families this dashboard reads:\nCluster metrics — the meter_elasticsearch_cluster_* family (health, node count, shard states, pending tasks, tripped breakers, average CPU / JVM memory / open files) for the cluster list and the Service dashboard.\nNode metrics — the meter_elasticsearch_node_* family (process / OS CPU, load averages, JVM memory and GC, translog, breakers, segments, disk, network, open files) for the Instance dashboard.\nIndex metrics — the meter_elasticsearch_index_* family (indexing and search rates, store size, document counts, per-phase search times) for the Endpoint dashboard.\nEach metric is queried at its own OAP scope — cluster metrics at service scope, node metrics at instance scope, index metrics at endpoint scope. OAP does not roll a metric up across scopes, so a node- or index-scope widget stays empty until that level of data is reported.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/elasticsearch/","title":"\u003c!--"},{"body":" Envoy AI Gateway The ENVOY_AI_GATEWAY layer monitors Envoy AI Gateway deployments — the Envoy-based gateway that fronts LLM providers and models, routing chat / completion traffic to OpenAI, Anthropic, and other backends. SkyWalking turns the gateway\u0026rsquo;s OpenTelemetry GenAI signals into request, latency, token, and streaming-quality metrics, broken down by provider and model, and lands them here.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Gateways. Its services are listed as AI Gateways and its instances as Nodes. The ENVOY_AI_GATEWAY layer enables the Service (AI Gateway) and Instance (Node) dashboards plus the Logs sub-tab. It does not ship an Endpoint dashboard, a topology / service-map view, or a Traces tab — the gateway is monitored entirely through its GenAI meter families, which carry no per-endpoint scope or call graph.\nThis page is the operator reference for the bundled ENVOY_AI_GATEWAY dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled ENVOY_AI_GATEWAY template; if an operator has published a customized template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a gateway, the layer landing page lists every AI Gateway with four sortable columns, sorted by traffic (RPM) by default:\nRPM — requests per minute across the gateway (meter_envoy_ai_gw_request_cpm).\nAvg Latency — average request latency in ms (meter_envoy_ai_gw_request_latency_avg).\nInput Tokens/min — input (prompt) token throughput per minute (meter_envoy_ai_gw_input_token_rate).\nOutput Tokens/min — output (completion) token throughput per minute (meter_envoy_ai_gw_output_token_rate).\nService dashboard The primary drill-down for one selected AI Gateway. Beyond the headline request and token widgets, it breaks traffic down by GenAI provider and model and exposes streaming-quality timings (TTFT / TPOT). The Model Context Protocol (MCP) widgets only appear when the gateway actually serves MCP traffic.\nRequests, latency, and tokens\nRequest RPM — requests per minute for the gateway (meter_envoy_ai_gw_request_cpm).\nRequest Latency Avg — average request latency in ms (meter_envoy_ai_gw_request_latency_avg).\nInput Token Rate — input (prompt) tokens per minute (meter_envoy_ai_gw_input_token_rate).\nOutput Token Rate — output (completion) tokens per minute (meter_envoy_ai_gw_output_token_rate).\nRequest Latency Percentile — p50 / p75 / p90 / p95 / p99 request latency, the tail of the latency distribution, in ms (meter_envoy_ai_gw_request_latency_percentile).\nStreaming quality\nTTFT (Time to First Token) — time to first token for streaming responses, shown as both the average and the percentile distribution in ms (meter_envoy_ai_gw_ttft_avg, meter_envoy_ai_gw_ttft_percentile).\nTPOT (Time Per Output Token) — time per output token (inter-token latency) for streaming responses, shown as both the average and the percentile distribution in ms (meter_envoy_ai_gw_tpot_avg, meter_envoy_ai_gw_tpot_percentile).\nBy provider — each widget is split per gen_ai_provider_name, so every upstream LLM provider the gateway routes to gets its own series:\nRPM by Provider — requests per minute per provider (meter_envoy_ai_gw_provider_request_cpm).\nTokens by Provider — token throughput per provider (meter_envoy_ai_gw_provider_token_rate).\nLatency Avg by Provider — average latency per provider, in ms (meter_envoy_ai_gw_provider_latency_avg).\nBy model — each widget is split per gen_ai_response_model, so every model the gateway answered with gets its own series:\nRPM by Model — requests per minute per model (meter_envoy_ai_gw_model_request_cpm).\nTokens by Model — token throughput per model (meter_envoy_ai_gw_model_token_rate).\nLatency Avg by Model — average latency per model, in ms (meter_envoy_ai_gw_model_latency_avg).\nTTFT by Model — average time to first token per model, in ms (meter_envoy_ai_gw_model_ttft_avg).\nTPOT by Model — average time per output token per model, in ms (meter_envoy_ai_gw_model_tpot_avg).\nMCP (Model Context Protocol) — these widgets render only when the gateway serves MCP traffic; on a gateway that never sees MCP requests they stay hidden rather than showing empty:\nMCP RPM — MCP requests per minute (meter_envoy_ai_gw_mcp_request_cpm).\nMCP Avg Latency — average MCP request latency in ms (meter_envoy_ai_gw_mcp_request_latency_avg).\nMCP Error RPM — MCP errors per minute (meter_envoy_ai_gw_mcp_error_cpm).\nMCP by Method — MCP requests per minute split per mcp_method_name (meter_envoy_ai_gw_mcp_method_cpm).\nMCP by Backend — MCP requests per minute split per mcp_backend (meter_envoy_ai_gw_mcp_backend_request_cpm).\nInstance dashboard For one selected Node of the gateway. The same request, latency, token, and streaming-quality timings as the service view, scoped to a single gateway node.\nRequest RPM — requests per minute for this node (meter_envoy_ai_gw_instance_request_cpm).\nRequest Latency Avg — average request latency for this node, in ms (meter_envoy_ai_gw_instance_request_latency_avg).\nInput Token Rate — input (prompt) tokens per minute for this node (meter_envoy_ai_gw_instance_input_token_rate).\nOutput Token Rate — output (completion) tokens per minute for this node (meter_envoy_ai_gw_instance_output_token_rate).\nRequest Latency Percentile — p50 / p75 / p90 / p95 / p99 request latency for this node, in ms (meter_envoy_ai_gw_instance_request_latency_percentile).\nTTFT — time to first token for this node, shown as both the average and the percentile distribution in ms (meter_envoy_ai_gw_instance_ttft_avg, meter_envoy_ai_gw_instance_ttft_percentile).\nTPOT — time per output token for this node, shown as both the average and the percentile distribution in ms (meter_envoy_ai_gw_instance_tpot_avg, meter_envoy_ai_gw_instance_tpot_percentile).\nRequirements The ENVOY_AI_GATEWAY dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the Envoy AI Gateway GenAI meter families, derived from the gateway\u0026rsquo;s OpenTelemetry GenAI signals:\nGateway (service) metrics — the meter_envoy_ai_gw_* family at service scope: request load (meter_envoy_ai_gw_request_cpm), latency average and percentile (meter_envoy_ai_gw_request_latency_avg, meter_envoy_ai_gw_request_latency_percentile), input / output token rates (meter_envoy_ai_gw_input_token_rate, meter_envoy_ai_gw_output_token_rate), and the streaming-quality timings (meter_envoy_ai_gw_ttft_*, meter_envoy_ai_gw_tpot_*).\nPer-provider and per-model metrics — the meter_envoy_ai_gw_provider_* and meter_envoy_ai_gw_model_* families, labelled by gen_ai_provider_name and gen_ai_response_model, for the provider and model breakdown widgets.\nMCP metrics — the meter_envoy_ai_gw_mcp_* family (request, latency, error, per-method, per-backend), reported only when the gateway serves Model Context Protocol traffic; the MCP widgets stay hidden until these arrive.\nNode (instance) metrics — the meter_envoy_ai_gw_instance_* family for the per-node widgets (request load, latency, tokens, percentile, TTFT, TPOT).\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a node-scope metric is empty until that level of data is reported. For how to enable the upstream collector, see the Envoy AI Gateway monitoring setup docs.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/envoy_ai_gateway/","title":"\u003c!--"},{"body":" Flink The FLINK layer monitors an Apache Flink stream-processing cluster: the JobManager that coordinates the cluster, the TaskManagers that run the work, and the Flink jobs themselves. It is sourced from Flink\u0026rsquo;s metric reporter via OpenTelemetry, so the dashboard reads the same JVM, slot, network, and checkpoint metrics Flink already exposes.\nIn Horizon\u0026rsquo;s sidebar this layer is named Flink. Its three scopes are aliased to Flink\u0026rsquo;s own vocabulary: services are listed as Flink JobManagers, instances as TaskManagers, and endpoints as Jobs. The FLINK layer enables the Service, Instance, and Endpoint sub-tabs only — it ships no topology, no traces, and no logs.\nThis page is the operator reference for the bundled FLINK dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled FLINK template; if an operator has published a customized FLINK template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a JobManager, the layer landing page lists every Flink JobManager with four sortable columns, sorted by Running Jobs by default. Each column is the latest reported value:\nRunning Jobs — jobs currently running on this JobManager (meter_flink_jobManager_running_job_number).\nTaskManagers — TaskManagers registered with this JobManager (meter_flink_jobManager_taskManagers_registered_number).\nSlots Available — free task slots across the registered TaskManagers (meter_flink_jobManager_taskManagers_slots_available).\nSlots Total — total task slots across the registered TaskManagers (meter_flink_jobManager_taskManagers_slots_total).\nService dashboard The primary drill-down for one selected JobManager. The top row is four single-value cards, followed by the JobManager\u0026rsquo;s JVM health, GC behavior, and a running-jobs ranking.\nRunning Jobs — jobs currently running (meter_flink_jobManager_running_job_number).\nTaskManagers — registered TaskManagers (meter_flink_jobManager_taskManagers_registered_number).\nSlots Total — total task slots (meter_flink_jobManager_taskManagers_slots_total).\nSlots Available — free task slots (meter_flink_jobManager_taskManagers_slots_available).\nJM JVM CPU Load (%) — JobManager JVM CPU load (meter_flink_jobManager_jvm_cpu_load).\nJM JVM Thread Count — live JVM threads in the JobManager (meter_flink_jobManager_jvm_thread_count).\nJM CPU Time (ms) — JobManager JVM CPU time in ms (meter_flink_jobManager_jvm_cpu_time).\nJM Heap (MB) — JobManager heap memory, used vs available in MB (meter_flink_jobManager_jvm_memory_heap_used, meter_flink_jobManager_jvm_memory_heap_available).\nJM NonHeap (MB) — JobManager non-heap memory, used vs available in MB (meter_flink_jobManager_jvm_memory_nonHeap_used, meter_flink_jobManager_jvm_memory_nonHeap_available).\nJM Metaspace (MB) — JobManager metaspace, used vs available in MB (meter_flink_jobManager_jvm_memory_metaspace_used, meter_flink_jobManager_jvm_memory_metaspace_available).\nG1 Young GC — G1 young-generation collections on a dual axis: count on the left, time in ms on the right (meter_flink_jobManager_jvm_g1_young_generation_count, meter_flink_jobManager_jvm_g1_young_generation_time).\nG1 Old GC — G1 old-generation collections, count and time in ms on a dual axis (meter_flink_jobManager_jvm_g1_old_generation_count, meter_flink_jobManager_jvm_g1_old_generation_time).\nAll GC — all garbage collectors combined, count and time in ms on a dual axis (meter_flink_jobManager_jvm_all_garbageCollector_count, meter_flink_jobManager_jvm_all_garbageCollector_time).\nTop 10 Running Jobs — the ten jobs with the longest running time, ranked descending (meter_flink_job_runningTime).\nInstance dashboard For one selected TaskManager — the JVM health and network/back-pressure detail of a single worker.\nJVM CPU Load (%) — TaskManager JVM CPU load (meter_flink_taskManager_jvm_cpu_load).\nJVM Thread Count — live JVM threads in the TaskManager (meter_flink_taskManager_jvm_thread_count).\nCPU Time (ms) — TaskManager JVM CPU time in ms (meter_flink_taskManager_jvm_cpu_time).\nBack Pressured — whether the TaskManager is currently back-pressured (meter_flink_taskManager_isBackPressured).\nHeap (MB) — TaskManager heap memory, used vs available in MB (meter_flink_taskManager_jvm_memory_heap_used, meter_flink_taskManager_jvm_memory_heap_available).\nNonHeap (MB) — TaskManager non-heap memory, used vs available in MB (meter_flink_taskManager_jvm_memory_nonHeap_used, meter_flink_taskManager_jvm_memory_nonHeap_available).\nMetaspace (MB) — TaskManager metaspace, used vs available in MB (meter_flink_taskManager_jvm_memory_metaspace_used, meter_flink_taskManager_jvm_memory_metaspace_available).\nRecords In / Out — records read in and written out by the TaskManager (meter_flink_taskManager_numRecordsIn, meter_flink_taskManager_numRecordsOut).\nBytes In / Out / s — bytes per second read in and written out (meter_flink_taskManager_numBytesInPerSecond, meter_flink_taskManager_numBytesOutPerSecond).\nNetty Memory (MB) — Netty network-shuffle memory, used vs available in MB (meter_flink_taskManager_netty_usedMemory, meter_flink_taskManager_netty_availableMemory).\nPool Usage (%) — input vs output buffer-pool usage (meter_flink_taskManager_inPoolUsage, meter_flink_taskManager_outPoolUsage).\nBack-Pressure Time (ms/s) — per-second time the TaskManager spent in each state: soft back-pressure, hard back-pressure, idle, and busy (meter_flink_taskManager_softBackPressuredTimeMsPerSecond, meter_flink_taskManager_hardBackPressuredTimeMsPerSecond, meter_flink_taskManager_idleTimeMsPerSecond, meter_flink_taskManager_busyTimeMsPerSecond).\nEndpoint dashboard For one selected Job — its lifecycle timing, checkpoint behavior, and throughput. The top row is four single-value cards.\nJob Running Time (min) — how long the job has been running, in minutes (meter_flink_job_runningTime).\nJob Restarting Time (min) — time the job has spent restarting, in minutes (meter_flink_job_restartingTime).\nJob Cancelling Time (min) — time the job has spent cancelling, in minutes (meter_flink_job_cancellingTime).\nJob Restarts — number of job restarts (meter_flink_job_restart_number).\nCheckpoints — checkpoint counts over the window: total, completed, failed, and in-progress (meter_flink_job_checkpoints_total, meter_flink_job_checkpoints_completed, meter_flink_job_checkpoints_failed, meter_flink_job_checkpoints_inProgress).\nLast Checkpoint — the most recent checkpoint on a dual axis: size in bytes on the left, duration in ms on the right (meter_flink_job_lastCheckpointSize, meter_flink_job_lastCheckpointDuration).\nCurrent Emit Event Time Lag (ms) — lag between event time and emit time, in ms (meter_flink_job_currentEmitEventTimeLag).\nRecords In / Out — records read in and written out by the job (meter_flink_job_numRecordsIn, meter_flink_job_numRecordsOut).\nBytes In / Out / s — bytes per second read in and written out (meter_flink_job_numBytesInPerSecond, meter_flink_job_numBytesOutPerSecond).\nRequirements The FLINK dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the Flink meter families produced from Flink\u0026rsquo;s OpenTelemetry metric export:\nJobManager metrics — the meter_flink_jobManager_* family (running jobs, registered TaskManagers, slot totals, and JVM CPU / thread / memory / GC detail), driving the Service list and Service dashboard.\nTaskManager metrics — the meter_flink_taskManager_* family (JVM detail, record / byte throughput, Netty and buffer-pool usage, and back-pressure timing), driving the Instance dashboard.\nJob metrics — the meter_flink_job_* family (running / restarting / cancelling time, restarts, checkpoints, emit-time lag, and throughput), driving the Endpoint dashboard and the Top 10 Running Jobs ranking.\nEach metric is queried at its own OAP scope, and OAP does not roll a metric up across scopes — a JobManager-, TaskManager-, or Job-scope metric is empty until that level of data is reported. To set up the Flink metric reporter and the OAP receiver, follow the Flink monitoring setup guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/flink/","title":"\u003c!--"},{"body":" General Service The GENERAL layer is where SkyWalking\u0026rsquo;s language agents report. Any service instrumented by a SkyWalking native agent — Java, .NET (CLR), Go, Python, Ruby, Node.js, PHP, and the Spring Boot / Spring Sleuth meter integrations — lands here, so it is the most-used layer and the reference dashboard every other layer\u0026rsquo;s dashboard is modelled on.\nIn Horizon\u0026rsquo;s sidebar this layer is named General Service. Its services are listed as Services, instances as Instances (each badged with the agent language), and endpoints as API — the endpoint-to-endpoint view is called API dependency. The GENERAL layer enables the full set of sub-tabs: Service, Instance, Endpoint, API dependency, Topology, Traces, Logs, and the profiling tabs (trace, eBPF, async, pprof).\nThis page is the operator reference for the bundled GENERAL dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled GENERAL template; if an operator has published a customized GENERAL template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a service, the layer landing page lists every GENERAL service with three sortable columns, sorted by traffic (RPM) by default:\nRPM — calls per minute (service_cpm). Apdex — user-satisfaction score on a 0 – 1 scale (service_apdex/10000). Error Rate — percent of failed calls (100 - service_sla/100). Service dashboard The primary drill-down for one selected service.\nTop 20 APIs — the busiest endpoints under this service, with tabs to re-rank by Traffic (endpoint_cpm, rpm), Slow (endpoint_resp_time, ms), and Successful Rate (endpoint_sla, %, worst-first). Click a row to jump into that endpoint. Traffic — calls per minute for the service (service_cpm). Error Rate — percent of failed calls (100 - service_sla/100). Apdex — user-satisfaction score on a 0 – 1 scale (service_apdex/10000). Response Time Percentile — p50 / p75 / p90 / p95 / p99 latency, the tail of the response-time distribution (service_percentile). Avg Response Time — mean latency in ms (service_resp_time). MQ Consume rate + latency — message-queue consume count and latency on a dual axis: count on the left, latency on the right (service_mq_consume_count, service_mq_consume_latency). Top 10 instances by load — this service\u0026rsquo;s instances ranked by traffic (service_instance_cpm, rpm). Top 10 slowest instances — instances ranked by average response time (service_instance_resp_time, ms). Top 10 instances by success rate — instances ranked worst-first by success rate (service_instance_sla, %). Slow Database Statements — the 20 slowest sampled database statements captured against this service (top_n_service_database_statement, ms). Each row carries the statement text and, when the sample has one, a jump-to-trace link. Shows no data when OAP captured no statements in the window. The latency and error widgets ship with the metric-to-trace drill enabled: click a data point on Avg Response Time or Response Time Percentile to open the slowest traces at that moment, or on Error Rate or Apdex to open the error traces. The same drill rides the instance latency / success-rate widgets and the endpoint latency, percentile, success-rate, and MQ-latency widgets below. See Dashboard Widgets → Metric-to-trace drill.\nInstance dashboard For one selected service instance. The first three widgets always render; the rest are runtime-specific and appear only when the instance actually reports those metrics — so a Java instance shows the JVM family, a Go instance the Golang family, and so on, without manual configuration.\nAlways shown\nService Instance Load — calls per minute against the instance (service_instance_cpm). Service Instance Latency — average response time in ms (service_instance_resp_time). Service Instance Success Rate — percent of successful calls (service_instance_sla/100). JVM (Java instances)\nJVM CPU — JVM CPU as reported by the agent (instance_jvm_cpu). JVM Memory — heap and non-heap used / max in MB (instance_jvm_memory_heap, instance_jvm_memory_heap_max, instance_jvm_memory_noheap, instance_jvm_memory_noheap_max). JVM Memory Detail — per-pool used memory in MB: code cache, newgen, oldgen, survivor, permgen, metaspace, plus the newer JVM pools (zheap, compressed class space, and the segmented codeheaps). Pools a JVM doesn\u0026rsquo;t expose stay at 0. JVM Thread Count — live / daemon / peak threads. JVM Thread State Count — threads by state: runnable / blocked / waiting / timed-waiting. JVM GC Time — young / old / normal GC time in ms. JVM GC Count — young / old / normal GC counts. JVM Class Count — loaded / total-loaded / total-unloaded classes. CLR (.NET instances)\nCLR CPU — process CPU percentage (instance_clr_cpu). CLR Thread — worker-available, completion-port-available, and completion-port-max threads. CLR Heap Memory — managed heap in MB. CLR GC — gen 0 / gen 1 / gen 2 collection counts. Spring (Spring Boot Actuator / Spring Sleuth meters)\nSpring HTTP Request Count and Spring HTTP Request Duration — http.server.requests count and latency. Spring Instance CPU Usage / Spring OS CPU Usage / Spring OS System Load — process CPU, OS CPU, and 1-minute load average. Spring OS Process Files — open vs max file descriptors. Spring JVM GC Pause Duration, Spring JVM Memory (used / max), Spring JVM Threads (live / daemon / peak), Spring JVM Classes (loaded / unloaded). Spring Database Connection Pool (HikariCP / datasource), Spring Thread Pool, Spring JDBC Connections (active / idle / max), Spring Tomcat Sessions (active / max / rejected). Golang (Go instances)\nGolang Goroutines / OS Threads, Golang GC Pause Time, Golang GC Count, Golang Heap Alloc, Golang Goroutine Schedule Time, Golang GC Free, Golang Alloc Size, Golang Free Size, Golang Heap Objects, Golang Heap, Golang Metadata Mspan, Golang Metadata Mcache, Golang GC Goal Size, and Golang CGO Calls — the Go runtime\u0026rsquo;s goroutine, scheduler, GC, and heap detail. Python (PVM instances)\nPython CPU Utilization and Python Memory Utilization — host vs process. Python Thread Count, Python GC Count (gen 0 / 1 / 2), and Python GC Time. Ruby instances\nRuby CPU Usage, Ruby Memory (RSS), Ruby Memory Usage, Ruby Thread Status (active / running), Ruby GC Count (total / minor / major), Ruby GC Time, Ruby Heap Usage, and Ruby Heap Slots (live / available). Node.js instances\nProcess CPU — process CPU percentage (meter_instance_nodejs_process_cpu). V8 Heap Used / V8 Heap Total / V8 Heap Limit — the V8 heap in MB: currently used, currently allocated, and the maximum the heap may grow to (meter_instance_nodejs_heap_used / _heap_total / _heap_limit). Process RSS — resident set size in MB (meter_instance_nodejs_rss). External Memory — memory held outside the V8 heap (buffers and native objects) in MB (meter_instance_nodejs_external_memory). Array Buffers — ArrayBuffer / SharedArrayBuffer memory in MB (meter_instance_nodejs_array_buffers). Process Uptime — days since the process started (meter_instance_nodejs_uptime/86400). Peak Malloced Memory / Malloced Memory — peak and current V8 malloced memory in MB (meter_instance_nodejs_peak_malloced_memory / _malloced_memory). Old Space Used / New Space Used — V8 old / new generation heap used in MB (meter_instance_nodejs_old_space_used / _new_space_used). PHP (PHM) instances\nPHP CPU Utilization — process CPU percentage (meter_instance_php_process_cpu_utilization). PHP Memory Used and PHP Memory Peak — current and peak process memory in MB (meter_instance_php_memory_used_mb, meter_instance_php_memory_peak_mb). PHP Virtual Memory — virtual memory size in MB (meter_instance_php_virtual_memory_mb). PHP Thread Count — live threads (meter_instance_php_thread_count). PHP Open FDs — open file descriptors (meter_instance_php_open_fd_count). Endpoint dashboard For one selected endpoint (an API).\nTraffic — calls per minute for the endpoint (endpoint_cpm). Response Time — average latency in ms (endpoint_resp_time). Success Rate — percent of successful calls (endpoint_sla/100). Response Time Percentile — p50 / p75 / p90 / p95 / p99 latency for the endpoint (endpoint_percentile). MQ Avg Consuming Latency — consume latency in ms, shown only for endpoints that serve message-queue traffic (endpoint_mq_consume_latency). Topology and maps The GENERAL layer ships a full set of maps.\nTopology (service map) — every service node is decorated with RPM (service_cpm), an SLA health ring (service_sla/100, colored green / amber / red — higher is better, so the ring turns red as success rate drops), and Latency (service_resp_time). Each call edge carries server-side and client-side RPM, Avg response time, p95, and SLA (service_relation_server_* and service_relation_client_*), shown aligned in the edge panel.\nInstance map — from a call between two services on the service map, drill into the instance-to-instance calls between them. The same node / edge metric set is evaluated at instance scope (service_instance_* and service_instance_relation_server/client_*).\nAPI dependency (endpoint map) — the endpoint-to-endpoint dependency view. Each endpoint node shows RPM (endpoint_cpm), an SLA ring (endpoint_sla/100), and Latency (endpoint_resp_time); each edge shows RPM, Avg response time, p95, and SLA (endpoint_relation_*). Endpoint relations are server-side only.\nFor how these maps are read and navigated, see the 3D Infrastructure Map for the cross-layer view, and the topology section of Layer Dashboard Templates for how the node and edge metrics are configured.\nRequirements The GENERAL dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nService / instance / endpoint metrics — the service_*, service_instance_*, and endpoint_* families (traffic, response time, SLA, apdex, percentiles), produced by OAP from agent-reported traces or meters. Relation metrics — service_relation_*, service_instance_relation_*, and endpoint_relation_* for the service map, instance map, and API-dependency views. Runtime metrics, for the runtime-specific instance widgets to appear: JVM (instance_jvm_*), CLR (instance_clr_*), the Spring meter family (meter_*), and the Golang / Python / Ruby / Node.js / PHP agent meter families. An instance only shows the families its agent emits. Sampled records — top_n_service_database_statement for the Slow Database Statements list, captured by OAP when slow-statement sampling is enabled. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so an instance- or endpoint-scope metric is empty until that level of data is reported. When a whole family is missing (for example a non-JVM runtime), its widgets are hidden rather than shown empty.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/general/","title":"\u003c!--"},{"body":" iOS The IOS layer is where SkyWalking\u0026rsquo;s iOS client-side monitoring reports. An iOS app instrumented with the SkyWalking iOS SDK surfaces Apple MetricKit diagnostics — app launch time, hang time, abnormal exits, OOM kills, peak memory, scroll responsiveness, and network transfer — alongside the latency and success rate of the HTTP calls the app makes out to your backends. It sits in the Mobile group of layers.\nIn Horizon\u0026rsquo;s sidebar this layer is named iOS. Its services are listed as Apps, instances as App Sessions, and endpoints as Outbound APIs — these are the names you see on the picker and column headers. The IOS layer enables the Service, Instance, and Endpoint sub-tabs plus Logs. It has no Topology, Traces, or endpoint-dependency view — iOS reports client-side device telemetry and outbound calls, not a server-side call graph.\nThis page is the operator reference for the bundled IOS dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled IOS template; if an operator has published a customized IOS template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nApp list Before opening an app, the layer landing page lists every IOS app with four sortable columns, sorted by Launch (P95) by default:\nLaunch (P95) — 95th-percentile app launch time in ms (meter_ios_app_launch_time_percentile{p='95'}), the tail of how long the app takes to become usable.\nHang Time — total time the main thread spent hung in the window, in ms (meter_ios_hang_time_sum).\nCrashes — abnormal exits, foreground and background summed (meter_ios_foreground_abnormal_exit_count + meter_ios_background_abnormal_exit_count).\nOutbound RPM — calls per minute the app makes to backends (service_cpm).\nApp (service) dashboard The primary drill-down for one selected app.\nApp Launch Time Percentile — p50 / p75 / p90 / p95 / p99 launch time in ms, the distribution of how long the app takes to start (meter_ios_app_launch_time_percentile).\nHang Time Percentile — p50 / p75 / p90 / p95 / p99 main-thread hang time in ms (meter_ios_hang_time_percentile).\nHang Time (sum) — total hang time over the window, in ms (meter_ios_hang_time_sum).\nAbnormal Exits (Crashes) — abnormal exits split into foreground and background series; MetricKit reports the two separately (meter_ios_foreground_abnormal_exit_count, meter_ios_background_abnormal_exit_count).\nOOM Kill Count — background out-of-memory kills, the iOS system reaping the app under memory pressure (meter_ios_background_oom_kill_count).\nPeak Memory — peak resident memory in bytes (meter_ios_peak_memory).\nScroll Hitch Ratio — the fraction of scroll frames classified as hitched; higher means a laggier scrolling UI (meter_ios_scroll_hitch_ratio).\nNetwork Transfer — bytes transferred over wifi vs cellular, download and upload, as four series (meter_ios_wifi_download, meter_ios_wifi_upload, meter_ios_cellular_download, meter_ios_cellular_upload).\nOutbound HTTP — the calls this app makes to backends, on a dual axis: RPM and Avg latency (ms) on the left axis, Success Rate (%) on the right (service_cpm, service_resp_time, service_sla/100).\nApp Session (instance) dashboard For one selected app session — the same MetricKit and outbound-HTTP families evaluated at session scope.\nLaunch Time Percentile — p50 / p75 / p90 / p95 / p99 launch time in ms for the session (meter_ios_instance_app_launch_time_percentile).\nHang Time Percentile — p50 / p75 / p90 / p95 / p99 hang time in ms for the session (meter_ios_instance_hang_time_percentile).\nAbnormal Exits — foreground vs background abnormal exits for the session (meter_ios_instance_foreground_abnormal_exit_count, meter_ios_instance_background_abnormal_exit_count).\nOOM Kill Count — background OOM kills for the session (meter_ios_instance_background_oom_kill_count).\nPeak Memory — peak resident memory in bytes for the session (meter_ios_instance_peak_memory).\nOutbound HTTP — the session\u0026rsquo;s outbound calls on a dual axis: RPM and Avg latency (ms) on the left axis, Success Rate (%) on the right (service_instance_cpm, service_instance_resp_time, service_instance_sla/100).\nOutbound API (endpoint) dashboard For one selected Outbound API — a backend endpoint the app calls.\nOutbound Load — calls per minute to the endpoint (endpoint_cpm).\nOutbound Avg Latency — average call latency in ms (endpoint_resp_time).\nOutbound Success Rate — percent of successful calls (endpoint_sla/100).\nOutbound Latency Percentile — p50 / p75 / p90 / p95 / p99 latency for the endpoint (endpoint_percentile).\nRequirements The IOS dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\niOS MetricKit metrics — the meter_ios_* family at service scope and the meter_ios_instance_* family at session scope: launch-time and hang-time percentiles, hang-time sum, foreground / background abnormal exits, background OOM kills, peak memory, scroll hitch ratio, and wifi / cellular network transfer. These are produced by OAP from the SkyWalking iOS SDK\u0026rsquo;s MetricKit reports.\nOutbound HTTP metrics — the service_*, service_instance_*, and endpoint_* families (traffic, response time, SLA, percentiles), produced by OAP from the calls the app makes to instrumented backends.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a session- or endpoint-scope metric is empty until that level of data is reported. When a family is missing, its widgets read no data rather than being shown with fabricated values.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/ios/","title":"\u003c!--"},{"body":" Kubernetes The K8S layer monitors Kubernetes clusters and the nodes inside them. SkyWalking builds this layer from cluster-state and node-resource telemetry collected through OpenTelemetry (kube-state-metrics and the node / cAdvisor metric pipelines scraped into OAP) and reshapes it into cluster-wide and per-node metrics. In the sidebar it groups under Kubernetes.\nIn Horizon\u0026rsquo;s sidebar this layer\u0026rsquo;s entities are renamed to fit the Kubernetes model: services are listed as Clusters and instances as Nodes. The K8S layer enables the Service (Cluster) and Instance (Node) scopes only — there is no endpoint scope, no topology, and no traces or logs tab, because this layer reports cluster-state and node-resource metrics rather than request traffic.\nThis page is the operator reference for the bundled K8S dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled K8S template; if an operator has published a customized K8S template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nCluster list Before opening a cluster, the layer landing page lists every Kubernetes cluster with four sortable columns, sorted by Pods by default. Each column shows the latest reading summed across the cluster:\nPods — total pods in the cluster (k8s_cluster_pod_total).\nNodes — total nodes in the cluster (k8s_cluster_node_total).\nNamespaces — total namespaces in the cluster (k8s_cluster_namespace_total).\nDeployments — total deployments in the cluster (k8s_cluster_deployment_total).\nCluster dashboard The primary drill-down for one selected cluster (a Service in OAP terms). It opens with a row of count cards summarizing the cluster\u0026rsquo;s object inventory, then resource trends, then status tables that break the cluster down per node, deployment, service, and pod.\nInventory cards — each is a single latest count:\nNode Total — nodes in the cluster (latest(k8s_cluster_node_total)).\nNamespace Total — namespaces in the cluster (latest(k8s_cluster_namespace_total)).\nDeployment Total — deployments in the cluster (latest(k8s_cluster_deployment_total)).\nStatefulSet Total — statefulsets in the cluster (latest(k8s_cluster_statefulset_total)).\nDaemonSet Total — daemonsets in the cluster (latest(k8s_cluster_daemonset_total)).\nService Total — Kubernetes services in the cluster (latest(k8s_cluster_service_total)).\nPod Total — pods in the cluster (latest(k8s_cluster_pod_total)).\nContainer Total — containers in the cluster (latest(k8s_cluster_container_total)).\nResource trends — cluster-wide capacity vs. demand over time:\nCPU Resources — cluster CPU capacity against requests, limits, and allocatable, in millicores (k8s_cluster_cpu_cores, k8s_cluster_cpu_cores_requests, k8s_cluster_cpu_cores_limits, k8s_cluster_cpu_cores_allocatable).\nMemory Resources — cluster memory requests, allocatable, limits, and total, in GiB (k8s_cluster_memory_requests, k8s_cluster_memory_allocatable, k8s_cluster_memory_limits, k8s_cluster_memory_total).\nStorage Resources — cluster ephemeral-storage total against allocatable, in GiB (k8s_cluster_storage_total, k8s_cluster_storage_allocatable).\nStatus tables — each lists the entities currently matching the condition; they read no data when nothing matches:\nNode Status — per-node Kubernetes conditions currently true or unknown — Ready, the various Pressure conditions, and so on (latest(k8s_cluster_node_status)).\nDeployment Status — deployments reporting the Available condition (latest(k8s_cluster_deployment_status)).\nDeployment Spec Replicas — desired replica count per deployment (latest(k8s_cluster_deployment_spec_replicas)).\nService Status — pods backing each Kubernetes service, grouped by pod phase — Running / Pending / Failed and so on (latest(k8s_cluster_service_pod_status)).\nPod Status Not Running — pods in any non-Running phase (latest(k8s_cluster_pod_status_not_running)).\nPod Status Waiting — containers in a waiting state, grouped by the waiting reason (latest(k8s_cluster_pod_status_waiting)).\nNode dashboard For one selected node (an Instance in OAP terms) — its scheduling state and CPU / memory / network / storage resources.\nNode Status — the node\u0026rsquo;s current status as a single latest reading (latest(k8s_node_node_status)).\nPods on Node — pods scheduled on the node over time (k8s_node_pod_total).\nPod Total — the current count of pods scheduled on the node, as a single latest reading (latest(k8s_node_pod_total)).\nNode CPU Usage — node CPU usage in millicores (k8s_node_cpu_usage).\nNode CPU Resources — node CPU total against allocatable, requests, and limits, in millicores (k8s_node_cpu_cores, k8s_node_cpu_cores_allocatable, k8s_node_cpu_cores_requests, k8s_node_cpu_cores_limits).\nNode Memory Usage — node memory usage in GiB (k8s_node_memory_usage).\nNode Memory Resources — node memory total against allocatable, requests, and limits, in GiB (k8s_node_memory_total, k8s_node_memory_allocatable, k8s_node_memory_requests, k8s_node_memory_limits).\nNode Network I/O — node receive and transmit throughput in KB/s (k8s_node_network_receive, k8s_node_network_transmit).\nNode Storage Resources — node storage total against allocatable, in GiB (k8s_node_storage_total, k8s_node_storage_allocatable).\nRequirements The K8S dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the Kubernetes monitoring metric families, fed in through the OpenTelemetry receiver from kube-state-metrics and the node / cAdvisor pipelines:\nCluster metrics — the k8s_cluster_* family at cluster scope: object-inventory totals (node / namespace / deployment / statefulset / daemonset / service / pod / container), CPU / memory / storage capacity-and-demand series, and the per-node, per-deployment, per-service, and per-pod status breakdowns.\nNode metrics — the k8s_node_* family at node scope: node status, pod count, and CPU / memory / network / storage usage and resource series.\nEach metric is queried at its own OAP scope (Cluster / Node); OAP does not roll a metric up across scopes, so a node-scope metric stays empty until that level of data is reported. For how to stand up the Kubernetes-to-OAP pipeline, see the layer\u0026rsquo;s setup documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/k8s/","title":"\u003c!--"},{"body":" Kubernetes Services The K8S_SERVICE layer monitors the network behavior of Kubernetes services, observed at the kernel level by SkyWalking Rover\u0026rsquo;s eBPF probes. It captures the HTTP and TCP traffic flowing in and out of each service\u0026rsquo;s pods — call rate, latency, status codes, header / body sizes, packet counts, and connection activity — without instrumenting the application. It belongs to the Kubernetes layer group.\nIn Horizon\u0026rsquo;s sidebar this layer is named Kubernetes Services. Its services are listed as K8s services, instances as Pods, and endpoints as Endpoints — service names are grouped and displayed by their Kubernetes namespace. The K8S_SERVICE layer enables the Service, Pod, Endpoint, Topology, eBPF Profiling, Network Profiling, and Pod Logs sub-tabs. It does not enable an endpoint-dependency map, Traces, or Logs.\nThis page is the operator reference for the bundled K8S_SERVICE dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled K8S_SERVICE template; if an operator has published a customized K8S_SERVICE template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a service, the layer landing page lists every K8S_SERVICE service with four sortable columns, sorted by HTTP traffic (HTTP RPM) by default:\nPods — number of pods backing the service, summed from the latest reading (latest(k8s_service_pod_total)).\nHTTP RPM — HTTP calls per minute, summed across the service (kubernetes_service_http_call_cpm).\nLatency — average HTTP response time in ms (kubernetes_service_http_call_time).\nSuccess Rate — percent of successful HTTP calls (kubernetes_service_http_call_successful_rate/100).\nService dashboard The primary drill-down for one selected service. It mixes pod-lifecycle and resource widgets with the HTTP and TCP traffic the service\u0026rsquo;s pods carry.\nPods and resources\nService Pods — pod count over time (k8s_service_pod_total).\nPods Waiting — a table of containers currently in the Waiting state, keyed by container · pod · reason (latest(k8s_service_pod_status_waiting)).\nPod Restarts — a table of pods by cumulative restart count (latest(k8s_service_pod_status_restarts_total)).\nCPU Resources — requested vs. limited CPU in millicores, as two series (k8s_service_cpu_cores_requests, k8s_service_cpu_cores_limits).\nMemory Resources — requested vs. limited memory in MiB (k8s_service_memory_requests, k8s_service_memory_limits).\nPod CPU Usage — actual CPU consumed by the pods in millicores (k8s_service_pod_cpu_usage).\nPod Memory Usage — actual memory consumed by the pods in MiB (k8s_service_pod_memory_usage).\nHTTP traffic\nHTTP Request RPM — HTTP calls per minute for the service (kubernetes_service_http_call_cpm).\nHTTP Response Time — average HTTP response time in ms (kubernetes_service_http_call_time).\nHTTP Status Code RPM — calls per minute broken out by status class: 1xx / 2xx / 3xx / 4xx / 5xx (kubernetes_service_http_status_1xx_cpm … kubernetes_service_http_status_5xx_cpm).\nHTTP Request / Response Size — average request and response header and body sizes in KB, as four series (kubernetes_service_http_avg_req_header_size, kubernetes_service_http_avg_req_body_size, kubernetes_service_http_avg_resp_header_size, kubernetes_service_http_avg_resp_body_size).\nTCP traffic\nTCP Connect — client-side connect attempts and successes per minute, as two series (kubernetes_service_connect_cpm, kubernetes_service_connect_success_cpm).\nTCP Connect Duration — average connect time in ns (kubernetes_service_connect_time).\nTCP Accept — server-side accept events per minute (kubernetes_service_accept_cpm).\nTCP Packets — read, write, and write-retransmit packet counts per minute, as three series (kubernetes_service_read_package_cpm, kubernetes_service_write_package_cpm, kubernetes_service_write_retrains_package_cpm).\nTCP Bytes — read vs. write payload bytes per minute in KB (kubernetes_service_read_package_size, kubernetes_service_write_package_size).\nPod dashboard For one selected pod (a service instance). The widgets mirror the service view at instance scope, covering the pod\u0026rsquo;s HTTP and TCP traffic.\nPod HTTP RPM — HTTP calls per minute for the pod (kubernetes_service_instance_http_call_cpm).\nPod HTTP Response Time — average HTTP response time in ms (kubernetes_service_instance_http_call_time).\nPod HTTP Status Code RPM — calls per minute by status class: 1xx / 2xx / 3xx / 4xx / 5xx (kubernetes_service_instance_http_status_1xx_cpm … kubernetes_service_instance_http_status_5xx_cpm).\nPod HTTP Sizes — average request and response header and body sizes in KB, as four series (kubernetes_service_instance_http_avg_req_header_size, kubernetes_service_instance_http_avg_req_body_size, kubernetes_service_instance_http_avg_resp_header_size, kubernetes_service_instance_http_avg_resp_body_size).\nPod TCP Connect — client-side connect attempts and successes per minute (kubernetes_service_instance_connect_cpm, kubernetes_service_instance_connect_success_cpm).\nPod TCP Packets — read, write, and write-retransmit packet counts per minute (kubernetes_service_instance_read_package_cpm, kubernetes_service_instance_write_package_cpm, kubernetes_service_instance_write_retrains_package_cpm).\nPod TCP Bytes — read vs. write payload bytes per minute in KB (kubernetes_service_instance_read_package_size, kubernetes_service_instance_write_package_size).\nEndpoint dashboard For one selected endpoint. K8S_SERVICE endpoints carry HTTP traffic, so this scope is HTTP-focused.\nEndpoint HTTP RPM — HTTP calls per minute for the endpoint (kubernetes_service_endpoint_http_call_cpm).\nEndpoint HTTP Response Time — average HTTP response time in ms (kubernetes_service_endpoint_http_call_time).\nEndpoint Status Code RPM — calls per minute by status class: 1xx / 2xx / 3xx / 4xx / 5xx (kubernetes_service_endpoint_http_status_1xx_cpm … kubernetes_service_endpoint_http_status_5xx_cpm).\nEndpoint Request Sizes — average request header and body sizes in KB (kubernetes_service_endpoint_http_avg_req_header_size, kubernetes_service_endpoint_http_avg_req_body_size).\nEndpoint Response Sizes — average response header and body sizes in KB (kubernetes_service_endpoint_http_avg_resp_header_size, kubernetes_service_endpoint_http_avg_resp_body_size).\nTopology and maps The K8S_SERVICE layer ships a service topology with an instance-level drill-down.\nTopology (service map) — each service node is decorated with RPM (kubernetes_service_http_call_cpm), a Success Rate health ring (kubernetes_service_http_call_successful_rate/100, colored green / amber / red — higher is better, so the ring turns red as the success rate drops), and Latency (kubernetes_service_http_call_time). Each call edge carries server-side and client-side RPM (kubernetes_service_relation_server_http_call_cpm, kubernetes_service_relation_client_http_call_cpm) and Avg response time (kubernetes_service_relation_server_http_call_time, kubernetes_service_relation_client_http_call_time).\nInstance map — from a call between two services on the service map, drill into the pod-to-pod calls between them. Each instance node shows RPM (kubernetes_service_instance_http_call_cpm), a Success Rate ring (kubernetes_service_instance_http_call_successful_rate/100), and Latency (kubernetes_service_instance_http_call_time); each edge carries server-side and client-side RPM (kubernetes_service_instance_relation_server_http_call_cpm, kubernetes_service_instance_relation_client_http_call_cpm) and Avg response time (kubernetes_service_instance_relation_server_http_call_time, kubernetes_service_instance_relation_client_http_call_time).\nFor how these maps are read and navigated, see the 3D Infrastructure Map for the cross-layer view, and the topology section of Layer Dashboard Templates for how the node and edge metrics are configured.\neBPF Profiling, Network Profiling, and Pod Logs Because K8S_SERVICE data comes from eBPF probes, this layer also enables three investigation tabs alongside the dashboards:\neBPF Profiling — on-CPU / off-CPU profiling tasks targeted at a selected service, with the flame-graph and span-attached results SkyWalking Rover reports.\nNetwork Profiling — the process-to-process network conversations within the service, rendered as a process-level topology, captured by SkyWalking Rover on a selected pod.\nPod Logs — the container logs collected from the service\u0026rsquo;s pods, with the same filtering and search the logs surface provides elsewhere.\nThese tabs query their own data on demand and are independent of the metric widgets above.\nRequirements The K8S_SERVICE dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Kubernetes network monitoring enabled, with SkyWalking Rover\u0026rsquo;s eBPF probes feeding traffic telemetry and a Kubernetes metrics source feeding pod / resource state. Specifically:\nPod and resource metrics — the k8s_service_* family (pod totals, waiting / restart status, CPU and memory requests / limits, and actual pod CPU / memory usage) for the service-scope lifecycle and resource widgets.\nHTTP metrics — the kubernetes_service_http_*, kubernetes_service_instance_http_*, and kubernetes_service_endpoint_http_* families covering call counts, response time, success rate, status classes, and header / body sizes, at their respective service / instance / endpoint scopes.\nTCP metrics — the kubernetes_service_* and kubernetes_service_instance_* connect, accept, packet, and byte families for the L4 widgets.\nRelation metrics — kubernetes_service_relation_server/client_http_* and kubernetes_service_instance_relation_server/client_http_* for the service map and instance map edges.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so an instance- or endpoint-scope metric is empty until that level of data is reported. For setup, see the Kubernetes network monitoring guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/k8s_service/","title":"\u003c!--"},{"body":" Kafka The KAFKA layer monitors Apache Kafka clusters. SkyWalking reads Kafka\u0026rsquo;s JMX metrics (via OpenTelemetry\u0026rsquo;s Kafka receiver or an equivalent collector) and turns them into per-cluster and per-broker meters, so this dashboard is a JMX-derived view of cluster health, partition / replication state, and broker throughput rather than agent-traced request data.\nIn Horizon\u0026rsquo;s sidebar this layer is named Kafka, grouped under MQ. Its services are listed as Kafka clusters and its instances as Brokers. The KAFKA layer enables only the Service (cluster) and Instance (broker) sub-tabs — it ships no endpoint scope, no topology, and no traces or logs, because Kafka\u0026rsquo;s JMX feed is metrics-only.\nThis page is the operator reference for the bundled KAFKA dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled KAFKA template; if an operator has published a customized KAFKA template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nCluster list Before opening a cluster, the layer landing page lists every Kafka cluster with four sortable columns, sorted by Partitions by default:\nPartitions — average partition count across the cluster (meter_kafka_partition_count).\nOffline Partitions — total partitions with no active leader, summed across the cluster (meter_kafka_offline_partitions_count). A non-zero value means data on those partitions is currently unavailable.\nMax Lag — the worst replica lag observed (meter_kafka_max_lag), the maximum of how far any follower trails its leader.\nLeaders — total partition leaders hosted across the cluster (meter_kafka_leader_count).\nCluster dashboard The primary drill-down for one selected Kafka cluster — the cluster-wide controller and partition health view.\nPartition Count — total partitions in the cluster (meter_kafka_partition_count).\nLeader Count — partition leaders in the cluster (meter_kafka_leader_count).\nActive Controllers — the count of active controllers (meter_kafka_active_controller_count). A healthy cluster has exactly one; zero or more than one signals a controller problem.\nMax Lag — the worst replica lag across the cluster (meter_kafka_max_lag).\nUnder-Replicated Partitions — partitions that have fewer in-sync replicas than configured (meter_kafka_under_replicated_partitions). Sustained non-zero values indicate replication is falling behind.\nOffline Partitions — partitions with no active leader (meter_kafka_offline_partitions_count).\nLeader Election Rate — partition-leader elections per second, split into two series: normal elections (meter_kafka_leader_election_rate) and unclean elections (meter_kafka_unclean_leader_elections_per_second). Unclean elections promote an out-of-sync replica and can lose data, so they should stay at zero.\nBroker dashboard For one selected broker. The widgets cover the broker\u0026rsquo;s CPU and memory, message and byte throughput, request handling, queue timings, replication, and partition/ISR state.\nCPU Usage — broker CPU percentage (meter_kafka_broker_cpu_time_total).\nIncoming Messages / s — messages produced into the broker per second (meter_kafka_broker_messages_per_second).\nBytes In / s — inbound throughput in bytes per second (meter_kafka_broker_bytes_in_per_second).\nBytes Out / s — outbound throughput in bytes per second (meter_kafka_broker_bytes_out_per_second).\nRequests / s — total requests handled per second (meter_kafka_broker_requests_per_second).\nPurgatory Size — requests parked in the broker\u0026rsquo;s request purgatory awaiting completion (meter_kafka_broker_purgatory_size).\nISR Shrinks/s — the latest rate at which in-sync-replica sets are shrinking (latest(meter_kafka_broker_isr_shrinks_per_second)), shown as a single number. Frequent shrinks mean replicas are repeatedly dropping out of sync.\nISR Expands/s — the latest rate at which in-sync-replica sets are re-expanding (latest(meter_kafka_broker_isr_expands_per_second)), shown as a single number.\nMemory Usage (%) — broker memory utilization (meter_kafka_broker_memory_usage_percentage).\nUnder-Replicated Partitions — the latest count of under-replicated partitions on this broker (latest(meter_kafka_broker_under_replicated_partitions)), shown as a single number.\nUnder Min-ISR Partitions — the latest count of partitions below their minimum in-sync-replica threshold on this broker (latest(meter_kafka_broker_under_min_isr_partition_count)), shown as a single number. These partitions reject produces under the default acks setting.\nPartitions + Leaders — partitions hosted on the broker (meter_kafka_broker_partition_count) overlaid with the partitions it currently leads (meter_kafka_broker_leader_count).\nQueue / Send Times — broker request latency breakdown in ms across four stages: request q (meter_kafka_broker_request_queue_time_ms), response q (meter_kafka_broker_response_queue_time_ms), response send (meter_kafka_broker_response_send_time_ms), and remote (meter_kafka_broker_remote_time_ms).\nTopic Rates — per-broker topic activity: produce req/s (meter_kafka_broker_topic_produce_requests_per_second), fetch req/s (meter_kafka_broker_topic_fetch_requests_per_second), and bytes-in/s (meter_kafka_broker_topic_bytesin_per_second).\nReplication — replication traffic in bytes per second between brokers, bytes in (meter_kafka_broker_replication_bytes_in_per_second) and bytes out (meter_kafka_broker_replication_bytes_out_per_second).\nGC Count — garbage-collection count for the broker JVM (meter_kafka_broker_garbage_collector_count).\nMax Lag (broker) — the broker\u0026rsquo;s total replica lag (sum(meter_kafka_broker_max_lag)), shown as a single number — the sum of how far this broker\u0026rsquo;s followers trail their leaders.\nRequirements The KAFKA dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Kafka\u0026rsquo;s JMX metrics ingested and aggregated into the two meter families this layer renders:\nCluster meters — the meter_kafka_* family at Service scope (partition, leader, controller, lag, under-replicated / offline partition, and leader-election metrics) for the cluster list and cluster dashboard.\nBroker meters — the meter_kafka_broker_* family at ServiceInstance scope (CPU, memory, message / byte / request throughput, purgatory, ISR, queue and send times, topic rates, replication, GC, and per-broker partition / leader / lag metrics) for the broker dashboard.\nThese meters are produced by SkyWalking\u0026rsquo;s Kafka monitoring, which reads Kafka\u0026rsquo;s JMX through OpenTelemetry\u0026rsquo;s Kafka receiver. See the Kafka monitoring setup for how to wire the collector to OAP. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a broker-scope meter is empty until per-broker data is reported.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/kafka/","title":"\u003c!--"},{"body":" Kong The KONG layer monitors Kong API gateways. Kong exposes its built-in Prometheus metrics, OpenTelemetry collects and forwards them to OAP, and OAP aggregates them into the meter_kong_* families this dashboard renders. The layer key is KONG, and in Horizon\u0026rsquo;s sidebar it is grouped under Gateways.\nIn the sidebar this layer\u0026rsquo;s services are listed as Kong services, its instances as Nodes (the individual Kong data-plane nodes), and its endpoints as Routes (the matched Kong routes). The KONG layer enables three scopes — Service, Instance (Node), and Endpoint (Route). It does not ship a topology, traces, or logs tab, so this dashboard is purely the metric drill-down across those three scopes.\nThis page is the operator reference for the bundled KONG dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled KONG template; if an operator has published a customized KONG template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a service, the layer landing page lists every Kong service, sorted by request rate (RPS) by default, with four columns:\nRPS — total HTTP requests per second across the service\u0026rsquo;s nodes (aggregate_labels(meter_kong_service_http_requests,sum)).\n200/s — 200-status responses per second (aggregate_labels(meter_kong_service_http_status{code='200'}, sum)).\n404/s — 404-status responses per second (aggregate_labels(meter_kong_service_http_status{code='404'}, sum)).\n500/s — 500-status responses per second (aggregate_labels(meter_kong_service_http_status{code='500'}, sum)).\nThe three status columns give an at-a-glance health read across the fleet — a service whose 500/s is climbing next to its 200/s is failing requests upstream.\nService dashboard The primary drill-down for one selected Kong service. Every widget aggregates across the service\u0026rsquo;s nodes.\nHTTP Request Trend — total requests per second for the service (aggregate_labels(meter_kong_service_http_requests,sum)).\nHTTP Status Trend — requests per second broken down by HTTP status code (aggregate_labels(meter_kong_service_http_status,sum(code)), one line per code).\nHTTP Bandwidth — ingress / egress bandwidth in KB/s, by direction (aggregate_labels(meter_kong_service_http_bandwidth,sum(direction)), divided to KB).\nKong Latency — the time spent inside Kong itself (plugins and routing), in ms, averaged across percentiles (aggregate_labels(meter_kong_service_kong_latency,avg(p))).\nRequest Latency — total request latency in ms — the time as seen by the client, averaged across percentiles (aggregate_labels(meter_kong_service_request_latency,avg(p))).\nUpstream Latency — the time spent waiting on the upstream service Kong proxies to, in ms, averaged across percentiles (aggregate_labels(meter_kong_service_upstream_latency,avg(p))). Comparing Kong Latency, Request Latency, and Upstream Latency tells you whether added latency is coming from the gateway or from the backend behind it.\nNginx Connections — Kong\u0026rsquo;s underlying Nginx connections by state (aggregate_labels(meter_kong_service_nginx_connections_total,sum(state)), one line per state).\nNginx Timers — Nginx timers by state — running vs pending (aggregate_labels(meter_kong_service_nginx_timers,sum(state)), one line per state).\nDatastore Reachable — a per-instance table of whether each node can reach Kong\u0026rsquo;s datastore (latest(aggregate_labels(meter_kong_service_datastore_reachable,sum(service_instance_id)))), with Instance and Reachable columns. A node that can\u0026rsquo;t reach the datastore is no longer receiving config updates.\nNginx Metric Errors — the latest count of errors Kong hit while exporting its own Nginx metrics (latest(aggregate_labels(meter_kong_service_nginx_metric_errors_total,sum))), shown as a single number — a non-zero value means metric collection on that service is degraded.\nInstance dashboard For one selected node. These widgets are reported per node, so they show how one Kong data-plane process is behaving.\nHTTP Request Trend — requests per second for the node (meter_kong_instance_http_requests).\nHTTP Status Trend — requests per second by status code for the node (meter_kong_instance_http_status).\nHTTP Bandwidth — bandwidth in KB/s for the node (meter_kong_instance_http_bandwidth, divided to KB).\nKong Latency — time spent inside Kong for the node, in ms (meter_kong_instance_kong_latency).\nRequest Latency — total request latency for the node, in ms (meter_kong_instance_request_latency).\nUpstream Latency — upstream wait time for the node, in ms (meter_kong_instance_upstream_latency).\nDatastore Reachable — the latest datastore-reachability reading for the node, shown as a single number (latest(meter_kong_instance_datastore_reachable)).\nNginx Connections — Nginx connections by state for the node (meter_kong_instance_nginx_connections_total).\nNginx Timers — Nginx timers by state for the node (meter_kong_instance_nginx_timers).\nShared Memory Usage — how full the node\u0026rsquo;s Nginx shared-memory dictionaries are, as a percentage of total (meter_kong_instance_shared_dict_bytes over meter_kong_instance_shared_dict_total_bytes). When this approaches 100% the node can no longer cache new entries.\nWorker Lua VM Usage — memory used by the worker processes\u0026rsquo; Lua VMs, in MB (meter_kong_instance_memory_workers_lua_vms_bytes, divided to MB).\nEndpoint dashboard For one selected route. Kong reports a tighter metric set at route scope — status, bandwidth, and the three latency views.\nHTTP Status Trend — requests per second by status code for the route (meter_kong_endpoint_http_status).\nTotal Bandwidth — ingress / egress bandwidth in KB/s for the route (meter_kong_endpoint_http_bandwidth, divided to KB).\nKong Latency — time spent inside Kong for the route, in ms (meter_kong_endpoint_kong_latency).\nRequest Latency — total request latency for the route, in ms (meter_kong_endpoint_request_latency).\nUpstream Latency — upstream wait time for the route, in ms (meter_kong_endpoint_upstream_latency).\nRequirements The KONG dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Kong metrics flowing in through the OpenTelemetry receiver:\nService metrics — the meter_kong_service_* family (requests, status, bandwidth, the Kong / request / upstream latency trio, Nginx connections and timers, datastore reachability, and the Nginx metric-error counter), aggregated by OAP across a service\u0026rsquo;s nodes.\nInstance (node) metrics — the meter_kong_instance_* family, including the node-only meter_kong_instance_shared_dict_* and meter_kong_instance_memory_workers_lua_vms_bytes health metrics.\nEndpoint (route) metrics — the meter_kong_endpoint_* family for the per-route status, bandwidth, and latency widgets.\nThese come from Kong\u0026rsquo;s Prometheus plugin scraped by an OpenTelemetry Collector and forwarded to OAP, which converts them into the meter_kong_* metrics through its meter-analysis rules. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a route-scope or node-scope metric is empty until that level of data is reported. For the end-to-end collector and OAP setup, see the Kong monitoring backend guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/kong/","title":"\u003c!--"},{"body":" Istio Managed Services The MESH layer is where SkyWalking observes services running inside an Istio service mesh. Telemetry comes from the Envoy sidecars via Envoy\u0026rsquo;s Access Log Service (ALS), so a service does not need a language agent to appear here — Envoy reports the traffic, latency, and Envoy-runtime metrics on its behalf. This makes MESH the natural home for any workload managed by Istio, instrumented or not.\nIn Horizon\u0026rsquo;s sidebar this layer is named Istio Managed Services. Its services are listed as Services, instances as Sidecars (one per Envoy proxy), and endpoints as Endpoints. Service names follow the Istio service.namespace convention, so the namespace is surfaced as a grouping value alongside the service name. The MESH layer enables the Service, Instance (Sidecar), Endpoint, Topology, Traces, and Logs sub-tabs, plus eBPF profiling, network profiling, and pod logs. There is no endpoint-to-endpoint dependency map for this layer.\nThis page is the operator reference for the bundled MESH dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled MESH template; if an operator has published a customized MESH template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a service, the layer landing page lists every MESH service with four sortable columns, sorted by traffic (Traffic) by default:\nTraffic — calls per minute observed by the sidecars (service_cpm).\nApdex — user-satisfaction score on a 0 – 1 scale (service_apdex/10000).\nLatency — average response time in ms (service_resp_time).\nError Rate — percent of failed calls (100 - service_sla/100).\nService dashboard The primary drill-down for one selected service.\nTop 20 APIs — the busiest endpoints under this service, with tabs to re-rank by Traffic (endpoint_cpm, rpm), Slow (endpoint_resp_time, ms), and Successful Rate (endpoint_sla, %, worst-first). Click a row to jump into that endpoint.\nTraffic — mesh-wide requests per minute observed by the Envoy sidecars (service_cpm).\nError Rate — percent of failed calls (100 - service_sla/100).\nApdex — user-satisfaction score on a 0 – 1 scale (service_apdex/10000).\nAvg Response Time — mean latency in ms (service_resp_time).\nResponse Time Percentile — p50 / p75 / p90 / p95 / p99 latency, the tail of the response-time distribution (service_percentile).\nService Throughput — bytes per minute through the sidecar, received and sent on the same chart (service_throughput_received, service_throughput_sent).\nSidecar Internal Latency — Envoy-internal request and response latency in nanoseconds. Useful when you suspect the sidecar itself is adding overhead (service_sidecar_internal_req_latency_nanos, service_sidecar_internal_resp_latency_nanos).\nTop 10 sidecars — this service\u0026rsquo;s sidecar instances ranked across three tabs: Traffic (service_instance_cpm, rpm), Slow (service_instance_resp_time, ms), and Successful Rate (service_instance_sla, %, worst-first).\nInstance dashboard For one selected sidecar instance. The first five widgets always render; the Envoy-runtime widgets that follow appear only when the sidecar actually reports those metrics, so a non-Envoy or partially-instrumented sidecar simply shows fewer panels.\nAlways shown\nSidecar Load — calls per minute against the selected sidecar instance (service_instance_cpm).\nSidecar Latency — average response time in ms (service_instance_resp_time).\nSidecar Success Rate — percent of successful calls (service_instance_sla/100).\nSidecar Throughput — bytes through the sidecar, received and sent (service_instance_throughput_received, service_instance_throughput_sent).\nSidecar Internal Latency — Envoy-internal request and response latency in nanoseconds (service_instance_sidecar_internal_req_latency_nanos, service_instance_sidecar_internal_resp_latency_nanos).\nEnvoy runtime (shown when the sidecar reports it)\nEnvoy Upstream Request Active — in-flight upstream requests per cluster (envoy_cluster_up_rq_active).\nEnvoy Upstream Request Increase — upstream requests added per minute (envoy_cluster_up_rq_incr).\nEnvoy Upstream Pending Active — pending upstream requests, a sign of connection-pool back-pressure (envoy_cluster_up_rq_pending_active).\nEnvoy Upstream Connection Active — active upstream connections per cluster (envoy_cluster_up_cx_active).\nEnvoy Upstream Connection Increase — upstream connections added per minute (envoy_cluster_up_cx_incr).\nEnvoy Cluster Healthy Membership — healthy upstream members per cluster; a non-trivial drop signals upstream churn (envoy_cluster_membership_healthy).\nEnvoy Total Connections — total vs parent connections in use (envoy_total_connections_used, envoy_parent_connections_used).\nEnvoy Heap Memory — Envoy memory in MB: heap used / max, allocated used / max, and physical size / max (envoy_heap_memory_used, envoy_heap_memory_max_used, envoy_memory_allocated, envoy_memory_allocated_max, envoy_memory_physical_size, envoy_memory_physical_size_max).\nEnvoy Worker Threads — live vs max worker threads (envoy_worker_threads, envoy_worker_threads_max).\nEnvoy Bug Failures — Envoy\u0026rsquo;s own assertion / bug counter; expected to be zero in healthy clusters (envoy_bug_failures).\nEndpoint dashboard For one selected endpoint.\nEndpoint Traffic — calls per minute for the endpoint (endpoint_cpm).\nEndpoint Avg Response Time — average latency in ms (endpoint_resp_time).\nEndpoint Success Rate — percent of successful calls (endpoint_sla/100).\nEndpoint Response Time Percentile — p50 / p75 / p90 / p95 / p99 latency for the endpoint (endpoint_percentile).\nSidecar Internal Latency (endpoint scope) — Envoy-internal request and response latency on this endpoint, in nanoseconds (endpoint_sidecar_internal_req_latency_nanos, endpoint_sidecar_internal_resp_latency_nanos).\nTopology and maps The MESH layer ships the service map and the instance (sidecar) map. There is no endpoint-dependency map for this layer.\nTopology (service map) — every service node is decorated with RPM (service_cpm), an SLA health ring (service_sla/100, colored green / amber / red — higher is better, so the ring turns red as success rate drops), and Latency (service_resp_time). Each call edge carries server-side and client-side RPM, Avg response time, p95, and SLA (service_relation_server_* and service_relation_client_*), shown aligned in the edge panel.\nInstance map (sidecars) — from a call between two services on the service map, drill into the sidecar-to-sidecar calls between them. The same node / edge metric set is evaluated at instance scope: node RPM (service_instance_cpm), SLA ring (service_instance_sla/100), and Latency (service_instance_resp_time); each edge shows server-side and client-side RPM, Avg response time, p95, and SLA (service_instance_relation_server/client_*).\nFor how these maps are read and navigated, see the 3D Infrastructure Map for the cross-layer view, and the topology section of Layer Dashboard Templates for how the node and edge metrics are configured.\nTraces MESH traces are served from Zipkin. Open the Traces tab to query the sidecar-reported spans; the workflow and filters are the same as any other layer\u0026rsquo;s trace view — see Traces.\nRequirements The MESH dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nEnvoy ALS enabled — the sidecars must stream access logs to OAP via the Access Log Service so it can derive the service / instance / endpoint traffic, latency, and SLA metrics. See the Envoy ALS setup guide.\nService / instance / endpoint metrics — the service_*, service_instance_*, and endpoint_* families (traffic, response time, SLA, apdex, percentiles), including the mesh-specific *_throughput_* and *_sidecar_internal_*_latency_nanos metrics produced from the ALS stream.\nRelation metrics — service_relation_* and service_instance_relation_* for the service map and the sidecar map.\nEnvoy-runtime metrics — the envoy_cluster_*, envoy_*_connections_used, envoy_*_memory_*, envoy_worker_threads*, and envoy_bug_failures families, for the Envoy-runtime instance widgets to appear. A sidecar only shows the families its Envoy build emits.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so an instance- or endpoint-scope metric is empty until that level of data is reported. When a whole family is missing (for example a sidecar that does not export Envoy-runtime metrics), its widgets are hidden rather than shown empty.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/mesh/","title":"\u003c!--"},{"body":" Istio Control Plane The MESH_CP layer monitors the Istio control plane — the istiod / Pilot process that distributes configuration to the data-plane proxies. SkyWalking scrapes the control-plane\u0026rsquo;s Prometheus metrics over OpenTelemetry and rolls them into per-control-plane meters, so operators can watch xDS push health, proxy convergence, configuration validation, and the Go runtime of istiod itself. See the upstream Istio monitoring setup for how to wire the telemetry pipeline.\nIn Horizon\u0026rsquo;s sidebar this layer is named Istio Control Plane (grouped under Istio). Its services are listed as Control Planes — each control plane is named service.namespace, with the namespace shown as its grouping alias. The MESH_CP layer enables only the Service sub-tab; it has no instance, endpoint, topology, traces, or logs view.\nThis page is the operator reference for the bundled MESH_CP dashboard: what you see on the service scope and what each widget means.\nThe widgets and metrics below are read from the bundled MESH_CP template; if an operator has published a customized MESH_CP template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a control plane, the layer landing page lists every Istio control plane with four sortable columns, sorted by CPU by default:\nCPU — average CPU usage of the control-plane process (meter_istio_cpu).\nGoroutines — total goroutines running in istiod (meter_istio_go_goroutines).\nPilot xDS — total xDS connections Pilot is serving (meter_istio_pilot_xds).\nServices — total services Pilot knows about (meter_istio_pilot_services).\nService dashboard The primary drill-down for one selected control plane, covering its Go runtime, xDS push pipeline, configuration validation, and proxy conflicts.\nCPU — CPU usage of the control-plane process over time (meter_istio_cpu).\nGoroutines — goroutines running in istiod (meter_istio_go_goroutines).\nIstio Versions — the reported Pilot / Istio version build info, so a version roll-out is visible on the timeline (meter_istio_pilot_version).\nMemory (MB) — the Go runtime\u0026rsquo;s memory footprint on one chart, in MB: allocated, heap in-use, stack in-use, virtual, and resident (meter_istio_go_alloc, meter_istio_go_heap_inuse, meter_istio_go_stack_inuse, meter_istio_virtual_memory, meter_istio_resident_memory, each /1024/1024).\nPilot Errors — xDS rejections and push timeouts that indicate the control plane could not deliver config: CDS / EDS / RDS / LDS rejects plus write timeouts (meter_istio_pilot_xds_cds_reject, meter_istio_pilot_xds_eds_reject, meter_istio_pilot_xds_rds_reject, meter_istio_pilot_xds_lds_reject, meter_istio_pilot_xds_write_timeout).\nProxy Push Time (percentile) — how long it takes to push config to proxies, as a latency percentile distribution in ms (meter_istio_pilot_proxy_push_percentile).\nPilot Pushes — the rate of xDS pushes Pilot sends to proxies (meter_istio_pilot_xds_pushes).\nSidecar Injection Success — successful sidecar-injection webhook calls (meter_istio_sidecar_injection_success_total).\nADS Monitoring — the aggregated discovery surface on one chart: xDS connections, known services, and virtual services (meter_istio_pilot_xds, meter_istio_pilot_services, meter_istio_pilot_virt_services).\nConfiguration Validation — Galley configuration-validation outcomes, passed vs failed (meter_istio_galley_validation_passed, meter_istio_galley_validation_failed).\nPilot Conflicts — listener conflicts Pilot detected while generating config, broken out by type: outbound TCP/TCP, inbound, outbound TCP/HTTP, and outbound HTTP/TCP (meter_istio_pilot_conflict_ol_tcp_tcp, meter_istio_pilot_conflict_il, meter_istio_pilot_conflict_ol_tcp_http, meter_istio_pilot_conflict_ol_http_tcp).\nRequirements The MESH_CP dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the Istio control-plane meter family, produced from istiod\u0026rsquo;s Prometheus metrics collected over OpenTelemetry:\nControl-plane metrics — the meter_istio_* family for the service list and the service dashboard: the Go runtime (meter_istio_cpu, meter_istio_go_goroutines, the meter_istio_go_* memory gauges, meter_istio_virtual_memory, meter_istio_resident_memory), the Pilot / xDS pipeline (meter_istio_pilot_xds, meter_istio_pilot_xds_pushes, the meter_istio_pilot_xds_*_reject rejection counters, meter_istio_pilot_xds_write_timeout, meter_istio_pilot_proxy_push_percentile, meter_istio_pilot_services, meter_istio_pilot_virt_services, meter_istio_pilot_version, the meter_istio_pilot_conflict_* counters), sidecar injection (meter_istio_sidecar_injection_success_total), and Galley validation (meter_istio_galley_validation_passed, meter_istio_galley_validation_failed). All MESH_CP metrics are reported at the control-plane Service scope; OAP does not roll a metric up across scopes, so the dashboard stays empty until the control plane\u0026rsquo;s telemetry is reported. See the upstream Istio monitoring setup for the collection pipeline that produces this family.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/mesh_cp/","title":"\u003c!--"},{"body":" Istio Data Plane The MESH_DP layer monitors the Istio data plane — the Envoy sidecar proxies that carry the mesh\u0026rsquo;s traffic. Where the service-mesh control-plane and request telemetry live in the MESH layer, MESH_DP is the proxy\u0026rsquo;s own view: the runtime health of each Envoy process and its upstream clusters, fed by Envoy\u0026rsquo;s metrics-service output. It is grouped under Istio in the sidebar.\nIn Horizon\u0026rsquo;s sidebar this layer\u0026rsquo;s entities are the Envoy sidecars themselves. Its top-level entities are listed as Sidecar services, and each sidecar process is a Sidecars instance — there is no separate per-application service or endpoint slot, so MESH_DP reads \u0026ldquo;Sidecar service / Sidecar\u0026rdquo; rather than the GENERAL layer\u0026rsquo;s \u0026ldquo;Service / Instance / Endpoint\u0026rdquo;. Sidecar names follow the Istio name.namespace convention, so the namespace is surfaced as the displayed grouping.\nMESH_DP enables the Sidecar (instance) dashboard plus the Logs and eBPF profiling tabs; pod logs are available for the sidecar. It does not ship a service dashboard, an endpoint dashboard, a topology / map view, or a traces tab — the layer is scoped to per-sidecar runtime metrics, so those sections are absent.\nThis page is the operator reference for the bundled MESH_DP dashboard: what you see on the sidecar scope and what each widget means.\nThe widgets and metrics below are read from the bundled MESH_DP template; if an operator has published a customized MESH_DP template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nSidecar service list The layer landing page lists every sidecar service. This layer defines no metric columns on the list, so the landing view is a plain, namespace-grouped roster of sidecar services — pick one to open its sidecars, then drill into a single sidecar\u0026rsquo;s dashboard.\nSidecar dashboard For one selected sidecar (an Envoy instance). The dashboard opens with four single-value status cards, then a set of time-series trends.\nStatus cards\nBug Failures — Envoy\u0026rsquo;s internal bug-failure counter; a non-zero value means an assertion or debug check tripped inside the proxy (envoy_bug_failures).\nMembership Healthy — the count of healthy endpoints across all of this Envoy\u0026rsquo;s upstream clusters (envoy_cluster_membership_healthy).\nWorker Threads — concurrent worker threads currently in use (envoy_worker_threads).\nUpstream Request Active — total active upstream requests across this Envoy\u0026rsquo;s clusters (envoy_cluster_up_rq_active).\nConnections and requests\nUpstream Connection Active — active upstream connections over time (envoy_cluster_up_cx_active).\nUpstream Request Pending — requests waiting in upstream queues (envoy_cluster_up_rq_pending_active).\nConnections Used — server-side connections in use, plotted as total and parent (envoy_total_connections_used, envoy_parent_connections_used).\nUpstream Connection Increase — new upstream connections opened per minute (envoy_cluster_up_cx_incr).\nUpstream Request Increase — new upstream requests per minute (envoy_cluster_up_rq_incr).\nThreads and memory\nWorker Threads (current vs max) — concurrent worker threads in use plotted against the window maximum, as current and max (envoy_worker_threads, envoy_worker_threads_max).\nServer Memory — the proxy\u0026rsquo;s memory footprint in bytes, each line paired with its window maximum: heap / heap max (envoy_heap_memory_used, envoy_heap_memory_max_used), allocated / allocated max (envoy_memory_allocated, envoy_memory_allocated_max), and physical / physical max (envoy_memory_physical_size, envoy_memory_physical_size_max).\nLogs and profiling Beyond the dashboard, the sidecar\u0026rsquo;s triage tabs are:\nLogs — the log stream is scoped to the sidecar (instance), so logs are read against the selected Envoy proxy rather than a higher-level service.\neBPF profiling — on-CPU / network profiling of the sidecar process via the eBPF profiling workflow.\nPod logs are also available for the sidecar, surfacing the underlying pod\u0026rsquo;s container output alongside the proxy\u0026rsquo;s own log stream.\nRequirements The MESH_DP dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the Envoy metrics-service receiver enabled and the sidecars configured to push their metrics to it. Every widget on the sidecar dashboard reads the envoy_* metric family at instance scope:\nProxy health — envoy_bug_failures, envoy_cluster_membership_healthy.\nWorker threads — envoy_worker_threads, envoy_worker_threads_max.\nUpstream clusters — envoy_cluster_up_rq_active, envoy_cluster_up_cx_active, envoy_cluster_up_rq_pending_active, envoy_cluster_up_cx_incr, envoy_cluster_up_rq_incr.\nServer connections — envoy_total_connections_used, envoy_parent_connections_used.\nServer memory — envoy_heap_memory_used, envoy_heap_memory_max_used, envoy_memory_allocated, envoy_memory_allocated_max, envoy_memory_physical_size, envoy_memory_physical_size_max.\nThese are instance-scope metrics; OAP does not roll a metric up across scopes, so the dashboard is empty until each sidecar\u0026rsquo;s Envoy is actually reporting to the metrics-service receiver. For how to point Envoy at OAP, see Envoy\u0026rsquo;s metrics service setting.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/mesh_dp/","title":"\u003c!--"},{"body":" MongoDB The MONGODB layer monitors MongoDB database clusters. SkyWalking collects MongoDB\u0026rsquo;s internal metrics — document and operation throughput, connections, cursors, replication lag and buffer, per-database data and index size, and per-node host stats — and Horizon renders them as a cluster-level and a node-level dashboard. This is a metrics-only layer: there are no traces, logs, endpoints, or a topology map for MongoDB.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Databases and named MongoDB. Its services are listed as MongoDB clusters and its instances as Nodes. Because the layer carries no endpoint, topology, trace, or log data, it enables only two sub-tabs: a Service (cluster) dashboard and an Instance (node) dashboard.\nThis page is the operator reference for the bundled MONGODB dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled MONGODB template; if an operator has published a customized MONGODB template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every MongoDB cluster with four sortable columns, sorted by document throughput (Doc QPS) by default:\nDoc QPS — document operations per second across the cluster, summed over the document-operation types (aggregate_labels(meter_mongodb_cluster_document_avg_qps, sum(doc_op_type))).\nOp QPS — database operations per second across the cluster, summed over the operation types (aggregate_labels(meter_mongodb_cluster_operation_avg_qps, sum(legacy_op_type))).\nConns — total open connections across the cluster (aggregate_labels(meter_mongodb_cluster_connections,sum)).\nRepl Lag — replication lag in ms (meter_mongodb_cluster_repl_lag).\nService dashboard The cluster-level drill-down for one selected MongoDB cluster. Most widgets aggregate across the cluster\u0026rsquo;s nodes with aggregate_labels(...).\nCluster Uptime (days) — how long the cluster has been running, as a single card, taking the max uptime across nodes and converting from seconds (latest(aggregate_labels(meter_mongodb_cluster_uptime,max))/3600/24).\nData Size (GB) — total stored data across the cluster, as a card, summed across nodes and converted from bytes (latest(aggregate_labels(meter_mongodb_cluster_data_size,sum))/1024/1024/1024).\nCollection Count — total number of collections across the cluster, as a card (latest(aggregate_labels(meter_mongodb_cluster_collection_count,sum))).\nObject Count — total number of objects (documents) across the cluster, as a card (latest(aggregate_labels(meter_mongodb_cluster_object_count,sum))).\nDocument QPS — document operations per second over time, summed over the document-operation types (aggregate_labels(meter_mongodb_cluster_document_avg_qps, sum(doc_op_type))).\nOperation QPS — database operations per second over time, summed over the operation types (aggregate_labels(meter_mongodb_cluster_operation_avg_qps, sum(legacy_op_type))).\nTotal Connections — open connections across the cluster over time (aggregate_labels(meter_mongodb_cluster_connections,sum)).\nCursor Total — open cursors across the cluster, summed over the cursor types (aggregate_labels(meter_mongodb_cluster_cursor_avg, sum(csr_type))).\nReplication Lag (ms) — replication lag in ms (meter_mongodb_cluster_repl_lag).\nDB Total Data (GB) — a per-database table of stored data size in GB, summed per database and converted from bytes, with columns Database and Data (GB) (latest(aggregate_labels(meter_mongodb_cluster_db_data_size, sum(database)))/1024/1024/1024).\nDB Total Index (GB) — a per-database table of index size in GB, summed per database and converted from bytes, with columns Database and Index (GB) (latest(aggregate_labels(meter_mongodb_cluster_db_index_size, sum(database)))/1024/1024/1024).\nInstance dashboard The node-level drill-down for one selected MongoDB node. These widgets read the per-node meter_mongodb_node_* family directly, with no cross-node aggregation.\nUptime (days) — how long the node has been running, as a card, converted from seconds (latest(meter_mongodb_node_uptime)/3600/24).\nQPS — total query throughput on the node (meter_mongodb_node_qps).\nReplSet State — a table of the node\u0026rsquo;s replica-set state, with columns Node and ReplSet state (latest(meter_mongodb_node_rs_state)).\nConnections — open connections on the node (meter_mongodb_node_connections).\nCPU Usage (%) — total CPU usage percentage on the node (meter_mongodb_node_cpu_total_percentage).\nMemory Usage — memory used by the node (meter_mongodb_node_memory_usage).\nMemory Free (GB) — free memory in GB as two series, mem and swap, each converted from KB (meter_mongodb_node_memory_free_kb/1024/1024, meter_mongodb_node_swap_memory_free_kb/1024/1024).\nDisk (GB) — filesystem used vs total in GB, converted from bytes (meter_mongodb_node_fs_used_size/1024/1024/1024, meter_mongodb_node_fs_total_size/1024/1024/1024).\nNetwork (KB/s) — network throughput in KB/s as in vs out, converted from bytes (meter_mongodb_node_network_bytes_in/1024, meter_mongodb_node_network_bytes_out/1024).\nActive Clients — active client connections as total, writers, and readers (meter_mongodb_node_active_total_num, meter_mongodb_node_active_writer_num, meter_mongodb_node_active_reader_num).\nDocument QPS — document operations per second on the node (meter_mongodb_node_document_qps).\nOperation QPS — database operations per second on the node (meter_mongodb_node_operation_qps).\nOp Latency (µs) — average operation latency in microseconds, computed as total latency divided by operation count, each summed over the operation types (aggregate_labels(meter_mongodb_node_latency_rate,sum(op_type))/aggregate_labels(meter_mongodb_node_op_rate,sum(op_type))).\nTransactions — active vs inactive transactions on the node (meter_mongodb_node_transactions_active, meter_mongodb_node_transactions_inactive).\nRepl Buffer — replication buffer count and size (MB), the size converted from bytes (meter_mongodb_node_repl_buffer_count, meter_mongodb_node_repl_buffer_size/1024/1024).\nQueued Operations — operations queued on the node (meter_mongodb_node_queued_operation).\nRequirements The MONGODB dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs MongoDB metrics, which OAP aggregates into the meter_mongodb_* families:\nCluster (service-scope) metrics — the meter_mongodb_cluster_* family (uptime, data and index size, collection and object counts, document and operation QPS, connections, cursors, and replication lag), which the cluster dashboard and the Service list aggregate across nodes.\nNode (instance-scope) metrics — the meter_mongodb_node_* family (uptime, QPS, replica-set state, connections, CPU, memory, disk, network, active clients, document and operation QPS, operation latency, transactions, replication buffer, and queued operations) for the node dashboard.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the node-scope widgets stay empty until per-node data is reported. Setting up MongoDB collection is described in the upstream MongoDB monitoring guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/mongodb/","title":"\u003c!--"},{"body":" MySQL / MariaDB The MYSQL layer monitors MySQL and MariaDB servers. It is populated by OAP\u0026rsquo;s MySQL/MariaDB monitoring, which scrapes a Prometheus-style mysqld-exporter and turns the result into SkyWalking meters — there is no language agent here, the data comes from the exporter.\nIn Horizon\u0026rsquo;s sidebar this layer lives under the Databases group and is named MySQL / MariaDB. A monitored cluster is listed as a MySQL cluster and each member server as a Node. This is a metrics-only layer: it enables the Service and Instance scopes and nothing else — there is no endpoint scope, no topology, and no traces or logs tabs.\nThis page is the operator reference for the bundled MySQL / MariaDB dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled MYSQL template; if an operator has published a customized MYSQL template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every MySQL cluster with four sortable columns, sorted by QPS by default. Each column aggregates the per-node meters across the whole cluster:\nQPS — queries per second across the cluster (aggregate_labels(meter_mysql_qps,sum)).\nTPS — transactions per second across the cluster (aggregate_labels(meter_mysql_tps,sum)).\nSlow QPS — slow queries per second across the cluster (aggregate_labels(meter_mysql_slow_queries_rate,sum)).\nConn Errors — connection-error rate, internal rejects plus max-connection rejects summed (aggregate_labels(meter_mysql_connection_errors_internal,sum) + aggregate_labels(meter_mysql_connection_errors_max_connections,sum)).\nService dashboard The primary drill-down for one selected cluster. Every widget here aggregates across the cluster\u0026rsquo;s nodes with aggregate_labels(..., sum).\nQPS — cluster-wide queries per second (aggregate_labels(meter_mysql_qps,sum)).\nTPS — cluster-wide transactions per second (aggregate_labels(meter_mysql_tps,sum)).\nSlow Queries / s — cluster-wide slow-query rate (aggregate_labels(meter_mysql_slow_queries_rate,sum)).\nConnection Errors — two series, internal rejects vs max-connection rejects (aggregate_labels(meter_mysql_connection_errors_internal,sum) and aggregate_labels(meter_mysql_connection_errors_max_connections,sum)).\nCommands Trend — rows-affected rate per command type: select / insert / update / delete (aggregate_labels(meter_mysql_commands_select_rate,sum), meter_mysql_commands_insert_rate, meter_mysql_commands_update_rate, meter_mysql_commands_delete_rate).\nThreads — thread counters: connected / running / cached / created (aggregate_labels(meter_mysql_threads_connected,sum), meter_mysql_threads_running, meter_mysql_threads_cached, meter_mysql_threads_created).\nSlow Statements — the top 20 slowest captured statements across the cluster, in ms, worst-first (top_n(top_n_database_statement, 20, des)). Each row carries the sampled statement text. Shows no data when OAP captured no statements in the window.\nInstance dashboard For one selected Node (a single MySQL/MariaDB server in the cluster). The four top cards are point-in-time configuration / status readings; the rest are per-node time series.\nStatus cards\nUptime — how long the server has been up, in days (latest(meter_mysql_instance_uptime)/3600/24).\nMax Connections — the server\u0026rsquo;s configured max_connections ceiling (latest(meter_mysql_instance_max_connections)).\nInnoDB Buffer Pool — the InnoDB buffer-pool size in MB (latest(meter_mysql_instance_innodb_buffer_pool_size)/1024/1024).\nThread Cache Size — the configured thread-cache size (latest(meter_mysql_instance_thread_cache_size)).\nTime series\nQPS / TPS — this node\u0026rsquo;s queries per second and transactions per second on one chart (meter_mysql_instance_qps, meter_mysql_instance_tps).\nSlow Queries / s — this node\u0026rsquo;s slow-query rate (meter_mysql_instance_slow_queries_rate).\nCommands Trend — rows-affected rate per command type for this node: select / insert / update / delete (meter_mysql_instance_commands_select_rate, meter_mysql_instance_commands_insert_rate, meter_mysql_instance_commands_update_rate, meter_mysql_instance_commands_delete_rate).\nThreads — this node\u0026rsquo;s thread counters: connected / running / cached / created (meter_mysql_instance_threads_connected, meter_mysql_instance_threads_running, meter_mysql_instance_threads_cached, meter_mysql_instance_threads_created).\nConnects — available vs aborted connection rate (meter_mysql_instance_connects_available, meter_mysql_instance_connects_aborted).\nConnection Errors — internal rejects vs max-connection rejects for this node (meter_mysql_instance_connection_errors_internal, meter_mysql_instance_connection_errors_max_connections).\nRequirements The MySQL / MariaDB dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs MySQL/MariaDB monitoring enabled so it scrapes a mysqld-exporter and produces:\nCluster (service-scope) meters — the meter_mysql_* family: QPS / TPS, slow-query rate, connection errors, the per-command rates, and the thread counters. These back the Service list and the Service dashboard.\nNode (instance-scope) meters — the meter_mysql_instance_* family: uptime, max-connections, InnoDB buffer-pool size, thread-cache size, and the per-node QPS/TPS, slow-query, command, thread, connect, and connection-error series. These back the Instance dashboard.\nSampled statements — top_n_database_statement for the Slow Statements list, captured by OAP when slow-statement sampling is configured.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the node-scope meter_mysql_instance_* series are empty until per-node data is reported. For the upstream setup steps — exporter configuration and which OAP rules to enable — see the MySQL/MariaDB monitoring documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/mysql/","title":"\u003c!--"},{"body":" Nginx The NGINX layer monitors Nginx servers and reverse proxies. Nginx, with the SkyWalking Lua module, reports request, latency, bandwidth, connection, status, and error-log telemetry, which OAP aggregates into the meter_nginx_* families this dashboard renders. The layer key is NGINX, and in Horizon\u0026rsquo;s sidebar it is grouped under Gateways.\nIn the sidebar this layer\u0026rsquo;s services are listed as Nginx services, its instances as Nodes (the individual Nginx server nodes), and its endpoints as Routes (the matched Nginx routes). The NGINX layer enables three metric scopes — Service, Instance (Node), and Endpoint (Route) — plus a Logs tab. It does not ship a topology or a traces tab, so apart from logs this dashboard is the metric drill-down across those three scopes.\nThis page is the operator reference for the bundled NGINX dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled NGINX template; if an operator has published a customized NGINX template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a service, the layer landing page lists every NGINX service, sorted by request rate (RPS) by default, with three columns:\nRPS — total HTTP requests per second across the service\u0026rsquo;s nodes (aggregate_labels(meter_nginx_service_http_requests, sum)).\n5xx % — percent of requests that returned a 5xx status, as a share of all requests over the window (aggregate_labels(meter_nginx_service_http_5xx_requests_increment, sum)/aggregate_labels(meter_nginx_service_http_requests_increment, sum)*100).\n4xx % — percent of requests that returned a 4xx status, as a share of all requests over the window (aggregate_labels(meter_nginx_service_http_4xx_requests_increment, sum)/aggregate_labels(meter_nginx_service_http_requests_increment, sum)*100).\nThe two error columns give an at-a-glance health read across the fleet — a service with a climbing 5xx % is failing requests at the proxy, a climbing 4xx % is rejecting client requests.\nService dashboard The primary drill-down for one selected Nginx service. All eight widgets aggregate across the service\u0026rsquo;s nodes.\nHTTP Request Trend — total requests per second for the service (aggregate_labels(meter_nginx_service_http_requests, sum)).\nHTTP Latency — request latency in ms, averaged across the reported percentiles (aggregate_labels(meter_nginx_service_http_latency, avg(p))).\nHTTP Bandwidth — bandwidth in KB/s, summed across the bandwidth types (aggregate_labels(meter_nginx_service_http_bandwidth, sum(type)), divided to KB/s).\nHTTP Connections — connections summed by state (aggregate_labels(meter_nginx_service_http_connections, sum(state)), one line per state).\nHTTP Status Trend — requests summed by HTTP status (aggregate_labels(meter_nginx_service_http_status, sum(status)), one line per status).\n4xx % / min — percent of requests returning a 4xx status per minute (aggregate_labels(meter_nginx_service_http_4xx_requests_increment, sum)/aggregate_labels(meter_nginx_service_http_requests_increment, sum)*100).\n5xx % / min — percent of requests returning a 5xx status per minute (aggregate_labels(meter_nginx_service_http_5xx_requests_increment, sum)/aggregate_labels(meter_nginx_service_http_requests_increment, sum)*100).\nError Log Count — count of error-log entries summed by log level (aggregate_labels(meter_nginx_service_error_log_count, sum(level)), one line per level).\nInstance dashboard For one selected node. These widgets are reported per node, so they show how one Nginx server process is behaving.\nHTTP Request Trend — requests per second for the node (meter_nginx_instance_http_requests).\nHTTP Latency — request latency in ms for the node (meter_nginx_instance_http_latency).\nHTTP Bandwidth — bandwidth in KB/s for the node (meter_nginx_instance_http_bandwidth, divided to KB/s).\nHTTP Connections — connections by state for the node (meter_nginx_instance_http_connections).\nHTTP Status Trend — requests by HTTP status for the node (meter_nginx_instance_http_status).\n4xx % / min — percent of requests returning a 4xx status per minute for the node ((meter_nginx_instance_http_4xx_requests_increment/meter_nginx_instance_http_requests_increment)*100).\n5xx % / min — percent of requests returning a 5xx status per minute for the node ((meter_nginx_instance_http_5xx_requests_increment/meter_nginx_instance_http_requests_increment)*100).\nError Log Count — count of error-log entries for the node (meter_nginx_instance_error_log_count).\nEndpoint dashboard For one selected route. Nginx reports a tighter metric set at route scope — requests, latency, bandwidth, status, and the per-route error rates.\nHTTP Request Trend — requests per second for the route (meter_nginx_endpoint_http_requests).\nHTTP Latency — request latency in ms for the route (meter_nginx_endpoint_http_latency).\nHTTP Bandwidth — bandwidth in KB for the route (meter_nginx_endpoint_http_bandwidth, divided to KB).\nHTTP Status Trend — requests by HTTP status for the route (meter_nginx_endpoint_http_status).\n4xx % / min — percent of requests returning a 4xx status per minute for the route ((meter_nginx_endpoint_http_4xx_requests_increment/meter_nginx_endpoint_http_requests_increment)*100).\n5xx % / min — percent of requests returning a 5xx status per minute for the route ((meter_nginx_endpoint_http_5xx_requests_increment/meter_nginx_endpoint_http_requests_increment)*100).\nLogs The NGINX layer enables the Logs tab. Nginx access and error logs forwarded to OAP are searchable here, scoped to the selected Nginx service, with the standard log filters and time range. This is the same logs experience as other log-enabled layers — see the layer logs view for how to filter, page, and inspect entries.\nRequirements The NGINX dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Nginx telemetry flowing in:\nService metrics — the meter_nginx_service_* family (requests, latency, bandwidth, connections, status, the 4xx / 5xx increment counters, and error-log count), aggregated by OAP across a service\u0026rsquo;s nodes.\nInstance (node) metrics — the meter_nginx_instance_* family for the per-node request, latency, bandwidth, connection, status, error-rate, and error-log widgets.\nEndpoint (route) metrics — the meter_nginx_endpoint_* family for the per-route request, latency, bandwidth, status, and error-rate widgets.\nLogs — Nginx access / error logs shipped to OAP, for the Logs tab.\nThese come from the SkyWalking Nginx Lua module emitting Nginx telemetry to OAP, which converts it into the meter_nginx_* metrics through its meter-analysis rules. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a route-scope or node-scope metric is empty until that level of data is reported. For the end-to-end setup, see the Nginx monitoring backend guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/nginx/","title":"\u003c!--"},{"body":" Linux The OS_LINUX layer monitors Linux hosts. It is populated by OAP\u0026rsquo;s VM monitoring, which scrapes a Prometheus node-exporter and turns the result into SkyWalking meters — there is no language agent here, the data comes from the exporter.\nIn Horizon\u0026rsquo;s sidebar this layer lives under the OS group and is named Linux. Each monitored host is listed as a Host. This is a metrics-only, single-scope layer: it enables the Service scope and nothing else — there is no instance scope, no endpoint scope, no topology, and no traces or logs tabs.\nThis page is the operator reference for the bundled Linux dashboard: what you see on the host scope and what each widget means.\nThe widgets and metrics below are read from the bundled OS_LINUX template; if an operator has published a customized OS_LINUX template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nHost list Before opening a host, the layer landing page lists every Linux host with four sortable columns, sorted by CPU % by default:\nCPU % — average CPU utilization across all cores (meter_vm_cpu_total_percentage).\nMemory MB — memory in use, in MB (meter_vm_memory_used/1024/1024).\nLoad 1m — the 1-minute load average (meter_vm_cpu_load1/100).\nFS % — filesystem space used, as a percent (meter_vm_filesystem_percentage).\nHost dashboard The primary drill-down for one selected host.\nCPU Average Used (%) — average CPU utilization across cores, as a percent (meter_vm_cpu_average_used).\nCPU Load — the load average at three windows: 1m / 5m / 15m (meter_vm_cpu_load1/100, meter_vm_cpu_load5/100, meter_vm_cpu_load15/100).\nFile FD Allocated — the number of allocated file descriptors (meter_vm_filefd_allocated).\nMemory RAM (MB) — four series in MB: used / total / available / buff/cache (meter_vm_memory_used/1024/1024, meter_vm_memory_total/1024/1024, meter_vm_memory_available/1024/1024, meter_vm_memory_buff_cache/1024/1024).\nMemory Swap (MB) — swap free vs swap total, in MB (meter_vm_memory_swap_free/1024/1024, meter_vm_memory_swap_total/1024/1024).\nNetwork Bandwidth (KB/s) — receive vs transmit throughput, in KB/s (meter_vm_network_receive/1024, meter_vm_network_transmit/1024).\nDisk R/W (KB/s) — disk read vs written throughput, in KB/s (meter_vm_disk_read/1024, meter_vm_disk_written/1024).\nFilesystem Usage (%) — filesystem space used, as a percent (meter_vm_filesystem_percentage).\nNetwork Status — five socket / TCP counters: established TCP connections, TCP time-wait, TCP alloc, sockets used, and UDP in-use (meter_vm_tcp_curr_estab, meter_vm_tcp_tw, meter_vm_tcp_alloc, meter_vm_sockets_used, meter_vm_udp_inuse).\nRequirements The Linux dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs VM monitoring enabled so it scrapes a node-exporter and produces:\nHost (service-scope) meters — the meter_vm_* family: CPU utilization and load average, memory (used / total / available / buff-cache / swap), file-descriptor allocation, network receive/transmit, disk read/written, filesystem usage, and the TCP / socket / UDP counters. These back the Host list and the Host dashboard. Each metric is queried at its own OAP scope; because this layer is service-scope only, every widget reads the host-level meter_vm_* series and there is no instance- or endpoint-level rollup. For the upstream setup steps — node-exporter configuration and which OAP rules to enable — see the VM monitoring documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/os_linux/","title":"\u003c!--"},{"body":" Windows The OS_WINDOWS layer monitors Windows hosts. It is populated by OAP\u0026rsquo;s Windows monitoring, which receives host telemetry (CPU, memory, network, disk) and turns it into SkyWalking meters — there is no language agent here, the data comes from the host telemetry.\nIn Horizon\u0026rsquo;s sidebar this layer lives under the OS group and is named Windows. Each monitored Windows machine is listed as a Host. This is a metrics-only, single-scope layer: it enables the Service scope and nothing else — there is no instance or endpoint scope, no topology, and no traces or logs tabs.\nThis page is the operator reference for the bundled Windows dashboard: what you see and what each widget means.\nThe widgets and metrics below are read from the bundled OS_WINDOWS template; if an operator has published a customized OS_WINDOWS template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nHost list Before opening a host, the layer landing page lists every Windows host with three sortable columns, sorted by CPU % by default:\nCPU % — average CPU utilization across the host (meter_win_cpu_total_percentage).\nMemory MB — physical memory used, in MB (meter_win_memory_used/1024/1024).\nVMem % — virtual-memory utilization percentage (avg(meter_win_memory_virtual_memory_percentage)).\nHost dashboard The primary drill-down for one selected Windows host.\nCPU Average Used (%) — average CPU utilization over the window (meter_win_cpu_average_used).\nMemory RAM (MB) — physical memory in MB, three series: used / total / available (meter_win_memory_used/1024/1024, meter_win_memory_total/1024/1024, meter_win_memory_available/1024/1024).\nVirtual Memory (MB) — virtual (page-file backed) memory in MB, free vs total (meter_win_memory_virtual_memory_free/1024/1024, meter_win_memory_virtual_memory_total/1024/1024).\nNetwork Bandwidth (KB/s) — network throughput in KB/s, receive vs transmit (meter_win_network_receive/1024, meter_win_network_transmit/1024).\nDisk R/W (KB/s) — disk throughput in KB/s, read vs written (meter_win_disk_read/1024, meter_win_disk_written/1024).\nRequirements The Windows dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Windows monitoring enabled so it ingests host telemetry and produces the host (service-scope) meter_win_* family:\nCPU — meter_win_cpu_total_percentage and meter_win_cpu_average_used back the CPU column and the CPU widget.\nMemory — meter_win_memory_used, meter_win_memory_total, meter_win_memory_available, and the meter_win_memory_virtual_memory_* series back the memory columns and the RAM / virtual-memory widgets.\nNetwork and disk — meter_win_network_receive / meter_win_network_transmit and meter_win_disk_read / meter_win_disk_written back the network and disk throughput widgets.\nEvery metric is queried at the Service (host) scope; OAP does not roll a metric up across scopes, so the dashboard stays empty until per-host data is reported. For the upstream setup steps — host-telemetry collection and which OAP rules to enable — see the Windows monitoring documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/os_windows/","title":"\u003c!--"},{"body":" Mesh Dashboard The Mesh Dashboard is the cross-layer overview for an Istio service mesh. Where the Services Dashboard centers on language-agent traffic, this one centers on the data plane: it pulls onto one screen the services routed through the Istio data plane, the Istio control-plane (pilot / xDS) push activity that keeps them configured, and — because a mesh always runs on Kubernetes — the same cluster capacity strip. It draws from the MESH, MESH_CP, and K8S layers.\nLike every overview, it sits at the top of the sidebar above the per-layer entries and appears only while at least one of its layers is reporting; a layer\u0026rsquo;s tile auto-hides when that layer has nothing reporting (refreshed on the same ~60-second cadence as the menu).\nThe widgets and metrics below are read from the bundled overview template; if an administrator has published a customized copy to OAP, the live page reflects that copy instead. These are editable defaults — reshape them in the Overview Templates admin page on a bundled-default → local-draft → Check diff \u0026amp; push flow. See Overview Templates for the editor and the stored format, and Overview Widgets for the widget vocabulary.\nMesh services row Istio-managed services (MESH) — a KPI tile with the mesh service count plus RPM (calls per minute, service_cpm), P95 (95th-percentile latency in ms, service_percentile{p='95'}), and SLA (percent successful, service_sla/100). This is the data-plane equivalent of the General-services tile. Istio pilot (MESH_CP) — a composite summarizing control-plane activity: xDS pushes (config pushes Pilot sent, meter_istio_pilot_xds_pushes), xDS connections (proxies currently connected to Pilot, meter_istio_pilot_xds), Services (the layer\u0026rsquo;s service count), and Pilot errors (rejected pushes + write timeouts across CDS / EDS / LDS / RDS, summed: meter_istio_pilot_xds_cds_reject+meter_istio_pilot_xds_eds_reject+meter_istio_pilot_xds_lds_reject+meter_istio_pilot_xds_rds_reject+meter_istio_pilot_xds_write_timeout). A climbing Pilot-errors number means the control plane is struggling to push valid config — a mesh-specific failure the data-plane tiles won\u0026rsquo;t surface. Topology \u0026amp; active alarms Mesh service topology — a live service map of the MESH layer, the bulk of the row. Same renderer as the per-layer Topology tab. Active alarms — the right-hand rail of alarms currently firing on mesh-reported services, up to 12. Read-only — alarm recovery is backend-automatic. Kubernetes Cluster capacity \u0026amp; utilisation — the same full-width K8S composite as the Services Dashboard: cluster inventory counts (Nodes, Namespaces, Deployments, StatefulSets, DaemonSets, Services, Containers) on the left, and CPU / Memory / Storage commitment bars on the right (same k8s_cluster_* metrics and 0 – 100 % scale). Mesh deployments always ride on Kubernetes, so the capacity block lives directly under the mesh health. Requirements An overview is a pure consumer of what OAP reports — it invents no data, and a tile or panel with no backing metric simply hides or reads no data. To populate the Mesh Dashboard, OAP needs:\nService-scope metrics on the MESH layer — the service_* family (traffic, response time, percentile, SLA), produced by OAP from the mesh-reported telemetry. Queried at its own OAP scope; OAP does not roll a metric up across scopes. Istio control-plane meters — the meter_istio_pilot_* family on the MESH_CP layer, for the Istio pilot composite. Relation metrics for the embedded service map — service_relation_* at the MESH layer. Alarm data — firing alarms scoped to the MESH layer, for the Active-alarms rail. Kubernetes cluster metrics — the k8s_cluster_* family on the K8S layer, for the capacity composite. When a whole layer is missing — no mesh, no Kubernetes monitoring — its tile or block is hidden rather than shown empty, and the overview drops out of the sidebar once none of its layers are reporting.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/overview-mesh/","title":"\u003c!--"},{"body":" Services Dashboard The Services Dashboard is Horizon\u0026rsquo;s default cross-layer overview — the \u0026ldquo;is everything OK?\u0026rdquo; war-room pane for your traced application services. It pulls several layers onto one screen at once: a row of count + health tiles for application and virtual-backend services, a live service map, the alarms firing right now, and the Kubernetes capacity underneath them all. It answers \u0026ldquo;how many services are up, how hard are they working, is anything on fire, and is the cluster running out of room\u0026rdquo; without you clicking into any one service.\nIt folds in the GENERAL, VIRTUAL_DATABASE, VIRTUAL_CACHE, VIRTUAL_MQ, VIRTUAL_GENAI, and K8S layers; any of those that isn\u0026rsquo;t reporting drops its tile automatically. Overviews are listed at the top of the sidebar, above the per-layer entries, and each appears only while at least one of its layers is reporting (refreshed on the same ~60-second cadence as the menu). For the service-mesh counterpart, see the Mesh Dashboard.\nThe widgets and metrics below are read from the bundled overview template; if an administrator has published a customized copy to OAP, the live page reflects that copy instead. These are editable defaults — reshape them (add / remove / resize widgets, swap MQE) in the Overview Templates admin page on a bundled-default → local-draft → Check diff \u0026amp; push flow. See Overview Templates for the editor and the stored format, and Overview Widgets for the widget vocabulary.\nServices row Five KPI tiles, one per service-class layer, each showing that layer\u0026rsquo;s reporting service count plus three headline numbers:\nGeneral services (GENERAL) — traced application services. RPM (total calls per minute, service_cpm), Latency (average response time in ms, service_resp_time), SLA (percent successful, service_sla/100). Virtual databases (VIRTUAL_DATABASE) — backend databases observed via client-side spans. RPM (database_access_cpm), Latency (database_access_resp_time), SLA (database_access_sla/100). Virtual caches (VIRTUAL_CACHE) — Redis / Memcached / … observed via client-side spans. RPM (cache_access_cpm), Latency (cache_access_resp_time), SLA (cache_access_sla/100). Virtual MQs (VIRTUAL_MQ) — message queues observed via consume + produce spans. Consume (consume rate per minute, mq_service_consume_cpm), Produce (produce rate per minute, mq_service_produce_cpm), Consume latency (ms, mq_service_consume_latency). Virtual GenAI (VIRTUAL_GENAI) — GenAI backends observed via instrumented client spans. RPM (gen_ai_provider_cpm), Latency (gen_ai_provider_resp_time), SLA (gen_ai_provider_sla/100). The RPM / consume / produce numbers are summed across the layer; latency and SLA are averaged. A layer with nothing reporting (no GenAI backends in this deployment, say) simply leaves its tile off the row.\nTopology \u0026amp; active alarms General service topology — a live service map of the GENERAL layer, taking up most of the row. Same map you see on the per-layer Topology tab, embedded here for the war-room at-a-glance view. Active alarms — a rail down the right side listing the alarms currently firing on agent-reported (GENERAL) services, up to 12 at a time. Read-only — alarm recovery is backend-automatic. Kubernetes Cluster capacity \u0026amp; utilisation — a full-width composite summarizing the K8S layer. On the left, the cluster inventory as latest counts: Nodes (k8s_cluster_node_total), Namespaces (k8s_cluster_namespace_total), Deployments (k8s_cluster_deployment_total), StatefulSets (k8s_cluster_statefulset_total), DaemonSets (k8s_cluster_daemonset_total), Services (k8s_cluster_service_total), and Containers (k8s_cluster_container_total). On the right, three utilisation bars showing how much of the cluster is already committed — CPU (requested cores over capacity, k8s_cluster_cpu_cores_requests/k8s_cluster_cpu_cores*100), Memory (requested over total, k8s_cluster_memory_requests/k8s_cluster_memory_total*100), and Storage (allocated over total, (k8s_cluster_storage_total-k8s_cluster_storage_allocatable)/k8s_cluster_storage_total*100), each on a 0 – 100 % scale. This block is the \u0026ldquo;are we about to run out of room\u0026rdquo; check that the service tiles above can\u0026rsquo;t tell you. Requirements An overview is a pure consumer of what OAP reports — it invents no data, and a tile or panel with no backing metric simply hides (the layer count drops to zero and the tile is omitted) or reads no data. To populate the Services Dashboard, OAP needs:\nService-scope metrics for each service-class layer — the service_* family for GENERAL (traffic, response time, SLA), and the virtual-backend families database_access_*, cache_access_*, mq_service_*, and gen_ai_provider_* for the virtual layers. Each is queried at its own OAP scope; OAP does not roll a metric up across scopes. Relation metrics for the embedded service map — service_relation_* at the GENERAL layer. Alarm data — firing alarms scoped to the layer, for the Active-alarms rail. Kubernetes cluster metrics — the k8s_cluster_* family (inventory totals plus CPU / memory / storage capacity), reported by the OAP Kubernetes monitoring on the K8S layer, for the capacity composite. When a whole layer is missing — no virtual MQs, no Kubernetes monitoring — its tile or block is hidden rather than shown empty, and the overview itself drops out of the sidebar once none of its layers are reporting.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/overview-services/","title":"\u003c!--"},{"body":" PostgreSQL The POSTGRESQL layer monitors PostgreSQL servers. It is populated by OAP\u0026rsquo;s PostgreSQL monitoring, which scrapes a Prometheus-style postgres-exporter and turns the result into SkyWalking meters — there is no language agent here, the data comes from the exporter.\nIn Horizon\u0026rsquo;s sidebar this layer lives under the Databases group and is named PostgreSQL. A monitored cluster is listed as a PostgreSQL cluster and each member server as a Node. This is a metrics-only layer: it enables the Service and Instance scopes and nothing else — there is no endpoint scope, no topology, and no traces or logs tabs.\nThis page is the operator reference for the bundled PostgreSQL dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled POSTGRESQL template; if an operator has published a customized POSTGRESQL template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every PostgreSQL cluster with four sortable columns, sorted by Fetched / s by default. Each column aggregates the per-node meters across the whole cluster:\nFetched / s — rows fetched per second across the cluster (aggregate_labels(meter_pg_fetched_rows_rate,sum)).\nInserted / s — rows inserted per second across the cluster (aggregate_labels(meter_pg_inserted_rows_rate,sum)).\nCache Hit — buffer-cache hit ratio across the cluster, in percent (aggregate_labels(meter_pg_cache_hit_rate,avg)).\nDeadlocks — deadlocks per second across the cluster (aggregate_labels(meter_pg_deadlocks_rate,sum)).\nService dashboard The primary drill-down for one selected cluster. Most widgets here aggregate across the cluster\u0026rsquo;s nodes with aggregate_labels(...).\nFetched Rows / s — cluster-wide rows fetched per second (aggregate_labels(meter_pg_fetched_rows_rate,sum)).\nInserted Rows / s — cluster-wide rows inserted per second (aggregate_labels(meter_pg_inserted_rows_rate,sum)).\nUpdated Rows / s — cluster-wide rows updated per second (aggregate_labels(meter_pg_updated_rows_rate,sum)).\nDeleted Rows / s — cluster-wide rows deleted per second (aggregate_labels(meter_pg_deleted_rows_rate,sum)).\nReturned Rows / s — cluster-wide rows returned per second (aggregate_labels(meter_pg_returned_rows_rate,sum)).\nTemporary Files / s — temporary files created per second across the cluster, a sign of queries spilling to disk (aggregate_labels(meter_pg_temporary_files_rate,sum)).\nCache Hit Rate — cluster-wide buffer-cache hit ratio in percent (aggregate_labels(meter_pg_cache_hit_rate,avg)).\nTransactions / s — committed vs rolled-back transactions per second (aggregate_labels(meter_pg_committed_transactions_rate,sum) and aggregate_labels(meter_pg_rolled_back_transactions_rate,sum)).\nConflicts + Deadlocks / s — two series, recovery conflicts vs deadlocks per second (aggregate_labels(meter_pg_conflicts_rate,sum) and aggregate_labels(meter_pg_deadlocks_rate,sum)).\nSessions — active vs idle sessions and the lock count across the cluster (aggregate_labels(meter_pg_active_sessions,sum), aggregate_labels(meter_pg_idle_sessions,sum), aggregate_labels(meter_pg_locks_count,sum)).\nBuffers / s — background-writer and checkpoint buffer activity: checkpoint / clean / backend fsync / alloc / backend (aggregate_labels(meter_pg_buffers_checkpoint,sum), aggregate_labels(meter_pg_buffers_clean,sum), aggregate_labels(meter_pg_buffers_backend_fsync,sum), aggregate_labels(meter_pg_buffers_alloc,sum), aggregate_labels(meter_pg_buffers_backend,sum)).\nCheckpoint Stats / s — checkpoint counters: timed / requested / write time / sync time (aggregate_labels(meter_pg_checkpoints_timed_rate,sum), aggregate_labels(meter_pg_checkpoint_req_rate,sum), aggregate_labels(meter_pg_checkpoint_write_time_rate,sum), aggregate_labels(meter_pg_checkpoint_sync_time_rate,sum)).\nSlow Statements — the top 20 slowest captured statements across the cluster, in ms, worst-first (top_n(top_n_database_statement, 20, des)). Each row carries the sampled statement text. Shows no data when OAP captured no statements in the window.\nInstance dashboard For one selected Node (a single PostgreSQL server in the cluster). The four top cards are point-in-time configuration readings; the rest are per-node time series.\nStatus cards\nShared Buffers — the node\u0026rsquo;s configured shared_buffers size in MB (latest(meter_pg_instance_shared_buffers)/1024/1024).\nEffective Cache — the node\u0026rsquo;s configured effective_cache_size in GB (latest(meter_pg_instance_effective_cache)/1024/1024/1024).\nWork Mem — the node\u0026rsquo;s configured work_mem in MB (latest(meter_pg_instance_work_mem)/1024/1024).\nMax WAL Size — the node\u0026rsquo;s configured max_wal_size in GB (latest(meter_pg_instance_max_wal_size)/1024/1024/1024).\nTime series\nFetched Rows / s — this node\u0026rsquo;s rows fetched per second (meter_pg_instance_fetched_rows_rate).\nInserted Rows / s — this node\u0026rsquo;s rows inserted per second (meter_pg_instance_inserted_rows_rate).\nCache Hit Rate — this node\u0026rsquo;s buffer-cache hit ratio in percent (meter_pg_instance_cache_hit_rate).\nSessions — this node\u0026rsquo;s active vs idle sessions and lock count (meter_pg_instance_active_sessions, meter_pg_instance_idle_sessions, meter_pg_instance_locks_count).\nConflicts + Deadlocks / s — recovery conflicts vs deadlocks per second for this node (meter_pg_instance_conflicts_rate, meter_pg_instance_deadlocks_rate).\nRequirements The PostgreSQL dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs PostgreSQL monitoring enabled so it scrapes a postgres-exporter and produces:\nCluster (service-scope) meters — the meter_pg_* family: the per-operation row rates (fetched / inserted / updated / deleted / returned), temporary-file rate, cache-hit ratio, committed and rolled-back transaction rates, conflicts and deadlocks, active / idle sessions and locks, and the background-writer buffer and checkpoint counters. These back the Service list and the Service dashboard.\nNode (instance-scope) meters — the meter_pg_instance_* family: the configured shared_buffers, effective_cache_size, work_mem, and max_wal_size readings, plus the per-node fetched / inserted row rates, cache-hit ratio, sessions and locks, and conflict / deadlock series. These back the Instance dashboard.\nSampled statements — top_n_database_statement for the Slow Statements list, captured by OAP when slow-statement sampling is configured.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the node-scope meter_pg_instance_* series are empty until per-node data is reported. For the upstream setup steps — exporter configuration and which OAP rules to enable — see the PostgreSQL monitoring documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/postgresql/","title":"\u003c!--"},{"body":" Pulsar The PULSAR layer monitors Apache Pulsar message brokers. SkyWalking collects Pulsar\u0026rsquo;s broker metrics through OpenTelemetry and renders each Pulsar cluster as a service, with its brokers as instances — so a cluster\u0026rsquo;s topic, subscription, and message-flow health sits beside the broker-level connection and JVM detail in one place.\nIn Horizon\u0026rsquo;s sidebar this layer is named Pulsar, grouped under MQ. Its services are listed as Pulsar clusters and its instances as Brokers. The PULSAR layer enables the Service and Instance sub-tabs only — there is no endpoint scope, no topology, and no traces or logs tab for this layer.\nThis page is the operator reference for the bundled PULSAR dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled PULSAR template; if an operator has published a customized PULSAR template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nPulsar cluster list Before opening a cluster, the layer landing page lists every Pulsar cluster with four sortable columns, sorted by Topics by default. Each column sums the per-label series across the cluster:\nTopics — total topics on the cluster (meter_pulsar_total_topics).\nSubscriptions — total subscriptions on the cluster (meter_pulsar_total_subscriptions).\nMsg In — incoming message rate (meter_pulsar_message_rate_in).\nMsg Out — outgoing message rate (meter_pulsar_message_rate_out).\nService dashboard The primary drill-down for one selected Pulsar cluster. Every widget aggregates the cluster\u0026rsquo;s per-label series with aggregate_labels(..., sum), giving cluster-wide totals.\nTotal Topics — number of topics on the cluster (meter_pulsar_total_topics).\nSubscriptions — number of subscriptions on the cluster (meter_pulsar_total_subscriptions).\nProducers — number of connected producers (meter_pulsar_total_producers).\nConsumers — number of connected consumers (meter_pulsar_total_consumers).\nMessage Rate — incoming vs outgoing message rate on one chart, plotted as in (meter_pulsar_message_rate_in) and out (meter_pulsar_message_rate_out).\nThroughput — incoming vs outgoing byte throughput on one chart, plotted as in (meter_pulsar_throughput_in) and out (meter_pulsar_throughput_out).\nStorage Read/Write Rate — bookkeeper storage read vs write rate, plotted as read (meter_pulsar_storage_read_rate) and write (meter_pulsar_storage_write_rate).\nStorage Size (MB) — physical vs logical storage size in MB, plotted as physical (meter_pulsar_storage_size) and logical (meter_pulsar_storage_logical_size); both are reported in bytes and divided by 1024 / 1024 for display.\nInstance dashboard For one selected broker. These widgets read the broker-scope meter_pulsar_broker_* family directly.\nActive Connections — connections currently open on the broker (meter_pulsar_broker_active_connections).\nTotal Connections — connections handled by the broker (meter_pulsar_broker_total_connections).\nConn Create Fail — failed connection-create attempts (meter_pulsar_broker_connection_create_fail_count).\nConn Create Success — successful connection-create attempts (meter_pulsar_broker_connection_create_success_count).\nConnection Closed — total connections closed (meter_pulsar_broker_connection_closed_total_count).\nJVM Buffer Pool (MB) — JVM buffer-pool bytes used by the broker, in MB (meter_pulsar_broker_jvm_buffer_pool_used_bytes, divided by 1024 / 1024).\nJVM Memory Pool Used (MB) — JVM memory-pool bytes used, in MB (meter_pulsar_broker_jvm_memory_pool_used, divided by 1024 / 1024).\nJVM Memory (MB) — JVM memory in MB plotted as used, committed, and init (meter_pulsar_broker_jvm_memory_used, meter_pulsar_broker_jvm_memory_committed, meter_pulsar_broker_jvm_memory_init, each divided by 1024 / 1024).\nJVM Threads — thread counts plotted as current, daemon, peak, and deadlocked (meter_pulsar_broker_jvm_threads_current, meter_pulsar_broker_jvm_threads_daemon, meter_pulsar_broker_jvm_threads_peak, meter_pulsar_broker_jvm_threads_deadlocked).\nGC — garbage-collection time vs count on a dual axis, with cumulative seconds on the left axis (meter_pulsar_broker_jvm_gc_collection_seconds_sum) and count on the right axis (meter_pulsar_broker_jvm_gc_collection_seconds_count).\nRequirements The PULSAR dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Pulsar monitoring enabled, so that the broker\u0026rsquo;s OpenTelemetry metrics reach OAP and are aggregated into the Pulsar meter families:\nCluster (service) metrics — the meter_pulsar_* family (topics, subscriptions, producers, consumers, message rate, throughput, and bookkeeper storage), which back the cluster list and the Service dashboard.\nBroker (instance) metrics — the meter_pulsar_broker_* family (connections and the broker JVM buffer / memory / thread / GC detail), which back the Instance dashboard.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a broker-scope metric is empty until that broker reports it. See Pulsar monitoring for how to wire a Pulsar deployment into OAP.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/pulsar/","title":"\u003c!--"},{"body":" RabbitMQ The RABBITMQ layer monitors RabbitMQ message brokers. OAP collects the metrics from RabbitMQ\u0026rsquo;s Prometheus / OpenMetrics endpoint, so each broker cluster and each broker node surfaces as a SkyWalking entity in this layer.\nIn Horizon\u0026rsquo;s sidebar this layer is named RabbitMQ. Its services are listed as RabbitMQ clusters and its instances as Nodes — a service is one RabbitMQ cluster, and each instance is one broker node inside it. The layer enables two scopes only: the Service (cluster) dashboard and the Instance (node) dashboard. There is no endpoint scope, no topology / map, and no traces or logs tab for this layer.\nThis page is the operator reference for the bundled RABBITMQ dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled RABBITMQ template; if an operator has published a customized RABBITMQ template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every RabbitMQ cluster with four sortable columns, sorted by Queues by default:\nQueues — total queues across the cluster (aggregate_labels(meter_rabbitmq_queues,sum)). Channels — total open channels across the cluster (aggregate_labels(meter_rabbitmq_channels,sum)). Connections — total open connections across the cluster (aggregate_labels(meter_rabbitmq_connections,sum)). Unconfirmed — publisher messages awaiting confirmation across the cluster (aggregate_labels(meter_rabbitmq_messages_unconfirmed,sum)). Service dashboard The cluster-level view for one selected RabbitMQ cluster.\nMemory Available Before Block (MB) — headroom in MB before the broker hits its memory high-watermark and starts blocking publishers (meter_rabbitmq_memory_available_before_publisher_blocked). Disk Available Before Block (GB) — headroom in GB before the broker hits its disk free-space limit and starts blocking publishers (meter_rabbitmq_disk_space_available_before_publisher_blocked). File Descriptors + Sockets — available file descriptors (fds) and available TCP sockets (sockets), the two resource pools that gate how many connections the broker can still accept (meter_rabbitmq_file_descriptors_available, meter_rabbitmq_tcp_socket_available). Ready Messages — messages ready to be delivered to consumers (meter_rabbitmq_message_ready_delivered_consumers). Pending Ack — messages delivered to consumers but not yet acknowledged (meter_rabbitmq_message_unacknowledged_delivered_consumers). Publish Pipeline — the publish path across four series: published, confirmed, routed, and unconfirmed (meter_rabbitmq_messages_published, meter_rabbitmq_messages_confirmed, meter_rabbitmq_messages_routed, meter_rabbitmq_messages_unconfirmed). A growing gap between published and confirmed/routed flags a routing or confirmation problem. Unroutable Messages — messages with no matching binding, split into dropped and returned (meter_rabbitmq_messages_unroutable_dropped, meter_rabbitmq_messages_unroutable_returned). Queues — queue lifecycle across the cluster: total currently present, plus the declared, created, and deleted running totals (meter_rabbitmq_queues, meter_rabbitmq_queues_declared_total, meter_rabbitmq_queues_created_total, meter_rabbitmq_queues_deleted_total). Channels — channel lifecycle: total currently open, plus the opened and closed running totals (meter_rabbitmq_channels, meter_rabbitmq_channels_opened_total, meter_rabbitmq_channels_closed_total). Connections — connection lifecycle: total currently open, plus the opened and closed running totals (meter_rabbitmq_connections, meter_rabbitmq_connections_opened_total, meter_rabbitmq_connections_closed_total). Instance dashboard The node-level view for one selected broker node. The cards across the top are single-value (latest) readings; the remaining widgets are time-series.\nReady Messages — messages ready for delivery on this node, latest value (latest(meter_rabbitmq_node_queue_messages_ready)). Incoming Messages — incoming message rate on this node, latest value (latest(meter_rabbitmq_node_incoming_messages)). Outgoing Messages — outgoing message total on this node, latest value (latest(meter_rabbitmq_node_outgoing_messages_total)). Unacknowledged Messages — delivered-but-unacknowledged messages on this node, latest value (latest(meter_rabbitmq_node_unacknowledged_messages)). Connections / Publishers / Consumers — the node\u0026rsquo;s connections, publishers, and consumers counts (latest(meter_rabbitmq_node_connections_total), latest(meter_rabbitmq_node_publisher_total), latest(meter_rabbitmq_node_consumer_total)). Channels + Queues — the node\u0026rsquo;s channels and queues counts (latest(meter_rabbitmq_node_channel_total), latest(meter_rabbitmq_node_queue_total)). Allocated Used % — percentage of the node\u0026rsquo;s allocated memory that is in use, latest value (latest(meter_rabbitmq_node_allocated_used_percent)). Memory (MB) — the node\u0026rsquo;s memory breakdown in MB: used, unused, resident, and total allocated (meter_rabbitmq_node_allocated_used_bytes, meter_rabbitmq_node_allocated_unused_bytes, meter_rabbitmq_node_process_resident_memory_bytes, meter_rabbitmq_node_allocated_total_bytes). Allocated By Type (MB) — allocated memory broken down by allocator type in MB, one series per type (meter_rabbitmq_node_allocated_by_type). Multi/Single-block Memory (MB) — allocator block usage in MB across multi used, multi unused, single used, and single unused (meter_rabbitmq_node_allocated_multiblock_used, meter_rabbitmq_node_allocated_multiblock_unused, meter_rabbitmq_node_allocated_singleblock_used, meter_rabbitmq_node_allocated_singleblock_unused). Requirements The RABBITMQ dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nCluster (service-scope) metrics — the meter_rabbitmq_* family (memory and disk headroom, file descriptors and sockets, ready / pending / unroutable messages, the publish pipeline, and queue / channel / connection lifecycle counters) that drives the service list and Service dashboard. Node (instance-scope) metrics — the meter_rabbitmq_node_* family (message counters, connections / publishers / consumers, channels / queues, and the allocator memory breakdown) that drives the Instance dashboard. These metrics come from OAP\u0026rsquo;s RabbitMQ monitoring, which scrapes the broker\u0026rsquo;s Prometheus / OpenMetrics endpoint. See the RabbitMQ monitoring setup in the SkyWalking backend documentation for how to enable it. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the node-scope widgets stay empty until per-node data is reported.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/rabbitmq/","title":"\u003c!--"},{"body":" Redis The REDIS layer monitors Redis deployments scraped through OpenTelemetry\u0026rsquo;s Redis receiver and forwarded to OAP as meters. It groups under Databases in the sidebar and is a metrics-only layer: each Redis cluster is a service, and the individual Redis processes under it are instances.\nIn Horizon\u0026rsquo;s sidebar this layer\u0026rsquo;s services are listed as Redis clusters, and the processes under a cluster as Nodes. The REDIS layer enables only the Service and Instance scopes — it has no endpoint dashboard, no topology or maps, and no Traces or Logs tabs.\nThis page is the operator reference for the bundled REDIS dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled REDIS template; if an operator has published a customized REDIS template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every Redis cluster with four sortable columns, sorted by command throughput (Commands/s) by default:\nCommands/s — total commands per second across the cluster, summed over command types (aggregate_labels(meter_redis_total_commands_rate,sum(cmd))).\nHit Rate — keyspace hit rate as a percentage, averaged across the cluster (aggregate_labels(meter_redis_hit_rate,avg)).\nMemory % — used memory as a percentage of max memory across the cluster (latest(aggregate_labels(meter_redis_memory_used_bytes,sum)/aggregate_labels(meter_redis_memory_max_bytes,sum))*100).\nClients — connected clients across the cluster (latest(aggregate_labels(meter_redis_connected_clients,sum))).\nService dashboard The primary drill-down for one selected Redis cluster. All cluster-scope widgets aggregate over the nodes that make up the cluster.\nStatus cards\nUptime (days) — cluster uptime in days, taken from the longest-running node (latest(aggregate_labels(meter_redis_uptime,max))/3600/24).\nConnected Clients — total connected clients across the cluster (latest(aggregate_labels(meter_redis_connected_clients,sum))).\nBlocked Clients — total clients blocked on a blocking call across the cluster (latest(aggregate_labels(meter_redis_blocked_clients,sum))).\nMemory Usage — used memory as a percentage of max memory across the cluster (latest(aggregate_labels(meter_redis_memory_used_bytes,sum)/aggregate_labels(meter_redis_memory_max_bytes,sum))*100).\nCharts\nTotal Commands / s — commands per second across the cluster, summed over command types (aggregate_labels(meter_redis_total_commands_rate,sum(cmd))).\nHit Rate — keyspace hit rate as a percentage (aggregate_labels(meter_redis_hit_rate,avg)).\nAvg Command Time / s — mean per-command duration, total command duration divided by total command count, summed over command types (aggregate_labels(meter_redis_commands_duration,sum(cmd))/aggregate_labels(meter_redis_commands_total,sum(cmd))).\nNet I/O (KB) — network throughput in KB, split into in and out (aggregate_labels(meter_redis_net_input_bytes_total,sum)/1024, aggregate_labels(meter_redis_net_output_bytes_total,sum)/1024).\nKeys — keyspace size over time, split into total keys, evicted keys, and expired keys (aggregate_labels(meter_redis_db_keys,sum), aggregate_labels(meter_redis_evicted_keys_total,sum), aggregate_labels(meter_redis_expired_keys_total,sum)).\nSlow Commands — the top 10 slowest captured commands in ms, sampled by the SkyWalking agent at the call site (top_n(top_n_database_statement,10,des)). Each row carries the command text. Shows no data when OAP captured no slow commands in the window.\nInstance dashboard For one selected node (a single Redis process under the cluster).\nStatus cards\nUptime (days) — node uptime in days (latest(meter_redis_instance_uptime)/3600/24).\nConnected Clients — clients connected to this node (latest(meter_redis_instance_connected_clients)).\nBlocked Clients — clients blocked on a blocking call on this node (latest(meter_redis_instance_redis_blocked_clients)).\nMemory Max (MB) — configured max memory for this node in MB (latest(meter_redis_instance_memory_max_bytes)/1000/1000).\nCharts\nMemory Usage (%) — used memory as a percentage of max for this node (meter_redis_instance_memory_usage).\nCommands / s — commands per second on this node (meter_redis_instance_total_commands_rate).\nHit Rate — keyspace hit rate as a percentage for this node (meter_redis_instance_hit_rate).\nNet I/O (KB) — network throughput in KB for this node, split into in and out (meter_redis_instance_net_input_bytes_total/1024, meter_redis_instance_net_output_bytes_total/1024).\nKeys — keyspace size over time for this node, split into total, evicted, and expired keys (meter_redis_instance_db_keys, meter_redis_instance_evicted_keys_total, meter_redis_instance_expired_keys_total).\nTotal Command Time (s) — total time spent on commands per second for this node (meter_redis_instance_commands_duration_seconds_total_rate).\nAvg Command Time — mean time spent per command on this node (meter_redis_instance_average_time_spent_by_command).\nRequirements The REDIS dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nCluster meters — the meter_redis_* family (commands rate, hit rate, used / max memory, connected and blocked clients, uptime, network bytes, keyspace counts, command duration and count), aggregated by command label where the metric is per-command. These back the service list and the cluster dashboard.\nNode meters — the meter_redis_instance_* family (the same measures at single-process scope), which back the node dashboard.\nSampled records — top_n_database_statement for the Slow Commands list, captured by the SkyWalking agent at the call site when slow-command sampling is enabled.\nThese meters come from the OpenTelemetry Redis receiver; see SkyWalking\u0026rsquo;s Redis monitoring setup for how to wire the collector to OAP. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a node-scope metric is empty until that level of data is reported.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/redis/","title":"\u003c!--"},{"body":" RocketMQ The ROCKETMQ layer monitors Apache RocketMQ message-queue clusters. SkyWalking collects RocketMQ metrics over OpenTelemetry and rolls them up into cluster-, broker-, and topic-scope metrics, so operators can watch produce / consume throughput, message size, consumer latency and backlog, and broker disk and thread-pool pressure alongside the rest of their estate. See the upstream RocketMQ monitoring setup for how to wire the telemetry pipeline.\nIn Horizon\u0026rsquo;s sidebar this layer is named RocketMQ. Its services are listed as RocketMQ clusters, its instances as Brokers, and its endpoints as Topics. The ROCKETMQ layer enables the Service, Instance, and Endpoint sub-tabs; it has no topology, traces, or logs view.\nThis page is the operator reference for the bundled ROCKETMQ dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled ROCKETMQ template; if an operator has published a customized ROCKETMQ template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every RocketMQ cluster with four sortable columns, sorted by Produce TPS by default:\nProduce TPS — messages produced per second across the cluster (meter_rocketmq_cluster_total_producer_tps).\nConsume TPS — messages consumed per second across the cluster (meter_rocketmq_cluster_total_consumer_tps).\nTopics — the latest topic count in the cluster (latest(meter_rocketmq_cluster_topic_count)).\nBrokers — the latest broker count in the cluster (latest(meter_rocketmq_cluster_broker_count)).\nService dashboard The primary drill-down for one selected RocketMQ cluster, mixing daily message volume, live throughput, disk and thread-pool health, and the cluster\u0026rsquo;s topic / broker totals.\nProduced Today — messages produced since the start of today (latest(meter_rocketmq_cluster_messages_produced_today)).\nConsumed Today — messages consumed since the start of today (latest(meter_rocketmq_cluster_messages_consumed_today)).\nProduced Yesterday — messages produced over the previous full day (latest(meter_rocketmq_cluster_messages_produced_until_yesterday)).\nConsumed Yesterday — messages consumed over the previous full day (latest(meter_rocketmq_cluster_messages_consumed_until_yesterday)).\nProducer / Consumer TPS — produce and consume throughput per second on one chart (meter_rocketmq_cluster_total_producer_tps, meter_rocketmq_cluster_total_consumer_tps).\nProducer / Consumer Message Size (MB) — produced and consumed message size, in MB (meter_rocketmq_cluster_producer_message_size/1024/1024, meter_rocketmq_cluster_consumer_message_size/1024/1024).\nMax Consumer Latency — the highest consumer latency seen across the cluster (latest(meter_rocketmq_cluster_max_consumer_latency)).\nCommitLog Disk Ratio (%) — how full the CommitLog disk is, in percent: the current ratio over time plus the latest maximum across brokers (meter_rocketmq_cluster_commitLog_disk_ratio, latest(meter_rocketmq_cluster_max_commitLog_disk_ratio)).\nThreadPool Queue Head Wait (ms) — how long the head request has waited in the pull and send broker thread-pool queues, in ms — a rising value signals broker back-pressure (meter_rocketmq_cluster_pull_threadPool_queue_head_wait_time, meter_rocketmq_cluster_send_threadPool_queue_head_wait_time).\nTopics — the latest topic count in the cluster (latest(meter_rocketmq_cluster_topic_count)).\nBrokers — the latest broker count in the cluster (latest(meter_rocketmq_cluster_broker_count)).\nInstance dashboard For one selected broker, focused on the broker\u0026rsquo;s produce / consume throughput and message size.\nProduce TPS — messages produced per second by this broker (meter_rocketmq_broker_produce_tps).\nConsume QPS — consume requests per second served by this broker (meter_rocketmq_broker_consume_qps).\nProducer Msg Size (MB) — produced message size on this broker, in MB (meter_rocketmq_broker_producer_message_size/1024/1024).\nConsumer Msg Size (MB) — consumed message size on this broker, in MB (meter_rocketmq_broker_consumer_message_size/1024/1024).\nEndpoint dashboard For one selected topic, covering producer / consumer-group throughput, message size, consumer latency, offsets, and lag.\nProducer / Consumer Group TPS — produce throughput and consumer-group consume throughput per second on one chart (meter_rocketmq_topic_producer_tps, meter_rocketmq_topic_consumer_group_tps).\nMessage Size (MB) — produced and consumed message size for the topic, in MB (meter_rocketmq_topic_producer_message_size/1024/1024, meter_rocketmq_topic_consumer_message_size/1024/1024).\nMax Message Size (MB) — the latest maximum produced and consumed message size for the topic, in MB (latest(meter_rocketmq_topic_max_producer_message_size)/1024/1024, latest(meter_rocketmq_topic_max_consumer_message_size)/1024/1024).\nConsumer Latency (s) — consumer latency for the topic, in seconds (meter_rocketmq_topic_consumer_latency/1000).\nProducer / Consumer Offsets — the topic\u0026rsquo;s producer offset and consumer-group offset over time (meter_rocketmq_topic_producer_offset, meter_rocketmq_topic_consumer_group_offset).\nBacklogged Messages — the topic lag: producer offset minus consumer-group offset, the count of produced messages a consumer group has not yet consumed (meter_rocketmq_topic_producer_offset-meter_rocketmq_topic_consumer_group_offset).\nConsumer Group Count — the latest number of consumer groups on the topic (latest(meter_rocketmq_topic_consumer_group_count)).\nBroker Count — the latest number of brokers serving the topic (latest(meter_rocketmq_topic_broker_count)).\nRequirements The ROCKETMQ dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the RocketMQ meter families, produced from cluster telemetry collected over OpenTelemetry:\nCluster metrics — the meter_rocketmq_cluster_* family (total producer / consumer TPS, messages produced / consumed today and yesterday, producer / consumer message size, max consumer latency, CommitLog disk ratio, the pull / send thread-pool queue head-wait timers, and the topic / broker counts) for the service list and the cluster dashboard.\nBroker metrics — the meter_rocketmq_broker_* family (produce TPS, consume QPS, producer / consumer message size) for the broker dashboard.\nTopic metrics — the meter_rocketmq_topic_* family (producer TPS and consumer-group TPS, producer / consumer message size and their maxima, consumer latency, producer and consumer-group offsets, and the consumer-group / broker counts) for the topic dashboard.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a broker- or topic-scope metric is empty until that level of data is reported. See the upstream RocketMQ monitoring setup for the collection pipeline that produces these families.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/rocketmq/","title":"\u003c!--"},{"body":" Go Agent (Self-Observability) The SO11Y_GO_AGENT layer is the self-observability view of the SkyWalking Go agent itself. It does not measure the application the agent instruments — it measures the agent\u0026rsquo;s own tracing machinery: how many tracing contexts it creates and finishes, how many it ignores, where contexts may have leaked, and how long the agent spends building them. Use it to confirm a Go agent is healthy and not accumulating leaked contexts or interceptor errors.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Self-Observability and named Go Agent. Its services are listed as Agent services and its instances as Agents — each Agent is one running Go process reporting these meters. This is an instance-only layer: it enables the Instance sub-tab and nothing else. There is no Service dashboard, no Endpoint dashboard, no Topology, and no Traces or Logs tabs — the agent reports a flat set of self-observability meters per process, with no service-, endpoint-, or relation-scoped data behind them.\nThis page is the operator reference for the bundled SO11Y_GO_AGENT dashboard: what you see on the Agent (instance) scope and what each widget means.\nThe widgets and metrics below are read from the bundled SO11Y_GO_AGENT template; if an operator has published a customized SO11Y_GO_AGENT template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nAgent list Selecting the layer lists the Agent services, and under one Agent service its Agents (instances) — one row per reporting Go process. This layer adds no extra landing columns, so the list is the plain name list; pick an Agent to open its dashboard.\nInstance dashboard For one selected Agent (instance). Every widget on this dashboard is a Go-agent self-observability meter, charted over the selected time window.\nTracing Context Creation / min — tracing contexts the agent created per minute (meter_sw_go_created_tracing_context_count). This is the agent\u0026rsquo;s working rate — how many trace contexts it is spinning up to follow requests.\nTracing Created + Finished / min — created vs finished tracing contexts per minute on one chart: the created series (aggregate_labels(meter_sw_go_created_tracing_context_count,sum)) against the finished series (meter_sw_go_finished_tracing_context_count). In a healthy agent the two lines track each other; a persistent gap (created running ahead of finished) is the signal that contexts are not being closed.\nIgnored Context Creation / min — contexts the agent created but deliberately ignored per minute (meter_sw_go_created_ignored_context_count), e.g. traffic filtered out of tracing.\nIgnored Created + Finished / min — the same created-vs-finished comparison for ignored contexts: created (aggregate_labels(meter_sw_go_created_ignored_context_count,sum)) against finished (meter_sw_go_finished_ignored_context_count).\nPossible Leaked Context / min — contexts the agent flags as possibly leaked per minute (meter_sw_go_possible_leaked_context_count). A non-zero, sustained line here points at instrumentation that opens a context without closing it — the key health signal on this dashboard.\nInterceptor Error Count / min — errors raised inside the agent\u0026rsquo;s interceptors per minute (meter_sw_go_interceptor_error_count). Rising values indicate the agent is failing while wrapping calls, which can mean lost or incomplete traces.\nTracing Context Execution Time (ms) — the time the agent spends building a tracing context, as a p50 / p75 / p90 / p95 / p99 latency distribution in milliseconds (relabels(meter_sw_go_tracing_context_execution_time_percentile,p='50,75,90,95,99',p='50,75,90,95,99')/1000000). The agent reports this percentile in nanoseconds, so the dashboard divides by 1,000,000 to display milliseconds. Watch the tail (p95 / p99) for instrumentation overhead.\nRequirements The SO11Y_GO_AGENT dashboard is a pure consumer of what the Go agent reports through OAP — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the Go agent\u0026rsquo;s self-observability meter family at instance scope:\nAgent self-observability meters — the meter_sw_go_* family: created / finished / ignored / leaked tracing-context counts, interceptor error count, and the tracing-context execution-time percentile. These are emitted by the SkyWalking Go agent\u0026rsquo;s own self-observability reporting, not derived from the traced application. Every metric here is queried at the ServiceInstance (Agent) scope; OAP does not roll a metric up across scopes, so the dashboard stays empty until a Go agent is actively reporting these meters. When the meter family is missing entirely — for example a Go agent build with self-observability disabled — the widgets render no data rather than failing.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/so11y_go_agent/","title":"\u003c!--"},{"body":" Java Agent (Self-Observability) The SO11Y_JAVA_AGENT layer is the self-observability view of the SkyWalking Java agent itself — not the services it instruments, but the health of the agent running inside each Java process. It surfaces the agent\u0026rsquo;s own internal counters: how many tracing contexts it creates and finishes, how many it ignores, how many may have leaked, how often its interceptors error, and how long its tracing context bookkeeping takes. Use it to confirm an agent is healthy and to catch agent-side problems (context leaks, interceptor failures) that would otherwise be invisible from the application\u0026rsquo;s own metrics.\nIn Horizon\u0026rsquo;s sidebar this layer lives under the Self-Observability group and is named Java Agent. It has no service-level page: the layer reports per-agent, so its services are listed as Agent services and its instances as Agents, and the only drill-down it enables is the Instance (per-agent) dashboard. There is no Service, Endpoint, Topology, Traces, Logs, or profiling tab in this layer — agent self-observability is purely instance-scoped runtime telemetry.\nThis page is the operator reference for the bundled SO11Y_JAVA_AGENT dashboard: what you see on the agent dashboard and what each widget means.\nThe widgets and metrics below are read from the bundled SO11Y_JAVA_AGENT template; if an operator has published a customized copy to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nAgent list The layer landing page lists every reporting agent (Agents). This layer defines no extra landing columns, so the list is the agent roster on its own — pick an agent to open its dashboard.\nAgent dashboard The per-agent drill-down. Every widget is a time series of the agent\u0026rsquo;s own internal counters; the counts are per-minute rates and the one timing widget is in milliseconds.\nTracing Context Creation / min — how many tracing contexts the agent created per minute (meter_java_agent_created_tracing_context_count). This is the agent\u0026rsquo;s working rate: each context corresponds to a traced execution it started tracking. Tracing Created + Finished / min — created vs. finished tracing contexts on one chart, so you can see the two lines track each other (aggregate_labels(meter_java_agent_created_tracing_context_count,sum) as created, meter_java_agent_finished_tracing_context_count as finished). A persistent gap where created outruns finished points at contexts that never closed. Ignored Context Creation / min — contexts the agent deliberately skipped tracing per minute (meter_java_agent_created_ignored_context_count), for example traffic matched by the agent\u0026rsquo;s ignore/exclusion rules. Ignored Created + Finished / min — the created vs. finished pair for ignored contexts (aggregate_labels(meter_java_agent_created_ignored_context_count,sum) as created, meter_java_agent_finished_ignored_context_count as finished), the same balance check applied to the ignored path. Possible Leaked Context / min — contexts the agent suspects were leaked per minute (meter_java_agent_possible_leaked_context_count). A sustained non-zero line here is the headline agent-health warning: it usually means trace contexts are not being cleaned up correctly in the instrumented application. Interceptor Error Count / min — errors raised inside the agent\u0026rsquo;s bytecode interceptors per minute (meter_java_agent_interceptor_error_count). Non-zero values flag a misbehaving or incompatible plugin and warrant a look at the agent log. Tracing Context Execution Time (ms) — the p50 / p75 / p90 / p95 / p99 distribution of how long the agent\u0026rsquo;s tracing-context handling takes, in milliseconds (relabels(meter_java_agent_tracing_context_execution_time_percentile,p='50,75,90,95,99',p='50,75,90,95,99')/1000000). This is the agent\u0026rsquo;s own overhead tail; the percentile values are converted from nanoseconds to milliseconds for display. Requirements The SO11Y_JAVA_AGENT dashboard is a pure consumer of what the Java agent reports about itself — it invents no data, and a widget with no backing data simply reads no data. To populate it, the Java agent must have its self-observability (so11y) meters enabled so OAP receives the meter_java_agent_* family:\nContext counters — meter_java_agent_created_tracing_context_count, meter_java_agent_finished_tracing_context_count, meter_java_agent_created_ignored_context_count, meter_java_agent_finished_ignored_context_count, and meter_java_agent_possible_leaked_context_count for the creation, created-vs-finished, ignored, and leaked widgets. Interceptor errors — meter_java_agent_interceptor_error_count for the interceptor error widget. Execution-time percentiles — meter_java_agent_tracing_context_execution_time_percentile for the execution-time tail. All of these are reported at the ServiceInstance scope (one agent = one instance), which is why this layer has only the agent dashboard and no service, endpoint, or topology view. An agent that does not emit the self-observability meter family will appear in the list but render no data on every widget.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/so11y_java_agent/","title":"\u003c!--"},{"body":" OAP (Self-Observability) The SO11Y_OAP layer is SkyWalking\u0026rsquo;s own self-observability — the OAP backend reporting metrics about itself. It answers \u0026ldquo;is the backend healthy?\u0026rdquo;: each OAP node\u0026rsquo;s JVM, the analysis pipelines it runs (trace / mesh / OTEL / K8s ALS), the GraphQL query surface the UI itself hits, and the storage backend it persists to. This is the layer you watch to tell whether OAP — not the services it monitors — is the bottleneck.\nIn Horizon\u0026rsquo;s sidebar this layer is named OAP, grouped under Self-Observability. Its services are listed as OAP services and its instances as OAP nodes — one node per running OAP backend in the cluster.\nUnlike the application layers, SO11Y_OAP is a node-only layer: it ships a single instance (OAP node) dashboard and no service, endpoint, topology, traces, or logs tabs. There is no per-node landing table — pick an OAP node and you land directly on its dashboard.\nThis page is the operator reference for the bundled SO11Y_OAP dashboard: what you see on the OAP-node dashboard and what each widget means.\nThe widgets and metrics below are read from the bundled SO11Y_OAP template; if an operator has published a customized SO11Y_OAP template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nOAP node dashboard For one selected OAP node. Every widget on this dashboard is OAP-node-scoped, fed by the meter_oap_* self-observability meter family.\nJVM health The runtime the OAP node runs on.\nCPU (%) — process CPU utilization for the OAP node (meter_oap_instance_cpu_percentage).\nJVM Memory (MB) — JVM heap memory used (meter_oap_instance_jvm_memory_bytes_used, converted to MB).\nGC Count / min — garbage-collection count per minute (meter_oap_instance_jvm_gc_count).\nGC Time (ms / min) — time spent in garbage collection per minute (meter_oap_instance_jvm_gc_time).\nBuffer Pool (MB) — JVM buffer-pool memory used (meter_oap_instance_jvm_buffer_pool_bytes_used, converted to MB).\nThread Count — JVM threads broken out as live, peak, and daemon (meter_oap_jvm_thread_live_count, meter_oap_jvm_thread_peak_count, meter_oap_jvm_thread_daemon_count).\nThread States — threads by state: runnable, timed-waiting, blocked, waiting (meter_oap_jvm_thread_runnable_count, meter_oap_jvm_thread_timed_waiting_count, meter_oap_jvm_thread_blocked_count, meter_oap_jvm_thread_waiting_count).\nClass Count — loaded, unloaded total, and loaded total classes (meter_oap_jvm_class_loaded_count, meter_oap_jvm_class_total_unloaded_count, meter_oap_jvm_class_total_loaded_count).\nMetrics aggregation and persistence How much work the analysis-and-write pipeline is doing on this node.\nAggregation / min — metrics aggregated per minute (meter_oap_instance_metrics_aggregation).\nPersistence Counts / min — persistence operations per minute, split into prepare and execute (meter_oap_instance_persistence_prepare_count, meter_oap_instance_persistence_execute_count).\nPersistent Cache / min — persistent-cache activity per minute (meter_oap_instance_metrics_persistent_cache).\nPersistence Prepare Latency (ms) — p50 / p75 / p90 / p95 / p99 latency of the persistence prepare phase (meter_oap_instance_persistence_prepare_percentile).\nPersistence Execute Latency (ms) — p50 / p75 / p90 / p95 / p99 latency of the persistence execute phase (meter_oap_instance_persistence_execute_percentile).\nAggregation Queue Usage (%) — fill level of the L1 and L2 metrics-aggregation queues, top-10 worst series each (meter_oap_instance_metrics_aggregation_queue_used_per_ten_thousand, level 1 and level 2). A queue trending toward 100% is back-pressure — OAP is ingesting faster than it can aggregate.\nQuery surface (GraphQL) The query API that Horizon (and any GraphQL client) hits.\nGraphQL Latency (ms) — p50 / p75 / p90 / p95 / p99 latency of GraphQL queries served by this node (meter_oap_graphql_query_latency_percentile).\nGraphQL Query Count — GraphQL queries per minute, split into total queries and errors (meter_oap_instance_graphql_query_count, meter_oap_instance_graphql_query_error_count).\nIngestion and analysis pipelines The receivers and analyzers turning raw telemetry into metrics.\nTrace Analysis / min — traces analyzed per minute, total vs errors (meter_oap_instance_trace_count, meter_oap_instance_trace_analysis_error_count).\nTrace Analysis Latency (ms) — p50 / p75 / p90 / p95 / p99 latency of trace analysis (meter_oap_instance_trace_latency_percentile).\nMesh Analysis / min — service-mesh telemetry analyzed per minute, total vs errors (meter_oap_instance_mesh_count, meter_oap_instance_mesh_analysis_error_count).\nMesh Analysis Latency (ms) — p50 / p75 / p90 / p95 / p99 latency of mesh analysis (meter_oap_instance_mesh_latency_percentile).\nOTEL Received / s — OpenTelemetry records received per second, broken out as metrics, logs, and spans (meter_oap_otel_metrics_received, meter_oap_otel_logs_received, meter_oap_otel_spans_received).\nK8S ALS — Kubernetes Access Log Service throughput: count, dropped, streams, and err streams (meter_oap_instance_k8s_als_count, meter_oap_instance_k8s_als_drop, meter_oap_instance_k8s_als_streams, meter_oap_instance_k8s_als_error_streams).\nWatermark Circuit Breaker — cumulative break and recover counters per listener; when OAP sheds load under memory pressure, breaks climb (meter_oap_instance_watermark_circuit_breaker_break_count, meter_oap_instance_watermark_circuit_breaker_recover_count).\nZipkin Spans Dropped — Zipkin spans dropped by this node, for deployments running the Zipkin receiver (meter_oap_instance_spans_dropped_count).\nStorage backend Write latency against whichever storage backend this OAP is configured with. These two widgets are storage-specific and only render when the matching backend is in use — a BanyanDB deployment shows the BanyanDB widget, an Elasticsearch deployment shows the Elasticsearch widget.\nBanyanDB Write Latency (ms) — write latency by catalog and operation: measure bulk, stream bulk, trace bulk, stream single, and property (meter_oap_banyandb_write_latency_percentile). Shown only when BanyanDB write metrics are present.\nElasticsearch Write Latency (ms) — write latency split into single (single write / update / delete) and bulk (meter_oap_elasticsearch_write_latency_percentile). Shown only when Elasticsearch write metrics are present.\nRequirements The SO11Y_OAP dashboard is a pure consumer of what OAP reports about itself — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs its self-observability telemetry enabled, which produces the meter_oap_* meter family:\nJVM and process metrics — meter_oap_instance_cpu_percentage, meter_oap_instance_jvm_*, and the meter_oap_jvm_thread_* / meter_oap_jvm_class_* families behind the JVM-health widgets.\nPipeline and persistence metrics — meter_oap_instance_metrics_aggregation, meter_oap_instance_persistence_*, meter_oap_instance_metrics_persistent_cache, and meter_oap_instance_metrics_aggregation_queue_used_per_ten_thousand for the aggregation / persistence widgets.\nQuery metrics — meter_oap_graphql_query_latency_percentile and meter_oap_instance_graphql_query_count / _error_count for the GraphQL surface.\nIngestion metrics — the trace, mesh, OTEL, K8s ALS, watermark, and Zipkin families (meter_oap_instance_trace_*, meter_oap_instance_mesh_*, meter_oap_otel_*, meter_oap_instance_k8s_als_*, meter_oap_instance_watermark_circuit_breaker_*, meter_oap_instance_spans_dropped_count). A pipeline that isn\u0026rsquo;t running on a given node simply reports nothing, and its widget reads no data.\nStorage metrics — meter_oap_banyandb_write_latency_percentile or meter_oap_elasticsearch_write_latency_percentile, depending on the configured storage backend; only the matching widget renders.\nEach metric is queried at the OAP-node (instance) scope; OAP does not roll a metric up across scopes, so the dashboard is empty until self-observability telemetry is reported by the OAP nodes themselves. See the OAP backend setup docs for enabling the self-observability telemetry source.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/so11y_oap/","title":"\u003c!--"},{"body":" Satellite (Self-Observability) The SO11Y_SATELLITE layer is SkyWalking\u0026rsquo;s self-observability view of Apache SkyWalking Satellite — the lightweight telemetry collector that sits in front of OAP, buffering and forwarding agent traffic. When a Satellite instance reports its own runtime metrics to OAP (via the OpenTelemetry receiver), each collector shows up here as a service so you can watch the collection tier the same way you watch instrumented applications.\nIn Horizon\u0026rsquo;s sidebar this layer is named Satellite, and it is grouped under Self-Observability alongside the other components SkyWalking monitors about itself. Its services are listed as Satellite services. This layer is intentionally focused: it enables only the Service scope — there are no instance, endpoint, topology, traces, or logs sub-tabs. Everything Satellite exposes is read at the service level.\nThis page is the operator reference for the bundled Satellite dashboard: what you see on the service scope and what each widget means.\nThe widgets and metrics below are read from the bundled SO11Y_SATELLITE template; if an operator has published a customized Satellite template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list The layer landing page lists every Satellite service that has reported. This layer defines no custom landing-page metric columns, so services are listed by name only — pick one to open its dashboard.\nService dashboard The dashboard for one selected Satellite collector. Every widget is a time-series line over the selected window, covering the collector\u0026rsquo;s connection load, host CPU, internal queue, and the four stages of its event pipeline. The queue and event widgets break their series out per Satellite pipeline (tracingpipe, jvmpipe, logpipe, meterpipe, …), so you can see which collection pipeline is driving the rate; Connection Count and CPU are single series.\nConnection Count — the number of gRPC connections the collector currently holds, i.e. how many upstream agents and downstream OAP links are attached (satellite_service_grpc_connect_count).\nCPU (%) — host CPU utilization of the process running the Satellite gRPC server, as a percentage (satellite_service_server_cpu_utilization).\nQueue Used — how much of the internal buffering queue is currently occupied. Watch this against the collector\u0026rsquo;s queue capacity — a queue that stays near full means Satellite is backing up and is at risk of dropping events (satellite_service_queue_used_count).\nReceive Events — events received from upstream agents per minute, the inbound rate into the collector (satellite_service_receive_event_count).\nFetch Events — events fetched into the pipeline per minute, the rate at which buffered data is pulled forward for processing (satellite_service_fetch_event_count).\nQueue Input / Output — two series on one chart that show whether the queue is keeping pace: input is events written into the queue per minute (satellite_service_queue_input_count) and output is events sent on to OAP per minute (satellite_service_send_event_count). When output tracks input the collector is draining as fast as it fills; a persistent gap is the same backlog signal as a full Queue Used.\nRequirements The Satellite dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs each Satellite instance to push its self-observability metrics to OAP\u0026rsquo;s OpenTelemetry receiver, where they are aggregated into the satellite_service_* family at Service scope:\nConnection and host metrics — satellite_service_grpc_connect_count (gRPC connections) and satellite_service_server_cpu_utilization (server-process CPU).\nQueue metrics — satellite_service_queue_used_count for current queue occupancy, plus satellite_service_queue_input_count for the inbound queue rate.\nEvent-pipeline metrics — satellite_service_receive_event_count, satellite_service_fetch_event_count, and satellite_service_send_event_count for the receive → fetch → send stages of the collection pipeline.\nEach metric is queried at its own OAP scope; this layer reports only at Service scope, so the dashboard stays empty until a Satellite instance is configured to export its runtime metrics and they reach OAP. For how to wire that export and the underlying metric rules, see the SkyWalking Satellite self-observability setup documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/so11y_satellite/","title":"\u003c!--"},{"body":" Virtual Cache The VIRTUAL_CACHE layer monitors the cache systems your services talk to — Redis, Memcached, and the like — as virtual targets. There is no agent inside the cache itself; the data is synthesized from the cache calls that instrumented services make, so each cache appears as a service whose traffic, latency, and success rate are reconstructed from the client side.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Virtual targets and named Virtual Cache. Its services are listed as Caches. The layer is single-scope: it ships only the Service (cache) dashboard — there are no instance, endpoint, topology, trace, or log tabs for virtual caches, so this page documents the Cache list and the Cache dashboard only.\nThis page is the operator reference for the bundled VIRTUAL_CACHE dashboard: what you see on the cache landing list and what each widget on the Cache dashboard means.\nThe widgets and metrics below are read from the bundled VIRTUAL_CACHE template; if an operator has published a customized VIRTUAL_CACHE template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nCache list Before opening a cache, the layer landing page lists every virtual cache with four sortable columns, sorted by access traffic (Access RPM) by default:\nAccess RPM — total cache accesses per minute (cache_access_cpm).\nLatency — average access latency in ms (cache_access_resp_time).\np95 — 95th-percentile access latency in ms (cache_access_percentile{p='95'}).\nError Rate — percent of failed accesses (100 - cache_access_sla/100).\nCache dashboard The drill-down for one selected cache. The dashboard splits into three views of the same traffic: the combined access (all operations), then read and write broken out separately, and finally the slowest captured commands.\nAccess (all operations)\nAccess Traffic — total cache accesses per minute (cache_access_cpm).\nAvg Access Latency — mean access latency in ms (cache_access_resp_time).\nAccess Success Rate — percent of successful accesses (cache_access_sla/100).\nAccess Latency Percentile — p50 / p75 / p90 / p95 / p99 access latency, the tail of the access-latency distribution (cache_access_percentile).\nRead\nRead Traffic — cache read operations per minute (cache_read_cpm).\nRead Avg Latency — mean read latency in ms (cache_read_resp_time).\nRead Success Rate — percent of successful reads (cache_read_sla/100).\nRead Latency Percentile — p50 / p75 / p90 / p95 / p99 read latency (cache_read_percentile).\nWrite\nWrite Traffic — cache write operations per minute (cache_write_cpm).\nWrite Avg Latency — mean write latency in ms (cache_write_resp_time).\nWrite Success Rate — percent of successful writes (cache_write_sla/100).\nWrite Latency Percentile — p50 / p75 / p90 / p95 / p99 write latency (cache_write_percentile).\nSlow commands\nSlow Read Commands — the 10 slowest captured read commands against this cache (top_n(top_n_cache_read_command, 10, des), ms). Each row is a single execution — click it to copy the command, or use the trace icon at the row head to open its originating trace (shown only when the sample carries one). Shows no data when OAP captured no slow read commands in the window.\nSlow Write Commands — the 10 slowest captured write commands against this cache (top_n(top_n_cache_write_command, 10, des), ms). Same row behavior as Slow Read Commands — click to copy, or open the originating trace when the sample has one. Shows no data when none were captured.\nRequirements The VIRTUAL_CACHE dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nCache-access metrics — the cache_access_* family (traffic, response time, SLA, percentiles), produced by OAP from the cache calls that instrumented services make.\nRead / write metrics — the cache_read_* and cache_write_* families, the same measures split by operation, for the Read and Write widgets.\nSampled records — top_n_cache_read_command and top_n_cache_write_command for the Slow Read / Write Commands lists, captured by OAP when slow-command sampling is enabled.\nEach metric is queried at the cache\u0026rsquo;s Service scope; OAP does not roll a metric up across scopes, so a widget stays empty until that measure is reported for the cache. Virtual-cache data only appears when the services calling the cache are instrumented and OAP\u0026rsquo;s virtual-cache analysis is enabled — see the Virtual Cache setup guide for how OAP derives these targets.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/virtual_cache/","title":"\u003c!--"},{"body":" Virtual Database The VIRTUAL_DATABASE layer is the conjugate view of database traffic: instead of monitoring the database server itself, it shows each database as a peer that your instrumented services talk to. SkyWalking\u0026rsquo;s language agents detect outbound database calls in their traces and synthesize a virtual database node from the connection\u0026rsquo;s peer address — so a database appears here whether or not it is independently monitored, reconstructed entirely from the caller\u0026rsquo;s perspective.\nIn Horizon\u0026rsquo;s sidebar this layer lives under the Virtual targets group and is named Virtual Database. Each synthesized database is listed as a Database. This is a virtual-target layer with a single scope: it enables only the Service scope — there is no instance or endpoint scope, no topology, and no traces or logs tabs. Everything you see is derived from the access traffic the calling agents reported, so the figures describe the database as seen by its clients, not by the database engine.\nThis page is the operator reference for the bundled Virtual Database dashboard: what you see on the scope and what each widget means.\nThe widgets and metrics below are read from the bundled VIRTUAL_DATABASE template; if an operator has published a customized VIRTUAL_DATABASE template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a database, the layer landing page lists every virtual database with four sortable columns, sorted by Access RPM by default:\nAccess RPM — accesses per minute against the database (database_access_cpm).\nLatency — average access latency in ms (database_access_resp_time).\np95 — 95th-percentile access latency in ms (database_access_percentile{p='95'}).\nError Rate — percent of accesses that threw (100 - database_access_sla/100).\nService dashboard The primary drill-down for one selected database.\nAccess Traffic — accesses per minute against the virtual database (database_access_cpm).\nAvg Response Time — mean access latency in ms (database_access_resp_time).\nSuccess Rate — percent of accesses that returned without throwing (database_access_sla/100).\nResponse Time Percentile — p50 / p75 / p90 / p95 / p99 access latency, the tail of the access-time distribution (database_access_percentile).\nSlow Statements — the top 20 slowest captured statements against this database, in ms, worst-first (top_n(top_n_database_statement, 20, des)). Each row is a single statement execution; click a row to copy the statement text, or use the trace icon at the row head to open its originating trace — shown only when the sample carries one. Reads no data when OAP captured no statements in the window.\nRequirements The Virtual Database dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs your services instrumented by SkyWalking language agents that capture database calls, which produces:\nDatabase access metrics — the database_access_* family: database_access_cpm (traffic), database_access_resp_time (latency), database_access_sla (success rate), and database_access_percentile (the latency tail). These back both the Service list and the Service dashboard.\nSampled statements — top_n_database_statement for the Slow Statements list, captured by OAP when slow-statement sampling is configured on the calling services.\nEach metric is queried at its own OAP scope; the whole layer lives at the service (database) scope, so the dashboard is empty until at least one instrumented service reports database access traffic. For the upstream setup — how virtual databases are detected and configured — see the virtual database documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/virtual_database/","title":"\u003c!--"},{"body":" Virtual GenAI The VIRTUAL_GENAI layer monitors the GenAI / LLM providers your services talk to — OpenAI, Anthropic, and other model backends — as virtual targets. There is no agent inside the provider; the data is synthesized from the GenAI calls that instrumented services make, so each provider appears as a service whose request load, latency, success rate, token throughput, and estimated cost are reconstructed from the client side, then broken down per model.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Virtual targets and named Virtual GenAI. Its services are listed as GenAI Providers and its instances as Models. The VIRTUAL_GENAI layer enables the Service (GenAI Provider) and Instance (Model) dashboards only — it does not ship an Endpoint dashboard, a topology / service-map view, or Traces / Logs tabs, because the providers are monitored entirely through their GenAI meter families, which carry no per-endpoint scope or call graph.\nThis page is the operator reference for the bundled VIRTUAL_GENAI dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled VIRTUAL_GENAI template; if an operator has published a customized VIRTUAL_GENAI template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a provider, the layer landing page lists every GenAI Provider with four sortable columns, sorted by traffic (RPM) by default:\nRPM — calls per minute to the provider (gen_ai_provider_cpm).\nLatency — average response time in ms (gen_ai_provider_resp_time).\nSuccess Rate — percent of successful calls (gen_ai_provider_sla/100).\nOutput Tokens — total output (completion) tokens produced over the window (latest(gen_ai_provider_output_tokens_sum)).\nService dashboard The primary drill-down for one selected GenAI Provider. The dashboard covers the request golden signals, the latency tail, token throughput split into input and output, and an estimated spend.\nCalls / min — calls per minute to the provider (gen_ai_provider_cpm).\nAvg Response Time — mean response time in ms (gen_ai_provider_resp_time).\nSuccess Rate — percent of successful calls (gen_ai_provider_sla/100).\nLatency Percentile — p50 / p75 / p90 / p95 / p99 response time, the tail of the latency distribution, in ms (gen_ai_provider_latency_percentile).\nInput Tokens — input (prompt) token throughput, shown as the running total over the window and the per-call average (latest(gen_ai_provider_input_tokens_sum), gen_ai_provider_input_tokens_avg).\nOutput Tokens — output (completion) token throughput, shown as the running total over the window and the per-call average (latest(gen_ai_provider_output_tokens_sum), gen_ai_provider_output_tokens_avg).\nEstimated Cost — estimated spend against the provider, shown as the total over the window and the per-call average (latest(gen_ai_provider_total_estimated_cost)/1000000, gen_ai_provider_avg_estimated_cost/1000000). OAP carries the cost in micro-units, so each series is divided by 1000000 to land in whole currency units.\nInstance dashboard For one selected Model of the provider. The same golden signals as the Service view, scoped to a single model, plus a streaming time-to-first-token timing.\nCalls / min — calls per minute to this model (gen_ai_model_call_cpm).\nAvg Latency — mean latency for this model, in ms (gen_ai_model_latency_avg).\nSuccess Rate — percent of successful calls to this model (gen_ai_model_sla/100).\nLatency Percentile — p50 / p75 / p90 / p95 / p99 latency for this model, in ms (gen_ai_model_latency_percentile).\nTTFT (Time to First Token) — time to first token for streaming responses, shown as both the average and the percentile distribution, in ms (gen_ai_model_ttft_avg, gen_ai_model_ttft_percentile).\nInput Tokens — input (prompt) token throughput for this model, shown as the running total over the window and the per-call average (latest(gen_ai_model_input_tokens_sum), gen_ai_model_input_tokens_avg).\nOutput Tokens — output (completion) token throughput for this model, shown as the running total over the window and the per-call average (latest(gen_ai_model_output_tokens_sum), gen_ai_model_output_tokens_avg).\nEstimated Cost — estimated spend against this model, shown as the total over the window and the per-call average (latest(gen_ai_model_total_estimated_cost)/1000000, gen_ai_model_avg_estimated_cost/1000000). As on the Service view, the micro-unit cost is divided by 1000000 to land in whole currency units.\nRequirements The VIRTUAL_GENAI dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nProvider (service) metrics — the gen_ai_provider_* family at Service scope: call load (gen_ai_provider_cpm), response time (gen_ai_provider_resp_time), SLA (gen_ai_provider_sla), latency percentile (gen_ai_provider_latency_percentile), input / output token sums and averages (gen_ai_provider_input_tokens_*, gen_ai_provider_output_tokens_*), and the estimated-cost totals and averages (gen_ai_provider_total_estimated_cost, gen_ai_provider_avg_estimated_cost).\nModel (instance) metrics — the gen_ai_model_* family at ServiceInstance scope: call load (gen_ai_model_call_cpm), latency average and percentile (gen_ai_model_latency_avg, gen_ai_model_latency_percentile), SLA (gen_ai_model_sla), the streaming time-to-first-token timings (gen_ai_model_ttft_avg, gen_ai_model_ttft_percentile), input / output token sums and averages (gen_ai_model_input_tokens_*, gen_ai_model_output_tokens_*), and the estimated-cost totals and averages (gen_ai_model_total_estimated_cost, gen_ai_model_avg_estimated_cost).\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a model-scope (instance) metric is empty until that level of data is reported. Virtual-GenAI data only appears when the services calling the provider are instrumented and OAP\u0026rsquo;s virtual-GenAI analysis is enabled — see the Virtual GenAI setup guide for how OAP derives these targets.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/virtual_genai/","title":"\u003c!--"},{"body":" Virtual MQ The VIRTUAL_MQ layer monitors the message-queue systems your services publish to and consume from — Kafka, RocketMQ, RabbitMQ, Pulsar, and the like — as virtual targets. There is no agent inside the broker itself; the data is synthesized from the produce and consume calls that instrumented services make, so each message-queue cluster appears as a service whose throughput, latency, and success rate are reconstructed from the client side.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Virtual targets and named Virtual MQ. Its services are listed as MQ clusters, and its endpoints — the queues / topics a cluster carries — are listed as Topics. The layer enables two scopes: the Service (MQ cluster) dashboard and the Endpoint (Topic) dashboard. There are no instance, topology, trace, or log tabs for virtual MQ, so this page documents the cluster list, the MQ cluster dashboard, and the Topic dashboard only.\nThis page is the operator reference for the bundled VIRTUAL_MQ dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled VIRTUAL_MQ template; if an operator has published a customized VIRTUAL_MQ template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nMQ cluster list Before opening a cluster, the layer landing page lists every MQ cluster with four sortable columns, sorted by consume throughput (Consume RPM) by default:\nConsume RPM — messages consumed per minute across the cluster (mq_service_consume_cpm).\nProduce RPM — messages produced per minute across the cluster (mq_service_produce_cpm).\nConsume Latency — average time between message production and consumption, in ms (mq_service_consume_latency).\nConsume Error Rate — percent of failed consume operations (100 - mq_service_consume_sla/100).\nMQ cluster dashboard The drill-down for one selected MQ cluster. The dashboard pairs the produce and consume sides of the cluster\u0026rsquo;s traffic — throughput, success rate, and the consume-latency profile.\nConsume Traffic — messages consumed per minute (mq_service_consume_cpm).\nProduce Traffic — messages produced per minute (mq_service_produce_cpm).\nConsume Avg Latency — average time between message production and consumption, in ms (mq_service_consume_latency).\nConsume Success Rate — percent of successful consume operations (mq_service_consume_sla/100).\nProduce Success Rate — percent of successful produce operations (mq_service_produce_sla/100).\nConsume Latency Percentile — p50 / p75 / p90 / p95 / p99 of consume latency, the tail of the consume-latency distribution (mq_service_consume_percentile).\nTopic dashboard For one selected Topic — a queue / topic under the cluster, on the Endpoint scope. It mirrors the cluster widgets at the per-topic level.\nTopic Consume Traffic — messages consumed per minute on the topic (mq_endpoint_consume_cpm).\nTopic Produce Traffic — messages produced per minute on the topic (mq_endpoint_produce_cpm).\nTopic Consume Avg Latency — average consume latency for the topic, in ms (mq_endpoint_consume_latency).\nTopic Consume Success Rate — percent of successful consume operations on the topic (mq_endpoint_consume_sla/100).\nTopic Produce Success Rate — percent of successful produce operations on the topic (mq_endpoint_produce_sla/100).\nTopic Consume Latency Percentile — p50 / p75 / p90 / p95 / p99 of the topic\u0026rsquo;s consume latency (mq_endpoint_consume_percentile).\nRequirements The VIRTUAL_MQ dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nMQ cluster metrics — the mq_service_* family (consume / produce throughput, consume latency, consume / produce SLA, consume percentiles), produced by OAP from the produce and consume calls that instrumented services make.\nTopic metrics — the mq_endpoint_* family, the same measures evaluated at the Endpoint (Topic) scope, for the Topic dashboard.\nEach metric is queried at its own OAP scope — the mq_service_* family at the MQ cluster\u0026rsquo;s Service scope and the mq_endpoint_* family at the Topic\u0026rsquo;s Endpoint scope. OAP does not roll a metric up across scopes, so a Topic widget stays empty until that measure is reported at the Topic level, independent of the cluster-scope data. Virtual-MQ data only appears when the services producing to and consuming from the broker are instrumented and OAP\u0026rsquo;s virtual-MQ analysis is enabled — see the Virtual MQ setup guide for how OAP derives these targets.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/virtual_mq/","title":"\u003c!--"},{"body":" WeChat Mini Program The WECHAT_MINI_PROGRAM layer holds WeChat (微信) Mini Programs monitored by the SkyWalking mini-program agent. The agent runs inside the mini-program runtime and reports client-side performance — app launch, first render, package load, page routing, script execution, and outbound request timing — so each mini-program lands here rather than in a server-side layer.\nIn Horizon\u0026rsquo;s sidebar this layer is named WeChat Mini Program (under the Mobile group). Its services are listed as Mini-programs, instances as Versions (one per released mini-program version), and endpoints as Pages (one per mini-program page). The layer enables the Service, Version, Page, Traces, and Logs sub-tabs. It has no service map, instance map, or page-dependency view — mini-program telemetry is client-side timing, with no inter-service call topology to draw.\nThis page is the operator reference for the bundled WECHAT_MINI_PROGRAM dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled WECHAT_MINI_PROGRAM template; if an operator has published a customized template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nMini-program list Before opening a mini-program, the layer landing page lists every WECHAT_MINI_PROGRAM service with four sortable columns, sorted by request traffic (Request RPM) by default:\nRequest RPM — outbound requests per minute (meter_wechat_mp_request_cpm).\nLaunch — average app-launch duration in ms (meter_wechat_mp_app_launch_duration).\nFirst Render — average first-render duration in ms (meter_wechat_mp_first_render_duration).\nErrors — count of reported errors (meter_wechat_mp_error_count).\nService dashboard The primary drill-down for one selected mini-program.\nApp Launch Duration — time to launch the mini-program, in ms (meter_wechat_mp_app_launch_duration).\nFirst Render Duration — time to the first render, in ms (meter_wechat_mp_first_render_duration).\nPackage Load Duration — time to download and parse the mini-program package bundle, in ms (meter_wechat_mp_package_load_duration).\nError Count — number of errors reported by the mini-program (meter_wechat_mp_error_count).\nRoute Duration — time spent in page-route transitions, in ms (meter_wechat_mp_route_duration).\nScript Duration — script-execution time, in ms (meter_wechat_mp_script_duration).\nRequest Load — outbound requests per minute (meter_wechat_mp_request_cpm).\nRequest Duration Percentile — p50 / p75 / p90 / p95 / p99 of outbound request duration, in ms — the tail of the request-timing distribution (meter_wechat_mp_request_duration_percentile).\nVersion dashboard For one selected released Version of the mini-program. The same timing families as the service dashboard, evaluated at version (instance) scope so you can compare one release against another.\nLaunch Duration — app-launch duration for this version, in ms (meter_wechat_mp_instance_app_launch_duration).\nFirst Render Duration — first-render duration for this version, in ms (meter_wechat_mp_instance_first_render_duration).\nPackage Load Duration — package download-and-parse time for this version, in ms (meter_wechat_mp_instance_package_load_duration).\nRequest Load — outbound requests per minute for this version (meter_wechat_mp_instance_request_cpm).\nRoute Duration — page-route transition time for this version, in ms (meter_wechat_mp_instance_route_duration).\nScript Duration — script-execution time for this version, in ms (meter_wechat_mp_instance_script_duration).\nRequest Duration Percentile — p50 / p75 / p90 / p95 / p99 of outbound request duration for this version, in ms (meter_wechat_mp_instance_request_duration_percentile).\nPage dashboard For one selected Page (endpoint) of the mini-program.\nLaunch Duration on Page — app-launch duration attributed to this page, in ms (meter_wechat_mp_endpoint_app_launch_duration).\nFirst Render Duration — first-render duration for this page, in ms (meter_wechat_mp_endpoint_first_render_duration).\nRequest Load — outbound requests per minute originating from this page (meter_wechat_mp_endpoint_request_cpm).\nRequest Duration Percentile — p50 / p75 / p90 / p95 / p99 of outbound request duration for this page, in ms (meter_wechat_mp_endpoint_request_duration_percentile).\nRequirements The WECHAT_MINI_PROGRAM dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nMini-program (service) metrics — the meter_wechat_mp_* family at service scope: app launch, first render, package load, route, script, error count, request load, and request-duration percentile.\nVersion (instance) metrics — the meter_wechat_mp_instance_* family, the same timings reported per released version.\nPage (endpoint) metrics — the meter_wechat_mp_endpoint_* family, the launch / first-render / request-load / request-percentile timings reported per page.\nThese metrics come from the WeChat Mini Program agent reporting client-side timing to OAP, where the mini-program meter rules aggregate them. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the Version and Page dashboards stay empty until that level of data is reported. See the WeChat Mini Program monitoring setup for enabling the receiver and meter rules on OAP.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/dashboards/wechat_mini_program/","title":"\u003c!--"},{"body":" AI Agent Conversations The Conversations tab of the AI Agents layer lists the conversations of long-lived AI agents that the SkyWalking AI Sessionizer pushed to OAP. A conversation is the Sessionizer\u0026rsquo;s unit of storage: one durable exchange between a person and an agent, however many sessions, context resets and child agents it spanned. This page is about the list; each row stands for one conversation as OAP holds it.\nIt is not the AI assistant. The assistant is Horizon\u0026rsquo;s own chat, where Horizon sends text to a model provider on your behalf. This tab reads stored transcripts of other agents and sends nothing anywhere.\nRequirements OAP 11.1.0 or later. Older OAPs have no AI_AGENT layer, so the layer never appears in the sidebar. A running AI Sessionizer configured to push to this OAP. Until it has pushed at least one conversation, the layer has no service and stays hidden. The ai-conversation:read permission, which the built-in viewer, maintainer and operator roles carry. See Roles and Permissions. Reading the list The agent runtime is the layer\u0026rsquo;s service — Claude Code, or the name the Sessionizer was configured with — and it is picked where every layer\u0026rsquo;s service is picked, in the header above the tabs. Pick a time range, optionally one sender (the Sessionizer processes that pushed to this runtime), then click Run query. The sender is part of the query: OAP lists only that sender\u0026rsquo;s conversations. The tab owns its own time range, like the Traces and Logs tabs: the global time picker and auto-refresh do not drive it. A conversation is in the window when its last activity is; the presets run from a day to 90 days because a conversation lives for days, not minutes.\nTwo more conditions travel with the query: Title contains keeps the conversations whose title contains the text, matched by OAP on each conversation\u0026rsquo;s newest title, and Conversation id asks OAP for that one conversation. Enter in either box runs the query.\nColumn Meaning Title The session\u0026rsquo;s title as the agent runtime recorded it. A conversation with no title shows (untitled); the conversation id sits under the title in either case. Sender Which Sessionizer pushed it — one process on one machine, user@host unless its operator named it. Talks Readable exchanges: one input from outside, the agent\u0026rsquo;s run, and its answer. Steps Everything the agent did: model calls, reasoning, tool uses, messages, agent launches, resets. Streams Execution streams: the main agent plus one per child agent it started. Segments Activity windows, split by idle time. Unresolved References the Sessionizer could not resolve — a tool result it never saw, a child whose stream never arrived. Worth a look when it is not zero. Span From the conversation\u0026rsquo;s first record to its last activity. Last activity When the newest record was written, in your browser\u0026rsquo;s time. Rows are ordered by it, newest first. The counts come from the conversation\u0026rsquo;s newest round as the Sessionizer wrote it; they are what the Sessionizer itself lists, not something Horizon computed.\nWhat the list can and cannot show The list is built from rounds, not conversations. The Sessionizer publishes a conversation as a chain of immutable rounds, a new round every few minutes while the agent is active, and OAP builds the list from the newest rounds in the window — up to a fixed budget (10,000 by default), folded to one row per conversation. The line above the table states that budget. A conversation whose newest round is older than every one of those rounds is not listed even though it is stored; narrowing the time range brings it back, and so does asking for a runtime with fewer conversations. OAP has no way to say whether the budget cut anything, so Horizon states the rule rather than guessing.\nRetention is OAP\u0026rsquo;s. On BanyanDB the conversation files live in their own group with their own lifetime (30 days by default); on other storages they follow the record retention. A conversation older than that is gone from the list because it is gone from OAP.\nOne conversation, one row, per sender. The same conversation pushed by two Sessionizers, or by one that was renamed between pushes, appears once per sender name.\nReading a conversation Click a row, or press Enter on it, and the conversation opens in a new browser tab of its own. The page reads the whole conversation from OAP in one document and shows the wait as it happens: first OAP is assembling the conversation with the seconds counting, because OAP folds the whole chain before it sends a byte; then the bytes coming in as n of m MB received with a percentage, the rate and the time left, and the talk, step and stream counts; then the parse and the draw. A conversation of a thousand talks and fifty thousand steps is tens of megabytes and draws in well under a second once the document has arrived.\nThe page\u0026rsquo;s address is the thing to share. It carries the conversation, its agent runtime and sender, and the reader\u0026rsquo;s position — the talk, the selected step and the stream being read — and it is updated in place as you move, so copying the address bar hands a colleague the same step. Opening it needs a Horizon sign-in and the ai-conversation:read permission; a signed-out reader is sent to the login page and back.\nThe page has three parts, and a header with the conversation\u0026rsquo;s title, runtime, sender and id, a link back to the list, the theme chip and the language picker (the page follows your Horizon theme, the light one included, and every text of it is translated):\nTranscript — the talks of one execution stream in reading order. What came from outside sits on the right; the agent\u0026rsquo;s replies sit on the left; the work between an input and its reply (model calls, reasoning, tool uses, agent launches, context resets) is folded under a show what the agent did row so a long run reads as a conversation until you open it. Long pauses are marked. Steps the Sessionizer could not place under any talk are listed in their own Outside any talk section rather than dropped. Flow timeline — the same stream on a time axis, one lane per kind of activity (external input, responses, context put in, model calls, tools, agent activity, runtime notices, nested streams). Busy stretches take the width; a long pause is cut to a marked gap. Selecting a step draws the relations that touch it: a solid line is an exact join, a dashed one was inferred, a faint curve is ownership (the model call that produced a step). A child agent appears as a nested stream you can select, then dive into; the header offers the way back to the step that opened it. j and k move through the steps, Enter dives in, Escape clears the selection. Inspector — the selected step\u0026rsquo;s Details (kind, lane, stream, segment, run, parent, time, token counts, request-to-result interval where the runtime recorded it), its Relations (what it opened, what opened it, what it joined with and how well), and its Evidence: the landed positions of the record the step came from and the text as the document carries it, with a note when the document clipped it. The status strip at the top states the document\u0026rsquo;s integrity — verified when every round\u0026rsquo;s digest chained, incomplete when rounds are missing, mismatch when a digest did not match — and lists the problems the Sessionizer recorded when it is not verified. The Overview button opens the summary cells and a filterable list of every talk in the conversation.\nWhat the page shows is exactly what the Sessionizer\u0026rsquo;s own viewer shows for the same conversation: the two draw the same document with the same renderer. The words the runtime uses for things — node kinds such as message.external, relation types, join qualities — are shown as the document carries them, in every locale.\nLimits One document, all at once. OAP sends the whole conversation, and its viewTimeoutMs budget (120 s by default, see Configuration File) covers the wait for the first byte. A conversation OAP cannot assemble within that time reports a timeout; try again, or raise the budget on both sides. Horizon holds the document before the browser starts receiving it, which is how the page knows the size to count against; that hold is a fifth of the document\u0026rsquo;s size per conversation being opened. Records are not on OAP. The Sessionizer\u0026rsquo;s own viewer can open the raw landed record behind a step; OAP stores the assembled document only, so this page shows the text the document carries and says when it was clipped. Troubleshooting The layer is missing from the sidebar. OAP reports no AI_AGENT layer: either it is older than 11.1.0, or nothing has been pushed yet. Check the Sessionizer\u0026rsquo;s push output for its receiver address and errors. The runtime is listed but the query returns nothing. The window may be too narrow for a conversation\u0026rsquo;s last activity, or every round in the window may belong to other conversations (see above). Try 90 days, and check the sender filter is not set. A conversation you know exists is not there. Its newest round may lie outside the round budget, or the conversation may be older than OAP\u0026rsquo;s retention. The Sessionizer\u0026rsquo;s own list page shows what it holds locally; compare the two. The conversation page says OAP holds no round for this runtime. The link names a runtime OAP has no round of this conversation for — the Sessionizer was renamed between pushes, or the sender in the link is not the one that pushed it. Open the conversation from the list again. The page waits a long time, then reports a timeout. OAP assembles the whole chain before answering, and a very long conversation can exceed the time budget. Try again; if it keeps happening, raise performance.aiConversation.viewTimeoutMs in Horizon and the matching viewRequestTimeout on OAP. A row does nothing when clicked. The browser blocked the new tab as a pop-up. Allow pop-ups for Horizon\u0026rsquo;s address, or open the row with Enter after focusing it. ","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/operate/ai-agent-conversations/","title":"\u003c!--"},{"body":" Alarms Path: /alarms. The page is read-only and needs no special permission to view.\nThe Alarms page is the triage surface for everything OAP\u0026rsquo;s alerting engine is firing right now, across every layer. It pulls the alarms OAP recorded over a recent window, groups the repeat firings of a rule on the same entity into a single incident, lays them out on a per-layer timeline, and shows the trigger expression and the captured metric snapshot for whichever alarm you select.\nAlarms are read-only here by design. OAP recovers an alarm automatically once the condition clears — there is no acknowledge, close, or silence action in the UI, and there is nothing to dismiss. A firing alarm stops firing when the underlying metric stops crossing the threshold; the page reflects that state, it does not drive it.\nThe time window The window picker offers three presets — 20m, 2h, 4h — plus a custom range capped at 4 hours.\nAlarms are second-precision events, and a long window pulls thousands of rows that some storage backends struggle to return; the 4-hour ceiling is enforced both in the picker and on the server, so a custom range wider than 4 hours is rejected. When the window genuinely holds more alarms than were fetched, the timeline header says so — narrow the window to see a complete slice. A window that exactly fills the fetch is complete and carries no notice.\nThe window\u0026rsquo;s starting preset can be set per deployment — see Alert page setup below.\nActive count and per-layer breakdown The KPI strip at the top counts what is actively firing, not the raw event count.\nActive — the total number of incidents that are currently firing. A fully recovered incident contributes nothing here, so this number answers \u0026ldquo;what is on fire right now?\u0026rdquo; rather than \u0026ldquo;what happened recently?\u0026rdquo;. Per-layer tiles — one tile per pinned layer (for example General, Mesh), each showing that layer\u0026rsquo;s active count. Pinned layers always render, even at zero, so the strip is stable across refreshes. Other — a read-only aggregate of active alarms in layers you did not pin, plus any alarm OAP could not attribute to a known layer. The arithmetic Active = (sum of pinned tiles) + Other always holds, so nothing hides off-screen. Overflow chips — below the tiles, the non-pinned layers that actually have an active alarm appear as small pills, sorted by count, as a filter shortcut. Clicking a tile, a chip, or a list tab narrows the timeline and the list to that layer; the selection is reflected in the URL, so a refresh or a shared link preserves it. Click the active tile again (or the Active tile) to clear the filter.\nFiltering Above the timeline is a filter row. What it offers depends on the connected OAP version:\nOn a current OAP, you get a cascading Layer → Service → Instance → Endpoint picker plus a free-text Keyword match on the alarm message. These filters are applied at the source, so the page only fetches the alarms that match. On an older OAP that does not support entity-scoped alarm queries, the row collapses to Keyword only, with a note inviting an upgrade for the full layer and entity filters. The filter is a draft until you press apply — nothing refires while you are composing it. clear resets every field.\nTimeline The timeline plots each alarm as a flag on a per-layer lane, so you can see at a glance when a burst happened and which layers it touched. It keeps every individual firing and recovery — not the merged incident — so a fire-then-recover pattern stays visible.\nTwo interactions:\nClick a flag to select that alarm and load its detail on the right. Brush a region to slice the list (and the counts) to that sub-window. The brushed rectangle is the only marker for the selection; the timeline itself still shows the full window so you can see other peaks to re-brush onto. reset clears the brushed range and the selected alarm.\nIncidents and the list OAP emits one alarm record per firing, so a rule that re-fires after its silence period produces several records. The list collapses the repeat firings of one rule on one entity into a single incident row, tagged with how many times it triggered. Each row carries a state:\nfiring — currently firing, and it never recovered within the window. unstable — currently firing, but it recovered at least once earlier in the window and fired again (a flapping rule). The badge shows how many of its firings are currently active versus recovered. Unstable still counts as active. recovered — the latest firing has cleared. Recovered incidents stay in the list as recent history but drop out of the Active count and the per-layer tiles — recovered is \u0026ldquo;no alarm\u0026rdquo;. For an incident that triggered more than once, the chevron at the end of the row expands a per-firing history: every individual firing and recovery on that entity and rule, in time order. Clicking a sub-entry loads that specific event into the detail panel. The list pages ten incidents at a time.\nAlarm detail Selecting an alarm — from a timeline flag, a list row, or an expanded history entry — opens the detail panel on the right:\nStatus — a firing or recovered pill, plus when the alarm started and (if cleared) when it recovered, and its layer. Message — the human-readable alarm text OAP formatted from the rule. Tags — any tags OAP attached to the alarm. Trigger expression — the MQE expression the rule evaluated, exactly as it fired. Rule — when the OAP admin port is reachable, the matched rule\u0026rsquo;s body: period, silence, recovery-obs, notification hooks, and the metrics it references. A \u0026ldquo;view in catalog\u0026rdquo; link jumps to the same rule on the Alerting rules page. When the admin port is unreachable, this section is omitted. Snapshot — one small chart per metric, plotting the values OAP captured at the firing moment so you can see what actually crossed the threshold. The trigger minute is marked, and the rule\u0026rsquo;s evaluation window is shaded when the rule body is available. An alarm recorded without an MQE snapshot (older OAP, or snapshot capture disabled in the rule) shows a note instead of charts. Admin: setup, pinned layers, and default window Which layers get their own KPI tile, and which window preset the page opens on, are configured on the Alert page setup admin page (/admin/alert-page-setup, verb alarm-setup:read), reachable from the page\u0026rsquo;s intro text.\nAlerting rules: the running context Path: /operate/alerting-rules. Verb: alarm-rule:read.\nThe Alerting rules page is a read-only catalog of every alarm rule loaded into the OAP cluster. Rules themselves are authored in OAP\u0026rsquo;s alarm-settings.yml and reloaded by OAP\u0026rsquo;s watcher — there is no add, edit, or delete here.\nEach rule lists its expression, window settings (period, silence, recovery-obs, and any additional period), the metrics it references, hooks, tags, entity include/exclude filters, and a per-node load state (loaded a/b) — because in a cluster each OAP instance loads the rule independently, and a partial count flags a node that has not picked it up.\nPer-entity running state Each OAP instance only evaluates a rule over the slice of entities it holds, so a rule\u0026rsquo;s Currently watching list is the union of evaluated entities across all nodes, with each entity tagged by the node watching it. Click an entity to open its live running context. Because the entity may be evaluated on only one node, the popup answers per node: the node actually evaluating it returns a populated body; the others read as \u0026ldquo;Not evaluated on this instance.\u0026rdquo;\nFor the evaluating node, the popup shows the rule\u0026rsquo;s current evaluation window (its size, the silence countdown, the recovery-observation countdown, and the window\u0026rsquo;s end time), the last alarm time and message, and a snapshot sparkline of the metric values in the window — each point annotated with its value and bucket time.\nThe headline of each node block is the rule\u0026rsquo;s current state for that entity. The states an operator will see:\nState Meaning FIRING The rule\u0026rsquo;s condition is currently met for this entity and the alarm is active. This is what surfaces as a firing alarm on the Alarms page. SILENCED_FIRING The condition is still met, but the alarm is inside its silence period after a recent firing, so OAP is holding off re-notifying. It is firing but quiet — no fresh notification goes out until the silence window elapses. OBSERVING_RECOVERY The condition has stopped being met and OAP is watching to confirm the recovery holds for the rule\u0026rsquo;s recovery-observation period before fully clearing the alarm. A flap back into breach during this window keeps the alarm active. These states are the live evaluation context behind the alarms you see on the Alarms page — they let you confirm that a rule is watching the entity you expect, see exactly where it is in the fire / silence / recover cycle, and read the very metric values it is acting on. The running context comes straight off OAP\u0026rsquo;s admin port; when that port is unreachable, the catalog surfaces a banner and the per-entity context is unavailable.\nRelated Runtime Rules (DSL) — runtime-editable MAL / LAL analysis rules that produce the metrics alarm rules evaluate. Metrics Inspect — browse the metric catalog and find which entities report a given metric. ","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/operate/alarms/","title":"\u003c!--"},{"body":" Events Events are the lifecycle records OAP has collected for a service — agent restarts, Kubernetes events, and other point-in-time facts reported by SkyWalking agents, the SkyWalking CLI, and the Kubernetes Event Exporter. Each event has a name, a type (Normal / Error), a message, and any reporter-supplied parameters. Events are distinct from alarms: an event records that something happened, not that a threshold was breached — for alerting, see Alarms.\nOpening the events popout Events are scoped to a single service and shown in a popout, so you review them without leaving the layer you\u0026rsquo;re on. On any layer drill-down, pick a service in the service banner at the top, then click the Events button next to the banner\u0026rsquo;s Share control. A modal opens for that service. The button appears only for users with the events:read permission (the built-in viewer, maintainer, and operator roles all have it).\nThe swimlane — instance × time The service is fixed (it\u0026rsquo;s in the popout title), so the view has two axes: each service instance is a row, and time runs left to right.\nAn event with a duration is a bar spanning its start to its end. An event with no end time is an instant marker (a small diamond). Each instance row is a distinct color, so the rows read apart at a glance. Error events carry a red ring so they stand out. If one instance reports overlapping events, they stack into sub-rows so nothing is hidden. A service that reports events without an instance shows a single row for the service. A rolling restart of a large service therefore shows as many bars at the same moment — one per instance — rather than a single summarised line. When a service runs many instances, use the search box at the top of the popout to filter the rows to the instances whose name matches.\nTime window and scrolling The popout owns its own window — 6h, 1d, 2d presets, plus a custom range — queried at second precision so the most recent events are never rounded out. The custom range takes an absolute start and end (entered in your browser\u0026rsquo;s local time) spanning up to 7 days; an invalid range — end before start, or a span past the 7-day cap — is rejected with the reason before anything is queried. A preset window is anchored to the moment you pick it, while a custom range is pinned exactly where you set it. Events are stored under OAP\u0026rsquo;s record retention; a window reaching past it simply returns fewer rows.\nScrolling stays inside the popout: the time-axis header stays pinned at the top and the instance column stays pinned at the left. A long range (a multi-day window) gets a wider, horizontally-scrollable canvas so bars keep a legible spacing instead of collapsing together, and the view opens scrolled to the newest events — scroll left for history. The time axis marks the date at day boundaries, so a range that crosses midnight is unambiguous.\nHow many events are shown The popout fetches the newest events up to a cap (200 by default; configurable under the server\u0026rsquo;s page-size limits). It tells you which case you\u0026rsquo;re in:\n\u0026ldquo;N events · all in range shown\u0026rdquo; — everything in the window is on screen. \u0026ldquo;Showing newest N — more available, narrow the range\u0026rdquo; — the window holds more than the cap; tighten the time range to reach older events. Event detail Click a bar to open the detail panel:\nHeader — the event type (Normal / Error) and name. Scope — the service, the instance (or \u0026ldquo;service-scoped\u0026rdquo;), the endpoint if present, and the layer. Started / Ended / Duration — for an event with a duration; a single Time for an instantaneous event. Message — the human-readable text the reporter attached. Parameters — the key/value details carried with the event (for example a Java agent\u0026rsquo;s startup options). Service names, instance names, messages, and parameter values are shown exactly as OAP reported them.\nRelated Alarms — threshold breaches from OAP\u0026rsquo;s alerting engine, a separate read-only triage surface. Traces and Logs — the other per-entity triage surfaces. ","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/operate/events/","title":"\u003c!--"},{"body":" 3D Infrastructure Map A single WebGL view of your whole deployment, stacked in 3D. Every SkyWalking layer\u0026rsquo;s services become cubes, grouped onto horizontal tiers, with live traffic, alarms, and call relationships drawn between them. It is the \u0026ldquo;stand back and look at everything at once\u0026rdquo; companion to the per-layer dashboards.\nOpen it from the 3D Infra pill in the topbar, or go directly to /3d/map. The map runs as a standalone full-screen view — no sidebar, no topbar, no global time picker — so the scene gets the whole viewport. The SkyWalking mark sits at the bottom-left; the × at the top-right returns you to the rest of Horizon.\nTiers A tier is a horizontal plane in the stack that groups related SkyWalking layers by their role in the system. Tiers are the spine of the map: they read top-to-bottom the way a request flows, from the apps a user touches down to the platform everything runs on.\nHorizon ships four bundled tiers:\nTier What lives here Examples Apps (top) The application surfaces and their direct dependencies as the app sees them General (agent) services, Browser/RUM, iOS, mini-programs, and the Virtual* targets (database / cache / MQ / gateway / GenAI) Middleware The data and messaging services, gateways, and self-observability MySQL, PostgreSQL, Redis, MongoDB, Elasticsearch, Kafka, RocketMQ, RabbitMQ, Pulsar, APISIX, Nginx, Kong, Flink, the SkyWalking SO11Y components, and cloud-managed data services Service Mesh The mesh that fronts the apps Istio managed services, Istio data plane (Envoy sidecars), Istio control plane, Cilium, Envoy AI Gateway Infra (bottom) The platform the rest runs on Kubernetes cluster + service, Linux/Windows hosts, virtual machines, EKS Every layer OAP reports is placed onto exactly one tier. A layer that Horizon hasn\u0026rsquo;t classified yet (for example a brand-new OAP layer) lands on the Middleware tier with an \u0026ldquo;unclassified\u0026rdquo; mark so an operator notices it and can re-assign it.\nThe tier list on the right-hand panel mirrors this stack. Click a tier row to fly the camera to it; use the eye toggle to show or hide every layer in that tier at once. The row also shows how many of the tier\u0026rsquo;s services are currently visible.\nReading the map Cubes Each cube is one service. Cubes are grouped into their layer\u0026rsquo;s zone on the tier, and each zone is colored with the layer\u0026rsquo;s brand color and stamped with the project\u0026rsquo;s logo (Istio\u0026rsquo;s sail, the Kubernetes helm wheel, a database cylinder, a queue, and so on) so you can identify a zone at a glance from any camera angle.\nLayers that ship a topology (General, Service Mesh, Kubernetes Service, Cilium) lay their cubes out by call dependency — upstream callers on one side, downstream services on the other — like the 2D service map. Layers without a topology pack their cubes into a tidy grid.\nTraffic A small pill under a cube shows that service\u0026rsquo;s live traffic — requests per minute for app and mesh services, queries or operations per second for data services, and so on, each with its own unit. The number is the service\u0026rsquo;s headline throughput metric for the current window.\nTraffic pills appear on cubes that are close enough to read; zoom out far enough and they fade away to keep the scene clean, then return as you zoom back in. A selected cube always shows its number.\nAlarms When a service has an alarm in the last 20 minutes, a small red beacon pulses on the top corner of its cube. The cube keeps its layer color — the beacon is the alert signal, so you can still tell which layer a troubled service belongs to. The alarm feed refreshes on its own while the map is open.\nConnections The map draws three kinds of lines:\nIn-layer calls — light cyan tubes between two services in the same layer, with animated packets flowing along them. This is each layer\u0026rsquo;s internal call graph. Cross-layer calls — soft orange arrows between services in different layers on the same tier (for example Browser → Frontend, or Frontend → Virtual Database). The arrow points from caller to callee. Hierarchy links — thicker gray tubes that connect the different views of the same logical service across tiers (for example a service seen by its agent, by the mesh, and as a Kubernetes service). These represent identity, not traffic, so they only appear when you select a cube, and show just that cube\u0026rsquo;s relatives — then disappear when you deselect. Interacting Camera — drag to rotate, scroll to zoom, and the on-screen toolbar (top-left) gives the same gestures as buttons. Arrow keys or WASD pan the view; hold Shift for a bigger step. Select a service — click a cube. It highlights, a detail card appears beside it (service name, layer, and an Open dashboard button that jumps to that service\u0026rsquo;s layer dashboard in a new tab), and its cross-tier hierarchy links light up. Click empty space, click another cube, or press Esc to deselect. Hover — hovering a cube shows a quick tooltip with the service\u0026rsquo;s name and layer next to it. Loading timeline Because a full deployment is too much to fetch in one request, the map loads in stages, and a slim timeline strip at the bottom shows the progress live:\nServices — the service roster and which layers they belong to. Templates — which layers carry a topology. Topologies — each topology-bearing layer\u0026rsquo;s call graph. Hierarchy — the cross-tier identity links between the different views of the same service. Only services that are new since the last run are fetched; the rest are reused, so a steady deployment costs nothing here on refresh. Layout — placing the cubes. Metrics — the per-service traffic numbers, fetched in batches so the cubes light up progressively. Each step shows its status as the map builds; click a step to open a drawer with its detail (services added/removed since last run, per-layer topology results, metric progress, and so on). A refresh button on the strip re-runs the whole sequence.\nConfiguration What the map shows is driven by a single configuration that an administrator edits in the UI at /admin/3d-map (linked under Dashboard setup in the sidebar). It is a structured editor — you work with tiers, layers, colors, and metrics through form controls, not raw JSON. Horizon ships a bundled default, seeded into OAP at first boot so the map is useful out of the box; your edits are kept as a local draft in your browser, and Check diff \u0026amp; push publishes them to OAP — the copy the map renders. In the default live template mode that OAP copy is the only source: if the template store cannot be read, the map reports that instead of rendering the bundled default. See Configuration File → Template source mode.\nFrom the editor you can:\nFilter layers — one global layer filter, written as a regex. A layer it excludes is dropped from the map entirely. This is the only filter; everything it admits is then placed on a tier. Arrange tiers — rename tiers, reorder them top-to-bottom, and pin each layer to a tier. A layer you don\u0026rsquo;t pin lands on the failover tier you nominate, so nothing silently falls off the map. Group layers — cluster several related layers (for example the SkyWalking self-observability components) into one labelled block on a tier, while each member keeps its own cube color. Color layers — pick each layer\u0026rsquo;s brand color (used for the cube, zone, and stamp). Choose a traffic metric — for each layer, set the single throughput metric its cubes display: the MQE expression, a display label, and a unit. The bundled defaults are seeded from each layer\u0026rsquo;s dashboard template, so most layers show a sensible number out of the box. A read-only Service-map layers list shows which layers lay their cubes out as a call graph — that comes from each layer\u0026rsquo;s template (its service-map capability), not from this page.\nPushed changes take effect the next time the map is opened. A Reset action reloads either the shipped bundled default or OAP\u0026rsquo;s current version, so you can start over before saving.\nExport downloads the map\u0026rsquo;s in-use configuration — the version live on OAP, or the bundled default when OAP has none — as a JSON file, for backup, sharing, or moving it to another OAP. Import reads a configuration JSON file and loads it as a local draft; preview it, then Check diff \u0026amp; push to publish. Import never writes OAP directly, and a file that isn\u0026rsquo;t a valid 3D-map configuration is rejected with a message.\nTuning the metric fan-out The map\u0026rsquo;s loading stages run in batches, several requests at once. How aggressively they do this is governed by the performance.bulk.infra3d block in horizon.yaml — an operator setting, not part of the map configuration, so it is not in the structured editor and does not travel with an exported / imported map. Edit horizon.yaml; the change is hot-reloaded and takes effect the next time the map is opened:\nmetricConcurrency — how many metric batches load at the same time. Default 4, range 1–8. Raise it to fill the cubes faster on a large deployment when OAP has headroom; lower it (toward 1) if a busy OAP rejects or slows the burst of metric requests during the Metrics step. metricBulkSize — how many services share one metric request. Default 6, range 1–12. Larger means fewer requests, but OAP rejects an oversized request, so this is capped — leave it at the default unless you have a reason to change it. topologyConcurrency — how many layer call-graphs load at once during the Topologies step. Default 4, range 1–16. templateConcurrency — how many layer templates load at once during the Templates step. Default 8, range 1–32. The defaults are tuned for a typical deployment; only revisit these if the loading timeline stalls on the Metrics, Topologies, or Templates step, or if OAP returns errors under the load.\nViewing the map needs read access (infra-3d:read, held by the built-in viewer role and above). A role without it does not get the topbar entry to the map at all. Editing and publishing the configuration needs infra-3d-setup:read to open, infra-3d-setup:write to publish (operators and admins hold both by default). See Roles and Permissions.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/operate/infra-3d-map/","title":"\u003c!--"},{"body":" Live Debugger Path: /operate/live-debug.\nThe Live Debugger captures, step by step, how a single analysis rule processes real data inside the connected OAP — so you can see why a metric comes out the way it does (or why it comes out empty) without reading the backend logs. You pick one rule, start a capture session, and OAP records each pipeline stage (input → filter → function → output) for a bounded number of executions. The capture runs on every reachable OAP node at once, so a rule that behaves differently on one node in a cluster is visible side by side. When you are done, you stop the session; captures are also saved locally so you can re-open them later.\nThis is a diagnostic surface for the same DSL families you edit on the Runtime Rules (DSL) page — it does not change any rule. Starting a session never alters collection; it attaches a recorder to the rule for the length of the session and detaches when you stop it or the retention window lapses.\nThe three DSL tabs The page is split into three tabs, one per DSL family. Each tab runs its own independent session, so you can have a MAL, a LAL, and an OAL capture going at the same time.\nTab DSL family What it debugs MAL Meter Analysis Language otel-rules, log-mal-rules, telegraf-rules, meter-analyzer-config — the meter pipeline for OTEL, log-derived, Telegraf, and agent-reported metrics. LAL Log Analysis Language lal — log parsing and extraction, capturable at block or statement granularity. OAL Observability Analysis Language the connected OAP\u0026rsquo;s OAL clauses — input source columns through aggregation and output. OAL rules are not runtime-editable (they are compiled into the OAP build), but they are still debuggable here — this is the one place you can watch an OAL clause execute against live source data.\nRunning a capture Pick a rule. For MAL, choose a rule file and then a specific metric inside it; for LAL choose the log rule (and block / statement granularity); for OAL choose the source and clause. Set the bounds. recordCap limits how many executions are captured (default and maximum 100). retention (min) is how long the session stays alive on OAP before it is reaped (default 5 minutes, maximum 60). Start. OAP installs the recorder across the cluster and begins collecting. The state pill moves through starting → capturing → captured. While capturing, the view refreshes about once a second; it stops polling on its own once every reachable node has finished (captured). Stop at any time to detach the recorder early. You do not have to wait for the retention window. Starting a new session for a rule that already has one running automatically replaces the prior session — the coverage strip notes how many prior sessions were stopped.\nCluster coverage strip Above the captured records, a per-node strip shows, for each OAP node, an install result (whether the recorder was accepted on that node) and a collect status (whether data came back). A rollup line summarizes how many nodes the session is live on (e.g. live on 2 of 3 nodes). Use it to spot a node that rejected the install or was unreachable — a missing node there explains a partial capture.\nReading the captured stages Each captured execution is shown as a chain of stages. Every stage reports an in → out count, so a stage that drops everything (a filter that matched nothing) is obvious at a glance. Clicking a stage highlights the matching fragment of the rule\u0026rsquo;s source text above the chain, tying the captured step back to the line of DSL that produced it.\nDiff-default label grouping When a stage emits many samples that share a metric name, they are grouped under a one-line summary rather than listed in full. Expanding a multi-sample group lands in diff mode by default: the labels that are identical across every sample collapse into a shared context shown once, and each sample row shows only the labels that differ. This makes \u0026ldquo;what distinguishes these series\u0026rdquo; the thing you see first. A toggle switches to the full per-sample label list when you want every label on every row. The same diff-first treatment applies to a run of output entities that share a metric — only the entity fields that vary are shown per row.\nVery large groups render a capped number of detail rows with a \u0026ldquo;+ N more\u0026rdquo; note; the summary count is always exact.\nThe LAL pipeline matrix A LAL capture renders as a grid — one column per captured record, one row per pipeline step (input, the per-statement or per-block function steps, output). The first column names each step and stays pinned as you scroll sideways through the records; each cell holds that record\u0026rsquo;s data at that step.\nIt reads any log format. A cell shows whatever fields OAP serialized for the record — a plain LogData input shows service / endpoint / tags / body, while an Envoy access-log (ALS) record shows its built snapshot (service, endpoint, response data, and the access-log content as JSON). When OAP cannot serialize a record\u0026rsquo;s raw input, the cell shows the reason (for example jsonformat-failed …) instead of rendering blank, and a small label names each cell\u0026rsquo;s payload class.\nFilter a row to the records that have data. A step row that has gaps carries a filter; turning it on narrows the grid to just the records that produced data for that step — for example the output row to only the records that emitted output (an abnormal-only rule aborts most records, so only a few reach output). The row count shows how many of all captured records reached that step.\nInspect and diff a cell. Each cell has a button — VIEW on the input row, DIFF on the builder rows — that opens the cell\u0026rsquo;s complete payload in a JSON viewer with the log content shown as formatted JSON. For the built-log snapshots you can compare stages: a picker presents the captured rule with each per-statement step on its line and the extractor / sink blocks as selectable ranges, and choosing one shows a side-by-side diff of the two snapshots — the quickest way to see which statement or stage added, changed, or dropped a field.\nEach OAP node renders its own matrix; filtering or selecting in one node\u0026rsquo;s grid does not affect another\u0026rsquo;s.\nCapture history Every session you run is saved to capture history, browse it at /operate/live-debug/history (or the history link on each tab). History is stored locally in your browser — it is not shared between users or machines and survives reloads, with the most recent captures kept per DSL family.\nFrom history you can:\nReplay a finished capture — re-open the recorded stages exactly as they were captured, without re-running anything on OAP. A banner marks that you are viewing a saved capture, with a back to live control to return. Resume a capture whose retention window has not yet lapsed — re-attach to the still-live OAP session and continue polling it. A capture that was archived before its first poll returned data shows as having no records; run a longer-lived capture to give the pipeline time to fire.\nRequirements The OAP dsl-debugging module must be loaded. This is the module that powers start / poll / stop across MAL / LAL / OAL; the page shows a warning banner when it is missing. See Required OAP Modules. The receiver-runtime-rule module must also be loaded — it backs the rule picker (the catalog of rules you choose from). It is a separate module from dsl-debugging: a deployment can have one without the other, in which case either the picker or the capture itself will be unavailable. OAP admin port reachable from Horizon. Access control Permission Grants live-debug:read View the Live Debugger, the active-session list, cluster status, and capture history. Nothing else is required to watch a capture. live-debug:write Start and stop capture sessions. Nothing else is required to run one — no rule:* grant takes part, and holding every rule verb without live-debug:* gets you nothing here. In the bundled roles, both are held by operator (and admin). A read-only viewer can be granted live-debug:read on its own to inspect existing sessions and history without being able to start new captures. See Roles and Permissions.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/operate/live-debugger/","title":"\u003c!--"},{"body":" Log Inspect Log Inspect (/operate/log-inspect) is the cross-layer log query tool in the sidebar. The per-layer Logs tab starts from a layer and the service picked in its header; this page starts from nothing. Open it, name any service — or no service at all — and query across everything the log store holds. It unifies three log sources on one page: the stored log stream, browser JavaScript errors, and on-demand Kubernetes pod tails.\nLike the other triage pages, it owns its own time range — the global topbar time picker does not apply — and (for the stored sources) nothing is fetched until you press Run query. Conditions are staged; switching source clears the previous result so streams never mix.\nSources The Source toggle at the top picks what kind of logs you are after:\nRaw — the logs SkyWalking has collected and stored: the same store as the per-layer Logs tab, queried across every layer. Browser — the JavaScript errors reported by the browser agent, with inline source-map management and per-row stack de-obfuscation. Kubernetes Pod logs — a live tail of one pod\u0026rsquo;s container output, pulled through OAP on demand and never persisted. Target — pick it, type it, or leave it blank For the Raw and Browser sources the Target is optional: blank queries every service in the window. Two modes scope it:\nPick — choose a Layer, then a Service from its catalog, then optionally an Instance and/or Endpoint. On the Browser source these last two are labelled Version and Page, because that is what a browser app\u0026rsquo;s instances and endpoints are. Type — enter a Service name directly, with a Real checkbox (off for a virtual/peer service), plus optional instance/endpoint (version/page) names. Typing needs no layer. The → edit as text link converts the current Pick selection into the Type form. Raw and Browser share one target, so switching between them keeps your pick; only crossing into or out of the pods source resets it.\nRaw — stored logs across layers Conditions for the stored stream:\nCondition What it does Tags Comma-separated key=value pairs, AND-joined, with autocomplete: type to see known keys, type = for that key\u0026rsquo;s known values, Enter commits the pair and primes a comma for the next. Filter by level with a level=… tag. Trace ID Show only the lines correlated with one trace. Time A rolling preset (15m, 30m, 1h, 3h, 6h, 12h, 24h) or Custom… with an absolute start/end pair. Second-precision, like the per-layer tab. Limit Result cap: 20, 50 (default), 100, or 200. The server additionally caps a single batch at its configured page-size limit (100 by default), so 200 only takes effect when that limit has been raised — see the query-limits section of the horizon.yaml reference. Run query fetches one batch of the newest matching lines. Rows render exactly as on the per-layer tab — timestamp, level, service, an ↗ trace link when trace-correlated, a format chip, and a one-line preview — and clicking a row opens the same full-payload popout: format-aware pretty-printing, Copy, the service/instance/endpoint/trace context, and the tag table. The ↗ trace links open the related trace\u0026rsquo;s waterfall in an overlay without leaving the page. Escape or the backdrop closes the popout, and a re-run that no longer contains the open row closes it too.\nUnlike the per-layer Logs tab there is no density histogram, no Levels strip, and no pager — this page returns a single batch capped by Limit. It trades the browsing chrome for reach: any service, any layer, or all of them at once.\nBrowser — JS errors with source-map resolve The Browser source queries the errors browser agents report, across every browser app at once if you leave the target blank. Its conditions are Category (All, or one of AJAX, RESOURCE, VUE, PROMISE, JS, UNKNOWN) plus the shared Time and Limit.\nBelow the conditions sits the source-map manager — the same map store the per-layer Browser Logs tab uses, managed inline so you never have to leave the page to make a stack readable. It lists the maps currently available (statically mounted ones and temporary uploads), shows the memory-usage bar, and offers Upload .map and per-upload remove. Uploaded maps live in server memory only; mounted maps cannot be removed here. If de-obfuscation is disabled on the server, the manager says so instead.\nResults render as a dense error list: time, category (color-keyed), page, app version, and the message. Click a row to open the browser-error popout — the error\u0026rsquo;s metadata and raw stack on one side, and the de-obfuscation control on the other: pick a hosted map (the first one is pre-selected), press Resolve, and read the original file/line/symbol frames with source snippets. Which map matches which build is your call — see Browser Logs \u0026amp; Source Maps for the matching rules and the resolvable categories.\nKubernetes Pod logs — live tails without entering a layer The pods source is the cross-layer twin of the per-layer Pod Logs tab: it tails one pod\u0026rsquo;s container output straight from the cluster through OAP. Nothing is persisted — each poll pulls the trailing window and discards it — so the pod must be currently running.\nUnlike the other two sources, the target here is required: a specific pod and container.\nPick a Layer and a Service (with exactly one Kubernetes-aware layer in your menu, the layer is pre-selected) — or switch the service field to Type and enter the service name directly, no layer needed. Pick the Pod — the service instance. A single-pod service is auto-selected. Pick the Container — the pod\u0026rsquo;s containers are listed and the first is auto-selected. Choose the trailing Window (Last 30s to Last 30m) and the poll Interval (2s–30s). Press Start to tail live, Pause to stop, or Refresh for a one-shot fetch (which also pauses a running tail). Include / Exclude chip fields narrow the lines: type a full-line regular expression (for example .*error.*) and press Enter to add it; the × removes a chip. Includes keep matching lines, excludes drop them, and changing them mid-tail re-runs with the new filters. Re-targeting the pod, container, or service stops the tail so a stale loop never bleeds across pods.\nOn-demand pod logs are disabled by default on OAP; when the feature is off or the pod no longer exists, the reason appears in a banner — see the pod-logs troubleshooting on the Logs page, which applies here unchanged.\nResolved query For the Raw and Browser sources, a Resolved query toggle appears after each run: it names the source and expands to the exact condition that was sent — resolved service ids, computed window, filled-in defaults. When a query returns something unexpected, read it first. Pod tails are live fetches rather than stored-store queries, so they have no resolved-query panel.\nPermissions The page and the raw/browser queries require the inspect:read permission. The tag autocomplete, the container list, and the pod tail additionally use logs:read; the source-map list and stack resolve use browser-errors:read (uploading or removing maps needs source-map:write); and the Pick-mode layer/service dropdowns use metrics:read. The bundled roles that grant inspect:read include the read verbs. See Roles and Permissions.\nRelated Logs — the per-layer stored-log stream and Pod Logs tab, with the full condition and troubleshooting reference. Browser Logs \u0026amp; Source Maps — source-map matching rules, static provisioning, and which error categories resolve. Trace Inspect — the cross-layer sibling for traces. Metrics Inspect — the cross-layer view of OAP\u0026rsquo;s metric catalog. ","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/operate/log-inspect/","title":"\u003c!--"},{"body":" Logs Horizon surfaces logs through two distinct tabs, each backed by a different OAP source.\nThe Logs tab queries the logs SkyWalking has collected and stored — application and service log records, indexed and filterable, correlated with traces. The Pod Logs tab does something different: it live-tails a Kubernetes pod\u0026rsquo;s container logs on demand, pulled straight from the Kubernetes API through OAP and never persisted. They appear as separate tabs because they answer different questions — \u0026ldquo;what did this service log over the last half hour?\u0026rdquo; versus \u0026ldquo;what is this pod printing to stdout right now?\u0026rdquo;.\nWhich tabs a layer shows depends on the layer template. The Logs tab appears on layers whose template enables it (for example GENERAL, MESH, MESH_DP, NGINX, ENVOY_AI_GATEWAY, the mini-program and mobile layers). The Pod Logs tab appears only on the Kubernetes-aware layers K8S_SERVICE, MESH, and MESH_DP.\nFor browser JavaScript errors reported by the browser agent — a separate stream with its own source-map de-obfuscation — see Browser Logs \u0026amp; Source Maps. That is not the same as the collected service logs described here.\nFor cross-layer digs — querying any service\u0026rsquo;s stored logs by name (or all services at once), browser errors, or a pod tail without entering a layer — see Log Inspect.\nStored logs Open a layer that has a Logs tab and pick a service in the header. The stored log stream loads for that service over the page\u0026rsquo;s own time range, newest first.\nScoping and filtering The conditions bar narrows the stream. Every filter is optional; together they are AND-joined.\nInstance — restrict to one service instance. The default is All. On a sidecar layer this picker is labelled Sidecar.\nEndpoint — restrict to one endpoint. Type to search the endpoint list, then click a result to pin it; the × clears it back to All.\nTrace ID — paste a trace id to show only the log lines correlated with that trace. Copy the id from a trace\u0026rsquo;s span detail and paste it here; there is no one-click jump from a trace to its logs.\nContent — words the log line must contain, space-separated for AND (timeout db matches only lines carrying both). This field appears only when your storage backend can search log content — ElasticSearch can, BanyanDB and the others cannot, and Horizon asks the connected OAP which it is. On a backend that cannot, the field is absent rather than present-and-ignored, because OAP accepts the condition there and returns the unfiltered stream — which reads as \u0026ldquo;everything matched\u0026rdquo;.\nTags — a single key=value field with autocomplete. Start typing a key to see suggested keys; type = to switch the suggestions to known values for that key; press Enter to commit the tag. Committed tags show as removable chips under the bar and ride along on the query as additional filters.\nLevel — the Levels strip above the stream doubles as a filter. Click error, warn, info, or debug to show only that level; click again to clear. The level filter is sent to OAP as a level tag, so pagination and counts reflect the filtered set. The other chip (lines whose level tag is missing or unrecognized) is informational only — it has no server-side value to filter on, so it is not clickable.\nThe stream queries on demand, not on every keystroke. Editing a condition stages it; nothing is fetched until you press Run query, which runs the query and resets to the first page. A freshly opened tab shows a Pick your conditions, then click Run query prompt rather than auto-loading, and switching service resets to that prompt — clearing the level and tag filters — so the previous service\u0026rsquo;s logs never linger under the new one. Paging and the page-size picker fetch immediately once you have run a query.\nTime range The Logs tab owns its own time range — the global topbar time picker is paused while you are here, so auto-refresh won\u0026rsquo;t shift the window mid-investigation. Pick a rolling preset (Last 15 min through Last 24 hours, default Last 30 min) or choose Custom… to pin an absolute start/end with two date-time inputs.\nLog queries use second-precision time windows. Logs are record-style data anchored at second granularity, so the window is not rounded to the minute — the most recent (and usually most interesting) lines are never chopped off. The window is capped at 7 days. A custom range longer than that is refused on the page, with the reason under the control, rather than being quietly shortened — a query made directly against the API is trimmed to the most recent week instead, so it still answers with the part that matters.\nReading the stream A density histogram sits above the stream: time on the x-axis, log count on the y-axis, each bar stacked by level (error / warn / info / debug / other) with the same colour as the legend. Hover a bar to see that bucket\u0026rsquo;s time range and per-level counts. The histogram is built from the currently loaded page, so it shows the shape of what is on screen, not the whole window.\nThe Levels strip carries a count per level next to each chip. Those counts come from a window-scoped sample (a few hundred of the most recent rows in the window, larger than one page), so they reflect the window\u0026rsquo;s level distribution rather than only the visible page. The strip notes the sample size it used, and says when the window held more rows than the sample counted — narrow the window if you need the counts to cover all of it.\nEach row shows the timestamp, the level, the service (with any group prefix decoded), an ↗ trace link when the line is trace-correlated, a format chip (JSON / YAML / TEXT), and a one-line preview of the content. Rows are colour-keyed by level.\nHorizon renders the payload according to its content. OAP labels payloads as JSON or plain text; on top of that, Horizon sniffs for JSON and YAML structure so an unlabelled-but-structured body still gets the right treatment. JSON is compacted to a single line in the preview and pretty-printed in the detail view; YAML keeps its keys; plain text is whitespace-collapsed.\nClick a row to open the full payload in a popout: the complete content, format-aware pretty-printing, a Copy button, the service / instance / endpoint / trace context, and a table of all tags on the line. If the line is trace-correlated, an ↗ trace button there (and the ↗ trace link on the row) opens the related trace\u0026rsquo;s waterfall in an overlay without leaving the log stream — the row\u0026rsquo;s timestamp is passed along so the trace is found even when it sits in a colder storage tier. Press Escape or click the backdrop to close.\nThe pager at the foot shows the current page and the row count on it; Prev / Next walk the pages, and the page size (20, 50, or 100) is set on the conditions bar. There is no \u0026ldquo;N of M\u0026rdquo; total, because the log query does not report one — Next is offered only when there really is another page with rows on it, so a full last page ends the walk instead of stepping onto an empty screen. Changing the page size restarts at page 1.\nTroubleshooting stored logs No rows returned. Confirm the service actually ships logs to OAP, that the storage backend has the logs module enabled, and that the time range covers when the logs were produced. Narrow filters (a tag, a level, an endpoint) can also empty the result — clear them and widen the window.\nA filter empties the stream. Tag and level filters are exact-match on indexed dimensions. A level value or tag value that doesn\u0026rsquo;t exist in the stored data returns nothing; check the value against what the Levels counts and the tag autocomplete actually offer.\nRun query is greyed out. The tab does not yet know which service to read, and says which case it is: Resolving service… while the picked service is being looked up, or a note that the selected service is not in this layer (it aged out of OAP, was renamed, or the link points elsewhere) — pick another one. The stream is always read for one service, so the tab waits instead of querying the whole layer.\nPod logs The Pod Logs tab tails a Kubernetes pod\u0026rsquo;s container logs live. There is no stored history to page through — each refresh pulls the trailing window straight from the Kubernetes API through OAP, shows it, and discards it. Nothing is persisted.\nStarting a tail Pick a service in the header, then pick a Pod (a service instance) — the page is pinned to one pod at a time. Pick a Container. Horizon lists the pod\u0026rsquo;s containers and auto-selects the first; switch if the pod runs more than one. Choose the look-back Window (Last 30s, 1m, 5m, 15m, or 30m) — how far back each poll reaches. Choose the poll Interval (2s, 5s, 10s, or 30s) — how often the window is re-fetched while live. Press Start. The trailing window streams into a read-only viewer and re-polls on the interval until you press Pause. The header shows a live indicator, the line count, and how long ago the view last updated. Changing the container, window, interval, or filters while tailing re-runs the query with the new settings. The viewer is read-only and keeps the newest line in view as fresh logs arrive.\nInclude and exclude filters Two filter rows narrow what the tail shows. Include keeps only lines that match; Exclude drops lines that match. Type an expression and press Enter to add it as a chip; the × on a chip removes it. Both are evaluated by OAP as full-line regular expressions (for example .*error.*), so they match against the whole log line, not a substring. Multiple expressions in a row stack as additional conditions.\nTime precision Pod-log windows are second-precision — this is a live tail, anchored at the current second. OAP caps a single tail window at 30 minutes; the longest selectable window is Last 30m.\nTroubleshooting pod logs On-demand pod logs are disabled by default on OAP because container logs can leak secrets. When the feature is off, or when the pod can\u0026rsquo;t be resolved, OAP returns a reason instead of data and Horizon shows it in a banner rather than an empty pane. Two common cases:\n\u0026ldquo;Logs unavailable\u0026rdquo; with a reason. If the reason indicates the feature is off, enable on-demand pod logs on the OAP side. If it indicates the pod wasn\u0026rsquo;t found, the instance you picked points at a pod that no longer exists (a finished rollout or a scaled-down replica) — pick a currently-running pod.\nThe tail stops on its own. A pod that vanishes mid-tail (a rollout or scale-down) makes the next poll fail; Horizon stops the loop and surfaces the reason rather than spinning on errors. Re-pick a live pod and Start again.\nPermissions Both tabs — stored log queries, tag autocomplete, the container list, and the on-demand tail — require the logs:read permission. See Roles and Permissions.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/operate/logs/","title":"\u003c!--"},{"body":" Profiling Profiling drills past metrics and traces into the call stacks, kernel events, and process-to-process conversations of a running service. Horizon surfaces SkyWalking\u0026rsquo;s profiling capabilities as a set of per-layer tabs on the service you have selected: Trace Profiling, eBPF Profiling, Async Profiling, Network Profiling, and pprof. Each profiling tab only appears on a layer when OAP reports that the service supports that kind of profiling, so the tabs you see depend on the agent and platform behind the service.\nEvery profiling tab follows the same shape: a task list on the left, a New Task control to start a profiling run, and a result panel on the right that renders the captured data once OAP has fanned the task out to the relevant instances or processes. Results are shown as an indented stack tree or a flame graph, with a toggle between the two where both apply.\nTask creation is consistent across every tab. The New Task control opens once you have selected a service; for Network Profiling, the target instance is picked inside the dialog. Inside the dialog, a target that cannot be profiled at all — no profilable processes for eBPF, or no instances on the service — disables Create with the reason shown next to it, rather than a silently greyed-out control; advisory checks (such as Network Profiling\u0026rsquo;s process list) warn without blocking. You always see why a task cannot be started.\nAccess control Profiling is gated by two distinct permissions:\nprofile:enable — required to start a profiling task (the New Task control). It is held by the operator role and above.\nprofile:read — required to view profiling results. It is part of the read-only data catalog held by viewer, maintainer, and operator.\nA viewer can therefore open a profiling tab and inspect existing results, but cannot create new tasks. See Roles and Permissions for the full permission catalog.\nTrace Profiling Trace Profiling samples the call stacks of slow trace segments. You start a task scoped to a service (and optionally a single endpoint), and the agent dumps CPU stacks from segments that exceed the task\u0026rsquo;s threshold while the task is running.\nTo start a task, open the New Task dialog and set:\nEndpoint name — restrict sampling to one endpoint, or leave it as (any) to profile all endpoints on the service.\nStart when — begin immediately (now) or at a scheduled time.\nDuration — how long the task runs, in minutes.\nMin threshold (ms) — only segments slower than this are sampled.\nDump period (ms) — how often a stack snapshot is taken while a sampled request runs.\nMax sampling count — the cap on how many segments the task collects.\nOnce the task has collected sampled traces, pick a trace from the Sampled traces list to load its spans. Select a profiled span and press Analyze to build its call tree. The result renders as either a Tree (indented stack table) or a Flame graph. A Data mode toggle switches between Include children (the whole span\u0026rsquo;s time) and Exclude children (only the time spent in the span itself, with child-span windows subtracted). The eye icon on a task opens a detail panel with the task\u0026rsquo;s parameters and the per-instance operation log.\neBPF Profiling eBPF Profiling samples kernel-level stacks from a process without an in-process agent, driven by SkyWalking Rover. It supports two capture targets:\nON_CPU — where the process spends CPU time.\nOFF_CPU — where the process is blocked off CPU (waiting on locks, I/O, scheduling).\nA task targets a service and, optionally, a set of process labels (leave the labels empty to profile all processes). You choose the target, a start time, and a duration in minutes. Open the New Task dialog from the selected service; if OAP reports no profilable processes for it, the dialog says so and Create stays disabled.\nWhen you select a task, the result auto-analyzes. The filter bar lets you narrow the view:\nLabels — restrict the aggregation to the chosen process labels.\nAggregate — Count (number of stack samples) or Duration. Duration is only available on OFF_CPU tasks, since off-CPU samples carry a blocked-time duration that on-CPU samples do not.\nProcesses — pin specific processes from the capture; pinning re-runs the analysis immediately.\nThe result is shown as a Flame graph or a Tree, with a banner stating the wall-clock window the capture covers and how many schedules contributed.\nAsync Profiling Async Profiling runs the async-profiler against a live Java service, capturing JVM-level stacks without restarting the process. A task targets one or more service instances and one or more event types. The supported events are:\nCPU ALLOC LOCK WALL CTIMER ITIMER You can select multiple instances and multiple events in a single task, with a duration from 30 seconds up to 15 minutes. After the task runs, choose which instances to include and which event type\u0026rsquo;s tree to render, then press Analyze. Because a single task can collect several event types, the result panel has an Event type selector — switching it re-draws the flame graph for the selected JVM event (for example EXECUTION_SAMPLE for CPU/Wall/Timer events, LOCK for lock contention, or one of the object-allocation event types for ALLOC).\npprof pprof profiles a live Go service through the standard Go runtime profiler. Unlike Async Profiling, a pprof task captures exactly one event type, chosen from:\nCPU HEAP BLOCK GOROUTINE MUTEX ALLOCS THREADCREATE The dialog adapts to the event you pick:\nCPU, BLOCK, and MUTEX are time-bounded captures and require a Duration (up to 15 minutes).\nBLOCK and MUTEX additionally take a Dump period sampling rate — for BLOCK it is a blocked-nanoseconds rate, for MUTEX a contention-occurrences rate; a value of 1 samples every event. Because lower means more samples, an invalid value is rejected with the reason rather than silently replaced with a default.\nHEAP, GOROUTINE, ALLOCS, and THREADCREATE are one-shot snapshots — they take no duration and no sampling rate, capturing the current state at the moment the task fires.\nA task can target multiple Go service instances. After it runs, select the instances to include and press Analyze to render the single result tree as a flame graph.\nNetwork Profiling Network Profiling captures the network conversations between processes of a service instance and renders them as a process-level topology. It mounts on a specific instance, which you pick inside the New Task dialog. The dialog lists the rover-monitored processes that recently reported on that instance — as advice, not a gate: an instance with no recently-reported process shows a warning that the task may collect nothing, but you can still create it and let OAP decide. Once an instance is chosen, the task defines which traffic to sample.\nEach sampling rule scopes the capture — by URI pattern, by HTTP 4xx / 5xx responses, or by a minimum duration — and controls how much of each request and response body is collected. OAP runs every network task for a fixed ten minutes and the create request carries no duration, so the New Task dialog defines the sampling rules rather than a run length.\nThe result is a honeycomb topology: each cell is a process, and the edges between them are the observed inter-process calls. Selecting an edge opens a detail panel with that process-to-process relation\u0026rsquo;s metrics (call rate, latency, and bytes transferred) charted over the task\u0026rsquo;s run window. The topology that drives this layout is the same process-relation data that powers the 3D Infrastructure Map.\nContinuous Profiling Everything above starts a profiling task on demand — you pick a target and start it. Continuous profiling is the opposite: you arm a policy once, and the profiling task starts by itself whenever a process crosses a threshold, with nobody present. It is how you catch a problem that only appears at 3 a.m.\nContinuous profiling is eBPF profiling only, and it requires Rover. A policy can trigger ON_CPU, OFF_CPU or NETWORK — the same three flavours as the eBPF and Network Profiling tabs above — and the Rover agent both evaluates the thresholds and runs the resulting task. There is no continuous trace, async-profiler or pprof profiling; those stay on demand. So a service with no Rover agent can hold a saved policy, but nothing will fire until one is deployed.\nPolicies are edited on the layer\u0026rsquo;s Continuous Profiling tab, beside the eBPF and Network Profiling tabs whose tasks they trigger. The tab has its own Target service picker: each service is labelled with the targets it already has armed, and the picker can be filtered by that — including no policy, which is the set you want when arming services that are not set up yet. Opening the tab selects the first service that already has a policy, or the first service in the layer if none does. Once a service is selected, the tab shows its policy plus the instances OAP is currently evaluating it against.\nNothing here is gated on the agent already being present, because arming a policy before deploying the agent is a valid order of work: the policy is backend configuration, and it simply starts firing once an eBPF agent begins reporting. If no process of the selected service has reported eBPF-profiling support recently, the tab says so as a warning and still lets you save.\nThe tab appears on a layer whose template enables the Continuous Profiling component (Layer Setup). It ships enabled on MESH, matching where the previous SkyWalking UI placed it. Rover registers its processes into MESH, MESH_DP and K8S_SERVICE by default (which layer is configurable per discovery analyzer), so those are the layers where enabling it is likely to be useful — turn it on there if your Rover deployment reports into them.\nA policy is a set of targets — ON_CPU, OFF_CPU, or NETWORK — and each target carries one or more conditions. A condition is:\na measurement (labelled that way on screen; OAP\u0026rsquo;s own name for it is ContinuousProfilingMonitorType) — PROCESS_CPU, PROCESS_THREAD_COUNT, SYSTEM_LOAD, HTTP_ERROR_RATE, or HTTP_AVG_RESPONSE_TIME; a threshold, whose unit follows the measurement — a percentage for CPU and error rate, a thread count, a load average, milliseconds for response time. Every threshold is a whole number: OAP parses all five as integers and rejects anything else, so 0.5% or 4.5 will not save. CPU percent and HTTP error rate must be 1–100; the rest must be greater than 0. The count cannot exceed the period, and one target cannot carry two conditions of the same measurement. a period, the number of seconds of metrics to evaluate; a count, how many matching evaluations must occur before profiling is triggered. The two HTTP monitors can additionally be scoped to specific traffic. Choose All traffic, URI list or URI regex — one or the other, never both. Nothing on the backend rejects a rule carrying both, but the agent applies the list and silently ignores the regex, so the form makes the choice explicit; switching away from a filter you have filled asks before erasing it.\nTwo things are worth knowing before you save:\nSaving replaces the service\u0026rsquo;s whole policy. OAP stores one policy per service, and the page sends everything you see. A target you delete is deleted; keep every rule you want to survive.\nA policy only evaluates processes an eBPF agent reports. Inside each target sits a paged Where it runs panel: the instances and processes OAP evaluates for that target, with how often each has actually triggered profiling recently, searchable by instance or process name, each row expanding to that instance\u0026rsquo;s processes. That trigger count is the thing to read: it is the difference between a policy that is stored and one that is working, and it is the only per-target signal here (the process list itself is the same for every target). An empty panel means nothing is reporting for the service at all.\nThe panel is not a Rover presence check — it lists a process whether or not that process can be eBPF-profiled. If the panel has rows and the warning above says no process reported eBPF-profiling support, the reading is \u0026ldquo;processes are there, but none are profilable\u0026rdquo;, which points at Rover\u0026rsquo;s configuration rather than its absence.\nTasks a policy starts appear in the eBPF Profiling and Network Profiling tabs alongside the ones you start by hand, so a fired policy is read the same way as an on-demand task.\nReading policies needs profile:read; saving one needs profile:enable, the same permission as starting a task by hand — because that is what a policy eventually does.\nTroubleshooting A continuous-profiling policy never fires — first check that it is actually applied: each target shows Applied or Not applied, and rules that have only been typed are not running. Then check the Where it runs panel. If it is empty, nothing is reporting for that service and the thresholds are irrelevant; deploy Rover for the service. If processes are listed but the trigger count stays at zero, the threshold is not being crossed — lower it, lengthen the period, or reduce the required count.\nNo profiling tabs on a layer — OAP did not report profiling support for that service. Each tab requires the corresponding capability (trace, eBPF, async-profiler, network, or pprof), which depends on the agent or Rover deployment behind the service.\nNew Task is unavailable — you have not selected a service, or you lack profile:enable.\nCreate is disabled inside the New Task dialog — the chosen target cannot be profiled, and the reason is shown next to the button: for eBPF, OAP reports no profilable processes for the service; for Async Profiling, pprof, and Network Profiling, the service has no instances. On Network Profiling, an instance whose processes have not reported recently is a warning, not a block — the task can still be created.\nTask list is empty after creating a task — the task is created, but results only appear once OAP has dispatched it to the instances or processes and they report back. The view polls for the new task briefly; use the refresh control if it does not appear.\nAnalyze returns no data — the task ran but collected no samples in the selected window or scope. For Trace Profiling, confirm the threshold was low enough to sample real traffic; for eBPF and pprof, confirm the chosen processes or instances were live during the capture.\nRelated Roles and Permissions — profile:enable and profile:read.\n3D Infrastructure Map — the process and instance topology that the network view draws on.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/operate/profiling/","title":"\u003c!--"},{"body":" Service Map \u0026amp; Topology The per-layer Topology tab draws a layer\u0026rsquo;s services as an interactive, directed call graph: who calls whom, how hard each call lane is running, and how healthy each service is. It is the per-layer companion to the deployment-wide 3D Infrastructure Map — same call relationships, but flat, focused on one layer, and clickable down to the individual instance.\nAround the service map sit three related views that share the same canvas and reading conventions: the instance map drill-down (instance-to-instance traffic across a service-pair), the Deployment tab (instance-to-instance traffic inside one service), and the API dependency tab (the same graph drawn at the endpoint level). All four are driven by each layer\u0026rsquo;s dashboard template, so what they measure varies by layer — but how you read and narrow them is identical.\nWhich tabs you see These views are layer capabilities, not global pages — a layer shows only the tabs its template enables:\nTopology appears for any layer whose template declares a service map.\nDeployment appears only for layers that configure an intra-service instance graph (for example a clustered store whose nodes call each other).\nDependency (API dependency) appears only for layers that configure endpoint-level dependencies.\nA layer with none of these declared shows no topology tabs at all. The map always opens on the layer\u0026rsquo;s own service map; the instance map is reached by drilling into an edge, not from the sidebar.\nReading the service map The graph flows left to right. Entry traffic (the synthetic User node, and other callers with no upstream service) anchors the left edge; each downstream hop sits one column to the right. Within a column, the busiest services are stacked toward the top so the heavy lanes line up across columns.\nNodes Each circle is one service. Three visual channels carry its numbers, and all three come from the layer\u0026rsquo;s template — nothing is hardcoded, so the exact metric and unit differ per layer:\nThe number inside the circle is the service\u0026rsquo;s headline throughput (requests per minute for app-style layers, queries or operations per second for data layers, and so on), shown with its configured unit.\nThe colored ring around the circle is the health band. It maps a health metric (SLA, success rate, Apdex, error rate, …) onto a green → yellow → orange → red ramp. The legend under the map names the metric, prints the four break points, and states the reading direction — higher = better for SLA / success-rate / Apdex style metrics, lower = better for error-rate style metrics.\nA technology badge floats above the circle, picked from the service\u0026rsquo;s detected component (the database, cache, queue, gateway, or framework SkyWalking identified). A service whose component SkyWalking could not resolve shows a neutral badge.\nTwo node shapes are not real services: the User entry node, and conjectured peers — external or unresolved callees (an address like localhost:-1 or rcmd:80 that SkyWalking observed traffic to but has no agent on). Conjectured peers are drawn as a cloud-with-? and carry no metrics of their own; they exist on the map only to complete a call lane. Selecting one shows a virtual tag in its detail panel.\nEdges A line is a call relationship. Its thickness tracks the call rate on that lane — heavier line, more traffic. The flow animation along the line shows direction (caller → callee). Edges are not colored by health; the ring on the nodes carries that signal.\nClick a line to open its detail panel. Each line metric is shown twice — Client (as the caller measured it) and Server (as the callee measured it) — side by side, each with a sparkline over the window so you can see the trend, not just the latest number. A lane may report only one side: a call into a conjectured peer has no server-side numbers, a call out from the User node has no client-side numbers, and the panel labels those client only / server only rather than showing a blank.\nNode detail Selecting a node opens a panel with its template metrics, its Upstream list (services it calls) and Downstream list (services calling it), and two jumps: Open service (its layer dashboard) and API map → (its endpoint dependency graph). The node and edge panels are independent — you can keep both open at once.\nCross-layer hierarchy (Smartscape) One logical service is often observed by several layers at once — the same workload seen by its in-process agent (GENERAL), by its sidecar (MESH / MESH_DP), and as a Kubernetes service (K8S_SERVICE). When the service you have selected has such cross-layer counterparts, a small chip appears on the selected node\u0026rsquo;s edge; clicking it opens the hierarchy overlay.\nThe map dims but stays visible for spatial context, the selected service lights up in place with a FOCUS tag, and its counterparts in other layers fan out around it — one labeled, layer-colored lane per layer, request-near layers above the focus and infrastructure-near layers below, with counterparts in the same lane spread side by side. Each counterpart is named the way its own layer\u0026rsquo;s map would name it, and one that SkyWalking knows only from observed traffic carries a virtual tag. Auto-refresh is paused while the overlay is open, so nothing shifts under you.\nNavigation is deliberately two-step so scanning never jumps you away: click a counterpart once to select it, then click the Open in \u0026lt;layer\u0026gt; chip beside it to open that layer\u0026rsquo;s drill-down in a new browser tab with the service pre-selected. A counterpart whose layer has no active layer template in Horizon is dimmed and cannot be opened — the service exists on OAP, but there is no page to land on. Close the overlay with the ×, the Esc key, or a click on the dimmed background.\nThe chip only appears when OAP reports cross-layer counterparts for the selected service, and it is not offered in the embedded overview-widget map — open the full Topology tab.\nFocusing and narrowing the map By default the map seeds from every service in the layer — the full layer overview. That is the right starting point for a small layer and the wrong one for a large estate. Two controls narrow it:\nFocus — open the service picker (top-right of the Topology toolbar) and select one or more services. The map then redraws around just those services and their neighbors. The picker supports search and selecting a whole service group at once.\nDepth — once at least one service is focused, a depth control appears: 1 hop, 2 hops, or 3 hops. Depth is how many call hops out from the focused service the map walks. Depth has no effect on the full-layer overview (it already includes everything), so the control is hidden until you focus a service.\nAdditional controls on the canvas:\nFilter (top-left) hides nodes by layer, or hides the User node, so a busy graph reading from several layers can be thinned to the layers you care about. The filter stores what is hidden, so a service that only appears after a depth or time change starts out visible. Reset clears it.\nZoom / Fit (top-right) and drag-to-pan move the camera; double-click the canvas to fit the whole graph. Drag a node to reposition it; the layout holds your placement.\nThe map honors the topbar time picker — change the window and every node and edge metric re-reads for that range.\nInstance map (drill-down) When a service-to-service edge is selected, the edge panel offers Instance map →. This opens the instance-to-instance graph for that one service pair: the caller\u0026rsquo;s instances in the left column, the callee\u0026rsquo;s instances in the right, and the instance-level call relationships between them. It is the view for answering \u0026ldquo;which instance is the slow one\u0026rdquo; once the service map has pointed at the lane.\nThe instance map keeps two service pickers at the top so you can swap either side to an adjacent service without returning to the service map, a Service map back link, and the same client | server line-metric panel as the edge detail. A picker is shown only when there is a real choice — if a side has a single counterpart, its name is simply printed.\nDeployment (intra-service topology) The Deployment tab draws the instance-to-instance call graph within a single service — the nodes of a clustered service talking to each other (for example a distributed store\u0026rsquo;s members). It shows the full container inventory for the service, grouped by cluster or by role, with per-node metrics; call relationships are drawn as edges where SkyWalking reports them. A container that exists but has no intra-service call in the window (an idle sidecar, say) still appears on the map as an inventory node rather than being hidden.\nGrouping (by cluster, by node role / node type) comes from the layer template. When a layer reports no intra-service relations, the tab is a grouped inventory of the service\u0026rsquo;s containers with their metrics — no edges — which is the expected, by-design state for those layers, not an error.\nAPI dependency (endpoint graph) The Dependency tab is the service map drawn one level down, at the endpoint (API) level. Pick a service in the header, search its endpoints, and select one — the map then shows that endpoint\u0026rsquo;s upstream and downstream endpoint dependencies as a directed graph, with the same node metrics, edge sparklines, and pan / zoom / focus conventions as the service map.\nOne difference is inherent to the data: endpoint-relation metrics are recorded by the callee only. Edges therefore carry server-side numbers, and an endpoint with no resolvable metric values in the window is dropped from the graph rather than drawn empty.\nTwo safeguards to know The maps protect you from two failure modes that would otherwise read as \u0026ldquo;the data\u0026rdquo;.\n\u0026ldquo;Topology too large to render\u0026rdquo; A graph that grows past 5,000 services or 15,000 calls cannot be drawn legibly and risks overwhelming the browser, so the map declines to draw a partial picture. Instead it shows a notice with the actual counts and the remedy:\nTopology too large to render — N services · M calls. Pick a specific service above, or lower the depth, to see a complete map.\nThis is almost always the full-layer overview of a large estate. Focus one or a few services, and/or lower the depth, and the map renders. (Inside the embedded overview-widget snapshot, the same notice points you to open the full Topology tab to narrow the scope.)\nPartial metrics Node and edge metrics are fetched from OAP in batches. When some of those batches fail (an OAP hiccup, a backend limit), the map still draws the graph but flags that the gaps are unknown, not zero:\nSome metrics could not be loaded (X of Y batches failed) — blank values may be unavailable, not zero.\nThis matters operationally: a blank ring or an empty traffic number under this banner means \u0026ldquo;we could not read it this time\u0026rdquo;, and you should re-run before concluding a service is idle or down. On the API dependency map the same banner is phrased for its data shape — some endpoints or links may be missing, because an endpoint whose metrics failed to load is dropped rather than drawn empty. Refresh to retry.\nAccess Viewing any of these maps — service map, instance map, deployment, API dependency — requires the topology:read permission, which the built-in viewer role and above hold. See Roles and Permissions.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/operate/service-map/","title":"\u003c!--"},{"body":" Trace Inspect Trace Inspect (/operate/trace-inspect) is the cross-layer trace query tool in the sidebar. The per-layer Traces tab starts from a layer and the service picked in its header; this page starts from nothing. Open it, name any service — or no service at all — and run one query across everything the trace store holds. It is built for the deep-dive that does not begin inside a dashboard: a trace id pasted from a log line or an alarm, a service you only know by name, or a \u0026ldquo;show me every error trace in the last hour, anywhere\u0026rdquo; sweep.\nLike the other triage pages, it owns its own time range — the global topbar time picker does not apply — and nothing is fetched until you press Run query. Every condition is staged: edit as much as you like, then run.\nSources The Source toggle at the top switches between the two trace stores:\nNative — SkyWalking\u0026rsquo;s own trace store, the same store the per-layer Traces tab queries. Zipkin — the Zipkin store behind OAP, with Zipkin\u0026rsquo;s own service universe and query conditions. On the per-layer tab, which store you see is decided by the layer template; here both are always one click away and you choose per query. Switching source clears the previous result so the two never mix.\nTarget — pick it, type it, or leave it blank For native traces the Target is optional: leaving it blank queries every service in the window. When you do want to scope it, there are two modes:\nPick — choose a Layer, then a Service from that layer\u0026rsquo;s catalog, then optionally narrow to one Instance and/or Endpoint. This is the discovery path: the dropdowns show you what exists. Type — enter a Service name directly, with a Real checkbox (leave it on for a normal instrumented service; turn it off for a virtual/peer service such as a database or remote endpoint that only exists as a conjectured node). Instance and Endpoint names are optional free text. Typing needs no layer at all — the name plus the Real flag identify the service. The → edit as text link converts the current Pick selection into the Type form — pick to discover, then tweak the name or flag by hand.\nThe Zipkin target is different because Zipkin has its own service universe (no layers, no SkyWalking ids): a Service field (blank means all services), plus Remote service and Span name narrowing fields whose suggestions load once a service is picked.\nConditions Native conditions:\nCondition What it does Trace ID Paste a known trace id for a direct lookup. Status ALL, SUCCESS, or ERROR. Order Newest (by start time) or Slowest (by duration). Duration (ms) Min–max trace duration bounds, in milliseconds. Tags Comma-separated key=value pairs, AND-joined, with autocomplete (below). Time A rolling preset (15m, 30m, 1h, 3h, 6h, 12h, 24h) or Custom…, which swaps in an absolute start/end pair. The × returns to presets. Limit Result cap: 20, 30 (default), 50, or 100. Zipkin conditions are the store\u0026rsquo;s own: Duration (ms) bounds, an Annotation query (error or key=value terms, AND-joined), plus the shared Time and Limit. There is no Trace ID field on the Zipkin side.\nThe Tags field autocompletes from the tags actually stored in the window: start typing to see known keys, type = to switch the suggestions to that key\u0026rsquo;s known values, and press Enter to commit the pair — the field then primes a comma so you can keep typing the next one. Time windows are evaluated at second precision, same as the per-layer tab, so a trace that just finished still falls inside the window.\nRun query and the resolved query Run query executes the staged conditions and replaces the result area. Next to it, a Resolved query toggle appears after each run: it names the source (and, for native, which trace query API answered) and expands to the exact condition that was sent — the service ids resolved from your picks or typed names, the computed window, and every filled-in default. When a query returns something unexpected, read this panel first: it shows what was actually asked, not what you meant.\nDistribution chart Beside the conditions, a Distribution chart plots one dot per result — start time on the X axis, colored by success/error, with the duration surfaced on hover. Click a dot to open that trace, or drag a rectangle to brush a subset: the list below narrows to the brushed traces and shows an N / total count with a clear control. Brushing filters what is already loaded; it does not re-query.\nResults — segments or whole traces For native traces, a banner above the results states which trace query API this OAP serves: on backends with whole-trace support (Trace Query v2) full traces come back inline; on any other backend (Trace Query v1) each row is a trace segment and clicking one fetches its full trace. This is a property of the storage backend, not a setting — see Traces for the full explanation.\nClicking a row opens the same trace detail the per-layer tab uses: the span waterfall with its Default / Tree / Statistics layouts, per-span detail (meta, tags, logs, cross-trace refs, attached events), and the id / url copy buttons — a copied shareable URL reopens the trace in an overlay for whoever you send it to, on either store. While a trace is open, the result list folds into a collapsible rail on the left so you can step through traces without losing the query. Escape closes the span panel first, then the trace, in that order. Zipkin results render with the Zipkin waterfall and keep their Zipkin span shape.\nHow it differs from the per-layer Traces tab No layer, no header picker. The target is part of the query form, optional, and can be a typed name — including services in layers you never open, or all services at once. Both stores on one page. Native vs. Zipkin is a per-query toggle here; on the layer tab it is fixed by the layer template. Built for id-first triage. Paste a trace id with no service at all and run — the common \u0026ldquo;a log/alarm gave me an id\u0026rdquo; entry point. The waterfall, the distribution chart, the staged Run-query flow, and the v1/v2 behavior are identical to the per-layer tab — this page changes how you scope the query, not how results render.\nPermissions The page and its queries require the inspect:read permission. Opening a trace\u0026rsquo;s waterfall and the tag / Zipkin suggestion lists additionally use traces:read, and the Pick-mode layer/service dropdowns use metrics:read — the bundled roles that grant inspect:read include these. See Roles and Permissions.\nRelated Traces — the per-layer trace explorer, with the full waterfall and condition reference. Log Inspect — the cross-layer sibling for logs, browser errors, and pod tails. Metrics Inspect — the cross-layer view of OAP\u0026rsquo;s metric catalog. ","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/operate/trace-inspect/","title":"\u003c!--"},{"body":" Traces The Traces tab is the distributed-trace explorer inside a layer. You pick a service, set conditions (status, sort, duration, tags, time window), run the query, then click a result to read its span timeline. It surfaces two trace stores — SkyWalking-native traces and Zipkin traces — depending on what the layer is configured for.\nTraces are triage data, so this tab owns its own time range and conditions. It is not driven by the global topbar time picker, and it does not auto-refresh: you set your conditions and press Run query. Nothing is fetched until you do — until then the list shows a \u0026ldquo;Pick your conditions, then click Run query.\u0026rdquo; prompt.\nWhich trace store appears A layer template carries a traces.source setting that decides which trace store the tab queries:\nnative (the default when a layer has no traces block) — only the SkyWalking-native trace explorer. zipkin — only the Zipkin trace explorer. both — two separate sidebar tabs, Trace (native) and Zipkin Trace. Native and Zipkin spans have different shapes and different query conditions, so they are kept as distinct tabs rather than one tab with a toggle. Mesh and Kubernetes-flavored layers commonly land on Zipkin; instrumented-agent layers land on native.\nNative traces The native explorer queries SkyWalking\u0026rsquo;s own trace store. The service is taken from the layer\u0026rsquo;s Service header picker at the top of the page; the in-tab conditions narrow within that service.\nConditions All conditions are staged in the toolbar and only take effect on Run query — editing a field does not refetch on its own.\nCondition What it does Instance Restrict to one service instance. Defaults to All. Resets when you switch service. Endpoint Restrict to one endpoint. Defaults to All. A dropdown of the service\u0026rsquo;s endpoints (capped at 50). Status ALL, SUCCESS, or ERROR — the trace state. Order BY_START_TIME (Newest) or BY_DURATION (Slowest). Limit Cap on result rows: 30 by default. The server caps a single page at 200. The list says when the window held more traces than the limit returned — there is no total on the wire, only \u0026ldquo;there is more\u0026rdquo;. Time range A rolling preset (Last 15 min through Last 24 hours) or a Custom… absolute start/end pair. Trace ID Paste a known trace id to look it up directly. Duration range (ms) Min–max trace duration, in milliseconds. Tag Free-form span tags as key=value (for example http.status_code=500). Press Enter to add; each committed tag shows as an Active-tag chip. Multiple tags are AND-joined. The time window is evaluated at second precision so a trace that just finished still falls inside it — minute rounding would drop the most recent (and usually most interesting) traces during triage.\nRicher vs. universal results, by storage backend What a result row represents depends on the storage backend behind OAP, and Horizon detects this automatically — you do not configure it:\nOn backends that support it, the explorer fetches whole traces with their spans inline. The list shows complete traces, and selecting one renders its waterfall immediately with no second round-trip. A banner reads \u0026ldquo;This OAP serves traces via Trace Query v2 API\u0026rdquo; and \u0026ldquo;Full traces are returned inline.\u0026rdquo; On any other backend, the explorer falls back to the universal basic query, which returns trace segments. Each row is one segment; the full trace is fetched on click. The banner reads \u0026ldquo;Trace Query v1 API\u0026rdquo; and \u0026ldquo;Each row is a trace segment — click one to fetch its full trace.\u0026rdquo; The banner stays visible across both the browse list and the open-trace view, so it is always clear what a row represents. The richer inline view is a property of the storage backend, not a setting — if your rows are segments, the backend does not support whole-trace queries.\nDuration distribution Beside the conditions, a Distribution chart plots one dot per result: the X axis is the trace\u0026rsquo;s start time, and the dot\u0026rsquo;s duration (the Y value) is surfaced on hover. Error traces are drawn in the error color, successful ones in the accent color.\nThe chart is an in-page filter. Click a dot — or drag a rectangle across several — to pick a subset; the result list then narrows to just the picked traces and the header switches to an \u0026ldquo;N picked\u0026rdquo; count with a Reset button. This filters what is already loaded; it does not issue a new query.\nResult list and the trace waterfall Each row in the result list shows the trace\u0026rsquo;s root endpoint, an OK/ERR status flag, the duration, and a bar sized relative to the slowest trace in the set. Click a row to open it.\nSelecting a trace opens the detail view, which offers three layouts:\nDefault — the span waterfall: an indented timeline, one row per span. Each row carries a service-colored bar positioned and sized by the span\u0026rsquo;s start offset and duration, a span-kind glyph, a component icon, the endpoint or peer name, and the span\u0026rsquo;s own duration. Errored spans are highlighted. A flag badge marks spans that carry attached events. Tree — the same spans drawn as a zoomable node graph. Statistics — spans rolled up by name, with count and total / average / maximum duration, sortable per column. Span kinds are grouped into entry (server), exit (client), local, producer, and consumer families, each with its own glyph and color. The waterfall stitches spans across segments using their parent references, so a single trace that spans multiple services renders as one connected timeline.\nClick any span row to open its detail panel:\nMeta — service, instance, endpoint, kind, component, peer, layer, start time, duration, and error flag. Cross-trace refs — when a span references a parent in a different trace, those references are listed with the parent trace id, parent segment, parent span, and ref type. The trace id is a link that opens that other trace. Tags — the span\u0026rsquo;s key/value tags. Logs — per-span log entries with their timestamps. Attached Events — named events on the span with their start/end times and summary key/values. The detail view\u0026rsquo;s header KPIs report the trace\u0026rsquo;s start time, total duration, span count, and the number of distinct services it touched. You can copy the trace id or a shareable URL from there; opening a shared ?traceId= link lands directly on the trace in an overlay.\nZipkin traces When a layer enables Zipkin, the Zipkin tab queries an upstream Zipkin store through OAP. Zipkin organizes data by its own service universe (the localEndpoint.serviceName reported on each span), which can drift from SkyWalking\u0026rsquo;s service list, so this tab carries its own service controls rather than binding to the shell\u0026rsquo;s Service picker.\nConditions Condition What it does Service Free-text service name (with suggestions). Empty means every service. Remote service Narrow to spans calling a given remote service. Requires a service to be picked first. Span name Narrow to one span/operation name. Requires a service to be picked first. Min duration (ms) / Max duration (ms) Duration bounds, entered in milliseconds. Annotations Zipkin annotation query — error or key=value terms, AND-joined. Open trace ID Paste a trace id to open it directly. Limit Result cap: 10, 30, 50, 100, or 200. The list says when the window held more traces than the limit returned. Time range A lookback preset (Last 15 min through Last 24 hours) or a Custom range… absolute window. As with the native tab, conditions are staged and only applied on Run query.\nEach Zipkin result shows its duration and error state, with a duration bar colored fast-to-slow (errored traces are forced to the error color). Selecting a trace renders the Zipkin span waterfall, and a span detail panel exposes the span\u0026rsquo;s duration, kind, and Zipkin tags. Because the two stores have different span formats, there is no field mapping between native and Zipkin results — Zipkin spans keep their Zipkin shape.\nTroubleshooting \u0026ldquo;No traces in window.\u0026rdquo; — the query ran but matched nothing. Widen the time range, relax the Status / Duration / Tag conditions, or confirm the service is actually reporting traces. An unreachable chip on the list — the trace store did not answer, and the reason is printed in a banner above the results. For native traces this points at OAP or its storage backend; for Zipkin it points at the configured Zipkin endpoint. The two stores fail independently — one being down does not blank the other. Run query is greyed out — the tab does not yet know which service to read. It says which: Resolving service… while the picked service is being looked up, or a note that the selected service is not in this layer (it aged out of OAP, was renamed, or the link points elsewhere) — pick another one. Traces are always read for one service, so the tab waits instead of querying the whole layer. Rows are segments, not whole traces — that is expected on storage backends without whole-trace support; the banner says so. Click a segment to fetch its full trace. A pasted trace id from a log row won\u0026rsquo;t resolve — older traces can sit outside the default lookup window or in a cold storage tier. Open the trace from the log row (which carries its timestamp) rather than pasting the id cold, so the lookup is widened around the right time. No data even with a valid service — double-check the time range first; this tab does not follow the global topbar, so the window is whatever the tab\u0026rsquo;s own Time range control says. Related Trace Inspect — the cross-layer trace query tool: look up a trace by id or query any service (picked, typed by name, or all of them) without entering a layer. 3D Infrastructure Map — topology-level view of the same services these traces flow through. Metrics Inspect — confirm which metrics a service is reporting when traces look incomplete. Layer Dashboard Templates — where a layer\u0026rsquo;s traces.source is configured. ","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/operate/traces/","title":"\u003c!--"},{"body":" ActiveMQ The ACTIVEMQ layer monitors Apache ActiveMQ message brokers. SkyWalking collects ActiveMQ metrics over OpenTelemetry and rolls them up into cluster-, broker-, and destination-scope metrics, so operators can watch queue depth, throughput, connection counts, and broker JVM health alongside the rest of their estate. See the upstream ActiveMQ monitoring setup for how to wire the telemetry pipeline.\nIn Horizon\u0026rsquo;s sidebar this layer is named ActiveMQ. Its services are listed as ActiveMQ clusters, its instances as Brokers, and its endpoints as Destinations (the queues and topics a broker serves). The ACTIVEMQ layer enables the Service, Instance, and Endpoint sub-tabs; it has no topology, traces, or logs view.\nThis page is the operator reference for the bundled ACTIVEMQ dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled ACTIVEMQ template; if an operator has published a customized ACTIVEMQ template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every ActiveMQ cluster with four sortable columns, sorted by Enqueue/s by default:\nEnqueue/s — messages enqueued per second across the cluster (meter_activemq_cluster_enqueue_rate).\nDequeue/s — messages dequeued per second across the cluster (meter_activemq_cluster_dequeue_rate).\nSystem Load — the cluster\u0026rsquo;s average system load (meter_activemq_cluster_system_load_average/10000).\nThreads — the cluster\u0026rsquo;s average thread count (meter_activemq_cluster_thread_count).\nService dashboard The primary drill-down for one selected ActiveMQ cluster, mixing throughput, message timing, and broker JVM health.\nSystem Load Average — the cluster\u0026rsquo;s system load over time (meter_activemq_cluster_system_load_average/10000).\nThread Count — live JVM threads across the cluster (meter_activemq_cluster_thread_count).\nHeap Used (MB) — JVM heap memory in use, in MB (meter_activemq_cluster_heap_memory_usage_used/1024/1024).\nHeap Max (MB) — the configured maximum heap across the cluster, summed and shown as the latest value in MB (latest(aggregate_labels(meter_activemq_cluster_heap_memory_usage_max,sum))/1024/1024).\nEnqueue / Dequeue / Dispatch /s — the three core message rates on one chart: messages enqueued, dequeued, and dispatched per second (meter_activemq_cluster_enqueue_rate, meter_activemq_cluster_dequeue_rate, meter_activemq_cluster_dispatch_rate).\nExpired /s — messages that expired before delivery, per second (meter_activemq_cluster_expired_rate).\nEnqueue Time — average and maximum time a message spends being enqueued, in seconds (meter_activemq_cluster_average_enqueue_time/1000, meter_activemq_cluster_max_enqueue_time/1000).\nGC Counts (G1+Parallel) — old- and young-generation garbage-collection counts, each combining the G1 and Parallel collectors so the chart reads correctly regardless of which collector the broker JVM uses (view_as_seq(meter_activemq_cluster_gc_g1_old_collection_count, meter_activemq_cluster_gc_parallel_old_collection_count), and the matching young-collection counters).\nGC Time (ms) — old- and young-generation GC time in ms, again combining the G1 and Parallel collectors (view_as_seq(meter_activemq_cluster_gc_g1_old_collection_time, meter_activemq_cluster_gc_parallel_old_collection_time), and the matching young-collection timers).\nInstance dashboard For one selected broker. A row of single-value cards summarizes the broker\u0026rsquo;s current state, followed by trend charts.\nSummary cards\nConnections — current TCP/JMS connections to this broker (latest(meter_activemq_broker_current_connections)).\nProducer Count — active producer sessions on this broker, summed across destinations (latest(aggregate_labels(meter_activemq_broker_current_producer_count,sum))).\nConsumer Count — active consumer sessions on this broker, summed across destinations (latest(aggregate_labels(meter_activemq_broker_current_consumer_count,sum))).\nUptime — broker uptime in hours since its last restart (latest(meter_activemq_broker_uptime)/1000/60/60).\nTrends\nConnections (trend) — the broker\u0026rsquo;s connection count over time (meter_activemq_broker_current_connections).\nEnqueue / Dequeue Count — per-minute enqueue and dequeue totals summed across the broker\u0026rsquo;s destinations (aggregate_labels(meter_activemq_broker_enqueue_count,sum), aggregate_labels(meter_activemq_broker_dequeue_count,sum)).\nProducer / Consumer Increase — new producer and consumer sessions opened per minute (aggregate_labels(meter_activemq_broker_producer_count,sum), aggregate_labels(meter_activemq_broker_consumer_count,sum)).\nMemory Usage — aggregate memory usage across destinations, in MB (aggregate_labels(meter_activemq_broker_memory_usage,sum)/1024/1024).\nMemory Limit — the configured memory ceiling across destinations, in GB (aggregate_labels(meter_activemq_broker_memory_limit,sum)/1024/1024/1024).\nAvg Message Size — average message size across destinations, in bytes (aggregate_labels(meter_activemq_broker_average_message_size,avg)).\nEndpoint dashboard For one selected destination (a queue or topic).\nProducer Count — producers attached to this destination (meter_activemq_destination_producer_count).\nConsumer Count — consumers attached to this destination (meter_activemq_destination_consumer_count).\nQueue Size — messages currently held in the destination (meter_activemq_destination_queue_size).\nMemory Usage (MB) — memory the destination is consuming, in MB (meter_activemq_destination_memory_usage/1024/1024).\nMessage Counts — the destination\u0026rsquo;s message lifecycle on one chart: enqueued, dequeued, dispatched, expired, and in-flight counts (meter_activemq_destination_enqueue_count, meter_activemq_destination_dequeue_count, meter_activemq_destination_dispatch_count, meter_activemq_destination_expired_count, meter_activemq_destination_inflight_count).\nEnqueue Time (s) — average and maximum enqueue time for the destination, in seconds (meter_activemq_destination_average_enqueue_time/1000, meter_activemq_destination_max_enqueue_time/1000).\nRequirements The ACTIVEMQ dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the ActiveMQ meter families, produced from broker telemetry collected over OpenTelemetry:\nCluster metrics — the meter_activemq_cluster_* family (enqueue / dequeue / dispatch / expired rates, enqueue time, system load, thread count, heap usage, and the G1 / Parallel GC counters) for the service list and the cluster dashboard.\nBroker metrics — the meter_activemq_broker_* family (current connections, producer / consumer counts, uptime, enqueue / dequeue counts, memory usage and limit, average message size) for the broker dashboard.\nDestination metrics — the meter_activemq_destination_* family (producer / consumer count, queue size, memory usage, the enqueue / dequeue / dispatch / expired / in-flight counts, and enqueue time) for the destination dashboard.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a broker- or destination-scope metric is empty until that level of data is reported. See the upstream ActiveMQ monitoring setup for the collection pipeline that produces these families.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/activemq/","title":"\u003c!--"},{"body":" Airflow The AIRFLOW layer monitors Apache Airflow workflow schedulers. OAP collects Airflow\u0026rsquo;s OpenTelemetry metrics and presents each monitored Airflow deployment as a cluster, so this layer is where you watch scheduler health, DAG parsing, executor and pool capacity, and triggerer activity.\nIn Horizon\u0026rsquo;s sidebar this layer is named Airflow, grouped under Workflow Scheduler. Its services are listed as Airflow clusters and the components that report into each cluster (the scheduler and triggerer processes) are listed as Components. The AIRFLOW layer enables only the Service and Instance scopes — there is no endpoint scope, no topology, and no traces or logs tab, because Airflow\u0026rsquo;s telemetry is scheduler-level meter data rather than request traffic.\nThis page is the operator reference for the bundled AIRFLOW dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled AIRFLOW template; if an operator has published a customized AIRFLOW template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every Airflow cluster with four sortable columns, sorted by DAG Bag Size by default:\nDAG Bag Size — number of DAGs found in the last scheduler scan (meter_airflow_dagbag_size).\nDAG Total Parse Time — seconds spent scanning and importing the queued DAG files (meter_airflow_dag_total_parse_time).\nExecutor Open Slots — free executor slots available to run tasks (meter_airflow_executor_open_slots).\nScheduled Slots — pool slots scheduled but not yet running, summed across pools (aggregate_labels(meter_airflow_pool_scheduled_slots, sum)).\nService dashboard The primary drill-down for one selected Airflow cluster. It opens with four single-value cards reporting the current scheduler and executor state, then a set of time-series charts tracking executor capacity and DAG-processing health.\nCards (current value)\nTasks Executable — tasks ready for execution across the cluster (latest(meter_airflow_scheduler_tasks_executable)).\nRunning Tasks — tasks currently running on the executor (latest(meter_airflow_executor_running_tasks)).\nScheduled Slots — pool slots scheduled but not yet running, aggregated across pools (latest(aggregate_labels(meter_airflow_pool_scheduled_slots, sum))).\nQueued Tasks — tasks waiting on the executor (latest(meter_airflow_executor_queued_tasks)).\nCharts (over time)\nExecutor Open Slots — free executor slots over the window (meter_airflow_executor_open_slots).\nDAG File Queue Size — DAG files pending a scan (meter_airflow_dag_file_queue_size).\nDAG Import Errors — DAG files that failed to parse (meter_airflow_dag_import_errors).\nDAG Bag Size — DAGs found in the last scheduler scan (meter_airflow_dagbag_size).\nDAG Total Parse Time — seconds to scan and import the queued DAG files (meter_airflow_dag_total_parse_time).\nDAG File Refresh Errors — DAG file load failures per minute (meter_airflow_dag_file_refresh_error).\nAsset Updates — asset update events per minute (meter_airflow_asset_updates).\nInstance dashboard For one selected Component of the cluster. Airflow reports different meters from different processes — the scheduler emits pool, executor, and asset metrics, while the triggerer emits trigger metrics — so each widget appears only when that component actually reports the metric behind it. A scheduler component therefore shows the pool / executor / heartbeat widgets, a triggerer component shows the triggerer widgets, and neither is shown empty for a component that does not emit it.\nScheduler component\nPool Open / Deferred / Running Slots — pool capacity on the scheduler, plotted as three series: open, deferred, and running slots (meter_airflow_instance_pool_open_slots, meter_airflow_instance_pool_deferred_slots, meter_airflow_instance_pool_running_slots).\nRunning Tasks / Scheduled Slots — executor running-task count against pool slots waiting to run (meter_airflow_instance_executor_running_tasks, meter_airflow_instance_pool_scheduled_slots).\nScheduler Heartbeat — scheduler heartbeats per minute (meter_airflow_instance_scheduler_heartbeat).\nExecutor Open / Queued Slots — executor capacity and queue depth on the scheduler (meter_airflow_instance_executor_open_slots, meter_airflow_instance_executor_queued_tasks).\nAsset Updates — asset update events on the scheduler, per minute (meter_airflow_instance_asset_updates).\nAsset Triggered DagRuns — DagRuns triggered by asset events on the scheduler, per minute (meter_airflow_instance_asset_triggered_dagruns).\nTriggerer component\nTriggerer Heartbeat — triggerer process heartbeats per minute (meter_airflow_instance_triggerer_heartbeat).\nTriggers Running / Capacity Left — live deferrable-trigger load on the triggerer: triggers running against capacity left (meter_airflow_instance_triggers_running, meter_airflow_instance_triggerer_capacity_left).\nTriggers Blocked / Failed / Succeeded — deferred-trigger outcomes on the triggerer host, per minute (meter_airflow_instance_triggers_blocked_main_thread, meter_airflow_instance_triggers_failed, meter_airflow_instance_triggers_succeeded).\nRequirements The AIRFLOW dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nCluster (service-scope) Airflow meters — the meter_airflow_* family (DAG bag size and parse time, DAG-processing queue and import / refresh errors, executor open / queued slots and running / executable tasks, pool scheduled slots, asset updates), aggregated per Airflow cluster.\nComponent (instance-scope) Airflow meters — the meter_airflow_instance_* family (per-component pool, executor, scheduler-heartbeat, asset, and triggerer metrics), reported by each scheduler or triggerer process.\nThese meters are produced by OAP from Airflow\u0026rsquo;s OpenTelemetry metric export — see Airflow monitoring for how to wire Airflow up to OAP. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so an instance-scope component metric is empty until that component reports it. When a component does not emit a family — a triggerer that reports no scheduler pool metrics, or a scheduler that reports no triggerer metrics — those widgets are hidden rather than shown empty.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/airflow/","title":"\u003c!--"},{"body":" Alipay Mini Program The ALIPAY_MINI_PROGRAM layer holds front-end real-user monitoring data reported from Alipay (支付宝) mini-programs. The mini-program monitoring agent feeds OAP launch, render, request, and error metrics from inside the Alipay container, and those land here.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Mobile. Its services are listed as Mini-programs, instances as Versions (each version of a published mini-program), and endpoints as Pages. The ALIPAY_MINI_PROGRAM layer enables the Service, Instance (Version), and Endpoint (Page) dashboards along with the Traces and Logs sub-tabs. It does not ship a topology / service-map view — mini-program RUM data has no call graph to draw.\nThis page is the operator reference for the bundled ALIPAY_MINI_PROGRAM dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled ALIPAY_MINI_PROGRAM template; if an operator has published a customized template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a mini-program, the layer landing page lists every Mini-program with four sortable columns, sorted by traffic (Request RPM) by default:\nRequest RPM — requests per minute (meter_alipay_mp_request_cpm).\nLaunch — average app-launch duration in ms (meter_alipay_mp_app_launch_duration).\nFirst Render — average first-render duration in ms (meter_alipay_mp_first_render_duration).\nErrors — error count over the window (meter_alipay_mp_error_count).\nService dashboard The primary drill-down for one selected mini-program.\nApp Launch Duration — the approximate cold-launch duration measured from the Alipay container, in ms (meter_alipay_mp_app_launch_duration).\nFirst Render Duration — time to first render, in ms (meter_alipay_mp_first_render_duration).\nError Count — number of reported front-end errors (meter_alipay_mp_error_count).\nRequest Load — requests per minute for the mini-program (meter_alipay_mp_request_cpm).\nRequest Duration Percentile — p50 / p75 / p90 / p95 / p99 of request duration, the tail of the request-time distribution, in ms (meter_alipay_mp_request_duration_percentile).\nInstance dashboard For one selected Version of the mini-program.\nLaunch Duration — app-launch duration for this version, in ms (meter_alipay_mp_instance_app_launch_duration).\nFirst Render Duration — first-render duration for this version, in ms (meter_alipay_mp_instance_first_render_duration).\nRequest Load — requests per minute for this version (meter_alipay_mp_instance_request_cpm).\nRequest Duration Percentile — p50 / p75 / p90 / p95 / p99 of request duration for this version, in ms (meter_alipay_mp_instance_request_duration_percentile).\nEndpoint dashboard For one selected Page.\nLaunch Duration on Page — app-launch duration attributed to this page, in ms (meter_alipay_mp_endpoint_app_launch_duration).\nFirst Render Duration — first-render duration for this page, in ms (meter_alipay_mp_endpoint_first_render_duration).\nRequest Load — requests per minute for this page (meter_alipay_mp_endpoint_request_cpm).\nRequest Duration Percentile — p50 / p75 / p90 / p95 / p99 of request duration for this page, in ms (meter_alipay_mp_endpoint_request_duration_percentile).\nRequirements The ALIPAY_MINI_PROGRAM dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the Alipay mini-program meter families, reported by the Alipay mini-program monitoring agent:\nMini-program (service) metrics — the meter_alipay_mp_* family at service scope: request load (meter_alipay_mp_request_cpm), launch and first-render duration (meter_alipay_mp_app_launch_duration, meter_alipay_mp_first_render_duration), error count (meter_alipay_mp_error_count), and the request-duration percentiles (meter_alipay_mp_request_duration_percentile).\nVersion (instance) metrics — the meter_alipay_mp_instance_* family for the per-version widgets (launch, first render, request load, request percentile).\nPage (endpoint) metrics — the meter_alipay_mp_endpoint_* family for the per-page widgets (launch, first render, request load, request percentile).\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a version- or page-scope metric is empty until that level of data is reported. For how to enable the upstream collector, see the Alipay Mini-Program monitoring setup docs.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/alipay_mini_program/","title":"\u003c!--"},{"body":" APISIX The APISIX layer monitors Apache APISIX API gateways. APISIX exposes its built-in Prometheus metrics, OpenTelemetry collects and forwards them to OAP, and OAP aggregates them into the meter_apisix_* families this dashboard renders. The layer key is APISIX, and in Horizon\u0026rsquo;s sidebar it is grouped under Gateways.\nIn the sidebar this layer\u0026rsquo;s services are listed as APISIX services, its instances as Nodes (the individual APISIX data-plane nodes), and its endpoints as Routes (the matched APISIX routes). The APISIX layer enables three scopes — Service, Instance (Node), and Endpoint (Route). It does not ship a topology, traces, or logs tab, so this dashboard is purely the metric drill-down across those three scopes.\nThis page is the operator reference for the bundled APISIX dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled APISIX template; if an operator has published a customized APISIX template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a service, the layer landing page lists every APISIX service, sorted by request rate (RPS) by default, with four columns:\nRPS — total HTTP requests per second across the service\u0026rsquo;s nodes (meter_apisix_sv_http_requests).\n200/s — 200-status responses per second (meter_apisix_sv_http_status_matched{code='200'}).\n404/s — 404-status responses per second (meter_apisix_sv_http_status_matched{code='404'}).\n503/s — 503-status responses per second (meter_apisix_sv_http_status_matched{code='503'}).\nThe three status columns give an at-a-glance health read across the fleet — a service with a climbing 503/s next to its 200/s is shedding load.\nService dashboard The primary drill-down for one selected APISIX service. All eight widgets aggregate across the service\u0026rsquo;s nodes.\nHTTP Request Trend — total requests per second for the service (meter_apisix_sv_http_requests).\nHTTP Status Trend — requests per second broken down by HTTP status code (meter_apisix_sv_http_status_matched, one line per code).\nHTTP Latency — request latency in ms, split by latency type and percentile (meter_apisix_sv_http_latency_matched).\nHTTP Bandwidth — ingress / egress bandwidth in KB, by type (meter_apisix_sv_bandwidth_matched, divided to KB).\nHTTP Connections — active connections by state — active, reading, writing, waiting (meter_apisix_sv_http_connections, one line per state).\nNon-matched Status Trend — requests per second by status code for traffic that hit no matching APISIX route (meter_apisix_sv_http_status_unmatched). Unmatched traffic is usually a misconfigured client or a probe; a rising line here is worth investigating.\nNon-matched Latency — latency in ms for the same no-matching-route traffic (meter_apisix_sv_http_latency_unmatched).\nNon-matched Bandwidth — bandwidth in KB for the same no-matching-route traffic (meter_apisix_sv_bandwidth_unmatched, divided to KB).\nInstance dashboard For one selected node. These widgets are reported per node, so they show how one APISIX data-plane process is behaving.\nHTTP Request Trend — requests per second for the node (meter_apisix_instance_http_requests).\nHTTP Status Trend — requests per second by status code for the node (meter_apisix_instance_http_status_matched).\nHTTP Latency — request latency in ms for the node (meter_apisix_instance_http_latency_matched).\nHTTP Bandwidth — bandwidth in KB for the node (meter_apisix_instance_bandwidth_matched, divided to KB).\nHTTP Connections — connections by state for the node (meter_apisix_instance_http_connections).\nShared Dict — the node\u0026rsquo;s shared-memory dictionary capacity vs. free space in MB (meter_apisix_instance_shared_dict_capacity_bytes and meter_apisix_instance_shared_dict_free_space_bytes, divided to MB, labelled capacity / free). When free space approaches zero the node can no longer cache new entries.\netcd — the node\u0026rsquo;s view of the control-plane etcd: the latest known etcd index and whether etcd is reachable (meter_apisix_instance_etcd_indexes and latest(meter_apisix_instance_etcd_reachable), labelled indexes / reachable). A node that can\u0026rsquo;t reach etcd is no longer receiving config updates.\nNon-matched Traffic — a combined view of no-matching-route activity for the node: status, latency in ms, and bandwidth in KB on one chart (meter_apisix_instance_http_status_unmatched, meter_apisix_instance_http_latency_unmatched, and meter_apisix_instance_bandwidth_unmatched divided to KB).\nEndpoint dashboard For one selected route. APISIX reports a tighter metric set at route scope — status, latency, and bandwidth.\nHTTP Status Trend — requests per second by status code for the route (meter_apisix_endpoint_http_status, one line per code).\nHTTP Latency — request latency in ms for the route, by type and percentile (meter_apisix_endpoint_http_latency).\nHTTP Bandwidth — bandwidth in KB for the route, by type (meter_apisix_endpoint_bandwidth, divided to KB).\nRequirements The APISIX dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs APISIX metrics flowing in through the OpenTelemetry receiver:\nService metrics — the meter_apisix_sv_* family (requests, status, latency, bandwidth, connections, and the unmatched-route counterparts), aggregated by OAP across a service\u0026rsquo;s nodes.\nInstance (node) metrics — the meter_apisix_instance_* family, including the node-only meter_apisix_instance_shared_dict_* and meter_apisix_instance_etcd_* health metrics.\nEndpoint (route) metrics — the meter_apisix_endpoint_* family for the per-route status, latency, and bandwidth widgets.\nThese come from APISIX\u0026rsquo;s Prometheus plugin scraped by an OpenTelemetry Collector and forwarded to OAP, which converts them into the meter_apisix_* metrics through its meter-analysis rules. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a route-scope or node-scope metric is empty until that level of data is reported. For the end-to-end collector and OAP setup, see the APISIX monitoring backend guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/apisix/","title":"\u003c!--"},{"body":" AWS DynamoDB The AWS_DYNAMODB layer monitors Amazon DynamoDB through CloudWatch metrics that OAP pulls in and aggregates. It is an agentless layer — there is no SkyWalking agent inside DynamoDB — so the dashboard is a read-only view of the throttling, error, capacity, and latency metrics CloudWatch exposes for your DynamoDB usage.\nIn Horizon\u0026rsquo;s sidebar this layer is named AWS DynamoDB. A service here represents one DynamoDB account, so services are listed as DynamoDB accounts; each account\u0026rsquo;s endpoints are its Tables. The layer enables only two scopes — the account-level Service dashboard and the per-table Endpoint dashboard. It has no instance scope, no topology / map, and no traces or logs tabs.\nThis page is the operator reference for the bundled AWS_DYNAMODB dashboard: what you see at the account level and per table, and what each widget means.\nThe widgets and metrics below are read from the bundled AWS_DYNAMODB template; if an operator has published a customized AWS_DYNAMODB template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening an account, the layer landing page lists every DynamoDB account with four sortable columns, sorted by Read Throttled by default. All four are window sums (aggregation: \u0026quot;sum\u0026quot;), so they surface the accounts taking the most throttling and system-error pressure over the selected range:\nRead Throttled — throttled read requests across the account (aws_dynamodb_read_throttled_requests).\nWrite Throttled — throttled write requests across the account (aws_dynamodb_write_throttled_requests).\nRead Sys Err — read requests that failed with a DynamoDB system error (aws_dynamodb_read_system_errors).\nWrite Sys Err — write requests that failed with a DynamoDB system error (aws_dynamodb_write_system_errors).\nService dashboard The account-level drill-down for one selected DynamoDB account. All widgets are time-series lines over the selected window.\nThrottled Requests — throttled read vs write requests for the account (aws_dynamodb_read_throttled_requests, aws_dynamodb_write_throttled_requests).\nThrottle Events — throttle events on read vs write, counted independently of throttled request volume (aws_dynamodb_read_throttle_events, aws_dynamodb_write_throttle_events).\nSystem Errors — read vs write requests that hit a DynamoDB-side system error (aws_dynamodb_read_system_errors, aws_dynamodb_write_system_errors).\nUser Errors — requests rejected for a client-side / user error such as a bad request (aws_dynamodb_user_errors).\nConditional Check Failed — write requests rejected because a conditional expression evaluated to false (aws_dynamodb_conditional_check_failed_requests).\nTransaction Conflict — transactional requests rejected due to a conflict with another in-flight transaction (aws_dynamodb_transaction_conflict).\nRead Capacity (unit/s) — provisioned read capacity vs consumed write capacity for the account (as the bundled template plots them), in capacity units per second (aws_dynamodb_provisioned_read_capacity_units, aws_dynamodb_consumed_write_capacity_units).\nWrite Capacity (unit/s) — provisioned vs consumed write capacity for the account, in capacity units per second (aws_dynamodb_provisioned_write_capacity_units, aws_dynamodb_consumed_write_capacity_units).\nSuccessful Request Latency (ms) — latency of successful operations, broken out by operation type — get / put / query / scan — in ms (aws_dynamodb_get_successful_request_latency, aws_dynamodb_put_successful_request_latency, aws_dynamodb_query_successful_request_latency, aws_dynamodb_scan_successful_request_latency).\nTTL Deleted Items — items removed by DynamoDB\u0026rsquo;s time-to-live expiry process (aws_dynamodb_time_to_live_deleted_item_count).\nScan Returned Items — items returned by Scan operations (aws_dynamodb_scan_returned_item_count).\nQuery Returned Items — items returned by Query operations (aws_dynamodb_query_returned_item_count).\nAccount Max Reads / Writes — the account-level capacity ceilings CloudWatch reports: table-level read / write maxima and account-wide read / write maxima (aws_dynamodb_account_max_table_level_reads, aws_dynamodb_account_max_table_level_writes, aws_dynamodb_account_max_reads, aws_dynamodb_account_max_writes).\nAccount Capacity Utilization — provisioned read vs write capacity utilization for the account, in percent (aws_dynamodb_account_provisioned_read_capacity_utilization, aws_dynamodb_account_provisioned_write_capacity_utilization).\nEndpoint dashboard For one selected Table under the account. These are the per-table counterparts of the account-level widgets, evaluated at endpoint scope (aws_dynamodb_endpoint_*), so you can see which individual table is responsible for the account\u0026rsquo;s throttling, errors, or capacity draw.\nThrottled Requests — throttled read vs write requests against the table (aws_dynamodb_endpoint_read_throttled_requests, aws_dynamodb_endpoint_write_throttled_requests).\nThrottle Events — read vs write throttle events on the table (aws_dynamodb_endpoint_read_throttle_events, aws_dynamodb_endpoint_write_throttle_events).\nSystem Errors — read vs write DynamoDB system errors on the table (aws_dynamodb_endpoint_read_system_errors, aws_dynamodb_endpoint_write_system_errors).\nConditional Check Failed — write requests on the table rejected by a failed conditional expression (aws_dynamodb_endpoint_conditional_check_failed_requests).\nTransaction Conflict — transactional requests on the table rejected due to a conflict (aws_dynamodb_endpoint_transaction_conflict).\nTTL Deleted Items — items removed from the table by time-to-live expiry (aws_dynamodb_endpoint_time_to_live_deleted_item_count).\nRead Capacity — provisioned vs consumed read capacity for the table (aws_dynamodb_endpoint_provisioned_read_capacity_units, aws_dynamodb_endpoint_consumed_read_capacity_units).\nWrite Capacity — provisioned vs consumed write capacity for the table (aws_dynamodb_endpoint_provisioned_write_capacity_units, aws_dynamodb_endpoint_consumed_write_capacity_units).\nSuccessful Request Latency (ms) — latency of successful operations on the table by operation type — get / put / query / scan — in ms (aws_dynamodb_endpoint_get_successful_request_latency, aws_dynamodb_endpoint_put_successful_request_latency, aws_dynamodb_endpoint_query_successful_request_latency, aws_dynamodb_endpoint_scan_successful_request_latency).\nScan Returned Items — items returned by Scan operations on the table (aws_dynamodb_endpoint_scan_returned_item_count).\nQuery Returned Items — items returned by Query operations on the table (aws_dynamodb_endpoint_query_returned_item_count).\nRequirements The AWS_DYNAMODB dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the DynamoDB metric families collected from CloudWatch and aggregated under this layer:\nAccount (service) metrics — the aws_dynamodb_* family: throttled requests and throttle events, read / write system errors, user errors, conditional-check failures, transaction conflicts, provisioned vs consumed read / write capacity, per-operation successful request latency (get / put / query / scan), TTL-deleted items, scan / query returned items, the account-level max read / write ceilings, and provisioned capacity utilization.\nTable (endpoint) metrics — the matching aws_dynamodb_endpoint_* family for the same throttling, error, capacity, latency, and returned-item metrics resolved per table.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the table-scope aws_dynamodb_endpoint_* metrics are empty until per-table data is reported, independently of the account-scope metrics. For how to collect these metrics into OAP, see the DynamoDB monitoring setup in the SkyWalking backend docs.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/aws_dynamodb/","title":"\u003c!--"},{"body":" AWS EKS The AWS_EKS layer monitors Amazon Elastic Kubernetes Service (EKS) clusters. SkyWalking ingests EKS observability data through OpenTelemetry — Container Insights / CloudWatch metrics scraped into OAP — and reshapes it into cluster, node, and pod metrics. It groups under AWS in the sidebar.\nIn Horizon\u0026rsquo;s sidebar this layer\u0026rsquo;s entities are renamed to fit the EKS model: services are listed as Clusters, instances as Nodes, and endpoints as EKS services (the Kubernetes services running inside the cluster). The AWS_EKS layer enables the Service (Cluster), Instance (Node), and Endpoint (EKS service) scopes; it does not enable a topology, traces, or logs tab — EKS reports metric data only.\nThis page is the operator reference for the bundled AWS_EKS dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled AWS_EKS template; if an operator has published a customized AWS_EKS template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nCluster list Before opening a cluster, the layer landing page lists every EKS cluster with four sortable columns, sorted by Nodes by default. Each column shows the latest reading averaged across the window:\nNodes — number of nodes in the cluster (latest(eks_cluster_node_count)).\nFailed Nodes — nodes currently in a failed state (latest(eks_cluster_failed_node_count)).\nNamespaces — Kubernetes namespaces in the cluster (latest(eks_cluster_namespace_count)).\nServices — Kubernetes services in the cluster (latest(eks_cluster_service_count)).\nCluster dashboard The primary drill-down for one selected cluster (a Service in OAP terms).\nNode Count — number of nodes over time (eks_cluster_node_count).\nFailed Nodes — nodes in a failed state over time (eks_cluster_failed_node_count).\nNamespace Count — Kubernetes namespaces in the cluster (eks_cluster_namespace_count).\nEKS Service Count — Kubernetes services in the cluster (eks_cluster_service_count).\nCluster Network Errors — cluster-wide receive and transmit error counts, plotted as two series, rx (eks_cluster_net_rx_error) and tx (eks_cluster_net_tx_error).\nCluster Network Drops — cluster-wide dropped packets on receive and transmit, rx (eks_cluster_net_rx_dropped) and tx (eks_cluster_net_tx_dropped).\nNode dashboard For one selected node (an Instance in OAP terms).\nPod Count — pods scheduled on the node (eks_cluster_node_pod_number).\nCPU Utilization (%) — node CPU utilization (eks_cluster_node_cpu_utilization).\nMemory Utilization (%) — node memory utilization (eks_cluster_node_memory_utilization).\nFS Utilization (%) — node filesystem utilization (eks_cluster_node_fs_utilization).\nNetwork RX (KB/s) — node receive throughput in KB/s (eks_cluster_node_net_rx_bytes/1024) on the left axis, with receive errors (eks_cluster_node_net_rx_error) on a second axis so the error count doesn\u0026rsquo;t get lost against the byte scale.\nNetwork TX (KB/s) — node transmit throughput in KB/s (eks_cluster_node_net_tx_bytes/1024) on the left axis, with transmit errors (eks_cluster_node_net_tx_error) on a second axis.\nDisk IO (B/s) — node disk read and write throughput in bytes/s, plotted as read (eks_cluster_node_disk_io_read) and write (eks_cluster_node_disk_io_write).\nPod CPU on Node — aggregate CPU utilization of the pods running on this node (eks_cluster_node_pod_cpu_utilization).\nPod Memory on Node — aggregate memory utilization of the pods running on this node (eks_cluster_node_pod_memory_utilization).\nEKS service dashboard For one selected EKS service (an Endpoint in OAP terms) — a Kubernetes service running inside the cluster, with its pod-level resource and network metrics.\nPod CPU Utilization (%) — CPU utilization across the service\u0026rsquo;s pods (eks_cluster_service_pod_cpu_utilization).\nPod Memory Utilization (%) — memory utilization across the service\u0026rsquo;s pods (eks_cluster_service_pod_memory_utilization).\nPod Network RX (KB/s) — pod receive throughput in KB/s (eks_cluster_service_pod_net_rx_bytes/1024).\nPod RX Errors / s — pod receive error rate (eks_cluster_service_pod_net_rx_error).\nPod Network TX (KB/s) — pod transmit throughput in KB/s (eks_cluster_service_pod_net_tx_bytes/1024).\nPod TX Errors / s — pod transmit error rate (eks_cluster_service_pod_net_tx_error).\nRequirements The AWS_EKS dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the EKS observability metric families, fed in through the OpenTelemetry receiver from Amazon CloudWatch / Container Insights:\nCluster metrics — the eks_cluster_* family at cluster scope: node / failed-node / namespace / service counts and cluster-wide network error and drop counters.\nNode metrics — the eks_cluster_node_* family at node scope: pod count, CPU / memory / filesystem utilization, network receive / transmit bytes and errors, disk read / write IO, and the per-node aggregate pod CPU / memory utilization.\nEKS service metrics — the eks_cluster_service_pod_* family at EKS-service scope: per-service pod CPU / memory utilization and pod network receive / transmit bytes and errors.\nEach metric is queried at its own OAP scope (Cluster / Node / EKS service); OAP does not roll a metric up across scopes, so a node- or service-scope metric stays empty until that level of data is reported. For how to stand up the EKS-to-OAP pipeline, see the layer\u0026rsquo;s setup documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/aws_eks/","title":"\u003c!--"},{"body":" AWS API Gateway The AWS_GATEWAY layer monitors Amazon API Gateway. OAP pulls per-gateway and per-route metrics from AWS CloudWatch — request counts, latency, error rates, cache behavior, and data volume — and presents each gateway as a service in SkyWalking.\nIn Horizon\u0026rsquo;s sidebar this layer is named AWS API Gateway. Its services are listed as AWS Gateways and its endpoints as Routes (each route is a method-plus-resource path on a gateway). The layer enables only the Service and Endpoint sub-tabs — there is no instance scope, no topology, and no traces or logs tab, because CloudWatch reports gateway- and route-level aggregates rather than per-instance, per-request, or relationship data.\nThis page is the operator reference for the bundled AWS_GATEWAY dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled AWS_GATEWAY template; if an operator has published a customized AWS_GATEWAY template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a gateway, the layer landing page lists every AWS Gateway with four sortable columns, sorted by request volume (Requests) by default:\nRequests — total request count over the window (aws_gateway_service_count).\nLatency — average request latency in ms (aws_gateway_service_latency).\n4xx — count of client-error (4xx) responses (aws_gateway_service_4xx).\n5xx — count of server-error (5xx) responses (aws_gateway_service_5xx).\nService dashboard The primary drill-down for one selected gateway.\nRequest Count — total requests handled by the gateway (aws_gateway_service_count).\n4xx Count — client-error responses (aws_gateway_service_4xx).\n5xx Count — server-error responses (aws_gateway_service_5xx).\nRequest Avg Latency — average end-to-end request latency in ms (aws_gateway_service_latency).\nIntegration Avg Latency — average latency between the gateway and its backend integration in ms, isolating backend time from gateway overhead (aws_gateway_service_integration_latency).\nData Processed (HTTP API only) — bytes processed by the gateway, shown in KB (aws_gateway_service_data_processed/1024). Populated only for HTTP API gateways.\nCache Hit Rate (REST API only) — percent of requests served from the gateway cache (aws_gateway_service_cache_hit_rate). Populated only for REST API gateways with caching enabled.\nCache Miss Rate (REST API only) — percent of requests that missed the gateway cache (aws_gateway_service_cache_miss_rate). Populated only for REST API gateways with caching enabled.\nEndpoint dashboard For one selected route (an endpoint under a gateway).\nRequest Count — total requests to the route (aws_gateway_endpoint_count).\n4xx Count — client-error responses on the route (aws_gateway_endpoint_4xx).\n5xx Count — server-error responses on the route (aws_gateway_endpoint_5xx).\nRequest Avg Latency — average request latency in ms (aws_gateway_endpoint_latency).\nIntegration Avg Latency — average gateway-to-backend integration latency in ms (aws_gateway_endpoint_integration_latency).\nData Processed — bytes processed for the route, shown in KB (aws_gateway_endpoint_DataProcessed/1024).\nCache Hit Rate — percent of requests served from cache (aws_gateway_endpoint_cache_hit_rate).\nCache Miss Rate — percent of requests that missed the cache (aws_gateway_endpoint_cache_miss_rate).\nRequirements The AWS_GATEWAY dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the AWS API Gateway receiver enabled and pulling CloudWatch metrics, which produces:\nGateway (service-scope) metrics — the aws_gateway_service_* family: request count, latency, integration latency, 4xx / 5xx counts, data processed, and cache hit / miss rates.\nRoute (endpoint-scope) metrics — the aws_gateway_endpoint_* family: the same measures at route granularity.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a route-scope metric is empty until route-level data is reported. Cache-rate and data-processed widgets stay empty for gateways whose API type (REST vs HTTP API) or configuration does not emit that CloudWatch metric.\nFor setting up the receiver, see the AWS API Gateway monitoring setup guide in the SkyWalking backend docs.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/aws_gateway/","title":"\u003c!--"},{"body":" AWS S3 The AWS_S3 layer monitors Amazon S3 storage by reading CloudWatch request metrics for your buckets, so each S3 bucket appears in SkyWalking as a service with its own request, error, latency, and transfer dashboard.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under AWS and its services are listed as S3 buckets — one entry per monitored bucket. The AWS_S3 layer is a metrics-only layer: it enables the Service scope alone. There is no instance, endpoint, topology, trace, or log sub-tab, because S3 monitoring is CloudWatch metric data rather than agent-instrumented traffic.\nThis page is the operator reference for the bundled AWS_S3 dashboard: what you see for each S3 bucket and what each widget means.\nThe widgets and metrics below are read from the bundled AWS_S3 template; if an operator has published a customized AWS_S3 template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a bucket, the layer landing page lists every S3 bucket with four sortable columns, sorted by request volume (Requests) by default:\nRequests — total request count for the bucket (aws_s3_all_requests).\nAvg Latency — average request latency in ms (aws_s3_request_latency).\n4xx — count of 4xx (client-error) responses (aws_s3_4xx).\n5xx — count of 5xx (server-error) responses (aws_s3_5xx).\nService dashboard The drill-down for one selected S3 bucket.\nAll Request Count — total requests against the bucket over the window (aws_s3_all_requests).\nGET Request Count — GET (read / download) requests (aws_s3_get_requests).\nPUT Request Count — PUT (write / upload) requests (aws_s3_put_requests).\nDELETE Request Count — DELETE requests (aws_s3_delete_requests).\n4xx Count — client-error responses, the 4xx family (aws_s3_4xx).\n5xx Count — server-error responses, the 5xx family (aws_s3_5xx).\nRequest Avg Latency — average total request latency in ms (aws_s3_request_latency).\nFirst Byte Avg Latency — average time to first byte in ms, the latency before any payload starts streaming back (aws_s3_first_latency_bytes).\nDownloaded (KB) — bytes downloaded from the bucket, in KB (aws_s3_downloaded_bytes).\nUploaded (KB) — bytes uploaded to the bucket, in KB (aws_s3_uploaded_bytes).\nRequirements The AWS_S3 dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the AWS S3 monitoring receiver enabled, pulling the bucket\u0026rsquo;s CloudWatch request metrics, which OAP turns into the aws_s3_* service-scope metric family: request counts (aws_s3_all_requests, aws_s3_get_requests, aws_s3_put_requests, aws_s3_delete_requests), error counts (aws_s3_4xx, aws_s3_5xx), latency (aws_s3_request_latency, aws_s3_first_latency_bytes), and transfer volume (aws_s3_downloaded_bytes, aws_s3_uploaded_bytes).\nEvery metric in this dashboard is queried at the Service scope — the S3 bucket — so each bucket you have configured CloudWatch monitoring for appears as one entry in the S3 buckets list. For the OAP-side setup (CloudWatch credentials, the buckets to watch, and the collection interval), follow the AWS S3 monitoring setup guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/aws_s3/","title":"\u003c!--"},{"body":" BanyanDB The BANYANDB layer is the self-observability dashboard for Apache SkyWalking BanyanDB, the native storage that backs an OAP cluster. It surfaces the health of the storage tier itself — write and query throughput, the liaison front door, the data nodes that hold the shards, the lifecycle sidecar that migrates data between tiers, and the per-group load across the measure / stream / trace / property data models.\nIn Horizon\u0026rsquo;s sidebar this layer sits under the Self-Observability group and is named BanyanDB. It maps BanyanDB\u0026rsquo;s own topology onto the standard entity slots: a BanyanDB cluster is a Cluster (the service slot), each running container — a liaison, data, or lifecycle process — is a Container (the instance slot, badged with its container_name), and each storage group is a Group (the endpoint slot). The layer enables the Cluster, Container, and Group dashboards plus a layer-specific Deployment tab; it ships no service topology, no API-dependency view, and no traces or logs tabs.\nThis page is the operator reference for the bundled BANYANDB dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled BANYANDB template; if an operator has published a customized BANYANDB template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nCluster list Before opening a cluster, the layer landing page lists every BanyanDB cluster with three sortable columns, sorted by write rate by default:\nWrite/s — cluster-wide writes per second (meter_banyandb_cluster_write_rate).\nQuery/s — cluster-wide query calls per second (meter_banyandb_cluster_query_rate).\nErrors — cluster-wide errors per minute (meter_banyandb_cluster_error_rate).\nCluster dashboard The primary drill-down for one selected cluster, summarizing the whole storage tier.\nWrite Rate / Query Rate / Error Rate — three headline cards: cluster-wide writes per second across the measure + stream + trace data models (meter_banyandb_cluster_write_rate), gRPC query calls per second seen at the liaison front door (meter_banyandb_cluster_query_rate), and errors per minute summed across the cluster (meter_banyandb_cluster_error_rate). On a healthy cluster the error card reads 0.\nCPU Cores / Memory Used / Disk Used — capacity cards rolled up across the cluster\u0026rsquo;s containers: total CPU cores visible (meter_banyandb_total_cpu_cores), total memory used in GB (meter_banyandb_total_memory_used), and total on-disk bytes used across the data paths in GB (meter_banyandb_total_disk_used).\nCluster Throughput — write rate versus query rate over time on one chart (meter_banyandb_cluster_write_rate, meter_banyandb_cluster_query_rate).\nCluster Errors / min — cluster-wide errors per minute over time (meter_banyandb_cluster_error_rate).\nContainers by Role — a table of the live container count per role (data / liaison), derived from the system uptime gauge (meter_banyandb_reporting_instances). The lifecycle sidecar runs no system collector, so it does not appear here.\nContainer dashboard For one selected container. Because the three BanyanDB roles report different metrics, most widgets are role-specific and appear only on the container they apply to — a liaison container shows the front-door widgets, a data container the storage-engine widgets, and the lifecycle sidecar its migration widgets. A handful of common runtime widgets render on every container, and a few host widgets appear only when that container\u0026rsquo;s system collector reports them.\nCommon runtime (every container)\nCPU Usage — process CPU consumption in cores (meter_banyandb_instance_cpu_usage).\nResident Memory — process resident memory in MB (meter_banyandb_instance_rss_memory).\nGoroutines — live goroutine count (meter_banyandb_instance_goroutines).\nGC Pause (avg) — average Go GC pause per cycle in ms (meter_banyandb_instance_gc_pause_avg).\nGo Heap — Go heap in-use versus next-GC threshold in MB (meter_banyandb_instance_heap_inuse, meter_banyandb_instance_heap_next_gc).\nGo Alloc Rate — Go allocation rate in MB/s (meter_banyandb_instance_alloc_rate).\nHost (when the system collector reports it)\nUptime — days since the node started (meter_banyandb_instance_node_uptime). Absent on the lifecycle sidecar, which runs no system collector.\nSystem Memory Used — host memory used as a percentage (meter_banyandb_instance_system_memory_percent).\nDisk Usage — used / total across the node\u0026rsquo;s data paths as a percentage (meter_banyandb_instance_disk_usage_percent).\nDisk Used / Total — used per data path against total filesystem capacity in GB (meter_banyandb_instance_disk_used_by_path, meter_banyandb_instance_disk_total_by_path). Paths that share one filesystem report identical figures.\nNetwork I/O — per-interface receive / send throughput in KB/s (meter_banyandb_instance_network_recv, meter_banyandb_instance_network_sent).\nLifecycle sidecar (container_name = lifecycle)\nTime Since Last Sync — how long ago the last migration cycle started, shown as a duration (meter_banyandb_instance_lifecycle_last_run). Appears once the first migration cycle has run.\nLast Sync — whether the last migration cycle succeeded (OK) or failed (meter_banyandb_instance_lifecycle_last_run_success).\nMigration Cycles — cumulative tier-migration cycles run by the sidecar (meter_banyandb_instance_lifecycle_migration_cycles).\nLiaison front door (container_name = liaison)\nQuery Rate by Service — gRPC query calls per second, split by data-model service (measure / stream / trace / property) (meter_banyandb_instance_liaison_query_rate).\ngRPC Errors / min — gRPC errors per minute, summed across total + registry + stream-msg (meter_banyandb_instance_liaison_grpc_error_rate). Lazily registered, so it reads 0 on a healthy liaison.\nRegistry Ops / s — schema-registry operations per second at the front door (meter_banyandb_instance_liaison_registry_op_rate).\nWrite Rate — writes per second at the front door across the three data models (meter_banyandb_instance_liaison_write_rate).\nPublish Throughput — the tier-2 publish pipeline (liaison → data) broken out by operation (meter_banyandb_instance_liaison_publish_throughput).\nPublish p99 Latency — p99 send latency of the publish pipeline, per operation (meter_banyandb_instance_liaison_publish_latency_p99).\nPart-sync Bytes — bytes per second streamed to data nodes on the part-sync (file-sync) path in KB/s (meter_banyandb_instance_liaison_publish_bytes). Only chunked file-sync increments this counter; regular write / query publishes are not counted.\nWrite Queue Pending — liaison write-buffer depth: records buffered at the front door before publish (meter_banyandb_instance_liaison_wqueue_pending).\nPublish Batch Throughput — batches published per second by operation (meter_banyandb_instance_liaison_publish_batch_throughput). Hidden until the cluster emits batch metrics.\nPublish Batch p99 — p99 latency of batch publishes in ms (meter_banyandb_instance_liaison_publish_batch_latency_p99).\nData node (container_name = data)\nStored Data Elements — total file elements stored across measure + stream + trace (meter_banyandb_instance_data_total_data).\nWrite Queue (wqueue) — the data-node write queue: pending records, on-disk file parts, and in-memory parts (meter_banyandb_instance_data_wqueue_pending, meter_banyandb_instance_data_wqueue_file_parts, meter_banyandb_instance_data_wqueue_mem_part).\nMerge Loop Rate — file merge-loop iterations per second (meter_banyandb_instance_data_merge_file_rate).\nMerge File Latency — average on-disk file-merge latency per merge loop in ms (meter_banyandb_instance_data_merge_file_latency).\nMerge Parts / Loop — average parts merged per on-disk merge loop (meter_banyandb_instance_data_merge_file_partitions).\nInverted Index Rate — series-index updates and term searches per second across measure + stream storage + stream tst (meter_banyandb_instance_data_series_write_rate, meter_banyandb_instance_data_series_term_search_rate, meter_banyandb_instance_data_stream_tst_write_rate, meter_banyandb_instance_data_stream_tst_term_search_rate).\nIndex Documents — total inverted-index documents, used as a series proxy (meter_banyandb_instance_data_total_series, meter_banyandb_instance_data_stream_tst_total_docs).\nSubscribe Throughput — subscribe-side queue throughput by operation (query / file-sync / batch-write / control) (meter_banyandb_instance_data_queue_sub_throughput).\nSubscribe p99 Latency — p99 latency of subscribe-side queue processing in ms (meter_banyandb_instance_data_queue_sub_latency_p99).\nRetention Disk Usage — per data-model retention disk-usage percentage (meter_banyandb_instance_data_retention_measure_disk_usage_percent, meter_banyandb_instance_data_retention_stream_disk_usage_percent, meter_banyandb_instance_data_retention_trace_disk_usage_percent).\nSubscribe Message Throughput — per-record processing rate the subscriber unpacks from batches in msgs/s (meter_banyandb_instance_data_queue_sub_message_throughput).\nGroup dashboard For one selected group — a BanyanDB storage group, mapped to the endpoint slot. The widgets are organized by data model (measure, stream, trace, property); each model\u0026rsquo;s widgets render only when that model\u0026rsquo;s group reports data, and a final set of queue widgets is common to every group.\nMeasure\nMeasure Write / s — writes per second for this group (meter_banyandb_endpoint_measure_write_rate).\nMeasure Query Latency — mean liaison query latency for this group in ms (meter_banyandb_endpoint_measure_query_latency).\nMeasure Total Data — current stored data elements for this group (meter_banyandb_endpoint_measure_total_data).\nMeasure Merge Rate — file merge-loop iterations per minute (meter_banyandb_endpoint_measure_merge_file_rate).\nMeasure Merge Latency — mean file-merge latency per merge loop in ms (meter_banyandb_endpoint_measure_merge_file_latency).\nMeasure Merge Partitions — average parts merged per file merge loop (meter_banyandb_endpoint_measure_merge_file_partitions).\nMeasure Series Write / s — inverted-index updates per second, used as a series-write proxy (meter_banyandb_endpoint_measure_series_write_rate).\nMeasure Term Search / s — inverted-index term-search invocations per second, the index read-pressure signal (meter_banyandb_endpoint_measure_series_term_search_rate).\nMeasure Total Series — total inverted-index documents for this group, used as a series proxy (meter_banyandb_endpoint_measure_total_series).\nStream\nStream Write / s — writes per second for this group (meter_banyandb_endpoint_stream_write_rate).\nStream Query Latency — mean liaison query latency for this group in ms (meter_banyandb_endpoint_stream_query_latency).\nStream Total Data — current stored data elements for this group (meter_banyandb_endpoint_stream_total_data).\nStream Merge Rate — file merge-loop iterations per minute (meter_banyandb_endpoint_stream_merge_file_rate).\nStream Merge Latency — mean file-merge latency per merge loop in ms (meter_banyandb_endpoint_stream_merge_file_latency).\nStream Merge Partitions — average parts merged per file merge loop (meter_banyandb_endpoint_stream_merge_file_partitions).\nStream Series Write / s — inverted-index updates per second, used as a series-write proxy (meter_banyandb_endpoint_stream_series_write_rate).\nStream TST Index Write / s — stream tst-scope inverted-index updates per second, distinct from the storage-scope series index (meter_banyandb_endpoint_stream_tst_index_write_rate).\nStream Term Search / s — inverted-index term-search invocations per second (meter_banyandb_endpoint_stream_series_term_search_rate).\nStream Total Series — total inverted-index documents for this group (meter_banyandb_endpoint_stream_total_series).\nStream TST Total Series — the stream tst-scope index document total, distinct from the storage-scope series index (meter_banyandb_endpoint_stream_tst_total_series).\nTrace\nTrace Write / s — writes per second for this group (meter_banyandb_endpoint_trace_write_rate).\nTrace Query Latency — mean liaison query latency for this group in ms (meter_banyandb_endpoint_trace_query_latency).\nTrace Total Data — current stored data elements for this group (meter_banyandb_endpoint_trace_total_data).\nTrace Merge Rate — file merge-loop iterations per minute (meter_banyandb_endpoint_trace_merge_file_rate).\nTrace Merge Latency — mean file-merge latency per merge loop in ms (meter_banyandb_endpoint_trace_merge_file_latency).\nTrace Merge Partitions — average parts merged per file merge loop (meter_banyandb_endpoint_trace_merge_file_partitions).\nTrace Series Write / s — inverted-index updates per second, used as a series-write proxy (meter_banyandb_endpoint_trace_series_write_rate).\nTrace Term Search / s — inverted-index term-search invocations per second (meter_banyandb_endpoint_trace_series_term_search_rate).\nTrace Total Series — total inverted-index documents for this group (meter_banyandb_endpoint_trace_total_series).\nProperty\nProperty Index Write / s — property-registry inverted-index updates per second, the property model\u0026rsquo;s write signal (meter_banyandb_endpoint_property_index_write_rate).\nProperty Index Merge Rate — property inverted-index segment merges per minute; property has no tst merge loop (meter_banyandb_endpoint_property_index_merge_rate).\nProperty Index Merge Latency — mean property inverted-index merge latency in ms (meter_banyandb_endpoint_property_index_merge_latency).\nProperty Term Search / s — property term-search invocations per second; property is read via the registry / term-search path rather than the liaison query method, so this is its read-load signal (meter_banyandb_endpoint_property_series_term_search_rate).\nProperty Total Series — total property inverted-index documents for this group (meter_banyandb_endpoint_property_total_series).\nQueue (every group)\nSubscribe Throughput — subscribe-side queue messages per second for this group, by operation (meter_banyandb_endpoint_queue_throughput).\nPublish p99 — publish-side queue p99 latency for this group in ms (meter_banyandb_endpoint_queue_latency_p99).\nBatch Throughput — per-group write-batch rate, by operation (meter_banyandb_endpoint_queue_batch_throughput).\nMessage Throughput — per-group per-record rate, by operation (meter_banyandb_endpoint_queue_message_throughput).\nPart-sync Bytes / s — part-sync (file-sync) bytes per second for this group in KB/s (meter_banyandb_endpoint_publish_bytes). Only the chunked part-streaming path increments this; regular write / query publishes are not counted.\nDeployment The BANYANDB layer enables the layer-specific Deployment tab — the deployment topology of one cluster\u0026rsquo;s own containers and the intra-cluster calls between them. Pick a cluster from the header and the tab draws its containers as health-ring nodes laid out left → right along the calls between them, with animated edge flow, a per-edge metric panel, and a node popover that opens the container dashboard. For how the Deployment tab is read and navigated in general, see the Deployment section of Layer Dashboard Templates.\nContainers are grouped into three roles by their node_role / node_type attributes:\nLiaison — the front door. Its node center shows Query/s (meter_banyandb_instance_liaison_query_rate) and its health ring tracks gRPC err/min (meter_banyandb_instance_liaison_grpc_error_rate).\nData — the storage nodes. Center shows Ingest/s (meter_banyandb_instance_data_queue_sub_throughput) and the ring tracks Disk % (meter_banyandb_instance_disk_usage_percent).\nLifecycle — the tier-migration sidecar. Center shows cumulative Cycles (meter_banyandb_instance_lifecycle_migration_cycles) and the ring tracks Last OK (meter_banyandb_instance_lifecycle_last_run_success).\nBecause role-pair edges are configured, the Deployment map gains a Flows sub-tab listing every edge grouped by role-pair. Each edge type carries its own client-side (publish) and server-side (subscribe) metrics, so a liaison → data call surfaces a different metric set than a liaison → liaison forward or a lifecycle → data migration:\nliaison → data — the main write / query path. Per-operation Write/s, Query/s, and Part-sync/s throughput; Write p99 and Query p99 latency; Part-sync B/s bytes; and Errors/s. Each is paired across the publish side (meter_banyandb_instance_relation_publish_*, filtered by operation) and the subscribe side (meter_banyandb_instance_relation_queue_sub_*).\nliaison → liaison — node-to-node forwarding. Forward/s and Forward p99 for the batch-write forward, Control/s for the control channel, and Errors/s (meter_banyandb_instance_relation_publish_throughput{operation='batch-write'} and the matching subscribe / control / error counters).\nlifecycle → data — the tier-migration path. Migrate/s throughput, Migrate p99 latency, Migrate B/s bytes, and Errors/s (meter_banyandb_instance_relation_migration_* on the publish side, meter_banyandb_instance_relation_queue_sub_* on the subscribe side).\nany other pair — a generic fallback showing aggregated Msg/s and p99 (aggregate_labels(meter_banyandb_instance_relation_publish_throughput,sum) and the matching latency / subscribe counters), so an edge that matches no specific role-pair still reports something.\nRequirements The BANYANDB dashboard is a pure consumer of what OAP reports about its BanyanDB storage tier — it invents no data, and a widget with no backing data simply reads no data (or 0 for the lazily-registered error counters). To populate it, OAP needs BanyanDB self-observability enabled so that BanyanDB exposes its metrics and OAP ingests them into the meter_banyandb_* families:\nCluster metrics — meter_banyandb_cluster_* and the meter_banyandb_total_* capacity rollups for the Cluster list and Cluster dashboard.\nContainer metrics — meter_banyandb_instance_* for the per-container runtime, host, liaison, data, and lifecycle widgets. A container only shows the families its role emits, and the host widgets need a running system collector (absent on the lifecycle sidecar).\nGroup metrics — the per-data-model meter_banyandb_endpoint_* families (measure / stream / trace / property, plus the shared queue counters) for the Group dashboard.\nRelation metrics — meter_banyandb_instance_relation_* (publish / subscribe / migration throughput, latency, bytes, and error counters) for the Deployment tab\u0026rsquo;s edges.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a container- or group-scope metric is empty until that level of data is reported, and an entire data model\u0026rsquo;s group widgets stay hidden until that model\u0026rsquo;s group reports.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/banyandb/","title":"\u003c!--"},{"body":" BookKeeper The BOOKKEEPER layer monitors Apache BookKeeper, the distributed write-ahead log storage that backs systems such as Apache Pulsar. OAP gathers BookKeeper\u0026rsquo;s metrics through the OpenTelemetry receiver and aggregates them per bookie node and per cluster.\nIn Horizon\u0026rsquo;s sidebar this layer is named BookKeeper. Its services are listed as BookKeeper clusters and its instances as Bookies — each bookie is one storage node in the cluster. This layer enables the Service and Instance scopes only: there is no endpoint scope, no topology, and no traces or logs tab, because BookKeeper reports node-level meters rather than request traffic.\nThis page is the operator reference for the bundled BOOKKEEPER dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled BOOKKEEPER template; if an operator has published a customized BOOKKEEPER template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every BookKeeper cluster with four sortable columns, sorted by Ledgers by default:\nLedgers — total ledgers held across the cluster\u0026rsquo;s bookies (meter_bookkeeper_bookie_ledgers_count, summed).\nEntries — total entries stored across the cluster\u0026rsquo;s bookies (meter_bookkeeper_bookie_entries_count, summed).\nWritable Dirs — number of ledger directories currently accepting writes (meter_bookkeeper_bookie_ledger_writable_dirs, summed).\nDir Usage — the ledger data directory\u0026rsquo;s fill level (meter_bookkeeper_bookie_ledger_dir_data_bookkeeper_ledgers_usage).\nService dashboard The primary drill-down for one selected BookKeeper cluster. Every widget aggregates the cluster\u0026rsquo;s bookies with aggregate_labels(..., sum).\nBookie Ledgers — ledgers held across the cluster over time (meter_bookkeeper_bookie_ledgers_count).\nBookie Entries — entries stored across the cluster over time (meter_bookkeeper_bookie_entries_count).\nWritable Ledger Dirs — ledger directories currently accepting writes (meter_bookkeeper_bookie_ledger_writable_dirs).\nWrite Cache — the bookie write cache on a dual axis: cache size in MB on the left (meter_bookkeeper_bookie_write_cache_size, divided to MB) and cached entry count on the right (meter_bookkeeper_bookie_write_cache_count).\nRead Cache — the bookie read cache on a dual axis: cache size in MB on the left (meter_bookkeeper_bookie_read_cache_size, divided to MB) and cached entry count on the right (meter_bookkeeper_bookie_read_cache_count).\nRead / Write Rate (B/s) — bytes per second served and ingested, plotted together: read (meter_bookkeeper_bookie_read_rate) and write (meter_bookkeeper_bookie_write_rate).\nLedger Dir Usage — fill level of the ledger data directory over time (meter_bookkeeper_bookie_ledger_dir_data_bookkeeper_ledgers_usage).\nInstance dashboard For one selected bookie. These widgets cover the bookie\u0026rsquo;s JVM runtime and its internal thread pools.\nJVM Memory Pool (MB) — used memory per JVM memory pool in MB (meter_bookkeeper_node_jvm_memory_pool_used).\nJVM Memory (MB) — JVM memory in MB: used, committed, and init (meter_bookkeeper_node_jvm_memory_used, meter_bookkeeper_node_jvm_memory_committed, meter_bookkeeper_node_jvm_memory_init).\nJVM Threads — thread counts: current, daemon, peak, and deadlocked (meter_bookkeeper_node_jvm_threads_current, meter_bookkeeper_node_jvm_threads_daemon, meter_bookkeeper_node_jvm_threads_peak, meter_bookkeeper_node_jvm_threads_deadlocked).\nGC — garbage-collection activity on a dual axis: cumulative GC seconds on the left (meter_bookkeeper_node_jvm_gc_collection_seconds_sum) and GC count on the right (meter_bookkeeper_node_jvm_gc_collection_seconds_count).\nThread Executor — the bookie\u0026rsquo;s task executor: completed, tasks completed, rejected, and failed (meter_bookkeeper_node_thread_executor_completed, meter_bookkeeper_node_thread_executor_tasks_completed, meter_bookkeeper_node_thread_executor_tasks_rejected, meter_bookkeeper_node_thread_executor_tasks_failed).\nPooled Threads — thread counts for the high-priority and read pools (meter_bookkeeper_node_high_priority_threads, meter_bookkeeper_node_read_thread_pool_threads).\nPool Max Queue Size — the maximum queue size of the high-priority and read thread pools (meter_bookkeeper_node_high_priority_thread_max_queue_size, meter_bookkeeper_node_read_thread_pool_max_queue_size).\nRequirements The BOOKKEEPER dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nBookie metrics — the meter_bookkeeper_bookie_* family (ledgers, entries, writable directories, directory usage, read/write caches, and read/write rates), aggregated at the BookKeeper cluster (Service) scope.\nBookie node metrics — the meter_bookkeeper_node_* family (JVM memory, threads, and GC, plus the bookie\u0026rsquo;s thread executor and thread pools), reported at the bookie (ServiceInstance) scope.\nThese metrics come from BookKeeper\u0026rsquo;s own OpenTelemetry export, gathered by OAP\u0026rsquo;s OpenTelemetry receiver — see the BookKeeper monitoring setup. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the instance-scope widgets stay empty until per-bookie data is reported.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/bookkeeper/","title":"\u003c!--"},{"body":" Browser The BROWSER layer is where SkyWalking\u0026rsquo;s browser agent (the client-side JavaScript SDK) reports. It is real-user monitoring: page views, front-end errors, page-load timing, and Core Web Vitals collected from the visitor\u0026rsquo;s browser rather than from a server-side agent.\nIn Horizon\u0026rsquo;s sidebar this layer is named Browser. Its top-level entities are web applications, listed as Apps; each app reports under one or more Versions (the instance slot), and each app serves a set of Pages (the endpoint slot). So where the GENERAL layer reads \u0026ldquo;Service / Instance / Endpoint\u0026rdquo;, BROWSER reads \u0026ldquo;App / Version / Page\u0026rdquo;. The layer enables the App, Version, and Page dashboards, plus the Traces tab and a Browser Logs tab — the per-page front-end error stream, which can de-obfuscate a minified JavaScript stack against a source map you upload. BROWSER has no service topology; there is no map view.\nThis page is the operator reference for the bundled BROWSER dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled BROWSER template; if an operator has published a customized BROWSER template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nApp list Before opening an app, the layer landing page lists every BROWSER app with three columns, sorted by traffic (Page Views) by default:\nPage Views — page views per minute (browser_app_pv).\nError Rate — percent of page views that recorded an error (browser_app_error_rate/100).\nErrors — total front-end errors in the window (browser_app_error_sum).\nApp dashboard The primary drill-down for one selected app.\nApp Load (PV) — page views per minute for the app (browser_app_pv).\nApp Error Rate — percent of page views that recorded an error (browser_app_error_rate/100).\nApp Error Count — total front-end errors per minute (browser_app_error_sum).\nTop Hot Pages — the app\u0026rsquo;s busiest pages, with tabs to re-rank by PV (browser_app_page_pv, /min), Errors (browser_app_page_error_sum), and Error Rate (browser_app_page_error_rate, %), worst-first. Click a row to jump into that page.\nTop Versions — the app\u0026rsquo;s versions broken down the same three ways: PV (browser_app_single_version_pv, /min), Errors (browser_app_single_version_error_sum), and Error Rate (browser_app_single_version_error_rate, %).\nVersion dashboard For one selected app Version — the per-release view of the same load and error signals.\nVersion PV — page views per minute for this version (browser_app_single_version_pv).\nVersion Error Rate — percent of this version\u0026rsquo;s page views that recorded an error (browser_app_single_version_error_rate/100).\nVersion Error Count — total front-end errors per minute for this version (browser_app_single_version_error_sum).\nPage dashboard For one selected Page — the deepest scope, where browser timing and Web Vitals live. This is the page-performance view: most of these metrics exist only at page scope.\nFirst Meaningful Paint Percentile — p50 / p75 / p90 / p95 / p99 of FMP latency for the page, in ms (browser_app_page_fmp_percentile). Below 1s at p75 is a common target.\nPage Load Percentile — p50 / p75 / p90 / p95 / p99 of full page-load time, in ms (browser_app_page_load_page_percentile).\nTime-to-Live Percentile — p50 / p75 / p90 / p95 / p99 of the page\u0026rsquo;s time-to-live, in ms (browser_app_page_ttl_percentile).\nFirst Pack Latency Percentile — p50 / p75 / p90 / p95 / p99 of first-pack latency, in ms (browser_app_page_first_pack_percentile).\nPage Performance Breakdown — average time spent in each phase of the page load, in ms, on one chart: DNS, redirect, TCP, TTFB, transfer, DOM analysis, DOM ready, FPT, load, and resource (browser_app_page_dns_avg, browser_app_page_redirect_avg, browser_app_page_tcp_avg, browser_app_page_ttfb_avg, browser_app_page_trans_avg, browser_app_page_dom_analysis_avg, browser_app_page_dom_ready_avg, browser_app_page_fpt_avg, browser_app_page_load_page_avg, browser_app_page_res_avg).\nPage Errors by Type — front-end error counters per minute split by source: resource, JS, AJAX, and unknown (browser_app_page_resource_error_sum, browser_app_page_js_error_sum, browser_app_page_ajax_error_sum, browser_app_page_unknown_error_sum).\nWeb Vitals — Core Web Vitals as averages per minute: FMP (ms), LCP (ms), and CLS (browser_app_web_vitals_fmp_avg, browser_app_web_vitals_lcp_avg, browser_app_web_vitals_cls_avg / 1000 — CLS is scaled down to its typical 0 – 1 score range).\nInteraction to Next Paint Percentile — p50 / p75 / p90 / p95 / p99 of INP, in ms (browser_app_web_interaction_inp_percentile). INP is the responsiveness metric that replaces FID.\nRequirements The BROWSER dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, your front-end must run the SkyWalking browser agent (the client-side JavaScript SDK) reporting to OAP, which produces:\nApp metrics — the browser_app_* family (page views, error rate, error sum), produced by OAP from browser-agent reports, for the App list and App dashboard.\nVersion metrics — browser_app_single_version_* (PV, error rate, error sum) for the Top Versions widget and the Version dashboard.\nPage metrics — browser_app_page_* (PV, errors, the timing percentiles, and the per-phase performance averages) and the browser_app_web_vitals_* / browser_app_web_interaction_* families for the Page dashboard.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a version- or page-scope metric is empty until that level of data is reported. BROWSER carries no relation metrics, so it has no topology or map view.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/browser/","title":"\u003c!--"},{"body":" Cilium Service The CILIUM_SERVICE layer monitors Kubernetes services observed through Cilium\u0026rsquo;s eBPF data plane. SkyWalking collects L4 (TCP) packet activity and L7 protocol telemetry (HTTP, DNS, Kafka) that Cilium reports for each service, giving you network-level and protocol-level visibility into the mesh without instrumenting the application. It belongs to the Kubernetes layer group.\nIn Horizon\u0026rsquo;s sidebar this layer is named Cilium Service. Its services are listed as Services, instances as Pods, and endpoints as Endpoints — service names are grouped and displayed by their Kubernetes namespace. The CILIUM_SERVICE layer enables the Service, Pod, Endpoint, and Topology sub-tabs. It does not enable Traces or Logs.\nThis page is the operator reference for the bundled CILIUM_SERVICE dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled CILIUM_SERVICE template; if an operator has published a customized CILIUM_SERVICE template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a service, the layer landing page lists every CILIUM_SERVICE service with four sortable columns, sorted by traffic (RPM) by default:\nRPM — protocol calls per minute (cilium_service_protocol_cpm).\nLatency — average protocol call duration in ms (cilium_service_protocol_call_duration/cilium_service_protocol_cpm/1000000).\nSuccess Rate — percent of successful protocol calls (cilium_service_protocol_call_success_count/cilium_service_protocol_cpm*100).\nTCP Drop — dropped read + write packets per minute at L4 (cilium_service_l4_read_pkg_drop_cpm + cilium_service_l4_write_pkg_drop_cpm).\nService dashboard The primary drill-down for one selected service. It splits into an L4 (TCP) row and per-protocol (HTTP, DNS, Kafka) groups.\nL4 (TCP)\nL4 Read Packages/min — inbound packets per minute (cilium_service_l4_read_pkg_cpm).\nL4 Write Packages/min — outbound packets per minute (cilium_service_l4_write_pkg_cpm).\nTCP Drop / min — dropped read and write packets per minute, plotted as two series (cilium_service_l4_read_pkg_drop_cpm, cilium_service_l4_write_pkg_drop_cpm).\nTCP Drop by Reason — dropped-packet count broken out by Cilium\u0026rsquo;s drop-reason label (cilium_service_l4_drop_reason_count).\nHTTP\nHTTP Load — HTTP calls per minute alongside the successful-call count (cilium_service_protocol_http_call_cpm, cilium_service_protocol_http_call_success_count).\nHTTP Duration — average HTTP call duration in ms (cilium_service_protocol_http_call_duration/cilium_service_protocol_http_call_cpm/1000000).\nHTTP Non-2xx Status — HTTP responses per minute by status class: 1xx / 3xx / 4xx / 5xx (cilium_service_protocol_http_status_1xx_cpm, cilium_service_protocol_http_status_3xx_cpm, cilium_service_protocol_http_status_4xx_cpm, cilium_service_protocol_http_status_5xx_cpm).\nDNS\nDNS Load — DNS calls per minute alongside the successful-call count (cilium_service_protocol_dns_call_cpm, cilium_service_protocol_dns_call_success_count).\nDNS Duration — average DNS call duration in ms (cilium_service_protocol_dns_call_duration/cilium_service_protocol_dns_call_cpm/1000000).\nDNS Errors / min — DNS error count per minute (cilium_service_protocol_dns_error_count).\nKafka\nKafka Load — Kafka calls per minute alongside the successful-call count (cilium_service_protocol_kafka_call_cpm, cilium_service_protocol_kafka_call_success_count).\nKafka Duration — average Kafka call duration in ms (cilium_service_protocol_kafka_call_duration/cilium_service_protocol_kafka_call_cpm/1000000).\nKafka Errors / min — Kafka error count per minute (cilium_service_protocol_kafka_call_error_count).\nPod dashboard For one selected pod (a service instance). The widgets mirror the service view at instance scope, covering L4 plus the HTTP, DNS, and Kafka protocols.\nL4 Read Packages/min — inbound packets per minute for the pod (cilium_service_instance_l4_read_pkg_cpm).\nL4 Write Packages/min — outbound packets per minute for the pod (cilium_service_instance_l4_write_pkg_cpm).\nHTTP Load — HTTP calls per minute alongside the successful-call count (cilium_service_instance_protocol_http_call_cpm, cilium_service_instance_protocol_http_call_success_count).\nHTTP Non-2xx Status — HTTP responses per minute by status class: 1xx / 3xx / 4xx / 5xx (cilium_service_instance_protocol_http_status_1xx_cpm, cilium_service_instance_protocol_http_status_3xx_cpm, cilium_service_instance_protocol_http_status_4xx_cpm, cilium_service_instance_protocol_http_status_5xx_cpm).\nDNS Load — DNS calls per minute alongside the successful-call count (cilium_service_instance_protocol_dns_call_cpm, cilium_service_instance_protocol_dns_call_success_count).\nDNS Errors — DNS error count (cilium_service_instance_protocol_dns_error_count).\nKafka Load — Kafka calls per minute alongside the successful-call count (cilium_service_instance_protocol_kafka_call_cpm, cilium_service_instance_protocol_kafka_call_success_count).\nEndpoint dashboard For one selected endpoint. Cilium endpoints carry L7 protocol traffic, so this scope is HTTP- and DNS-focused.\nHTTP Load — HTTP calls per minute alongside the successful-call count (cilium_endpoint_protocol_http_call_cpm, cilium_endpoint_protocol_http_call_success_count).\nHTTP Duration — average HTTP call duration in ms (cilium_endpoint_protocol_http_call_duration/cilium_endpoint_protocol_http_call_cpm/1000000).\nHTTP Non-2xx Status — HTTP responses per minute by status class: 1xx / 3xx / 4xx / 5xx (cilium_endpoint_protocol_http_status_1xx_cpm, cilium_endpoint_protocol_http_status_3xx_cpm, cilium_endpoint_protocol_http_status_4xx_cpm, cilium_endpoint_protocol_http_status_5xx_cpm).\nDNS Load — DNS calls per minute alongside the successful-call count (cilium_endpoint_protocol_dns_call_cpm, cilium_endpoint_protocol_dns_call_success_count).\nDNS Duration — average DNS call duration in ms (cilium_endpoint_protocol_dns_call_duration/cilium_endpoint_protocol_dns_call_cpm/1000000).\nDNS Errors — DNS error count (cilium_endpoint_protocol_dns_error_count).\nTopology and maps The CILIUM_SERVICE layer ships a service topology with an instance-level drill-down.\nTopology (service map) — each service node is decorated with RPM (cilium_service_protocol_cpm), a Success % health ring (cilium_service_protocol_call_success_count/cilium_service_protocol_cpm*100, colored green / amber / red — higher is better, so the ring turns red as the success rate drops), and Latency (cilium_service_protocol_call_duration/cilium_service_protocol_cpm/1000000). Each call edge carries server-side HTTP RPM (cilium_service_relation_server_protocol_http_call_cpm) and Avg Latency (cilium_service_relation_server_protocol_http_call_duration/cilium_service_relation_server_protocol_http_call_cpm/1000000).\nInstance map — from a call between two services on the service map, drill into the pod-to-pod calls between them. Each instance node shows HTTP RPM (cilium_service_instance_protocol_http_call_cpm) and Avg Latency (cilium_service_instance_protocol_http_call_duration/cilium_service_instance_protocol_http_call_cpm/1000000); each edge carries server-side HTTP RPM (cilium_service_instance_relation_server_protocol_http_call_cpm) and Avg Latency (cilium_service_instance_relation_server_protocol_http_call_duration/cilium_service_instance_relation_server_protocol_http_call_cpm/1000000).\nFor how these maps are read and navigated, see the 3D Infrastructure Map for the cross-layer view, and the topology section of Layer Dashboard Templates for how the node and edge metrics are configured.\nRequirements The CILIUM_SERVICE dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Cilium monitoring enabled, with the Cilium-Hubble fetcher feeding SkyWalking. Specifically:\nL4 metrics — the cilium_service_l4_* family (read / write packet rates, packet drops, and drop-reason breakdown) for the service-scope TCP widgets.\nProtocol metrics — the cilium_service_protocol_*, cilium_service_instance_protocol_*, and cilium_endpoint_protocol_* families covering HTTP, DNS, and Kafka call counts, durations, success counts, status classes, and errors, at their respective service / instance / endpoint scopes.\nRelation metrics — cilium_service_relation_server_protocol_* and cilium_service_instance_relation_server_protocol_* for the service map and instance map edges.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so an instance- or endpoint-scope metric is empty until that level of data is reported. A pod or endpoint that carries only one protocol shows no data for the others. For setup, see the Cilium monitoring guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/cilium_service/","title":"\u003c!--"},{"body":" ClickHouse The CLICKHOUSE layer monitors ClickHouse database clusters. SkyWalking collects ClickHouse\u0026rsquo;s internal metrics — queries, query latency, merges and mutations, data parts, replication, ZooKeeper / Keeper coordination, and per-node host stats — through OpenTelemetry, and Horizon renders them as a cluster-level and a node-level dashboard. This is a metrics-only layer: there are no traces, logs, endpoints, or a topology map for ClickHouse.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Databases and named ClickHouse. Its services are listed as ClickHouse clusters and its instances as Nodes. Because the layer carries no endpoint, topology, trace, or log data, it enables only two sub-tabs: a Service (cluster) dashboard and an Instance (node) dashboard.\nThis page is the operator reference for the bundled CLICKHOUSE dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled CLICKHOUSE template; if an operator has published a customized CLICKHOUSE template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every ClickHouse cluster with four sortable columns, sorted by select rate (Select / s) by default:\nSelect / s — SELECT queries per second across the cluster (aggregate_labels(meter_clickhouse_query_select_rate,sum)).\nInsert / s — INSERT queries per second across the cluster (aggregate_labels(meter_clickhouse_query_insert_rate,sum)).\nSlow Reads — slow file reads across the cluster (aggregate_labels(meter_clickhouse_query_slow,sum)).\nOpen Files — the latest count of open files across the cluster (latest(aggregate_labels(meter_clickhouse_file_open,sum))).\nService dashboard The cluster-level drill-down for one selected ClickHouse cluster. Every widget aggregates across the cluster\u0026rsquo;s nodes with aggregate_labels(...,sum).\nFiles Open — the latest number of open files in the cluster, as a single card (latest(aggregate_labels(meter_clickhouse_file_open,sum))).\nQPS — query rate per second, plotted as two series: select (aggregate_labels(meter_clickhouse_query_select_rate,sum)) and insert (aggregate_labels(meter_clickhouse_query_insert_rate,sum)).\nQueries — query counts split into total, select, and insert (aggregate_labels(meter_clickhouse_query,sum), aggregate_labels(meter_clickhouse_query_select,sum), aggregate_labels(meter_clickhouse_query_insert,sum)).\nQuery Time (ms) — average time per query in ms, as avg, select, and insert series, each computed as total query microseconds divided by query count and converted to ms (aggregate_labels(meter_clickhouse_querytime_microseconds,sum)/aggregate_labels(meter_clickhouse_query,sum)/1000 and the matching _select_ / _insert_ pair).\nConnections — open client connections by protocol: TCP (aggregate_labels(meter_clickhouse_tcp_connections,sum)) and HTTP (aggregate_labels(meter_clickhouse_http_connections,sum)).\nSlow Reads — slow file reads across the cluster (aggregate_labels(meter_clickhouse_query_slow,sum)).\nMerge / Mutations — background merge operations (aggregate_labels(meter_clickhouse_background_merge,sum)) and mutations (aggregate_labels(meter_clickhouse_mutations,sum)).\nInsert Throughput — insert volume on a dual axis: bytes/s on the left (aggregate_labels(meter_clickhouse_inserted_bytes,sum)) and rows/s on the right (aggregate_labels(meter_clickhouse_inserted_rows,sum)).\nDelayed Inserts (s) — inserts that were throttled / delayed (aggregate_labels(meter_clickhouse_delayed_inserts,sum)).\nActive Data Parts — the number of active MergeTree data parts in the cluster (aggregate_labels(meter_clickhouse_parts_active,sum)).\nReplicated Fetch / Send — replication traffic between replicas: fetch (aggregate_labels(meter_clickhouse_replicated_fetch,sum)) and send (aggregate_labels(meter_clickhouse_replicated_send,sum)).\nZookeeper Activity — the coordination layer\u0026rsquo;s health, with the latest sessions and watches (latest(aggregate_labels(meter_clickhouse_zookeeper_session,sum)), latest(aggregate_labels(meter_clickhouse_zookeeper_watch,sum))) plus bytes sent and bytes recv over time (aggregate_labels(meter_clickhouse_zookeeper_bytes_sent,sum), aggregate_labels(meter_clickhouse_zookeeper_bytes_received,sum)).\nKeeper Alive Conns — the latest count of alive ClickHouse Keeper connections, as a single card (latest(aggregate_labels(meter_clickhouse_keeper_connections_alive,sum))).\nKeeper Outstanding Requests — the latest count of outstanding ClickHouse Keeper requests, as a single card (latest(aggregate_labels(meter_clickhouse_keeper_outstanding_requests,sum))).\nInstance dashboard The node-level drill-down for one selected ClickHouse node. These widgets read the per-node meter_clickhouse_instance_* family directly, with no cross-node aggregation.\nUptime (days) — how long the node has been running, as a card, converted from seconds (latest(meter_clickhouse_instance_uptime)/3600/24).\nVersion — the node\u0026rsquo;s ClickHouse version, as a card (latest(meter_clickhouse_instance_version)).\nCPU (cores) — CPU consumption expressed in cores (meter_clickhouse_instance_cpu_usage/1000000).\nMemory (%) — used vs available memory percentage (meter_clickhouse_instance_memory_usage, meter_clickhouse_instance_memory_available).\nNetwork (B) — bytes receive vs send on the node (meter_clickhouse_instance_network_receive_bytes, meter_clickhouse_instance_network_send_bytes).\nConnections — open client connections by protocol: TCP (meter_clickhouse_instance_tcp_connections) and HTTP (meter_clickhouse_instance_http_connections).\nQueries — query counts split into total, select, and insert (meter_clickhouse_instance_query, meter_clickhouse_instance_query_select, meter_clickhouse_instance_query_insert).\nQPS — query rate per second, as select and insert series (meter_clickhouse_instance_query_select_rate, meter_clickhouse_instance_query_insert_rate).\nQuery Time (ms) — average time per query in ms, as avg, select, and insert series, each total query microseconds divided by query count and converted to ms (meter_clickhouse_instance_querytime_microseconds/meter_clickhouse_instance_query/1000 and the matching _select_ / _insert_ pair).\nFile Slow Read — slow file reads on the node (meter_clickhouse_instance_query_slow).\nBackground Merge — background merge operations on the node (meter_clickhouse_instance_background_merge).\nMutations — mutation operations on the node (meter_clickhouse_instance_mutations).\nFiles Open — the latest number of open files on the node, as a card (latest(meter_clickhouse_instance_file_open)).\nInsert Throughput — insert volume on a dual axis: bytes/s on the left (meter_clickhouse_instance_inserted_bytes) and rows/s on the right (meter_clickhouse_instance_inserted_rows).\nDelayed Inserts (s) — inserts that were throttled / delayed on the node (meter_clickhouse_instance_delayed_inserts).\nRequirements The CLICKHOUSE dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs ClickHouse metrics delivered through the OpenTelemetry receiver, which OAP aggregates into the meter_clickhouse_* families:\nCluster (service-scope) metrics — the meter_clickhouse_* family (queries, query rate, query time, connections, slow reads, merges, mutations, data parts, insert throughput, delayed inserts, replication, ZooKeeper, and Keeper), which the cluster dashboard and the Service list aggregate across nodes.\nNode (instance-scope) metrics — the meter_clickhouse_instance_* family (uptime, version, CPU, memory, network, connections, queries, query rate, query time, slow reads, merges, mutations, open files, and insert throughput) for the node dashboard.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the node-scope widgets stay empty until per-node data is reported. Setting up the ClickHouse OpenTelemetry collection is described in the upstream ClickHouse monitoring guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/clickhouse/","title":"\u003c!--"},{"body":" Elasticsearch The ELASTICSEARCH layer monitors Elasticsearch clusters that OAP scrapes through its Elasticsearch monitoring receiver. It groups under Databases in the sidebar and gives an operator the cluster-health, node-runtime, and per-index view that an Elasticsearch admin expects.\nIn Horizon\u0026rsquo;s sidebar this layer carries the display name Elasticsearch. An Elasticsearch cluster maps onto SkyWalking\u0026rsquo;s entity scopes, and the layer renames each slot to match: services are listed as ES clusters, instances as Nodes, and endpoints as Indices. The layer enables three drill-down tabs — Service (the cluster dashboard), Instance (a node), and Endpoint (an index). It ships no topology, traces, or logs tabs.\nThis page is the operator reference for the bundled ELASTICSEARCH dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled ELASTICSEARCH template; if an operator has published a customized ELASTICSEARCH template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nES clusters list Before opening a cluster, the layer landing page lists every Elasticsearch cluster with four sortable columns, sorted by Shards by default:\nHealth — the cluster health status (meter_elasticsearch_cluster_health_status), averaged over the window.\nShards — total active shards across the cluster (meter_elasticsearch_cluster_shards_total, latest value).\nNodes — number of nodes in the cluster (meter_elasticsearch_cluster_nodes, latest value).\nUnassigned — shards the cluster has not yet placed on a node (meter_elasticsearch_cluster_unassigned_shards_total, latest value) — a non-zero value is the usual first sign of a cluster under stress.\nService dashboard (cluster) The primary drill-down for one selected cluster — the cluster-wide health and capacity picture.\nCluster Health — a table of the current health status and value (meter_elasticsearch_cluster_health_status, latest), the green / yellow / red rollup Elasticsearch reports for the cluster.\nNodes — number of nodes currently in the cluster (meter_elasticsearch_cluster_nodes, latest).\nPending Tasks — average count of cluster-level tasks queued for the master (meter_elasticsearch_cluster_pending_tasks_total) — a rising queue points at master-node pressure.\nPrimary Shards — total primary shards (meter_elasticsearch_cluster_primary_shards_total, latest).\nActive Shards — total active shards (meter_elasticsearch_cluster_shards_total, latest).\nInitializing — shards currently initializing (meter_elasticsearch_cluster_initializing_shards_total, latest).\nRelocating — shards being moved between nodes (meter_elasticsearch_cluster_relocating_shards_total, latest).\nUnassigned — shards not assigned to any node (meter_elasticsearch_cluster_unassigned_shards_total, latest).\nDelayed Unassigned — unassigned shards whose reassignment is being delayed (meter_elasticsearch_cluster_delayed_unassigned_shards_total, latest).\nTripped Breakers — count of tripped circuit breakers across the cluster (meter_elasticsearch_cluster_breakers_tripped, latest), a memory-protection signal.\nCluster CPU Avg — average CPU usage across the cluster\u0026rsquo;s nodes in percent (meter_elasticsearch_cluster_cpu_usage_avg).\nJVM Memory Used Avg — average JVM heap memory used across the cluster (meter_elasticsearch_cluster_jvm_memory_used_avg).\nOpen Files Avg — average open file-descriptor count across the cluster (meter_elasticsearch_cluster_open_file_count).\nInstance dashboard (node) For one selected node — the per-node OS, JVM, and storage detail.\nProcess CPU (%) — CPU consumed by the Elasticsearch process (meter_elasticsearch_node_process_cpu_percent).\nOS CPU (%) — host CPU usage on the node (meter_elasticsearch_node_os_cpu_percent).\nLoad Average — the node\u0026rsquo;s 1-minute, 5-minute, and 15-minute OS load averages (meter_elasticsearch_node_os_load1, meter_elasticsearch_node_os_load5, meter_elasticsearch_node_os_load15).\nJVM Memory (MB) — heap used, heap max, and non-heap used in MB (meter_elasticsearch_node_jvm_memory_heap_used, meter_elasticsearch_node_jvm_memory_heap_max, meter_elasticsearch_node_jvm_memory_nonheap_used).\nGC — garbage-collection activity on a dual axis: GC count on the left, GC time in ms/min on the right (meter_elasticsearch_node_jvm_gc_count, meter_elasticsearch_node_jvm_gc_time).\nTranslog — transaction-log operations and translog size in MB on a dual axis (meter_elasticsearch_node_indices_translog_operations, meter_elasticsearch_node_indices_translog_size).\nBreakers — tripped circuit breakers and the estimated breaker size in MB on this node (meter_elasticsearch_node_breakers_tripped, meter_elasticsearch_node_breakers_estimated_size).\nSegments — Lucene segment count and segment memory in MB on a dual axis (meter_elasticsearch_node_segment_count, meter_elasticsearch_node_segment_memory).\nDisk Usage — disk used in GB and disk-used percent on a dual axis (meter_elasticsearch_node_disk_usage, meter_elasticsearch_node_disk_usage_percent).\nNetwork — bytes sent and received on the node (meter_elasticsearch_node_network_send_bytes, meter_elasticsearch_node_network_receive_bytes).\nOpen Files — average open file-descriptor count on the node (meter_elasticsearch_node_open_file_count).\nEndpoint dashboard (index) For one selected index — indexing throughput, search throughput, size, and document counts.\nIndexing Rate — indexing requests vs. processed operations (meter_elasticsearch_index_stats_indexing_index_total_req_rate, meter_elasticsearch_index_stats_indexing_index_total_proc_rate).\nSearch Rate — search-query requests vs. processed operations (meter_elasticsearch_index_stats_search_query_total_req_rate, meter_elasticsearch_index_stats_search_query_total_proc_rate).\nIndex Size (all shards) — total store size of the index across all shards in GB (meter_elasticsearch_index_indices_store_size_bytes_total, latest).\nIndex Size (primary) — store size of the index\u0026rsquo;s primary shards in GB (meter_elasticsearch_index_indices_store_size_bytes_primary, latest).\nDocuments — document counts: all, primary, and deleted (meter_elasticsearch_index_indices_docs_total, meter_elasticsearch_index_indices_docs_primary, meter_elasticsearch_index_indices_deleted_docs_primary).\nAvg Search Time / Req (s) — average per-request time in seconds for each search phase: fetch, query, scroll, and suggest (meter_elasticsearch_index_search_fetch_avg_time, meter_elasticsearch_index_search_query_avg_time, meter_elasticsearch_index_search_scroll_avg_time, meter_elasticsearch_index_search_suggest_avg_time).\nRequirements The ELASTICSEARCH dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Elasticsearch monitoring enabled (see the upstream Elasticsearch monitoring setup), which feeds the three metric families this dashboard reads:\nCluster metrics — the meter_elasticsearch_cluster_* family (health, node count, shard states, pending tasks, tripped breakers, average CPU / JVM memory / open files) for the cluster list and the Service dashboard.\nNode metrics — the meter_elasticsearch_node_* family (process / OS CPU, load averages, JVM memory and GC, translog, breakers, segments, disk, network, open files) for the Instance dashboard.\nIndex metrics — the meter_elasticsearch_index_* family (indexing and search rates, store size, document counts, per-phase search times) for the Endpoint dashboard.\nEach metric is queried at its own OAP scope — cluster metrics at service scope, node metrics at instance scope, index metrics at endpoint scope. OAP does not roll a metric up across scopes, so a node- or index-scope widget stays empty until that level of data is reported.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/elasticsearch/","title":"\u003c!--"},{"body":" Envoy AI Gateway The ENVOY_AI_GATEWAY layer monitors Envoy AI Gateway deployments — the Envoy-based gateway that fronts LLM providers and models, routing chat / completion traffic to OpenAI, Anthropic, and other backends. SkyWalking turns the gateway\u0026rsquo;s OpenTelemetry GenAI signals into request, latency, token, and streaming-quality metrics, broken down by provider and model, and lands them here.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Gateways. Its services are listed as AI Gateways and its instances as Nodes. The ENVOY_AI_GATEWAY layer enables the Service (AI Gateway) and Instance (Node) dashboards plus the Logs sub-tab. It does not ship an Endpoint dashboard, a topology / service-map view, or a Traces tab — the gateway is monitored entirely through its GenAI meter families, which carry no per-endpoint scope or call graph.\nThis page is the operator reference for the bundled ENVOY_AI_GATEWAY dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled ENVOY_AI_GATEWAY template; if an operator has published a customized template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a gateway, the layer landing page lists every AI Gateway with four sortable columns, sorted by traffic (RPM) by default:\nRPM — requests per minute across the gateway (meter_envoy_ai_gw_request_cpm).\nAvg Latency — average request latency in ms (meter_envoy_ai_gw_request_latency_avg).\nInput Tokens/min — input (prompt) token throughput per minute (meter_envoy_ai_gw_input_token_rate).\nOutput Tokens/min — output (completion) token throughput per minute (meter_envoy_ai_gw_output_token_rate).\nService dashboard The primary drill-down for one selected AI Gateway. Beyond the headline request and token widgets, it breaks traffic down by GenAI provider and model and exposes streaming-quality timings (TTFT / TPOT). The Model Context Protocol (MCP) widgets only appear when the gateway actually serves MCP traffic.\nRequests, latency, and tokens\nRequest RPM — requests per minute for the gateway (meter_envoy_ai_gw_request_cpm).\nRequest Latency Avg — average request latency in ms (meter_envoy_ai_gw_request_latency_avg).\nInput Token Rate — input (prompt) tokens per minute (meter_envoy_ai_gw_input_token_rate).\nOutput Token Rate — output (completion) tokens per minute (meter_envoy_ai_gw_output_token_rate).\nRequest Latency Percentile — p50 / p75 / p90 / p95 / p99 request latency, the tail of the latency distribution, in ms (meter_envoy_ai_gw_request_latency_percentile).\nStreaming quality\nTTFT (Time to First Token) — time to first token for streaming responses, shown as both the average and the percentile distribution in ms (meter_envoy_ai_gw_ttft_avg, meter_envoy_ai_gw_ttft_percentile).\nTPOT (Time Per Output Token) — time per output token (inter-token latency) for streaming responses, shown as both the average and the percentile distribution in ms (meter_envoy_ai_gw_tpot_avg, meter_envoy_ai_gw_tpot_percentile).\nBy provider — each widget is split per gen_ai_provider_name, so every upstream LLM provider the gateway routes to gets its own series:\nRPM by Provider — requests per minute per provider (meter_envoy_ai_gw_provider_request_cpm).\nTokens by Provider — token throughput per provider (meter_envoy_ai_gw_provider_token_rate).\nLatency Avg by Provider — average latency per provider, in ms (meter_envoy_ai_gw_provider_latency_avg).\nBy model — each widget is split per gen_ai_response_model, so every model the gateway answered with gets its own series:\nRPM by Model — requests per minute per model (meter_envoy_ai_gw_model_request_cpm).\nTokens by Model — token throughput per model (meter_envoy_ai_gw_model_token_rate).\nLatency Avg by Model — average latency per model, in ms (meter_envoy_ai_gw_model_latency_avg).\nTTFT by Model — average time to first token per model, in ms (meter_envoy_ai_gw_model_ttft_avg).\nTPOT by Model — average time per output token per model, in ms (meter_envoy_ai_gw_model_tpot_avg).\nMCP (Model Context Protocol) — these widgets render only when the gateway serves MCP traffic; on a gateway that never sees MCP requests they stay hidden rather than showing empty:\nMCP RPM — MCP requests per minute (meter_envoy_ai_gw_mcp_request_cpm).\nMCP Avg Latency — average MCP request latency in ms (meter_envoy_ai_gw_mcp_request_latency_avg).\nMCP Error RPM — MCP errors per minute (meter_envoy_ai_gw_mcp_error_cpm).\nMCP by Method — MCP requests per minute split per mcp_method_name (meter_envoy_ai_gw_mcp_method_cpm).\nMCP by Backend — MCP requests per minute split per mcp_backend (meter_envoy_ai_gw_mcp_backend_request_cpm).\nInstance dashboard For one selected Node of the gateway. The same request, latency, token, and streaming-quality timings as the service view, scoped to a single gateway node.\nRequest RPM — requests per minute for this node (meter_envoy_ai_gw_instance_request_cpm).\nRequest Latency Avg — average request latency for this node, in ms (meter_envoy_ai_gw_instance_request_latency_avg).\nInput Token Rate — input (prompt) tokens per minute for this node (meter_envoy_ai_gw_instance_input_token_rate).\nOutput Token Rate — output (completion) tokens per minute for this node (meter_envoy_ai_gw_instance_output_token_rate).\nRequest Latency Percentile — p50 / p75 / p90 / p95 / p99 request latency for this node, in ms (meter_envoy_ai_gw_instance_request_latency_percentile).\nTTFT — time to first token for this node, shown as both the average and the percentile distribution in ms (meter_envoy_ai_gw_instance_ttft_avg, meter_envoy_ai_gw_instance_ttft_percentile).\nTPOT — time per output token for this node, shown as both the average and the percentile distribution in ms (meter_envoy_ai_gw_instance_tpot_avg, meter_envoy_ai_gw_instance_tpot_percentile).\nRequirements The ENVOY_AI_GATEWAY dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the Envoy AI Gateway GenAI meter families, derived from the gateway\u0026rsquo;s OpenTelemetry GenAI signals:\nGateway (service) metrics — the meter_envoy_ai_gw_* family at service scope: request load (meter_envoy_ai_gw_request_cpm), latency average and percentile (meter_envoy_ai_gw_request_latency_avg, meter_envoy_ai_gw_request_latency_percentile), input / output token rates (meter_envoy_ai_gw_input_token_rate, meter_envoy_ai_gw_output_token_rate), and the streaming-quality timings (meter_envoy_ai_gw_ttft_*, meter_envoy_ai_gw_tpot_*).\nPer-provider and per-model metrics — the meter_envoy_ai_gw_provider_* and meter_envoy_ai_gw_model_* families, labelled by gen_ai_provider_name and gen_ai_response_model, for the provider and model breakdown widgets.\nMCP metrics — the meter_envoy_ai_gw_mcp_* family (request, latency, error, per-method, per-backend), reported only when the gateway serves Model Context Protocol traffic; the MCP widgets stay hidden until these arrive.\nNode (instance) metrics — the meter_envoy_ai_gw_instance_* family for the per-node widgets (request load, latency, tokens, percentile, TTFT, TPOT).\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a node-scope metric is empty until that level of data is reported. For how to enable the upstream collector, see the Envoy AI Gateway monitoring setup docs.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/envoy_ai_gateway/","title":"\u003c!--"},{"body":" Flink The FLINK layer monitors an Apache Flink stream-processing cluster: the JobManager that coordinates the cluster, the TaskManagers that run the work, and the Flink jobs themselves. It is sourced from Flink\u0026rsquo;s metric reporter via OpenTelemetry, so the dashboard reads the same JVM, slot, network, and checkpoint metrics Flink already exposes.\nIn Horizon\u0026rsquo;s sidebar this layer is named Flink. Its three scopes are aliased to Flink\u0026rsquo;s own vocabulary: services are listed as Flink JobManagers, instances as TaskManagers, and endpoints as Jobs. The FLINK layer enables the Service, Instance, and Endpoint sub-tabs only — it ships no topology, no traces, and no logs.\nThis page is the operator reference for the bundled FLINK dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled FLINK template; if an operator has published a customized FLINK template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a JobManager, the layer landing page lists every Flink JobManager with four sortable columns, sorted by Running Jobs by default. Each column is the latest reported value:\nRunning Jobs — jobs currently running on this JobManager (meter_flink_jobManager_running_job_number).\nTaskManagers — TaskManagers registered with this JobManager (meter_flink_jobManager_taskManagers_registered_number).\nSlots Available — free task slots across the registered TaskManagers (meter_flink_jobManager_taskManagers_slots_available).\nSlots Total — total task slots across the registered TaskManagers (meter_flink_jobManager_taskManagers_slots_total).\nService dashboard The primary drill-down for one selected JobManager. The top row is four single-value cards, followed by the JobManager\u0026rsquo;s JVM health, GC behavior, and a running-jobs ranking.\nRunning Jobs — jobs currently running (meter_flink_jobManager_running_job_number).\nTaskManagers — registered TaskManagers (meter_flink_jobManager_taskManagers_registered_number).\nSlots Total — total task slots (meter_flink_jobManager_taskManagers_slots_total).\nSlots Available — free task slots (meter_flink_jobManager_taskManagers_slots_available).\nJM JVM CPU Load (%) — JobManager JVM CPU load (meter_flink_jobManager_jvm_cpu_load).\nJM JVM Thread Count — live JVM threads in the JobManager (meter_flink_jobManager_jvm_thread_count).\nJM CPU Time (ms) — JobManager JVM CPU time in ms (meter_flink_jobManager_jvm_cpu_time).\nJM Heap (MB) — JobManager heap memory, used vs available in MB (meter_flink_jobManager_jvm_memory_heap_used, meter_flink_jobManager_jvm_memory_heap_available).\nJM NonHeap (MB) — JobManager non-heap memory, used vs available in MB (meter_flink_jobManager_jvm_memory_nonHeap_used, meter_flink_jobManager_jvm_memory_nonHeap_available).\nJM Metaspace (MB) — JobManager metaspace, used vs available in MB (meter_flink_jobManager_jvm_memory_metaspace_used, meter_flink_jobManager_jvm_memory_metaspace_available).\nG1 Young GC — G1 young-generation collections on a dual axis: count on the left, time in ms on the right (meter_flink_jobManager_jvm_g1_young_generation_count, meter_flink_jobManager_jvm_g1_young_generation_time).\nG1 Old GC — G1 old-generation collections, count and time in ms on a dual axis (meter_flink_jobManager_jvm_g1_old_generation_count, meter_flink_jobManager_jvm_g1_old_generation_time).\nAll GC — all garbage collectors combined, count and time in ms on a dual axis (meter_flink_jobManager_jvm_all_garbageCollector_count, meter_flink_jobManager_jvm_all_garbageCollector_time).\nTop 10 Running Jobs — the ten jobs with the longest running time, ranked descending (meter_flink_job_runningTime).\nInstance dashboard For one selected TaskManager — the JVM health and network/back-pressure detail of a single worker.\nJVM CPU Load (%) — TaskManager JVM CPU load (meter_flink_taskManager_jvm_cpu_load).\nJVM Thread Count — live JVM threads in the TaskManager (meter_flink_taskManager_jvm_thread_count).\nCPU Time (ms) — TaskManager JVM CPU time in ms (meter_flink_taskManager_jvm_cpu_time).\nBack Pressured — whether the TaskManager is currently back-pressured (meter_flink_taskManager_isBackPressured).\nHeap (MB) — TaskManager heap memory, used vs available in MB (meter_flink_taskManager_jvm_memory_heap_used, meter_flink_taskManager_jvm_memory_heap_available).\nNonHeap (MB) — TaskManager non-heap memory, used vs available in MB (meter_flink_taskManager_jvm_memory_nonHeap_used, meter_flink_taskManager_jvm_memory_nonHeap_available).\nMetaspace (MB) — TaskManager metaspace, used vs available in MB (meter_flink_taskManager_jvm_memory_metaspace_used, meter_flink_taskManager_jvm_memory_metaspace_available).\nRecords In / Out — records read in and written out by the TaskManager (meter_flink_taskManager_numRecordsIn, meter_flink_taskManager_numRecordsOut).\nBytes In / Out / s — bytes per second read in and written out (meter_flink_taskManager_numBytesInPerSecond, meter_flink_taskManager_numBytesOutPerSecond).\nNetty Memory (MB) — Netty network-shuffle memory, used vs available in MB (meter_flink_taskManager_netty_usedMemory, meter_flink_taskManager_netty_availableMemory).\nPool Usage (%) — input vs output buffer-pool usage (meter_flink_taskManager_inPoolUsage, meter_flink_taskManager_outPoolUsage).\nBack-Pressure Time (ms/s) — per-second time the TaskManager spent in each state: soft back-pressure, hard back-pressure, idle, and busy (meter_flink_taskManager_softBackPressuredTimeMsPerSecond, meter_flink_taskManager_hardBackPressuredTimeMsPerSecond, meter_flink_taskManager_idleTimeMsPerSecond, meter_flink_taskManager_busyTimeMsPerSecond).\nEndpoint dashboard For one selected Job — its lifecycle timing, checkpoint behavior, and throughput. The top row is four single-value cards.\nJob Running Time (min) — how long the job has been running, in minutes (meter_flink_job_runningTime).\nJob Restarting Time (min) — time the job has spent restarting, in minutes (meter_flink_job_restartingTime).\nJob Cancelling Time (min) — time the job has spent cancelling, in minutes (meter_flink_job_cancellingTime).\nJob Restarts — number of job restarts (meter_flink_job_restart_number).\nCheckpoints — checkpoint counts over the window: total, completed, failed, and in-progress (meter_flink_job_checkpoints_total, meter_flink_job_checkpoints_completed, meter_flink_job_checkpoints_failed, meter_flink_job_checkpoints_inProgress).\nLast Checkpoint — the most recent checkpoint on a dual axis: size in bytes on the left, duration in ms on the right (meter_flink_job_lastCheckpointSize, meter_flink_job_lastCheckpointDuration).\nCurrent Emit Event Time Lag (ms) — lag between event time and emit time, in ms (meter_flink_job_currentEmitEventTimeLag).\nRecords In / Out — records read in and written out by the job (meter_flink_job_numRecordsIn, meter_flink_job_numRecordsOut).\nBytes In / Out / s — bytes per second read in and written out (meter_flink_job_numBytesInPerSecond, meter_flink_job_numBytesOutPerSecond).\nRequirements The FLINK dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the Flink meter families produced from Flink\u0026rsquo;s OpenTelemetry metric export:\nJobManager metrics — the meter_flink_jobManager_* family (running jobs, registered TaskManagers, slot totals, and JVM CPU / thread / memory / GC detail), driving the Service list and Service dashboard.\nTaskManager metrics — the meter_flink_taskManager_* family (JVM detail, record / byte throughput, Netty and buffer-pool usage, and back-pressure timing), driving the Instance dashboard.\nJob metrics — the meter_flink_job_* family (running / restarting / cancelling time, restarts, checkpoints, emit-time lag, and throughput), driving the Endpoint dashboard and the Top 10 Running Jobs ranking.\nEach metric is queried at its own OAP scope, and OAP does not roll a metric up across scopes — a JobManager-, TaskManager-, or Job-scope metric is empty until that level of data is reported. To set up the Flink metric reporter and the OAP receiver, follow the Flink monitoring setup guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/flink/","title":"\u003c!--"},{"body":" General Service The GENERAL layer is where SkyWalking\u0026rsquo;s language agents report. Any service instrumented by a SkyWalking native agent — Java, .NET (CLR), Go, Python, Ruby, Node.js, PHP, and the Spring Boot / Spring Sleuth meter integrations — lands here, so it is the most-used layer and the reference dashboard every other layer\u0026rsquo;s dashboard is modelled on.\nIn Horizon\u0026rsquo;s sidebar this layer is named General Service. Its services are listed as Services, instances as Instances (each badged with the agent language), and endpoints as API — the endpoint-to-endpoint view is called API dependency. The GENERAL layer enables the full set of sub-tabs: Service, Instance, Endpoint, API dependency, Topology, Traces, Logs, and the profiling tabs (trace, eBPF, async, pprof).\nThis page is the operator reference for the bundled GENERAL dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled GENERAL template; if an operator has published a customized GENERAL template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a service, the layer landing page lists every GENERAL service with three sortable columns, sorted by traffic (RPM) by default:\nRPM — calls per minute (service_cpm). Apdex — user-satisfaction score on a 0 – 1 scale (service_apdex/10000). Error Rate — percent of failed calls (100 - service_sla/100). Service dashboard The primary drill-down for one selected service.\nTop 20 APIs — the busiest endpoints under this service, with tabs to re-rank by Traffic (endpoint_cpm, rpm), Slow (endpoint_resp_time, ms), and Successful Rate (endpoint_sla, %, worst-first). Click a row to jump into that endpoint. Traffic — calls per minute for the service (service_cpm). Error Rate — percent of failed calls (100 - service_sla/100). Apdex — user-satisfaction score on a 0 – 1 scale (service_apdex/10000). Response Time Percentile — p50 / p75 / p90 / p95 / p99 latency, the tail of the response-time distribution (service_percentile). Avg Response Time — mean latency in ms (service_resp_time). MQ Consume rate + latency — message-queue consume count and latency on a dual axis: count on the left, latency on the right (service_mq_consume_count, service_mq_consume_latency). Top 10 instances by load — this service\u0026rsquo;s instances ranked by traffic (service_instance_cpm, rpm). Top 10 slowest instances — instances ranked by average response time (service_instance_resp_time, ms). Top 10 instances by success rate — instances ranked worst-first by success rate (service_instance_sla, %). Slow Database Statements — the 20 slowest sampled database statements captured against this service (top_n_service_database_statement, ms). Each row carries the statement text and, when the sample has one, a jump-to-trace link. Shows no data when OAP captured no statements in the window. The latency and error widgets ship with the metric-to-trace drill enabled: click a data point on Avg Response Time or Response Time Percentile to open the slowest traces at that moment, or on Error Rate or Apdex to open the error traces. The same drill rides the instance latency / success-rate widgets and the endpoint latency, percentile, success-rate, and MQ-latency widgets below. See Dashboard Widgets → Metric-to-trace drill.\nInstance dashboard For one selected service instance. The first three widgets always render; the rest are runtime-specific and appear only when the instance actually reports those metrics — so a Java instance shows the JVM family, a Go instance the Golang family, and so on, without manual configuration.\nAlways shown\nService Instance Load — calls per minute against the instance (service_instance_cpm). Service Instance Latency — average response time in ms (service_instance_resp_time). Service Instance Success Rate — percent of successful calls (service_instance_sla/100). JVM (Java instances)\nJVM CPU — JVM CPU as reported by the agent (instance_jvm_cpu). JVM Memory — heap and non-heap used / max in MB (instance_jvm_memory_heap, instance_jvm_memory_heap_max, instance_jvm_memory_noheap, instance_jvm_memory_noheap_max). JVM Memory Detail — per-pool used memory in MB: code cache, newgen, oldgen, survivor, permgen, metaspace, plus the newer JVM pools (zheap, compressed class space, and the segmented codeheaps). Pools a JVM doesn\u0026rsquo;t expose stay at 0. JVM Thread Count — live / daemon / peak threads. JVM Thread State Count — threads by state: runnable / blocked / waiting / timed-waiting. JVM GC Time — young / old / normal GC time in ms. JVM GC Count — young / old / normal GC counts. JVM Class Count — loaded / total-loaded / total-unloaded classes. CLR (.NET instances)\nCLR CPU — process CPU percentage (instance_clr_cpu). CLR Thread — worker-available, completion-port-available, and completion-port-max threads. CLR Heap Memory — managed heap in MB. CLR GC — gen 0 / gen 1 / gen 2 collection counts. Spring (Spring Boot Actuator / Spring Sleuth meters)\nSpring HTTP Request Count and Spring HTTP Request Duration — http.server.requests count and latency. Spring Instance CPU Usage / Spring OS CPU Usage / Spring OS System Load — process CPU, OS CPU, and 1-minute load average. Spring OS Process Files — open vs max file descriptors. Spring JVM GC Pause Duration, Spring JVM Memory (used / max), Spring JVM Threads (live / daemon / peak), Spring JVM Classes (loaded / unloaded). Spring Database Connection Pool (HikariCP / datasource), Spring Thread Pool, Spring JDBC Connections (active / idle / max), Spring Tomcat Sessions (active / max / rejected). Golang (Go instances)\nGolang Goroutines / OS Threads, Golang GC Pause Time, Golang GC Count, Golang Heap Alloc, Golang Goroutine Schedule Time, Golang GC Free, Golang Alloc Size, Golang Free Size, Golang Heap Objects, Golang Heap, Golang Metadata Mspan, Golang Metadata Mcache, Golang GC Goal Size, and Golang CGO Calls — the Go runtime\u0026rsquo;s goroutine, scheduler, GC, and heap detail. Python (PVM instances)\nPython CPU Utilization and Python Memory Utilization — host vs process. Python Thread Count, Python GC Count (gen 0 / 1 / 2), and Python GC Time. Ruby instances\nRuby CPU Usage, Ruby Memory (RSS), Ruby Memory Usage, Ruby Thread Status (active / running), Ruby GC Count (total / minor / major), Ruby GC Time, Ruby Heap Usage, and Ruby Heap Slots (live / available). Node.js instances\nProcess CPU — process CPU percentage (meter_instance_nodejs_process_cpu). V8 Heap Used / V8 Heap Total / V8 Heap Limit — the V8 heap in MB: currently used, currently allocated, and the maximum the heap may grow to (meter_instance_nodejs_heap_used / _heap_total / _heap_limit). Process RSS — resident set size in MB (meter_instance_nodejs_rss). External Memory — memory held outside the V8 heap (buffers and native objects) in MB (meter_instance_nodejs_external_memory). Array Buffers — ArrayBuffer / SharedArrayBuffer memory in MB (meter_instance_nodejs_array_buffers). Process Uptime — days since the process started (meter_instance_nodejs_uptime/86400). Peak Malloced Memory / Malloced Memory — peak and current V8 malloced memory in MB (meter_instance_nodejs_peak_malloced_memory / _malloced_memory). Old Space Used / New Space Used — V8 old / new generation heap used in MB (meter_instance_nodejs_old_space_used / _new_space_used). PHP (PHM) instances\nPHP CPU Utilization — process CPU percentage (meter_instance_php_process_cpu_utilization). PHP Memory Used and PHP Memory Peak — current and peak process memory in MB (meter_instance_php_memory_used_mb, meter_instance_php_memory_peak_mb). PHP Virtual Memory — virtual memory size in MB (meter_instance_php_virtual_memory_mb). PHP Thread Count — live threads (meter_instance_php_thread_count). PHP Open FDs — open file descriptors (meter_instance_php_open_fd_count). Endpoint dashboard For one selected endpoint (an API).\nTraffic — calls per minute for the endpoint (endpoint_cpm). Response Time — average latency in ms (endpoint_resp_time). Success Rate — percent of successful calls (endpoint_sla/100). Response Time Percentile — p50 / p75 / p90 / p95 / p99 latency for the endpoint (endpoint_percentile). MQ Avg Consuming Latency — consume latency in ms, shown only for endpoints that serve message-queue traffic (endpoint_mq_consume_latency). Topology and maps The GENERAL layer ships a full set of maps.\nTopology (service map) — every service node is decorated with RPM (service_cpm), an SLA health ring (service_sla/100, colored green / amber / red — higher is better, so the ring turns red as success rate drops), and Latency (service_resp_time). Each call edge carries server-side and client-side RPM, Avg response time, p95, and SLA (service_relation_server_* and service_relation_client_*), shown aligned in the edge panel.\nInstance map — from a call between two services on the service map, drill into the instance-to-instance calls between them. The same node / edge metric set is evaluated at instance scope (service_instance_* and service_instance_relation_server/client_*).\nAPI dependency (endpoint map) — the endpoint-to-endpoint dependency view. Each endpoint node shows RPM (endpoint_cpm), an SLA ring (endpoint_sla/100), and Latency (endpoint_resp_time); each edge shows RPM, Avg response time, p95, and SLA (endpoint_relation_*). Endpoint relations are server-side only.\nFor how these maps are read and navigated, see the 3D Infrastructure Map for the cross-layer view, and the topology section of Layer Dashboard Templates for how the node and edge metrics are configured.\nRequirements The GENERAL dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nService / instance / endpoint metrics — the service_*, service_instance_*, and endpoint_* families (traffic, response time, SLA, apdex, percentiles), produced by OAP from agent-reported traces or meters. Relation metrics — service_relation_*, service_instance_relation_*, and endpoint_relation_* for the service map, instance map, and API-dependency views. Runtime metrics, for the runtime-specific instance widgets to appear: JVM (instance_jvm_*), CLR (instance_clr_*), the Spring meter family (meter_*), and the Golang / Python / Ruby / Node.js / PHP agent meter families. An instance only shows the families its agent emits. Sampled records — top_n_service_database_statement for the Slow Database Statements list, captured by OAP when slow-statement sampling is enabled. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so an instance- or endpoint-scope metric is empty until that level of data is reported. When a whole family is missing (for example a non-JVM runtime), its widgets are hidden rather than shown empty.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/general/","title":"\u003c!--"},{"body":" iOS The IOS layer is where SkyWalking\u0026rsquo;s iOS client-side monitoring reports. An iOS app instrumented with the SkyWalking iOS SDK surfaces Apple MetricKit diagnostics — app launch time, hang time, abnormal exits, OOM kills, peak memory, scroll responsiveness, and network transfer — alongside the latency and success rate of the HTTP calls the app makes out to your backends. It sits in the Mobile group of layers.\nIn Horizon\u0026rsquo;s sidebar this layer is named iOS. Its services are listed as Apps, instances as App Sessions, and endpoints as Outbound APIs — these are the names you see on the picker and column headers. The IOS layer enables the Service, Instance, and Endpoint sub-tabs plus Logs. It has no Topology, Traces, or endpoint-dependency view — iOS reports client-side device telemetry and outbound calls, not a server-side call graph.\nThis page is the operator reference for the bundled IOS dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled IOS template; if an operator has published a customized IOS template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nApp list Before opening an app, the layer landing page lists every IOS app with four sortable columns, sorted by Launch (P95) by default:\nLaunch (P95) — 95th-percentile app launch time in ms (meter_ios_app_launch_time_percentile{p='95'}), the tail of how long the app takes to become usable.\nHang Time — total time the main thread spent hung in the window, in ms (meter_ios_hang_time_sum).\nCrashes — abnormal exits, foreground and background summed (meter_ios_foreground_abnormal_exit_count + meter_ios_background_abnormal_exit_count).\nOutbound RPM — calls per minute the app makes to backends (service_cpm).\nApp (service) dashboard The primary drill-down for one selected app.\nApp Launch Time Percentile — p50 / p75 / p90 / p95 / p99 launch time in ms, the distribution of how long the app takes to start (meter_ios_app_launch_time_percentile).\nHang Time Percentile — p50 / p75 / p90 / p95 / p99 main-thread hang time in ms (meter_ios_hang_time_percentile).\nHang Time (sum) — total hang time over the window, in ms (meter_ios_hang_time_sum).\nAbnormal Exits (Crashes) — abnormal exits split into foreground and background series; MetricKit reports the two separately (meter_ios_foreground_abnormal_exit_count, meter_ios_background_abnormal_exit_count).\nOOM Kill Count — background out-of-memory kills, the iOS system reaping the app under memory pressure (meter_ios_background_oom_kill_count).\nPeak Memory — peak resident memory in bytes (meter_ios_peak_memory).\nScroll Hitch Ratio — the fraction of scroll frames classified as hitched; higher means a laggier scrolling UI (meter_ios_scroll_hitch_ratio).\nNetwork Transfer — bytes transferred over wifi vs cellular, download and upload, as four series (meter_ios_wifi_download, meter_ios_wifi_upload, meter_ios_cellular_download, meter_ios_cellular_upload).\nOutbound HTTP — the calls this app makes to backends, on a dual axis: RPM and Avg latency (ms) on the left axis, Success Rate (%) on the right (service_cpm, service_resp_time, service_sla/100).\nApp Session (instance) dashboard For one selected app session — the same MetricKit and outbound-HTTP families evaluated at session scope.\nLaunch Time Percentile — p50 / p75 / p90 / p95 / p99 launch time in ms for the session (meter_ios_instance_app_launch_time_percentile).\nHang Time Percentile — p50 / p75 / p90 / p95 / p99 hang time in ms for the session (meter_ios_instance_hang_time_percentile).\nAbnormal Exits — foreground vs background abnormal exits for the session (meter_ios_instance_foreground_abnormal_exit_count, meter_ios_instance_background_abnormal_exit_count).\nOOM Kill Count — background OOM kills for the session (meter_ios_instance_background_oom_kill_count).\nPeak Memory — peak resident memory in bytes for the session (meter_ios_instance_peak_memory).\nOutbound HTTP — the session\u0026rsquo;s outbound calls on a dual axis: RPM and Avg latency (ms) on the left axis, Success Rate (%) on the right (service_instance_cpm, service_instance_resp_time, service_instance_sla/100).\nOutbound API (endpoint) dashboard For one selected Outbound API — a backend endpoint the app calls.\nOutbound Load — calls per minute to the endpoint (endpoint_cpm).\nOutbound Avg Latency — average call latency in ms (endpoint_resp_time).\nOutbound Success Rate — percent of successful calls (endpoint_sla/100).\nOutbound Latency Percentile — p50 / p75 / p90 / p95 / p99 latency for the endpoint (endpoint_percentile).\nRequirements The IOS dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\niOS MetricKit metrics — the meter_ios_* family at service scope and the meter_ios_instance_* family at session scope: launch-time and hang-time percentiles, hang-time sum, foreground / background abnormal exits, background OOM kills, peak memory, scroll hitch ratio, and wifi / cellular network transfer. These are produced by OAP from the SkyWalking iOS SDK\u0026rsquo;s MetricKit reports.\nOutbound HTTP metrics — the service_*, service_instance_*, and endpoint_* families (traffic, response time, SLA, percentiles), produced by OAP from the calls the app makes to instrumented backends.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a session- or endpoint-scope metric is empty until that level of data is reported. When a family is missing, its widgets read no data rather than being shown with fabricated values.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/ios/","title":"\u003c!--"},{"body":" Kubernetes The K8S layer monitors Kubernetes clusters and the nodes inside them. SkyWalking builds this layer from cluster-state and node-resource telemetry collected through OpenTelemetry (kube-state-metrics and the node / cAdvisor metric pipelines scraped into OAP) and reshapes it into cluster-wide and per-node metrics. In the sidebar it groups under Kubernetes.\nIn Horizon\u0026rsquo;s sidebar this layer\u0026rsquo;s entities are renamed to fit the Kubernetes model: services are listed as Clusters and instances as Nodes. The K8S layer enables the Service (Cluster) and Instance (Node) scopes only — there is no endpoint scope, no topology, and no traces or logs tab, because this layer reports cluster-state and node-resource metrics rather than request traffic.\nThis page is the operator reference for the bundled K8S dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled K8S template; if an operator has published a customized K8S template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nCluster list Before opening a cluster, the layer landing page lists every Kubernetes cluster with four sortable columns, sorted by Pods by default. Each column shows the latest reading summed across the cluster:\nPods — total pods in the cluster (k8s_cluster_pod_total).\nNodes — total nodes in the cluster (k8s_cluster_node_total).\nNamespaces — total namespaces in the cluster (k8s_cluster_namespace_total).\nDeployments — total deployments in the cluster (k8s_cluster_deployment_total).\nCluster dashboard The primary drill-down for one selected cluster (a Service in OAP terms). It opens with a row of count cards summarizing the cluster\u0026rsquo;s object inventory, then resource trends, then status tables that break the cluster down per node, deployment, service, and pod.\nInventory cards — each is a single latest count:\nNode Total — nodes in the cluster (latest(k8s_cluster_node_total)).\nNamespace Total — namespaces in the cluster (latest(k8s_cluster_namespace_total)).\nDeployment Total — deployments in the cluster (latest(k8s_cluster_deployment_total)).\nStatefulSet Total — statefulsets in the cluster (latest(k8s_cluster_statefulset_total)).\nDaemonSet Total — daemonsets in the cluster (latest(k8s_cluster_daemonset_total)).\nService Total — Kubernetes services in the cluster (latest(k8s_cluster_service_total)).\nPod Total — pods in the cluster (latest(k8s_cluster_pod_total)).\nContainer Total — containers in the cluster (latest(k8s_cluster_container_total)).\nResource trends — cluster-wide capacity vs. demand over time:\nCPU Resources — cluster CPU capacity against requests, limits, and allocatable, in millicores (k8s_cluster_cpu_cores, k8s_cluster_cpu_cores_requests, k8s_cluster_cpu_cores_limits, k8s_cluster_cpu_cores_allocatable).\nMemory Resources — cluster memory requests, allocatable, limits, and total, in GiB (k8s_cluster_memory_requests, k8s_cluster_memory_allocatable, k8s_cluster_memory_limits, k8s_cluster_memory_total).\nStorage Resources — cluster ephemeral-storage total against allocatable, in GiB (k8s_cluster_storage_total, k8s_cluster_storage_allocatable).\nStatus tables — each lists the entities currently matching the condition; they read no data when nothing matches:\nNode Status — per-node Kubernetes conditions currently true or unknown — Ready, the various Pressure conditions, and so on (latest(k8s_cluster_node_status)).\nDeployment Status — deployments reporting the Available condition (latest(k8s_cluster_deployment_status)).\nDeployment Spec Replicas — desired replica count per deployment (latest(k8s_cluster_deployment_spec_replicas)).\nService Status — pods backing each Kubernetes service, grouped by pod phase — Running / Pending / Failed and so on (latest(k8s_cluster_service_pod_status)).\nPod Status Not Running — pods in any non-Running phase (latest(k8s_cluster_pod_status_not_running)).\nPod Status Waiting — containers in a waiting state, grouped by the waiting reason (latest(k8s_cluster_pod_status_waiting)).\nNode dashboard For one selected node (an Instance in OAP terms) — its scheduling state and CPU / memory / network / storage resources.\nNode Status — the node\u0026rsquo;s current status as a single latest reading (latest(k8s_node_node_status)).\nPods on Node — pods scheduled on the node over time (k8s_node_pod_total).\nPod Total — the current count of pods scheduled on the node, as a single latest reading (latest(k8s_node_pod_total)).\nNode CPU Usage — node CPU usage in millicores (k8s_node_cpu_usage).\nNode CPU Resources — node CPU total against allocatable, requests, and limits, in millicores (k8s_node_cpu_cores, k8s_node_cpu_cores_allocatable, k8s_node_cpu_cores_requests, k8s_node_cpu_cores_limits).\nNode Memory Usage — node memory usage in GiB (k8s_node_memory_usage).\nNode Memory Resources — node memory total against allocatable, requests, and limits, in GiB (k8s_node_memory_total, k8s_node_memory_allocatable, k8s_node_memory_requests, k8s_node_memory_limits).\nNode Network I/O — node receive and transmit throughput in KB/s (k8s_node_network_receive, k8s_node_network_transmit).\nNode Storage Resources — node storage total against allocatable, in GiB (k8s_node_storage_total, k8s_node_storage_allocatable).\nRequirements The K8S dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the Kubernetes monitoring metric families, fed in through the OpenTelemetry receiver from kube-state-metrics and the node / cAdvisor pipelines:\nCluster metrics — the k8s_cluster_* family at cluster scope: object-inventory totals (node / namespace / deployment / statefulset / daemonset / service / pod / container), CPU / memory / storage capacity-and-demand series, and the per-node, per-deployment, per-service, and per-pod status breakdowns.\nNode metrics — the k8s_node_* family at node scope: node status, pod count, and CPU / memory / network / storage usage and resource series.\nEach metric is queried at its own OAP scope (Cluster / Node); OAP does not roll a metric up across scopes, so a node-scope metric stays empty until that level of data is reported. For how to stand up the Kubernetes-to-OAP pipeline, see the layer\u0026rsquo;s setup documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/k8s/","title":"\u003c!--"},{"body":" Kubernetes Services The K8S_SERVICE layer monitors the network behavior of Kubernetes services, observed at the kernel level by SkyWalking Rover\u0026rsquo;s eBPF probes. It captures the HTTP and TCP traffic flowing in and out of each service\u0026rsquo;s pods — call rate, latency, status codes, header / body sizes, packet counts, and connection activity — without instrumenting the application. It belongs to the Kubernetes layer group.\nIn Horizon\u0026rsquo;s sidebar this layer is named Kubernetes Services. Its services are listed as K8s services, instances as Pods, and endpoints as Endpoints — service names are grouped and displayed by their Kubernetes namespace. The K8S_SERVICE layer enables the Service, Pod, Endpoint, Topology, eBPF Profiling, Network Profiling, and Pod Logs sub-tabs. It does not enable an endpoint-dependency map, Traces, or Logs.\nThis page is the operator reference for the bundled K8S_SERVICE dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled K8S_SERVICE template; if an operator has published a customized K8S_SERVICE template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a service, the layer landing page lists every K8S_SERVICE service with four sortable columns, sorted by HTTP traffic (HTTP RPM) by default:\nPods — number of pods backing the service, summed from the latest reading (latest(k8s_service_pod_total)).\nHTTP RPM — HTTP calls per minute, summed across the service (kubernetes_service_http_call_cpm).\nLatency — average HTTP response time in ms (kubernetes_service_http_call_time).\nSuccess Rate — percent of successful HTTP calls (kubernetes_service_http_call_successful_rate/100).\nService dashboard The primary drill-down for one selected service. It mixes pod-lifecycle and resource widgets with the HTTP and TCP traffic the service\u0026rsquo;s pods carry.\nPods and resources\nService Pods — pod count over time (k8s_service_pod_total).\nPods Waiting — a table of containers currently in the Waiting state, keyed by container · pod · reason (latest(k8s_service_pod_status_waiting)).\nPod Restarts — a table of pods by cumulative restart count (latest(k8s_service_pod_status_restarts_total)).\nCPU Resources — requested vs. limited CPU in millicores, as two series (k8s_service_cpu_cores_requests, k8s_service_cpu_cores_limits).\nMemory Resources — requested vs. limited memory in MiB (k8s_service_memory_requests, k8s_service_memory_limits).\nPod CPU Usage — actual CPU consumed by the pods in millicores (k8s_service_pod_cpu_usage).\nPod Memory Usage — actual memory consumed by the pods in MiB (k8s_service_pod_memory_usage).\nHTTP traffic\nHTTP Request RPM — HTTP calls per minute for the service (kubernetes_service_http_call_cpm).\nHTTP Response Time — average HTTP response time in ms (kubernetes_service_http_call_time).\nHTTP Status Code RPM — calls per minute broken out by status class: 1xx / 2xx / 3xx / 4xx / 5xx (kubernetes_service_http_status_1xx_cpm … kubernetes_service_http_status_5xx_cpm).\nHTTP Request / Response Size — average request and response header and body sizes in KB, as four series (kubernetes_service_http_avg_req_header_size, kubernetes_service_http_avg_req_body_size, kubernetes_service_http_avg_resp_header_size, kubernetes_service_http_avg_resp_body_size).\nTCP traffic\nTCP Connect — client-side connect attempts and successes per minute, as two series (kubernetes_service_connect_cpm, kubernetes_service_connect_success_cpm).\nTCP Connect Duration — average connect time in ns (kubernetes_service_connect_time).\nTCP Accept — server-side accept events per minute (kubernetes_service_accept_cpm).\nTCP Packets — read, write, and write-retransmit packet counts per minute, as three series (kubernetes_service_read_package_cpm, kubernetes_service_write_package_cpm, kubernetes_service_write_retrains_package_cpm).\nTCP Bytes — read vs. write payload bytes per minute in KB (kubernetes_service_read_package_size, kubernetes_service_write_package_size).\nPod dashboard For one selected pod (a service instance). The widgets mirror the service view at instance scope, covering the pod\u0026rsquo;s HTTP and TCP traffic.\nPod HTTP RPM — HTTP calls per minute for the pod (kubernetes_service_instance_http_call_cpm).\nPod HTTP Response Time — average HTTP response time in ms (kubernetes_service_instance_http_call_time).\nPod HTTP Status Code RPM — calls per minute by status class: 1xx / 2xx / 3xx / 4xx / 5xx (kubernetes_service_instance_http_status_1xx_cpm … kubernetes_service_instance_http_status_5xx_cpm).\nPod HTTP Sizes — average request and response header and body sizes in KB, as four series (kubernetes_service_instance_http_avg_req_header_size, kubernetes_service_instance_http_avg_req_body_size, kubernetes_service_instance_http_avg_resp_header_size, kubernetes_service_instance_http_avg_resp_body_size).\nPod TCP Connect — client-side connect attempts and successes per minute (kubernetes_service_instance_connect_cpm, kubernetes_service_instance_connect_success_cpm).\nPod TCP Packets — read, write, and write-retransmit packet counts per minute (kubernetes_service_instance_read_package_cpm, kubernetes_service_instance_write_package_cpm, kubernetes_service_instance_write_retrains_package_cpm).\nPod TCP Bytes — read vs. write payload bytes per minute in KB (kubernetes_service_instance_read_package_size, kubernetes_service_instance_write_package_size).\nEndpoint dashboard For one selected endpoint. K8S_SERVICE endpoints carry HTTP traffic, so this scope is HTTP-focused.\nEndpoint HTTP RPM — HTTP calls per minute for the endpoint (kubernetes_service_endpoint_http_call_cpm).\nEndpoint HTTP Response Time — average HTTP response time in ms (kubernetes_service_endpoint_http_call_time).\nEndpoint Status Code RPM — calls per minute by status class: 1xx / 2xx / 3xx / 4xx / 5xx (kubernetes_service_endpoint_http_status_1xx_cpm … kubernetes_service_endpoint_http_status_5xx_cpm).\nEndpoint Request Sizes — average request header and body sizes in KB (kubernetes_service_endpoint_http_avg_req_header_size, kubernetes_service_endpoint_http_avg_req_body_size).\nEndpoint Response Sizes — average response header and body sizes in KB (kubernetes_service_endpoint_http_avg_resp_header_size, kubernetes_service_endpoint_http_avg_resp_body_size).\nTopology and maps The K8S_SERVICE layer ships a service topology with an instance-level drill-down.\nTopology (service map) — each service node is decorated with RPM (kubernetes_service_http_call_cpm), a Success Rate health ring (kubernetes_service_http_call_successful_rate/100, colored green / amber / red — higher is better, so the ring turns red as the success rate drops), and Latency (kubernetes_service_http_call_time). Each call edge carries server-side and client-side RPM (kubernetes_service_relation_server_http_call_cpm, kubernetes_service_relation_client_http_call_cpm) and Avg response time (kubernetes_service_relation_server_http_call_time, kubernetes_service_relation_client_http_call_time).\nInstance map — from a call between two services on the service map, drill into the pod-to-pod calls between them. Each instance node shows RPM (kubernetes_service_instance_http_call_cpm), a Success Rate ring (kubernetes_service_instance_http_call_successful_rate/100), and Latency (kubernetes_service_instance_http_call_time); each edge carries server-side and client-side RPM (kubernetes_service_instance_relation_server_http_call_cpm, kubernetes_service_instance_relation_client_http_call_cpm) and Avg response time (kubernetes_service_instance_relation_server_http_call_time, kubernetes_service_instance_relation_client_http_call_time).\nFor how these maps are read and navigated, see the 3D Infrastructure Map for the cross-layer view, and the topology section of Layer Dashboard Templates for how the node and edge metrics are configured.\neBPF Profiling, Network Profiling, and Pod Logs Because K8S_SERVICE data comes from eBPF probes, this layer also enables three investigation tabs alongside the dashboards:\neBPF Profiling — on-CPU / off-CPU profiling tasks targeted at a selected service, with the flame-graph and span-attached results SkyWalking Rover reports.\nNetwork Profiling — the process-to-process network conversations within the service, rendered as a process-level topology, captured by SkyWalking Rover on a selected pod.\nPod Logs — the container logs collected from the service\u0026rsquo;s pods, with the same filtering and search the logs surface provides elsewhere.\nThese tabs query their own data on demand and are independent of the metric widgets above.\nRequirements The K8S_SERVICE dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Kubernetes network monitoring enabled, with SkyWalking Rover\u0026rsquo;s eBPF probes feeding traffic telemetry and a Kubernetes metrics source feeding pod / resource state. Specifically:\nPod and resource metrics — the k8s_service_* family (pod totals, waiting / restart status, CPU and memory requests / limits, and actual pod CPU / memory usage) for the service-scope lifecycle and resource widgets.\nHTTP metrics — the kubernetes_service_http_*, kubernetes_service_instance_http_*, and kubernetes_service_endpoint_http_* families covering call counts, response time, success rate, status classes, and header / body sizes, at their respective service / instance / endpoint scopes.\nTCP metrics — the kubernetes_service_* and kubernetes_service_instance_* connect, accept, packet, and byte families for the L4 widgets.\nRelation metrics — kubernetes_service_relation_server/client_http_* and kubernetes_service_instance_relation_server/client_http_* for the service map and instance map edges.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so an instance- or endpoint-scope metric is empty until that level of data is reported. For setup, see the Kubernetes network monitoring guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/k8s_service/","title":"\u003c!--"},{"body":" Kafka The KAFKA layer monitors Apache Kafka clusters. SkyWalking reads Kafka\u0026rsquo;s JMX metrics (via OpenTelemetry\u0026rsquo;s Kafka receiver or an equivalent collector) and turns them into per-cluster and per-broker meters, so this dashboard is a JMX-derived view of cluster health, partition / replication state, and broker throughput rather than agent-traced request data.\nIn Horizon\u0026rsquo;s sidebar this layer is named Kafka, grouped under MQ. Its services are listed as Kafka clusters and its instances as Brokers. The KAFKA layer enables only the Service (cluster) and Instance (broker) sub-tabs — it ships no endpoint scope, no topology, and no traces or logs, because Kafka\u0026rsquo;s JMX feed is metrics-only.\nThis page is the operator reference for the bundled KAFKA dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled KAFKA template; if an operator has published a customized KAFKA template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nCluster list Before opening a cluster, the layer landing page lists every Kafka cluster with four sortable columns, sorted by Partitions by default:\nPartitions — average partition count across the cluster (meter_kafka_partition_count).\nOffline Partitions — total partitions with no active leader, summed across the cluster (meter_kafka_offline_partitions_count). A non-zero value means data on those partitions is currently unavailable.\nMax Lag — the worst replica lag observed (meter_kafka_max_lag), the maximum of how far any follower trails its leader.\nLeaders — total partition leaders hosted across the cluster (meter_kafka_leader_count).\nCluster dashboard The primary drill-down for one selected Kafka cluster — the cluster-wide controller and partition health view.\nPartition Count — total partitions in the cluster (meter_kafka_partition_count).\nLeader Count — partition leaders in the cluster (meter_kafka_leader_count).\nActive Controllers — the count of active controllers (meter_kafka_active_controller_count). A healthy cluster has exactly one; zero or more than one signals a controller problem.\nMax Lag — the worst replica lag across the cluster (meter_kafka_max_lag).\nUnder-Replicated Partitions — partitions that have fewer in-sync replicas than configured (meter_kafka_under_replicated_partitions). Sustained non-zero values indicate replication is falling behind.\nOffline Partitions — partitions with no active leader (meter_kafka_offline_partitions_count).\nLeader Election Rate — partition-leader elections per second, split into two series: normal elections (meter_kafka_leader_election_rate) and unclean elections (meter_kafka_unclean_leader_elections_per_second). Unclean elections promote an out-of-sync replica and can lose data, so they should stay at zero.\nBroker dashboard For one selected broker. The widgets cover the broker\u0026rsquo;s CPU and memory, message and byte throughput, request handling, queue timings, replication, and partition/ISR state.\nCPU Usage — broker CPU percentage (meter_kafka_broker_cpu_time_total).\nIncoming Messages / s — messages produced into the broker per second (meter_kafka_broker_messages_per_second).\nBytes In / s — inbound throughput in bytes per second (meter_kafka_broker_bytes_in_per_second).\nBytes Out / s — outbound throughput in bytes per second (meter_kafka_broker_bytes_out_per_second).\nRequests / s — total requests handled per second (meter_kafka_broker_requests_per_second).\nPurgatory Size — requests parked in the broker\u0026rsquo;s request purgatory awaiting completion (meter_kafka_broker_purgatory_size).\nISR Shrinks/s — the latest rate at which in-sync-replica sets are shrinking (latest(meter_kafka_broker_isr_shrinks_per_second)), shown as a single number. Frequent shrinks mean replicas are repeatedly dropping out of sync.\nISR Expands/s — the latest rate at which in-sync-replica sets are re-expanding (latest(meter_kafka_broker_isr_expands_per_second)), shown as a single number.\nMemory Usage (%) — broker memory utilization (meter_kafka_broker_memory_usage_percentage).\nUnder-Replicated Partitions — the latest count of under-replicated partitions on this broker (latest(meter_kafka_broker_under_replicated_partitions)), shown as a single number.\nUnder Min-ISR Partitions — the latest count of partitions below their minimum in-sync-replica threshold on this broker (latest(meter_kafka_broker_under_min_isr_partition_count)), shown as a single number. These partitions reject produces under the default acks setting.\nPartitions + Leaders — partitions hosted on the broker (meter_kafka_broker_partition_count) overlaid with the partitions it currently leads (meter_kafka_broker_leader_count).\nQueue / Send Times — broker request latency breakdown in ms across four stages: request q (meter_kafka_broker_request_queue_time_ms), response q (meter_kafka_broker_response_queue_time_ms), response send (meter_kafka_broker_response_send_time_ms), and remote (meter_kafka_broker_remote_time_ms).\nTopic Rates — per-broker topic activity: produce req/s (meter_kafka_broker_topic_produce_requests_per_second), fetch req/s (meter_kafka_broker_topic_fetch_requests_per_second), and bytes-in/s (meter_kafka_broker_topic_bytesin_per_second).\nReplication — replication traffic in bytes per second between brokers, bytes in (meter_kafka_broker_replication_bytes_in_per_second) and bytes out (meter_kafka_broker_replication_bytes_out_per_second).\nGC Count — garbage-collection count for the broker JVM (meter_kafka_broker_garbage_collector_count).\nMax Lag (broker) — the broker\u0026rsquo;s total replica lag (sum(meter_kafka_broker_max_lag)), shown as a single number — the sum of how far this broker\u0026rsquo;s followers trail their leaders.\nRequirements The KAFKA dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Kafka\u0026rsquo;s JMX metrics ingested and aggregated into the two meter families this layer renders:\nCluster meters — the meter_kafka_* family at Service scope (partition, leader, controller, lag, under-replicated / offline partition, and leader-election metrics) for the cluster list and cluster dashboard.\nBroker meters — the meter_kafka_broker_* family at ServiceInstance scope (CPU, memory, message / byte / request throughput, purgatory, ISR, queue and send times, topic rates, replication, GC, and per-broker partition / leader / lag metrics) for the broker dashboard.\nThese meters are produced by SkyWalking\u0026rsquo;s Kafka monitoring, which reads Kafka\u0026rsquo;s JMX through OpenTelemetry\u0026rsquo;s Kafka receiver. See the Kafka monitoring setup for how to wire the collector to OAP. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a broker-scope meter is empty until per-broker data is reported.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/kafka/","title":"\u003c!--"},{"body":" Kong The KONG layer monitors Kong API gateways. Kong exposes its built-in Prometheus metrics, OpenTelemetry collects and forwards them to OAP, and OAP aggregates them into the meter_kong_* families this dashboard renders. The layer key is KONG, and in Horizon\u0026rsquo;s sidebar it is grouped under Gateways.\nIn the sidebar this layer\u0026rsquo;s services are listed as Kong services, its instances as Nodes (the individual Kong data-plane nodes), and its endpoints as Routes (the matched Kong routes). The KONG layer enables three scopes — Service, Instance (Node), and Endpoint (Route). It does not ship a topology, traces, or logs tab, so this dashboard is purely the metric drill-down across those three scopes.\nThis page is the operator reference for the bundled KONG dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled KONG template; if an operator has published a customized KONG template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a service, the layer landing page lists every Kong service, sorted by request rate (RPS) by default, with four columns:\nRPS — total HTTP requests per second across the service\u0026rsquo;s nodes (aggregate_labels(meter_kong_service_http_requests,sum)).\n200/s — 200-status responses per second (aggregate_labels(meter_kong_service_http_status{code='200'}, sum)).\n404/s — 404-status responses per second (aggregate_labels(meter_kong_service_http_status{code='404'}, sum)).\n500/s — 500-status responses per second (aggregate_labels(meter_kong_service_http_status{code='500'}, sum)).\nThe three status columns give an at-a-glance health read across the fleet — a service whose 500/s is climbing next to its 200/s is failing requests upstream.\nService dashboard The primary drill-down for one selected Kong service. Every widget aggregates across the service\u0026rsquo;s nodes.\nHTTP Request Trend — total requests per second for the service (aggregate_labels(meter_kong_service_http_requests,sum)).\nHTTP Status Trend — requests per second broken down by HTTP status code (aggregate_labels(meter_kong_service_http_status,sum(code)), one line per code).\nHTTP Bandwidth — ingress / egress bandwidth in KB/s, by direction (aggregate_labels(meter_kong_service_http_bandwidth,sum(direction)), divided to KB).\nKong Latency — the time spent inside Kong itself (plugins and routing), in ms, averaged across percentiles (aggregate_labels(meter_kong_service_kong_latency,avg(p))).\nRequest Latency — total request latency in ms — the time as seen by the client, averaged across percentiles (aggregate_labels(meter_kong_service_request_latency,avg(p))).\nUpstream Latency — the time spent waiting on the upstream service Kong proxies to, in ms, averaged across percentiles (aggregate_labels(meter_kong_service_upstream_latency,avg(p))). Comparing Kong Latency, Request Latency, and Upstream Latency tells you whether added latency is coming from the gateway or from the backend behind it.\nNginx Connections — Kong\u0026rsquo;s underlying Nginx connections by state (aggregate_labels(meter_kong_service_nginx_connections_total,sum(state)), one line per state).\nNginx Timers — Nginx timers by state — running vs pending (aggregate_labels(meter_kong_service_nginx_timers,sum(state)), one line per state).\nDatastore Reachable — a per-instance table of whether each node can reach Kong\u0026rsquo;s datastore (latest(aggregate_labels(meter_kong_service_datastore_reachable,sum(service_instance_id)))), with Instance and Reachable columns. A node that can\u0026rsquo;t reach the datastore is no longer receiving config updates.\nNginx Metric Errors — the latest count of errors Kong hit while exporting its own Nginx metrics (latest(aggregate_labels(meter_kong_service_nginx_metric_errors_total,sum))), shown as a single number — a non-zero value means metric collection on that service is degraded.\nInstance dashboard For one selected node. These widgets are reported per node, so they show how one Kong data-plane process is behaving.\nHTTP Request Trend — requests per second for the node (meter_kong_instance_http_requests).\nHTTP Status Trend — requests per second by status code for the node (meter_kong_instance_http_status).\nHTTP Bandwidth — bandwidth in KB/s for the node (meter_kong_instance_http_bandwidth, divided to KB).\nKong Latency — time spent inside Kong for the node, in ms (meter_kong_instance_kong_latency).\nRequest Latency — total request latency for the node, in ms (meter_kong_instance_request_latency).\nUpstream Latency — upstream wait time for the node, in ms (meter_kong_instance_upstream_latency).\nDatastore Reachable — the latest datastore-reachability reading for the node, shown as a single number (latest(meter_kong_instance_datastore_reachable)).\nNginx Connections — Nginx connections by state for the node (meter_kong_instance_nginx_connections_total).\nNginx Timers — Nginx timers by state for the node (meter_kong_instance_nginx_timers).\nShared Memory Usage — how full the node\u0026rsquo;s Nginx shared-memory dictionaries are, as a percentage of total (meter_kong_instance_shared_dict_bytes over meter_kong_instance_shared_dict_total_bytes). When this approaches 100% the node can no longer cache new entries.\nWorker Lua VM Usage — memory used by the worker processes\u0026rsquo; Lua VMs, in MB (meter_kong_instance_memory_workers_lua_vms_bytes, divided to MB).\nEndpoint dashboard For one selected route. Kong reports a tighter metric set at route scope — status, bandwidth, and the three latency views.\nHTTP Status Trend — requests per second by status code for the route (meter_kong_endpoint_http_status).\nTotal Bandwidth — ingress / egress bandwidth in KB/s for the route (meter_kong_endpoint_http_bandwidth, divided to KB).\nKong Latency — time spent inside Kong for the route, in ms (meter_kong_endpoint_kong_latency).\nRequest Latency — total request latency for the route, in ms (meter_kong_endpoint_request_latency).\nUpstream Latency — upstream wait time for the route, in ms (meter_kong_endpoint_upstream_latency).\nRequirements The KONG dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Kong metrics flowing in through the OpenTelemetry receiver:\nService metrics — the meter_kong_service_* family (requests, status, bandwidth, the Kong / request / upstream latency trio, Nginx connections and timers, datastore reachability, and the Nginx metric-error counter), aggregated by OAP across a service\u0026rsquo;s nodes.\nInstance (node) metrics — the meter_kong_instance_* family, including the node-only meter_kong_instance_shared_dict_* and meter_kong_instance_memory_workers_lua_vms_bytes health metrics.\nEndpoint (route) metrics — the meter_kong_endpoint_* family for the per-route status, bandwidth, and latency widgets.\nThese come from Kong\u0026rsquo;s Prometheus plugin scraped by an OpenTelemetry Collector and forwarded to OAP, which converts them into the meter_kong_* metrics through its meter-analysis rules. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a route-scope or node-scope metric is empty until that level of data is reported. For the end-to-end collector and OAP setup, see the Kong monitoring backend guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/kong/","title":"\u003c!--"},{"body":" Istio Managed Services The MESH layer is where SkyWalking observes services running inside an Istio service mesh. Telemetry comes from the Envoy sidecars via Envoy\u0026rsquo;s Access Log Service (ALS), so a service does not need a language agent to appear here — Envoy reports the traffic, latency, and Envoy-runtime metrics on its behalf. This makes MESH the natural home for any workload managed by Istio, instrumented or not.\nIn Horizon\u0026rsquo;s sidebar this layer is named Istio Managed Services. Its services are listed as Services, instances as Sidecars (one per Envoy proxy), and endpoints as Endpoints. Service names follow the Istio service.namespace convention, so the namespace is surfaced as a grouping value alongside the service name. The MESH layer enables the Service, Instance (Sidecar), Endpoint, Topology, Traces, and Logs sub-tabs, plus eBPF profiling, network profiling, and pod logs. There is no endpoint-to-endpoint dependency map for this layer.\nThis page is the operator reference for the bundled MESH dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled MESH template; if an operator has published a customized MESH template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a service, the layer landing page lists every MESH service with four sortable columns, sorted by traffic (Traffic) by default:\nTraffic — calls per minute observed by the sidecars (service_cpm).\nApdex — user-satisfaction score on a 0 – 1 scale (service_apdex/10000).\nLatency — average response time in ms (service_resp_time).\nError Rate — percent of failed calls (100 - service_sla/100).\nService dashboard The primary drill-down for one selected service.\nTop 20 APIs — the busiest endpoints under this service, with tabs to re-rank by Traffic (endpoint_cpm, rpm), Slow (endpoint_resp_time, ms), and Successful Rate (endpoint_sla, %, worst-first). Click a row to jump into that endpoint.\nTraffic — mesh-wide requests per minute observed by the Envoy sidecars (service_cpm).\nError Rate — percent of failed calls (100 - service_sla/100).\nApdex — user-satisfaction score on a 0 – 1 scale (service_apdex/10000).\nAvg Response Time — mean latency in ms (service_resp_time).\nResponse Time Percentile — p50 / p75 / p90 / p95 / p99 latency, the tail of the response-time distribution (service_percentile).\nService Throughput — bytes per minute through the sidecar, received and sent on the same chart (service_throughput_received, service_throughput_sent).\nSidecar Internal Latency — Envoy-internal request and response latency in nanoseconds. Useful when you suspect the sidecar itself is adding overhead (service_sidecar_internal_req_latency_nanos, service_sidecar_internal_resp_latency_nanos).\nTop 10 sidecars — this service\u0026rsquo;s sidecar instances ranked across three tabs: Traffic (service_instance_cpm, rpm), Slow (service_instance_resp_time, ms), and Successful Rate (service_instance_sla, %, worst-first).\nInstance dashboard For one selected sidecar instance. The first five widgets always render; the Envoy-runtime widgets that follow appear only when the sidecar actually reports those metrics, so a non-Envoy or partially-instrumented sidecar simply shows fewer panels.\nAlways shown\nSidecar Load — calls per minute against the selected sidecar instance (service_instance_cpm).\nSidecar Latency — average response time in ms (service_instance_resp_time).\nSidecar Success Rate — percent of successful calls (service_instance_sla/100).\nSidecar Throughput — bytes through the sidecar, received and sent (service_instance_throughput_received, service_instance_throughput_sent).\nSidecar Internal Latency — Envoy-internal request and response latency in nanoseconds (service_instance_sidecar_internal_req_latency_nanos, service_instance_sidecar_internal_resp_latency_nanos).\nEnvoy runtime (shown when the sidecar reports it)\nEnvoy Upstream Request Active — in-flight upstream requests per cluster (envoy_cluster_up_rq_active).\nEnvoy Upstream Request Increase — upstream requests added per minute (envoy_cluster_up_rq_incr).\nEnvoy Upstream Pending Active — pending upstream requests, a sign of connection-pool back-pressure (envoy_cluster_up_rq_pending_active).\nEnvoy Upstream Connection Active — active upstream connections per cluster (envoy_cluster_up_cx_active).\nEnvoy Upstream Connection Increase — upstream connections added per minute (envoy_cluster_up_cx_incr).\nEnvoy Cluster Healthy Membership — healthy upstream members per cluster; a non-trivial drop signals upstream churn (envoy_cluster_membership_healthy).\nEnvoy Total Connections — total vs parent connections in use (envoy_total_connections_used, envoy_parent_connections_used).\nEnvoy Heap Memory — Envoy memory in MB: heap used / max, allocated used / max, and physical size / max (envoy_heap_memory_used, envoy_heap_memory_max_used, envoy_memory_allocated, envoy_memory_allocated_max, envoy_memory_physical_size, envoy_memory_physical_size_max).\nEnvoy Worker Threads — live vs max worker threads (envoy_worker_threads, envoy_worker_threads_max).\nEnvoy Bug Failures — Envoy\u0026rsquo;s own assertion / bug counter; expected to be zero in healthy clusters (envoy_bug_failures).\nEndpoint dashboard For one selected endpoint.\nEndpoint Traffic — calls per minute for the endpoint (endpoint_cpm).\nEndpoint Avg Response Time — average latency in ms (endpoint_resp_time).\nEndpoint Success Rate — percent of successful calls (endpoint_sla/100).\nEndpoint Response Time Percentile — p50 / p75 / p90 / p95 / p99 latency for the endpoint (endpoint_percentile).\nSidecar Internal Latency (endpoint scope) — Envoy-internal request and response latency on this endpoint, in nanoseconds (endpoint_sidecar_internal_req_latency_nanos, endpoint_sidecar_internal_resp_latency_nanos).\nTopology and maps The MESH layer ships the service map and the instance (sidecar) map. There is no endpoint-dependency map for this layer.\nTopology (service map) — every service node is decorated with RPM (service_cpm), an SLA health ring (service_sla/100, colored green / amber / red — higher is better, so the ring turns red as success rate drops), and Latency (service_resp_time). Each call edge carries server-side and client-side RPM, Avg response time, p95, and SLA (service_relation_server_* and service_relation_client_*), shown aligned in the edge panel.\nInstance map (sidecars) — from a call between two services on the service map, drill into the sidecar-to-sidecar calls between them. The same node / edge metric set is evaluated at instance scope: node RPM (service_instance_cpm), SLA ring (service_instance_sla/100), and Latency (service_instance_resp_time); each edge shows server-side and client-side RPM, Avg response time, p95, and SLA (service_instance_relation_server/client_*).\nFor how these maps are read and navigated, see the 3D Infrastructure Map for the cross-layer view, and the topology section of Layer Dashboard Templates for how the node and edge metrics are configured.\nTraces MESH traces are served from Zipkin. Open the Traces tab to query the sidecar-reported spans; the workflow and filters are the same as any other layer\u0026rsquo;s trace view — see Traces.\nRequirements The MESH dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nEnvoy ALS enabled — the sidecars must stream access logs to OAP via the Access Log Service so it can derive the service / instance / endpoint traffic, latency, and SLA metrics. See the Envoy ALS setup guide.\nService / instance / endpoint metrics — the service_*, service_instance_*, and endpoint_* families (traffic, response time, SLA, apdex, percentiles), including the mesh-specific *_throughput_* and *_sidecar_internal_*_latency_nanos metrics produced from the ALS stream.\nRelation metrics — service_relation_* and service_instance_relation_* for the service map and the sidecar map.\nEnvoy-runtime metrics — the envoy_cluster_*, envoy_*_connections_used, envoy_*_memory_*, envoy_worker_threads*, and envoy_bug_failures families, for the Envoy-runtime instance widgets to appear. A sidecar only shows the families its Envoy build emits.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so an instance- or endpoint-scope metric is empty until that level of data is reported. When a whole family is missing (for example a sidecar that does not export Envoy-runtime metrics), its widgets are hidden rather than shown empty.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/mesh/","title":"\u003c!--"},{"body":" Istio Control Plane The MESH_CP layer monitors the Istio control plane — the istiod / Pilot process that distributes configuration to the data-plane proxies. SkyWalking scrapes the control-plane\u0026rsquo;s Prometheus metrics over OpenTelemetry and rolls them into per-control-plane meters, so operators can watch xDS push health, proxy convergence, configuration validation, and the Go runtime of istiod itself. See the upstream Istio monitoring setup for how to wire the telemetry pipeline.\nIn Horizon\u0026rsquo;s sidebar this layer is named Istio Control Plane (grouped under Istio). Its services are listed as Control Planes — each control plane is named service.namespace, with the namespace shown as its grouping alias. The MESH_CP layer enables only the Service sub-tab; it has no instance, endpoint, topology, traces, or logs view.\nThis page is the operator reference for the bundled MESH_CP dashboard: what you see on the service scope and what each widget means.\nThe widgets and metrics below are read from the bundled MESH_CP template; if an operator has published a customized MESH_CP template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a control plane, the layer landing page lists every Istio control plane with four sortable columns, sorted by CPU by default:\nCPU — average CPU usage of the control-plane process (meter_istio_cpu).\nGoroutines — total goroutines running in istiod (meter_istio_go_goroutines).\nPilot xDS — total xDS connections Pilot is serving (meter_istio_pilot_xds).\nServices — total services Pilot knows about (meter_istio_pilot_services).\nService dashboard The primary drill-down for one selected control plane, covering its Go runtime, xDS push pipeline, configuration validation, and proxy conflicts.\nCPU — CPU usage of the control-plane process over time (meter_istio_cpu).\nGoroutines — goroutines running in istiod (meter_istio_go_goroutines).\nIstio Versions — the reported Pilot / Istio version build info, so a version roll-out is visible on the timeline (meter_istio_pilot_version).\nMemory (MB) — the Go runtime\u0026rsquo;s memory footprint on one chart, in MB: allocated, heap in-use, stack in-use, virtual, and resident (meter_istio_go_alloc, meter_istio_go_heap_inuse, meter_istio_go_stack_inuse, meter_istio_virtual_memory, meter_istio_resident_memory, each /1024/1024).\nPilot Errors — xDS rejections and push timeouts that indicate the control plane could not deliver config: CDS / EDS / RDS / LDS rejects plus write timeouts (meter_istio_pilot_xds_cds_reject, meter_istio_pilot_xds_eds_reject, meter_istio_pilot_xds_rds_reject, meter_istio_pilot_xds_lds_reject, meter_istio_pilot_xds_write_timeout).\nProxy Push Time (percentile) — how long it takes to push config to proxies, as a latency percentile distribution in ms (meter_istio_pilot_proxy_push_percentile).\nPilot Pushes — the rate of xDS pushes Pilot sends to proxies (meter_istio_pilot_xds_pushes).\nSidecar Injection Success — successful sidecar-injection webhook calls (meter_istio_sidecar_injection_success_total).\nADS Monitoring — the aggregated discovery surface on one chart: xDS connections, known services, and virtual services (meter_istio_pilot_xds, meter_istio_pilot_services, meter_istio_pilot_virt_services).\nConfiguration Validation — Galley configuration-validation outcomes, passed vs failed (meter_istio_galley_validation_passed, meter_istio_galley_validation_failed).\nPilot Conflicts — listener conflicts Pilot detected while generating config, broken out by type: outbound TCP/TCP, inbound, outbound TCP/HTTP, and outbound HTTP/TCP (meter_istio_pilot_conflict_ol_tcp_tcp, meter_istio_pilot_conflict_il, meter_istio_pilot_conflict_ol_tcp_http, meter_istio_pilot_conflict_ol_http_tcp).\nRequirements The MESH_CP dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the Istio control-plane meter family, produced from istiod\u0026rsquo;s Prometheus metrics collected over OpenTelemetry:\nControl-plane metrics — the meter_istio_* family for the service list and the service dashboard: the Go runtime (meter_istio_cpu, meter_istio_go_goroutines, the meter_istio_go_* memory gauges, meter_istio_virtual_memory, meter_istio_resident_memory), the Pilot / xDS pipeline (meter_istio_pilot_xds, meter_istio_pilot_xds_pushes, the meter_istio_pilot_xds_*_reject rejection counters, meter_istio_pilot_xds_write_timeout, meter_istio_pilot_proxy_push_percentile, meter_istio_pilot_services, meter_istio_pilot_virt_services, meter_istio_pilot_version, the meter_istio_pilot_conflict_* counters), sidecar injection (meter_istio_sidecar_injection_success_total), and Galley validation (meter_istio_galley_validation_passed, meter_istio_galley_validation_failed). All MESH_CP metrics are reported at the control-plane Service scope; OAP does not roll a metric up across scopes, so the dashboard stays empty until the control plane\u0026rsquo;s telemetry is reported. See the upstream Istio monitoring setup for the collection pipeline that produces this family.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/mesh_cp/","title":"\u003c!--"},{"body":" Istio Data Plane The MESH_DP layer monitors the Istio data plane — the Envoy sidecar proxies that carry the mesh\u0026rsquo;s traffic. Where the service-mesh control-plane and request telemetry live in the MESH layer, MESH_DP is the proxy\u0026rsquo;s own view: the runtime health of each Envoy process and its upstream clusters, fed by Envoy\u0026rsquo;s metrics-service output. It is grouped under Istio in the sidebar.\nIn Horizon\u0026rsquo;s sidebar this layer\u0026rsquo;s entities are the Envoy sidecars themselves. Its top-level entities are listed as Sidecar services, and each sidecar process is a Sidecars instance — there is no separate per-application service or endpoint slot, so MESH_DP reads \u0026ldquo;Sidecar service / Sidecar\u0026rdquo; rather than the GENERAL layer\u0026rsquo;s \u0026ldquo;Service / Instance / Endpoint\u0026rdquo;. Sidecar names follow the Istio name.namespace convention, so the namespace is surfaced as the displayed grouping.\nMESH_DP enables the Sidecar (instance) dashboard plus the Logs and eBPF profiling tabs; pod logs are available for the sidecar. It does not ship a service dashboard, an endpoint dashboard, a topology / map view, or a traces tab — the layer is scoped to per-sidecar runtime metrics, so those sections are absent.\nThis page is the operator reference for the bundled MESH_DP dashboard: what you see on the sidecar scope and what each widget means.\nThe widgets and metrics below are read from the bundled MESH_DP template; if an operator has published a customized MESH_DP template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nSidecar service list The layer landing page lists every sidecar service. This layer defines no metric columns on the list, so the landing view is a plain, namespace-grouped roster of sidecar services — pick one to open its sidecars, then drill into a single sidecar\u0026rsquo;s dashboard.\nSidecar dashboard For one selected sidecar (an Envoy instance). The dashboard opens with four single-value status cards, then a set of time-series trends.\nStatus cards\nBug Failures — Envoy\u0026rsquo;s internal bug-failure counter; a non-zero value means an assertion or debug check tripped inside the proxy (envoy_bug_failures).\nMembership Healthy — the count of healthy endpoints across all of this Envoy\u0026rsquo;s upstream clusters (envoy_cluster_membership_healthy).\nWorker Threads — concurrent worker threads currently in use (envoy_worker_threads).\nUpstream Request Active — total active upstream requests across this Envoy\u0026rsquo;s clusters (envoy_cluster_up_rq_active).\nConnections and requests\nUpstream Connection Active — active upstream connections over time (envoy_cluster_up_cx_active).\nUpstream Request Pending — requests waiting in upstream queues (envoy_cluster_up_rq_pending_active).\nConnections Used — server-side connections in use, plotted as total and parent (envoy_total_connections_used, envoy_parent_connections_used).\nUpstream Connection Increase — new upstream connections opened per minute (envoy_cluster_up_cx_incr).\nUpstream Request Increase — new upstream requests per minute (envoy_cluster_up_rq_incr).\nThreads and memory\nWorker Threads (current vs max) — concurrent worker threads in use plotted against the window maximum, as current and max (envoy_worker_threads, envoy_worker_threads_max).\nServer Memory — the proxy\u0026rsquo;s memory footprint in bytes, each line paired with its window maximum: heap / heap max (envoy_heap_memory_used, envoy_heap_memory_max_used), allocated / allocated max (envoy_memory_allocated, envoy_memory_allocated_max), and physical / physical max (envoy_memory_physical_size, envoy_memory_physical_size_max).\nLogs and profiling Beyond the dashboard, the sidecar\u0026rsquo;s triage tabs are:\nLogs — the log stream is scoped to the sidecar (instance), so logs are read against the selected Envoy proxy rather than a higher-level service.\neBPF profiling — on-CPU / network profiling of the sidecar process via the eBPF profiling workflow.\nPod logs are also available for the sidecar, surfacing the underlying pod\u0026rsquo;s container output alongside the proxy\u0026rsquo;s own log stream.\nRequirements The MESH_DP dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the Envoy metrics-service receiver enabled and the sidecars configured to push their metrics to it. Every widget on the sidecar dashboard reads the envoy_* metric family at instance scope:\nProxy health — envoy_bug_failures, envoy_cluster_membership_healthy.\nWorker threads — envoy_worker_threads, envoy_worker_threads_max.\nUpstream clusters — envoy_cluster_up_rq_active, envoy_cluster_up_cx_active, envoy_cluster_up_rq_pending_active, envoy_cluster_up_cx_incr, envoy_cluster_up_rq_incr.\nServer connections — envoy_total_connections_used, envoy_parent_connections_used.\nServer memory — envoy_heap_memory_used, envoy_heap_memory_max_used, envoy_memory_allocated, envoy_memory_allocated_max, envoy_memory_physical_size, envoy_memory_physical_size_max.\nThese are instance-scope metrics; OAP does not roll a metric up across scopes, so the dashboard is empty until each sidecar\u0026rsquo;s Envoy is actually reporting to the metrics-service receiver. For how to point Envoy at OAP, see Envoy\u0026rsquo;s metrics service setting.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/mesh_dp/","title":"\u003c!--"},{"body":" MongoDB The MONGODB layer monitors MongoDB database clusters. SkyWalking collects MongoDB\u0026rsquo;s internal metrics — document and operation throughput, connections, cursors, replication lag and buffer, per-database data and index size, and per-node host stats — and Horizon renders them as a cluster-level and a node-level dashboard. This is a metrics-only layer: there are no traces, logs, endpoints, or a topology map for MongoDB.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Databases and named MongoDB. Its services are listed as MongoDB clusters and its instances as Nodes. Because the layer carries no endpoint, topology, trace, or log data, it enables only two sub-tabs: a Service (cluster) dashboard and an Instance (node) dashboard.\nThis page is the operator reference for the bundled MONGODB dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled MONGODB template; if an operator has published a customized MONGODB template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every MongoDB cluster with four sortable columns, sorted by document throughput (Doc QPS) by default:\nDoc QPS — document operations per second across the cluster, summed over the document-operation types (aggregate_labels(meter_mongodb_cluster_document_avg_qps, sum(doc_op_type))).\nOp QPS — database operations per second across the cluster, summed over the operation types (aggregate_labels(meter_mongodb_cluster_operation_avg_qps, sum(legacy_op_type))).\nConns — total open connections across the cluster (aggregate_labels(meter_mongodb_cluster_connections,sum)).\nRepl Lag — replication lag in ms (meter_mongodb_cluster_repl_lag).\nService dashboard The cluster-level drill-down for one selected MongoDB cluster. Most widgets aggregate across the cluster\u0026rsquo;s nodes with aggregate_labels(...).\nCluster Uptime (days) — how long the cluster has been running, as a single card, taking the max uptime across nodes and converting from seconds (latest(aggregate_labels(meter_mongodb_cluster_uptime,max))/3600/24).\nData Size (GB) — total stored data across the cluster, as a card, summed across nodes and converted from bytes (latest(aggregate_labels(meter_mongodb_cluster_data_size,sum))/1024/1024/1024).\nCollection Count — total number of collections across the cluster, as a card (latest(aggregate_labels(meter_mongodb_cluster_collection_count,sum))).\nObject Count — total number of objects (documents) across the cluster, as a card (latest(aggregate_labels(meter_mongodb_cluster_object_count,sum))).\nDocument QPS — document operations per second over time, summed over the document-operation types (aggregate_labels(meter_mongodb_cluster_document_avg_qps, sum(doc_op_type))).\nOperation QPS — database operations per second over time, summed over the operation types (aggregate_labels(meter_mongodb_cluster_operation_avg_qps, sum(legacy_op_type))).\nTotal Connections — open connections across the cluster over time (aggregate_labels(meter_mongodb_cluster_connections,sum)).\nCursor Total — open cursors across the cluster, summed over the cursor types (aggregate_labels(meter_mongodb_cluster_cursor_avg, sum(csr_type))).\nReplication Lag (ms) — replication lag in ms (meter_mongodb_cluster_repl_lag).\nDB Total Data (GB) — a per-database table of stored data size in GB, summed per database and converted from bytes, with columns Database and Data (GB) (latest(aggregate_labels(meter_mongodb_cluster_db_data_size, sum(database)))/1024/1024/1024).\nDB Total Index (GB) — a per-database table of index size in GB, summed per database and converted from bytes, with columns Database and Index (GB) (latest(aggregate_labels(meter_mongodb_cluster_db_index_size, sum(database)))/1024/1024/1024).\nInstance dashboard The node-level drill-down for one selected MongoDB node. These widgets read the per-node meter_mongodb_node_* family directly, with no cross-node aggregation.\nUptime (days) — how long the node has been running, as a card, converted from seconds (latest(meter_mongodb_node_uptime)/3600/24).\nQPS — total query throughput on the node (meter_mongodb_node_qps).\nReplSet State — a table of the node\u0026rsquo;s replica-set state, with columns Node and ReplSet state (latest(meter_mongodb_node_rs_state)).\nConnections — open connections on the node (meter_mongodb_node_connections).\nCPU Usage (%) — total CPU usage percentage on the node (meter_mongodb_node_cpu_total_percentage).\nMemory Usage — memory used by the node (meter_mongodb_node_memory_usage).\nMemory Free (GB) — free memory in GB as two series, mem and swap, each converted from KB (meter_mongodb_node_memory_free_kb/1024/1024, meter_mongodb_node_swap_memory_free_kb/1024/1024).\nDisk (GB) — filesystem used vs total in GB, converted from bytes (meter_mongodb_node_fs_used_size/1024/1024/1024, meter_mongodb_node_fs_total_size/1024/1024/1024).\nNetwork (KB/s) — network throughput in KB/s as in vs out, converted from bytes (meter_mongodb_node_network_bytes_in/1024, meter_mongodb_node_network_bytes_out/1024).\nActive Clients — active client connections as total, writers, and readers (meter_mongodb_node_active_total_num, meter_mongodb_node_active_writer_num, meter_mongodb_node_active_reader_num).\nDocument QPS — document operations per second on the node (meter_mongodb_node_document_qps).\nOperation QPS — database operations per second on the node (meter_mongodb_node_operation_qps).\nOp Latency (µs) — average operation latency in microseconds, computed as total latency divided by operation count, each summed over the operation types (aggregate_labels(meter_mongodb_node_latency_rate,sum(op_type))/aggregate_labels(meter_mongodb_node_op_rate,sum(op_type))).\nTransactions — active vs inactive transactions on the node (meter_mongodb_node_transactions_active, meter_mongodb_node_transactions_inactive).\nRepl Buffer — replication buffer count and size (MB), the size converted from bytes (meter_mongodb_node_repl_buffer_count, meter_mongodb_node_repl_buffer_size/1024/1024).\nQueued Operations — operations queued on the node (meter_mongodb_node_queued_operation).\nRequirements The MONGODB dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs MongoDB metrics, which OAP aggregates into the meter_mongodb_* families:\nCluster (service-scope) metrics — the meter_mongodb_cluster_* family (uptime, data and index size, collection and object counts, document and operation QPS, connections, cursors, and replication lag), which the cluster dashboard and the Service list aggregate across nodes.\nNode (instance-scope) metrics — the meter_mongodb_node_* family (uptime, QPS, replica-set state, connections, CPU, memory, disk, network, active clients, document and operation QPS, operation latency, transactions, replication buffer, and queued operations) for the node dashboard.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the node-scope widgets stay empty until per-node data is reported. Setting up MongoDB collection is described in the upstream MongoDB monitoring guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/mongodb/","title":"\u003c!--"},{"body":" MySQL / MariaDB The MYSQL layer monitors MySQL and MariaDB servers. It is populated by OAP\u0026rsquo;s MySQL/MariaDB monitoring, which scrapes a Prometheus-style mysqld-exporter and turns the result into SkyWalking meters — there is no language agent here, the data comes from the exporter.\nIn Horizon\u0026rsquo;s sidebar this layer lives under the Databases group and is named MySQL / MariaDB. A monitored cluster is listed as a MySQL cluster and each member server as a Node. This is a metrics-only layer: it enables the Service and Instance scopes and nothing else — there is no endpoint scope, no topology, and no traces or logs tabs.\nThis page is the operator reference for the bundled MySQL / MariaDB dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled MYSQL template; if an operator has published a customized MYSQL template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every MySQL cluster with four sortable columns, sorted by QPS by default. Each column aggregates the per-node meters across the whole cluster:\nQPS — queries per second across the cluster (aggregate_labels(meter_mysql_qps,sum)).\nTPS — transactions per second across the cluster (aggregate_labels(meter_mysql_tps,sum)).\nSlow QPS — slow queries per second across the cluster (aggregate_labels(meter_mysql_slow_queries_rate,sum)).\nConn Errors — connection-error rate, internal rejects plus max-connection rejects summed (aggregate_labels(meter_mysql_connection_errors_internal,sum) + aggregate_labels(meter_mysql_connection_errors_max_connections,sum)).\nService dashboard The primary drill-down for one selected cluster. Every widget here aggregates across the cluster\u0026rsquo;s nodes with aggregate_labels(..., sum).\nQPS — cluster-wide queries per second (aggregate_labels(meter_mysql_qps,sum)).\nTPS — cluster-wide transactions per second (aggregate_labels(meter_mysql_tps,sum)).\nSlow Queries / s — cluster-wide slow-query rate (aggregate_labels(meter_mysql_slow_queries_rate,sum)).\nConnection Errors — two series, internal rejects vs max-connection rejects (aggregate_labels(meter_mysql_connection_errors_internal,sum) and aggregate_labels(meter_mysql_connection_errors_max_connections,sum)).\nCommands Trend — rows-affected rate per command type: select / insert / update / delete (aggregate_labels(meter_mysql_commands_select_rate,sum), meter_mysql_commands_insert_rate, meter_mysql_commands_update_rate, meter_mysql_commands_delete_rate).\nThreads — thread counters: connected / running / cached / created (aggregate_labels(meter_mysql_threads_connected,sum), meter_mysql_threads_running, meter_mysql_threads_cached, meter_mysql_threads_created).\nSlow Statements — the top 20 slowest captured statements across the cluster, in ms, worst-first (top_n(top_n_database_statement, 20, des)). Each row carries the sampled statement text. Shows no data when OAP captured no statements in the window.\nInstance dashboard For one selected Node (a single MySQL/MariaDB server in the cluster). The four top cards are point-in-time configuration / status readings; the rest are per-node time series.\nStatus cards\nUptime — how long the server has been up, in days (latest(meter_mysql_instance_uptime)/3600/24).\nMax Connections — the server\u0026rsquo;s configured max_connections ceiling (latest(meter_mysql_instance_max_connections)).\nInnoDB Buffer Pool — the InnoDB buffer-pool size in MB (latest(meter_mysql_instance_innodb_buffer_pool_size)/1024/1024).\nThread Cache Size — the configured thread-cache size (latest(meter_mysql_instance_thread_cache_size)).\nTime series\nQPS / TPS — this node\u0026rsquo;s queries per second and transactions per second on one chart (meter_mysql_instance_qps, meter_mysql_instance_tps).\nSlow Queries / s — this node\u0026rsquo;s slow-query rate (meter_mysql_instance_slow_queries_rate).\nCommands Trend — rows-affected rate per command type for this node: select / insert / update / delete (meter_mysql_instance_commands_select_rate, meter_mysql_instance_commands_insert_rate, meter_mysql_instance_commands_update_rate, meter_mysql_instance_commands_delete_rate).\nThreads — this node\u0026rsquo;s thread counters: connected / running / cached / created (meter_mysql_instance_threads_connected, meter_mysql_instance_threads_running, meter_mysql_instance_threads_cached, meter_mysql_instance_threads_created).\nConnects — available vs aborted connection rate (meter_mysql_instance_connects_available, meter_mysql_instance_connects_aborted).\nConnection Errors — internal rejects vs max-connection rejects for this node (meter_mysql_instance_connection_errors_internal, meter_mysql_instance_connection_errors_max_connections).\nRequirements The MySQL / MariaDB dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs MySQL/MariaDB monitoring enabled so it scrapes a mysqld-exporter and produces:\nCluster (service-scope) meters — the meter_mysql_* family: QPS / TPS, slow-query rate, connection errors, the per-command rates, and the thread counters. These back the Service list and the Service dashboard.\nNode (instance-scope) meters — the meter_mysql_instance_* family: uptime, max-connections, InnoDB buffer-pool size, thread-cache size, and the per-node QPS/TPS, slow-query, command, thread, connect, and connection-error series. These back the Instance dashboard.\nSampled statements — top_n_database_statement for the Slow Statements list, captured by OAP when slow-statement sampling is configured.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the node-scope meter_mysql_instance_* series are empty until per-node data is reported. For the upstream setup steps — exporter configuration and which OAP rules to enable — see the MySQL/MariaDB monitoring documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/mysql/","title":"\u003c!--"},{"body":" Nginx The NGINX layer monitors Nginx servers and reverse proxies. Nginx, with the SkyWalking Lua module, reports request, latency, bandwidth, connection, status, and error-log telemetry, which OAP aggregates into the meter_nginx_* families this dashboard renders. The layer key is NGINX, and in Horizon\u0026rsquo;s sidebar it is grouped under Gateways.\nIn the sidebar this layer\u0026rsquo;s services are listed as Nginx services, its instances as Nodes (the individual Nginx server nodes), and its endpoints as Routes (the matched Nginx routes). The NGINX layer enables three metric scopes — Service, Instance (Node), and Endpoint (Route) — plus a Logs tab. It does not ship a topology or a traces tab, so apart from logs this dashboard is the metric drill-down across those three scopes.\nThis page is the operator reference for the bundled NGINX dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled NGINX template; if an operator has published a customized NGINX template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a service, the layer landing page lists every NGINX service, sorted by request rate (RPS) by default, with three columns:\nRPS — total HTTP requests per second across the service\u0026rsquo;s nodes (aggregate_labels(meter_nginx_service_http_requests, sum)).\n5xx % — percent of requests that returned a 5xx status, as a share of all requests over the window (aggregate_labels(meter_nginx_service_http_5xx_requests_increment, sum)/aggregate_labels(meter_nginx_service_http_requests_increment, sum)*100).\n4xx % — percent of requests that returned a 4xx status, as a share of all requests over the window (aggregate_labels(meter_nginx_service_http_4xx_requests_increment, sum)/aggregate_labels(meter_nginx_service_http_requests_increment, sum)*100).\nThe two error columns give an at-a-glance health read across the fleet — a service with a climbing 5xx % is failing requests at the proxy, a climbing 4xx % is rejecting client requests.\nService dashboard The primary drill-down for one selected Nginx service. All eight widgets aggregate across the service\u0026rsquo;s nodes.\nHTTP Request Trend — total requests per second for the service (aggregate_labels(meter_nginx_service_http_requests, sum)).\nHTTP Latency — request latency in ms, averaged across the reported percentiles (aggregate_labels(meter_nginx_service_http_latency, avg(p))).\nHTTP Bandwidth — bandwidth in KB/s, summed across the bandwidth types (aggregate_labels(meter_nginx_service_http_bandwidth, sum(type)), divided to KB/s).\nHTTP Connections — connections summed by state (aggregate_labels(meter_nginx_service_http_connections, sum(state)), one line per state).\nHTTP Status Trend — requests summed by HTTP status (aggregate_labels(meter_nginx_service_http_status, sum(status)), one line per status).\n4xx % / min — percent of requests returning a 4xx status per minute (aggregate_labels(meter_nginx_service_http_4xx_requests_increment, sum)/aggregate_labels(meter_nginx_service_http_requests_increment, sum)*100).\n5xx % / min — percent of requests returning a 5xx status per minute (aggregate_labels(meter_nginx_service_http_5xx_requests_increment, sum)/aggregate_labels(meter_nginx_service_http_requests_increment, sum)*100).\nError Log Count — count of error-log entries summed by log level (aggregate_labels(meter_nginx_service_error_log_count, sum(level)), one line per level).\nInstance dashboard For one selected node. These widgets are reported per node, so they show how one Nginx server process is behaving.\nHTTP Request Trend — requests per second for the node (meter_nginx_instance_http_requests).\nHTTP Latency — request latency in ms for the node (meter_nginx_instance_http_latency).\nHTTP Bandwidth — bandwidth in KB/s for the node (meter_nginx_instance_http_bandwidth, divided to KB/s).\nHTTP Connections — connections by state for the node (meter_nginx_instance_http_connections).\nHTTP Status Trend — requests by HTTP status for the node (meter_nginx_instance_http_status).\n4xx % / min — percent of requests returning a 4xx status per minute for the node ((meter_nginx_instance_http_4xx_requests_increment/meter_nginx_instance_http_requests_increment)*100).\n5xx % / min — percent of requests returning a 5xx status per minute for the node ((meter_nginx_instance_http_5xx_requests_increment/meter_nginx_instance_http_requests_increment)*100).\nError Log Count — count of error-log entries for the node (meter_nginx_instance_error_log_count).\nEndpoint dashboard For one selected route. Nginx reports a tighter metric set at route scope — requests, latency, bandwidth, status, and the per-route error rates.\nHTTP Request Trend — requests per second for the route (meter_nginx_endpoint_http_requests).\nHTTP Latency — request latency in ms for the route (meter_nginx_endpoint_http_latency).\nHTTP Bandwidth — bandwidth in KB for the route (meter_nginx_endpoint_http_bandwidth, divided to KB).\nHTTP Status Trend — requests by HTTP status for the route (meter_nginx_endpoint_http_status).\n4xx % / min — percent of requests returning a 4xx status per minute for the route ((meter_nginx_endpoint_http_4xx_requests_increment/meter_nginx_endpoint_http_requests_increment)*100).\n5xx % / min — percent of requests returning a 5xx status per minute for the route ((meter_nginx_endpoint_http_5xx_requests_increment/meter_nginx_endpoint_http_requests_increment)*100).\nLogs The NGINX layer enables the Logs tab. Nginx access and error logs forwarded to OAP are searchable here, scoped to the selected Nginx service, with the standard log filters and time range. This is the same logs experience as other log-enabled layers — see the layer logs view for how to filter, page, and inspect entries.\nRequirements The NGINX dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Nginx telemetry flowing in:\nService metrics — the meter_nginx_service_* family (requests, latency, bandwidth, connections, status, the 4xx / 5xx increment counters, and error-log count), aggregated by OAP across a service\u0026rsquo;s nodes.\nInstance (node) metrics — the meter_nginx_instance_* family for the per-node request, latency, bandwidth, connection, status, error-rate, and error-log widgets.\nEndpoint (route) metrics — the meter_nginx_endpoint_* family for the per-route request, latency, bandwidth, status, and error-rate widgets.\nLogs — Nginx access / error logs shipped to OAP, for the Logs tab.\nThese come from the SkyWalking Nginx Lua module emitting Nginx telemetry to OAP, which converts it into the meter_nginx_* metrics through its meter-analysis rules. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a route-scope or node-scope metric is empty until that level of data is reported. For the end-to-end setup, see the Nginx monitoring backend guide.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/nginx/","title":"\u003c!--"},{"body":" Linux The OS_LINUX layer monitors Linux hosts. It is populated by OAP\u0026rsquo;s VM monitoring, which scrapes a Prometheus node-exporter and turns the result into SkyWalking meters — there is no language agent here, the data comes from the exporter.\nIn Horizon\u0026rsquo;s sidebar this layer lives under the OS group and is named Linux. Each monitored host is listed as a Host. This is a metrics-only, single-scope layer: it enables the Service scope and nothing else — there is no instance scope, no endpoint scope, no topology, and no traces or logs tabs.\nThis page is the operator reference for the bundled Linux dashboard: what you see on the host scope and what each widget means.\nThe widgets and metrics below are read from the bundled OS_LINUX template; if an operator has published a customized OS_LINUX template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nHost list Before opening a host, the layer landing page lists every Linux host with four sortable columns, sorted by CPU % by default:\nCPU % — average CPU utilization across all cores (meter_vm_cpu_total_percentage).\nMemory MB — memory in use, in MB (meter_vm_memory_used/1024/1024).\nLoad 1m — the 1-minute load average (meter_vm_cpu_load1/100).\nFS % — filesystem space used, as a percent (meter_vm_filesystem_percentage).\nHost dashboard The primary drill-down for one selected host.\nCPU Average Used (%) — average CPU utilization across cores, as a percent (meter_vm_cpu_average_used).\nCPU Load — the load average at three windows: 1m / 5m / 15m (meter_vm_cpu_load1/100, meter_vm_cpu_load5/100, meter_vm_cpu_load15/100).\nFile FD Allocated — the number of allocated file descriptors (meter_vm_filefd_allocated).\nMemory RAM (MB) — four series in MB: used / total / available / buff/cache (meter_vm_memory_used/1024/1024, meter_vm_memory_total/1024/1024, meter_vm_memory_available/1024/1024, meter_vm_memory_buff_cache/1024/1024).\nMemory Swap (MB) — swap free vs swap total, in MB (meter_vm_memory_swap_free/1024/1024, meter_vm_memory_swap_total/1024/1024).\nNetwork Bandwidth (KB/s) — receive vs transmit throughput, in KB/s (meter_vm_network_receive/1024, meter_vm_network_transmit/1024).\nDisk R/W (KB/s) — disk read vs written throughput, in KB/s (meter_vm_disk_read/1024, meter_vm_disk_written/1024).\nFilesystem Usage (%) — filesystem space used, as a percent (meter_vm_filesystem_percentage).\nNetwork Status — five socket / TCP counters: established TCP connections, TCP time-wait, TCP alloc, sockets used, and UDP in-use (meter_vm_tcp_curr_estab, meter_vm_tcp_tw, meter_vm_tcp_alloc, meter_vm_sockets_used, meter_vm_udp_inuse).\nRequirements The Linux dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs VM monitoring enabled so it scrapes a node-exporter and produces:\nHost (service-scope) meters — the meter_vm_* family: CPU utilization and load average, memory (used / total / available / buff-cache / swap), file-descriptor allocation, network receive/transmit, disk read/written, filesystem usage, and the TCP / socket / UDP counters. These back the Host list and the Host dashboard. Each metric is queried at its own OAP scope; because this layer is service-scope only, every widget reads the host-level meter_vm_* series and there is no instance- or endpoint-level rollup. For the upstream setup steps — node-exporter configuration and which OAP rules to enable — see the VM monitoring documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/os_linux/","title":"\u003c!--"},{"body":" Windows The OS_WINDOWS layer monitors Windows hosts. It is populated by OAP\u0026rsquo;s Windows monitoring, which receives host telemetry (CPU, memory, network, disk) and turns it into SkyWalking meters — there is no language agent here, the data comes from the host telemetry.\nIn Horizon\u0026rsquo;s sidebar this layer lives under the OS group and is named Windows. Each monitored Windows machine is listed as a Host. This is a metrics-only, single-scope layer: it enables the Service scope and nothing else — there is no instance or endpoint scope, no topology, and no traces or logs tabs.\nThis page is the operator reference for the bundled Windows dashboard: what you see and what each widget means.\nThe widgets and metrics below are read from the bundled OS_WINDOWS template; if an operator has published a customized OS_WINDOWS template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nHost list Before opening a host, the layer landing page lists every Windows host with three sortable columns, sorted by CPU % by default:\nCPU % — average CPU utilization across the host (meter_win_cpu_total_percentage).\nMemory MB — physical memory used, in MB (meter_win_memory_used/1024/1024).\nVMem % — virtual-memory utilization percentage (avg(meter_win_memory_virtual_memory_percentage)).\nHost dashboard The primary drill-down for one selected Windows host.\nCPU Average Used (%) — average CPU utilization over the window (meter_win_cpu_average_used).\nMemory RAM (MB) — physical memory in MB, three series: used / total / available (meter_win_memory_used/1024/1024, meter_win_memory_total/1024/1024, meter_win_memory_available/1024/1024).\nVirtual Memory (MB) — virtual (page-file backed) memory in MB, free vs total (meter_win_memory_virtual_memory_free/1024/1024, meter_win_memory_virtual_memory_total/1024/1024).\nNetwork Bandwidth (KB/s) — network throughput in KB/s, receive vs transmit (meter_win_network_receive/1024, meter_win_network_transmit/1024).\nDisk R/W (KB/s) — disk throughput in KB/s, read vs written (meter_win_disk_read/1024, meter_win_disk_written/1024).\nRequirements The Windows dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Windows monitoring enabled so it ingests host telemetry and produces the host (service-scope) meter_win_* family:\nCPU — meter_win_cpu_total_percentage and meter_win_cpu_average_used back the CPU column and the CPU widget.\nMemory — meter_win_memory_used, meter_win_memory_total, meter_win_memory_available, and the meter_win_memory_virtual_memory_* series back the memory columns and the RAM / virtual-memory widgets.\nNetwork and disk — meter_win_network_receive / meter_win_network_transmit and meter_win_disk_read / meter_win_disk_written back the network and disk throughput widgets.\nEvery metric is queried at the Service (host) scope; OAP does not roll a metric up across scopes, so the dashboard stays empty until per-host data is reported. For the upstream setup steps — host-telemetry collection and which OAP rules to enable — see the Windows monitoring documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/os_windows/","title":"\u003c!--"},{"body":" Mesh Dashboard The Mesh Dashboard is the cross-layer overview for an Istio service mesh. Where the Services Dashboard centers on language-agent traffic, this one centers on the data plane: it pulls onto one screen the services routed through the Istio data plane, the Istio control-plane (pilot / xDS) push activity that keeps them configured, and — because a mesh always runs on Kubernetes — the same cluster capacity strip. It draws from the MESH, MESH_CP, and K8S layers.\nLike every overview, it sits at the top of the sidebar above the per-layer entries and appears only while at least one of its layers is reporting; a layer\u0026rsquo;s tile auto-hides when that layer has nothing reporting (refreshed on the same ~60-second cadence as the menu).\nThe widgets and metrics below are read from the bundled overview template; if an administrator has published a customized copy to OAP, the live page reflects that copy instead. These are editable defaults — reshape them in the Overview Templates admin page on a bundled-default → local-draft → Check diff \u0026amp; push flow. See Overview Templates for the editor and the stored format, and Overview Widgets for the widget vocabulary.\nMesh services row Istio-managed services (MESH) — a KPI tile with the mesh service count plus RPM (calls per minute, service_cpm), P95 (95th-percentile latency in ms, service_percentile{p='95'}), and SLA (percent successful, service_sla/100). This is the data-plane equivalent of the General-services tile. Istio pilot (MESH_CP) — a composite summarizing control-plane activity: xDS pushes (config pushes Pilot sent, meter_istio_pilot_xds_pushes), xDS connections (proxies currently connected to Pilot, meter_istio_pilot_xds), Services (the layer\u0026rsquo;s service count), and Pilot errors (rejected pushes + write timeouts across CDS / EDS / LDS / RDS, summed: meter_istio_pilot_xds_cds_reject+meter_istio_pilot_xds_eds_reject+meter_istio_pilot_xds_lds_reject+meter_istio_pilot_xds_rds_reject+meter_istio_pilot_xds_write_timeout). A climbing Pilot-errors number means the control plane is struggling to push valid config — a mesh-specific failure the data-plane tiles won\u0026rsquo;t surface. Topology \u0026amp; active alarms Mesh service topology — a live service map of the MESH layer, the bulk of the row. Same renderer as the per-layer Topology tab. Active alarms — the right-hand rail of alarms currently firing on mesh-reported services, up to 12. Read-only — alarm recovery is backend-automatic. Kubernetes Cluster capacity \u0026amp; utilisation — the same full-width K8S composite as the Services Dashboard: cluster inventory counts (Nodes, Namespaces, Deployments, StatefulSets, DaemonSets, Services, Containers) on the left, and CPU / Memory / Storage commitment bars on the right (same k8s_cluster_* metrics and 0 – 100 % scale). Mesh deployments always ride on Kubernetes, so the capacity block lives directly under the mesh health. Requirements An overview is a pure consumer of what OAP reports — it invents no data, and a tile or panel with no backing metric simply hides or reads no data. To populate the Mesh Dashboard, OAP needs:\nService-scope metrics on the MESH layer — the service_* family (traffic, response time, percentile, SLA), produced by OAP from the mesh-reported telemetry. Queried at its own OAP scope; OAP does not roll a metric up across scopes. Istio control-plane meters — the meter_istio_pilot_* family on the MESH_CP layer, for the Istio pilot composite. Relation metrics for the embedded service map — service_relation_* at the MESH layer. Alarm data — firing alarms scoped to the MESH layer, for the Active-alarms rail. Kubernetes cluster metrics — the k8s_cluster_* family on the K8S layer, for the capacity composite. When a whole layer is missing — no mesh, no Kubernetes monitoring — its tile or block is hidden rather than shown empty, and the overview drops out of the sidebar once none of its layers are reporting.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/overview-mesh/","title":"\u003c!--"},{"body":" Services Dashboard The Services Dashboard is Horizon\u0026rsquo;s default cross-layer overview — the \u0026ldquo;is everything OK?\u0026rdquo; war-room pane for your traced application services. It pulls several layers onto one screen at once: a row of count + health tiles for application and virtual-backend services, a live service map, the alarms firing right now, and the Kubernetes capacity underneath them all. It answers \u0026ldquo;how many services are up, how hard are they working, is anything on fire, and is the cluster running out of room\u0026rdquo; without you clicking into any one service.\nIt folds in the GENERAL, VIRTUAL_DATABASE, VIRTUAL_CACHE, VIRTUAL_MQ, VIRTUAL_GENAI, and K8S layers; any of those that isn\u0026rsquo;t reporting drops its tile automatically. Overviews are listed at the top of the sidebar, above the per-layer entries, and each appears only while at least one of its layers is reporting (refreshed on the same ~60-second cadence as the menu). For the service-mesh counterpart, see the Mesh Dashboard.\nThe widgets and metrics below are read from the bundled overview template; if an administrator has published a customized copy to OAP, the live page reflects that copy instead. These are editable defaults — reshape them (add / remove / resize widgets, swap MQE) in the Overview Templates admin page on a bundled-default → local-draft → Check diff \u0026amp; push flow. See Overview Templates for the editor and the stored format, and Overview Widgets for the widget vocabulary.\nServices row Five KPI tiles, one per service-class layer, each showing that layer\u0026rsquo;s reporting service count plus three headline numbers:\nGeneral services (GENERAL) — traced application services. RPM (total calls per minute, service_cpm), Latency (average response time in ms, service_resp_time), SLA (percent successful, service_sla/100). Virtual databases (VIRTUAL_DATABASE) — backend databases observed via client-side spans. RPM (database_access_cpm), Latency (database_access_resp_time), SLA (database_access_sla/100). Virtual caches (VIRTUAL_CACHE) — Redis / Memcached / … observed via client-side spans. RPM (cache_access_cpm), Latency (cache_access_resp_time), SLA (cache_access_sla/100). Virtual MQs (VIRTUAL_MQ) — message queues observed via consume + produce spans. Consume (consume rate per minute, mq_service_consume_cpm), Produce (produce rate per minute, mq_service_produce_cpm), Consume latency (ms, mq_service_consume_latency). Virtual GenAI (VIRTUAL_GENAI) — GenAI backends observed via instrumented client spans. RPM (gen_ai_provider_cpm), Latency (gen_ai_provider_resp_time), SLA (gen_ai_provider_sla/100). The RPM / consume / produce numbers are summed across the layer; latency and SLA are averaged. A layer with nothing reporting (no GenAI backends in this deployment, say) simply leaves its tile off the row.\nTopology \u0026amp; active alarms General service topology — a live service map of the GENERAL layer, taking up most of the row. Same map you see on the per-layer Topology tab, embedded here for the war-room at-a-glance view. Active alarms — a rail down the right side listing the alarms currently firing on agent-reported (GENERAL) services, up to 12 at a time. Read-only — alarm recovery is backend-automatic. Kubernetes Cluster capacity \u0026amp; utilisation — a full-width composite summarizing the K8S layer. On the left, the cluster inventory as latest counts: Nodes (k8s_cluster_node_total), Namespaces (k8s_cluster_namespace_total), Deployments (k8s_cluster_deployment_total), StatefulSets (k8s_cluster_statefulset_total), DaemonSets (k8s_cluster_daemonset_total), Services (k8s_cluster_service_total), and Containers (k8s_cluster_container_total). On the right, three utilisation bars showing how much of the cluster is already committed — CPU (requested cores over capacity, k8s_cluster_cpu_cores_requests/k8s_cluster_cpu_cores*100), Memory (requested over total, k8s_cluster_memory_requests/k8s_cluster_memory_total*100), and Storage (allocated over total, (k8s_cluster_storage_total-k8s_cluster_storage_allocatable)/k8s_cluster_storage_total*100), each on a 0 – 100 % scale. This block is the \u0026ldquo;are we about to run out of room\u0026rdquo; check that the service tiles above can\u0026rsquo;t tell you. Requirements An overview is a pure consumer of what OAP reports — it invents no data, and a tile or panel with no backing metric simply hides (the layer count drops to zero and the tile is omitted) or reads no data. To populate the Services Dashboard, OAP needs:\nService-scope metrics for each service-class layer — the service_* family for GENERAL (traffic, response time, SLA), and the virtual-backend families database_access_*, cache_access_*, mq_service_*, and gen_ai_provider_* for the virtual layers. Each is queried at its own OAP scope; OAP does not roll a metric up across scopes. Relation metrics for the embedded service map — service_relation_* at the GENERAL layer. Alarm data — firing alarms scoped to the layer, for the Active-alarms rail. Kubernetes cluster metrics — the k8s_cluster_* family (inventory totals plus CPU / memory / storage capacity), reported by the OAP Kubernetes monitoring on the K8S layer, for the capacity composite. When a whole layer is missing — no virtual MQs, no Kubernetes monitoring — its tile or block is hidden rather than shown empty, and the overview itself drops out of the sidebar once none of its layers are reporting.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/overview-services/","title":"\u003c!--"},{"body":" PostgreSQL The POSTGRESQL layer monitors PostgreSQL servers. It is populated by OAP\u0026rsquo;s PostgreSQL monitoring, which scrapes a Prometheus-style postgres-exporter and turns the result into SkyWalking meters — there is no language agent here, the data comes from the exporter.\nIn Horizon\u0026rsquo;s sidebar this layer lives under the Databases group and is named PostgreSQL. A monitored cluster is listed as a PostgreSQL cluster and each member server as a Node. This is a metrics-only layer: it enables the Service and Instance scopes and nothing else — there is no endpoint scope, no topology, and no traces or logs tabs.\nThis page is the operator reference for the bundled PostgreSQL dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled POSTGRESQL template; if an operator has published a customized POSTGRESQL template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every PostgreSQL cluster with four sortable columns, sorted by Fetched / s by default. Each column aggregates the per-node meters across the whole cluster:\nFetched / s — rows fetched per second across the cluster (aggregate_labels(meter_pg_fetched_rows_rate,sum)).\nInserted / s — rows inserted per second across the cluster (aggregate_labels(meter_pg_inserted_rows_rate,sum)).\nCache Hit — buffer-cache hit ratio across the cluster, in percent (aggregate_labels(meter_pg_cache_hit_rate,avg)).\nDeadlocks — deadlocks per second across the cluster (aggregate_labels(meter_pg_deadlocks_rate,sum)).\nService dashboard The primary drill-down for one selected cluster. Most widgets here aggregate across the cluster\u0026rsquo;s nodes with aggregate_labels(...).\nFetched Rows / s — cluster-wide rows fetched per second (aggregate_labels(meter_pg_fetched_rows_rate,sum)).\nInserted Rows / s — cluster-wide rows inserted per second (aggregate_labels(meter_pg_inserted_rows_rate,sum)).\nUpdated Rows / s — cluster-wide rows updated per second (aggregate_labels(meter_pg_updated_rows_rate,sum)).\nDeleted Rows / s — cluster-wide rows deleted per second (aggregate_labels(meter_pg_deleted_rows_rate,sum)).\nReturned Rows / s — cluster-wide rows returned per second (aggregate_labels(meter_pg_returned_rows_rate,sum)).\nTemporary Files / s — temporary files created per second across the cluster, a sign of queries spilling to disk (aggregate_labels(meter_pg_temporary_files_rate,sum)).\nCache Hit Rate — cluster-wide buffer-cache hit ratio in percent (aggregate_labels(meter_pg_cache_hit_rate,avg)).\nTransactions / s — committed vs rolled-back transactions per second (aggregate_labels(meter_pg_committed_transactions_rate,sum) and aggregate_labels(meter_pg_rolled_back_transactions_rate,sum)).\nConflicts + Deadlocks / s — two series, recovery conflicts vs deadlocks per second (aggregate_labels(meter_pg_conflicts_rate,sum) and aggregate_labels(meter_pg_deadlocks_rate,sum)).\nSessions — active vs idle sessions and the lock count across the cluster (aggregate_labels(meter_pg_active_sessions,sum), aggregate_labels(meter_pg_idle_sessions,sum), aggregate_labels(meter_pg_locks_count,sum)).\nBuffers / s — background-writer and checkpoint buffer activity: checkpoint / clean / backend fsync / alloc / backend (aggregate_labels(meter_pg_buffers_checkpoint,sum), aggregate_labels(meter_pg_buffers_clean,sum), aggregate_labels(meter_pg_buffers_backend_fsync,sum), aggregate_labels(meter_pg_buffers_alloc,sum), aggregate_labels(meter_pg_buffers_backend,sum)).\nCheckpoint Stats / s — checkpoint counters: timed / requested / write time / sync time (aggregate_labels(meter_pg_checkpoints_timed_rate,sum), aggregate_labels(meter_pg_checkpoint_req_rate,sum), aggregate_labels(meter_pg_checkpoint_write_time_rate,sum), aggregate_labels(meter_pg_checkpoint_sync_time_rate,sum)).\nSlow Statements — the top 20 slowest captured statements across the cluster, in ms, worst-first (top_n(top_n_database_statement, 20, des)). Each row carries the sampled statement text. Shows no data when OAP captured no statements in the window.\nInstance dashboard For one selected Node (a single PostgreSQL server in the cluster). The four top cards are point-in-time configuration readings; the rest are per-node time series.\nStatus cards\nShared Buffers — the node\u0026rsquo;s configured shared_buffers size in MB (latest(meter_pg_instance_shared_buffers)/1024/1024).\nEffective Cache — the node\u0026rsquo;s configured effective_cache_size in GB (latest(meter_pg_instance_effective_cache)/1024/1024/1024).\nWork Mem — the node\u0026rsquo;s configured work_mem in MB (latest(meter_pg_instance_work_mem)/1024/1024).\nMax WAL Size — the node\u0026rsquo;s configured max_wal_size in GB (latest(meter_pg_instance_max_wal_size)/1024/1024/1024).\nTime series\nFetched Rows / s — this node\u0026rsquo;s rows fetched per second (meter_pg_instance_fetched_rows_rate).\nInserted Rows / s — this node\u0026rsquo;s rows inserted per second (meter_pg_instance_inserted_rows_rate).\nCache Hit Rate — this node\u0026rsquo;s buffer-cache hit ratio in percent (meter_pg_instance_cache_hit_rate).\nSessions — this node\u0026rsquo;s active vs idle sessions and lock count (meter_pg_instance_active_sessions, meter_pg_instance_idle_sessions, meter_pg_instance_locks_count).\nConflicts + Deadlocks / s — recovery conflicts vs deadlocks per second for this node (meter_pg_instance_conflicts_rate, meter_pg_instance_deadlocks_rate).\nRequirements The PostgreSQL dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs PostgreSQL monitoring enabled so it scrapes a postgres-exporter and produces:\nCluster (service-scope) meters — the meter_pg_* family: the per-operation row rates (fetched / inserted / updated / deleted / returned), temporary-file rate, cache-hit ratio, committed and rolled-back transaction rates, conflicts and deadlocks, active / idle sessions and locks, and the background-writer buffer and checkpoint counters. These back the Service list and the Service dashboard.\nNode (instance-scope) meters — the meter_pg_instance_* family: the configured shared_buffers, effective_cache_size, work_mem, and max_wal_size readings, plus the per-node fetched / inserted row rates, cache-hit ratio, sessions and locks, and conflict / deadlock series. These back the Instance dashboard.\nSampled statements — top_n_database_statement for the Slow Statements list, captured by OAP when slow-statement sampling is configured.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the node-scope meter_pg_instance_* series are empty until per-node data is reported. For the upstream setup steps — exporter configuration and which OAP rules to enable — see the PostgreSQL monitoring documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/postgresql/","title":"\u003c!--"},{"body":" Pulsar The PULSAR layer monitors Apache Pulsar message brokers. SkyWalking collects Pulsar\u0026rsquo;s broker metrics through OpenTelemetry and renders each Pulsar cluster as a service, with its brokers as instances — so a cluster\u0026rsquo;s topic, subscription, and message-flow health sits beside the broker-level connection and JVM detail in one place.\nIn Horizon\u0026rsquo;s sidebar this layer is named Pulsar, grouped under MQ. Its services are listed as Pulsar clusters and its instances as Brokers. The PULSAR layer enables the Service and Instance sub-tabs only — there is no endpoint scope, no topology, and no traces or logs tab for this layer.\nThis page is the operator reference for the bundled PULSAR dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled PULSAR template; if an operator has published a customized PULSAR template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nPulsar cluster list Before opening a cluster, the layer landing page lists every Pulsar cluster with four sortable columns, sorted by Topics by default. Each column sums the per-label series across the cluster:\nTopics — total topics on the cluster (meter_pulsar_total_topics).\nSubscriptions — total subscriptions on the cluster (meter_pulsar_total_subscriptions).\nMsg In — incoming message rate (meter_pulsar_message_rate_in).\nMsg Out — outgoing message rate (meter_pulsar_message_rate_out).\nService dashboard The primary drill-down for one selected Pulsar cluster. Every widget aggregates the cluster\u0026rsquo;s per-label series with aggregate_labels(..., sum), giving cluster-wide totals.\nTotal Topics — number of topics on the cluster (meter_pulsar_total_topics).\nSubscriptions — number of subscriptions on the cluster (meter_pulsar_total_subscriptions).\nProducers — number of connected producers (meter_pulsar_total_producers).\nConsumers — number of connected consumers (meter_pulsar_total_consumers).\nMessage Rate — incoming vs outgoing message rate on one chart, plotted as in (meter_pulsar_message_rate_in) and out (meter_pulsar_message_rate_out).\nThroughput — incoming vs outgoing byte throughput on one chart, plotted as in (meter_pulsar_throughput_in) and out (meter_pulsar_throughput_out).\nStorage Read/Write Rate — bookkeeper storage read vs write rate, plotted as read (meter_pulsar_storage_read_rate) and write (meter_pulsar_storage_write_rate).\nStorage Size (MB) — physical vs logical storage size in MB, plotted as physical (meter_pulsar_storage_size) and logical (meter_pulsar_storage_logical_size); both are reported in bytes and divided by 1024 / 1024 for display.\nInstance dashboard For one selected broker. These widgets read the broker-scope meter_pulsar_broker_* family directly.\nActive Connections — connections currently open on the broker (meter_pulsar_broker_active_connections).\nTotal Connections — connections handled by the broker (meter_pulsar_broker_total_connections).\nConn Create Fail — failed connection-create attempts (meter_pulsar_broker_connection_create_fail_count).\nConn Create Success — successful connection-create attempts (meter_pulsar_broker_connection_create_success_count).\nConnection Closed — total connections closed (meter_pulsar_broker_connection_closed_total_count).\nJVM Buffer Pool (MB) — JVM buffer-pool bytes used by the broker, in MB (meter_pulsar_broker_jvm_buffer_pool_used_bytes, divided by 1024 / 1024).\nJVM Memory Pool Used (MB) — JVM memory-pool bytes used, in MB (meter_pulsar_broker_jvm_memory_pool_used, divided by 1024 / 1024).\nJVM Memory (MB) — JVM memory in MB plotted as used, committed, and init (meter_pulsar_broker_jvm_memory_used, meter_pulsar_broker_jvm_memory_committed, meter_pulsar_broker_jvm_memory_init, each divided by 1024 / 1024).\nJVM Threads — thread counts plotted as current, daemon, peak, and deadlocked (meter_pulsar_broker_jvm_threads_current, meter_pulsar_broker_jvm_threads_daemon, meter_pulsar_broker_jvm_threads_peak, meter_pulsar_broker_jvm_threads_deadlocked).\nGC — garbage-collection time vs count on a dual axis, with cumulative seconds on the left axis (meter_pulsar_broker_jvm_gc_collection_seconds_sum) and count on the right axis (meter_pulsar_broker_jvm_gc_collection_seconds_count).\nRequirements The PULSAR dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs Pulsar monitoring enabled, so that the broker\u0026rsquo;s OpenTelemetry metrics reach OAP and are aggregated into the Pulsar meter families:\nCluster (service) metrics — the meter_pulsar_* family (topics, subscriptions, producers, consumers, message rate, throughput, and bookkeeper storage), which back the cluster list and the Service dashboard.\nBroker (instance) metrics — the meter_pulsar_broker_* family (connections and the broker JVM buffer / memory / thread / GC detail), which back the Instance dashboard.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a broker-scope metric is empty until that broker reports it. See Pulsar monitoring for how to wire a Pulsar deployment into OAP.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/pulsar/","title":"\u003c!--"},{"body":" RabbitMQ The RABBITMQ layer monitors RabbitMQ message brokers. OAP collects the metrics from RabbitMQ\u0026rsquo;s Prometheus / OpenMetrics endpoint, so each broker cluster and each broker node surfaces as a SkyWalking entity in this layer.\nIn Horizon\u0026rsquo;s sidebar this layer is named RabbitMQ. Its services are listed as RabbitMQ clusters and its instances as Nodes — a service is one RabbitMQ cluster, and each instance is one broker node inside it. The layer enables two scopes only: the Service (cluster) dashboard and the Instance (node) dashboard. There is no endpoint scope, no topology / map, and no traces or logs tab for this layer.\nThis page is the operator reference for the bundled RABBITMQ dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled RABBITMQ template; if an operator has published a customized RABBITMQ template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every RabbitMQ cluster with four sortable columns, sorted by Queues by default:\nQueues — total queues across the cluster (aggregate_labels(meter_rabbitmq_queues,sum)). Channels — total open channels across the cluster (aggregate_labels(meter_rabbitmq_channels,sum)). Connections — total open connections across the cluster (aggregate_labels(meter_rabbitmq_connections,sum)). Unconfirmed — publisher messages awaiting confirmation across the cluster (aggregate_labels(meter_rabbitmq_messages_unconfirmed,sum)). Service dashboard The cluster-level view for one selected RabbitMQ cluster.\nMemory Available Before Block (MB) — headroom in MB before the broker hits its memory high-watermark and starts blocking publishers (meter_rabbitmq_memory_available_before_publisher_blocked). Disk Available Before Block (GB) — headroom in GB before the broker hits its disk free-space limit and starts blocking publishers (meter_rabbitmq_disk_space_available_before_publisher_blocked). File Descriptors + Sockets — available file descriptors (fds) and available TCP sockets (sockets), the two resource pools that gate how many connections the broker can still accept (meter_rabbitmq_file_descriptors_available, meter_rabbitmq_tcp_socket_available). Ready Messages — messages ready to be delivered to consumers (meter_rabbitmq_message_ready_delivered_consumers). Pending Ack — messages delivered to consumers but not yet acknowledged (meter_rabbitmq_message_unacknowledged_delivered_consumers). Publish Pipeline — the publish path across four series: published, confirmed, routed, and unconfirmed (meter_rabbitmq_messages_published, meter_rabbitmq_messages_confirmed, meter_rabbitmq_messages_routed, meter_rabbitmq_messages_unconfirmed). A growing gap between published and confirmed/routed flags a routing or confirmation problem. Unroutable Messages — messages with no matching binding, split into dropped and returned (meter_rabbitmq_messages_unroutable_dropped, meter_rabbitmq_messages_unroutable_returned). Queues — queue lifecycle across the cluster: total currently present, plus the declared, created, and deleted running totals (meter_rabbitmq_queues, meter_rabbitmq_queues_declared_total, meter_rabbitmq_queues_created_total, meter_rabbitmq_queues_deleted_total). Channels — channel lifecycle: total currently open, plus the opened and closed running totals (meter_rabbitmq_channels, meter_rabbitmq_channels_opened_total, meter_rabbitmq_channels_closed_total). Connections — connection lifecycle: total currently open, plus the opened and closed running totals (meter_rabbitmq_connections, meter_rabbitmq_connections_opened_total, meter_rabbitmq_connections_closed_total). Instance dashboard The node-level view for one selected broker node. The cards across the top are single-value (latest) readings; the remaining widgets are time-series.\nReady Messages — messages ready for delivery on this node, latest value (latest(meter_rabbitmq_node_queue_messages_ready)). Incoming Messages — incoming message rate on this node, latest value (latest(meter_rabbitmq_node_incoming_messages)). Outgoing Messages — outgoing message total on this node, latest value (latest(meter_rabbitmq_node_outgoing_messages_total)). Unacknowledged Messages — delivered-but-unacknowledged messages on this node, latest value (latest(meter_rabbitmq_node_unacknowledged_messages)). Connections / Publishers / Consumers — the node\u0026rsquo;s connections, publishers, and consumers counts (latest(meter_rabbitmq_node_connections_total), latest(meter_rabbitmq_node_publisher_total), latest(meter_rabbitmq_node_consumer_total)). Channels + Queues — the node\u0026rsquo;s channels and queues counts (latest(meter_rabbitmq_node_channel_total), latest(meter_rabbitmq_node_queue_total)). Allocated Used % — percentage of the node\u0026rsquo;s allocated memory that is in use, latest value (latest(meter_rabbitmq_node_allocated_used_percent)). Memory (MB) — the node\u0026rsquo;s memory breakdown in MB: used, unused, resident, and total allocated (meter_rabbitmq_node_allocated_used_bytes, meter_rabbitmq_node_allocated_unused_bytes, meter_rabbitmq_node_process_resident_memory_bytes, meter_rabbitmq_node_allocated_total_bytes). Allocated By Type (MB) — allocated memory broken down by allocator type in MB, one series per type (meter_rabbitmq_node_allocated_by_type). Multi/Single-block Memory (MB) — allocator block usage in MB across multi used, multi unused, single used, and single unused (meter_rabbitmq_node_allocated_multiblock_used, meter_rabbitmq_node_allocated_multiblock_unused, meter_rabbitmq_node_allocated_singleblock_used, meter_rabbitmq_node_allocated_singleblock_unused). Requirements The RABBITMQ dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nCluster (service-scope) metrics — the meter_rabbitmq_* family (memory and disk headroom, file descriptors and sockets, ready / pending / unroutable messages, the publish pipeline, and queue / channel / connection lifecycle counters) that drives the service list and Service dashboard. Node (instance-scope) metrics — the meter_rabbitmq_node_* family (message counters, connections / publishers / consumers, channels / queues, and the allocator memory breakdown) that drives the Instance dashboard. These metrics come from OAP\u0026rsquo;s RabbitMQ monitoring, which scrapes the broker\u0026rsquo;s Prometheus / OpenMetrics endpoint. See the RabbitMQ monitoring setup in the SkyWalking backend documentation for how to enable it. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the node-scope widgets stay empty until per-node data is reported.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/rabbitmq/","title":"\u003c!--"},{"body":" Redis The REDIS layer monitors Redis deployments scraped through OpenTelemetry\u0026rsquo;s Redis receiver and forwarded to OAP as meters. It groups under Databases in the sidebar and is a metrics-only layer: each Redis cluster is a service, and the individual Redis processes under it are instances.\nIn Horizon\u0026rsquo;s sidebar this layer\u0026rsquo;s services are listed as Redis clusters, and the processes under a cluster as Nodes. The REDIS layer enables only the Service and Instance scopes — it has no endpoint dashboard, no topology or maps, and no Traces or Logs tabs.\nThis page is the operator reference for the bundled REDIS dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled REDIS template; if an operator has published a customized REDIS template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every Redis cluster with four sortable columns, sorted by command throughput (Commands/s) by default:\nCommands/s — total commands per second across the cluster, summed over command types (aggregate_labels(meter_redis_total_commands_rate,sum(cmd))).\nHit Rate — keyspace hit rate as a percentage, averaged across the cluster (aggregate_labels(meter_redis_hit_rate,avg)).\nMemory % — used memory as a percentage of max memory across the cluster (latest(aggregate_labels(meter_redis_memory_used_bytes,sum)/aggregate_labels(meter_redis_memory_max_bytes,sum))*100).\nClients — connected clients across the cluster (latest(aggregate_labels(meter_redis_connected_clients,sum))).\nService dashboard The primary drill-down for one selected Redis cluster. All cluster-scope widgets aggregate over the nodes that make up the cluster.\nStatus cards\nUptime (days) — cluster uptime in days, taken from the longest-running node (latest(aggregate_labels(meter_redis_uptime,max))/3600/24).\nConnected Clients — total connected clients across the cluster (latest(aggregate_labels(meter_redis_connected_clients,sum))).\nBlocked Clients — total clients blocked on a blocking call across the cluster (latest(aggregate_labels(meter_redis_blocked_clients,sum))).\nMemory Usage — used memory as a percentage of max memory across the cluster (latest(aggregate_labels(meter_redis_memory_used_bytes,sum)/aggregate_labels(meter_redis_memory_max_bytes,sum))*100).\nCharts\nTotal Commands / s — commands per second across the cluster, summed over command types (aggregate_labels(meter_redis_total_commands_rate,sum(cmd))).\nHit Rate — keyspace hit rate as a percentage (aggregate_labels(meter_redis_hit_rate,avg)).\nAvg Command Time / s — mean per-command duration, total command duration divided by total command count, summed over command types (aggregate_labels(meter_redis_commands_duration,sum(cmd))/aggregate_labels(meter_redis_commands_total,sum(cmd))).\nNet I/O (KB) — network throughput in KB, split into in and out (aggregate_labels(meter_redis_net_input_bytes_total,sum)/1024, aggregate_labels(meter_redis_net_output_bytes_total,sum)/1024).\nKeys — keyspace size over time, split into total keys, evicted keys, and expired keys (aggregate_labels(meter_redis_db_keys,sum), aggregate_labels(meter_redis_evicted_keys_total,sum), aggregate_labels(meter_redis_expired_keys_total,sum)).\nSlow Commands — the top 10 slowest captured commands in ms, sampled by the SkyWalking agent at the call site (top_n(top_n_database_statement,10,des)). Each row carries the command text. Shows no data when OAP captured no slow commands in the window.\nInstance dashboard For one selected node (a single Redis process under the cluster).\nStatus cards\nUptime (days) — node uptime in days (latest(meter_redis_instance_uptime)/3600/24).\nConnected Clients — clients connected to this node (latest(meter_redis_instance_connected_clients)).\nBlocked Clients — clients blocked on a blocking call on this node (latest(meter_redis_instance_redis_blocked_clients)).\nMemory Max (MB) — configured max memory for this node in MB (latest(meter_redis_instance_memory_max_bytes)/1000/1000).\nCharts\nMemory Usage (%) — used memory as a percentage of max for this node (meter_redis_instance_memory_usage).\nCommands / s — commands per second on this node (meter_redis_instance_total_commands_rate).\nHit Rate — keyspace hit rate as a percentage for this node (meter_redis_instance_hit_rate).\nNet I/O (KB) — network throughput in KB for this node, split into in and out (meter_redis_instance_net_input_bytes_total/1024, meter_redis_instance_net_output_bytes_total/1024).\nKeys — keyspace size over time for this node, split into total, evicted, and expired keys (meter_redis_instance_db_keys, meter_redis_instance_evicted_keys_total, meter_redis_instance_expired_keys_total).\nTotal Command Time (s) — total time spent on commands per second for this node (meter_redis_instance_commands_duration_seconds_total_rate).\nAvg Command Time — mean time spent per command on this node (meter_redis_instance_average_time_spent_by_command).\nRequirements The REDIS dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nCluster meters — the meter_redis_* family (commands rate, hit rate, used / max memory, connected and blocked clients, uptime, network bytes, keyspace counts, command duration and count), aggregated by command label where the metric is per-command. These back the service list and the cluster dashboard.\nNode meters — the meter_redis_instance_* family (the same measures at single-process scope), which back the node dashboard.\nSampled records — top_n_database_statement for the Slow Commands list, captured by the SkyWalking agent at the call site when slow-command sampling is enabled.\nThese meters come from the OpenTelemetry Redis receiver; see SkyWalking\u0026rsquo;s Redis monitoring setup for how to wire the collector to OAP. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a node-scope metric is empty until that level of data is reported.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/redis/","title":"\u003c!--"},{"body":" RocketMQ The ROCKETMQ layer monitors Apache RocketMQ message-queue clusters. SkyWalking collects RocketMQ metrics over OpenTelemetry and rolls them up into cluster-, broker-, and topic-scope metrics, so operators can watch produce / consume throughput, message size, consumer latency and backlog, and broker disk and thread-pool pressure alongside the rest of their estate. See the upstream RocketMQ monitoring setup for how to wire the telemetry pipeline.\nIn Horizon\u0026rsquo;s sidebar this layer is named RocketMQ. Its services are listed as RocketMQ clusters, its instances as Brokers, and its endpoints as Topics. The ROCKETMQ layer enables the Service, Instance, and Endpoint sub-tabs; it has no topology, traces, or logs view.\nThis page is the operator reference for the bundled ROCKETMQ dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled ROCKETMQ template; if an operator has published a customized ROCKETMQ template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a cluster, the layer landing page lists every RocketMQ cluster with four sortable columns, sorted by Produce TPS by default:\nProduce TPS — messages produced per second across the cluster (meter_rocketmq_cluster_total_producer_tps).\nConsume TPS — messages consumed per second across the cluster (meter_rocketmq_cluster_total_consumer_tps).\nTopics — the latest topic count in the cluster (latest(meter_rocketmq_cluster_topic_count)).\nBrokers — the latest broker count in the cluster (latest(meter_rocketmq_cluster_broker_count)).\nService dashboard The primary drill-down for one selected RocketMQ cluster, mixing daily message volume, live throughput, disk and thread-pool health, and the cluster\u0026rsquo;s topic / broker totals.\nProduced Today — messages produced since the start of today (latest(meter_rocketmq_cluster_messages_produced_today)).\nConsumed Today — messages consumed since the start of today (latest(meter_rocketmq_cluster_messages_consumed_today)).\nProduced Yesterday — messages produced over the previous full day (latest(meter_rocketmq_cluster_messages_produced_until_yesterday)).\nConsumed Yesterday — messages consumed over the previous full day (latest(meter_rocketmq_cluster_messages_consumed_until_yesterday)).\nProducer / Consumer TPS — produce and consume throughput per second on one chart (meter_rocketmq_cluster_total_producer_tps, meter_rocketmq_cluster_total_consumer_tps).\nProducer / Consumer Message Size (MB) — produced and consumed message size, in MB (meter_rocketmq_cluster_producer_message_size/1024/1024, meter_rocketmq_cluster_consumer_message_size/1024/1024).\nMax Consumer Latency — the highest consumer latency seen across the cluster (latest(meter_rocketmq_cluster_max_consumer_latency)).\nCommitLog Disk Ratio (%) — how full the CommitLog disk is, in percent: the current ratio over time plus the latest maximum across brokers (meter_rocketmq_cluster_commitLog_disk_ratio, latest(meter_rocketmq_cluster_max_commitLog_disk_ratio)).\nThreadPool Queue Head Wait (ms) — how long the head request has waited in the pull and send broker thread-pool queues, in ms — a rising value signals broker back-pressure (meter_rocketmq_cluster_pull_threadPool_queue_head_wait_time, meter_rocketmq_cluster_send_threadPool_queue_head_wait_time).\nTopics — the latest topic count in the cluster (latest(meter_rocketmq_cluster_topic_count)).\nBrokers — the latest broker count in the cluster (latest(meter_rocketmq_cluster_broker_count)).\nInstance dashboard For one selected broker, focused on the broker\u0026rsquo;s produce / consume throughput and message size.\nProduce TPS — messages produced per second by this broker (meter_rocketmq_broker_produce_tps).\nConsume QPS — consume requests per second served by this broker (meter_rocketmq_broker_consume_qps).\nProducer Msg Size (MB) — produced message size on this broker, in MB (meter_rocketmq_broker_producer_message_size/1024/1024).\nConsumer Msg Size (MB) — consumed message size on this broker, in MB (meter_rocketmq_broker_consumer_message_size/1024/1024).\nEndpoint dashboard For one selected topic, covering producer / consumer-group throughput, message size, consumer latency, offsets, and lag.\nProducer / Consumer Group TPS — produce throughput and consumer-group consume throughput per second on one chart (meter_rocketmq_topic_producer_tps, meter_rocketmq_topic_consumer_group_tps).\nMessage Size (MB) — produced and consumed message size for the topic, in MB (meter_rocketmq_topic_producer_message_size/1024/1024, meter_rocketmq_topic_consumer_message_size/1024/1024).\nMax Message Size (MB) — the latest maximum produced and consumed message size for the topic, in MB (latest(meter_rocketmq_topic_max_producer_message_size)/1024/1024, latest(meter_rocketmq_topic_max_consumer_message_size)/1024/1024).\nConsumer Latency (s) — consumer latency for the topic, in seconds (meter_rocketmq_topic_consumer_latency/1000).\nProducer / Consumer Offsets — the topic\u0026rsquo;s producer offset and consumer-group offset over time (meter_rocketmq_topic_producer_offset, meter_rocketmq_topic_consumer_group_offset).\nBacklogged Messages — the topic lag: producer offset minus consumer-group offset, the count of produced messages a consumer group has not yet consumed (meter_rocketmq_topic_producer_offset-meter_rocketmq_topic_consumer_group_offset).\nConsumer Group Count — the latest number of consumer groups on the topic (latest(meter_rocketmq_topic_consumer_group_count)).\nBroker Count — the latest number of brokers serving the topic (latest(meter_rocketmq_topic_broker_count)).\nRequirements The ROCKETMQ dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the RocketMQ meter families, produced from cluster telemetry collected over OpenTelemetry:\nCluster metrics — the meter_rocketmq_cluster_* family (total producer / consumer TPS, messages produced / consumed today and yesterday, producer / consumer message size, max consumer latency, CommitLog disk ratio, the pull / send thread-pool queue head-wait timers, and the topic / broker counts) for the service list and the cluster dashboard.\nBroker metrics — the meter_rocketmq_broker_* family (produce TPS, consume QPS, producer / consumer message size) for the broker dashboard.\nTopic metrics — the meter_rocketmq_topic_* family (producer TPS and consumer-group TPS, producer / consumer message size and their maxima, consumer latency, producer and consumer-group offsets, and the consumer-group / broker counts) for the topic dashboard.\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a broker- or topic-scope metric is empty until that level of data is reported. See the upstream RocketMQ monitoring setup for the collection pipeline that produces these families.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/rocketmq/","title":"\u003c!--"},{"body":" Go Agent (Self-Observability) The SO11Y_GO_AGENT layer is the self-observability view of the SkyWalking Go agent itself. It does not measure the application the agent instruments — it measures the agent\u0026rsquo;s own tracing machinery: how many tracing contexts it creates and finishes, how many it ignores, where contexts may have leaked, and how long the agent spends building them. Use it to confirm a Go agent is healthy and not accumulating leaked contexts or interceptor errors.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Self-Observability and named Go Agent. Its services are listed as Agent services and its instances as Agents — each Agent is one running Go process reporting these meters. This is an instance-only layer: it enables the Instance sub-tab and nothing else. There is no Service dashboard, no Endpoint dashboard, no Topology, and no Traces or Logs tabs — the agent reports a flat set of self-observability meters per process, with no service-, endpoint-, or relation-scoped data behind them.\nThis page is the operator reference for the bundled SO11Y_GO_AGENT dashboard: what you see on the Agent (instance) scope and what each widget means.\nThe widgets and metrics below are read from the bundled SO11Y_GO_AGENT template; if an operator has published a customized SO11Y_GO_AGENT template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nAgent list Selecting the layer lists the Agent services, and under one Agent service its Agents (instances) — one row per reporting Go process. This layer adds no extra landing columns, so the list is the plain name list; pick an Agent to open its dashboard.\nInstance dashboard For one selected Agent (instance). Every widget on this dashboard is a Go-agent self-observability meter, charted over the selected time window.\nTracing Context Creation / min — tracing contexts the agent created per minute (meter_sw_go_created_tracing_context_count). This is the agent\u0026rsquo;s working rate — how many trace contexts it is spinning up to follow requests.\nTracing Created + Finished / min — created vs finished tracing contexts per minute on one chart: the created series (aggregate_labels(meter_sw_go_created_tracing_context_count,sum)) against the finished series (meter_sw_go_finished_tracing_context_count). In a healthy agent the two lines track each other; a persistent gap (created running ahead of finished) is the signal that contexts are not being closed.\nIgnored Context Creation / min — contexts the agent created but deliberately ignored per minute (meter_sw_go_created_ignored_context_count), e.g. traffic filtered out of tracing.\nIgnored Created + Finished / min — the same created-vs-finished comparison for ignored contexts: created (aggregate_labels(meter_sw_go_created_ignored_context_count,sum)) against finished (meter_sw_go_finished_ignored_context_count).\nPossible Leaked Context / min — contexts the agent flags as possibly leaked per minute (meter_sw_go_possible_leaked_context_count). A non-zero, sustained line here points at instrumentation that opens a context without closing it — the key health signal on this dashboard.\nInterceptor Error Count / min — errors raised inside the agent\u0026rsquo;s interceptors per minute (meter_sw_go_interceptor_error_count). Rising values indicate the agent is failing while wrapping calls, which can mean lost or incomplete traces.\nTracing Context Execution Time (ms) — the time the agent spends building a tracing context, as a p50 / p75 / p90 / p95 / p99 latency distribution in milliseconds (relabels(meter_sw_go_tracing_context_execution_time_percentile,p='50,75,90,95,99',p='50,75,90,95,99')/1000000). The agent reports this percentile in nanoseconds, so the dashboard divides by 1,000,000 to display milliseconds. Watch the tail (p95 / p99) for instrumentation overhead.\nRequirements The SO11Y_GO_AGENT dashboard is a pure consumer of what the Go agent reports through OAP — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs the Go agent\u0026rsquo;s self-observability meter family at instance scope:\nAgent self-observability meters — the meter_sw_go_* family: created / finished / ignored / leaked tracing-context counts, interceptor error count, and the tracing-context execution-time percentile. These are emitted by the SkyWalking Go agent\u0026rsquo;s own self-observability reporting, not derived from the traced application. Every metric here is queried at the ServiceInstance (Agent) scope; OAP does not roll a metric up across scopes, so the dashboard stays empty until a Go agent is actively reporting these meters. When the meter family is missing entirely — for example a Go agent build with self-observability disabled — the widgets render no data rather than failing.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/so11y_go_agent/","title":"\u003c!--"},{"body":" Java Agent (Self-Observability) The SO11Y_JAVA_AGENT layer is the self-observability view of the SkyWalking Java agent itself — not the services it instruments, but the health of the agent running inside each Java process. It surfaces the agent\u0026rsquo;s own internal counters: how many tracing contexts it creates and finishes, how many it ignores, how many may have leaked, how often its interceptors error, and how long its tracing context bookkeeping takes. Use it to confirm an agent is healthy and to catch agent-side problems (context leaks, interceptor failures) that would otherwise be invisible from the application\u0026rsquo;s own metrics.\nIn Horizon\u0026rsquo;s sidebar this layer lives under the Self-Observability group and is named Java Agent. It has no service-level page: the layer reports per-agent, so its services are listed as Agent services and its instances as Agents, and the only drill-down it enables is the Instance (per-agent) dashboard. There is no Service, Endpoint, Topology, Traces, Logs, or profiling tab in this layer — agent self-observability is purely instance-scoped runtime telemetry.\nThis page is the operator reference for the bundled SO11Y_JAVA_AGENT dashboard: what you see on the agent dashboard and what each widget means.\nThe widgets and metrics below are read from the bundled SO11Y_JAVA_AGENT template; if an operator has published a customized copy to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nAgent list The layer landing page lists every reporting agent (Agents). This layer defines no extra landing columns, so the list is the agent roster on its own — pick an agent to open its dashboard.\nAgent dashboard The per-agent drill-down. Every widget is a time series of the agent\u0026rsquo;s own internal counters; the counts are per-minute rates and the one timing widget is in milliseconds.\nTracing Context Creation / min — how many tracing contexts the agent created per minute (meter_java_agent_created_tracing_context_count). This is the agent\u0026rsquo;s working rate: each context corresponds to a traced execution it started tracking. Tracing Created + Finished / min — created vs. finished tracing contexts on one chart, so you can see the two lines track each other (aggregate_labels(meter_java_agent_created_tracing_context_count,sum) as created, meter_java_agent_finished_tracing_context_count as finished). A persistent gap where created outruns finished points at contexts that never closed. Ignored Context Creation / min — contexts the agent deliberately skipped tracing per minute (meter_java_agent_created_ignored_context_count), for example traffic matched by the agent\u0026rsquo;s ignore/exclusion rules. Ignored Created + Finished / min — the created vs. finished pair for ignored contexts (aggregate_labels(meter_java_agent_created_ignored_context_count,sum) as created, meter_java_agent_finished_ignored_context_count as finished), the same balance check applied to the ignored path. Possible Leaked Context / min — contexts the agent suspects were leaked per minute (meter_java_agent_possible_leaked_context_count). A sustained non-zero line here is the headline agent-health warning: it usually means trace contexts are not being cleaned up correctly in the instrumented application. Interceptor Error Count / min — errors raised inside the agent\u0026rsquo;s bytecode interceptors per minute (meter_java_agent_interceptor_error_count). Non-zero values flag a misbehaving or incompatible plugin and warrant a look at the agent log. Tracing Context Execution Time (ms) — the p50 / p75 / p90 / p95 / p99 distribution of how long the agent\u0026rsquo;s tracing-context handling takes, in milliseconds (relabels(meter_java_agent_tracing_context_execution_time_percentile,p='50,75,90,95,99',p='50,75,90,95,99')/1000000). This is the agent\u0026rsquo;s own overhead tail; the percentile values are converted from nanoseconds to milliseconds for display. Requirements The SO11Y_JAVA_AGENT dashboard is a pure consumer of what the Java agent reports about itself — it invents no data, and a widget with no backing data simply reads no data. To populate it, the Java agent must have its self-observability (so11y) meters enabled so OAP receives the meter_java_agent_* family:\nContext counters — meter_java_agent_created_tracing_context_count, meter_java_agent_finished_tracing_context_count, meter_java_agent_created_ignored_context_count, meter_java_agent_finished_ignored_context_count, and meter_java_agent_possible_leaked_context_count for the creation, created-vs-finished, ignored, and leaked widgets. Interceptor errors — meter_java_agent_interceptor_error_count for the interceptor error widget. Execution-time percentiles — meter_java_agent_tracing_context_execution_time_percentile for the execution-time tail. All of these are reported at the ServiceInstance scope (one agent = one instance), which is why this layer has only the agent dashboard and no service, endpoint, or topology view. An agent that does not emit the self-observability meter family will appear in the list but render no data on every widget.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/so11y_java_agent/","title":"\u003c!--"},{"body":" OAP (Self-Observability) The SO11Y_OAP layer is SkyWalking\u0026rsquo;s own self-observability — the OAP backend reporting metrics about itself. It answers \u0026ldquo;is the backend healthy?\u0026rdquo;: each OAP node\u0026rsquo;s JVM, the analysis pipelines it runs (trace / mesh / OTEL / K8s ALS), the GraphQL query surface the UI itself hits, and the storage backend it persists to. This is the layer you watch to tell whether OAP — not the services it monitors — is the bottleneck.\nIn Horizon\u0026rsquo;s sidebar this layer is named OAP, grouped under Self-Observability. Its services are listed as OAP services and its instances as OAP nodes — one node per running OAP backend in the cluster.\nUnlike the application layers, SO11Y_OAP is a node-only layer: it ships a single instance (OAP node) dashboard and no service, endpoint, topology, traces, or logs tabs. There is no per-node landing table — pick an OAP node and you land directly on its dashboard.\nThis page is the operator reference for the bundled SO11Y_OAP dashboard: what you see on the OAP-node dashboard and what each widget means.\nThe widgets and metrics below are read from the bundled SO11Y_OAP template; if an operator has published a customized SO11Y_OAP template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nOAP node dashboard For one selected OAP node. Every widget on this dashboard is OAP-node-scoped, fed by the meter_oap_* self-observability meter family.\nJVM health The runtime the OAP node runs on.\nCPU (%) — process CPU utilization for the OAP node (meter_oap_instance_cpu_percentage).\nJVM Memory (MB) — JVM heap memory used (meter_oap_instance_jvm_memory_bytes_used, converted to MB).\nGC Count / min — garbage-collection count per minute (meter_oap_instance_jvm_gc_count).\nGC Time (ms / min) — time spent in garbage collection per minute (meter_oap_instance_jvm_gc_time).\nBuffer Pool (MB) — JVM buffer-pool memory used (meter_oap_instance_jvm_buffer_pool_bytes_used, converted to MB).\nThread Count — JVM threads broken out as live, peak, and daemon (meter_oap_jvm_thread_live_count, meter_oap_jvm_thread_peak_count, meter_oap_jvm_thread_daemon_count).\nThread States — threads by state: runnable, timed-waiting, blocked, waiting (meter_oap_jvm_thread_runnable_count, meter_oap_jvm_thread_timed_waiting_count, meter_oap_jvm_thread_blocked_count, meter_oap_jvm_thread_waiting_count).\nClass Count — loaded, unloaded total, and loaded total classes (meter_oap_jvm_class_loaded_count, meter_oap_jvm_class_total_unloaded_count, meter_oap_jvm_class_total_loaded_count).\nMetrics aggregation and persistence How much work the analysis-and-write pipeline is doing on this node.\nAggregation / min — metrics aggregated per minute (meter_oap_instance_metrics_aggregation).\nPersistence Counts / min — persistence operations per minute, split into prepare and execute (meter_oap_instance_persistence_prepare_count, meter_oap_instance_persistence_execute_count).\nPersistent Cache / min — persistent-cache activity per minute (meter_oap_instance_metrics_persistent_cache).\nPersistence Prepare Latency (ms) — p50 / p75 / p90 / p95 / p99 latency of the persistence prepare phase (meter_oap_instance_persistence_prepare_percentile).\nPersistence Execute Latency (ms) — p50 / p75 / p90 / p95 / p99 latency of the persistence execute phase (meter_oap_instance_persistence_execute_percentile).\nAggregation Queue Usage (%) — fill level of the L1 and L2 metrics-aggregation queues, top-10 worst series each (meter_oap_instance_metrics_aggregation_queue_used_per_ten_thousand, level 1 and level 2). A queue trending toward 100% is back-pressure — OAP is ingesting faster than it can aggregate.\nQuery surface (GraphQL) The query API that Horizon (and any GraphQL client) hits.\nGraphQL Latency (ms) — p50 / p75 / p90 / p95 / p99 latency of GraphQL queries served by this node (meter_oap_graphql_query_latency_percentile).\nGraphQL Query Count — GraphQL queries per minute, split into total queries and errors (meter_oap_instance_graphql_query_count, meter_oap_instance_graphql_query_error_count).\nIngestion and analysis pipelines The receivers and analyzers turning raw telemetry into metrics.\nTrace Analysis / min — traces analyzed per minute, total vs errors (meter_oap_instance_trace_count, meter_oap_instance_trace_analysis_error_count).\nTrace Analysis Latency (ms) — p50 / p75 / p90 / p95 / p99 latency of trace analysis (meter_oap_instance_trace_latency_percentile).\nMesh Analysis / min — service-mesh telemetry analyzed per minute, total vs errors (meter_oap_instance_mesh_count, meter_oap_instance_mesh_analysis_error_count).\nMesh Analysis Latency (ms) — p50 / p75 / p90 / p95 / p99 latency of mesh analysis (meter_oap_instance_mesh_latency_percentile).\nOTEL Received / s — OpenTelemetry records received per second, broken out as metrics, logs, and spans (meter_oap_otel_metrics_received, meter_oap_otel_logs_received, meter_oap_otel_spans_received).\nK8S ALS — Kubernetes Access Log Service throughput: count, dropped, streams, and err streams (meter_oap_instance_k8s_als_count, meter_oap_instance_k8s_als_drop, meter_oap_instance_k8s_als_streams, meter_oap_instance_k8s_als_error_streams).\nWatermark Circuit Breaker — cumulative break and recover counters per listener; when OAP sheds load under memory pressure, breaks climb (meter_oap_instance_watermark_circuit_breaker_break_count, meter_oap_instance_watermark_circuit_breaker_recover_count).\nZipkin Spans Dropped — Zipkin spans dropped by this node, for deployments running the Zipkin receiver (meter_oap_instance_spans_dropped_count).\nStorage backend Write latency against whichever storage backend this OAP is configured with. These two widgets are storage-specific and only render when the matching backend is in use — a BanyanDB deployment shows the BanyanDB widget, an Elasticsearch deployment shows the Elasticsearch widget.\nBanyanDB Write Latency (ms) — write latency by catalog and operation: measure bulk, stream bulk, trace bulk, stream single, and property (meter_oap_banyandb_write_latency_percentile). Shown only when BanyanDB write metrics are present.\nElasticsearch Write Latency (ms) — write latency split into single (single write / update / delete) and bulk (meter_oap_elasticsearch_write_latency_percentile). Shown only when Elasticsearch write metrics are present.\nRequirements The SO11Y_OAP dashboard is a pure consumer of what OAP reports about itself — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs its self-observability telemetry enabled, which produces the meter_oap_* meter family:\nJVM and process metrics — meter_oap_instance_cpu_percentage, meter_oap_instance_jvm_*, and the meter_oap_jvm_thread_* / meter_oap_jvm_class_* families behind the JVM-health widgets.\nPipeline and persistence metrics — meter_oap_instance_metrics_aggregation, meter_oap_instance_persistence_*, meter_oap_instance_metrics_persistent_cache, and meter_oap_instance_metrics_aggregation_queue_used_per_ten_thousand for the aggregation / persistence widgets.\nQuery metrics — meter_oap_graphql_query_latency_percentile and meter_oap_instance_graphql_query_count / _error_count for the GraphQL surface.\nIngestion metrics — the trace, mesh, OTEL, K8s ALS, watermark, and Zipkin families (meter_oap_instance_trace_*, meter_oap_instance_mesh_*, meter_oap_otel_*, meter_oap_instance_k8s_als_*, meter_oap_instance_watermark_circuit_breaker_*, meter_oap_instance_spans_dropped_count). A pipeline that isn\u0026rsquo;t running on a given node simply reports nothing, and its widget reads no data.\nStorage metrics — meter_oap_banyandb_write_latency_percentile or meter_oap_elasticsearch_write_latency_percentile, depending on the configured storage backend; only the matching widget renders.\nEach metric is queried at the OAP-node (instance) scope; OAP does not roll a metric up across scopes, so the dashboard is empty until self-observability telemetry is reported by the OAP nodes themselves. See the OAP backend setup docs for enabling the self-observability telemetry source.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/so11y_oap/","title":"\u003c!--"},{"body":" Satellite (Self-Observability) The SO11Y_SATELLITE layer is SkyWalking\u0026rsquo;s self-observability view of Apache SkyWalking Satellite — the lightweight telemetry collector that sits in front of OAP, buffering and forwarding agent traffic. When a Satellite instance reports its own runtime metrics to OAP (via the OpenTelemetry receiver), each collector shows up here as a service so you can watch the collection tier the same way you watch instrumented applications.\nIn Horizon\u0026rsquo;s sidebar this layer is named Satellite, and it is grouped under Self-Observability alongside the other components SkyWalking monitors about itself. Its services are listed as Satellite services. This layer is intentionally focused: it enables only the Service scope — there are no instance, endpoint, topology, traces, or logs sub-tabs. Everything Satellite exposes is read at the service level.\nThis page is the operator reference for the bundled Satellite dashboard: what you see on the service scope and what each widget means.\nThe widgets and metrics below are read from the bundled SO11Y_SATELLITE template; if an operator has published a customized Satellite template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list The layer landing page lists every Satellite service that has reported. This layer defines no custom landing-page metric columns, so services are listed by name only — pick one to open its dashboard.\nService dashboard The dashboard for one selected Satellite collector. Every widget is a time-series line over the selected window, covering the collector\u0026rsquo;s connection load, host CPU, internal queue, and the four stages of its event pipeline. The queue and event widgets break their series out per Satellite pipeline (tracingpipe, jvmpipe, logpipe, meterpipe, …), so you can see which collection pipeline is driving the rate; Connection Count and CPU are single series.\nConnection Count — the number of gRPC connections the collector currently holds, i.e. how many upstream agents and downstream OAP links are attached (satellite_service_grpc_connect_count).\nCPU (%) — host CPU utilization of the process running the Satellite gRPC server, as a percentage (satellite_service_server_cpu_utilization).\nQueue Used — how much of the internal buffering queue is currently occupied. Watch this against the collector\u0026rsquo;s queue capacity — a queue that stays near full means Satellite is backing up and is at risk of dropping events (satellite_service_queue_used_count).\nReceive Events — events received from upstream agents per minute, the inbound rate into the collector (satellite_service_receive_event_count).\nFetch Events — events fetched into the pipeline per minute, the rate at which buffered data is pulled forward for processing (satellite_service_fetch_event_count).\nQueue Input / Output — two series on one chart that show whether the queue is keeping pace: input is events written into the queue per minute (satellite_service_queue_input_count) and output is events sent on to OAP per minute (satellite_service_send_event_count). When output tracks input the collector is draining as fast as it fills; a persistent gap is the same backlog signal as a full Queue Used.\nRequirements The Satellite dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs each Satellite instance to push its self-observability metrics to OAP\u0026rsquo;s OpenTelemetry receiver, where they are aggregated into the satellite_service_* family at Service scope:\nConnection and host metrics — satellite_service_grpc_connect_count (gRPC connections) and satellite_service_server_cpu_utilization (server-process CPU).\nQueue metrics — satellite_service_queue_used_count for current queue occupancy, plus satellite_service_queue_input_count for the inbound queue rate.\nEvent-pipeline metrics — satellite_service_receive_event_count, satellite_service_fetch_event_count, and satellite_service_send_event_count for the receive → fetch → send stages of the collection pipeline.\nEach metric is queried at its own OAP scope; this layer reports only at Service scope, so the dashboard stays empty until a Satellite instance is configured to export its runtime metrics and they reach OAP. For how to wire that export and the underlying metric rules, see the SkyWalking Satellite self-observability setup documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/so11y_satellite/","title":"\u003c!--"},{"body":" Virtual Cache The VIRTUAL_CACHE layer monitors the cache systems your services talk to — Redis, Memcached, and the like — as virtual targets. There is no agent inside the cache itself; the data is synthesized from the cache calls that instrumented services make, so each cache appears as a service whose traffic, latency, and success rate are reconstructed from the client side.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Virtual targets and named Virtual Cache. Its services are listed as Caches. The layer is single-scope: it ships only the Service (cache) dashboard — there are no instance, endpoint, topology, trace, or log tabs for virtual caches, so this page documents the Cache list and the Cache dashboard only.\nThis page is the operator reference for the bundled VIRTUAL_CACHE dashboard: what you see on the cache landing list and what each widget on the Cache dashboard means.\nThe widgets and metrics below are read from the bundled VIRTUAL_CACHE template; if an operator has published a customized VIRTUAL_CACHE template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nCache list Before opening a cache, the layer landing page lists every virtual cache with four sortable columns, sorted by access traffic (Access RPM) by default:\nAccess RPM — total cache accesses per minute (cache_access_cpm).\nLatency — average access latency in ms (cache_access_resp_time).\np95 — 95th-percentile access latency in ms (cache_access_percentile{p='95'}).\nError Rate — percent of failed accesses (100 - cache_access_sla/100).\nCache dashboard The drill-down for one selected cache. The dashboard splits into three views of the same traffic: the combined access (all operations), then read and write broken out separately, and finally the slowest captured commands.\nAccess (all operations)\nAccess Traffic — total cache accesses per minute (cache_access_cpm).\nAvg Access Latency — mean access latency in ms (cache_access_resp_time).\nAccess Success Rate — percent of successful accesses (cache_access_sla/100).\nAccess Latency Percentile — p50 / p75 / p90 / p95 / p99 access latency, the tail of the access-latency distribution (cache_access_percentile).\nRead\nRead Traffic — cache read operations per minute (cache_read_cpm).\nRead Avg Latency — mean read latency in ms (cache_read_resp_time).\nRead Success Rate — percent of successful reads (cache_read_sla/100).\nRead Latency Percentile — p50 / p75 / p90 / p95 / p99 read latency (cache_read_percentile).\nWrite\nWrite Traffic — cache write operations per minute (cache_write_cpm).\nWrite Avg Latency — mean write latency in ms (cache_write_resp_time).\nWrite Success Rate — percent of successful writes (cache_write_sla/100).\nWrite Latency Percentile — p50 / p75 / p90 / p95 / p99 write latency (cache_write_percentile).\nSlow commands\nSlow Read Commands — the 10 slowest captured read commands against this cache (top_n(top_n_cache_read_command, 10, des), ms). Each row is a single execution — click it to copy the command, or use the trace icon at the row head to open its originating trace (shown only when the sample carries one). Shows no data when OAP captured no slow read commands in the window.\nSlow Write Commands — the 10 slowest captured write commands against this cache (top_n(top_n_cache_write_command, 10, des), ms). Same row behavior as Slow Read Commands — click to copy, or open the originating trace when the sample has one. Shows no data when none were captured.\nRequirements The VIRTUAL_CACHE dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nCache-access metrics — the cache_access_* family (traffic, response time, SLA, percentiles), produced by OAP from the cache calls that instrumented services make.\nRead / write metrics — the cache_read_* and cache_write_* families, the same measures split by operation, for the Read and Write widgets.\nSampled records — top_n_cache_read_command and top_n_cache_write_command for the Slow Read / Write Commands lists, captured by OAP when slow-command sampling is enabled.\nEach metric is queried at the cache\u0026rsquo;s Service scope; OAP does not roll a metric up across scopes, so a widget stays empty until that measure is reported for the cache. Virtual-cache data only appears when the services calling the cache are instrumented and OAP\u0026rsquo;s virtual-cache analysis is enabled — see the Virtual Cache setup guide for how OAP derives these targets.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/virtual_cache/","title":"\u003c!--"},{"body":" Virtual Database The VIRTUAL_DATABASE layer is the conjugate view of database traffic: instead of monitoring the database server itself, it shows each database as a peer that your instrumented services talk to. SkyWalking\u0026rsquo;s language agents detect outbound database calls in their traces and synthesize a virtual database node from the connection\u0026rsquo;s peer address — so a database appears here whether or not it is independently monitored, reconstructed entirely from the caller\u0026rsquo;s perspective.\nIn Horizon\u0026rsquo;s sidebar this layer lives under the Virtual targets group and is named Virtual Database. Each synthesized database is listed as a Database. This is a virtual-target layer with a single scope: it enables only the Service scope — there is no instance or endpoint scope, no topology, and no traces or logs tabs. Everything you see is derived from the access traffic the calling agents reported, so the figures describe the database as seen by its clients, not by the database engine.\nThis page is the operator reference for the bundled Virtual Database dashboard: what you see on the scope and what each widget means.\nThe widgets and metrics below are read from the bundled VIRTUAL_DATABASE template; if an operator has published a customized VIRTUAL_DATABASE template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a database, the layer landing page lists every virtual database with four sortable columns, sorted by Access RPM by default:\nAccess RPM — accesses per minute against the database (database_access_cpm).\nLatency — average access latency in ms (database_access_resp_time).\np95 — 95th-percentile access latency in ms (database_access_percentile{p='95'}).\nError Rate — percent of accesses that threw (100 - database_access_sla/100).\nService dashboard The primary drill-down for one selected database.\nAccess Traffic — accesses per minute against the virtual database (database_access_cpm).\nAvg Response Time — mean access latency in ms (database_access_resp_time).\nSuccess Rate — percent of accesses that returned without throwing (database_access_sla/100).\nResponse Time Percentile — p50 / p75 / p90 / p95 / p99 access latency, the tail of the access-time distribution (database_access_percentile).\nSlow Statements — the top 20 slowest captured statements against this database, in ms, worst-first (top_n(top_n_database_statement, 20, des)). Each row is a single statement execution; click a row to copy the statement text, or use the trace icon at the row head to open its originating trace — shown only when the sample carries one. Reads no data when OAP captured no statements in the window.\nRequirements The Virtual Database dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs your services instrumented by SkyWalking language agents that capture database calls, which produces:\nDatabase access metrics — the database_access_* family: database_access_cpm (traffic), database_access_resp_time (latency), database_access_sla (success rate), and database_access_percentile (the latency tail). These back both the Service list and the Service dashboard.\nSampled statements — top_n_database_statement for the Slow Statements list, captured by OAP when slow-statement sampling is configured on the calling services.\nEach metric is queried at its own OAP scope; the whole layer lives at the service (database) scope, so the dashboard is empty until at least one instrumented service reports database access traffic. For the upstream setup — how virtual databases are detected and configured — see the virtual database documentation.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/virtual_database/","title":"\u003c!--"},{"body":" Virtual GenAI The VIRTUAL_GENAI layer monitors the GenAI / LLM providers your services talk to — OpenAI, Anthropic, and other model backends — as virtual targets. There is no agent inside the provider; the data is synthesized from the GenAI calls that instrumented services make, so each provider appears as a service whose request load, latency, success rate, token throughput, and estimated cost are reconstructed from the client side, then broken down per model.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Virtual targets and named Virtual GenAI. Its services are listed as GenAI Providers and its instances as Models. The VIRTUAL_GENAI layer enables the Service (GenAI Provider) and Instance (Model) dashboards only — it does not ship an Endpoint dashboard, a topology / service-map view, or Traces / Logs tabs, because the providers are monitored entirely through their GenAI meter families, which carry no per-endpoint scope or call graph.\nThis page is the operator reference for the bundled VIRTUAL_GENAI dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled VIRTUAL_GENAI template; if an operator has published a customized VIRTUAL_GENAI template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nService list Before opening a provider, the layer landing page lists every GenAI Provider with four sortable columns, sorted by traffic (RPM) by default:\nRPM — calls per minute to the provider (gen_ai_provider_cpm).\nLatency — average response time in ms (gen_ai_provider_resp_time).\nSuccess Rate — percent of successful calls (gen_ai_provider_sla/100).\nOutput Tokens — total output (completion) tokens produced over the window (latest(gen_ai_provider_output_tokens_sum)).\nService dashboard The primary drill-down for one selected GenAI Provider. The dashboard covers the request golden signals, the latency tail, token throughput split into input and output, and an estimated spend.\nCalls / min — calls per minute to the provider (gen_ai_provider_cpm).\nAvg Response Time — mean response time in ms (gen_ai_provider_resp_time).\nSuccess Rate — percent of successful calls (gen_ai_provider_sla/100).\nLatency Percentile — p50 / p75 / p90 / p95 / p99 response time, the tail of the latency distribution, in ms (gen_ai_provider_latency_percentile).\nInput Tokens — input (prompt) token throughput, shown as the running total over the window and the per-call average (latest(gen_ai_provider_input_tokens_sum), gen_ai_provider_input_tokens_avg).\nOutput Tokens — output (completion) token throughput, shown as the running total over the window and the per-call average (latest(gen_ai_provider_output_tokens_sum), gen_ai_provider_output_tokens_avg).\nEstimated Cost — estimated spend against the provider, shown as the total over the window and the per-call average (latest(gen_ai_provider_total_estimated_cost)/1000000, gen_ai_provider_avg_estimated_cost/1000000). OAP carries the cost in micro-units, so each series is divided by 1000000 to land in whole currency units.\nInstance dashboard For one selected Model of the provider. The same golden signals as the Service view, scoped to a single model, plus a streaming time-to-first-token timing.\nCalls / min — calls per minute to this model (gen_ai_model_call_cpm).\nAvg Latency — mean latency for this model, in ms (gen_ai_model_latency_avg).\nSuccess Rate — percent of successful calls to this model (gen_ai_model_sla/100).\nLatency Percentile — p50 / p75 / p90 / p95 / p99 latency for this model, in ms (gen_ai_model_latency_percentile).\nTTFT (Time to First Token) — time to first token for streaming responses, shown as both the average and the percentile distribution, in ms (gen_ai_model_ttft_avg, gen_ai_model_ttft_percentile).\nInput Tokens — input (prompt) token throughput for this model, shown as the running total over the window and the per-call average (latest(gen_ai_model_input_tokens_sum), gen_ai_model_input_tokens_avg).\nOutput Tokens — output (completion) token throughput for this model, shown as the running total over the window and the per-call average (latest(gen_ai_model_output_tokens_sum), gen_ai_model_output_tokens_avg).\nEstimated Cost — estimated spend against this model, shown as the total over the window and the per-call average (latest(gen_ai_model_total_estimated_cost)/1000000, gen_ai_model_avg_estimated_cost/1000000). As on the Service view, the micro-unit cost is divided by 1000000 to land in whole currency units.\nRequirements The VIRTUAL_GENAI dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nProvider (service) metrics — the gen_ai_provider_* family at Service scope: call load (gen_ai_provider_cpm), response time (gen_ai_provider_resp_time), SLA (gen_ai_provider_sla), latency percentile (gen_ai_provider_latency_percentile), input / output token sums and averages (gen_ai_provider_input_tokens_*, gen_ai_provider_output_tokens_*), and the estimated-cost totals and averages (gen_ai_provider_total_estimated_cost, gen_ai_provider_avg_estimated_cost).\nModel (instance) metrics — the gen_ai_model_* family at ServiceInstance scope: call load (gen_ai_model_call_cpm), latency average and percentile (gen_ai_model_latency_avg, gen_ai_model_latency_percentile), SLA (gen_ai_model_sla), the streaming time-to-first-token timings (gen_ai_model_ttft_avg, gen_ai_model_ttft_percentile), input / output token sums and averages (gen_ai_model_input_tokens_*, gen_ai_model_output_tokens_*), and the estimated-cost totals and averages (gen_ai_model_total_estimated_cost, gen_ai_model_avg_estimated_cost).\nEach metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so a model-scope (instance) metric is empty until that level of data is reported. Virtual-GenAI data only appears when the services calling the provider are instrumented and OAP\u0026rsquo;s virtual-GenAI analysis is enabled — see the Virtual GenAI setup guide for how OAP derives these targets.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/virtual_genai/","title":"\u003c!--"},{"body":" Virtual MQ The VIRTUAL_MQ layer monitors the message-queue systems your services publish to and consume from — Kafka, RocketMQ, RabbitMQ, Pulsar, and the like — as virtual targets. There is no agent inside the broker itself; the data is synthesized from the produce and consume calls that instrumented services make, so each message-queue cluster appears as a service whose throughput, latency, and success rate are reconstructed from the client side.\nIn Horizon\u0026rsquo;s sidebar this layer is grouped under Virtual targets and named Virtual MQ. Its services are listed as MQ clusters, and its endpoints — the queues / topics a cluster carries — are listed as Topics. The layer enables two scopes: the Service (MQ cluster) dashboard and the Endpoint (Topic) dashboard. There are no instance, topology, trace, or log tabs for virtual MQ, so this page documents the cluster list, the MQ cluster dashboard, and the Topic dashboard only.\nThis page is the operator reference for the bundled VIRTUAL_MQ dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled VIRTUAL_MQ template; if an operator has published a customized VIRTUAL_MQ template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nMQ cluster list Before opening a cluster, the layer landing page lists every MQ cluster with four sortable columns, sorted by consume throughput (Consume RPM) by default:\nConsume RPM — messages consumed per minute across the cluster (mq_service_consume_cpm).\nProduce RPM — messages produced per minute across the cluster (mq_service_produce_cpm).\nConsume Latency — average time between message production and consumption, in ms (mq_service_consume_latency).\nConsume Error Rate — percent of failed consume operations (100 - mq_service_consume_sla/100).\nMQ cluster dashboard The drill-down for one selected MQ cluster. The dashboard pairs the produce and consume sides of the cluster\u0026rsquo;s traffic — throughput, success rate, and the consume-latency profile.\nConsume Traffic — messages consumed per minute (mq_service_consume_cpm).\nProduce Traffic — messages produced per minute (mq_service_produce_cpm).\nConsume Avg Latency — average time between message production and consumption, in ms (mq_service_consume_latency).\nConsume Success Rate — percent of successful consume operations (mq_service_consume_sla/100).\nProduce Success Rate — percent of successful produce operations (mq_service_produce_sla/100).\nConsume Latency Percentile — p50 / p75 / p90 / p95 / p99 of consume latency, the tail of the consume-latency distribution (mq_service_consume_percentile).\nTopic dashboard For one selected Topic — a queue / topic under the cluster, on the Endpoint scope. It mirrors the cluster widgets at the per-topic level.\nTopic Consume Traffic — messages consumed per minute on the topic (mq_endpoint_consume_cpm).\nTopic Produce Traffic — messages produced per minute on the topic (mq_endpoint_produce_cpm).\nTopic Consume Avg Latency — average consume latency for the topic, in ms (mq_endpoint_consume_latency).\nTopic Consume Success Rate — percent of successful consume operations on the topic (mq_endpoint_consume_sla/100).\nTopic Produce Success Rate — percent of successful produce operations on the topic (mq_endpoint_produce_sla/100).\nTopic Consume Latency Percentile — p50 / p75 / p90 / p95 / p99 of the topic\u0026rsquo;s consume latency (mq_endpoint_consume_percentile).\nRequirements The VIRTUAL_MQ dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nMQ cluster metrics — the mq_service_* family (consume / produce throughput, consume latency, consume / produce SLA, consume percentiles), produced by OAP from the produce and consume calls that instrumented services make.\nTopic metrics — the mq_endpoint_* family, the same measures evaluated at the Endpoint (Topic) scope, for the Topic dashboard.\nEach metric is queried at its own OAP scope — the mq_service_* family at the MQ cluster\u0026rsquo;s Service scope and the mq_endpoint_* family at the Topic\u0026rsquo;s Endpoint scope. OAP does not roll a metric up across scopes, so a Topic widget stays empty until that measure is reported at the Topic level, independent of the cluster-scope data. Virtual-MQ data only appears when the services producing to and consuming from the broker are instrumented and OAP\u0026rsquo;s virtual-MQ analysis is enabled — see the Virtual MQ setup guide for how OAP derives these targets.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/virtual_mq/","title":"\u003c!--"},{"body":" WeChat Mini Program The WECHAT_MINI_PROGRAM layer holds WeChat (微信) Mini Programs monitored by the SkyWalking mini-program agent. The agent runs inside the mini-program runtime and reports client-side performance — app launch, first render, package load, page routing, script execution, and outbound request timing — so each mini-program lands here rather than in a server-side layer.\nIn Horizon\u0026rsquo;s sidebar this layer is named WeChat Mini Program (under the Mobile group). Its services are listed as Mini-programs, instances as Versions (one per released mini-program version), and endpoints as Pages (one per mini-program page). The layer enables the Service, Version, Page, Traces, and Logs sub-tabs. It has no service map, instance map, or page-dependency view — mini-program telemetry is client-side timing, with no inter-service call topology to draw.\nThis page is the operator reference for the bundled WECHAT_MINI_PROGRAM dashboard: what you see on each scope and what each widget means.\nThe widgets and metrics below are read from the bundled WECHAT_MINI_PROGRAM template; if an operator has published a customized template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.\nMini-program list Before opening a mini-program, the layer landing page lists every WECHAT_MINI_PROGRAM service with four sortable columns, sorted by request traffic (Request RPM) by default:\nRequest RPM — outbound requests per minute (meter_wechat_mp_request_cpm).\nLaunch — average app-launch duration in ms (meter_wechat_mp_app_launch_duration).\nFirst Render — average first-render duration in ms (meter_wechat_mp_first_render_duration).\nErrors — count of reported errors (meter_wechat_mp_error_count).\nService dashboard The primary drill-down for one selected mini-program.\nApp Launch Duration — time to launch the mini-program, in ms (meter_wechat_mp_app_launch_duration).\nFirst Render Duration — time to the first render, in ms (meter_wechat_mp_first_render_duration).\nPackage Load Duration — time to download and parse the mini-program package bundle, in ms (meter_wechat_mp_package_load_duration).\nError Count — number of errors reported by the mini-program (meter_wechat_mp_error_count).\nRoute Duration — time spent in page-route transitions, in ms (meter_wechat_mp_route_duration).\nScript Duration — script-execution time, in ms (meter_wechat_mp_script_duration).\nRequest Load — outbound requests per minute (meter_wechat_mp_request_cpm).\nRequest Duration Percentile — p50 / p75 / p90 / p95 / p99 of outbound request duration, in ms — the tail of the request-timing distribution (meter_wechat_mp_request_duration_percentile).\nVersion dashboard For one selected released Version of the mini-program. The same timing families as the service dashboard, evaluated at version (instance) scope so you can compare one release against another.\nLaunch Duration — app-launch duration for this version, in ms (meter_wechat_mp_instance_app_launch_duration).\nFirst Render Duration — first-render duration for this version, in ms (meter_wechat_mp_instance_first_render_duration).\nPackage Load Duration — package download-and-parse time for this version, in ms (meter_wechat_mp_instance_package_load_duration).\nRequest Load — outbound requests per minute for this version (meter_wechat_mp_instance_request_cpm).\nRoute Duration — page-route transition time for this version, in ms (meter_wechat_mp_instance_route_duration).\nScript Duration — script-execution time for this version, in ms (meter_wechat_mp_instance_script_duration).\nRequest Duration Percentile — p50 / p75 / p90 / p95 / p99 of outbound request duration for this version, in ms (meter_wechat_mp_instance_request_duration_percentile).\nPage dashboard For one selected Page (endpoint) of the mini-program.\nLaunch Duration on Page — app-launch duration attributed to this page, in ms (meter_wechat_mp_endpoint_app_launch_duration).\nFirst Render Duration — first-render duration for this page, in ms (meter_wechat_mp_endpoint_first_render_duration).\nRequest Load — outbound requests per minute originating from this page (meter_wechat_mp_endpoint_request_cpm).\nRequest Duration Percentile — p50 / p75 / p90 / p95 / p99 of outbound request duration for this page, in ms (meter_wechat_mp_endpoint_request_duration_percentile).\nRequirements The WECHAT_MINI_PROGRAM dashboard is a pure consumer of what OAP reports — it invents no data, and a widget with no backing data simply reads no data. To populate it, OAP needs:\nMini-program (service) metrics — the meter_wechat_mp_* family at service scope: app launch, first render, package load, route, script, error count, request load, and request-duration percentile.\nVersion (instance) metrics — the meter_wechat_mp_instance_* family, the same timings reported per released version.\nPage (endpoint) metrics — the meter_wechat_mp_endpoint_* family, the launch / first-render / request-load / request-percentile timings reported per page.\nThese metrics come from the WeChat Mini Program agent reporting client-side timing to OAP, where the mini-program meter rules aggregate them. Each metric is queried at its own OAP scope; OAP does not roll a metric up across scopes, so the Version and Page dashboards stay empty until that level of data is reported. See the WeChat Mini Program monitoring setup for enabling the receiver and meter rules on OAP.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/dashboards/wechat_mini_program/","title":"\u003c!--"},{"body":" Alarms Path: /alarms. The page is read-only and needs no special permission to view.\nThe Alarms page is the triage surface for everything OAP\u0026rsquo;s alerting engine is firing right now, across every layer. It pulls the alarms OAP recorded over a recent window, groups the repeat firings of a rule on the same entity into a single incident, lays them out on a per-layer timeline, and shows the trigger expression and the captured metric snapshot for whichever alarm you select.\nAlarms are read-only here by design. OAP recovers an alarm automatically once the condition clears — there is no acknowledge, close, or silence action in the UI, and there is nothing to dismiss. A firing alarm stops firing when the underlying metric stops crossing the threshold; the page reflects that state, it does not drive it.\nThe time window The window picker offers three presets — 20m, 2h, 4h — plus a custom range capped at 4 hours.\nAlarms are second-precision events, and a long window pulls thousands of rows that some storage backends struggle to return; the 4-hour ceiling is enforced both in the picker and on the server, so a custom range wider than 4 hours is rejected. When the window genuinely holds more alarms than were fetched, the timeline header says so — narrow the window to see a complete slice. A window that exactly fills the fetch is complete and carries no notice.\nThe window\u0026rsquo;s starting preset can be set per deployment — see Alert page setup below.\nActive count and per-layer breakdown The KPI strip at the top counts what is actively firing, not the raw event count.\nActive — the total number of incidents that are currently firing. A fully recovered incident contributes nothing here, so this number answers \u0026ldquo;what is on fire right now?\u0026rdquo; rather than \u0026ldquo;what happened recently?\u0026rdquo;. Per-layer tiles — one tile per pinned layer (for example General, Mesh), each showing that layer\u0026rsquo;s active count. Pinned layers always render, even at zero, so the strip is stable across refreshes. Other — a read-only aggregate of active alarms in layers you did not pin, plus any alarm OAP could not attribute to a known layer. The arithmetic Active = (sum of pinned tiles) + Other always holds, so nothing hides off-screen. Overflow chips — below the tiles, the non-pinned layers that actually have an active alarm appear as small pills, sorted by count, as a filter shortcut. Clicking a tile, a chip, or a list tab narrows the timeline and the list to that layer; the selection is reflected in the URL, so a refresh or a shared link preserves it. Click the active tile again (or the Active tile) to clear the filter.\nFiltering Above the timeline is a filter row. What it offers depends on the connected OAP version:\nOn a current OAP, you get a cascading Layer → Service → Instance → Endpoint picker plus a free-text Keyword match on the alarm message. These filters are applied at the source, so the page only fetches the alarms that match. On an older OAP that does not support entity-scoped alarm queries, the row collapses to Keyword only, with a note inviting an upgrade for the full layer and entity filters. The filter is a draft until you press apply — nothing refires while you are composing it. clear resets every field.\nTimeline The timeline plots each alarm as a flag on a per-layer lane, so you can see at a glance when a burst happened and which layers it touched. It keeps every individual firing and recovery — not the merged incident — so a fire-then-recover pattern stays visible.\nTwo interactions:\nClick a flag to select that alarm and load its detail on the right. Brush a region to slice the list (and the counts) to that sub-window. The brushed rectangle is the only marker for the selection; the timeline itself still shows the full window so you can see other peaks to re-brush onto. reset clears the brushed range and the selected alarm.\nIncidents and the list OAP emits one alarm record per firing, so a rule that re-fires after its silence period produces several records. The list collapses the repeat firings of one rule on one entity into a single incident row, tagged with how many times it triggered. Each row carries a state:\nfiring — currently firing, and it never recovered within the window. unstable — currently firing, but it recovered at least once earlier in the window and fired again (a flapping rule). The badge shows how many of its firings are currently active versus recovered. Unstable still counts as active. recovered — the latest firing has cleared. Recovered incidents stay in the list as recent history but drop out of the Active count and the per-layer tiles — recovered is \u0026ldquo;no alarm\u0026rdquo;. For an incident that triggered more than once, the chevron at the end of the row expands a per-firing history: every individual firing and recovery on that entity and rule, in time order. Clicking a sub-entry loads that specific event into the detail panel. The list pages ten incidents at a time.\nAlarm detail Selecting an alarm — from a timeline flag, a list row, or an expanded history entry — opens the detail panel on the right:\nStatus — a firing or recovered pill, plus when the alarm started and (if cleared) when it recovered, and its layer. Message — the human-readable alarm text OAP formatted from the rule. Tags — any tags OAP attached to the alarm. Trigger expression — the MQE expression the rule evaluated, exactly as it fired. Rule — when the OAP admin port is reachable, the matched rule\u0026rsquo;s body: period, silence, recovery-obs, notification hooks, and the metrics it references. A \u0026ldquo;view in catalog\u0026rdquo; link jumps to the same rule on the Alerting rules page. When the admin port is unreachable, this section is omitted. Snapshot — one small chart per metric, plotting the values OAP captured at the firing moment so you can see what actually crossed the threshold. The trigger minute is marked, and the rule\u0026rsquo;s evaluation window is shaded when the rule body is available. An alarm recorded without an MQE snapshot (older OAP, or snapshot capture disabled in the rule) shows a note instead of charts. Admin: setup, pinned layers, and default window Which layers get their own KPI tile, and which window preset the page opens on, are configured on the Alert page setup admin page (/admin/alert-page-setup, verb alarm-setup:read), reachable from the page\u0026rsquo;s intro text.\nAlerting rules: the running context Path: /operate/alerting-rules. Verb: alarm-rule:read.\nThe Alerting rules page is a read-only catalog of every alarm rule loaded into the OAP cluster. Rules themselves are authored in OAP\u0026rsquo;s alarm-settings.yml and reloaded by OAP\u0026rsquo;s watcher — there is no add, edit, or delete here.\nEach rule lists its expression, window settings (period, silence, recovery-obs, and any additional period), the metrics it references, hooks, tags, entity include/exclude filters, and a per-node load state (loaded a/b) — because in a cluster each OAP instance loads the rule independently, and a partial count flags a node that has not picked it up.\nPer-entity running state Each OAP instance only evaluates a rule over the slice of entities it holds, so a rule\u0026rsquo;s Currently watching list is the union of evaluated entities across all nodes, with each entity tagged by the node watching it. Click an entity to open its live running context. Because the entity may be evaluated on only one node, the popup answers per node: the node actually evaluating it returns a populated body; the others read as \u0026ldquo;Not evaluated on this instance.\u0026rdquo;\nFor the evaluating node, the popup shows the rule\u0026rsquo;s current evaluation window (its size, the silence countdown, the recovery-observation countdown, and the window\u0026rsquo;s end time), the last alarm time and message, and a snapshot sparkline of the metric values in the window — each point annotated with its value and bucket time.\nThe headline of each node block is the rule\u0026rsquo;s current state for that entity. The states an operator will see:\nState Meaning FIRING The rule\u0026rsquo;s condition is currently met for this entity and the alarm is active. This is what surfaces as a firing alarm on the Alarms page. SILENCED_FIRING The condition is still met, but the alarm is inside its silence period after a recent firing, so OAP is holding off re-notifying. It is firing but quiet — no fresh notification goes out until the silence window elapses. OBSERVING_RECOVERY The condition has stopped being met and OAP is watching to confirm the recovery holds for the rule\u0026rsquo;s recovery-observation period before fully clearing the alarm. A flap back into breach during this window keeps the alarm active. These states are the live evaluation context behind the alarms you see on the Alarms page — they let you confirm that a rule is watching the entity you expect, see exactly where it is in the fire / silence / recover cycle, and read the very metric values it is acting on. The running context comes straight off OAP\u0026rsquo;s admin port; when that port is unreachable, the catalog surfaces a banner and the per-entity context is unavailable.\nRelated Runtime Rules (DSL) — runtime-editable MAL / LAL analysis rules that produce the metrics alarm rules evaluate. Metrics Inspect — browse the metric catalog and find which entities report a given metric. ","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/operate/alarms/","title":"\u003c!--"},{"body":" Events Events are the lifecycle records OAP has collected for a service — agent restarts, Kubernetes events, and other point-in-time facts reported by SkyWalking agents, the SkyWalking CLI, and the Kubernetes Event Exporter. Each event has a name, a type (Normal / Error), a message, and any reporter-supplied parameters. Events are distinct from alarms: an event records that something happened, not that a threshold was breached — for alerting, see Alarms.\nOpening the events popout Events are scoped to a single service and shown in a popout, so you review them without leaving the layer you\u0026rsquo;re on. On any layer drill-down, pick a service in the service banner at the top, then click the Events button next to the banner\u0026rsquo;s Share control. A modal opens for that service. The button appears only for users with the events:read permission (the built-in viewer, maintainer, and operator roles all have it).\nThe swimlane — instance × time The service is fixed (it\u0026rsquo;s in the popout title), so the view has two axes: each service instance is a row, and time runs left to right.\nAn event with a duration is a bar spanning its start to its end. An event with no end time is an instant marker (a small diamond). Each instance row is a distinct color, so the rows read apart at a glance. Error events carry a red ring so they stand out. If one instance reports overlapping events, they stack into sub-rows so nothing is hidden. A service that reports events without an instance shows a single row for the service. A rolling restart of a large service therefore shows as many bars at the same moment — one per instance — rather than a single summarised line. When a service runs many instances, use the search box at the top of the popout to filter the rows to the instances whose name matches.\nTime window and scrolling The popout owns its own window — 6h, 1d, 2d presets, plus a custom range — queried at second precision so the most recent events are never rounded out. The custom range takes an absolute start and end (entered in your browser\u0026rsquo;s local time) spanning up to 7 days; an invalid range — end before start, or a span past the 7-day cap — is rejected with the reason before anything is queried. A preset window is anchored to the moment you pick it, while a custom range is pinned exactly where you set it. Events are stored under OAP\u0026rsquo;s record retention; a window reaching past it simply returns fewer rows.\nScrolling stays inside the popout: the time-axis header stays pinned at the top and the instance column stays pinned at the left. A long range (a multi-day window) gets a wider, horizontally-scrollable canvas so bars keep a legible spacing instead of collapsing together, and the view opens scrolled to the newest events — scroll left for history. The time axis marks the date at day boundaries, so a range that crosses midnight is unambiguous.\nHow many events are shown The popout fetches the newest events up to a cap (200 by default; configurable under the server\u0026rsquo;s page-size limits). It tells you which case you\u0026rsquo;re in:\n\u0026ldquo;N events · all in range shown\u0026rdquo; — everything in the window is on screen. \u0026ldquo;Showing newest N — more available, narrow the range\u0026rdquo; — the window holds more than the cap; tighten the time range to reach older events. Event detail Click a bar to open the detail panel:\nHeader — the event type (Normal / Error) and name. Scope — the service, the instance (or \u0026ldquo;service-scoped\u0026rdquo;), the endpoint if present, and the layer. Started / Ended / Duration — for an event with a duration; a single Time for an instantaneous event. Message — the human-readable text the reporter attached. Parameters — the key/value details carried with the event (for example a Java agent\u0026rsquo;s startup options). Service names, instance names, messages, and parameter values are shown exactly as OAP reported them.\nRelated Alarms — threshold breaches from OAP\u0026rsquo;s alerting engine, a separate read-only triage surface. Traces and Logs — the other per-entity triage surfaces. ","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/operate/events/","title":"\u003c!--"},{"body":" 3D Infrastructure Map A single WebGL view of your whole deployment, stacked in 3D. Every SkyWalking layer\u0026rsquo;s services become cubes, grouped onto horizontal tiers, with live traffic, alarms, and call relationships drawn between them. It is the \u0026ldquo;stand back and look at everything at once\u0026rdquo; companion to the per-layer dashboards.\nOpen it from the 3D Infra pill in the topbar, or go directly to /3d/map. The map runs as a standalone full-screen view — no sidebar, no topbar, no global time picker — so the scene gets the whole viewport. The SkyWalking mark sits at the bottom-left; the × at the top-right returns you to the rest of Horizon.\nTiers A tier is a horizontal plane in the stack that groups related SkyWalking layers by their role in the system. Tiers are the spine of the map: they read top-to-bottom the way a request flows, from the apps a user touches down to the platform everything runs on.\nHorizon ships four bundled tiers:\nTier What lives here Examples Apps (top) The application surfaces and their direct dependencies as the app sees them General (agent) services, Browser/RUM, iOS, mini-programs, and the Virtual* targets (database / cache / MQ / gateway / GenAI) Middleware The data and messaging services, gateways, and self-observability MySQL, PostgreSQL, Redis, MongoDB, Elasticsearch, Kafka, RocketMQ, RabbitMQ, Pulsar, APISIX, Nginx, Kong, Flink, the SkyWalking SO11Y components, and cloud-managed data services Service Mesh The mesh that fronts the apps Istio managed services, Istio data plane (Envoy sidecars), Istio control plane, Cilium, Envoy AI Gateway Infra (bottom) The platform the rest runs on Kubernetes cluster + service, Linux/Windows hosts, virtual machines, EKS Every layer OAP reports is placed onto exactly one tier. A layer that Horizon hasn\u0026rsquo;t classified yet (for example a brand-new OAP layer) lands on the Middleware tier with an \u0026ldquo;unclassified\u0026rdquo; mark so an operator notices it and can re-assign it.\nThe tier list on the right-hand panel mirrors this stack. Click a tier row to fly the camera to it; use the eye toggle to show or hide every layer in that tier at once. The row also shows how many of the tier\u0026rsquo;s services are currently visible.\nReading the map Cubes Each cube is one service. Cubes are grouped into their layer\u0026rsquo;s zone on the tier, and each zone is colored with the layer\u0026rsquo;s brand color and stamped with the project\u0026rsquo;s logo (Istio\u0026rsquo;s sail, the Kubernetes helm wheel, a database cylinder, a queue, and so on) so you can identify a zone at a glance from any camera angle.\nLayers that ship a topology (General, Service Mesh, Kubernetes Service, Cilium) lay their cubes out by call dependency — upstream callers on one side, downstream services on the other — like the 2D service map. Layers without a topology pack their cubes into a tidy grid.\nTraffic A small pill under a cube shows that service\u0026rsquo;s live traffic — requests per minute for app and mesh services, queries or operations per second for data services, and so on, each with its own unit. The number is the service\u0026rsquo;s headline throughput metric for the current window.\nTraffic pills appear on cubes that are close enough to read; zoom out far enough and they fade away to keep the scene clean, then return as you zoom back in. A selected cube always shows its number.\nAlarms When a service has an alarm in the last 20 minutes, a small red beacon pulses on the top corner of its cube. The cube keeps its layer color — the beacon is the alert signal, so you can still tell which layer a troubled service belongs to. The alarm feed refreshes on its own while the map is open.\nConnections The map draws three kinds of lines:\nIn-layer calls — light cyan tubes between two services in the same layer, with animated packets flowing along them. This is each layer\u0026rsquo;s internal call graph. Cross-layer calls — soft orange arrows between services in different layers on the same tier (for example Browser → Frontend, or Frontend → Virtual Database). The arrow points from caller to callee. Hierarchy links — thicker gray tubes that connect the different views of the same logical service across tiers (for example a service seen by its agent, by the mesh, and as a Kubernetes service). These represent identity, not traffic, so they only appear when you select a cube, and show just that cube\u0026rsquo;s relatives — then disappear when you deselect. Interacting Camera — drag to rotate, scroll to zoom, and the on-screen toolbar (top-left) gives the same gestures as buttons. Arrow keys or WASD pan the view; hold Shift for a bigger step. Select a service — click a cube. It highlights, a detail card appears beside it (service name, layer, and an Open dashboard button that jumps to that service\u0026rsquo;s layer dashboard in a new tab), and its cross-tier hierarchy links light up. Click empty space, click another cube, or press Esc to deselect. Hover — hovering a cube shows a quick tooltip with the service\u0026rsquo;s name and layer next to it. Loading timeline Because a full deployment is too much to fetch in one request, the map loads in stages, and a slim timeline strip at the bottom shows the progress live:\nServices — the service roster and which layers they belong to. Templates — which layers carry a topology. Topologies — each topology-bearing layer\u0026rsquo;s call graph. Hierarchy — the cross-tier identity links between the different views of the same service. Only services that are new since the last run are fetched; the rest are reused, so a steady deployment costs nothing here on refresh. Layout — placing the cubes. Metrics — the per-service traffic numbers, fetched in batches so the cubes light up progressively. Each step shows its status as the map builds; click a step to open a drawer with its detail (services added/removed since last run, per-layer topology results, metric progress, and so on). A refresh button on the strip re-runs the whole sequence.\nConfiguration What the map shows is driven by a single configuration that an administrator edits in the UI at /admin/3d-map (linked under Dashboard setup in the sidebar). It is a structured editor — you work with tiers, layers, colors, and metrics through form controls, not raw JSON. Horizon ships a bundled default, seeded into OAP at first boot so the map is useful out of the box; your edits are kept as a local draft in your browser, and Check diff \u0026amp; push publishes them to OAP — the copy the map renders. In the default live template mode that OAP copy is the only source: if the template store cannot be read, the map reports that instead of rendering the bundled default. See Configuration File → Template source mode.\nFrom the editor you can:\nFilter layers — one global layer filter, written as a regex. A layer it excludes is dropped from the map entirely. This is the only filter; everything it admits is then placed on a tier. Arrange tiers — rename tiers, reorder them top-to-bottom, and pin each layer to a tier. A layer you don\u0026rsquo;t pin lands on the failover tier you nominate, so nothing silently falls off the map. Group layers — cluster several related layers (for example the SkyWalking self-observability components) into one labelled block on a tier, while each member keeps its own cube color. Color layers — pick each layer\u0026rsquo;s brand color (used for the cube, zone, and stamp). Choose a traffic metric — for each layer, set the single throughput metric its cubes display: the MQE expression, a display label, and a unit. The bundled defaults are seeded from each layer\u0026rsquo;s dashboard template, so most layers show a sensible number out of the box. A read-only Service-map layers list shows which layers lay their cubes out as a call graph — that comes from each layer\u0026rsquo;s template (its service-map capability), not from this page.\nPushed changes take effect the next time the map is opened. A Reset action reloads either the shipped bundled default or OAP\u0026rsquo;s current version, so you can start over before saving.\nExport downloads the map\u0026rsquo;s in-use configuration — the version live on OAP, or the bundled default when OAP has none — as a JSON file, for backup, sharing, or moving it to another OAP. Import reads a configuration JSON file and loads it as a local draft; preview it, then Check diff \u0026amp; push to publish. Import never writes OAP directly, and a file that isn\u0026rsquo;t a valid 3D-map configuration is rejected with a message.\nTuning the metric fan-out The map\u0026rsquo;s loading stages run in batches, several requests at once. How aggressively they do this is governed by the performance.bulk.infra3d block in horizon.yaml — an operator setting, not part of the map configuration, so it is not in the structured editor and does not travel with an exported / imported map. Edit horizon.yaml; the change is hot-reloaded and takes effect the next time the map is opened:\nmetricConcurrency — how many metric batches load at the same time. Default 4, range 1–8. Raise it to fill the cubes faster on a large deployment when OAP has headroom; lower it (toward 1) if a busy OAP rejects or slows the burst of metric requests during the Metrics step. metricBulkSize — how many services share one metric request. Default 6, range 1–12. Larger means fewer requests, but OAP rejects an oversized request, so this is capped — leave it at the default unless you have a reason to change it. topologyConcurrency — how many layer call-graphs load at once during the Topologies step. Default 4, range 1–16. templateConcurrency — how many layer templates load at once during the Templates step. Default 8, range 1–32. The defaults are tuned for a typical deployment; only revisit these if the loading timeline stalls on the Metrics, Topologies, or Templates step, or if OAP returns errors under the load.\nViewing the map needs read access (infra-3d:read, held by the built-in viewer role and above). A role without it does not get the topbar entry to the map at all. Editing and publishing the configuration needs overview:write (operators and admins by default). See Roles and Permissions.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/operate/infra-3d-map/","title":"\u003c!--"},{"body":" Live Debugger Path: /operate/live-debug.\nThe Live Debugger captures, step by step, how a single analysis rule processes real data inside the connected OAP — so you can see why a metric comes out the way it does (or why it comes out empty) without reading the backend logs. You pick one rule, start a capture session, and OAP records each pipeline stage (input → filter → function → output) for a bounded number of executions. The capture runs on every reachable OAP node at once, so a rule that behaves differently on one node in a cluster is visible side by side. When you are done, you stop the session; captures are also saved locally so you can re-open them later.\nThis is a diagnostic surface for the same DSL families you edit on the Runtime Rules (DSL) page — it does not change any rule. Starting a session never alters collection; it attaches a recorder to the rule for the length of the session and detaches when you stop it or the retention window lapses.\nThe three DSL tabs The page is split into three tabs, one per DSL family. Each tab runs its own independent session, so you can have a MAL, a LAL, and an OAL capture going at the same time.\nTab DSL family What it debugs MAL Meter Analysis Language otel-rules, log-mal-rules, telegraf-rules, meter-analyzer-config — the meter pipeline for OTEL, log-derived, Telegraf, and agent-reported metrics. LAL Log Analysis Language lal — log parsing and extraction, capturable at block or statement granularity. OAL Observability Analysis Language the connected OAP\u0026rsquo;s OAL clauses — input source columns through aggregation and output. OAL rules are not runtime-editable (they are compiled into the OAP build), but they are still debuggable here — this is the one place you can watch an OAL clause execute against live source data.\nRunning a capture Pick a rule. For MAL, choose a rule file and then a specific metric inside it; for LAL choose the log rule (and block / statement granularity); for OAL choose the source and clause. Set the bounds. recordCap limits how many executions are captured (default and maximum 100). retention (min) is how long the session stays alive on OAP before it is reaped (default 5 minutes, maximum 60). Start. OAP installs the recorder across the cluster and begins collecting. The state pill moves through starting → capturing → captured. While capturing, the view refreshes about once a second; it stops polling on its own once every reachable node has finished (captured). Stop at any time to detach the recorder early. You do not have to wait for the retention window. Starting a new session for a rule that already has one running automatically replaces the prior session — the coverage strip notes how many prior sessions were stopped.\nCluster coverage strip Above the captured records, a per-node strip shows, for each OAP node, an install result (whether the recorder was accepted on that node) and a collect status (whether data came back). A rollup line summarizes how many nodes the session is live on (e.g. live on 2 of 3 nodes). Use it to spot a node that rejected the install or was unreachable — a missing node there explains a partial capture.\nReading the captured stages Each captured execution is shown as a chain of stages. Every stage reports an in → out count, so a stage that drops everything (a filter that matched nothing) is obvious at a glance. Clicking a stage highlights the matching fragment of the rule\u0026rsquo;s source text above the chain, tying the captured step back to the line of DSL that produced it.\nDiff-default label grouping When a stage emits many samples that share a metric name, they are grouped under a one-line summary rather than listed in full. Expanding a multi-sample group lands in diff mode by default: the labels that are identical across every sample collapse into a shared context shown once, and each sample row shows only the labels that differ. This makes \u0026ldquo;what distinguishes these series\u0026rdquo; the thing you see first. A toggle switches to the full per-sample label list when you want every label on every row. The same diff-first treatment applies to a run of output entities that share a metric — only the entity fields that vary are shown per row.\nVery large groups render a capped number of detail rows with a \u0026ldquo;+ N more\u0026rdquo; note; the summary count is always exact.\nThe LAL pipeline matrix A LAL capture renders as a grid — one column per captured record, one row per pipeline step (input, the per-statement or per-block function steps, output). The first column names each step and stays pinned as you scroll sideways through the records; each cell holds that record\u0026rsquo;s data at that step.\nIt reads any log format. A cell shows whatever fields OAP serialized for the record — a plain LogData input shows service / endpoint / tags / body, while an Envoy access-log (ALS) record shows its built snapshot (service, endpoint, response data, and the access-log content as JSON). When OAP cannot serialize a record\u0026rsquo;s raw input, the cell shows the reason (for example jsonformat-failed …) instead of rendering blank, and a small label names each cell\u0026rsquo;s payload class.\nFilter a row to the records that have data. A step row that has gaps carries a filter; turning it on narrows the grid to just the records that produced data for that step — for example the output row to only the records that emitted output (an abnormal-only rule aborts most records, so only a few reach output). The row count shows how many of all captured records reached that step.\nInspect and diff a cell. Each cell has a button — VIEW on the input row, DIFF on the builder rows — that opens the cell\u0026rsquo;s complete payload in a JSON viewer with the log content shown as formatted JSON. For the built-log snapshots you can compare stages: a picker presents the captured rule with each per-statement step on its line and the extractor / sink blocks as selectable ranges, and choosing one shows a side-by-side diff of the two snapshots — the quickest way to see which statement or stage added, changed, or dropped a field.\nEach OAP node renders its own matrix; filtering or selecting in one node\u0026rsquo;s grid does not affect another\u0026rsquo;s.\nCapture history Every session you run is saved to capture history, browse it at /operate/live-debug/history (or the history link on each tab). History is stored locally in your browser — it is not shared between users or machines and survives reloads, with the most recent captures kept per DSL family.\nFrom history you can:\nReplay a finished capture — re-open the recorded stages exactly as they were captured, without re-running anything on OAP. A banner marks that you are viewing a saved capture, with a back to live control to return. Resume a capture whose retention window has not yet lapsed — re-attach to the still-live OAP session and continue polling it. A capture that was archived before its first poll returned data shows as having no records; run a longer-lived capture to give the pipeline time to fire.\nRequirements The OAP dsl-debugging module must be loaded. This is the module that powers start / poll / stop across MAL / LAL / OAL; the page shows a warning banner when it is missing. See Required OAP Modules. The receiver-runtime-rule module must also be loaded — it backs the rule picker (the catalog of rules you choose from). It is a separate module from dsl-debugging: a deployment can have one without the other, in which case either the picker or the capture itself will be unavailable. OAP admin port reachable from Horizon. Access control Permission Grants live-debug:read View the Live Debugger, the active-session list, cluster status, and capture history. Nothing else is required to watch a capture. live-debug:write Start and stop capture sessions. Nothing else is required to run one — no rule:* grant takes part, and holding every rule verb without live-debug:* gets you nothing here. In the bundled roles, both are held by operator (and admin). A read-only viewer can be granted live-debug:read on its own to inspect existing sessions and history without being able to start new captures. See Roles and Permissions.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/operate/live-debugger/","title":"\u003c!--"},{"body":" Log Inspect Log Inspect (/operate/log-inspect) is the cross-layer log query tool in the sidebar. The per-layer Logs tab starts from a layer and the service picked in its header; this page starts from nothing. Open it, name any service — or no service at all — and query across everything the log store holds. It unifies three log sources on one page: the stored log stream, browser JavaScript errors, and on-demand Kubernetes pod tails.\nLike the other triage pages, it owns its own time range — the global topbar time picker does not apply — and (for the stored sources) nothing is fetched until you press Run query. Conditions are staged; switching source clears the previous result so streams never mix.\nSources The Source toggle at the top picks what kind of logs you are after:\nRaw — the logs SkyWalking has collected and stored: the same store as the per-layer Logs tab, queried across every layer. Browser — the JavaScript errors reported by the browser agent, with inline source-map management and per-row stack de-obfuscation. Kubernetes Pod logs — a live tail of one pod\u0026rsquo;s container output, pulled through OAP on demand and never persisted. Target — pick it, type it, or leave it blank For the Raw and Browser sources the Target is optional: blank queries every service in the window. Two modes scope it:\nPick — choose a Layer, then a Service from its catalog, then optionally an Instance and/or Endpoint. On the Browser source these last two are labelled Version and Page, because that is what a browser app\u0026rsquo;s instances and endpoints are. Type — enter a Service name directly, with a Real checkbox (off for a virtual/peer service), plus optional instance/endpoint (version/page) names. Typing needs no layer. The → edit as text link converts the current Pick selection into the Type form. Raw and Browser share one target, so switching between them keeps your pick; only crossing into or out of the pods source resets it.\nRaw — stored logs across layers Conditions for the stored stream:\nCondition What it does Tags Comma-separated key=value pairs, AND-joined, with autocomplete: type to see known keys, type = for that key\u0026rsquo;s known values, Enter commits the pair and primes a comma for the next. Filter by level with a level=… tag. Trace ID Show only the lines correlated with one trace. Time A rolling preset (15m, 30m, 1h, 3h, 6h, 12h, 24h) or Custom… with an absolute start/end pair. Second-precision, like the per-layer tab. Limit Result cap: 20, 50 (default), 100, or 200. The server additionally caps a single batch at its configured page-size limit (100 by default), so 200 only takes effect when that limit has been raised — see the query-limits section of the horizon.yaml reference. Run query fetches one batch of the newest matching lines. Rows render exactly as on the per-layer tab — timestamp, level, service, an ↗ trace link when trace-correlated, a format chip, and a one-line preview — and clicking a row opens the same full-payload popout: format-aware pretty-printing, Copy, the service/instance/endpoint/trace context, and the tag table. The ↗ trace links open the related trace\u0026rsquo;s waterfall in an overlay without leaving the page. Escape or the backdrop closes the popout, and a re-run that no longer contains the open row closes it too.\nUnlike the per-layer Logs tab there is no density histogram, no Levels strip, and no pager — this page returns a single batch capped by Limit. It trades the browsing chrome for reach: any service, any layer, or all of them at once.\nBrowser — JS errors with source-map resolve The Browser source queries the errors browser agents report, across every browser app at once if you leave the target blank. Its conditions are Category (All, or one of AJAX, RESOURCE, VUE, PROMISE, JS, UNKNOWN) plus the shared Time and Limit.\nBelow the conditions sits the source-map manager — the same map store the per-layer Browser Logs tab uses, managed inline so you never have to leave the page to make a stack readable. It lists the maps currently available (statically mounted ones and temporary uploads), shows the memory-usage bar, and offers Upload .map and per-upload remove. Uploaded maps live in server memory only; mounted maps cannot be removed here. If de-obfuscation is disabled on the server, the manager says so instead.\nResults render as a dense error list: time, category (color-keyed), page, app version, and the message. Click a row to open the browser-error popout — the error\u0026rsquo;s metadata and raw stack on one side, and the de-obfuscation control on the other: pick a hosted map (the first one is pre-selected), press Resolve, and read the original file/line/symbol frames with source snippets. Which map matches which build is your call — see Browser Logs \u0026amp; Source Maps for the matching rules and the resolvable categories.\nKubernetes Pod logs — live tails without entering a layer The pods source is the cross-layer twin of the per-layer Pod Logs tab: it tails one pod\u0026rsquo;s container output straight from the cluster through OAP. Nothing is persisted — each poll pulls the trailing window and discards it — so the pod must be currently running.\nUnlike the other two sources, the target here is required: a specific pod and container.\nPick a Layer and a Service (with exactly one Kubernetes-aware layer in your menu, the layer is pre-selected) — or switch the service field to Type and enter the service name directly, no layer needed. Pick the Pod — the service instance. A single-pod service is auto-selected. Pick the Container — the pod\u0026rsquo;s containers are listed and the first is auto-selected. Choose the trailing Window (Last 30s to Last 30m) and the poll Interval (2s–30s). Press Start to tail live, Pause to stop, or Refresh for a one-shot fetch (which also pauses a running tail). Include / Exclude chip fields narrow the lines: type a full-line regular expression (for example .*error.*) and press Enter to add it; the × removes a chip. Includes keep matching lines, excludes drop them, and changing them mid-tail re-runs with the new filters. Re-targeting the pod, container, or service stops the tail so a stale loop never bleeds across pods.\nOn-demand pod logs are disabled by default on OAP; when the feature is off or the pod no longer exists, the reason appears in a banner — see the pod-logs troubleshooting on the Logs page, which applies here unchanged.\nResolved query For the Raw and Browser sources, a Resolved query toggle appears after each run: it names the source and expands to the exact condition that was sent — resolved service ids, computed window, filled-in defaults. When a query returns something unexpected, read it first. Pod tails are live fetches rather than stored-store queries, so they have no resolved-query panel.\nPermissions The page and the raw/browser queries require the inspect:read permission. The tag autocomplete, the container list, and the pod tail additionally use logs:read; the source-map list and stack resolve use browser-errors:read (uploading or removing maps needs source-map:write); and the Pick-mode layer/service dropdowns use metrics:read. The bundled roles that grant inspect:read include the read verbs. See Roles and Permissions.\nRelated Logs — the per-layer stored-log stream and Pod Logs tab, with the full condition and troubleshooting reference. Browser Logs \u0026amp; Source Maps — source-map matching rules, static provisioning, and which error categories resolve. Trace Inspect — the cross-layer sibling for traces. Metrics Inspect — the cross-layer view of OAP\u0026rsquo;s metric catalog. ","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/operate/log-inspect/","title":"\u003c!--"},{"body":" Logs Horizon surfaces logs through two distinct tabs, each backed by a different OAP source.\nThe Logs tab queries the logs SkyWalking has collected and stored — application and service log records, indexed and filterable, correlated with traces. The Pod Logs tab does something different: it live-tails a Kubernetes pod\u0026rsquo;s container logs on demand, pulled straight from the Kubernetes API through OAP and never persisted. They appear as separate tabs because they answer different questions — \u0026ldquo;what did this service log over the last half hour?\u0026rdquo; versus \u0026ldquo;what is this pod printing to stdout right now?\u0026rdquo;.\nWhich tabs a layer shows depends on the layer template. The Logs tab appears on layers whose template enables it (for example GENERAL, MESH, MESH_DP, NGINX, ENVOY_AI_GATEWAY, the mini-program and mobile layers). The Pod Logs tab appears only on the Kubernetes-aware layers K8S_SERVICE, MESH, and MESH_DP.\nFor browser JavaScript errors reported by the browser agent — a separate stream with its own source-map de-obfuscation — see Browser Logs \u0026amp; Source Maps. That is not the same as the collected service logs described here.\nFor cross-layer digs — querying any service\u0026rsquo;s stored logs by name (or all services at once), browser errors, or a pod tail without entering a layer — see Log Inspect.\nStored logs Open a layer that has a Logs tab and pick a service in the header. The stored log stream loads for that service over the page\u0026rsquo;s own time range, newest first.\nScoping and filtering The conditions bar narrows the stream. Every filter is optional; together they are AND-joined.\nInstance — restrict to one service instance. The default is All. On a sidecar layer this picker is labelled Sidecar.\nEndpoint — restrict to one endpoint. Type to search the endpoint list, then click a result to pin it; the × clears it back to All.\nTrace ID — paste a trace id to show only the log lines correlated with that trace. Copy the id from a trace\u0026rsquo;s span detail and paste it here; there is no one-click jump from a trace to its logs.\nContent — words the log line must contain, space-separated for AND (timeout db matches only lines carrying both). This field appears only when your storage backend can search log content — ElasticSearch can, BanyanDB and the others cannot, and Horizon asks the connected OAP which it is. On a backend that cannot, the field is absent rather than present-and-ignored, because OAP accepts the condition there and returns the unfiltered stream — which reads as \u0026ldquo;everything matched\u0026rdquo;.\nTags — a single key=value field with autocomplete. Start typing a key to see suggested keys; type = to switch the suggestions to known values for that key; press Enter to commit the tag. Committed tags show as removable chips under the bar and ride along on the query as additional filters.\nLevel — the Levels strip above the stream doubles as a filter. Click error, warn, info, or debug to show only that level; click again to clear. The level filter is sent to OAP as a level tag, so pagination and counts reflect the filtered set. The other chip (lines whose level tag is missing or unrecognized) is informational only — it has no server-side value to filter on, so it is not clickable.\nThe stream queries on demand, not on every keystroke. Editing a condition stages it; nothing is fetched until you press Run query, which runs the query and resets to the first page. A freshly opened tab shows a Pick your conditions, then click Run query prompt rather than auto-loading, and switching service resets to that prompt — clearing the level and tag filters — so the previous service\u0026rsquo;s logs never linger under the new one. Paging and the page-size picker fetch immediately once you have run a query.\nTime range The Logs tab owns its own time range — the global topbar time picker is paused while you are here, so auto-refresh won\u0026rsquo;t shift the window mid-investigation. Pick a rolling preset (Last 15 min through Last 24 hours, default Last 30 min) or choose Custom… to pin an absolute start/end with two date-time inputs.\nLog queries use second-precision time windows. Logs are record-style data anchored at second granularity, so the window is not rounded to the minute — the most recent (and usually most interesting) lines are never chopped off. The window is capped at 7 days. A custom range longer than that is refused on the page, with the reason under the control, rather than being quietly shortened — a query made directly against the API is trimmed to the most recent week instead, so it still answers with the part that matters.\nReading the stream A density histogram sits above the stream: time on the x-axis, log count on the y-axis, each bar stacked by level (error / warn / info / debug / other) with the same colour as the legend. Hover a bar to see that bucket\u0026rsquo;s time range and per-level counts. The histogram is built from the currently loaded page, so it shows the shape of what is on screen, not the whole window.\nThe Levels strip carries a count per level next to each chip. Those counts come from a window-scoped sample (a few hundred of the most recent rows in the window, larger than one page), so they reflect the window\u0026rsquo;s level distribution rather than only the visible page. The strip notes the sample size it used, and says when the window held more rows than the sample counted — narrow the window if you need the counts to cover all of it.\nEach row shows the timestamp, the level, the service (with any group prefix decoded), an ↗ trace link when the line is trace-correlated, a format chip (JSON / YAML / TEXT), and a one-line preview of the content. Rows are colour-keyed by level.\nHorizon renders the payload according to its content. OAP labels payloads as JSON or plain text; on top of that, Horizon sniffs for JSON and YAML structure so an unlabelled-but-structured body still gets the right treatment. JSON is compacted to a single line in the preview and pretty-printed in the detail view; YAML keeps its keys; plain text is whitespace-collapsed.\nClick a row to open the full payload in a popout: the complete content, format-aware pretty-printing, a Copy button, the service / instance / endpoint / trace context, and a table of all tags on the line. If the line is trace-correlated, an ↗ trace button there (and the ↗ trace link on the row) opens the related trace\u0026rsquo;s waterfall in an overlay without leaving the log stream — the row\u0026rsquo;s timestamp is passed along so the trace is found even when it sits in a colder storage tier. Press Escape or click the backdrop to close.\nThe pager at the foot shows the current page and the row count on it; Prev / Next walk the pages, and the page size (20, 50, or 100) is set on the conditions bar. There is no \u0026ldquo;N of M\u0026rdquo; total, because the log query does not report one — Next is offered only when there really is another page with rows on it, so a full last page ends the walk instead of stepping onto an empty screen. Changing the page size restarts at page 1.\nTroubleshooting stored logs No rows returned. Confirm the service actually ships logs to OAP, that the storage backend has the logs module enabled, and that the time range covers when the logs were produced. Narrow filters (a tag, a level, an endpoint) can also empty the result — clear them and widen the window.\nA filter empties the stream. Tag and level filters are exact-match on indexed dimensions. A level value or tag value that doesn\u0026rsquo;t exist in the stored data returns nothing; check the value against what the Levels counts and the tag autocomplete actually offer.\nRun query is greyed out. The tab does not yet know which service to read, and says which case it is: Resolving service… while the picked service is being looked up, or a note that the selected service is not in this layer (it aged out of OAP, was renamed, or the link points elsewhere) — pick another one. The stream is always read for one service, so the tab waits instead of querying the whole layer.\nPod logs The Pod Logs tab tails a Kubernetes pod\u0026rsquo;s container logs live. There is no stored history to page through — each refresh pulls the trailing window straight from the Kubernetes API through OAP, shows it, and discards it. Nothing is persisted.\nStarting a tail Pick a service in the header, then pick a Pod (a service instance) — the page is pinned to one pod at a time. Pick a Container. Horizon lists the pod\u0026rsquo;s containers and auto-selects the first; switch if the pod runs more than one. Choose the look-back Window (Last 30s, 1m, 5m, 15m, or 30m) — how far back each poll reaches. Choose the poll Interval (2s, 5s, 10s, or 30s) — how often the window is re-fetched while live. Press Start. The trailing window streams into a read-only viewer and re-polls on the interval until you press Pause. The header shows a live indicator, the line count, and how long ago the view last updated. Changing the container, window, interval, or filters while tailing re-runs the query with the new settings. The viewer is read-only and keeps the newest line in view as fresh logs arrive.\nInclude and exclude filters Two filter rows narrow what the tail shows. Include keeps only lines that match; Exclude drops lines that match. Type an expression and press Enter to add it as a chip; the × on a chip removes it. Both are evaluated by OAP as full-line regular expressions (for example .*error.*), so they match against the whole log line, not a substring. Multiple expressions in a row stack as additional conditions.\nTime precision Pod-log windows are second-precision — this is a live tail, anchored at the current second. OAP caps a single tail window at 30 minutes; the longest selectable window is Last 30m.\nTroubleshooting pod logs On-demand pod logs are disabled by default on OAP because container logs can leak secrets. When the feature is off, or when the pod can\u0026rsquo;t be resolved, OAP returns a reason instead of data and Horizon shows it in a banner rather than an empty pane. Two common cases:\n\u0026ldquo;Logs unavailable\u0026rdquo; with a reason. If the reason indicates the feature is off, enable on-demand pod logs on the OAP side. If it indicates the pod wasn\u0026rsquo;t found, the instance you picked points at a pod that no longer exists (a finished rollout or a scaled-down replica) — pick a currently-running pod.\nThe tail stops on its own. A pod that vanishes mid-tail (a rollout or scale-down) makes the next poll fail; Horizon stops the loop and surfaces the reason rather than spinning on errors. Re-pick a live pod and Start again.\nPermissions Both tabs — stored log queries, tag autocomplete, the container list, and the on-demand tail — require the logs:read permission. See Roles and Permissions.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/operate/logs/","title":"\u003c!--"},{"body":" Profiling Profiling drills past metrics and traces into the call stacks, kernel events, and process-to-process conversations of a running service. Horizon surfaces SkyWalking\u0026rsquo;s profiling capabilities as a set of per-layer tabs on the service you have selected: Trace Profiling, eBPF Profiling, Async Profiling, Network Profiling, and pprof. Each profiling tab only appears on a layer when OAP reports that the service supports that kind of profiling, so the tabs you see depend on the agent and platform behind the service.\nEvery profiling tab follows the same shape: a task list on the left, a New Task control to start a profiling run, and a result panel on the right that renders the captured data once OAP has fanned the task out to the relevant instances or processes. Results are shown as an indented stack tree or a flame graph, with a toggle between the two where both apply.\nTask creation is consistent across every tab. The New Task control opens once you have selected a service; for Network Profiling, the target instance is picked inside the dialog. Inside the dialog, a target that cannot be profiled at all — no profilable processes for eBPF, or no instances on the service — disables Create with the reason shown next to it, rather than a silently greyed-out control; advisory checks (such as Network Profiling\u0026rsquo;s process list) warn without blocking. You always see why a task cannot be started.\nAccess control Profiling is gated by two distinct permissions:\nprofile:enable — required to start a profiling task (the New Task control). It is held by the operator role and above.\nprofile:read — required to view profiling results. It is part of the read-only data catalog held by viewer, maintainer, and operator.\nA viewer can therefore open a profiling tab and inspect existing results, but cannot create new tasks. See Roles and Permissions for the full permission catalog.\nTrace Profiling Trace Profiling samples the call stacks of slow trace segments. You start a task scoped to a service (and optionally a single endpoint), and the agent dumps CPU stacks from segments that exceed the task\u0026rsquo;s threshold while the task is running.\nTo start a task, open the New Task dialog and set:\nEndpoint name — restrict sampling to one endpoint, or leave it as (any) to profile all endpoints on the service.\nStart when — begin immediately (now) or at a scheduled time.\nDuration — how long the task runs, in minutes.\nMin threshold (ms) — only segments slower than this are sampled.\nDump period (ms) — how often a stack snapshot is taken while a sampled request runs.\nMax sampling count — the cap on how many segments the task collects.\nOnce the task has collected sampled traces, pick a trace from the Sampled traces list to load its spans. Select a profiled span and press Analyze to build its call tree. The result renders as either a Tree (indented stack table) or a Flame graph. A Data mode toggle switches between Include children (the whole span\u0026rsquo;s time) and Exclude children (only the time spent in the span itself, with child-span windows subtracted). The eye icon on a task opens a detail panel with the task\u0026rsquo;s parameters and the per-instance operation log.\neBPF Profiling eBPF Profiling samples kernel-level stacks from a process without an in-process agent, driven by SkyWalking Rover. It supports two capture targets:\nON_CPU — where the process spends CPU time.\nOFF_CPU — where the process is blocked off CPU (waiting on locks, I/O, scheduling).\nA task targets a service and, optionally, a set of process labels (leave the labels empty to profile all processes). You choose the target, a start time, and a duration in minutes. Open the New Task dialog from the selected service; if OAP reports no profilable processes for it, the dialog says so and Create stays disabled.\nWhen you select a task, the result auto-analyzes. The filter bar lets you narrow the view:\nLabels — restrict the aggregation to the chosen process labels.\nAggregate — Count (number of stack samples) or Duration. Duration is only available on OFF_CPU tasks, since off-CPU samples carry a blocked-time duration that on-CPU samples do not.\nProcesses — pin specific processes from the capture; pinning re-runs the analysis immediately.\nThe result is shown as a Flame graph or a Tree, with a banner stating the wall-clock window the capture covers and how many schedules contributed.\nAsync Profiling Async Profiling runs the async-profiler against a live Java service, capturing JVM-level stacks without restarting the process. A task targets one or more service instances and one or more event types. The supported events are:\nCPU ALLOC LOCK WALL CTIMER ITIMER You can select multiple instances and multiple events in a single task, with a duration from 30 seconds up to 15 minutes. After the task runs, choose which instances to include and which event type\u0026rsquo;s tree to render, then press Analyze. Because a single task can collect several event types, the result panel has an Event type selector — switching it re-draws the flame graph for the selected JVM event (for example EXECUTION_SAMPLE for CPU/Wall/Timer events, LOCK for lock contention, or one of the object-allocation event types for ALLOC).\npprof pprof profiles a live Go service through the standard Go runtime profiler. Unlike Async Profiling, a pprof task captures exactly one event type, chosen from:\nCPU HEAP BLOCK GOROUTINE MUTEX ALLOCS THREADCREATE The dialog adapts to the event you pick:\nCPU, BLOCK, and MUTEX are time-bounded captures and require a Duration (up to 15 minutes).\nBLOCK and MUTEX additionally take a Dump period sampling rate — for BLOCK it is a blocked-nanoseconds rate, for MUTEX a contention-occurrences rate; a value of 1 samples every event. Because lower means more samples, an invalid value is rejected with the reason rather than silently replaced with a default.\nHEAP, GOROUTINE, ALLOCS, and THREADCREATE are one-shot snapshots — they take no duration and no sampling rate, capturing the current state at the moment the task fires.\nA task can target multiple Go service instances. After it runs, select the instances to include and press Analyze to render the single result tree as a flame graph.\nNetwork Profiling Network Profiling captures the network conversations between processes of a service instance and renders them as a process-level topology. It mounts on a specific instance, which you pick inside the New Task dialog. The dialog lists the rover-monitored processes that recently reported on that instance — as advice, not a gate: an instance with no recently-reported process shows a warning that the task may collect nothing, but you can still create it and let OAP decide. Once an instance is chosen, the task defines which traffic to sample.\nEach sampling rule scopes the capture — by URI pattern, by HTTP 4xx / 5xx responses, or by a minimum duration — and controls how much of each request and response body is collected. OAP runs every network task for a fixed ten minutes and the create request carries no duration, so the New Task dialog defines the sampling rules rather than a run length.\nThe result is a honeycomb topology: each cell is a process, and the edges between them are the observed inter-process calls. Selecting an edge opens a detail panel with that process-to-process relation\u0026rsquo;s metrics (call rate, latency, and bytes transferred) charted over the task\u0026rsquo;s run window. The topology that drives this layout is the same process-relation data that powers the 3D Infrastructure Map.\nContinuous Profiling Everything above starts a profiling task on demand — you pick a target and start it. Continuous profiling is the opposite: you arm a policy once, and the profiling task starts by itself whenever a process crosses a threshold, with nobody present. It is how you catch a problem that only appears at 3 a.m.\nContinuous profiling is eBPF profiling only, and it requires Rover. A policy can trigger ON_CPU, OFF_CPU or NETWORK — the same three flavours as the eBPF and Network Profiling tabs above — and the Rover agent both evaluates the thresholds and runs the resulting task. There is no continuous trace, async-profiler or pprof profiling; those stay on demand. So a service with no Rover agent can hold a saved policy, but nothing will fire until one is deployed.\nPolicies are edited on the layer\u0026rsquo;s Continuous Profiling tab, beside the eBPF and Network Profiling tabs whose tasks they trigger. The tab has its own Target service picker: each service is labelled with the targets it already has armed, and the picker can be filtered by that — including no policy, which is the set you want when arming services that are not set up yet. Opening the tab selects the first service that already has a policy, or the first service in the layer if none does. Once a service is selected, the tab shows its policy plus the instances OAP is currently evaluating it against.\nNothing here is gated on the agent already being present, because arming a policy before deploying the agent is a valid order of work: the policy is backend configuration, and it simply starts firing once an eBPF agent begins reporting. If no process of the selected service has reported eBPF-profiling support recently, the tab says so as a warning and still lets you save.\nThe tab appears on a layer whose template enables the Continuous Profiling component (Layer Setup). It ships enabled on MESH, matching where the previous SkyWalking UI placed it. Rover registers its processes into MESH, MESH_DP and K8S_SERVICE by default (which layer is configurable per discovery analyzer), so those are the layers where enabling it is likely to be useful — turn it on there if your Rover deployment reports into them.\nA policy is a set of targets — ON_CPU, OFF_CPU, or NETWORK — and each target carries one or more conditions. A condition is:\na measurement (labelled that way on screen; OAP\u0026rsquo;s own name for it is ContinuousProfilingMonitorType) — PROCESS_CPU, PROCESS_THREAD_COUNT, SYSTEM_LOAD, HTTP_ERROR_RATE, or HTTP_AVG_RESPONSE_TIME; a threshold, whose unit follows the measurement — a percentage for CPU and error rate, a thread count, a load average, milliseconds for response time. Every threshold is a whole number: OAP parses all five as integers and rejects anything else, so 0.5% or 4.5 will not save. CPU percent and HTTP error rate must be 1–100; the rest must be greater than 0. The count cannot exceed the period, and one target cannot carry two conditions of the same measurement. a period, the number of seconds of metrics to evaluate; a count, how many matching evaluations must occur before profiling is triggered. The two HTTP monitors can additionally be scoped to specific traffic. Choose All traffic, URI list or URI regex — one or the other, never both. Nothing on the backend rejects a rule carrying both, but the agent applies the list and silently ignores the regex, so the form makes the choice explicit; switching away from a filter you have filled asks before erasing it.\nTwo things are worth knowing before you save:\nSaving replaces the service\u0026rsquo;s whole policy. OAP stores one policy per service, and the page sends everything you see. A target you delete is deleted; keep every rule you want to survive.\nA policy only evaluates processes an eBPF agent reports. Inside each target sits a paged Where it runs panel: the instances and processes OAP evaluates for that target, with how often each has actually triggered profiling recently, searchable by instance or process name, each row expanding to that instance\u0026rsquo;s processes. That trigger count is the thing to read: it is the difference between a policy that is stored and one that is working, and it is the only per-target signal here (the process list itself is the same for every target). An empty panel means nothing is reporting for the service at all.\nThe panel is not a Rover presence check — it lists a process whether or not that process can be eBPF-profiled. If the panel has rows and the warning above says no process reported eBPF-profiling support, the reading is \u0026ldquo;processes are there, but none are profilable\u0026rdquo;, which points at Rover\u0026rsquo;s configuration rather than its absence.\nTasks a policy starts appear in the eBPF Profiling and Network Profiling tabs alongside the ones you start by hand, so a fired policy is read the same way as an on-demand task.\nReading policies needs profile:read; saving one needs profile:enable, the same permission as starting a task by hand — because that is what a policy eventually does.\nTroubleshooting A continuous-profiling policy never fires — first check that it is actually applied: each target shows Applied or Not applied, and rules that have only been typed are not running. Then check the Where it runs panel. If it is empty, nothing is reporting for that service and the thresholds are irrelevant; deploy Rover for the service. If processes are listed but the trigger count stays at zero, the threshold is not being crossed — lower it, lengthen the period, or reduce the required count.\nNo profiling tabs on a layer — OAP did not report profiling support for that service. Each tab requires the corresponding capability (trace, eBPF, async-profiler, network, or pprof), which depends on the agent or Rover deployment behind the service.\nNew Task is unavailable — you have not selected a service, or you lack profile:enable.\nCreate is disabled inside the New Task dialog — the chosen target cannot be profiled, and the reason is shown next to the button: for eBPF, OAP reports no profilable processes for the service; for Async Profiling, pprof, and Network Profiling, the service has no instances. On Network Profiling, an instance whose processes have not reported recently is a warning, not a block — the task can still be created.\nTask list is empty after creating a task — the task is created, but results only appear once OAP has dispatched it to the instances or processes and they report back. The view polls for the new task briefly; use the refresh control if it does not appear.\nAnalyze returns no data — the task ran but collected no samples in the selected window or scope. For Trace Profiling, confirm the threshold was low enough to sample real traffic; for eBPF and pprof, confirm the chosen processes or instances were live during the capture.\nRelated Roles and Permissions — profile:enable and profile:read.\n3D Infrastructure Map — the process and instance topology that the network view draws on.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/operate/profiling/","title":"\u003c!--"},{"body":" Service Map \u0026amp; Topology The per-layer Topology tab draws a layer\u0026rsquo;s services as an interactive, directed call graph: who calls whom, how hard each call lane is running, and how healthy each service is. It is the per-layer companion to the deployment-wide 3D Infrastructure Map — same call relationships, but flat, focused on one layer, and clickable down to the individual instance.\nAround the service map sit three related views that share the same canvas and reading conventions: the instance map drill-down (instance-to-instance traffic across a service-pair), the Deployment tab (instance-to-instance traffic inside one service), and the API dependency tab (the same graph drawn at the endpoint level). All four are driven by each layer\u0026rsquo;s dashboard template, so what they measure varies by layer — but how you read and narrow them is identical.\nWhich tabs you see These views are layer capabilities, not global pages — a layer shows only the tabs its template enables:\nTopology appears for any layer whose template declares a service map.\nDeployment appears only for layers that configure an intra-service instance graph (for example a clustered store whose nodes call each other).\nDependency (API dependency) appears only for layers that configure endpoint-level dependencies.\nA layer with none of these declared shows no topology tabs at all. The map always opens on the layer\u0026rsquo;s own service map; the instance map is reached by drilling into an edge, not from the sidebar.\nReading the service map The graph flows left to right. Entry traffic (the synthetic User node, and other callers with no upstream service) anchors the left edge; each downstream hop sits one column to the right. Within a column, the busiest services are stacked toward the top so the heavy lanes line up across columns.\nNodes Each circle is one service. Three visual channels carry its numbers, and all three come from the layer\u0026rsquo;s template — nothing is hardcoded, so the exact metric and unit differ per layer:\nThe number inside the circle is the service\u0026rsquo;s headline throughput (requests per minute for app-style layers, queries or operations per second for data layers, and so on), shown with its configured unit.\nThe colored ring around the circle is the health band. It maps a health metric (SLA, success rate, Apdex, error rate, …) onto a green → yellow → orange → red ramp. The legend under the map names the metric, prints the four break points, and states the reading direction — higher = better for SLA / success-rate / Apdex style metrics, lower = better for error-rate style metrics.\nA technology badge floats above the circle, picked from the service\u0026rsquo;s detected component (the database, cache, queue, gateway, or framework SkyWalking identified). A service whose component SkyWalking could not resolve shows a neutral badge.\nTwo node shapes are not real services: the User entry node, and conjectured peers — external or unresolved callees (an address like localhost:-1 or rcmd:80 that SkyWalking observed traffic to but has no agent on). Conjectured peers are drawn as a cloud-with-? and carry no metrics of their own; they exist on the map only to complete a call lane. Selecting one shows a virtual tag in its detail panel.\nEdges A line is a call relationship. Its thickness tracks the call rate on that lane — heavier line, more traffic. The flow animation along the line shows direction (caller → callee). Edges are not colored by health; the ring on the nodes carries that signal.\nClick a line to open its detail panel. Each line metric is shown twice — Client (as the caller measured it) and Server (as the callee measured it) — side by side, each with a sparkline over the window so you can see the trend, not just the latest number. A lane may report only one side: a call into a conjectured peer has no server-side numbers, a call out from the User node has no client-side numbers, and the panel labels those client only / server only rather than showing a blank.\nNode detail Selecting a node opens a panel with its template metrics, its Upstream list (services it calls) and Downstream list (services calling it), and two jumps: Open service (its layer dashboard) and API map → (its endpoint dependency graph). The node and edge panels are independent — you can keep both open at once.\nCross-layer hierarchy (Smartscape) One logical service is often observed by several layers at once — the same workload seen by its in-process agent (GENERAL), by its sidecar (MESH / MESH_DP), and as a Kubernetes service (K8S_SERVICE). When the service you have selected has such cross-layer counterparts, a small chip appears on the selected node\u0026rsquo;s edge; clicking it opens the hierarchy overlay.\nThe map dims but stays visible for spatial context, the selected service lights up in place with a FOCUS tag, and its counterparts in other layers fan out around it — one labeled, layer-colored lane per layer, request-near layers above the focus and infrastructure-near layers below, with counterparts in the same lane spread side by side. Each counterpart is named the way its own layer\u0026rsquo;s map would name it, and one that SkyWalking knows only from observed traffic carries a virtual tag. Auto-refresh is paused while the overlay is open, so nothing shifts under you.\nNavigation is deliberately two-step so scanning never jumps you away: click a counterpart once to select it, then click the Open in \u0026lt;layer\u0026gt; chip beside it to open that layer\u0026rsquo;s drill-down in a new browser tab with the service pre-selected. A counterpart whose layer has no active layer template in Horizon is dimmed and cannot be opened — the service exists on OAP, but there is no page to land on. Close the overlay with the ×, the Esc key, or a click on the dimmed background.\nThe chip only appears when OAP reports cross-layer counterparts for the selected service, and it is not offered in the embedded overview-widget map — open the full Topology tab.\nFocusing and narrowing the map By default the map seeds from every service in the layer — the full layer overview. That is the right starting point for a small layer and the wrong one for a large estate. Two controls narrow it:\nFocus — open the service picker (top-right of the Topology toolbar) and select one or more services. The map then redraws around just those services and their neighbors. The picker supports search and selecting a whole service group at once.\nDepth — once at least one service is focused, a depth control appears: 1 hop, 2 hops, or 3 hops. Depth is how many call hops out from the focused service the map walks. Depth has no effect on the full-layer overview (it already includes everything), so the control is hidden until you focus a service.\nAdditional controls on the canvas:\nFilter (top-left) hides nodes by layer, or hides the User node, so a busy graph reading from several layers can be thinned to the layers you care about. The filter stores what is hidden, so a service that only appears after a depth or time change starts out visible. Reset clears it.\nZoom / Fit (top-right) and drag-to-pan move the camera; double-click the canvas to fit the whole graph. Drag a node to reposition it; the layout holds your placement.\nThe map honors the topbar time picker — change the window and every node and edge metric re-reads for that range.\nInstance map (drill-down) When a service-to-service edge is selected, the edge panel offers Instance map →. This opens the instance-to-instance graph for that one service pair: the caller\u0026rsquo;s instances in the left column, the callee\u0026rsquo;s instances in the right, and the instance-level call relationships between them. It is the view for answering \u0026ldquo;which instance is the slow one\u0026rdquo; once the service map has pointed at the lane.\nThe instance map keeps two service pickers at the top so you can swap either side to an adjacent service without returning to the service map, a Service map back link, and the same client | server line-metric panel as the edge detail. A picker is shown only when there is a real choice — if a side has a single counterpart, its name is simply printed.\nDeployment (intra-service topology) The Deployment tab draws the instance-to-instance call graph within a single service — the nodes of a clustered service talking to each other (for example a distributed store\u0026rsquo;s members). It shows the full container inventory for the service, grouped by cluster or by role, with per-node metrics; call relationships are drawn as edges where SkyWalking reports them. A container that exists but has no intra-service call in the window (an idle sidecar, say) still appears on the map as an inventory node rather than being hidden.\nGrouping (by cluster, by node role / node type) comes from the layer template. When a layer reports no intra-service relations, the tab is a grouped inventory of the service\u0026rsquo;s containers with their metrics — no edges — which is the expected, by-design state for those layers, not an error.\nAPI dependency (endpoint graph) The Dependency tab is the service map drawn one level down, at the endpoint (API) level. Pick a service in the header, search its endpoints, and select one — the map then shows that endpoint\u0026rsquo;s upstream and downstream endpoint dependencies as a directed graph, with the same node metrics, edge sparklines, and pan / zoom / focus conventions as the service map.\nOne difference is inherent to the data: endpoint-relation metrics are recorded by the callee only. Edges therefore carry server-side numbers, and an endpoint with no resolvable metric values in the window is dropped from the graph rather than drawn empty.\nTwo safeguards to know The maps protect you from two failure modes that would otherwise read as \u0026ldquo;the data\u0026rdquo;.\n\u0026ldquo;Topology too large to render\u0026rdquo; A graph that grows past 5,000 services or 15,000 calls cannot be drawn legibly and risks overwhelming the browser, so the map declines to draw a partial picture. Instead it shows a notice with the actual counts and the remedy:\nTopology too large to render — N services · M calls. Pick a specific service above, or lower the depth, to see a complete map.\nThis is almost always the full-layer overview of a large estate. Focus one or a few services, and/or lower the depth, and the map renders. (Inside the embedded overview-widget snapshot, the same notice points you to open the full Topology tab to narrow the scope.)\nPartial metrics Node and edge metrics are fetched from OAP in batches. When some of those batches fail (an OAP hiccup, a backend limit), the map still draws the graph but flags that the gaps are unknown, not zero:\nSome metrics could not be loaded (X of Y batches failed) — blank values may be unavailable, not zero.\nThis matters operationally: a blank ring or an empty traffic number under this banner means \u0026ldquo;we could not read it this time\u0026rdquo;, and you should re-run before concluding a service is idle or down. On the API dependency map the same banner is phrased for its data shape — some endpoints or links may be missing, because an endpoint whose metrics failed to load is dropped rather than drawn empty. Refresh to retry.\nAccess Viewing any of these maps — service map, instance map, deployment, API dependency — requires the topology:read permission, which the built-in viewer role and above hold. See Roles and Permissions.\n","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/operate/service-map/","title":"\u003c!--"},{"body":" Trace Inspect Trace Inspect (/operate/trace-inspect) is the cross-layer trace query tool in the sidebar. The per-layer Traces tab starts from a layer and the service picked in its header; this page starts from nothing. Open it, name any service — or no service at all — and run one query across everything the trace store holds. It is built for the deep-dive that does not begin inside a dashboard: a trace id pasted from a log line or an alarm, a service you only know by name, or a \u0026ldquo;show me every error trace in the last hour, anywhere\u0026rdquo; sweep.\nLike the other triage pages, it owns its own time range — the global topbar time picker does not apply — and nothing is fetched until you press Run query. Every condition is staged: edit as much as you like, then run.\nSources The Source toggle at the top switches between the two trace stores:\nNative — SkyWalking\u0026rsquo;s own trace store, the same store the per-layer Traces tab queries. Zipkin — the Zipkin store behind OAP, with Zipkin\u0026rsquo;s own service universe and query conditions. On the per-layer tab, which store you see is decided by the layer template; here both are always one click away and you choose per query. Switching source clears the previous result so the two never mix.\nTarget — pick it, type it, or leave it blank For native traces the Target is optional: leaving it blank queries every service in the window. When you do want to scope it, there are two modes:\nPick — choose a Layer, then a Service from that layer\u0026rsquo;s catalog, then optionally narrow to one Instance and/or Endpoint. This is the discovery path: the dropdowns show you what exists. Type — enter a Service name directly, with a Real checkbox (leave it on for a normal instrumented service; turn it off for a virtual/peer service such as a database or remote endpoint that only exists as a conjectured node). Instance and Endpoint names are optional free text. Typing needs no layer at all — the name plus the Real flag identify the service. The → edit as text link converts the current Pick selection into the Type form — pick to discover, then tweak the name or flag by hand.\nThe Zipkin target is different because Zipkin has its own service universe (no layers, no SkyWalking ids): a Service field (blank means all services), plus Remote service and Span name narrowing fields whose suggestions load once a service is picked.\nConditions Native conditions:\nCondition What it does Trace ID Paste a known trace id for a direct lookup. Status ALL, SUCCESS, or ERROR. Order Newest (by start time) or Slowest (by duration). Duration (ms) Min–max trace duration bounds, in milliseconds. Tags Comma-separated key=value pairs, AND-joined, with autocomplete (below). Time A rolling preset (15m, 30m, 1h, 3h, 6h, 12h, 24h) or Custom…, which swaps in an absolute start/end pair. The × returns to presets. Limit Result cap: 20, 30 (default), 50, or 100. Zipkin conditions are the store\u0026rsquo;s own: Duration (ms) bounds, an Annotation query (error or key=value terms, AND-joined), plus the shared Time and Limit. There is no Trace ID field on the Zipkin side.\nThe Tags field autocompletes from the tags actually stored in the window: start typing to see known keys, type = to switch the suggestions to that key\u0026rsquo;s known values, and press Enter to commit the pair — the field then primes a comma so you can keep typing the next one. Time windows are evaluated at second precision, same as the per-layer tab, so a trace that just finished still falls inside the window.\nRun query and the resolved query Run query executes the staged conditions and replaces the result area. Next to it, a Resolved query toggle appears after each run: it names the source (and, for native, which trace query API answered) and expands to the exact condition that was sent — the service ids resolved from your picks or typed names, the computed window, and every filled-in default. When a query returns something unexpected, read this panel first: it shows what was actually asked, not what you meant.\nDistribution chart Beside the conditions, a Distribution chart plots one dot per result — start time on the X axis, colored by success/error, with the duration surfaced on hover. Click a dot to open that trace, or drag a rectangle to brush a subset: the list below narrows to the brushed traces and shows an N / total count with a clear control. Brushing filters what is already loaded; it does not re-query.\nResults — segments or whole traces For native traces, a banner above the results states which trace query API this OAP serves: on backends with whole-trace support (Trace Query v2) full traces come back inline; on any other backend (Trace Query v1) each row is a trace segment and clicking one fetches its full trace. This is a property of the storage backend, not a setting — see Traces for the full explanation.\nClicking a row opens the same trace detail the per-layer tab uses: the span waterfall with its Default / Tree / Statistics layouts, per-span detail (meta, tags, logs, cross-trace refs, attached events), and the id / url copy buttons — a copied shareable URL reopens the trace in an overlay for whoever you send it to, on either store. While a trace is open, the result list folds into a collapsible rail on the left so you can step through traces without losing the query. Escape closes the span panel first, then the trace, in that order. Zipkin results render with the Zipkin waterfall and keep their Zipkin span shape.\nHow it differs from the per-layer Traces tab No layer, no header picker. The target is part of the query form, optional, and can be a typed name — including services in layers you never open, or all services at once. Both stores on one page. Native vs. Zipkin is a per-query toggle here; on the layer tab it is fixed by the layer template. Built for id-first triage. Paste a trace id with no service at all and run — the common \u0026ldquo;a log/alarm gave me an id\u0026rdquo; entry point. The waterfall, the distribution chart, the staged Run-query flow, and the v1/v2 behavior are identical to the per-layer tab — this page changes how you scope the query, not how results render.\nPermissions The page and its queries require the inspect:read permission. Opening a trace\u0026rsquo;s waterfall and the tag / Zipkin suggestion lists additionally use traces:read, and the Pick-mode layer/service dropdowns use metrics:read — the bundled roles that grant inspect:read include these. See Roles and Permissions.\nRelated Traces — the per-layer trace explorer, with the full waterfall and condition reference. Log Inspect — the cross-layer sibling for logs, browser errors, and pod tails. Metrics Inspect — the cross-layer view of OAP\u0026rsquo;s metric catalog. ","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/operate/trace-inspect/","title":"\u003c!--"},{"body":" Traces The Traces tab is the distributed-trace explorer inside a layer. You pick a service, set conditions (status, sort, duration, tags, time window), run the query, then click a result to read its span timeline. It surfaces two trace stores — SkyWalking-native traces and Zipkin traces — depending on what the layer is configured for.\nTraces are triage data, so this tab owns its own time range and conditions. It is not driven by the global topbar time picker, and it does not auto-refresh: you set your conditions and press Run query. Nothing is fetched until you do — until then the list shows a \u0026ldquo;Pick your conditions, then click Run query.\u0026rdquo; prompt.\nWhich trace store appears A layer template carries a traces.source setting that decides which trace store the tab queries:\nnative (the default when a layer has no traces block) — only the SkyWalking-native trace explorer. zipkin — only the Zipkin trace explorer. both — two separate sidebar tabs, Trace (native) and Zipkin Trace. Native and Zipkin spans have different shapes and different query conditions, so they are kept as distinct tabs rather than one tab with a toggle. Mesh and Kubernetes-flavored layers commonly land on Zipkin; instrumented-agent layers land on native.\nNative traces The native explorer queries SkyWalking\u0026rsquo;s own trace store. The service is taken from the layer\u0026rsquo;s Service header picker at the top of the page; the in-tab conditions narrow within that service.\nConditions All conditions are staged in the toolbar and only take effect on Run query — editing a field does not refetch on its own.\nCondition What it does Instance Restrict to one service instance. Defaults to All. Resets when you switch service. Endpoint Restrict to one endpoint. Defaults to All. A dropdown of the service\u0026rsquo;s endpoints (capped at 50). Status ALL, SUCCESS, or ERROR — the trace state. Order BY_START_TIME (Newest) or BY_DURATION (Slowest). Limit Cap on result rows: 30 by default. The server caps a single page at 200. The list says when the window held more traces than the limit returned — there is no total on the wire, only \u0026ldquo;there is more\u0026rdquo;. Time range A rolling preset (Last 15 min through Last 24 hours) or a Custom… absolute start/end pair. Trace ID Paste a known trace id to look it up directly. Duration range (ms) Min–max trace duration, in milliseconds. Tag Free-form span tags as key=value (for example http.status_code=500). Press Enter to add; each committed tag shows as an Active-tag chip. Multiple tags are AND-joined. The time window is evaluated at second precision so a trace that just finished still falls inside it — minute rounding would drop the most recent (and usually most interesting) traces during triage.\nRicher vs. universal results, by storage backend What a result row represents depends on the storage backend behind OAP, and Horizon detects this automatically — you do not configure it:\nOn backends that support it, the explorer fetches whole traces with their spans inline. The list shows complete traces, and selecting one renders its waterfall immediately with no second round-trip. A banner reads \u0026ldquo;This OAP serves traces via Trace Query v2 API\u0026rdquo; and \u0026ldquo;Full traces are returned inline.\u0026rdquo; On any other backend, the explorer falls back to the universal basic query, which returns trace segments. Each row is one segment; the full trace is fetched on click. The banner reads \u0026ldquo;Trace Query v1 API\u0026rdquo; and \u0026ldquo;Each row is a trace segment — click one to fetch its full trace.\u0026rdquo; The banner stays visible across both the browse list and the open-trace view, so it is always clear what a row represents. The richer inline view is a property of the storage backend, not a setting — if your rows are segments, the backend does not support whole-trace queries.\nDuration distribution Beside the conditions, a Distribution chart plots one dot per result: the X axis is the trace\u0026rsquo;s start time, and the dot\u0026rsquo;s duration (the Y value) is surfaced on hover. Error traces are drawn in the error color, successful ones in the accent color.\nThe chart is an in-page filter. Click a dot — or drag a rectangle across several — to pick a subset; the result list then narrows to just the picked traces and the header switches to an \u0026ldquo;N picked\u0026rdquo; count with a Reset button. This filters what is already loaded; it does not issue a new query.\nResult list and the trace waterfall Each row in the result list shows the trace\u0026rsquo;s root endpoint, an OK/ERR status flag, the duration, and a bar sized relative to the slowest trace in the set. Click a row to open it.\nSelecting a trace opens the detail view, which offers three layouts:\nDefault — the span waterfall: an indented timeline, one row per span. Each row carries a service-colored bar positioned and sized by the span\u0026rsquo;s start offset and duration, a span-kind glyph, a component icon, the endpoint or peer name, and the span\u0026rsquo;s own duration. Errored spans are highlighted. A flag badge marks spans that carry attached events. Tree — the same spans drawn as a zoomable node graph. Statistics — spans rolled up by name, with count and total / average / maximum duration, sortable per column. Span kinds are grouped into entry (server), exit (client), local, producer, and consumer families, each with its own glyph and color. The waterfall stitches spans across segments using their parent references, so a single trace that spans multiple services renders as one connected timeline.\nClick any span row to open its detail panel:\nMeta — service, instance, endpoint, kind, component, peer, layer, start time, duration, and error flag. Cross-trace refs — when a span references a parent in a different trace, those references are listed with the parent trace id, parent segment, parent span, and ref type. The trace id is a link that opens that other trace. Tags — the span\u0026rsquo;s key/value tags. Logs — per-span log entries with their timestamps. Attached Events — named events on the span with their start/end times and summary key/values. The detail view\u0026rsquo;s header KPIs report the trace\u0026rsquo;s start time, total duration, span count, and the number of distinct services it touched. You can copy the trace id or a shareable URL from there; opening a shared ?traceId= link lands directly on the trace in an overlay.\nZipkin traces When a layer enables Zipkin, the Zipkin tab queries an upstream Zipkin store through OAP. Zipkin organizes data by its own service universe (the localEndpoint.serviceName reported on each span), which can drift from SkyWalking\u0026rsquo;s service list, so this tab carries its own service controls rather than binding to the shell\u0026rsquo;s Service picker.\nConditions Condition What it does Service Free-text service name (with suggestions). Empty means every service. Remote service Narrow to spans calling a given remote service. Requires a service to be picked first. Span name Narrow to one span/operation name. Requires a service to be picked first. Min duration (ms) / Max duration (ms) Duration bounds, entered in milliseconds. Annotations Zipkin annotation query — error or key=value terms, AND-joined. Open trace ID Paste a trace id to open it directly. Limit Result cap: 10, 30, 50, 100, or 200. The list says when the window held more traces than the limit returned. Time range A lookback preset (Last 15 min through Last 24 hours) or a Custom range… absolute window. As with the native tab, conditions are staged and only applied on Run query.\nEach Zipkin result shows its duration and error state, with a duration bar colored fast-to-slow (errored traces are forced to the error color). Selecting a trace renders the Zipkin span waterfall, and a span detail panel exposes the span\u0026rsquo;s duration, kind, and Zipkin tags. Because the two stores have different span formats, there is no field mapping between native and Zipkin results — Zipkin spans keep their Zipkin shape.\nTroubleshooting \u0026ldquo;No traces in window.\u0026rdquo; — the query ran but matched nothing. Widen the time range, relax the Status / Duration / Tag conditions, or confirm the service is actually reporting traces. An unreachable chip on the list — the trace store did not answer, and the reason is printed in a banner above the results. For native traces this points at OAP or its storage backend; for Zipkin it points at the configured Zipkin endpoint. The two stores fail independently — one being down does not blank the other. Run query is greyed out — the tab does not yet know which service to read. It says which: Resolving service… while the picked service is being looked up, or a note that the selected service is not in this layer (it aged out of OAP, was renamed, or the link points elsewhere) — pick another one. Traces are always read for one service, so the tab waits instead of querying the whole layer. Rows are segments, not whole traces — that is expected on storage backends without whole-trace support; the banner says so. Click a segment to fetch its full trace. A pasted trace id from a log row won\u0026rsquo;t resolve — older traces can sit outside the default lookup window or in a cold storage tier. Open the trace from the log row (which carries its timestamp) rather than pasting the id cold, so the lookup is widened around the right time. No data even with a valid service — double-check the time range first; this tab does not follow the global topbar, so the window is whatever the tab\u0026rsquo;s own Time range control says. Related Trace Inspect — the cross-layer trace query tool: look up a trace by id or query any service (picked, typed by name, or all of them) without entering a layer. 3D Infrastructure Map — topology-level view of the same services these traces flow through. Metrics Inspect — confirm which metrics a service is reporting when traces look incomplete. Layer Dashboard Templates — where a layer\u0026rsquo;s traces.source is configured. ","excerpt":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license …","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/operate/traces/","title":"\u003c!--"},{"body":"0.1.0 Foundational scaffolding. The shell renders, auth works, OAP is reachable, and CI is green. No operator-facing data surfaces yet.\npnpm monorepo: apps/ui (Vue 3 + Vite), apps/bff (Fastify), shared packages/api-client (typed REST + GraphQL clients), shared packages/design-tokens (CSS custom properties). BFF — Fastify skeleton with horizon.yaml config + hot reload, local auth (argon2 + cookie sessions), RBAC verb gating + JSONL audit log, OAP proxy with cluster fan-out + preflight. UI — AppShell (sidebar, topbar) with design tokens, Pinia auth store with on-401 redirect, login view with route guard + sign-out, stub admin / operate pages. CI — monorepo workspace build + dependency license check via skywalking-eyes. ","excerpt":"\u003ch1 id=\"010\"\u003e0.1.0\u003c/h1\u003e\n\u003cp\u003eFoundational scaffolding. The shell renders, auth works, OAP is reachable, and CI is green. No …\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/changelog/0.1.0/","title":"0.1.0"},{"body":"0.1.0 Foundational scaffolding. The shell renders, auth works, OAP is reachable, and CI is green. No operator-facing data surfaces yet.\npnpm monorepo: apps/ui (Vue 3 + Vite), apps/bff (Fastify), shared packages/api-client (typed REST + GraphQL clients), shared packages/design-tokens (CSS custom properties). BFF — Fastify skeleton with horizon.yaml config + hot reload, local auth (argon2 + cookie sessions), RBAC verb gating + JSONL audit log, OAP proxy with cluster fan-out + preflight. UI — AppShell (sidebar, topbar) with design tokens, Pinia auth store with on-401 redirect, login view with route guard + sign-out, stub admin / operate pages. CI — monorepo workspace build + dependency license check via skywalking-eyes. ","excerpt":"\u003ch1 id=\"010\"\u003e0.1.0\u003c/h1\u003e\n\u003cp\u003eFoundational scaffolding. The shell renders, auth works, OAP is reachable, and CI is green. No …\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/changelog/0.1.0/","title":"0.1.0"},{"body":"0.1.0 Foundational scaffolding. The shell renders, auth works, OAP is reachable, and CI is green. No operator-facing data surfaces yet.\npnpm monorepo: apps/ui (Vue 3 + Vite), apps/bff (Fastify), shared packages/api-client (typed REST + GraphQL clients), shared packages/design-tokens (CSS custom properties). BFF — Fastify skeleton with horizon.yaml config + hot reload, local auth (argon2 + cookie sessions), RBAC verb gating + JSONL audit log, OAP proxy with cluster fan-out + preflight. UI — AppShell (sidebar, topbar) with design tokens, Pinia auth store with on-401 redirect, login view with route guard + sign-out, stub admin / operate pages. CI — monorepo workspace build + dependency license check via skywalking-eyes. ","excerpt":"\u003ch1 id=\"010\"\u003e0.1.0\u003c/h1\u003e\n\u003cp\u003eFoundational scaffolding. The shell renders, auth works, OAP is reachable, and CI is green. No …\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/changelog/0.1.0/","title":"0.1.0"},{"body":"0.1.0 Features Add OAPServer CRDs and controller. Chores Set up GitHub actions to build from sources, check code styles, licenses. ","excerpt":"\u003ch2 id=\"010\"\u003e0.1.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd OAPServer CRDs and controller.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"chores\"\u003eChores\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSet up GitHub actions to build from …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/latest/en/changes/changes-0.1.0/","title":"0.1.0"},{"body":"0.1.0 Features Add OAPServer CRDs and controller. Chores Set up GitHub actions to build from sources, check code styles, licenses. ","excerpt":"\u003ch2 id=\"010\"\u003e0.1.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd OAPServer CRDs and controller.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"chores\"\u003eChores\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSet up GitHub actions to build from …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/next/en/changes/changes-0.1.0/","title":"0.1.0"},{"body":"0.1.0 Features Add OAPServer CRDs and controller. Chores Set up GitHub actions to build from sources, check code styles, licenses. ","excerpt":"\u003ch2 id=\"010\"\u003e0.1.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd OAPServer CRDs and controller.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"chores\"\u003eChores\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSet up GitHub actions to build from …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/v0.11.0/en/changes/changes-0.1.0/","title":"0.1.0"},{"body":"0.10.0 Features Support the Horizon UI in the UI CRD through a spec.kind discriminator. Publish Docker images to ghcr.io on every push to master. Bugs Fix the LAL rule missing the layer property, for compatibility with the latest OAP. Replace the deprecated scheme.Builder and update Go to 1.26. Chores Bump Go to 1.26.3 in the adapter and 1.25.9 in the operator, to fix stdlib CVEs. Bump golang.org/x/net to v0.53.0 to fix CVE-2026-33814. Bump go.opentelemetry.io/otel/sdk to v1.43.0, sigs.k8s.io/controller-runtime and other dependencies. ","excerpt":"\u003ch2 id=\"0100\"\u003e0.10.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport the Horizon UI in the UI CRD through a \u003ccode\u003espec.kind\u003c/code\u003e discriminator.\u003c/li\u003e\n\u003cli\u003ePublish …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/latest/en/changes/changes-0.10.0/","title":"0.10.0"},{"body":"0.10.0 Features Support the Horizon UI in the UI CRD through a spec.kind discriminator. Publish Docker images to ghcr.io on every push to master. Bugs Fix the LAL rule missing the layer property, for compatibility with the latest OAP. Replace the deprecated scheme.Builder and update Go to 1.26. Chores Bump Go to 1.26.3 in the adapter and 1.25.9 in the operator, to fix stdlib CVEs. Bump golang.org/x/net to v0.53.0 to fix CVE-2026-33814. Bump go.opentelemetry.io/otel/sdk to v1.43.0, sigs.k8s.io/controller-runtime and other dependencies. ","excerpt":"\u003ch2 id=\"0100\"\u003e0.10.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport the Horizon UI in the UI CRD through a \u003ccode\u003espec.kind\u003c/code\u003e discriminator.\u003c/li\u003e\n\u003cli\u003ePublish …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/next/en/changes/changes-0.10.0/","title":"0.10.0"},{"body":"0.10.0 Features Support the Horizon UI in the UI CRD through a spec.kind discriminator. Publish Docker images to ghcr.io on every push to master. Bugs Fix the LAL rule missing the layer property, for compatibility with the latest OAP. Replace the deprecated scheme.Builder and update Go to 1.26. Chores Bump Go to 1.26.3 in the adapter and 1.25.9 in the operator, to fix stdlib CVEs. Bump golang.org/x/net to v0.53.0 to fix CVE-2026-33814. Bump go.opentelemetry.io/otel/sdk to v1.43.0, sigs.k8s.io/controller-runtime and other dependencies. ","excerpt":"\u003ch2 id=\"0100\"\u003e0.10.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport the Horizon UI in the UI CRD through a \u003ccode\u003espec.kind\u003c/code\u003e discriminator.\u003c/li\u003e\n\u003cli\u003ePublish …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/v0.11.0/en/changes/changes-0.10.0/","title":"0.10.0"},{"body":"0.10.0 Design Documents KTM FODC proxy FODC watchdog and flight recorder Context-aware panic diagnostics ","excerpt":"\u003ch1 id=\"0100-design-documents\"\u003e0.10.0 Design Documents\u003c/h1\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"../ktm\"\u003eKTM\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"../fodc/proxy\"\u003eFODC proxy\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"../fodc/watchdog-and-flight-recoder\"\u003eFODC watchdog and flight recorder\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"../fodc/context-aware-panic-diagnostics\"\u003eContext-aware panic …\u003c/a\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-banyandb/next/design/archive/0.10.0/readme/","title":"0.10.0 Design Documents"},{"body":"0.11.0 Bugs Add tools/releasing/preflight.sh, which checks everything a release needs before release.sh does anything irreversible: the tools, the signing key and whether it is in the published KEYS file, gh authentication, the dist URLs, that the version agrees between Chart.yaml and the changelog and that its tag is free, that no abandoned candidate is sitting in dist/dev, and that the tree is clean with the generated chart files in sync. release.sh runs it as its first step, before asking about the signing key, and takes the resolved key from it rather than detecting one itself. It reports every problem rather than stopping at the first, and is explicit about the three things it cannot check from a developer machine. Let tools/releasing/release.sh run when the tree already carries the release version. It committed the version bump with a plain git commit, which exits non-zero with nothing staged, and the script runs under set -e \u0026ndash; so a release cut the documented way, with Chart.yaml already updated and the kustomize image tags already set by the same function, died at \u0026ldquo;nothing to commit, working tree clean\u0026rdquo; before tagging or building anything. Both release commits allow an empty diff now, which also keeps the regeneration --amend pointed at the script\u0026rsquo;s own commit rather than at whatever master happened to be. Build genuinely multi-architecture images, again. The previous fix declared ARG TARGETARCH=amd64 in each Dockerfile, and giving a predefined platform argument a default makes BuildKit use that default instead of the target\u0026rsquo;s architecture \u0026ndash; so TARGETARCH read amd64 even when building for linux/arm64, the builder stage ran once, and the amd64 binary was copied into the arm64 manifest. The publish workflow\u0026rsquo;s own ELF check caught it on the first run that was ever able to start. The argument is declared with no default now, and the shell falls back to the native architecture for a plain docker build, which was also producing amd64 binaries on an arm64 host. Features Release the skywalking-swck Helm chart from this repository. One chart installs the operator and, behind a values flag, the custom metrics adapter. The CRDs, the operator\u0026rsquo;s ClusterRole and the admission webhook configurations it ships are generated from the operator sources by make chart-manifests, and CI fails on any drift. Bugs Stop reconciling a UI whose kind is no longer supported. Narrowing the CRD enum to horizon only rejects new resources \u0026ndash; schema validation runs on admission, never on read \u0026ndash; so a UI stored as kind: booster by an earlier operator survives the upgrade and still reconciles. With the templates now unconditionally Horizon\u0026rsquo;s, reconciling one rewrote a running Booster Deployment into a shape its image cannot serve and took the UI down on the first pass after upgrade. Such a resource is now left untouched, with a UnsupportedKind event saying what to do. Stop applying an OAPServer Deployment when the Storage it names cannot be read. Every lookup error was logged and ignored, and the Deployment was applied anyway \u0026ndash; without SW_STORAGE, targets, credentials or TLS volumes \u0026ndash; so a Storage briefly deleted and recreated replaced a working OAP with one that never becomes ready. The reconcile now leaves the running Deployment alone, emits a StorageUnresolved event and requeues. Keep the storage TLS volume when an OAPServerConfig mounts static files. The overlay assigned over the pod\u0026rsquo;s volume and mount lists, and ApplyOverlay is an RFC 7386 merge patch under which an array replaces rather than merges \u0026ndash; so the certificate volume disappeared and SW_STORAGE_BANYANDB_SSL_TRUST_CA_PATH pointed at nothing. The lists are merged by name now, and the mount is reconciled on every pass rather than skipped whenever the file content is unchanged, which had left it lost for good after any re-render. Roll the OAP when its credential Secret is rotated. Environment variables taken from a Secret are resolved once, when the container starts, so moving credentials to secretKeyRef meant a rotation went unnoticed until something restarted the pod. The controller now watches Secrets and carries the referenced Secret\u0026rsquo;s resourceVersion in a pod-template annotation \u0026ndash; an opaque token, not a digest of the credential. Create certificate signing requests through certificates.k8s.io/v1. The v1beta1 API this used was removed in Kubernetes 1.22, so internal Elasticsearch TLS could not obtain a certificate on any cluster newer than that and the workload waited on a Secret nothing produced. The wait loop is also bounded now, and sleeps \u0026ndash; it used to spin on Get with no delay and no limit. Require SW_STORAGE to carry a value. The mandatory-storage check accepted an entry named SW_STORAGE with nothing behind it, which reaches the OAP as an empty selector and produces exactly the never-ready state the check exists to prevent. Require a published GitHub release before publishing convenience binaries. Both the publish workflow and release-passed.sh tested only whether gh release view succeeded, and that resolves drafts \u0026ndash; so a dispatch against a draft could put official version-tagged images on GHCR before the vote, and release-passed.sh mistook a draft for a finished release and never published it. Both now require isDraft: false and a publication time; the workflow rejects prereleases too. Stop truncating rendered manifests at the first #. Every manifest was cut line-by-line at its first hash with no awareness of YAML quoting, so any value containing one \u0026ndash; a password, an AI prompt, a URL fragment \u0026ndash; was severed mid-string and the resulting manifest no longer parsed. Only whole-line comments are dropped now. This became reachable for user-supplied values with spec.env. Deep-copy the new env and envFrom fields. zz_generated.deepcopy.go had not been regenerated, so those slices were shared with the objects controller-runtime\u0026rsquo;s cache hands out. Render an OAPServer whose Storage cannot be read yet. spec.storage.name is now mandatory, but the operator fills in the resolved Storage only once it can read it, and the deployment template reached through the nil \u0026ndash; so an OAPServer applied before its Storage failed to render at all rather than waiting for it. Reference the Elasticsearch credentials from the Storage controller too. The OAPServer side stopped copying them out of the Secret; the Storage controller still did, putting the password into the resource and the Elasticsearch StatefulSet it renders. Document BanyanDB storage: the endpoint format and its gRPC port, cluster targets, authentication, persistence, and the flags BanyanDB 0.11 renamed. See docs/en/setup/banyandb.md. Pass BanyanDB credentials from a Storage\u0026rsquo;s security.user.secretName, as the Elasticsearch path already did. Raise the OAP startup probe budget from 110 seconds to 10 minutes. SkyWalking 11 has no embedded storage, so every start installs a schema into BanyanDB or Elasticsearch \u0026ndash; work that overran the old probe on a cold cluster, and being killed mid-schema turned a slow first boot into a crash loop. Wire BanyanDB storage. The operator could only configure Elasticsearch, so an OAPServer on BanyanDB had to carry the storage environment by hand \u0026ndash; and the variable it needed changed name between SkyWalking 9.x and 11.x. A Storage of type: banyandb now yields SW_STORAGE=banyandb and SW_STORAGE_BANYANDB_TARGETS. Without this, SWCK cannot deploy a working OAP at all across the supported range: SkyWalking removed H2 permanently in 10.2.0, so there is no fallback and an OAPServer with no storage never becomes ready. Stop deriving the Horizon admin and Zipkin URLs. The OAP admin host arrived in 11.x, and on 10.x port 17128 is the AI-pipeline URI-recognition server, so a derived oap.adminUrl pointed Horizon at a live endpoint that was the wrong service; the OAPServer this operator deploys exposes no Zipkin port at all. Both are now emitted only when spec.OAPServerAdminAddress / spec.OAPServerZipkinAddress are set, matching what skywalking-helm does. Fix the default image for kind: horizon UIs. It was apache/skywalking-horizon-ui:\u0026lt;version\u0026gt;, a Docker Hub repository that does not exist \u0026ndash; Horizon releases share apache/skywalking-ui with the legacy Booster UI and are told apart by a horizon- tag prefix. Since horizon is the default kind, every UI created without an explicit image could never pull. Covered by a unit test; the samples and docs carried the same wrong name. Build genuinely multi-architecture images. operator/Dockerfile, adapter/Dockerfile and build/images/Dockerfile.release hardcoded GOARCH=amd64 and -linux-amd64 while the publish workflow advertised linux/arm64, so apache/skywalking-swck:0.10.0 shipped an arm64 manifest holding x86-64 binaries and an arm64 node got exec format error. The release now builds a binary per architecture, and the publish workflow pulls every advertised platform back and checks the ELF machine type before the release completes. Ship the eventexporter admission webhook in the chart, and drop the duplicate meventexporter.kb.io entry that the API server rejects. Both come out of generating the webhook configurations from the operator sources rather than hand-copying them. Chores Seed the e2e login from a Secret through UI.spec.envFrom, so the new envFrom surface is exercised against a real cluster rather than only unit-tested: if it did not reach the container there would be no user and oap-ui-agent\u0026rsquo;s login would fail. Split the e2e suite by configuration path. Almost every case now runs the Horizon UI on HORIZON_* environment variables with no ConfigMap at all, and oap-ui-agent proves they arrive by logging in and reading the OAP hosts back through the BFF. Exactly one case, oap-ui-agent-oapserverconfig-oapserverdynamicconfig, takes the override path: the UI carries a whole horizon.yaml in spec.config while its OAPServerConfig overlays a static file on the OAP. Configure the Horizon UI with environment variables instead of a generated file. Horizon\u0026rsquo;s image bakes a fully tokenised horizon.yaml, so every setting it has is reachable as HORIZON_* \u0026ndash; but the operator mounted its own file over that one, which replaced every token and left the container with no env: at all, so no variable could reach any setting. UI.spec.env and UI.spec.envFrom now carry them, the operator sets only what it derives, and the ConfigMap is mounted only when spec.config supplies a whole file. A setting added in a future Horizon release works without an SWCK release. Add envFrom to OAPServer and Satellite, which are configured entirely through environment variables and previously had no way to take one from a Secret. Reference storage credentials instead of copying them. The operator read the Storage\u0026rsquo;s user secret and wrote the username and password in as literal env values, so they appeared in both the OAPServer and its Deployment for anyone with read access. They are now secretKeyRefs resolved by the kubelet. This also needed a template fix: the OAP deployment rendered only name/value and silently dropped valueFrom, so a secret reference could not have worked at all. Wire BanyanDB TLS. security.tls with security.tlsSecretName mounts the CA at /skywalking/bydb-tls and sets SW_STORAGE_BANYANDB_SSL_TRUST_CA_PATH. Previously tls: true on a banyandb Storage was accepted, wired no TLS, and mounted the Elasticsearch keystore secret skywalking-storage, leaving the OAP pod waiting on a secret nothing creates; tlsSecretName is now required for that combination. Add UI.spec.templatesMode, emitted as HORIZON_TEMPLATES_MODE. Left unset it follows the admin address: live reads OAP\u0026rsquo;s template store over the OAP admin host, so it is chosen only when spec.OAPServerAdminAddress is set, and readonly \u0026ndash; which renders the templates bundled in the image \u0026ndash; otherwise. Defaulting to live regardless would leave every UI probing 127.0.0.1:17128, failing Horizon\u0026rsquo;s ui-management preflight and blocking every layer-driven page, Traces most visibly. OAP 10.x needs readonly in any case: it manages templates over legacy query-port GraphQL and Horizon speaks only OAP 11\u0026rsquo;s REST protocol. Stop maintaining a copy of Horizon\u0026rsquo;s configuration schema. The config the operator used to generate restated Horizon\u0026rsquo;s own defaults, and the copy had drifted: viewer was granted 6 of the 12 permissions Horizon gives that role, and the admin landing route was /admin/cluster, which Horizon has no route for \u0026ndash; signing in as admin landed on \u0026ldquo;No route matches\u0026rdquo;. It also carried audit.file, setup and alarms, keys Horizon 1.0.0\u0026rsquo;s schema does not have and whose presence stops the BFF booting at all. Refuse an OAPServer that has no storage. SkyWalking removed the embedded H2 permanently in 10.2.0, so no version this operator supports has one and there is nothing to fall back to: an OAPServer with nowhere to write starts, dials a BanyanDB on 127.0.0.1:17912 and never becomes ready. The webhook now says so at admission instead. Setting SW_STORAGE directly in spec.config still counts as having chosen a storage. Breaking: an OAPServer with neither spec.storage.name nor SW_STORAGE is rejected \u0026ndash; it could never have worked. Publish the operator image, the metrics adapter image and the Helm chart from the release publish workflow, triggered by publishing a GitHub release, instead of pushing tags by hand. Pushes to master keep publishing SHA-tagged snapshots to GHCR. Add tools/releasing/release.sh and tools/releasing/release-passed.sh, automating the release either side of the vote. Ship the Helm chart tarball as a signed, voted artifact on dist.apache.org, alongside the source and binary tarballs. Support OAP 10.4.0 and later, with 11.0.0 recommended, matching skywalking-helm. An OAPServer below that is admitted with a warning rather than rejected. Deploy only the Horizon UI. spec.kind on the UI resource now accepts horizon alone \u0026ndash; apache/skywalking removed the legacy Booster UI in 11.0.0 and no longer builds an image for it. Breaking: a UI with kind: booster is rejected, with a message saying what to use instead. Default OAPServerConfig and OAPServerDynamicConfig to version 11.0.0, was 9.5.0. These match an OAPServer by exact version string, so a config that omits version previously only attached to an OAP explicitly pinned at 9.5.0. Behaviour change: set spec.version explicitly if you run an older OAP. Verify the Horizon UI over its own API rather than a GraphQL proxy it does not have: every UI case asserts the auth backend at the public /api/auth/health, and oap-ui-agent logs in with a seeded user and calls an RBAC-protected route. Document that Horizon ships with no users, so a UI refuses every login until one is seeded through HORIZON_AUTH_LOCAL_USERS. Move the e2e suite to the current SkyWalking stack: OAP 11.0.0, Horizon UI 1.0.0 and BanyanDB 0.11.0, which have to move together because OAP 11.0.0 accepts BanyanDB server API 0.11 only and Horizon 1.0.0\u0026rsquo;s admin host is an OAP 11 addition. The UI cases previously deployed the legacy Booster UI, which apache/skywalking no longer builds an image for. Pin every image the e2e suite deploys in test/e2e/env, substituted with envsubst, replacing apache/skywalking-banyandb:latest and centralising nine images that were spread across fifteen manifests. Install the operator with the Helm chart in ten of the twelve e2e cases, and add a case that tests the chart\u0026rsquo;s own lifecycle: installing the packaged tarball, CRDs, webhook CA injection, agent injection, HPA metrics, upgrade in both directions, and uninstall without taking the CRDs with it. test/e2e/oap-ui-agent stays on kustomize so that install path remains covered. Split hack/ by purpose: developer and build tooling moved to tools/, test tooling to test/tools/. Restructure the documentation into docs/en/{concepts-and-designs,setup,examples,guides,changes}, following the layout of apache/skywalking, and move the changelog from CHANGES.md into docs/en/changes/. Bump go.opentelemetry.io/otel to v1.44.0 to fix CVE-2026-41178. Bump golang.org/x/net to v0.55.0, golang.org/x/crypto to v0.53.0 and golang.org/x/sys to v0.46.0. Bump software.sslmate.com/src/go-pkcs12, github.com/sirupsen/logrus, github.com/go-logr/logr, google.golang.org/grpc, golang.org/x/text and the Kubernetes dependencies. Bump the actions-deps group across the repository. ","excerpt":"\u003ch2 id=\"0110\"\u003e0.11.0\u003c/h2\u003e\n\u003ch4 id=\"bugs\"\u003eBugs\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd \u003ccode\u003etools/releasing/preflight.sh\u003c/code\u003e, which checks everything a release needs before …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/latest/en/changes/changes/","title":"0.11.0"},{"body":"0.11.0 Bugs Add tools/releasing/preflight.sh, which checks everything a release needs before release.sh does anything irreversible: the tools, the signing key and whether it is in the published KEYS file, gh authentication, the dist URLs, that the version agrees between Chart.yaml and the changelog and that its tag is free, that no abandoned candidate is sitting in dist/dev, and that the tree is clean with the generated chart files in sync. release.sh runs it as its first step, before asking about the signing key, and takes the resolved key from it rather than detecting one itself. It reports every problem rather than stopping at the first, and is explicit about the three things it cannot check from a developer machine. Let tools/releasing/release.sh run when the tree already carries the release version. It committed the version bump with a plain git commit, which exits non-zero with nothing staged, and the script runs under set -e \u0026ndash; so a release cut the documented way, with Chart.yaml already updated and the kustomize image tags already set by the same function, died at \u0026ldquo;nothing to commit, working tree clean\u0026rdquo; before tagging or building anything. Both release commits allow an empty diff now, which also keeps the regeneration --amend pointed at the script\u0026rsquo;s own commit rather than at whatever master happened to be. Build genuinely multi-architecture images, again. The previous fix declared ARG TARGETARCH=amd64 in each Dockerfile, and giving a predefined platform argument a default makes BuildKit use that default instead of the target\u0026rsquo;s architecture \u0026ndash; so TARGETARCH read amd64 even when building for linux/arm64, the builder stage ran once, and the amd64 binary was copied into the arm64 manifest. The publish workflow\u0026rsquo;s own ELF check caught it on the first run that was ever able to start. The argument is declared with no default now, and the shell falls back to the native architecture for a plain docker build, which was also producing amd64 binaries on an arm64 host. Features Release the skywalking-swck Helm chart from this repository. One chart installs the operator and, behind a values flag, the custom metrics adapter. The CRDs, the operator\u0026rsquo;s ClusterRole and the admission webhook configurations it ships are generated from the operator sources by make chart-manifests, and CI fails on any drift. Bugs Stop reconciling a UI whose kind is no longer supported. Narrowing the CRD enum to horizon only rejects new resources \u0026ndash; schema validation runs on admission, never on read \u0026ndash; so a UI stored as kind: booster by an earlier operator survives the upgrade and still reconciles. With the templates now unconditionally Horizon\u0026rsquo;s, reconciling one rewrote a running Booster Deployment into a shape its image cannot serve and took the UI down on the first pass after upgrade. Such a resource is now left untouched, with a UnsupportedKind event saying what to do. Stop applying an OAPServer Deployment when the Storage it names cannot be read. Every lookup error was logged and ignored, and the Deployment was applied anyway \u0026ndash; without SW_STORAGE, targets, credentials or TLS volumes \u0026ndash; so a Storage briefly deleted and recreated replaced a working OAP with one that never becomes ready. The reconcile now leaves the running Deployment alone, emits a StorageUnresolved event and requeues. Keep the storage TLS volume when an OAPServerConfig mounts static files. The overlay assigned over the pod\u0026rsquo;s volume and mount lists, and ApplyOverlay is an RFC 7386 merge patch under which an array replaces rather than merges \u0026ndash; so the certificate volume disappeared and SW_STORAGE_BANYANDB_SSL_TRUST_CA_PATH pointed at nothing. The lists are merged by name now, and the mount is reconciled on every pass rather than skipped whenever the file content is unchanged, which had left it lost for good after any re-render. Roll the OAP when its credential Secret is rotated. Environment variables taken from a Secret are resolved once, when the container starts, so moving credentials to secretKeyRef meant a rotation went unnoticed until something restarted the pod. The controller now watches Secrets and carries the referenced Secret\u0026rsquo;s resourceVersion in a pod-template annotation \u0026ndash; an opaque token, not a digest of the credential. Create certificate signing requests through certificates.k8s.io/v1. The v1beta1 API this used was removed in Kubernetes 1.22, so internal Elasticsearch TLS could not obtain a certificate on any cluster newer than that and the workload waited on a Secret nothing produced. The wait loop is also bounded now, and sleeps \u0026ndash; it used to spin on Get with no delay and no limit. Require SW_STORAGE to carry a value. The mandatory-storage check accepted an entry named SW_STORAGE with nothing behind it, which reaches the OAP as an empty selector and produces exactly the never-ready state the check exists to prevent. Require a published GitHub release before publishing convenience binaries. Both the publish workflow and release-passed.sh tested only whether gh release view succeeded, and that resolves drafts \u0026ndash; so a dispatch against a draft could put official version-tagged images on GHCR before the vote, and release-passed.sh mistook a draft for a finished release and never published it. Both now require isDraft: false and a publication time; the workflow rejects prereleases too. Stop truncating rendered manifests at the first #. Every manifest was cut line-by-line at its first hash with no awareness of YAML quoting, so any value containing one \u0026ndash; a password, an AI prompt, a URL fragment \u0026ndash; was severed mid-string and the resulting manifest no longer parsed. Only whole-line comments are dropped now. This became reachable for user-supplied values with spec.env. Deep-copy the new env and envFrom fields. zz_generated.deepcopy.go had not been regenerated, so those slices were shared with the objects controller-runtime\u0026rsquo;s cache hands out. Render an OAPServer whose Storage cannot be read yet. spec.storage.name is now mandatory, but the operator fills in the resolved Storage only once it can read it, and the deployment template reached through the nil \u0026ndash; so an OAPServer applied before its Storage failed to render at all rather than waiting for it. Reference the Elasticsearch credentials from the Storage controller too. The OAPServer side stopped copying them out of the Secret; the Storage controller still did, putting the password into the resource and the Elasticsearch StatefulSet it renders. Document BanyanDB storage: the endpoint format and its gRPC port, cluster targets, authentication, persistence, and the flags BanyanDB 0.11 renamed. See docs/en/setup/banyandb.md. Pass BanyanDB credentials from a Storage\u0026rsquo;s security.user.secretName, as the Elasticsearch path already did. Raise the OAP startup probe budget from 110 seconds to 10 minutes. SkyWalking 11 has no embedded storage, so every start installs a schema into BanyanDB or Elasticsearch \u0026ndash; work that overran the old probe on a cold cluster, and being killed mid-schema turned a slow first boot into a crash loop. Wire BanyanDB storage. The operator could only configure Elasticsearch, so an OAPServer on BanyanDB had to carry the storage environment by hand \u0026ndash; and the variable it needed changed name between SkyWalking 9.x and 11.x. A Storage of type: banyandb now yields SW_STORAGE=banyandb and SW_STORAGE_BANYANDB_TARGETS. Without this, SWCK cannot deploy a working OAP at all across the supported range: SkyWalking removed H2 permanently in 10.2.0, so there is no fallback and an OAPServer with no storage never becomes ready. Stop deriving the Horizon admin and Zipkin URLs. The OAP admin host arrived in 11.x, and on 10.x port 17128 is the AI-pipeline URI-recognition server, so a derived oap.adminUrl pointed Horizon at a live endpoint that was the wrong service; the OAPServer this operator deploys exposes no Zipkin port at all. Both are now emitted only when spec.OAPServerAdminAddress / spec.OAPServerZipkinAddress are set, matching what skywalking-helm does. Fix the default image for kind: horizon UIs. It was apache/skywalking-horizon-ui:\u0026lt;version\u0026gt;, a Docker Hub repository that does not exist \u0026ndash; Horizon releases share apache/skywalking-ui with the legacy Booster UI and are told apart by a horizon- tag prefix. Since horizon is the default kind, every UI created without an explicit image could never pull. Covered by a unit test; the samples and docs carried the same wrong name. Build genuinely multi-architecture images. operator/Dockerfile, adapter/Dockerfile and build/images/Dockerfile.release hardcoded GOARCH=amd64 and -linux-amd64 while the publish workflow advertised linux/arm64, so apache/skywalking-swck:0.10.0 shipped an arm64 manifest holding x86-64 binaries and an arm64 node got exec format error. The release now builds a binary per architecture, and the publish workflow pulls every advertised platform back and checks the ELF machine type before the release completes. Ship the eventexporter admission webhook in the chart, and drop the duplicate meventexporter.kb.io entry that the API server rejects. Both come out of generating the webhook configurations from the operator sources rather than hand-copying them. Chores Seed the e2e login from a Secret through UI.spec.envFrom, so the new envFrom surface is exercised against a real cluster rather than only unit-tested: if it did not reach the container there would be no user and oap-ui-agent\u0026rsquo;s login would fail. Split the e2e suite by configuration path. Almost every case now runs the Horizon UI on HORIZON_* environment variables with no ConfigMap at all, and oap-ui-agent proves they arrive by logging in and reading the OAP hosts back through the BFF. Exactly one case, oap-ui-agent-oapserverconfig-oapserverdynamicconfig, takes the override path: the UI carries a whole horizon.yaml in spec.config while its OAPServerConfig overlays a static file on the OAP. Configure the Horizon UI with environment variables instead of a generated file. Horizon\u0026rsquo;s image bakes a fully tokenised horizon.yaml, so every setting it has is reachable as HORIZON_* \u0026ndash; but the operator mounted its own file over that one, which replaced every token and left the container with no env: at all, so no variable could reach any setting. UI.spec.env and UI.spec.envFrom now carry them, the operator sets only what it derives, and the ConfigMap is mounted only when spec.config supplies a whole file. A setting added in a future Horizon release works without an SWCK release. Add envFrom to OAPServer and Satellite, which are configured entirely through environment variables and previously had no way to take one from a Secret. Reference storage credentials instead of copying them. The operator read the Storage\u0026rsquo;s user secret and wrote the username and password in as literal env values, so they appeared in both the OAPServer and its Deployment for anyone with read access. They are now secretKeyRefs resolved by the kubelet. This also needed a template fix: the OAP deployment rendered only name/value and silently dropped valueFrom, so a secret reference could not have worked at all. Wire BanyanDB TLS. security.tls with security.tlsSecretName mounts the CA at /skywalking/bydb-tls and sets SW_STORAGE_BANYANDB_SSL_TRUST_CA_PATH. Previously tls: true on a banyandb Storage was accepted, wired no TLS, and mounted the Elasticsearch keystore secret skywalking-storage, leaving the OAP pod waiting on a secret nothing creates; tlsSecretName is now required for that combination. Add UI.spec.templatesMode, emitted as HORIZON_TEMPLATES_MODE. Left unset it follows the admin address: live reads OAP\u0026rsquo;s template store over the OAP admin host, so it is chosen only when spec.OAPServerAdminAddress is set, and readonly \u0026ndash; which renders the templates bundled in the image \u0026ndash; otherwise. Defaulting to live regardless would leave every UI probing 127.0.0.1:17128, failing Horizon\u0026rsquo;s ui-management preflight and blocking every layer-driven page, Traces most visibly. OAP 10.x needs readonly in any case: it manages templates over legacy query-port GraphQL and Horizon speaks only OAP 11\u0026rsquo;s REST protocol. Stop maintaining a copy of Horizon\u0026rsquo;s configuration schema. The config the operator used to generate restated Horizon\u0026rsquo;s own defaults, and the copy had drifted: viewer was granted 6 of the 12 permissions Horizon gives that role, and the admin landing route was /admin/cluster, which Horizon has no route for \u0026ndash; signing in as admin landed on \u0026ldquo;No route matches\u0026rdquo;. It also carried audit.file, setup and alarms, keys Horizon 1.0.0\u0026rsquo;s schema does not have and whose presence stops the BFF booting at all. Refuse an OAPServer that has no storage. SkyWalking removed the embedded H2 permanently in 10.2.0, so no version this operator supports has one and there is nothing to fall back to: an OAPServer with nowhere to write starts, dials a BanyanDB on 127.0.0.1:17912 and never becomes ready. The webhook now says so at admission instead. Setting SW_STORAGE directly in spec.config still counts as having chosen a storage. Breaking: an OAPServer with neither spec.storage.name nor SW_STORAGE is rejected \u0026ndash; it could never have worked. Publish the operator image, the metrics adapter image and the Helm chart from the release publish workflow, triggered by publishing a GitHub release, instead of pushing tags by hand. Pushes to master keep publishing SHA-tagged snapshots to GHCR. Add tools/releasing/release.sh and tools/releasing/release-passed.sh, automating the release either side of the vote. Ship the Helm chart tarball as a signed, voted artifact on dist.apache.org, alongside the source and binary tarballs. Support OAP 10.4.0 and later, with 11.0.0 recommended, matching skywalking-helm. An OAPServer below that is admitted with a warning rather than rejected. Deploy only the Horizon UI. spec.kind on the UI resource now accepts horizon alone \u0026ndash; apache/skywalking removed the legacy Booster UI in 11.0.0 and no longer builds an image for it. Breaking: a UI with kind: booster is rejected, with a message saying what to use instead. Default OAPServerConfig and OAPServerDynamicConfig to version 11.0.0, was 9.5.0. These match an OAPServer by exact version string, so a config that omits version previously only attached to an OAP explicitly pinned at 9.5.0. Behaviour change: set spec.version explicitly if you run an older OAP. Verify the Horizon UI over its own API rather than a GraphQL proxy it does not have: every UI case asserts the auth backend at the public /api/auth/health, and oap-ui-agent logs in with a seeded user and calls an RBAC-protected route. Document that Horizon ships with no users, so a UI refuses every login until one is seeded through HORIZON_AUTH_LOCAL_USERS. Move the e2e suite to the current SkyWalking stack: OAP 11.0.0, Horizon UI 1.0.0 and BanyanDB 0.11.0, which have to move together because OAP 11.0.0 accepts BanyanDB server API 0.11 only and Horizon 1.0.0\u0026rsquo;s admin host is an OAP 11 addition. The UI cases previously deployed the legacy Booster UI, which apache/skywalking no longer builds an image for. Pin every image the e2e suite deploys in test/e2e/env, substituted with envsubst, replacing apache/skywalking-banyandb:latest and centralising nine images that were spread across fifteen manifests. Install the operator with the Helm chart in ten of the twelve e2e cases, and add a case that tests the chart\u0026rsquo;s own lifecycle: installing the packaged tarball, CRDs, webhook CA injection, agent injection, HPA metrics, upgrade in both directions, and uninstall without taking the CRDs with it. test/e2e/oap-ui-agent stays on kustomize so that install path remains covered. Split hack/ by purpose: developer and build tooling moved to tools/, test tooling to test/tools/. Restructure the documentation into docs/en/{concepts-and-designs,setup,examples,guides,changes}, following the layout of apache/skywalking, and move the changelog from CHANGES.md into docs/en/changes/. Bump go.opentelemetry.io/otel to v1.44.0 to fix CVE-2026-41178. Bump golang.org/x/net to v0.55.0, golang.org/x/crypto to v0.53.0 and golang.org/x/sys to v0.46.0. Bump software.sslmate.com/src/go-pkcs12, github.com/sirupsen/logrus, github.com/go-logr/logr, google.golang.org/grpc, golang.org/x/text and the Kubernetes dependencies. Bump the actions-deps group across the repository. ","excerpt":"\u003ch2 id=\"0110\"\u003e0.11.0\u003c/h2\u003e\n\u003ch4 id=\"bugs\"\u003eBugs\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd \u003ccode\u003etools/releasing/preflight.sh\u003c/code\u003e, which checks everything a release needs before …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/next/en/changes/changes-0.11.0/","title":"0.11.0"},{"body":"0.11.0 Bugs Add tools/releasing/preflight.sh, which checks everything a release needs before release.sh does anything irreversible: the tools, the signing key and whether it is in the published KEYS file, gh authentication, the dist URLs, that the version agrees between Chart.yaml and the changelog and that its tag is free, that no abandoned candidate is sitting in dist/dev, and that the tree is clean with the generated chart files in sync. release.sh runs it as its first step, before asking about the signing key, and takes the resolved key from it rather than detecting one itself. It reports every problem rather than stopping at the first, and is explicit about the three things it cannot check from a developer machine. Let tools/releasing/release.sh run when the tree already carries the release version. It committed the version bump with a plain git commit, which exits non-zero with nothing staged, and the script runs under set -e \u0026ndash; so a release cut the documented way, with Chart.yaml already updated and the kustomize image tags already set by the same function, died at \u0026ldquo;nothing to commit, working tree clean\u0026rdquo; before tagging or building anything. Both release commits allow an empty diff now, which also keeps the regeneration --amend pointed at the script\u0026rsquo;s own commit rather than at whatever master happened to be. Build genuinely multi-architecture images, again. The previous fix declared ARG TARGETARCH=amd64 in each Dockerfile, and giving a predefined platform argument a default makes BuildKit use that default instead of the target\u0026rsquo;s architecture \u0026ndash; so TARGETARCH read amd64 even when building for linux/arm64, the builder stage ran once, and the amd64 binary was copied into the arm64 manifest. The publish workflow\u0026rsquo;s own ELF check caught it on the first run that was ever able to start. The argument is declared with no default now, and the shell falls back to the native architecture for a plain docker build, which was also producing amd64 binaries on an arm64 host. Features Release the skywalking-swck Helm chart from this repository. One chart installs the operator and, behind a values flag, the custom metrics adapter. The CRDs, the operator\u0026rsquo;s ClusterRole and the admission webhook configurations it ships are generated from the operator sources by make chart-manifests, and CI fails on any drift. Bugs Stop reconciling a UI whose kind is no longer supported. Narrowing the CRD enum to horizon only rejects new resources \u0026ndash; schema validation runs on admission, never on read \u0026ndash; so a UI stored as kind: booster by an earlier operator survives the upgrade and still reconciles. With the templates now unconditionally Horizon\u0026rsquo;s, reconciling one rewrote a running Booster Deployment into a shape its image cannot serve and took the UI down on the first pass after upgrade. Such a resource is now left untouched, with a UnsupportedKind event saying what to do. Stop applying an OAPServer Deployment when the Storage it names cannot be read. Every lookup error was logged and ignored, and the Deployment was applied anyway \u0026ndash; without SW_STORAGE, targets, credentials or TLS volumes \u0026ndash; so a Storage briefly deleted and recreated replaced a working OAP with one that never becomes ready. The reconcile now leaves the running Deployment alone, emits a StorageUnresolved event and requeues. Keep the storage TLS volume when an OAPServerConfig mounts static files. The overlay assigned over the pod\u0026rsquo;s volume and mount lists, and ApplyOverlay is an RFC 7386 merge patch under which an array replaces rather than merges \u0026ndash; so the certificate volume disappeared and SW_STORAGE_BANYANDB_SSL_TRUST_CA_PATH pointed at nothing. The lists are merged by name now, and the mount is reconciled on every pass rather than skipped whenever the file content is unchanged, which had left it lost for good after any re-render. Roll the OAP when its credential Secret is rotated. Environment variables taken from a Secret are resolved once, when the container starts, so moving credentials to secretKeyRef meant a rotation went unnoticed until something restarted the pod. The controller now watches Secrets and carries the referenced Secret\u0026rsquo;s resourceVersion in a pod-template annotation \u0026ndash; an opaque token, not a digest of the credential. Create certificate signing requests through certificates.k8s.io/v1. The v1beta1 API this used was removed in Kubernetes 1.22, so internal Elasticsearch TLS could not obtain a certificate on any cluster newer than that and the workload waited on a Secret nothing produced. The wait loop is also bounded now, and sleeps \u0026ndash; it used to spin on Get with no delay and no limit. Require SW_STORAGE to carry a value. The mandatory-storage check accepted an entry named SW_STORAGE with nothing behind it, which reaches the OAP as an empty selector and produces exactly the never-ready state the check exists to prevent. Require a published GitHub release before publishing convenience binaries. Both the publish workflow and release-passed.sh tested only whether gh release view succeeded, and that resolves drafts \u0026ndash; so a dispatch against a draft could put official version-tagged images on GHCR before the vote, and release-passed.sh mistook a draft for a finished release and never published it. Both now require isDraft: false and a publication time; the workflow rejects prereleases too. Stop truncating rendered manifests at the first #. Every manifest was cut line-by-line at its first hash with no awareness of YAML quoting, so any value containing one \u0026ndash; a password, an AI prompt, a URL fragment \u0026ndash; was severed mid-string and the resulting manifest no longer parsed. Only whole-line comments are dropped now. This became reachable for user-supplied values with spec.env. Deep-copy the new env and envFrom fields. zz_generated.deepcopy.go had not been regenerated, so those slices were shared with the objects controller-runtime\u0026rsquo;s cache hands out. Render an OAPServer whose Storage cannot be read yet. spec.storage.name is now mandatory, but the operator fills in the resolved Storage only once it can read it, and the deployment template reached through the nil \u0026ndash; so an OAPServer applied before its Storage failed to render at all rather than waiting for it. Reference the Elasticsearch credentials from the Storage controller too. The OAPServer side stopped copying them out of the Secret; the Storage controller still did, putting the password into the resource and the Elasticsearch StatefulSet it renders. Document BanyanDB storage: the endpoint format and its gRPC port, cluster targets, authentication, persistence, and the flags BanyanDB 0.11 renamed. See docs/en/setup/banyandb.md. Pass BanyanDB credentials from a Storage\u0026rsquo;s security.user.secretName, as the Elasticsearch path already did. Raise the OAP startup probe budget from 110 seconds to 10 minutes. SkyWalking 11 has no embedded storage, so every start installs a schema into BanyanDB or Elasticsearch \u0026ndash; work that overran the old probe on a cold cluster, and being killed mid-schema turned a slow first boot into a crash loop. Wire BanyanDB storage. The operator could only configure Elasticsearch, so an OAPServer on BanyanDB had to carry the storage environment by hand \u0026ndash; and the variable it needed changed name between SkyWalking 9.x and 11.x. A Storage of type: banyandb now yields SW_STORAGE=banyandb and SW_STORAGE_BANYANDB_TARGETS. Without this, SWCK cannot deploy a working OAP at all across the supported range: SkyWalking removed H2 permanently in 10.2.0, so there is no fallback and an OAPServer with no storage never becomes ready. Stop deriving the Horizon admin and Zipkin URLs. The OAP admin host arrived in 11.x, and on 10.x port 17128 is the AI-pipeline URI-recognition server, so a derived oap.adminUrl pointed Horizon at a live endpoint that was the wrong service; the OAPServer this operator deploys exposes no Zipkin port at all. Both are now emitted only when spec.OAPServerAdminAddress / spec.OAPServerZipkinAddress are set, matching what skywalking-helm does. Fix the default image for kind: horizon UIs. It was apache/skywalking-horizon-ui:\u0026lt;version\u0026gt;, a Docker Hub repository that does not exist \u0026ndash; Horizon releases share apache/skywalking-ui with the legacy Booster UI and are told apart by a horizon- tag prefix. Since horizon is the default kind, every UI created without an explicit image could never pull. Covered by a unit test; the samples and docs carried the same wrong name. Build genuinely multi-architecture images. operator/Dockerfile, adapter/Dockerfile and build/images/Dockerfile.release hardcoded GOARCH=amd64 and -linux-amd64 while the publish workflow advertised linux/arm64, so apache/skywalking-swck:0.10.0 shipped an arm64 manifest holding x86-64 binaries and an arm64 node got exec format error. The release now builds a binary per architecture, and the publish workflow pulls every advertised platform back and checks the ELF machine type before the release completes. Ship the eventexporter admission webhook in the chart, and drop the duplicate meventexporter.kb.io entry that the API server rejects. Both come out of generating the webhook configurations from the operator sources rather than hand-copying them. Chores Seed the e2e login from a Secret through UI.spec.envFrom, so the new envFrom surface is exercised against a real cluster rather than only unit-tested: if it did not reach the container there would be no user and oap-ui-agent\u0026rsquo;s login would fail. Split the e2e suite by configuration path. Almost every case now runs the Horizon UI on HORIZON_* environment variables with no ConfigMap at all, and oap-ui-agent proves they arrive by logging in and reading the OAP hosts back through the BFF. Exactly one case, oap-ui-agent-oapserverconfig-oapserverdynamicconfig, takes the override path: the UI carries a whole horizon.yaml in spec.config while its OAPServerConfig overlays a static file on the OAP. Configure the Horizon UI with environment variables instead of a generated file. Horizon\u0026rsquo;s image bakes a fully tokenised horizon.yaml, so every setting it has is reachable as HORIZON_* \u0026ndash; but the operator mounted its own file over that one, which replaced every token and left the container with no env: at all, so no variable could reach any setting. UI.spec.env and UI.spec.envFrom now carry them, the operator sets only what it derives, and the ConfigMap is mounted only when spec.config supplies a whole file. A setting added in a future Horizon release works without an SWCK release. Add envFrom to OAPServer and Satellite, which are configured entirely through environment variables and previously had no way to take one from a Secret. Reference storage credentials instead of copying them. The operator read the Storage\u0026rsquo;s user secret and wrote the username and password in as literal env values, so they appeared in both the OAPServer and its Deployment for anyone with read access. They are now secretKeyRefs resolved by the kubelet. This also needed a template fix: the OAP deployment rendered only name/value and silently dropped valueFrom, so a secret reference could not have worked at all. Wire BanyanDB TLS. security.tls with security.tlsSecretName mounts the CA at /skywalking/bydb-tls and sets SW_STORAGE_BANYANDB_SSL_TRUST_CA_PATH. Previously tls: true on a banyandb Storage was accepted, wired no TLS, and mounted the Elasticsearch keystore secret skywalking-storage, leaving the OAP pod waiting on a secret nothing creates; tlsSecretName is now required for that combination. Add UI.spec.templatesMode, emitted as HORIZON_TEMPLATES_MODE. Left unset it follows the admin address: live reads OAP\u0026rsquo;s template store over the OAP admin host, so it is chosen only when spec.OAPServerAdminAddress is set, and readonly \u0026ndash; which renders the templates bundled in the image \u0026ndash; otherwise. Defaulting to live regardless would leave every UI probing 127.0.0.1:17128, failing Horizon\u0026rsquo;s ui-management preflight and blocking every layer-driven page, Traces most visibly. OAP 10.x needs readonly in any case: it manages templates over legacy query-port GraphQL and Horizon speaks only OAP 11\u0026rsquo;s REST protocol. Stop maintaining a copy of Horizon\u0026rsquo;s configuration schema. The config the operator used to generate restated Horizon\u0026rsquo;s own defaults, and the copy had drifted: viewer was granted 6 of the 12 permissions Horizon gives that role, and the admin landing route was /admin/cluster, which Horizon has no route for \u0026ndash; signing in as admin landed on \u0026ldquo;No route matches\u0026rdquo;. It also carried audit.file, setup and alarms, keys Horizon 1.0.0\u0026rsquo;s schema does not have and whose presence stops the BFF booting at all. Refuse an OAPServer that has no storage. SkyWalking removed the embedded H2 permanently in 10.2.0, so no version this operator supports has one and there is nothing to fall back to: an OAPServer with nowhere to write starts, dials a BanyanDB on 127.0.0.1:17912 and never becomes ready. The webhook now says so at admission instead. Setting SW_STORAGE directly in spec.config still counts as having chosen a storage. Breaking: an OAPServer with neither spec.storage.name nor SW_STORAGE is rejected \u0026ndash; it could never have worked. Publish the operator image, the metrics adapter image and the Helm chart from the release publish workflow, triggered by publishing a GitHub release, instead of pushing tags by hand. Pushes to master keep publishing SHA-tagged snapshots to GHCR. Add tools/releasing/release.sh and tools/releasing/release-passed.sh, automating the release either side of the vote. Ship the Helm chart tarball as a signed, voted artifact on dist.apache.org, alongside the source and binary tarballs. Support OAP 10.4.0 and later, with 11.0.0 recommended, matching skywalking-helm. An OAPServer below that is admitted with a warning rather than rejected. Deploy only the Horizon UI. spec.kind on the UI resource now accepts horizon alone \u0026ndash; apache/skywalking removed the legacy Booster UI in 11.0.0 and no longer builds an image for it. Breaking: a UI with kind: booster is rejected, with a message saying what to use instead. Default OAPServerConfig and OAPServerDynamicConfig to version 11.0.0, was 9.5.0. These match an OAPServer by exact version string, so a config that omits version previously only attached to an OAP explicitly pinned at 9.5.0. Behaviour change: set spec.version explicitly if you run an older OAP. Verify the Horizon UI over its own API rather than a GraphQL proxy it does not have: every UI case asserts the auth backend at the public /api/auth/health, and oap-ui-agent logs in with a seeded user and calls an RBAC-protected route. Document that Horizon ships with no users, so a UI refuses every login until one is seeded through HORIZON_AUTH_LOCAL_USERS. Move the e2e suite to the current SkyWalking stack: OAP 11.0.0, Horizon UI 1.0.0 and BanyanDB 0.11.0, which have to move together because OAP 11.0.0 accepts BanyanDB server API 0.11 only and Horizon 1.0.0\u0026rsquo;s admin host is an OAP 11 addition. The UI cases previously deployed the legacy Booster UI, which apache/skywalking no longer builds an image for. Pin every image the e2e suite deploys in test/e2e/env, substituted with envsubst, replacing apache/skywalking-banyandb:latest and centralising nine images that were spread across fifteen manifests. Install the operator with the Helm chart in ten of the twelve e2e cases, and add a case that tests the chart\u0026rsquo;s own lifecycle: installing the packaged tarball, CRDs, webhook CA injection, agent injection, HPA metrics, upgrade in both directions, and uninstall without taking the CRDs with it. test/e2e/oap-ui-agent stays on kustomize so that install path remains covered. Split hack/ by purpose: developer and build tooling moved to tools/, test tooling to test/tools/. Restructure the documentation into docs/en/{concepts-and-designs,setup,examples,guides,changes}, following the layout of apache/skywalking, and move the changelog from CHANGES.md into docs/en/changes/. Bump go.opentelemetry.io/otel to v1.44.0 to fix CVE-2026-41178. Bump golang.org/x/net to v0.55.0, golang.org/x/crypto to v0.53.0 and golang.org/x/sys to v0.46.0. Bump software.sslmate.com/src/go-pkcs12, github.com/sirupsen/logrus, github.com/go-logr/logr, google.golang.org/grpc, golang.org/x/text and the Kubernetes dependencies. Bump the actions-deps group across the repository. ","excerpt":"\u003ch2 id=\"0110\"\u003e0.11.0\u003c/h2\u003e\n\u003ch4 id=\"bugs\"\u003eBugs\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd \u003ccode\u003etools/releasing/preflight.sh\u003c/code\u003e, which checks everything a release needs before …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/v0.11.0/en/changes/changes/","title":"0.11.0"},{"body":"0.11.0 Design Documents Storage-node post-trace pipeline Trace fragment sampling guard Trace-pipeline merge optimization plan Trace-pipeline merge performance test Trace drop-set bounding plan Trace drop-set bounding ","excerpt":"\u003ch1 id=\"0110-design-documents\"\u003e0.11.0 Design Documents\u003c/h1\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"../post-trace-pipeline\"\u003eStorage-node post-trace pipeline\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"../trace-fragment-sampling-guard\"\u003eTrace fragment sampling guard …\u003c/a\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-banyandb/next/design/archive/0.11.0/readme/","title":"0.11.0 Design Documents"},{"body":"0.12.0 Features Bugs Log in to Docker Hub as docker.io, not registry-1.docker.io. Both names resolve to the same registry and either can be pulled from anonymously, but the credential is stored under the host you logged in as while a push normalises the reference back to Docker Hub\u0026rsquo;s canonical host \u0026ndash; so the chart push looked up a credential that had never been written there and would have failed with a 401. apache/skywalking uses docker.io throughout for this reason. The docs and the chart\u0026rsquo;s NOTES use the same name now. Publish snapshots to GHCR and releases to Docker Hub, and nothing to both, with the workflow\u0026rsquo;s jobs named for the path they serve \u0026ndash; publish-snapshot-images, publish-snapshot-chart, publish-release-image, publish-release-chart. The per-component images and the GHCR chart were pushed on the release path too, so a release also produced ghcr.io/apache/skywalking-swck/operator:\u0026lt;version\u0026gt; \u0026ndash; artifacts nobody asked for, published from a job whose failure then blocked the release. They are snapshot-only now, and a release publishes the combined image and the chart to Docker Hub. Fetch the release tarball from dist/release, which is where the svn move puts it and is servable immediately. Dockerfile.release downloaded it from archive.apache.org instead \u0026ndash; a copy that holds every release ever made but takes hours to receive a new one \u0026ndash; so a just-voted release could not be built until the archive caught up, and both release-passed.sh and the publish workflow sat in long polls waiting for it, up to an hour and thirty minutes respectively. Neither wait was for anything the build reads. Take release-passed.sh\u0026rsquo;s default version from dist/dev rather than from Chart.yaml. By the time a vote passes, release.sh has already opened the next-version PR and it is usually merged, so Chart.yaml holds the version after the one being released \u0026ndash; pressing Enter at the prompt tried to publish a candidate that does not exist. It failed, but several prompts later and with an svn path error rather than an explanation. The script now reads what is actually waiting in dist/dev, refuses to guess when there is more than one candidate, and checks the chosen version exists before asking whether the vote passed. Chores ","excerpt":"\u003ch2 id=\"0120\"\u003e0.12.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003ch4 id=\"bugs\"\u003eBugs\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eLog in to Docker Hub as \u003ccode\u003edocker.io\u003c/code\u003e, not \u003ccode\u003eregistry-1.docker.io\u003c/code\u003e. Both names …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/next/en/changes/changes/","title":"0.12.0"},{"body":"0.12.0 Design Documents Native inverted-index replacement ","excerpt":"\u003ch1 id=\"0120-design-documents\"\u003e0.12.0 Design Documents\u003c/h1\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"../native-inverted-index/readme\"\u003eNative inverted-index replacement\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-banyandb/next/design/archive/0.12.0/readme/","title":"0.12.0 Design Documents"},{"body":"0.2.0 Per-layer dashboards become real, the layer-template editor ships, and topology gets its booster-ui port.\nPer-layer dashboards Real widget grid per layer driven by JSON templates. 43 layer dashboards migrated from booster-ui. Per-scope widget sets: each layer template defines its own service, instance, endpoint, topology, traces, logs, profiling variants. Visibility predicates per widget (visibleWhen) so MQ / DB widgets only render when the relevant metrics are reporting. Layer admin Read-only template browser, then full edit UI: components editor (toggle which per-layer views exist), metrics editor (header columns), separate Overview tile card, scope-aware visibleWhen hints. Service deep-dive APIs widget (formerly Services), MQ widgets gated by visibleWhen, TopList multi-expression switcher with MQE preview in tooltip, smaller widget height, per-metric color alignment, dual-axis MQ. Topology Polished linear-chain variant, dual-panel detail, per-side line charts. Drag-to-move + barycentric layout for smaller graphs. RPM-only chip variant. Istio renamed. Logs Legend at top of table (drop service facet duplication), workflow notes. Charting TimeChart: legend formatting fix for dual-axis widgets, value dots, tooltip escape for clipped charts, no more legend / axis-name crowding at chart top. Sidebar + chrome Group toggle + group click cascades to first layer\u0026rsquo;s first tab. Topbar 60m widget format hints (int / decimal / compact). Per-layer image pipeline (icons) shipped. ","excerpt":"\u003ch1 id=\"020\"\u003e0.2.0\u003c/h1\u003e\n\u003cp\u003ePer-layer dashboards become real, the layer-template editor ships, and topology gets its …\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/changelog/0.2.0/","title":"0.2.0"},{"body":"0.2.0 Per-layer dashboards become real, the layer-template editor ships, and topology gets its booster-ui port.\nPer-layer dashboards Real widget grid per layer driven by JSON templates. 43 layer dashboards migrated from booster-ui. Per-scope widget sets: each layer template defines its own service, instance, endpoint, topology, traces, logs, profiling variants. Visibility predicates per widget (visibleWhen) so MQ / DB widgets only render when the relevant metrics are reporting. Layer admin Read-only template browser, then full edit UI: components editor (toggle which per-layer views exist), metrics editor (header columns), separate Overview tile card, scope-aware visibleWhen hints. Service deep-dive APIs widget (formerly Services), MQ widgets gated by visibleWhen, TopList multi-expression switcher with MQE preview in tooltip, smaller widget height, per-metric color alignment, dual-axis MQ. Topology Polished linear-chain variant, dual-panel detail, per-side line charts. Drag-to-move + barycentric layout for smaller graphs. RPM-only chip variant. Istio renamed. Logs Legend at top of table (drop service facet duplication), workflow notes. Charting TimeChart: legend formatting fix for dual-axis widgets, value dots, tooltip escape for clipped charts, no more legend / axis-name crowding at chart top. Sidebar + chrome Group toggle + group click cascades to first layer\u0026rsquo;s first tab. Topbar 60m widget format hints (int / decimal / compact). Per-layer image pipeline (icons) shipped. ","excerpt":"\u003ch1 id=\"020\"\u003e0.2.0\u003c/h1\u003e\n\u003cp\u003ePer-layer dashboards become real, the layer-template editor ships, and topology gets its …\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/changelog/0.2.0/","title":"0.2.0"},{"body":"0.2.0 Per-layer dashboards become real, the layer-template editor ships, and topology gets its booster-ui port.\nPer-layer dashboards Real widget grid per layer driven by JSON templates. 43 layer dashboards migrated from booster-ui. Per-scope widget sets: each layer template defines its own service, instance, endpoint, topology, traces, logs, profiling variants. Visibility predicates per widget (visibleWhen) so MQ / DB widgets only render when the relevant metrics are reporting. Layer admin Read-only template browser, then full edit UI: components editor (toggle which per-layer views exist), metrics editor (header columns), separate Overview tile card, scope-aware visibleWhen hints. Service deep-dive APIs widget (formerly Services), MQ widgets gated by visibleWhen, TopList multi-expression switcher with MQE preview in tooltip, smaller widget height, per-metric color alignment, dual-axis MQ. Topology Polished linear-chain variant, dual-panel detail, per-side line charts. Drag-to-move + barycentric layout for smaller graphs. RPM-only chip variant. Istio renamed. Logs Legend at top of table (drop service facet duplication), workflow notes. Charting TimeChart: legend formatting fix for dual-axis widgets, value dots, tooltip escape for clipped charts, no more legend / axis-name crowding at chart top. Sidebar + chrome Group toggle + group click cascades to first layer\u0026rsquo;s first tab. Topbar 60m widget format hints (int / decimal / compact). Per-layer image pipeline (icons) shipped. ","excerpt":"\u003ch1 id=\"020\"\u003e0.2.0\u003c/h1\u003e\n\u003cp\u003ePer-layer dashboards become real, the layer-template editor ships, and topology gets its …\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/changelog/0.2.0/","title":"0.2.0"},{"body":"0.2.0 Features Introduce custom metrics adapter to SkyWalking OAP cluster for Kubernetes HPA autoscaling. Add RBAC files and service account to support Kubernetes coordination. Add default and validation webhooks to operator controllers. Add UI CRD to deploy skywalking UI server. Add Fetcher CRD to fetch metrics from other telemetry system, for example, Prometheus. Chores Transform project layers to support multiple applications. Introduce unit test to verify the operator. ","excerpt":"\u003ch2 id=\"020\"\u003e0.2.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eIntroduce custom metrics adapter to SkyWalking OAP cluster for Kubernetes HPA …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/latest/en/changes/changes-0.2.0/","title":"0.2.0"},{"body":"0.2.0 Features Introduce custom metrics adapter to SkyWalking OAP cluster for Kubernetes HPA autoscaling. Add RBAC files and service account to support Kubernetes coordination. Add default and validation webhooks to operator controllers. Add UI CRD to deploy skywalking UI server. Add Fetcher CRD to fetch metrics from other telemetry system, for example, Prometheus. Chores Transform project layers to support multiple applications. Introduce unit test to verify the operator. ","excerpt":"\u003ch2 id=\"020\"\u003e0.2.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eIntroduce custom metrics adapter to SkyWalking OAP cluster for Kubernetes HPA …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/next/en/changes/changes-0.2.0/","title":"0.2.0"},{"body":"0.2.0 Features Introduce custom metrics adapter to SkyWalking OAP cluster for Kubernetes HPA autoscaling. Add RBAC files and service account to support Kubernetes coordination. Add default and validation webhooks to operator controllers. Add UI CRD to deploy skywalking UI server. Add Fetcher CRD to fetch metrics from other telemetry system, for example, Prometheus. Chores Transform project layers to support multiple applications. Introduce unit test to verify the operator. ","excerpt":"\u003ch2 id=\"020\"\u003e0.2.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eIntroduce custom metrics adapter to SkyWalking OAP cluster for Kubernetes HPA …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/v0.11.0/en/changes/changes-0.2.0/","title":"0.2.0"},{"body":"0.3.0 The shell unifies, the operate stack lands, and the first round of public documentation ships.\nOperate stack Alarms page — incident-merged active-alarms view, severity tabs, alarm list with right-side detail (trigger expression, channel routing), inline Live Debug card (Run / Step / Pause / Copy as MQE, execution-trace ladder with per-step output + latency, matched entities, eval-window chart, raw OAP response). Inspect — metric catalog + entity enumerator with search, type filter, scope (Service / Instance / Endpoint / Process / All), and source attribution. Live Debugger — MAL / LAL / OAL session start, poll, stop. Per-node status fan-out, sample payloads, capture history with replay-ready recordings. Profiling — flame graph + stack table over five profilers: trace-driven thread profiling, eBPF CPU/off-CPU, JVM async-profiler, network profiling (process conversation graph), Go pprof. Zipkin trace explorer — service / span search, waterfall popout with per-service color bands, sticky time-axis. Overview dashboards — cross-layer war-room views (Services, Mesh) with per-layer KPI tiles, alarm rails, and the existing chart widgets. Auth + access control Local + LDAP authentication backends. Break-glass admin honored only when backend: ldap AND the LDAP probe is failing. Three admin pages — Users, Auth status, Roles \u0026amp; permissions. 4 built-in roles (viewer / maintainer / operator / admin) and a 28-verb permission model. Every BFF route gated by a single policy table. Login view redesigned (canyon hero, status pill, configured-backend banner). Reliability + UX Cascade-clear, then load — every dependent area visibly resets and shows \u0026ldquo;Reading data…\u0026rdquo; between an upstream control change (service / instance / endpoint pick, time-range change, layer / scope nav) and the new data landing. No silent freezes; no stale value sitting under a spinner. Global time picker in the topbar wired into the landing + widget query keys; the picker only applies to dashboards / overviews (triage pages keep their own per-page time). Single-shot bundle preload: layer dashboards + overview list arrive in one round-trip, cached in localStorage with ETag revalidation. Framework event ticker in the topbar replaces breadcrumb+search; Admin-toggled debug panel surfaces a 200-event buffer with operator click capture. Auto-pick first instance / endpoint when a scope needs one and the list is non-empty. Topology + dashboard fixes, multi-layer service attribution, sticky service selection across navigations. Documentation First public docs tree (docs/) — Setup, Compatibility, Access Control, Customization, Components, Operate. Lives in-repo and publishes to skywalking.apache.org. Container + CI Real packages/* builds + self-contained dist/ + copy-in image (no compile in the container). Zero-config boot: image defaults HORIZON_SERVER_HOST=0.0.0.0. Multi-arch publish-image — native amd64 + arm64 builds, OCI manifest list. Unit-test job in CI; 107 UTs covering entity-scope construction + routing decisions. ","excerpt":"\u003ch1 id=\"030\"\u003e0.3.0\u003c/h1\u003e\n\u003cp\u003eThe shell unifies, the operate stack lands, and the first round of public documentation ships. …\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/changelog/0.3.0/","title":"0.3.0"},{"body":"0.3.0 The shell unifies, the operate stack lands, and the first round of public documentation ships.\nOperate stack Alarms page — incident-merged active-alarms view, severity tabs, alarm list with right-side detail (trigger expression, channel routing), inline Live Debug card (Run / Step / Pause / Copy as MQE, execution-trace ladder with per-step output + latency, matched entities, eval-window chart, raw OAP response). Inspect — metric catalog + entity enumerator with search, type filter, scope (Service / Instance / Endpoint / Process / All), and source attribution. Live Debugger — MAL / LAL / OAL session start, poll, stop. Per-node status fan-out, sample payloads, capture history with replay-ready recordings. Profiling — flame graph + stack table over five profilers: trace-driven thread profiling, eBPF CPU/off-CPU, JVM async-profiler, network profiling (process conversation graph), Go pprof. Zipkin trace explorer — service / span search, waterfall popout with per-service color bands, sticky time-axis. Overview dashboards — cross-layer war-room views (Services, Mesh) with per-layer KPI tiles, alarm rails, and the existing chart widgets. Auth + access control Local + LDAP authentication backends. Break-glass admin honored only when backend: ldap AND the LDAP probe is failing. Three admin pages — Users, Auth status, Roles \u0026amp; permissions. 4 built-in roles (viewer / maintainer / operator / admin) and a 28-verb permission model. Every BFF route gated by a single policy table. Login view redesigned (canyon hero, status pill, configured-backend banner). Reliability + UX Cascade-clear, then load — every dependent area visibly resets and shows \u0026ldquo;Reading data…\u0026rdquo; between an upstream control change (service / instance / endpoint pick, time-range change, layer / scope nav) and the new data landing. No silent freezes; no stale value sitting under a spinner. Global time picker in the topbar wired into the landing + widget query keys; the picker only applies to dashboards / overviews (triage pages keep their own per-page time). Single-shot bundle preload: layer dashboards + overview list arrive in one round-trip, cached in localStorage with ETag revalidation. Framework event ticker in the topbar replaces breadcrumb+search; Admin-toggled debug panel surfaces a 200-event buffer with operator click capture. Auto-pick first instance / endpoint when a scope needs one and the list is non-empty. Topology + dashboard fixes, multi-layer service attribution, sticky service selection across navigations. Documentation First public docs tree (docs/) — Setup, Compatibility, Access Control, Customization, Components, Operate. Lives in-repo and publishes to skywalking.apache.org. Container + CI Real packages/* builds + self-contained dist/ + copy-in image (no compile in the container). Zero-config boot: image defaults HORIZON_SERVER_HOST=0.0.0.0. Multi-arch publish-image — native amd64 + arm64 builds, OCI manifest list. Unit-test job in CI; 107 UTs covering entity-scope construction + routing decisions. ","excerpt":"\u003ch1 id=\"030\"\u003e0.3.0\u003c/h1\u003e\n\u003cp\u003eThe shell unifies, the operate stack lands, and the first round of public documentation ships. …\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/changelog/0.3.0/","title":"0.3.0"},{"body":"0.3.0 The shell unifies, the operate stack lands, and the first round of public documentation ships.\nOperate stack Alarms page — incident-merged active-alarms view, severity tabs, alarm list with right-side detail (trigger expression, channel routing), inline Live Debug card (Run / Step / Pause / Copy as MQE, execution-trace ladder with per-step output + latency, matched entities, eval-window chart, raw OAP response). Inspect — metric catalog + entity enumerator with search, type filter, scope (Service / Instance / Endpoint / Process / All), and source attribution. Live Debugger — MAL / LAL / OAL session start, poll, stop. Per-node status fan-out, sample payloads, capture history with replay-ready recordings. Profiling — flame graph + stack table over five profilers: trace-driven thread profiling, eBPF CPU/off-CPU, JVM async-profiler, network profiling (process conversation graph), Go pprof. Zipkin trace explorer — service / span search, waterfall popout with per-service color bands, sticky time-axis. Overview dashboards — cross-layer war-room views (Services, Mesh) with per-layer KPI tiles, alarm rails, and the existing chart widgets. Auth + access control Local + LDAP authentication backends. Break-glass admin honored only when backend: ldap AND the LDAP probe is failing. Three admin pages — Users, Auth status, Roles \u0026amp; permissions. 4 built-in roles (viewer / maintainer / operator / admin) and a 28-verb permission model. Every BFF route gated by a single policy table. Login view redesigned (canyon hero, status pill, configured-backend banner). Reliability + UX Cascade-clear, then load — every dependent area visibly resets and shows \u0026ldquo;Reading data…\u0026rdquo; between an upstream control change (service / instance / endpoint pick, time-range change, layer / scope nav) and the new data landing. No silent freezes; no stale value sitting under a spinner. Global time picker in the topbar wired into the landing + widget query keys; the picker only applies to dashboards / overviews (triage pages keep their own per-page time). Single-shot bundle preload: layer dashboards + overview list arrive in one round-trip, cached in localStorage with ETag revalidation. Framework event ticker in the topbar replaces breadcrumb+search; Admin-toggled debug panel surfaces a 200-event buffer with operator click capture. Auto-pick first instance / endpoint when a scope needs one and the list is non-empty. Topology + dashboard fixes, multi-layer service attribution, sticky service selection across navigations. Documentation First public docs tree (docs/) — Setup, Compatibility, Access Control, Customization, Components, Operate. Lives in-repo and publishes to skywalking.apache.org. Container + CI Real packages/* builds + self-contained dist/ + copy-in image (no compile in the container). Zero-config boot: image defaults HORIZON_SERVER_HOST=0.0.0.0. Multi-arch publish-image — native amd64 + arm64 builds, OCI manifest list. Unit-test job in CI; 107 UTs covering entity-scope construction + routing decisions. ","excerpt":"\u003ch1 id=\"030\"\u003e0.3.0\u003c/h1\u003e\n\u003cp\u003eThe shell unifies, the operate stack lands, and the first round of public documentation ships. …\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/changelog/0.3.0/","title":"0.3.0"},{"body":"0.3.0 Features Support special characters in the metric selector of HPA metric adapter. Add the namespace to HPA metric name. Chores Upgrade skywalking-cli dependency. ","excerpt":"\u003ch2 id=\"030\"\u003e0.3.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport special characters in the metric selector of HPA metric adapter.\u003c/li\u003e\n\u003cli\u003eAdd the …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/latest/en/changes/changes-0.3.0/","title":"0.3.0"},{"body":"0.3.0 Features Support special characters in the metric selector of HPA metric adapter. Add the namespace to HPA metric name. Chores Upgrade skywalking-cli dependency. ","excerpt":"\u003ch2 id=\"030\"\u003e0.3.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport special characters in the metric selector of HPA metric adapter.\u003c/li\u003e\n\u003cli\u003eAdd the …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/next/en/changes/changes-0.3.0/","title":"0.3.0"},{"body":"0.3.0 Features Support special characters in the metric selector of HPA metric adapter. Add the namespace to HPA metric name. Chores Upgrade skywalking-cli dependency. ","excerpt":"\u003ch2 id=\"030\"\u003e0.3.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport special characters in the metric selector of HPA metric adapter.\u003c/li\u003e\n\u003cli\u003eAdd the …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/v0.11.0/en/changes/changes-0.3.0/","title":"0.3.0"},{"body":"0.4.0 OAP becomes the runtime source of truth for UI templates, the 5-theme system lands, and the app supports being served behind a gateway prefix.\nTemplates synced to OAP Five reserved template families now live on OAP\u0026rsquo;s UI-template REST surface (/ui-management/templates* on the admin port): overview dashboards, per-layer dashboards, alert page setup, theme selection, time-defaults. Bundled JSON ships as the seed + read-only fallback. One-shot seed on BFF boot pushes any missing bundled template to OAP; runtime sync is read-only with a 30-second single-flight cache. New admin endpoints: GET /api/admin/templates/sync-status, POST /api/admin/templates/save, POST /api/admin/templates/resync, POST /api/admin/templates/:name/push-bundled. When the admin port is unreachable, every admin page goes read-only with a red banner; Save / Create / Delete are disabled; render falls back to bundled. Diverged rows surface a \u0026ldquo;Show diff \u0026amp; reset\u0026rdquo; Monaco modal with a destructive-confirm (type the template key to arm reset). Themes Five bundled themes — Horizon (default), Meridian, Obsidian, Daybreak, Aurora — each shipping a complete token set (bg, fg, accent, info/ok/warn/err, font, radius, density). New /admin/global-defaults admin page replaces the old \u0026ldquo;Setup\u0026rdquo; link. Theme picker uses preview cards lifted from the design (hero strip, mini-app mockup with Primary/Tonal/Ghost buttons, KPI tiles, sparkline, density/font/radius badges). Per-user theme override via a labelled topbar chip — three-tier resolution localStorage user → OAP org default → bundled, written to \u0026lt;html data-theme\u0026gt; / \u0026lt;html data-appearance\u0026gt; synchronously on boot so the pre-auth login page already respects the local override. Sidebar SkyWalking logo swaps to the official brand blue (#1368B3) on light-appearance themes. Widget series colors (Zipkin trace palette, AlarmSnapshotChart, AlarmsTimeline) track the active theme\u0026rsquo;s --sw-accent via a shared readAccent() util. Sign-in button gradient derives both stops from the theme accent. Time defaults /admin/global-defaults also owns the global picker\u0026rsquo;s default window (60 minutes shipped). OAP step precision is derived from window size — ≤ 4 h MINUTE, 6 h–14 d HOUR, ≥ 30 d DAY — and surfaced inline on the page. Per-user override in the topbar time picker: \u0026ldquo;Save as my default\u0026rdquo; / \u0026ldquo;Reset to org default\u0026rdquo;. Reliability + diagnostics Topology cluster boundary now grows to encompass dragged nodes; the chip moved inside the cluster header so it stays visible at any drag position. Alarms page gains an Other KPI tile that surfaces the residual count between Active and the sum of pinned-layer chips — Active = General + Mesh + Other reconciles even when alarms land in unmapped layers. Overview \u0026ldquo;Active alarms\u0026rdquo; widget now reads the admin\u0026rsquo;s configured defaultWindowMs from /admin/alert-page-setup; all three alarm surfaces (overview widget, alarms page, topbar badge) share one window. Every backend call failure (network throw or non-2xx) writes a pushEvent('api', 'err', …) into the debug event log with the BFF\u0026rsquo;s code / message envelope inlined when present. Dashboards with more than 40 widgets (e.g. the General/instance page, 56 widgets) now succeed: the UI splits oversize requests into ≤40-widget chunks fired in parallel, then merges results. Deployment Gateway-prefix support: BffClient.request() prepends import.meta.env.BASE_URL to every API path. Build with vite build --base=/horizon/ and a gateway that strips the prefix and the SPA + every API call resolves cleanly under the sub-path. Cluster Status route corrected from /admin/cluster → /operate/cluster (the prior default 404\u0026rsquo;d because no route by that name existed). Cleanup Documentation rewritten as an orientation map; the left-side menu is the canonical navigation now. All SWIP-* references removed from user-visible text and docs. \u0026ldquo;Coming in Phase 6 / 7\u0026rdquo; placeholder strip on Cluster Status removed. Dead code dropped — LandingView.vue, LayerTabPlaceholder.vue, the orphaned disk-write template routes (POST /api/admin/overview-templates/:id + POST /api/admin/layer-templates/:key), and stale Phase X markers across BFF + UI + docs. The OAP UTC-offset chip is gone from the topbar; the health dot stays. ","excerpt":"\u003ch1 id=\"040\"\u003e0.4.0\u003c/h1\u003e\n\u003cp\u003eOAP becomes the runtime source of truth for UI templates, the 5-theme system lands, and the …\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/changelog/0.4.0/","title":"0.4.0"},{"body":"0.4.0 OAP becomes the runtime source of truth for UI templates, the 5-theme system lands, and the app supports being served behind a gateway prefix.\nTemplates synced to OAP Five reserved template families now live on OAP\u0026rsquo;s UI-template REST surface (/ui-management/templates* on the admin port): overview dashboards, per-layer dashboards, alert page setup, theme selection, time-defaults. Bundled JSON ships as the seed + read-only fallback. One-shot seed on BFF boot pushes any missing bundled template to OAP; runtime sync is read-only with a 30-second single-flight cache. New admin endpoints: GET /api/admin/templates/sync-status, POST /api/admin/templates/save, POST /api/admin/templates/resync, POST /api/admin/templates/:name/push-bundled. When the admin port is unreachable, every admin page goes read-only with a red banner; Save / Create / Delete are disabled; render falls back to bundled. Diverged rows surface a \u0026ldquo;Show diff \u0026amp; reset\u0026rdquo; Monaco modal with a destructive-confirm (type the template key to arm reset). Themes Five bundled themes — Horizon (default), Meridian, Obsidian, Daybreak, Aurora — each shipping a complete token set (bg, fg, accent, info/ok/warn/err, font, radius, density). New /admin/global-defaults admin page replaces the old \u0026ldquo;Setup\u0026rdquo; link. Theme picker uses preview cards lifted from the design (hero strip, mini-app mockup with Primary/Tonal/Ghost buttons, KPI tiles, sparkline, density/font/radius badges). Per-user theme override via a labelled topbar chip — three-tier resolution localStorage user → OAP org default → bundled, written to \u0026lt;html data-theme\u0026gt; / \u0026lt;html data-appearance\u0026gt; synchronously on boot so the pre-auth login page already respects the local override. Sidebar SkyWalking logo swaps to the official brand blue (#1368B3) on light-appearance themes. Widget series colors (Zipkin trace palette, AlarmSnapshotChart, AlarmsTimeline) track the active theme\u0026rsquo;s --sw-accent via a shared readAccent() util. Sign-in button gradient derives both stops from the theme accent. Time defaults /admin/global-defaults also owns the global picker\u0026rsquo;s default window (60 minutes shipped). OAP step precision is derived from window size — ≤ 4 h MINUTE, 6 h–14 d HOUR, ≥ 30 d DAY — and surfaced inline on the page. Per-user override in the topbar time picker: \u0026ldquo;Save as my default\u0026rdquo; / \u0026ldquo;Reset to org default\u0026rdquo;. Reliability + diagnostics Topology cluster boundary now grows to encompass dragged nodes; the chip moved inside the cluster header so it stays visible at any drag position. Alarms page gains an Other KPI tile that surfaces the residual count between Active and the sum of pinned-layer chips — Active = General + Mesh + Other reconciles even when alarms land in unmapped layers. Overview \u0026ldquo;Active alarms\u0026rdquo; widget now reads the admin\u0026rsquo;s configured defaultWindowMs from /admin/alert-page-setup; all three alarm surfaces (overview widget, alarms page, topbar badge) share one window. Every backend call failure (network throw or non-2xx) writes a pushEvent('api', 'err', …) into the debug event log with the BFF\u0026rsquo;s code / message envelope inlined when present. Dashboards with more than 40 widgets (e.g. the General/instance page, 56 widgets) now succeed: the UI splits oversize requests into ≤40-widget chunks fired in parallel, then merges results. Deployment Gateway-prefix support: BffClient.request() prepends import.meta.env.BASE_URL to every API path. Build with vite build --base=/horizon/ and a gateway that strips the prefix and the SPA + every API call resolves cleanly under the sub-path. Cluster Status route corrected from /admin/cluster → /operate/cluster (the prior default 404\u0026rsquo;d because no route by that name existed). Cleanup Documentation rewritten as an orientation map; the left-side menu is the canonical navigation now. All SWIP-* references removed from user-visible text and docs. \u0026ldquo;Coming in Phase 6 / 7\u0026rdquo; placeholder strip on Cluster Status removed. Dead code dropped — LandingView.vue, LayerTabPlaceholder.vue, the orphaned disk-write template routes (POST /api/admin/overview-templates/:id + POST /api/admin/layer-templates/:key), and stale Phase X markers across BFF + UI + docs. The OAP UTC-offset chip is gone from the topbar; the health dot stays. ","excerpt":"\u003ch1 id=\"040\"\u003e0.4.0\u003c/h1\u003e\n\u003cp\u003eOAP becomes the runtime source of truth for UI templates, the 5-theme system lands, and the …\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/changelog/0.4.0/","title":"0.4.0"},{"body":"0.4.0 OAP becomes the runtime source of truth for UI templates, the 5-theme system lands, and the app supports being served behind a gateway prefix.\nTemplates synced to OAP Five reserved template families now live on OAP\u0026rsquo;s UI-template REST surface (/ui-management/templates* on the admin port): overview dashboards, per-layer dashboards, alert page setup, theme selection, time-defaults. Bundled JSON ships as the seed + read-only fallback. One-shot seed on BFF boot pushes any missing bundled template to OAP; runtime sync is read-only with a 30-second single-flight cache. New admin endpoints: GET /api/admin/templates/sync-status, POST /api/admin/templates/save, POST /api/admin/templates/resync, POST /api/admin/templates/:name/push-bundled. When the admin port is unreachable, every admin page goes read-only with a red banner; Save / Create / Delete are disabled; render falls back to bundled. Diverged rows surface a \u0026ldquo;Show diff \u0026amp; reset\u0026rdquo; Monaco modal with a destructive-confirm (type the template key to arm reset). Themes Five bundled themes — Horizon (default), Meridian, Obsidian, Daybreak, Aurora — each shipping a complete token set (bg, fg, accent, info/ok/warn/err, font, radius, density). New /admin/global-defaults admin page replaces the old \u0026ldquo;Setup\u0026rdquo; link. Theme picker uses preview cards lifted from the design (hero strip, mini-app mockup with Primary/Tonal/Ghost buttons, KPI tiles, sparkline, density/font/radius badges). Per-user theme override via a labelled topbar chip — three-tier resolution localStorage user → OAP org default → bundled, written to \u0026lt;html data-theme\u0026gt; / \u0026lt;html data-appearance\u0026gt; synchronously on boot so the pre-auth login page already respects the local override. Sidebar SkyWalking logo swaps to the official brand blue (#1368B3) on light-appearance themes. Widget series colors (Zipkin trace palette, AlarmSnapshotChart, AlarmsTimeline) track the active theme\u0026rsquo;s --sw-accent via a shared readAccent() util. Sign-in button gradient derives both stops from the theme accent. Time defaults /admin/global-defaults also owns the global picker\u0026rsquo;s default window (60 minutes shipped). OAP step precision is derived from window size — ≤ 4 h MINUTE, 6 h–14 d HOUR, ≥ 30 d DAY — and surfaced inline on the page. Per-user override in the topbar time picker: \u0026ldquo;Save as my default\u0026rdquo; / \u0026ldquo;Reset to org default\u0026rdquo;. Reliability + diagnostics Topology cluster boundary now grows to encompass dragged nodes; the chip moved inside the cluster header so it stays visible at any drag position. Alarms page gains an Other KPI tile that surfaces the residual count between Active and the sum of pinned-layer chips — Active = General + Mesh + Other reconciles even when alarms land in unmapped layers. Overview \u0026ldquo;Active alarms\u0026rdquo; widget now reads the admin\u0026rsquo;s configured defaultWindowMs from /admin/alert-page-setup; all three alarm surfaces (overview widget, alarms page, topbar badge) share one window. Every backend call failure (network throw or non-2xx) writes a pushEvent('api', 'err', …) into the debug event log with the BFF\u0026rsquo;s code / message envelope inlined when present. Dashboards with more than 40 widgets (e.g. the General/instance page, 56 widgets) now succeed: the UI splits oversize requests into ≤40-widget chunks fired in parallel, then merges results. Deployment Gateway-prefix support: BffClient.request() prepends import.meta.env.BASE_URL to every API path. Build with vite build --base=/horizon/ and a gateway that strips the prefix and the SPA + every API call resolves cleanly under the sub-path. Cluster Status route corrected from /admin/cluster → /operate/cluster (the prior default 404\u0026rsquo;d because no route by that name existed). Cleanup Documentation rewritten as an orientation map; the left-side menu is the canonical navigation now. All SWIP-* references removed from user-visible text and docs. \u0026ldquo;Coming in Phase 6 / 7\u0026rdquo; placeholder strip on Cluster Status removed. Dead code dropped — LandingView.vue, LayerTabPlaceholder.vue, the orphaned disk-write template routes (POST /api/admin/overview-templates/:id + POST /api/admin/layer-templates/:key), and stale Phase X markers across BFF + UI + docs. The OAP UTC-offset chip is gone from the topbar; the health dot stays. ","excerpt":"\u003ch1 id=\"040\"\u003e0.4.0\u003c/h1\u003e\n\u003cp\u003eOAP becomes the runtime source of truth for UI templates, the 5-theme system lands, and the …\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/changelog/0.4.0/","title":"0.4.0"},{"body":"0.4.0 Features Add Java agent injector. Add JavaAgent and Storage CRDs of the operator. Vulnerabilities CVE-2021-3121: An issue was discovered in GoGo Protobuf before 1.3.2. plugin/unmarshal/unmarshal.go lacks certain index validation CVE-2020-29652: A nil pointer dereference in the golang.org/x/crypto/ssh component through v0.0.0-20201203163018-be400aefbc4c for Go allows remote attackers to cause a denial of service against SSH servers. Chores Bump up GO to 1.17. Bump up k8s api to 0.20.11. Polish documents. Bump up SkyWalking OAP to 8.8.1. ","excerpt":"\u003ch2 id=\"040\"\u003e0.4.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd Java agent injector.\u003c/li\u003e\n\u003cli\u003eAdd JavaAgent and Storage CRDs of the operator. …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/latest/en/changes/changes-0.4.0/","title":"0.4.0"},{"body":"0.4.0 Features Add Java agent injector. Add JavaAgent and Storage CRDs of the operator. Vulnerabilities CVE-2021-3121: An issue was discovered in GoGo Protobuf before 1.3.2. plugin/unmarshal/unmarshal.go lacks certain index validation CVE-2020-29652: A nil pointer dereference in the golang.org/x/crypto/ssh component through v0.0.0-20201203163018-be400aefbc4c for Go allows remote attackers to cause a denial of service against SSH servers. Chores Bump up GO to 1.17. Bump up k8s api to 0.20.11. Polish documents. Bump up SkyWalking OAP to 8.8.1. ","excerpt":"\u003ch2 id=\"040\"\u003e0.4.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd Java agent injector.\u003c/li\u003e\n\u003cli\u003eAdd JavaAgent and Storage CRDs of the operator. …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/next/en/changes/changes-0.4.0/","title":"0.4.0"},{"body":"0.4.0 Features Add Java agent injector. Add JavaAgent and Storage CRDs of the operator. Vulnerabilities CVE-2021-3121: An issue was discovered in GoGo Protobuf before 1.3.2. plugin/unmarshal/unmarshal.go lacks certain index validation CVE-2020-29652: A nil pointer dereference in the golang.org/x/crypto/ssh component through v0.0.0-20201203163018-be400aefbc4c for Go allows remote attackers to cause a denial of service against SSH servers. Chores Bump up GO to 1.17. Bump up k8s api to 0.20.11. Polish documents. Bump up SkyWalking OAP to 8.8.1. ","excerpt":"\u003ch2 id=\"040\"\u003e0.4.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd Java agent injector.\u003c/li\u003e\n\u003cli\u003eAdd JavaAgent and Storage CRDs of the operator. …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/v0.11.0/en/changes/changes-0.4.0/","title":"0.4.0"},{"body":"0.5.0 First Apache-style release cut from this repo: source + binary tarballs, GPG-signed and SHA-512 checksummed, with a self-contained binary that boots via node server.js and no pnpm install step. Binary distribution ships a regenerated LICENSE + NOTICE that enumerate every bundled third-party package — produced by scripts/collect-dist-licenses.mjs during packaging and validated against a deny-list before signing.\nProfiling pprof (Go) profiling is fully wired: pick one event per task (CPU / HEAP / BLOCK / MUTEX / GOROUTINE / ALLOCS / THREADCREATE), with duration shown for CPU/BLOCK/MUTEX and a sampling-rate field for BLOCK/MUTEX. Create and analyze both match OAP\u0026rsquo;s single-event pprof schema. eBPF profiling gets a reworked process picker — click a row to expand its full attributes, selection lives on the checkbox, anchored pop-out — a refresh button on every task list, Intl-formatted times, and a hover-info frame on the flame graph. Flame-graph thrash on re-analyze is gone. The shared flame graph fixes \u0026ldquo;% of root\u0026rdquo; (it read a never-aggregated count), highlights the selected frame across all four profilers, and shows a single hover card (the library\u0026rsquo;s duplicate native tooltip is suppressed). After creating any profiling task (trace / async / eBPF / network / pprof) the list now polls up to 4× at 10s until the new task shows up, instead of leaving a stale pre-create list. Network profiling \u0026amp; process topology A booster-style honeycomb process topology: pods as hexagons, peers hugging the boundary, animated protocol-coloured edges (HTTP/TCP/TLS), a node pop-over, and a wide client | server edge-metric dashboard. Network task creation and the task-list query now use OAP\u0026rsquo;s schema field names. Platform monitoring (operate) Two new read-only operate pages: Data retention (TTL — getRecordsTTL / getMetricsTTL) and OAP configuration (the admin-port config dump, with OAP-masked secrets). Gated on new ttl:read / config:read verbs granted to maintainer and above. Data retention now loads on non-BanyanDB backends too (the metadata TTL field is optional). The operate sidebar now leads with a single Platform monitoring group (cluster status, data retention, OAP configuration) above the per-layer self-observability dashboards. Dashboards \u0026amp; templates The global time picker now drives dashboards. Layer dashboards query OAP at the picker\u0026rsquo;s window and precision (MINUTE / HOUR / DAY) instead of a fixed last-hour minute window, and line charts label the x-axis with real times per step (e.g. MM-DD for a 30-day view) rather than -Nm. New table widget for label-dimensioned metrics — pod phase per service, node condition, deployment replicas, etc. — rendered as one column per label (e.g. Condition | Node) instead of a scalar card or a misleading flat line. The K8S dashboards (and kong / mongodb / elasticsearch) now use it where upstream booster-ui does; widgets that were charting a single latest(…) value as a line are now cards. The K8S Cluster view is realigned to the upstream layout (totals cards · resource lines · status tables). Edit locally, publish on your terms. Saving a dashboard/overview template now writes the local bundled copy (so the edit renders immediately for preview) and marks it diverged — nothing reaches OAP until you press Sync all to OAP, which pushes only the templates that differ, behind a confirmation listing exactly what will be written. A post-save tip spells out that the change is local-only until published. Local-vs-remote, made explicit. When local edits diverge from OAP, a per-session prompt (by menu name, not file name) asks which to render — keep my local edits (preview) or use live (which overwrites the local copy with the remote version, confirmed). The layer-templates admin page carries the same Local/Remote display toggle next to Sync all, and a Diverged only filter; each diverged layer shows a yellow warning icon in the sidebar. Traces The native trace view auto-selects OAP\u0026rsquo;s trace-query API — queryTraces (whole trace inline, BanyanDB) vs queryBasicTraces (segment list + a per-trace fetch on click, every other backend) — and a banner states which is in use; in segment-list mode the list reads \u0026ldquo;Segments\u0026rdquo; and a click loads the full trace. Span kind (Entry / Exit / Local) renders as a colored word, not a filled pill. Auth, RBAC \u0026amp; resilience Every OAP call — GraphQL, admin REST, and Zipkin — now carries the configured basic-auth credentials, so a secured OAP no longer 401s pages. The sidebar is RBAC-gated by read verb, the Roles page shows a per-role menu-visibility matrix, and the Users page labels per-node \u0026ldquo;Active (24h)\u0026rdquo; / \u0026ldquo;Last seen\u0026rdquo; honestly (these are tracked per BFF replica, not cluster-wide). Routes are verb-gated, not just menus. A user without the required read verb is bounced from a restricted page (e.g. a viewer can no longer reach Cluster Status via the topbar OAP chip or a direct URL); the chip only links there for cluster:read. This sits on top of the existing per-route BFF verb enforcement. LDAP resolves group membership with the service account, not the logging-in user — directories that hide the group subtree from ordinary users no longer collapse every login to the fallback role. When OAP is unreachable the menu and admin loaders fall back to bundled templates, and non-JSON OAP responses surface a clear diagnostic. Smaller touches Top-N widgets get hover tooltips for long names and a title-bar pop-out to the full ranked list; redundant single-service name prefixes are dropped. The admin template-diff modal is a wide side-by-side view with labelled bundled-vs-OAP columns and an explanation of what the template drives; the layer-dashboards admin rail gains an in-page search. Per-layer alarm filtering uses the singular queryAlarms layer condition. Dependency hygiene for the release: dompurify ≥ 3.3.2 and @fastify/static ≥ 9.1.1 (clears the known advisories); the general layer drops networkProfiling, which is instance-scoped to k8s / mesh. ","excerpt":"\u003ch1 id=\"050\"\u003e0.5.0\u003c/h1\u003e\n\u003cp\u003eFirst Apache-style release cut from this repo: source + binary tarballs, GPG-signed and …\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/changelog/0.5.0/","title":"0.5.0"},{"body":"0.5.0 First Apache-style release cut from this repo: source + binary tarballs, GPG-signed and SHA-512 checksummed, with a self-contained binary that boots via node server.js and no pnpm install step. Binary distribution ships a regenerated LICENSE + NOTICE that enumerate every bundled third-party package — produced by scripts/collect-dist-licenses.mjs during packaging and validated against a deny-list before signing.\nProfiling pprof (Go) profiling is fully wired: pick one event per task (CPU / HEAP / BLOCK / MUTEX / GOROUTINE / ALLOCS / THREADCREATE), with duration shown for CPU/BLOCK/MUTEX and a sampling-rate field for BLOCK/MUTEX. Create and analyze both match OAP\u0026rsquo;s single-event pprof schema. eBPF profiling gets a reworked process picker — click a row to expand its full attributes, selection lives on the checkbox, anchored pop-out — a refresh button on every task list, Intl-formatted times, and a hover-info frame on the flame graph. Flame-graph thrash on re-analyze is gone. The shared flame graph fixes \u0026ldquo;% of root\u0026rdquo; (it read a never-aggregated count), highlights the selected frame across all four profilers, and shows a single hover card (the library\u0026rsquo;s duplicate native tooltip is suppressed). After creating any profiling task (trace / async / eBPF / network / pprof) the list now polls up to 4× at 10s until the new task shows up, instead of leaving a stale pre-create list. Network profiling \u0026amp; process topology A booster-style honeycomb process topology: pods as hexagons, peers hugging the boundary, animated protocol-coloured edges (HTTP/TCP/TLS), a node pop-over, and a wide client | server edge-metric dashboard. Network task creation and the task-list query now use OAP\u0026rsquo;s schema field names. Platform monitoring (operate) Two new read-only operate pages: Data retention (TTL — getRecordsTTL / getMetricsTTL) and OAP configuration (the admin-port config dump, with OAP-masked secrets). Gated on new ttl:read / config:read verbs granted to maintainer and above. Data retention now loads on non-BanyanDB backends too (the metadata TTL field is optional). The operate sidebar now leads with a single Platform monitoring group (cluster status, data retention, OAP configuration) above the per-layer self-observability dashboards. Dashboards \u0026amp; templates The global time picker now drives dashboards. Layer dashboards query OAP at the picker\u0026rsquo;s window and precision (MINUTE / HOUR / DAY) instead of a fixed last-hour minute window, and line charts label the x-axis with real times per step (e.g. MM-DD for a 30-day view) rather than -Nm. New table widget for label-dimensioned metrics — pod phase per service, node condition, deployment replicas, etc. — rendered as one column per label (e.g. Condition | Node) instead of a scalar card or a misleading flat line. The K8S dashboards (and kong / mongodb / elasticsearch) now use it where upstream booster-ui does; widgets that were charting a single latest(…) value as a line are now cards. The K8S Cluster view is realigned to the upstream layout (totals cards · resource lines · status tables). Edit locally, publish on your terms. Saving a dashboard/overview template now writes the local bundled copy (so the edit renders immediately for preview) and marks it diverged — nothing reaches OAP until you press Sync all to OAP, which pushes only the templates that differ, behind a confirmation listing exactly what will be written. A post-save tip spells out that the change is local-only until published. Local-vs-remote, made explicit. When local edits diverge from OAP, a per-session prompt (by menu name, not file name) asks which to render — keep my local edits (preview) or use live (which overwrites the local copy with the remote version, confirmed). The layer-templates admin page carries the same Local/Remote display toggle next to Sync all, and a Diverged only filter; each diverged layer shows a yellow warning icon in the sidebar. Traces The native trace view auto-selects OAP\u0026rsquo;s trace-query API — queryTraces (whole trace inline, BanyanDB) vs queryBasicTraces (segment list + a per-trace fetch on click, every other backend) — and a banner states which is in use; in segment-list mode the list reads \u0026ldquo;Segments\u0026rdquo; and a click loads the full trace. Span kind (Entry / Exit / Local) renders as a colored word, not a filled pill. Auth, RBAC \u0026amp; resilience Every OAP call — GraphQL, admin REST, and Zipkin — now carries the configured basic-auth credentials, so a secured OAP no longer 401s pages. The sidebar is RBAC-gated by read verb, the Roles page shows a per-role menu-visibility matrix, and the Users page labels per-node \u0026ldquo;Active (24h)\u0026rdquo; / \u0026ldquo;Last seen\u0026rdquo; honestly (these are tracked per BFF replica, not cluster-wide). Routes are verb-gated, not just menus. A user without the required read verb is bounced from a restricted page (e.g. a viewer can no longer reach Cluster Status via the topbar OAP chip or a direct URL); the chip only links there for cluster:read. This sits on top of the existing per-route BFF verb enforcement. LDAP resolves group membership with the service account, not the logging-in user — directories that hide the group subtree from ordinary users no longer collapse every login to the fallback role. When OAP is unreachable the menu and admin loaders fall back to bundled templates, and non-JSON OAP responses surface a clear diagnostic. Smaller touches Top-N widgets get hover tooltips for long names and a title-bar pop-out to the full ranked list; redundant single-service name prefixes are dropped. The admin template-diff modal is a wide side-by-side view with labelled bundled-vs-OAP columns and an explanation of what the template drives; the layer-dashboards admin rail gains an in-page search. Per-layer alarm filtering uses the singular queryAlarms layer condition. Dependency hygiene for the release: dompurify ≥ 3.3.2 and @fastify/static ≥ 9.1.1 (clears the known advisories); the general layer drops networkProfiling, which is instance-scoped to k8s / mesh. ","excerpt":"\u003ch1 id=\"050\"\u003e0.5.0\u003c/h1\u003e\n\u003cp\u003eFirst Apache-style release cut from this repo: source + binary tarballs, GPG-signed and …\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/changelog/0.5.0/","title":"0.5.0"},{"body":"0.5.0 First Apache-style release cut from this repo: source + binary tarballs, GPG-signed and SHA-512 checksummed, with a self-contained binary that boots via node server.js and no pnpm install step. Binary distribution ships a regenerated LICENSE + NOTICE that enumerate every bundled third-party package — produced by scripts/collect-dist-licenses.mjs during packaging and validated against a deny-list before signing.\nProfiling pprof (Go) profiling is fully wired: pick one event per task (CPU / HEAP / BLOCK / MUTEX / GOROUTINE / ALLOCS / THREADCREATE), with duration shown for CPU/BLOCK/MUTEX and a sampling-rate field for BLOCK/MUTEX. Create and analyze both match OAP\u0026rsquo;s single-event pprof schema. eBPF profiling gets a reworked process picker — click a row to expand its full attributes, selection lives on the checkbox, anchored pop-out — a refresh button on every task list, Intl-formatted times, and a hover-info frame on the flame graph. Flame-graph thrash on re-analyze is gone. The shared flame graph fixes \u0026ldquo;% of root\u0026rdquo; (it read a never-aggregated count), highlights the selected frame across all four profilers, and shows a single hover card (the library\u0026rsquo;s duplicate native tooltip is suppressed). After creating any profiling task (trace / async / eBPF / network / pprof) the list now polls up to 4× at 10s until the new task shows up, instead of leaving a stale pre-create list. Network profiling \u0026amp; process topology A booster-style honeycomb process topology: pods as hexagons, peers hugging the boundary, animated protocol-coloured edges (HTTP/TCP/TLS), a node pop-over, and a wide client | server edge-metric dashboard. Network task creation and the task-list query now use OAP\u0026rsquo;s schema field names. Platform monitoring (operate) Two new read-only operate pages: Data retention (TTL — getRecordsTTL / getMetricsTTL) and OAP configuration (the admin-port config dump, with OAP-masked secrets). Gated on new ttl:read / config:read verbs granted to maintainer and above. Data retention now loads on non-BanyanDB backends too (the metadata TTL field is optional). The operate sidebar now leads with a single Platform monitoring group (cluster status, data retention, OAP configuration) above the per-layer self-observability dashboards. Dashboards \u0026amp; templates The global time picker now drives dashboards. Layer dashboards query OAP at the picker\u0026rsquo;s window and precision (MINUTE / HOUR / DAY) instead of a fixed last-hour minute window, and line charts label the x-axis with real times per step (e.g. MM-DD for a 30-day view) rather than -Nm. New table widget for label-dimensioned metrics — pod phase per service, node condition, deployment replicas, etc. — rendered as one column per label (e.g. Condition | Node) instead of a scalar card or a misleading flat line. The K8S dashboards (and kong / mongodb / elasticsearch) now use it where upstream booster-ui does; widgets that were charting a single latest(…) value as a line are now cards. The K8S Cluster view is realigned to the upstream layout (totals cards · resource lines · status tables). Edit locally, publish on your terms. Saving a dashboard/overview template now writes the local bundled copy (so the edit renders immediately for preview) and marks it diverged — nothing reaches OAP until you press Sync all to OAP, which pushes only the templates that differ, behind a confirmation listing exactly what will be written. A post-save tip spells out that the change is local-only until published. Local-vs-remote, made explicit. When local edits diverge from OAP, a per-session prompt (by menu name, not file name) asks which to render — keep my local edits (preview) or use live (which overwrites the local copy with the remote version, confirmed). The layer-templates admin page carries the same Local/Remote display toggle next to Sync all, and a Diverged only filter; each diverged layer shows a yellow warning icon in the sidebar. Traces The native trace view auto-selects OAP\u0026rsquo;s trace-query API — queryTraces (whole trace inline, BanyanDB) vs queryBasicTraces (segment list + a per-trace fetch on click, every other backend) — and a banner states which is in use; in segment-list mode the list reads \u0026ldquo;Segments\u0026rdquo; and a click loads the full trace. Span kind (Entry / Exit / Local) renders as a colored word, not a filled pill. Auth, RBAC \u0026amp; resilience Every OAP call — GraphQL, admin REST, and Zipkin — now carries the configured basic-auth credentials, so a secured OAP no longer 401s pages. The sidebar is RBAC-gated by read verb, the Roles page shows a per-role menu-visibility matrix, and the Users page labels per-node \u0026ldquo;Active (24h)\u0026rdquo; / \u0026ldquo;Last seen\u0026rdquo; honestly (these are tracked per BFF replica, not cluster-wide). Routes are verb-gated, not just menus. A user without the required read verb is bounced from a restricted page (e.g. a viewer can no longer reach Cluster Status via the topbar OAP chip or a direct URL); the chip only links there for cluster:read. This sits on top of the existing per-route BFF verb enforcement. LDAP resolves group membership with the service account, not the logging-in user — directories that hide the group subtree from ordinary users no longer collapse every login to the fallback role. When OAP is unreachable the menu and admin loaders fall back to bundled templates, and non-JSON OAP responses surface a clear diagnostic. Smaller touches Top-N widgets get hover tooltips for long names and a title-bar pop-out to the full ranked list; redundant single-service name prefixes are dropped. The admin template-diff modal is a wide side-by-side view with labelled bundled-vs-OAP columns and an explanation of what the template drives; the layer-dashboards admin rail gains an in-page search. Per-layer alarm filtering uses the singular queryAlarms layer condition. Dependency hygiene for the release: dompurify ≥ 3.3.2 and @fastify/static ≥ 9.1.1 (clears the known advisories); the general layer drops networkProfiling, which is instance-scoped to k8s / mesh. ","excerpt":"\u003ch1 id=\"050\"\u003e0.5.0\u003c/h1\u003e\n\u003cp\u003eFirst Apache-style release cut from this repo: source + binary tarballs, GPG-signed and …\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/changelog/0.5.0/","title":"0.5.0"},{"body":"0.5.0 Features Add E2E test cases to verify OAPServer, UI, Java agent and Storage components. Add the Satellite component. Bugs Fix operator role patch issues Fix invalid CSR signername Fix bug in the configmap controller Chores Bump up KubeBuilder to V3 Bump up metric adapter server to v1.21.0 Split mono-project to two independent projects ","excerpt":"\u003ch2 id=\"050\"\u003e0.5.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd E2E test cases to verify OAPServer, UI, Java agent and Storage components.\u003c/li\u003e\n\u003cli\u003eAdd …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/latest/en/changes/changes-0.5.0/","title":"0.5.0"},{"body":"0.5.0 Features Add E2E test cases to verify OAPServer, UI, Java agent and Storage components. Add the Satellite component. Bugs Fix operator role patch issues Fix invalid CSR signername Fix bug in the configmap controller Chores Bump up KubeBuilder to V3 Bump up metric adapter server to v1.21.0 Split mono-project to two independent projects ","excerpt":"\u003ch2 id=\"050\"\u003e0.5.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd E2E test cases to verify OAPServer, UI, Java agent and Storage components.\u003c/li\u003e\n\u003cli\u003eAdd …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/next/en/changes/changes-0.5.0/","title":"0.5.0"},{"body":"0.5.0 Features Add E2E test cases to verify OAPServer, UI, Java agent and Storage components. Add the Satellite component. Bugs Fix operator role patch issues Fix invalid CSR signername Fix bug in the configmap controller Chores Bump up KubeBuilder to V3 Bump up metric adapter server to v1.21.0 Split mono-project to two independent projects ","excerpt":"\u003ch2 id=\"050\"\u003e0.5.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd E2E test cases to verify OAPServer, UI, Java agent and Storage components.\u003c/li\u003e\n\u003cli\u003eAdd …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/v0.11.0/en/changes/changes-0.5.0/","title":"0.5.0"},{"body":"0.6.0 This release is the production-readiness pass for Horizon UI: every page now renders correctly across the eight supported languages on non-UTC OAP deployments, with deliberate caps and validation on the load surfaces that operators reach. The pillars below describe the operator-visible result.\nEight-locale internationalization Horizon now ships with eight first-class UI languages — English (source) plus zh-CN, ja, ko, es, pt, de, fr — selectable from the top-bar locale chip on every page (including the pre-auth login). The choice persists per device.\nUI chrome. Every routed page and every shared sub-component renders through vue-i18n; non-English locales now cover every admin page (Roles, Users, Auth status, Alert page setup, Global defaults, 3D-map config), every operate page (Alerting rules, DSL catalog / editor / dump, OAL catalog, Live debugger + MAL / LAL / OAL, Capture history, Metrics inspect, OAP config, TTL), the alarms surface, and the shared modals. Long lede paragraphs that previously rendered as English | one translated word | English mid-sentence are now single translation units — inline \u0026lt;code\u0026gt; and links interpolate without splitting the prose. Missing leaves still fall back to English so partial catalogs degrade invisibly. BFF-shipped templates. All 42 layer dashboards and both overview dashboards carry per-locale overlay catalogs alongside the source template. Coverage is ~2,300 translatable leaves per non-English locale across the layer set. The BFF picks the locale from the request\u0026rsquo;s X-Horizon-Locale header (auto-set by the SPA), merges the overlay onto the source, and serves the localised template to the renderer — translation resolves once on the BFF, never on every chart mount. Operator-runnable Translations page. A new admin surface (Dashboard setup → Translations) edits the per-locale overlays through the live preview: pick a target language, click any widget in the rendered dashboard, type the translation. Per-locale status chips on the template picker show at a glance which dashboards have drafts, which are synced, which diverge from disk, and which are empty for a given locale. Push writes the sibling overlay row on OAP; pushing zh-CN never touches ja. Tech-term policy. Product, project, and protocol names (SkyWalking, Kubernetes, OAP, MQE, eBPF, Zipkin, OpenTelemetry, Istio, GraphQL, etc.), OAP scope enums (Service, ServiceInstance, Endpoint, Process), layer keys, MQE function names, env vars, HTTP status codes, and per-language runtimes (JVM, Go, Python, …) stay verbatim in every locale per CLAUDE.md. Phrases containing tech terms are translated around the term (HTTP Connections → HTTP 连接 / HTTP 接続), not transliterated. OAP-supplied data is never translated. Service names, alarm rule names, trace span operation names, log messages — anything arriving over the OAP wire — render verbatim regardless of locale. Validator gate. i18n:validate is stricter: every source template must have a sibling overlay file per advertised locale, and empty {} overlays are now a finding (used to pass silently — surfaced as \u0026ldquo;structurally complete\u0026rdquo; while every translatable string still rendered in English). Typography + self-hosted fonts Inter + JetBrains Mono are now self-hosted. The Google Fonts CDN dependency is gone — air-gapped or firewalled deployments render the intended typography instead of silently falling back to system fonts. One typescale across every page. Older admin pages that drifted to a mixed pixel palette (9.5 / 10 / 10.5 / 11 / 11.5 / 12 / 14 / 18 / 20 / 22) now share the same six-step scale + uppercase-label vocabulary as the newer dashboards. Sidebar, kpi labels, table headers, kickers all line up. Wire-correctness on non-UTC OAP Every BFF query route now spells Duration.start / end in the OAP server\u0026rsquo;s timezone (probed once per minute, cached). Previously only the alarms route did this; dashboards / landing / topology / endpoint / endpoint-dependency / instance / eBPF / traces / logs / trace-tag all emitted UTC, which silently shifted every query on non-UTC OAP installs by the server\u0026rsquo;s offset.\nTraces and logs additionally query at SECOND precision now (records, not metric buckets) — a trace that just finished falls inside the window instead of getting rounded off the MINUTE boundary.\nPerformance hardening Landing batches no longer 5xx on wide layers. The per-layer landing route used to build one GraphQL with up to 250 aliased fragments (25 services × 10 metric columns) and trip OAP\u0026rsquo;s per-request complexity ceiling, blanking every cell. Chunks at 6 services per round-trip and fires them in parallel — same pattern the dashboard route already uses. Trace waterfall opens fast on huge traces. Rows render lazily via the browser\u0026rsquo;s content-visibility window — a 5000-span trace no longer freezes the main thread on open. Backgrounded tabs stop polling. The shared auto-refresh ticker pauses when the tab is hidden and resumes (with one immediate tick) on return. An unattended browser no longer streams queries at the topbar interval × every subscribed widget. RBAC + input-validation hardening /api/health no longer leaks the active session count to unauthenticated callers — the public liveness probe returns only status + version. The authenticated /api/auth/health surface still carries detail. pageSize capped server-side on every trace / log route (trace 200, log 100). OAP forwards paging.pageSize straight to the storage LIMIT, so a client posting pageSize: 50000 previously cascaded the load to OAP. The UI picker\u0026rsquo;s matching cap is now defended at the BFF boundary too. Profiling task bodies validated. Async-profiler, pprof, eBPF fixed-task, and network-profiling create routes now sanitize and bound their bodies — duration caps, target-instance and event-list caps, payload-size clamps. Closes a DoS vector where a user with profile:enable could submit a multi-hour profile that pegs the target instance\u0026rsquo;s CPU. Diff modal console error fixed The four admin \u0026ldquo;Check diff \u0026amp; push\u0026rdquo; modals (Layer dashboards, Overview templates, Translations, the shared TemplateDiffModal) used to log Uncaught (in promise) Error: Missing requestHandler or method: resetSchema (two per modal — one per diff pane) the moment the operator opened them. Monaco\u0026rsquo;s JSON language service was sending the message to a worker that didn\u0026rsquo;t know how to handle it. Now wired correctly — the diff itself rendered all along, but the console is quiet again.\nDSL / OAL catalog renames + admin editor seeding Catalog pages spell out the language name in the header: Metrics Analysis Language - OpenTelemetry Rules (and … - Telegraf Rules / … - Log MAL Rules); LAL renders as Log Analysis Language; the OAL browse as Observability Analysis Language. The sidebar keeps the abbreviated MAL · OTEL form for space. Layer Dashboards + Overview Templates editor opens REMOTE on diverged rows. The runtime menu already renders remote content when the row is diverged, but the editor used to seed from bundled — so operators were editing a copy that silently disagreed with what end users saw. Priority is now local → remote (diverged / remote-only) → bundled; the source pill on both editors reads from local / from remote / from bundled consistently. Rule card arrow stays next to the rule name. Short names like default / mesh-dp / vm used to float in the middle of the card with a big gap from the ▶ run arrow — the arrow + name are now grouped on the left, with the status pill alone pushed right. Sync-status counters don\u0026rsquo;t include translation rows Per-locale overlay rows on OAP share their parent template\u0026rsquo;s kind, so they were inflating the remote-only and diverged counts on the Overview Templates and Layer Dashboards admin pages (the banners would read 14 remote-only / 294 remote-only when most of those entries were translation rows that belong on the Translations page only). Filtered at bundle assembly so the sync-status banners count source rows only.\nCluster Status + admin polish Cluster Status — Pane B + Pane C fully translatable. Module column headers, gate descriptions (the SWIP-13 affects strings), per-row enabled/missing badges, the admin-host status badge (loading… / unreachable / all selectors on / {n} selectors off), the admin-host-unreachable hint, the Zipkin / OTLP pane lede, the Endpoint card heading, the Zipkin badge and the Zipkin-unreachable hint all now render in the active locale. Hide redundant BUNDLED chrome when synced. On the Overview templates and Layer dashboards admin pages, when the row\u0026rsquo;s sync badge is synced the BUNDLED and REMOTE versions are byte-equal, so the from {source} pill and the Reset to / Preview Bundled dropdown items add no information and are hidden. LOCAL drafts always show; BUNDLED resurfaces the moment the row diverges. Translations picker prefers the English bundle. REMOTE-only rows with non-English titles (legacy duplicates from prior import cycles) no longer appear as separate dashboards in the picker — the picker lists the canonical English bundled dashboards once each, and the preview renders the English source as the baseline. Alerting rules — running entities show their OAP node The Operate › Alerting rules detail pane\u0026rsquo;s Currently watching list now spans the whole cluster and tags each entity with the OAP node evaluating it. Each OAP instance evaluates a rule independently over the slice of entities it holds, so the watched set is the union across nodes — the page previously showed only the first responding node\u0026rsquo;s entities, which misread as \u0026ldquo;these are all the entities the rule watches.\u0026rdquo; The list now aggregates every instance\u0026rsquo;s entities and labels each row with its node (e.g. SERVICE agent::app NODE 10.116.3.26_11800), with the per-entity alarm message on hover. The per-node load-state table is unchanged. Single-instance deployments simply show one node label per row.\nClicking a watched entity now opens a running-context popup — the live evaluation window the rule is computing for that entity, per OAP node. It shows the current state (FIRING / SILENCED_FIRING / RECOVERY_OBSERVATION), the window size and silence / recovery countdowns, the window end, the last-alarm time and message, and the per-metric snapshot the expression was evaluated against — rendered as a sparkline plus per-bucket values so an operator can see exactly why a rule is (or isn\u0026rsquo;t) firing. Nodes not evaluating the entity are marked as such, and a raw-JSON disclosure carries the full payload.\nLive debugger fixes A clutch of small but visible bugs were caught while exercising the i18n surfaces:\nHistory → debug deep-link rendered blank. MAL and LAL views crashed silently on ?historyId=… because loadHistorical reset refs (selectedRow, expandedEntities, foldedRecords; selectedCell for LAL) that the file declared further down. The watch(historyId, …, { immediate: true }) fires during setup, so the TDZ ReferenceError aborted setup before the page rendered, with no console trace. Resettable refs are now hoisted above their consumer. Captured records wiped on stop. After stop(), the per-DSL view did one final session() refresh against an already-cleaned- up OAP, which returned nodes-only-with-empty-records and overwrote the rich live snapshot in localStorage. save() now refuses to shrink an existing entry\u0026rsquo;s recordCount for the same sessionId — only metadata (retentionDeadline, retentionMillis) updates. Stable node ordering. MAL / LAL / OAL node cards now sort by nodeId ?? peer so they don\u0026rsquo;t reshuffle between polls. Tab buttons jumped nowhere. LiveDebuggerView.selectTab pushed /debug/\u0026lt;tab\u0026gt; (a path that doesn\u0026rsquo;t exist); fixed to /operate/live-debug/\u0026lt;tab\u0026gt;. Same correction in DebugHistoryView.loadEntry deep-links and surrounding doc-comments in RuleCard.vue / DslEditorView.vue. Empty-capture placeholder. When a saved session has zero populated nodes, DebugView now shows an explicit \u0026ldquo;This capture has no records\u0026rdquo; rather than rendering blank, so an honest empty capture is visibly different from a bug. Per-locale lookup widened. Per-DSL views look up the historyId via history.all (not the widget-filtered history.entries); the route already pinned us to the right widget, and double-filtering by widget was a silent way for a stale field to drop the entry. Other small fixes MAL / LAL editor gutter glyph. The green ▶ live-debug entrance on every - name: row was referencing an unstyled CSS class — Monaco reserved the gutter and wired clicks, but the icon was invisible. Restored. Rule catalog cards. Duplicate BUNDLED pill removed; the header status pill already conveyed it. K8s instance Node Status. Switched from a single-scalar card (rendering just 1) to a table of currently-true Kubernetes conditions, matching the cluster-scope sibling and aligning widget heights. Upstream ui-management compatibility Aligned with upstream skywalking#13884 (\u0026ldquo;remove auto generate id for UI templates\u0026rdquo;): Horizon now sends id = \u0026lt;envelope name\u0026gt; on POST /ui-management/templates. Current OAP requires it (POSTs without id are rejected); legacy OAP releases ignored the field and auto-generated UUIDs, so the same payload works against both. Horizon already treated r.id as opaque; mixed-id deployments self-heal via reconcileDuplicates on the next boot.\nSmartscape service hierarchy OAP 10\u0026rsquo;s cross-layer service hierarchy is now reachable from any layer\u0026rsquo;s service map — a logical service projected across observation layers (GENERAL agent ↔ MESH sidecar ↔ MESH_DP data-plane ↔ K8S_SERVICE pod) is one click away on every selected hex.\nLazy-probed chip on the selected hex. Picking a node fires one getServiceHierarchy call; if the service has cross-layer peers, a small chevron-stack chip clips to the hex\u0026rsquo;s right edge. No probe, no chip on services with no peers. Focus + context + suggestions overlay. Click the chip and the topology dims under a transparent canvas; the focused hex re-renders bright at the exact same screen position and scale as the underlying hex (the topology\u0026rsquo;s d3 zoom transform is mirrored onto the overlay). Peers fan vertically from the focus column using OAP\u0026rsquo;s listLayerLevels order — higher-level (request-near) layers above, lower-level (infra-near) layers below, matching booster-ui\u0026rsquo;s hierarchy rendering rule. Auto-refresh pauses while the overlay is open so the background topology and KPI panels don\u0026rsquo;t shift under the operator. Closing the overlay (× button, ESC, or click-on-dim) resumes the ticker and fires one immediate tick so the page snaps back to live data. Two-step peer open. First click on a peer hex arms it (selection halo + side ↗ Open in \u0026lt;Layer\u0026gt; action chip); second click on the chip opens the destination layer in a new browser tab, pre-selecting the peer service. Peers in layers Horizon has no template for render dimmed with a cursor: not-allowed; clicking them logs \u0026ldquo;No layer template configured for \u0026lt;Layer\u0026gt;\u0026rdquo; to the event log instead. URL-pinned service validator on the destination tab. Every per-layer page now validates the URL-hydrated ?service=\u0026lt;id\u0026gt; against the layer\u0026rsquo;s real service roster (the new GET /api/layer/:key/services, served from the BFF\u0026rsquo;s 60s catalog cache so it adds no extra OAP traffic). A genuinely missing id pops a Service not found in this layer modal with a one-click fallback to the first available service; a valid id is trusted even when landing\u0026rsquo;s top-N rollup doesn\u0026rsquo;t sample it (the cause of the previous silent service-swap on low-traffic deep links). Service-name resolution on the layer dashboard now consults the roster after landing\u0026rsquo;s top-N, so deep links to low-traffic services no longer sit on \u0026ldquo;Resolving service…\u0026rdquo; forever waiting for a row that won\u0026rsquo;t arrive. On-demand pod logs (live tail) A new per-layer Pod Logs tab live-tails a Kubernetes pod\u0026rsquo;s container logs, pulled on demand from the K8s API through OAP and never persisted.\nInstance-pinned tail. Pick a pod, pick one of its containers, press Start; the trailing window (30s / 1m / 5m / 15m / 30m) streams into a read-only log pane and refreshes on a chosen interval (2s / 5s / 10s / 30s) until paused. A header strip shows the container, line count, a live dot, and \u0026ldquo;updated Ns ago\u0026rdquo;. Include / Exclude filtering forwards to OAP\u0026rsquo;s content keyword filters — full-line regex, so a substring match reads .*error.*. Enabled on the Kubernetes-deployed layers — Kubernetes Services (K8S_SERVICE), Istio Managed Services (MESH), and Istio Data Plane (MESH_DP) — whose service instances resolve to a pod. The tab is gated by a new podLogs component flag added to those bundled layer templates; an existing OAP whose stored template predates the flag still gets the tab, because the flag is back-filled from the bundled default (no re-push needed). The page owns its own refresh — the global auto-refresh ticker and the topbar time picker are paused while on it, the same as Traces / Logs. When the selected instance carries no pod metadata (or the pod has rotated away), OAP\u0026rsquo;s reason is shown verbatim with a hint to pick a currently-running pod or enable the feature on OAP. BanyanDB cold-stage query The cold lifecycle stage is now reachable from the UI on BanyanDB deployments — operators can query data that has aged past the hot + warm window without leaving the page.\nTopbar Cold pill appears only when the connected OAP is BanyanDB. Toggling it switches every page to read from the cold stage instead of hot + warm — it replaces the read, it does NOT union the two stages, so the pill label flips to Cold only while on. The choice is sticky per browser and re-runs every visible query so what you see matches the new mode immediately. Cold-trap banner. When the pill is on AND the current time range is within the hot + warm window, a yellow strip appears under the topbar: \u0026ldquo;Cold-only read is active — your time range is within the last N d (hot + warm), where the cold stage returns nothing.\u0026rdquo; A one-click Turn Cold off sits on the right. Trace lookup from a log row now passes the row\u0026rsquo;s timestamp through the popout so the trace lookup spans a window around that timestamp instead of OAP\u0026rsquo;s default last-1-day search — paired with the Cold pill, a trace that lives in cold resolves from a cold-era log row instead of silently failing to load. Data retention page Per-data-class data lifecycle bar. One row per data class (Normal / Trace / Zipkin trace / Log / Browser error log / Metadata / Minute metric / Hour metric / Day metric) with proportional Hot+Warm and (when configured) Cold segments. Widths are proportional to total retention across all rows, so a class retained longer visibly stretches further than its peers. When every class in a category shares the same TTL pair, the rows collapse to All records (5) / All metrics (4) — the page never renders nine identical bars. The page branches sharply by backend. BanyanDB shows the full lifecycle bar + stage vocabulary; on any other backend, the page renders a single Retention pane with per-class values and skips the stage vocabulary entirely. A footer note names the wire-level truth: OAP\u0026rsquo;s TTL response collapses hot + warm into one number per class, BanyanDB migrates between stages in segments so records near a boundary may briefly exist in both, and property data is omitted (forever-retained, no TTL reported). Time picker Custom range seeds from the last applied range when you re-open the picker, instead of resetting to \u0026ldquo;half the max ending now\u0026rdquo;. Reopening also auto-expands the Custom form on the matching precision tab when the current range is custom. Locale-bleed fix on the alarms page custom-range stamp and the log row date column (was rendering 5月08日 on zh-CN browsers; now uniform MM-DD). Overview widgets follow the global time picker The Services Dashboard (and any overview using metric / KPI / table widgets) now honors the topbar time picker. Previously the per-layer landing and topology routes were hardcoded to the last 60 minutes, so picking 12 days back kept showing recent numbers; now picker + Cold pill flow end-to-end. The layer dashboard\u0026rsquo;s header KPIs follow the picker too (was showing live numbers while the body honored the picker). The Active alarms widget title now shows the actual window (e.g. · last 10m) and the empty-state copy uses the same value instead of a hardcoded \u0026ldquo;last 60m\u0026rdquo;. Polish Sentence-case fixes on a couple of leftover Title-Case labels (DSL management, Metrics inspect) so the menu and roles tables read consistently; acronyms (DSL / OAP / MAL / LAL / OAL) stay uppercase. Public-demo (demo.skywalking.apache.org) references removed from setup docs — the demo doesn\u0026rsquo;t accept anonymous traffic. Dashboard authoring The Layer dashboards and Overview templates admin pages now share one editing model where your work-in-progress lives in your browser, and the live, shared version is whatever OAP serves.\nEdits are local to your browser. \u0026ldquo;Save (local)\u0026rdquo; stores your draft in this browser only — it is never written to the server and nobody else sees it. Everyone (including you, in normal viewing) keeps seeing the published remote dashboard until you publish. Load any source into the editor. A single Reset to ▾ control loads the Bundled (shipped default) or Remote (OAP live) version into the editor; editing from there becomes your local draft. Preview any source on the real page. A Preview ▾ control opens the actual layer / overview page in a new tab rendering your Local draft, the Bundled default, or Remote — via ?mode=preview\u0026amp;source=…, which stays in the URL and propagates as you navigate the menu. A banner names what you\u0026rsquo;re previewing (dismiss with × or Esc). Publish with a diff. Check diff \u0026amp; push shows a side-by-side local→remote diff and publishes to OAP; it\u0026rsquo;s enabled only when your local draft actually differs from remote. Bundled can also be pushed straight to OAP. Resetting to remote clears the local draft. Reset to bundled then publishes correctly when the bundled default differs from remote — Save (local) and Check diff \u0026amp; push now compare the editor against remote, not just against what was first loaded, so a bundled-vs-remote divergence is no longer mistaken for \u0026ldquo;no changes\u0026rdquo; (layer + overview editors). Preview faithfully reflects your draft\u0026rsquo;s enabled components / menu labels — disabled tabs disappear and renamed nouns (\u0026ldquo;Nodes\u0026rdquo;, \u0026ldquo;Topics\u0026rdquo;) show through — without pushing anything to the server. Preview works even for layers OAP currently reports no services for. An editors-only reminder lists any unpublished local drafts with quick links to the relevant edit page (no more \u0026ldquo;use local vs remote\u0026rdquo; prompt — remote is always the live source). Create mirrors edit. \u0026ldquo;+ New dashboard\u0026rdquo; writes a local draft (the id is the template name, checked unique) — edit and preview it, then Check diff \u0026amp; push publishes it. A pushed dashboard with no bundled default is remote-only and now renders everywhere (live page + sidebar), not just in the editor. Delete = soft-disable (OAP has no hard delete). A local-only draft is removed from the browser; a dashboard on OAP is disabled — dropped from the picker\u0026rsquo;s live state, the sidebar, and the live page. A disabled status chip shows it. Confirmations are styled in-app dialogs, not the browser\u0026rsquo;s native box. Layer dashboards editor The layer picker is a single filterable dropdown showing alias + key + sync status. A live menu preview sits beside the Alias / Components / Menu labels (per-layer slot aliases) editor; clicking a menu item jumps to that component\u0026rsquo;s config. Scope tabs and section headings read in the layer\u0026rsquo;s own vocabulary. The service-list metrics editor gains a sample-data preview (plus a faithful landing KPI tile preview that reuses the real header components), the column remove-button alignment is fixed, and the landing KPI tile config makes clear it just picks which existing column feeds the headline + sparkline. The picker\u0026rsquo;s Diverged / Local filters now sit inside the dropdown (next to the search), and the editor header reads on one line — Layer: \u0026lt;name\u0026gt; \u0026lt;key\u0026gt; \u0026lt;status\u0026gt; — showing the same sync-status chip the picker does. Disable / Reactivate a layer. Disabling soft-disables the layer on OAP and drops it from the sidebar (the menu honors disabled templates); a disabled layer offers Reactivate, which re-enables it from the bundled default (the OAP update path clears the disabled flag). Overview templates editor Rebuilt as a layer-style canvas: a 12-column grid you drag to reorder and corner-drag to resize, click-to-edit in a right drawer that appears on selection (Esc / deselect hides it), with section-breaks and the dashboard title selectable and unselected widgets hinted by a dashed outline. The canvas mirrors the live grid (fixed row height), so the layout matches the real page — including side-by-side widgets like topology + alarms. The composite-metrics KPI editor stacks each row as a card (label / source / MQE / unit / aggr / style / max) so nothing truncates in the narrow drawer. Navigation \u0026amp; shell The main sidebar folds to a narrow rail to reclaim width; the logo moves into the topbar while folded (the original wordmark is unchanged). The active menu item now reliably expands, highlights, and scrolls into view on entering a page (route matching is case-insensitive). Fixed a regression where the sidebar scrolled to the very bottom on every navigation (the \u0026ldquo;Debug events\u0026rdquo; toggle\u0026rsquo;s active state was being treated as the scroll target). Overview dashboards appear in the sidebar only when their layers are reporting services. Visibility is derived from each dashboard\u0026rsquo;s widgets (their layer field) ∪ the explicit layers[] list, gated against the live availableLayers. A dashboard you create via \u0026ldquo;+ New\u0026rdquo; inherits this automatically — no need to maintain the layers[] field by hand. Polls on the 60s menu cadence + window focus, so entries appear / disappear as services start and stop reporting. Smarter landing. Root / cascades through a sensible chain so the user never sees a blank page: first available public overview → first layer with services → the empty landing (/landing-empty). The cascade only lands on destinations that are also in the sidebar — a bundled-but-inactive layer (no services yet) is deliberately not a fallback, since it would put the user on a page they can\u0026rsquo;t navigate back to via the menu. The empty page is also a real bookmarkable route, with two distinct copies — \u0026ldquo;No data is flowing yet\u0026rdquo; (no agents/receivers reporting) vs \u0026ldquo;No dashboard configured yet\u0026rdquo; (services exist but no overview is set up) — each with the right operations-team handoff and no action buttons (a viewer\u0026rsquo;s role doesn\u0026rsquo;t include the verbs the old buttons jumped to). Debug events panel now defaults OFF on every host (was on for localhost). Same baseline for operators and developers so reproductions match what operators see. Zipkin trace mode drops the per-layer service-KPI header — the Zipkin explorer is a self-contained, cross-service view. 3D Infrastructure Map A standalone, bird\u0026rsquo;s-eye view of the deployment at /3d/map: services render as cubes on stacked tier-planes (apps · service mesh · middleware · infra), each tier subdivided into per-layer zones with the layer\u0026rsquo;s brand mark stamped on its colored swatch. Drag to rotate, scroll to zoom, arrow keys / WASD to pan; click a cube for its detail card and a link into that layer\u0026rsquo;s dashboard.\nLive data windows. The map auto-refreshes every minute — per-cube traffic rolls up the last 2h of metrics (HOUR step) and alarmed services light up from the last 20m of alarms. A toolbar chip shows the active scopes (metrics 2h · alarms 20m · ↻ 1m). An alarmed cube burns red with a radiating ripple, matched to its service by (layer, name) so only the firing service in the right tier is flagged. Live topology. The deployment structure is read live from OAP rather than a bundled snapshot: each layer\u0026rsquo;s service roster and service map are fetched one at a time (low concurrency) and assembled into the scene, so the map is correct on any deployment. It refreshes on the same one-minute cycle — an unchanged structure updates metrics/alarms in place without disturbing the camera, while a service appearing or disappearing rebuilds the affected tier. The load progresses stage by stage in the status strip. Beacon mode. A toolbar toggle dims every healthy cube to a wireframe ghost and lets only alarming cubes glow, so the services that are firing jump out instantly during an incident. Logic groups. Related layers can be clustered into a single labelled block on a tier — the bundled config ships a Self-Observability group (OAP, Satellite, BanyanDB, and the Java / Go agents) on the middleware tier. Members keep their own cube colors but read as one block on the map. Configurable tiers + layers. Tier order, per-layer plane mapping, cube colors, the traffic MQE per layer, and the logic groups are all driven by the 3D map config, edited on a structured admin page at /admin/3d-map. Pin each layer to a tier (with a single global layer filter as the top-level gate), edit each layer\u0026rsquo;s color + traffic metric, manage logic groups (members, color, icon, tier), and choose the single failover tier for anything unpinned. The config is published to OAP and shared across the deployment the same way as dashboards: edits save to a local draft in your browser, then Check diff \u0026amp; push publishes to OAP — the map renders the remote, with the bundled defaults as fallback. Topology clustering. Within layers that carry a service map, services group into named clusters drawn as a wireframe frame with the cluster name baked into the frame\u0026rsquo;s lower-left corner — service-mesh services cluster by their showcase group, Kubernetes services by namespace. Clustering follows each layer\u0026rsquo;s naming rule, so layers without one keep rendering flat. Navigate by tier. The right-side Tiers panel is a two-level tree (tier → layer / logic-group). Clicking any entry resets the view, glides the camera to face that region, zooms in, and flashes the region for a few seconds so it is easy to spot. A Reset button in the panel header restores the initial framing. Hover = preview. Hovering a cube shows the same detail card as a click — tier, layer, service, and (when present) group and cluster — anchored at the cube, minus the open-in-dashboard link. Call-direction flow. Call relationships animate directional particles so the direction of traffic reads at a glance. Focus one layer. The per-layer service map gains a View in 3D link that opens the map focused on just that layer. Refresh countdown. The load status strip shows a live countdown to the next refresh, anchored to the stage that will run next. Reliability When the BFF is unreachable the UI now shows a clear \u0026ldquo;Cannot reach the server\u0026rdquo; message instead of the cryptic \u0026ldquo;body stream already read\u0026rdquo; — the API client reads each error response body once and surfaces the real status/text (or a wrapped network error). Server-global service-by-layer catalog. One singleton on the BFF (60s TTL + single-flight) now owns the listLayers + aliased listServices(layer) fan-out. The sidebar menu\u0026rsquo;s per-layer counts and the alarms layer-tagger share this one cache instead of each running their own poll, so OAP sees at most one fan-out per minute regardless of how many routes are polling — and the two views can no longer drift by 60s relative to each other. The brand link and the post-login redirect no longer resolve to an empty address — both now land on the operator\u0026rsquo;s actual landing route. Trace span detail now labels the span direction as Kind (the noun) rather than the mistranslated verb form, and long tag keys wrap inside the panel instead of overflowing their column. ","excerpt":"\u003ch1 id=\"060\"\u003e0.6.0\u003c/h1\u003e\n\u003cp\u003eThis release is the production-readiness pass for Horizon UI: every page now renders correctly …\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/changelog/0.6.0/","title":"0.6.0"},{"body":"0.6.0 This release is the production-readiness pass for Horizon UI: every page now renders correctly across the eight supported languages on non-UTC OAP deployments, with deliberate caps and validation on the load surfaces that operators reach. The pillars below describe the operator-visible result.\nEight-locale internationalization Horizon now ships with eight first-class UI languages — English (source) plus zh-CN, ja, ko, es, pt, de, fr — selectable from the top-bar locale chip on every page (including the pre-auth login). The choice persists per device.\nUI chrome. Every routed page and every shared sub-component renders through vue-i18n; non-English locales now cover every admin page (Roles, Users, Auth status, Alert page setup, Global defaults, 3D-map config), every operate page (Alerting rules, DSL catalog / editor / dump, OAL catalog, Live debugger + MAL / LAL / OAL, Capture history, Metrics inspect, OAP config, TTL), the alarms surface, and the shared modals. Long lede paragraphs that previously rendered as English | one translated word | English mid-sentence are now single translation units — inline \u0026lt;code\u0026gt; and links interpolate without splitting the prose. Missing leaves still fall back to English so partial catalogs degrade invisibly. BFF-shipped templates. All 42 layer dashboards and both overview dashboards carry per-locale overlay catalogs alongside the source template. Coverage is ~2,300 translatable leaves per non-English locale across the layer set. The BFF picks the locale from the request\u0026rsquo;s X-Horizon-Locale header (auto-set by the SPA), merges the overlay onto the source, and serves the localised template to the renderer — translation resolves once on the BFF, never on every chart mount. Operator-runnable Translations page. A new admin surface (Dashboard setup → Translations) edits the per-locale overlays through the live preview: pick a target language, click any widget in the rendered dashboard, type the translation. Per-locale status chips on the template picker show at a glance which dashboards have drafts, which are synced, which diverge from disk, and which are empty for a given locale. Push writes the sibling overlay row on OAP; pushing zh-CN never touches ja. Tech-term policy. Product, project, and protocol names (SkyWalking, Kubernetes, OAP, MQE, eBPF, Zipkin, OpenTelemetry, Istio, GraphQL, etc.), OAP scope enums (Service, ServiceInstance, Endpoint, Process), layer keys, MQE function names, env vars, HTTP status codes, and per-language runtimes (JVM, Go, Python, …) stay verbatim in every locale per CLAUDE.md. Phrases containing tech terms are translated around the term (HTTP Connections → HTTP 连接 / HTTP 接続), not transliterated. OAP-supplied data is never translated. Service names, alarm rule names, trace span operation names, log messages — anything arriving over the OAP wire — render verbatim regardless of locale. Validator gate. i18n:validate is stricter: every source template must have a sibling overlay file per advertised locale, and empty {} overlays are now a finding (used to pass silently — surfaced as \u0026ldquo;structurally complete\u0026rdquo; while every translatable string still rendered in English). Typography + self-hosted fonts Inter + JetBrains Mono are now self-hosted. The Google Fonts CDN dependency is gone — air-gapped or firewalled deployments render the intended typography instead of silently falling back to system fonts. One typescale across every page. Older admin pages that drifted to a mixed pixel palette (9.5 / 10 / 10.5 / 11 / 11.5 / 12 / 14 / 18 / 20 / 22) now share the same six-step scale + uppercase-label vocabulary as the newer dashboards. Sidebar, kpi labels, table headers, kickers all line up. Wire-correctness on non-UTC OAP Every BFF query route now spells Duration.start / end in the OAP server\u0026rsquo;s timezone (probed once per minute, cached). Previously only the alarms route did this; dashboards / landing / topology / endpoint / endpoint-dependency / instance / eBPF / traces / logs / trace-tag all emitted UTC, which silently shifted every query on non-UTC OAP installs by the server\u0026rsquo;s offset.\nTraces and logs additionally query at SECOND precision now (records, not metric buckets) — a trace that just finished falls inside the window instead of getting rounded off the MINUTE boundary.\nPerformance hardening Landing batches no longer 5xx on wide layers. The per-layer landing route used to build one GraphQL with up to 250 aliased fragments (25 services × 10 metric columns) and trip OAP\u0026rsquo;s per-request complexity ceiling, blanking every cell. Chunks at 6 services per round-trip and fires them in parallel — same pattern the dashboard route already uses. Trace waterfall opens fast on huge traces. Rows render lazily via the browser\u0026rsquo;s content-visibility window — a 5000-span trace no longer freezes the main thread on open. Backgrounded tabs stop polling. The shared auto-refresh ticker pauses when the tab is hidden and resumes (with one immediate tick) on return. An unattended browser no longer streams queries at the topbar interval × every subscribed widget. RBAC + input-validation hardening /api/health no longer leaks the active session count to unauthenticated callers — the public liveness probe returns only status + version. The authenticated /api/auth/health surface still carries detail. pageSize capped server-side on every trace / log route (trace 200, log 100). OAP forwards paging.pageSize straight to the storage LIMIT, so a client posting pageSize: 50000 previously cascaded the load to OAP. The UI picker\u0026rsquo;s matching cap is now defended at the BFF boundary too. Profiling task bodies validated. Async-profiler, pprof, eBPF fixed-task, and network-profiling create routes now sanitize and bound their bodies — duration caps, target-instance and event-list caps, payload-size clamps. Closes a DoS vector where a user with profile:enable could submit a multi-hour profile that pegs the target instance\u0026rsquo;s CPU. Diff modal console error fixed The four admin \u0026ldquo;Check diff \u0026amp; push\u0026rdquo; modals (Layer dashboards, Overview templates, Translations, the shared TemplateDiffModal) used to log Uncaught (in promise) Error: Missing requestHandler or method: resetSchema (two per modal — one per diff pane) the moment the operator opened them. Monaco\u0026rsquo;s JSON language service was sending the message to a worker that didn\u0026rsquo;t know how to handle it. Now wired correctly — the diff itself rendered all along, but the console is quiet again.\nDSL / OAL catalog renames + admin editor seeding Catalog pages spell out the language name in the header: Metrics Analysis Language - OpenTelemetry Rules (and … - Telegraf Rules / … - Log MAL Rules); LAL renders as Log Analysis Language; the OAL browse as Observability Analysis Language. The sidebar keeps the abbreviated MAL · OTEL form for space. Layer Dashboards + Overview Templates editor opens REMOTE on diverged rows. The runtime menu already renders remote content when the row is diverged, but the editor used to seed from bundled — so operators were editing a copy that silently disagreed with what end users saw. Priority is now local → remote (diverged / remote-only) → bundled; the source pill on both editors reads from local / from remote / from bundled consistently. Rule card arrow stays next to the rule name. Short names like default / mesh-dp / vm used to float in the middle of the card with a big gap from the ▶ run arrow — the arrow + name are now grouped on the left, with the status pill alone pushed right. Sync-status counters don\u0026rsquo;t include translation rows Per-locale overlay rows on OAP share their parent template\u0026rsquo;s kind, so they were inflating the remote-only and diverged counts on the Overview Templates and Layer Dashboards admin pages (the banners would read 14 remote-only / 294 remote-only when most of those entries were translation rows that belong on the Translations page only). Filtered at bundle assembly so the sync-status banners count source rows only.\nCluster Status + admin polish Cluster Status — Pane B + Pane C fully translatable. Module column headers, gate descriptions (the SWIP-13 affects strings), per-row enabled/missing badges, the admin-host status badge (loading… / unreachable / all selectors on / {n} selectors off), the admin-host-unreachable hint, the Zipkin / OTLP pane lede, the Endpoint card heading, the Zipkin badge and the Zipkin-unreachable hint all now render in the active locale. Hide redundant BUNDLED chrome when synced. On the Overview templates and Layer dashboards admin pages, when the row\u0026rsquo;s sync badge is synced the BUNDLED and REMOTE versions are byte-equal, so the from {source} pill and the Reset to / Preview Bundled dropdown items add no information and are hidden. LOCAL drafts always show; BUNDLED resurfaces the moment the row diverges. Translations picker prefers the English bundle. REMOTE-only rows with non-English titles (legacy duplicates from prior import cycles) no longer appear as separate dashboards in the picker — the picker lists the canonical English bundled dashboards once each, and the preview renders the English source as the baseline. Alerting rules — running entities show their OAP node The Operate › Alerting rules detail pane\u0026rsquo;s Currently watching list now spans the whole cluster and tags each entity with the OAP node evaluating it. Each OAP instance evaluates a rule independently over the slice of entities it holds, so the watched set is the union across nodes — the page previously showed only the first responding node\u0026rsquo;s entities, which misread as \u0026ldquo;these are all the entities the rule watches.\u0026rdquo; The list now aggregates every instance\u0026rsquo;s entities and labels each row with its node (e.g. SERVICE agent::app NODE 10.116.3.26_11800), with the per-entity alarm message on hover. The per-node load-state table is unchanged. Single-instance deployments simply show one node label per row.\nClicking a watched entity now opens a running-context popup — the live evaluation window the rule is computing for that entity, per OAP node. It shows the current state (FIRING / SILENCED_FIRING / RECOVERY_OBSERVATION), the window size and silence / recovery countdowns, the window end, the last-alarm time and message, and the per-metric snapshot the expression was evaluated against — rendered as a sparkline plus per-bucket values so an operator can see exactly why a rule is (or isn\u0026rsquo;t) firing. Nodes not evaluating the entity are marked as such, and a raw-JSON disclosure carries the full payload.\nLive debugger fixes A clutch of small but visible bugs were caught while exercising the i18n surfaces:\nHistory → debug deep-link rendered blank. MAL and LAL views crashed silently on ?historyId=… because loadHistorical reset refs (selectedRow, expandedEntities, foldedRecords; selectedCell for LAL) that the file declared further down. The watch(historyId, …, { immediate: true }) fires during setup, so the TDZ ReferenceError aborted setup before the page rendered, with no console trace. Resettable refs are now hoisted above their consumer. Captured records wiped on stop. After stop(), the per-DSL view did one final session() refresh against an already-cleaned- up OAP, which returned nodes-only-with-empty-records and overwrote the rich live snapshot in localStorage. save() now refuses to shrink an existing entry\u0026rsquo;s recordCount for the same sessionId — only metadata (retentionDeadline, retentionMillis) updates. Stable node ordering. MAL / LAL / OAL node cards now sort by nodeId ?? peer so they don\u0026rsquo;t reshuffle between polls. Tab buttons jumped nowhere. LiveDebuggerView.selectTab pushed /debug/\u0026lt;tab\u0026gt; (a path that doesn\u0026rsquo;t exist); fixed to /operate/live-debug/\u0026lt;tab\u0026gt;. Same correction in DebugHistoryView.loadEntry deep-links and surrounding doc-comments in RuleCard.vue / DslEditorView.vue. Empty-capture placeholder. When a saved session has zero populated nodes, DebugView now shows an explicit \u0026ldquo;This capture has no records\u0026rdquo; rather than rendering blank, so an honest empty capture is visibly different from a bug. Per-locale lookup widened. Per-DSL views look up the historyId via history.all (not the widget-filtered history.entries); the route already pinned us to the right widget, and double-filtering by widget was a silent way for a stale field to drop the entry. Other small fixes MAL / LAL editor gutter glyph. The green ▶ live-debug entrance on every - name: row was referencing an unstyled CSS class — Monaco reserved the gutter and wired clicks, but the icon was invisible. Restored. Rule catalog cards. Duplicate BUNDLED pill removed; the header status pill already conveyed it. K8s instance Node Status. Switched from a single-scalar card (rendering just 1) to a table of currently-true Kubernetes conditions, matching the cluster-scope sibling and aligning widget heights. Upstream ui-management compatibility Aligned with upstream skywalking#13884 (\u0026ldquo;remove auto generate id for UI templates\u0026rdquo;): Horizon now sends id = \u0026lt;envelope name\u0026gt; on POST /ui-management/templates. Current OAP requires it (POSTs without id are rejected); legacy OAP releases ignored the field and auto-generated UUIDs, so the same payload works against both. Horizon already treated r.id as opaque; mixed-id deployments self-heal via reconcileDuplicates on the next boot.\nSmartscape service hierarchy OAP 10\u0026rsquo;s cross-layer service hierarchy is now reachable from any layer\u0026rsquo;s service map — a logical service projected across observation layers (GENERAL agent ↔ MESH sidecar ↔ MESH_DP data-plane ↔ K8S_SERVICE pod) is one click away on every selected hex.\nLazy-probed chip on the selected hex. Picking a node fires one getServiceHierarchy call; if the service has cross-layer peers, a small chevron-stack chip clips to the hex\u0026rsquo;s right edge. No probe, no chip on services with no peers. Focus + context + suggestions overlay. Click the chip and the topology dims under a transparent canvas; the focused hex re-renders bright at the exact same screen position and scale as the underlying hex (the topology\u0026rsquo;s d3 zoom transform is mirrored onto the overlay). Peers fan vertically from the focus column using OAP\u0026rsquo;s listLayerLevels order — higher-level (request-near) layers above, lower-level (infra-near) layers below, matching booster-ui\u0026rsquo;s hierarchy rendering rule. Auto-refresh pauses while the overlay is open so the background topology and KPI panels don\u0026rsquo;t shift under the operator. Closing the overlay (× button, ESC, or click-on-dim) resumes the ticker and fires one immediate tick so the page snaps back to live data. Two-step peer open. First click on a peer hex arms it (selection halo + side ↗ Open in \u0026lt;Layer\u0026gt; action chip); second click on the chip opens the destination layer in a new browser tab, pre-selecting the peer service. Peers in layers Horizon has no template for render dimmed with a cursor: not-allowed; clicking them logs \u0026ldquo;No layer template configured for \u0026lt;Layer\u0026gt;\u0026rdquo; to the event log instead. URL-pinned service validator on the destination tab. Every per-layer page now validates the URL-hydrated ?service=\u0026lt;id\u0026gt; against the layer\u0026rsquo;s real service roster (the new GET /api/layer/:key/services, served from the BFF\u0026rsquo;s 60s catalog cache so it adds no extra OAP traffic). A genuinely missing id pops a Service not found in this layer modal with a one-click fallback to the first available service; a valid id is trusted even when landing\u0026rsquo;s top-N rollup doesn\u0026rsquo;t sample it (the cause of the previous silent service-swap on low-traffic deep links). Service-name resolution on the layer dashboard now consults the roster after landing\u0026rsquo;s top-N, so deep links to low-traffic services no longer sit on \u0026ldquo;Resolving service…\u0026rdquo; forever waiting for a row that won\u0026rsquo;t arrive. On-demand pod logs (live tail) A new per-layer Pod Logs tab live-tails a Kubernetes pod\u0026rsquo;s container logs, pulled on demand from the K8s API through OAP and never persisted.\nInstance-pinned tail. Pick a pod, pick one of its containers, press Start; the trailing window (30s / 1m / 5m / 15m / 30m) streams into a read-only log pane and refreshes on a chosen interval (2s / 5s / 10s / 30s) until paused. A header strip shows the container, line count, a live dot, and \u0026ldquo;updated Ns ago\u0026rdquo;. Include / Exclude filtering forwards to OAP\u0026rsquo;s content keyword filters — full-line regex, so a substring match reads .*error.*. Enabled on the Kubernetes-deployed layers — Kubernetes Services (K8S_SERVICE), Istio Managed Services (MESH), and Istio Data Plane (MESH_DP) — whose service instances resolve to a pod. The tab is gated by a new podLogs component flag added to those bundled layer templates; an existing OAP whose stored template predates the flag still gets the tab, because the flag is back-filled from the bundled default (no re-push needed). The page owns its own refresh — the global auto-refresh ticker and the topbar time picker are paused while on it, the same as Traces / Logs. When the selected instance carries no pod metadata (or the pod has rotated away), OAP\u0026rsquo;s reason is shown verbatim with a hint to pick a currently-running pod or enable the feature on OAP. BanyanDB cold-stage query The cold lifecycle stage is now reachable from the UI on BanyanDB deployments — operators can query data that has aged past the hot + warm window without leaving the page.\nTopbar Cold pill appears only when the connected OAP is BanyanDB. Toggling it switches every page to read from the cold stage instead of hot + warm — it replaces the read, it does NOT union the two stages, so the pill label flips to Cold only while on. The choice is sticky per browser and re-runs every visible query so what you see matches the new mode immediately. Cold-trap banner. When the pill is on AND the current time range is within the hot + warm window, a yellow strip appears under the topbar: \u0026ldquo;Cold-only read is active — your time range is within the last N d (hot + warm), where the cold stage returns nothing.\u0026rdquo; A one-click Turn Cold off sits on the right. Trace lookup from a log row now passes the row\u0026rsquo;s timestamp through the popout so the trace lookup spans a window around that timestamp instead of OAP\u0026rsquo;s default last-1-day search — paired with the Cold pill, a trace that lives in cold resolves from a cold-era log row instead of silently failing to load. Data retention page Per-data-class data lifecycle bar. One row per data class (Normal / Trace / Zipkin trace / Log / Browser error log / Metadata / Minute metric / Hour metric / Day metric) with proportional Hot+Warm and (when configured) Cold segments. Widths are proportional to total retention across all rows, so a class retained longer visibly stretches further than its peers. When every class in a category shares the same TTL pair, the rows collapse to All records (5) / All metrics (4) — the page never renders nine identical bars. The page branches sharply by backend. BanyanDB shows the full lifecycle bar + stage vocabulary; on any other backend, the page renders a single Retention pane with per-class values and skips the stage vocabulary entirely. A footer note names the wire-level truth: OAP\u0026rsquo;s TTL response collapses hot + warm into one number per class, BanyanDB migrates between stages in segments so records near a boundary may briefly exist in both, and property data is omitted (forever-retained, no TTL reported). Time picker Custom range seeds from the last applied range when you re-open the picker, instead of resetting to \u0026ldquo;half the max ending now\u0026rdquo;. Reopening also auto-expands the Custom form on the matching precision tab when the current range is custom. Locale-bleed fix on the alarms page custom-range stamp and the log row date column (was rendering 5月08日 on zh-CN browsers; now uniform MM-DD). Overview widgets follow the global time picker The Services Dashboard (and any overview using metric / KPI / table widgets) now honors the topbar time picker. Previously the per-layer landing and topology routes were hardcoded to the last 60 minutes, so picking 12 days back kept showing recent numbers; now picker + Cold pill flow end-to-end. The layer dashboard\u0026rsquo;s header KPIs follow the picker too (was showing live numbers while the body honored the picker). The Active alarms widget title now shows the actual window (e.g. · last 10m) and the empty-state copy uses the same value instead of a hardcoded \u0026ldquo;last 60m\u0026rdquo;. Polish Sentence-case fixes on a couple of leftover Title-Case labels (DSL management, Metrics inspect) so the menu and roles tables read consistently; acronyms (DSL / OAP / MAL / LAL / OAL) stay uppercase. Public-demo (demo.skywalking.apache.org) references removed from setup docs — the demo doesn\u0026rsquo;t accept anonymous traffic. Dashboard authoring The Layer dashboards and Overview templates admin pages now share one editing model where your work-in-progress lives in your browser, and the live, shared version is whatever OAP serves.\nEdits are local to your browser. \u0026ldquo;Save (local)\u0026rdquo; stores your draft in this browser only — it is never written to the server and nobody else sees it. Everyone (including you, in normal viewing) keeps seeing the published remote dashboard until you publish. Load any source into the editor. A single Reset to ▾ control loads the Bundled (shipped default) or Remote (OAP live) version into the editor; editing from there becomes your local draft. Preview any source on the real page. A Preview ▾ control opens the actual layer / overview page in a new tab rendering your Local draft, the Bundled default, or Remote — via ?mode=preview\u0026amp;source=…, which stays in the URL and propagates as you navigate the menu. A banner names what you\u0026rsquo;re previewing (dismiss with × or Esc). Publish with a diff. Check diff \u0026amp; push shows a side-by-side local→remote diff and publishes to OAP; it\u0026rsquo;s enabled only when your local draft actually differs from remote. Bundled can also be pushed straight to OAP. Resetting to remote clears the local draft. Reset to bundled then publishes correctly when the bundled default differs from remote — Save (local) and Check diff \u0026amp; push now compare the editor against remote, not just against what was first loaded, so a bundled-vs-remote divergence is no longer mistaken for \u0026ldquo;no changes\u0026rdquo; (layer + overview editors). Preview faithfully reflects your draft\u0026rsquo;s enabled components / menu labels — disabled tabs disappear and renamed nouns (\u0026ldquo;Nodes\u0026rdquo;, \u0026ldquo;Topics\u0026rdquo;) show through — without pushing anything to the server. Preview works even for layers OAP currently reports no services for. An editors-only reminder lists any unpublished local drafts with quick links to the relevant edit page (no more \u0026ldquo;use local vs remote\u0026rdquo; prompt — remote is always the live source). Create mirrors edit. \u0026ldquo;+ New dashboard\u0026rdquo; writes a local draft (the id is the template name, checked unique) — edit and preview it, then Check diff \u0026amp; push publishes it. A pushed dashboard with no bundled default is remote-only and now renders everywhere (live page + sidebar), not just in the editor. Delete = soft-disable (OAP has no hard delete). A local-only draft is removed from the browser; a dashboard on OAP is disabled — dropped from the picker\u0026rsquo;s live state, the sidebar, and the live page. A disabled status chip shows it. Confirmations are styled in-app dialogs, not the browser\u0026rsquo;s native box. Layer dashboards editor The layer picker is a single filterable dropdown showing alias + key + sync status. A live menu preview sits beside the Alias / Components / Menu labels (per-layer slot aliases) editor; clicking a menu item jumps to that component\u0026rsquo;s config. Scope tabs and section headings read in the layer\u0026rsquo;s own vocabulary. The service-list metrics editor gains a sample-data preview (plus a faithful landing KPI tile preview that reuses the real header components), the column remove-button alignment is fixed, and the landing KPI tile config makes clear it just picks which existing column feeds the headline + sparkline. The picker\u0026rsquo;s Diverged / Local filters now sit inside the dropdown (next to the search), and the editor header reads on one line — Layer: \u0026lt;name\u0026gt; \u0026lt;key\u0026gt; \u0026lt;status\u0026gt; — showing the same sync-status chip the picker does. Disable / Reactivate a layer. Disabling soft-disables the layer on OAP and drops it from the sidebar (the menu honors disabled templates); a disabled layer offers Reactivate, which re-enables it from the bundled default (the OAP update path clears the disabled flag). Overview templates editor Rebuilt as a layer-style canvas: a 12-column grid you drag to reorder and corner-drag to resize, click-to-edit in a right drawer that appears on selection (Esc / deselect hides it), with section-breaks and the dashboard title selectable and unselected widgets hinted by a dashed outline. The canvas mirrors the live grid (fixed row height), so the layout matches the real page — including side-by-side widgets like topology + alarms. The composite-metrics KPI editor stacks each row as a card (label / source / MQE / unit / aggr / style / max) so nothing truncates in the narrow drawer. Navigation \u0026amp; shell The main sidebar folds to a narrow rail to reclaim width; the logo moves into the topbar while folded (the original wordmark is unchanged). The active menu item now reliably expands, highlights, and scrolls into view on entering a page (route matching is case-insensitive). Fixed a regression where the sidebar scrolled to the very bottom on every navigation (the \u0026ldquo;Debug events\u0026rdquo; toggle\u0026rsquo;s active state was being treated as the scroll target). Overview dashboards appear in the sidebar only when their layers are reporting services. Visibility is derived from each dashboard\u0026rsquo;s widgets (their layer field) ∪ the explicit layers[] list, gated against the live availableLayers. A dashboard you create via \u0026ldquo;+ New\u0026rdquo; inherits this automatically — no need to maintain the layers[] field by hand. Polls on the 60s menu cadence + window focus, so entries appear / disappear as services start and stop reporting. Smarter landing. Root / cascades through a sensible chain so the user never sees a blank page: first available public overview → first layer with services → the empty landing (/landing-empty). The cascade only lands on destinations that are also in the sidebar — a bundled-but-inactive layer (no services yet) is deliberately not a fallback, since it would put the user on a page they can\u0026rsquo;t navigate back to via the menu. The empty page is also a real bookmarkable route, with two distinct copies — \u0026ldquo;No data is flowing yet\u0026rdquo; (no agents/receivers reporting) vs \u0026ldquo;No dashboard configured yet\u0026rdquo; (services exist but no overview is set up) — each with the right operations-team handoff and no action buttons (a viewer\u0026rsquo;s role doesn\u0026rsquo;t include the verbs the old buttons jumped to). Debug events panel now defaults OFF on every host (was on for localhost). Same baseline for operators and developers so reproductions match what operators see. Zipkin trace mode drops the per-layer service-KPI header — the Zipkin explorer is a self-contained, cross-service view. 3D Infrastructure Map A standalone, bird\u0026rsquo;s-eye view of the deployment at /3d/map: services render as cubes on stacked tier-planes (apps · service mesh · middleware · infra), each tier subdivided into per-layer zones with the layer\u0026rsquo;s brand mark stamped on its colored swatch. Drag to rotate, scroll to zoom, arrow keys / WASD to pan; click a cube for its detail card and a link into that layer\u0026rsquo;s dashboard.\nLive data windows. The map auto-refreshes every minute — per-cube traffic rolls up the last 2h of metrics (HOUR step) and alarmed services light up from the last 20m of alarms. A toolbar chip shows the active scopes (metrics 2h · alarms 20m · ↻ 1m). An alarmed cube burns red with a radiating ripple, matched to its service by (layer, name) so only the firing service in the right tier is flagged. Live topology. The deployment structure is read live from OAP rather than a bundled snapshot: each layer\u0026rsquo;s service roster and service map are fetched one at a time (low concurrency) and assembled into the scene, so the map is correct on any deployment. It refreshes on the same one-minute cycle — an unchanged structure updates metrics/alarms in place without disturbing the camera, while a service appearing or disappearing rebuilds the affected tier. The load progresses stage by stage in the status strip. Beacon mode. A toolbar toggle dims every healthy cube to a wireframe ghost and lets only alarming cubes glow, so the services that are firing jump out instantly during an incident. Logic groups. Related layers can be clustered into a single labelled block on a tier — the bundled config ships a Self-Observability group (OAP, Satellite, BanyanDB, and the Java / Go agents) on the middleware tier. Members keep their own cube colors but read as one block on the map. Configurable tiers + layers. Tier order, per-layer plane mapping, cube colors, the traffic MQE per layer, and the logic groups are all driven by the 3D map config, edited on a structured admin page at /admin/3d-map. Pin each layer to a tier (with a single global layer filter as the top-level gate), edit each layer\u0026rsquo;s color + traffic metric, manage logic groups (members, color, icon, tier), and choose the single failover tier for anything unpinned. The config is published to OAP and shared across the deployment the same way as dashboards: edits save to a local draft in your browser, then Check diff \u0026amp; push publishes to OAP — the map renders the remote, with the bundled defaults as fallback. Topology clustering. Within layers that carry a service map, services group into named clusters drawn as a wireframe frame with the cluster name baked into the frame\u0026rsquo;s lower-left corner — service-mesh services cluster by their showcase group, Kubernetes services by namespace. Clustering follows each layer\u0026rsquo;s naming rule, so layers without one keep rendering flat. Navigate by tier. The right-side Tiers panel is a two-level tree (tier → layer / logic-group). Clicking any entry resets the view, glides the camera to face that region, zooms in, and flashes the region for a few seconds so it is easy to spot. A Reset button in the panel header restores the initial framing. Hover = preview. Hovering a cube shows the same detail card as a click — tier, layer, service, and (when present) group and cluster — anchored at the cube, minus the open-in-dashboard link. Call-direction flow. Call relationships animate directional particles so the direction of traffic reads at a glance. Focus one layer. The per-layer service map gains a View in 3D link that opens the map focused on just that layer. Refresh countdown. The load status strip shows a live countdown to the next refresh, anchored to the stage that will run next. Reliability When the BFF is unreachable the UI now shows a clear \u0026ldquo;Cannot reach the server\u0026rdquo; message instead of the cryptic \u0026ldquo;body stream already read\u0026rdquo; — the API client reads each error response body once and surfaces the real status/text (or a wrapped network error). Server-global service-by-layer catalog. One singleton on the BFF (60s TTL + single-flight) now owns the listLayers + aliased listServices(layer) fan-out. The sidebar menu\u0026rsquo;s per-layer counts and the alarms layer-tagger share this one cache instead of each running their own poll, so OAP sees at most one fan-out per minute regardless of how many routes are polling — and the two views can no longer drift by 60s relative to each other. The brand link and the post-login redirect no longer resolve to an empty address — both now land on the operator\u0026rsquo;s actual landing route. Trace span detail now labels the span direction as Kind (the noun) rather than the mistranslated verb form, and long tag keys wrap inside the panel instead of overflowing their column. ","excerpt":"\u003ch1 id=\"060\"\u003e0.6.0\u003c/h1\u003e\n\u003cp\u003eThis release is the production-readiness pass for Horizon UI: every page now renders correctly …\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/changelog/0.6.0/","title":"0.6.0"},{"body":"0.6.0 This release is the production-readiness pass for Horizon UI: every page now renders correctly across the eight supported languages on non-UTC OAP deployments, with deliberate caps and validation on the load surfaces that operators reach. The pillars below describe the operator-visible result.\nEight-locale internationalization Horizon now ships with eight first-class UI languages — English (source) plus zh-CN, ja, ko, es, pt, de, fr — selectable from the top-bar locale chip on every page (including the pre-auth login). The choice persists per device.\nUI chrome. Every routed page and every shared sub-component renders through vue-i18n; non-English locales now cover every admin page (Roles, Users, Auth status, Alert page setup, Global defaults, 3D-map config), every operate page (Alerting rules, DSL catalog / editor / dump, OAL catalog, Live debugger + MAL / LAL / OAL, Capture history, Metrics inspect, OAP config, TTL), the alarms surface, and the shared modals. Long lede paragraphs that previously rendered as English | one translated word | English mid-sentence are now single translation units — inline \u0026lt;code\u0026gt; and links interpolate without splitting the prose. Missing leaves still fall back to English so partial catalogs degrade invisibly. BFF-shipped templates. All 42 layer dashboards and both overview dashboards carry per-locale overlay catalogs alongside the source template. Coverage is ~2,300 translatable leaves per non-English locale across the layer set. The BFF picks the locale from the request\u0026rsquo;s X-Horizon-Locale header (auto-set by the SPA), merges the overlay onto the source, and serves the localised template to the renderer — translation resolves once on the BFF, never on every chart mount. Operator-runnable Translations page. A new admin surface (Dashboard setup → Translations) edits the per-locale overlays through the live preview: pick a target language, click any widget in the rendered dashboard, type the translation. Per-locale status chips on the template picker show at a glance which dashboards have drafts, which are synced, which diverge from disk, and which are empty for a given locale. Push writes the sibling overlay row on OAP; pushing zh-CN never touches ja. Tech-term policy. Product, project, and protocol names (SkyWalking, Kubernetes, OAP, MQE, eBPF, Zipkin, OpenTelemetry, Istio, GraphQL, etc.), OAP scope enums (Service, ServiceInstance, Endpoint, Process), layer keys, MQE function names, env vars, HTTP status codes, and per-language runtimes (JVM, Go, Python, …) stay verbatim in every locale per CLAUDE.md. Phrases containing tech terms are translated around the term (HTTP Connections → HTTP 连接 / HTTP 接続), not transliterated. OAP-supplied data is never translated. Service names, alarm rule names, trace span operation names, log messages — anything arriving over the OAP wire — render verbatim regardless of locale. Validator gate. i18n:validate is stricter: every source template must have a sibling overlay file per advertised locale, and empty {} overlays are now a finding (used to pass silently — surfaced as \u0026ldquo;structurally complete\u0026rdquo; while every translatable string still rendered in English). Typography + self-hosted fonts Inter + JetBrains Mono are now self-hosted. The Google Fonts CDN dependency is gone — air-gapped or firewalled deployments render the intended typography instead of silently falling back to system fonts. One typescale across every page. Older admin pages that drifted to a mixed pixel palette (9.5 / 10 / 10.5 / 11 / 11.5 / 12 / 14 / 18 / 20 / 22) now share the same six-step scale + uppercase-label vocabulary as the newer dashboards. Sidebar, kpi labels, table headers, kickers all line up. Wire-correctness on non-UTC OAP Every BFF query route now spells Duration.start / end in the OAP server\u0026rsquo;s timezone (probed once per minute, cached). Previously only the alarms route did this; dashboards / landing / topology / endpoint / endpoint-dependency / instance / eBPF / traces / logs / trace-tag all emitted UTC, which silently shifted every query on non-UTC OAP installs by the server\u0026rsquo;s offset.\nTraces and logs additionally query at SECOND precision now (records, not metric buckets) — a trace that just finished falls inside the window instead of getting rounded off the MINUTE boundary.\nPerformance hardening Landing batches no longer 5xx on wide layers. The per-layer landing route used to build one GraphQL with up to 250 aliased fragments (25 services × 10 metric columns) and trip OAP\u0026rsquo;s per-request complexity ceiling, blanking every cell. Chunks at 6 services per round-trip and fires them in parallel — same pattern the dashboard route already uses. Trace waterfall opens fast on huge traces. Rows render lazily via the browser\u0026rsquo;s content-visibility window — a 5000-span trace no longer freezes the main thread on open. Backgrounded tabs stop polling. The shared auto-refresh ticker pauses when the tab is hidden and resumes (with one immediate tick) on return. An unattended browser no longer streams queries at the topbar interval × every subscribed widget. RBAC + input-validation hardening /api/health no longer leaks the active session count to unauthenticated callers — the public liveness probe returns only status + version. The authenticated /api/auth/health surface still carries detail. pageSize capped server-side on every trace / log route (trace 200, log 100). OAP forwards paging.pageSize straight to the storage LIMIT, so a client posting pageSize: 50000 previously cascaded the load to OAP. The UI picker\u0026rsquo;s matching cap is now defended at the BFF boundary too. Profiling task bodies validated. Async-profiler, pprof, eBPF fixed-task, and network-profiling create routes now sanitize and bound their bodies — duration caps, target-instance and event-list caps, payload-size clamps. Closes a DoS vector where a user with profile:enable could submit a multi-hour profile that pegs the target instance\u0026rsquo;s CPU. Diff modal console error fixed The four admin \u0026ldquo;Check diff \u0026amp; push\u0026rdquo; modals (Layer dashboards, Overview templates, Translations, the shared TemplateDiffModal) used to log Uncaught (in promise) Error: Missing requestHandler or method: resetSchema (two per modal — one per diff pane) the moment the operator opened them. Monaco\u0026rsquo;s JSON language service was sending the message to a worker that didn\u0026rsquo;t know how to handle it. Now wired correctly — the diff itself rendered all along, but the console is quiet again.\nDSL / OAL catalog renames + admin editor seeding Catalog pages spell out the language name in the header: Metrics Analysis Language - OpenTelemetry Rules (and … - Telegraf Rules / … - Log MAL Rules); LAL renders as Log Analysis Language; the OAL browse as Observability Analysis Language. The sidebar keeps the abbreviated MAL · OTEL form for space. Layer Dashboards + Overview Templates editor opens REMOTE on diverged rows. The runtime menu already renders remote content when the row is diverged, but the editor used to seed from bundled — so operators were editing a copy that silently disagreed with what end users saw. Priority is now local → remote (diverged / remote-only) → bundled; the source pill on both editors reads from local / from remote / from bundled consistently. Rule card arrow stays next to the rule name. Short names like default / mesh-dp / vm used to float in the middle of the card with a big gap from the ▶ run arrow — the arrow + name are now grouped on the left, with the status pill alone pushed right. Sync-status counters don\u0026rsquo;t include translation rows Per-locale overlay rows on OAP share their parent template\u0026rsquo;s kind, so they were inflating the remote-only and diverged counts on the Overview Templates and Layer Dashboards admin pages (the banners would read 14 remote-only / 294 remote-only when most of those entries were translation rows that belong on the Translations page only). Filtered at bundle assembly so the sync-status banners count source rows only.\nCluster Status + admin polish Cluster Status — Pane B + Pane C fully translatable. Module column headers, gate descriptions (the SWIP-13 affects strings), per-row enabled/missing badges, the admin-host status badge (loading… / unreachable / all selectors on / {n} selectors off), the admin-host-unreachable hint, the Zipkin / OTLP pane lede, the Endpoint card heading, the Zipkin badge and the Zipkin-unreachable hint all now render in the active locale. Hide redundant BUNDLED chrome when synced. On the Overview templates and Layer dashboards admin pages, when the row\u0026rsquo;s sync badge is synced the BUNDLED and REMOTE versions are byte-equal, so the from {source} pill and the Reset to / Preview Bundled dropdown items add no information and are hidden. LOCAL drafts always show; BUNDLED resurfaces the moment the row diverges. Translations picker prefers the English bundle. REMOTE-only rows with non-English titles (legacy duplicates from prior import cycles) no longer appear as separate dashboards in the picker — the picker lists the canonical English bundled dashboards once each, and the preview renders the English source as the baseline. Alerting rules — running entities show their OAP node The Operate › Alerting rules detail pane\u0026rsquo;s Currently watching list now spans the whole cluster and tags each entity with the OAP node evaluating it. Each OAP instance evaluates a rule independently over the slice of entities it holds, so the watched set is the union across nodes — the page previously showed only the first responding node\u0026rsquo;s entities, which misread as \u0026ldquo;these are all the entities the rule watches.\u0026rdquo; The list now aggregates every instance\u0026rsquo;s entities and labels each row with its node (e.g. SERVICE agent::app NODE 10.116.3.26_11800), with the per-entity alarm message on hover. The per-node load-state table is unchanged. Single-instance deployments simply show one node label per row.\nClicking a watched entity now opens a running-context popup — the live evaluation window the rule is computing for that entity, per OAP node. It shows the current state (FIRING / SILENCED_FIRING / RECOVERY_OBSERVATION), the window size and silence / recovery countdowns, the window end, the last-alarm time and message, and the per-metric snapshot the expression was evaluated against — rendered as a sparkline plus per-bucket values so an operator can see exactly why a rule is (or isn\u0026rsquo;t) firing. Nodes not evaluating the entity are marked as such, and a raw-JSON disclosure carries the full payload.\nLive debugger fixes A clutch of small but visible bugs were caught while exercising the i18n surfaces:\nHistory → debug deep-link rendered blank. MAL and LAL views crashed silently on ?historyId=… because loadHistorical reset refs (selectedRow, expandedEntities, foldedRecords; selectedCell for LAL) that the file declared further down. The watch(historyId, …, { immediate: true }) fires during setup, so the TDZ ReferenceError aborted setup before the page rendered, with no console trace. Resettable refs are now hoisted above their consumer. Captured records wiped on stop. After stop(), the per-DSL view did one final session() refresh against an already-cleaned- up OAP, which returned nodes-only-with-empty-records and overwrote the rich live snapshot in localStorage. save() now refuses to shrink an existing entry\u0026rsquo;s recordCount for the same sessionId — only metadata (retentionDeadline, retentionMillis) updates. Stable node ordering. MAL / LAL / OAL node cards now sort by nodeId ?? peer so they don\u0026rsquo;t reshuffle between polls. Tab buttons jumped nowhere. LiveDebuggerView.selectTab pushed /debug/\u0026lt;tab\u0026gt; (a path that doesn\u0026rsquo;t exist); fixed to /operate/live-debug/\u0026lt;tab\u0026gt;. Same correction in DebugHistoryView.loadEntry deep-links and surrounding doc-comments in RuleCard.vue / DslEditorView.vue. Empty-capture placeholder. When a saved session has zero populated nodes, DebugView now shows an explicit \u0026ldquo;This capture has no records\u0026rdquo; rather than rendering blank, so an honest empty capture is visibly different from a bug. Per-locale lookup widened. Per-DSL views look up the historyId via history.all (not the widget-filtered history.entries); the route already pinned us to the right widget, and double-filtering by widget was a silent way for a stale field to drop the entry. Other small fixes MAL / LAL editor gutter glyph. The green ▶ live-debug entrance on every - name: row was referencing an unstyled CSS class — Monaco reserved the gutter and wired clicks, but the icon was invisible. Restored. Rule catalog cards. Duplicate BUNDLED pill removed; the header status pill already conveyed it. K8s instance Node Status. Switched from a single-scalar card (rendering just 1) to a table of currently-true Kubernetes conditions, matching the cluster-scope sibling and aligning widget heights. Upstream ui-management compatibility Aligned with upstream skywalking#13884 (\u0026ldquo;remove auto generate id for UI templates\u0026rdquo;): Horizon now sends id = \u0026lt;envelope name\u0026gt; on POST /ui-management/templates. Current OAP requires it (POSTs without id are rejected); legacy OAP releases ignored the field and auto-generated UUIDs, so the same payload works against both. Horizon already treated r.id as opaque; mixed-id deployments self-heal via reconcileDuplicates on the next boot.\nSmartscape service hierarchy OAP 10\u0026rsquo;s cross-layer service hierarchy is now reachable from any layer\u0026rsquo;s service map — a logical service projected across observation layers (GENERAL agent ↔ MESH sidecar ↔ MESH_DP data-plane ↔ K8S_SERVICE pod) is one click away on every selected hex.\nLazy-probed chip on the selected hex. Picking a node fires one getServiceHierarchy call; if the service has cross-layer peers, a small chevron-stack chip clips to the hex\u0026rsquo;s right edge. No probe, no chip on services with no peers. Focus + context + suggestions overlay. Click the chip and the topology dims under a transparent canvas; the focused hex re-renders bright at the exact same screen position and scale as the underlying hex (the topology\u0026rsquo;s d3 zoom transform is mirrored onto the overlay). Peers fan vertically from the focus column using OAP\u0026rsquo;s listLayerLevels order — higher-level (request-near) layers above, lower-level (infra-near) layers below, matching booster-ui\u0026rsquo;s hierarchy rendering rule. Auto-refresh pauses while the overlay is open so the background topology and KPI panels don\u0026rsquo;t shift under the operator. Closing the overlay (× button, ESC, or click-on-dim) resumes the ticker and fires one immediate tick so the page snaps back to live data. Two-step peer open. First click on a peer hex arms it (selection halo + side ↗ Open in \u0026lt;Layer\u0026gt; action chip); second click on the chip opens the destination layer in a new browser tab, pre-selecting the peer service. Peers in layers Horizon has no template for render dimmed with a cursor: not-allowed; clicking them logs \u0026ldquo;No layer template configured for \u0026lt;Layer\u0026gt;\u0026rdquo; to the event log instead. URL-pinned service validator on the destination tab. Every per-layer page now validates the URL-hydrated ?service=\u0026lt;id\u0026gt; against the layer\u0026rsquo;s real service roster (the new GET /api/layer/:key/services, served from the BFF\u0026rsquo;s 60s catalog cache so it adds no extra OAP traffic). A genuinely missing id pops a Service not found in this layer modal with a one-click fallback to the first available service; a valid id is trusted even when landing\u0026rsquo;s top-N rollup doesn\u0026rsquo;t sample it (the cause of the previous silent service-swap on low-traffic deep links). Service-name resolution on the layer dashboard now consults the roster after landing\u0026rsquo;s top-N, so deep links to low-traffic services no longer sit on \u0026ldquo;Resolving service…\u0026rdquo; forever waiting for a row that won\u0026rsquo;t arrive. On-demand pod logs (live tail) A new per-layer Pod Logs tab live-tails a Kubernetes pod\u0026rsquo;s container logs, pulled on demand from the K8s API through OAP and never persisted.\nInstance-pinned tail. Pick a pod, pick one of its containers, press Start; the trailing window (30s / 1m / 5m / 15m / 30m) streams into a read-only log pane and refreshes on a chosen interval (2s / 5s / 10s / 30s) until paused. A header strip shows the container, line count, a live dot, and \u0026ldquo;updated Ns ago\u0026rdquo;. Include / Exclude filtering forwards to OAP\u0026rsquo;s content keyword filters — full-line regex, so a substring match reads .*error.*. Enabled on the Kubernetes-deployed layers — Kubernetes Services (K8S_SERVICE), Istio Managed Services (MESH), and Istio Data Plane (MESH_DP) — whose service instances resolve to a pod. The tab is gated by a new podLogs component flag added to those bundled layer templates; an existing OAP whose stored template predates the flag still gets the tab, because the flag is back-filled from the bundled default (no re-push needed). The page owns its own refresh — the global auto-refresh ticker and the topbar time picker are paused while on it, the same as Traces / Logs. When the selected instance carries no pod metadata (or the pod has rotated away), OAP\u0026rsquo;s reason is shown verbatim with a hint to pick a currently-running pod or enable the feature on OAP. BanyanDB cold-stage query The cold lifecycle stage is now reachable from the UI on BanyanDB deployments — operators can query data that has aged past the hot + warm window without leaving the page.\nTopbar Cold pill appears only when the connected OAP is BanyanDB. Toggling it switches every page to read from the cold stage instead of hot + warm — it replaces the read, it does NOT union the two stages, so the pill label flips to Cold only while on. The choice is sticky per browser and re-runs every visible query so what you see matches the new mode immediately. Cold-trap banner. When the pill is on AND the current time range is within the hot + warm window, a yellow strip appears under the topbar: \u0026ldquo;Cold-only read is active — your time range is within the last N d (hot + warm), where the cold stage returns nothing.\u0026rdquo; A one-click Turn Cold off sits on the right. Trace lookup from a log row now passes the row\u0026rsquo;s timestamp through the popout so the trace lookup spans a window around that timestamp instead of OAP\u0026rsquo;s default last-1-day search — paired with the Cold pill, a trace that lives in cold resolves from a cold-era log row instead of silently failing to load. Data retention page Per-data-class data lifecycle bar. One row per data class (Normal / Trace / Zipkin trace / Log / Browser error log / Metadata / Minute metric / Hour metric / Day metric) with proportional Hot+Warm and (when configured) Cold segments. Widths are proportional to total retention across all rows, so a class retained longer visibly stretches further than its peers. When every class in a category shares the same TTL pair, the rows collapse to All records (5) / All metrics (4) — the page never renders nine identical bars. The page branches sharply by backend. BanyanDB shows the full lifecycle bar + stage vocabulary; on any other backend, the page renders a single Retention pane with per-class values and skips the stage vocabulary entirely. A footer note names the wire-level truth: OAP\u0026rsquo;s TTL response collapses hot + warm into one number per class, BanyanDB migrates between stages in segments so records near a boundary may briefly exist in both, and property data is omitted (forever-retained, no TTL reported). Time picker Custom range seeds from the last applied range when you re-open the picker, instead of resetting to \u0026ldquo;half the max ending now\u0026rdquo;. Reopening also auto-expands the Custom form on the matching precision tab when the current range is custom. Locale-bleed fix on the alarms page custom-range stamp and the log row date column (was rendering 5月08日 on zh-CN browsers; now uniform MM-DD). Overview widgets follow the global time picker The Services Dashboard (and any overview using metric / KPI / table widgets) now honors the topbar time picker. Previously the per-layer landing and topology routes were hardcoded to the last 60 minutes, so picking 12 days back kept showing recent numbers; now picker + Cold pill flow end-to-end. The layer dashboard\u0026rsquo;s header KPIs follow the picker too (was showing live numbers while the body honored the picker). The Active alarms widget title now shows the actual window (e.g. · last 10m) and the empty-state copy uses the same value instead of a hardcoded \u0026ldquo;last 60m\u0026rdquo;. Polish Sentence-case fixes on a couple of leftover Title-Case labels (DSL management, Metrics inspect) so the menu and roles tables read consistently; acronyms (DSL / OAP / MAL / LAL / OAL) stay uppercase. Public-demo (demo.skywalking.apache.org) references removed from setup docs — the demo doesn\u0026rsquo;t accept anonymous traffic. Dashboard authoring The Layer dashboards and Overview templates admin pages now share one editing model where your work-in-progress lives in your browser, and the live, shared version is whatever OAP serves.\nEdits are local to your browser. \u0026ldquo;Save (local)\u0026rdquo; stores your draft in this browser only — it is never written to the server and nobody else sees it. Everyone (including you, in normal viewing) keeps seeing the published remote dashboard until you publish. Load any source into the editor. A single Reset to ▾ control loads the Bundled (shipped default) or Remote (OAP live) version into the editor; editing from there becomes your local draft. Preview any source on the real page. A Preview ▾ control opens the actual layer / overview page in a new tab rendering your Local draft, the Bundled default, or Remote — via ?mode=preview\u0026amp;source=…, which stays in the URL and propagates as you navigate the menu. A banner names what you\u0026rsquo;re previewing (dismiss with × or Esc). Publish with a diff. Check diff \u0026amp; push shows a side-by-side local→remote diff and publishes to OAP; it\u0026rsquo;s enabled only when your local draft actually differs from remote. Bundled can also be pushed straight to OAP. Resetting to remote clears the local draft. Reset to bundled then publishes correctly when the bundled default differs from remote — Save (local) and Check diff \u0026amp; push now compare the editor against remote, not just against what was first loaded, so a bundled-vs-remote divergence is no longer mistaken for \u0026ldquo;no changes\u0026rdquo; (layer + overview editors). Preview faithfully reflects your draft\u0026rsquo;s enabled components / menu labels — disabled tabs disappear and renamed nouns (\u0026ldquo;Nodes\u0026rdquo;, \u0026ldquo;Topics\u0026rdquo;) show through — without pushing anything to the server. Preview works even for layers OAP currently reports no services for. An editors-only reminder lists any unpublished local drafts with quick links to the relevant edit page (no more \u0026ldquo;use local vs remote\u0026rdquo; prompt — remote is always the live source). Create mirrors edit. \u0026ldquo;+ New dashboard\u0026rdquo; writes a local draft (the id is the template name, checked unique) — edit and preview it, then Check diff \u0026amp; push publishes it. A pushed dashboard with no bundled default is remote-only and now renders everywhere (live page + sidebar), not just in the editor. Delete = soft-disable (OAP has no hard delete). A local-only draft is removed from the browser; a dashboard on OAP is disabled — dropped from the picker\u0026rsquo;s live state, the sidebar, and the live page. A disabled status chip shows it. Confirmations are styled in-app dialogs, not the browser\u0026rsquo;s native box. Layer dashboards editor The layer picker is a single filterable dropdown showing alias + key + sync status. A live menu preview sits beside the Alias / Components / Menu labels (per-layer slot aliases) editor; clicking a menu item jumps to that component\u0026rsquo;s config. Scope tabs and section headings read in the layer\u0026rsquo;s own vocabulary. The service-list metrics editor gains a sample-data preview (plus a faithful landing KPI tile preview that reuses the real header components), the column remove-button alignment is fixed, and the landing KPI tile config makes clear it just picks which existing column feeds the headline + sparkline. The picker\u0026rsquo;s Diverged / Local filters now sit inside the dropdown (next to the search), and the editor header reads on one line — Layer: \u0026lt;name\u0026gt; \u0026lt;key\u0026gt; \u0026lt;status\u0026gt; — showing the same sync-status chip the picker does. Disable / Reactivate a layer. Disabling soft-disables the layer on OAP and drops it from the sidebar (the menu honors disabled templates); a disabled layer offers Reactivate, which re-enables it from the bundled default (the OAP update path clears the disabled flag). Overview templates editor Rebuilt as a layer-style canvas: a 12-column grid you drag to reorder and corner-drag to resize, click-to-edit in a right drawer that appears on selection (Esc / deselect hides it), with section-breaks and the dashboard title selectable and unselected widgets hinted by a dashed outline. The canvas mirrors the live grid (fixed row height), so the layout matches the real page — including side-by-side widgets like topology + alarms. The composite-metrics KPI editor stacks each row as a card (label / source / MQE / unit / aggr / style / max) so nothing truncates in the narrow drawer. Navigation \u0026amp; shell The main sidebar folds to a narrow rail to reclaim width; the logo moves into the topbar while folded (the original wordmark is unchanged). The active menu item now reliably expands, highlights, and scrolls into view on entering a page (route matching is case-insensitive). Fixed a regression where the sidebar scrolled to the very bottom on every navigation (the \u0026ldquo;Debug events\u0026rdquo; toggle\u0026rsquo;s active state was being treated as the scroll target). Overview dashboards appear in the sidebar only when their layers are reporting services. Visibility is derived from each dashboard\u0026rsquo;s widgets (their layer field) ∪ the explicit layers[] list, gated against the live availableLayers. A dashboard you create via \u0026ldquo;+ New\u0026rdquo; inherits this automatically — no need to maintain the layers[] field by hand. Polls on the 60s menu cadence + window focus, so entries appear / disappear as services start and stop reporting. Smarter landing. Root / cascades through a sensible chain so the user never sees a blank page: first available public overview → first layer with services → the empty landing (/landing-empty). The cascade only lands on destinations that are also in the sidebar — a bundled-but-inactive layer (no services yet) is deliberately not a fallback, since it would put the user on a page they can\u0026rsquo;t navigate back to via the menu. The empty page is also a real bookmarkable route, with two distinct copies — \u0026ldquo;No data is flowing yet\u0026rdquo; (no agents/receivers reporting) vs \u0026ldquo;No dashboard configured yet\u0026rdquo; (services exist but no overview is set up) — each with the right operations-team handoff and no action buttons (a viewer\u0026rsquo;s role doesn\u0026rsquo;t include the verbs the old buttons jumped to). Debug events panel now defaults OFF on every host (was on for localhost). Same baseline for operators and developers so reproductions match what operators see. Zipkin trace mode drops the per-layer service-KPI header — the Zipkin explorer is a self-contained, cross-service view. 3D Infrastructure Map A standalone, bird\u0026rsquo;s-eye view of the deployment at /3d/map: services render as cubes on stacked tier-planes (apps · service mesh · middleware · infra), each tier subdivided into per-layer zones with the layer\u0026rsquo;s brand mark stamped on its colored swatch. Drag to rotate, scroll to zoom, arrow keys / WASD to pan; click a cube for its detail card and a link into that layer\u0026rsquo;s dashboard.\nLive data windows. The map auto-refreshes every minute — per-cube traffic rolls up the last 2h of metrics (HOUR step) and alarmed services light up from the last 20m of alarms. A toolbar chip shows the active scopes (metrics 2h · alarms 20m · ↻ 1m). An alarmed cube burns red with a radiating ripple, matched to its service by (layer, name) so only the firing service in the right tier is flagged. Live topology. The deployment structure is read live from OAP rather than a bundled snapshot: each layer\u0026rsquo;s service roster and service map are fetched one at a time (low concurrency) and assembled into the scene, so the map is correct on any deployment. It refreshes on the same one-minute cycle — an unchanged structure updates metrics/alarms in place without disturbing the camera, while a service appearing or disappearing rebuilds the affected tier. The load progresses stage by stage in the status strip. Beacon mode. A toolbar toggle dims every healthy cube to a wireframe ghost and lets only alarming cubes glow, so the services that are firing jump out instantly during an incident. Logic groups. Related layers can be clustered into a single labelled block on a tier — the bundled config ships a Self-Observability group (OAP, Satellite, BanyanDB, and the Java / Go agents) on the middleware tier. Members keep their own cube colors but read as one block on the map. Configurable tiers + layers. Tier order, per-layer plane mapping, cube colors, the traffic MQE per layer, and the logic groups are all driven by the 3D map config, edited on a structured admin page at /admin/3d-map. Pin each layer to a tier (with a single global layer filter as the top-level gate), edit each layer\u0026rsquo;s color + traffic metric, manage logic groups (members, color, icon, tier), and choose the single failover tier for anything unpinned. The config is published to OAP and shared across the deployment the same way as dashboards: edits save to a local draft in your browser, then Check diff \u0026amp; push publishes to OAP — the map renders the remote, with the bundled defaults as fallback. Topology clustering. Within layers that carry a service map, services group into named clusters drawn as a wireframe frame with the cluster name baked into the frame\u0026rsquo;s lower-left corner — service-mesh services cluster by their showcase group, Kubernetes services by namespace. Clustering follows each layer\u0026rsquo;s naming rule, so layers without one keep rendering flat. Navigate by tier. The right-side Tiers panel is a two-level tree (tier → layer / logic-group). Clicking any entry resets the view, glides the camera to face that region, zooms in, and flashes the region for a few seconds so it is easy to spot. A Reset button in the panel header restores the initial framing. Hover = preview. Hovering a cube shows the same detail card as a click — tier, layer, service, and (when present) group and cluster — anchored at the cube, minus the open-in-dashboard link. Call-direction flow. Call relationships animate directional particles so the direction of traffic reads at a glance. Focus one layer. The per-layer service map gains a View in 3D link that opens the map focused on just that layer. Refresh countdown. The load status strip shows a live countdown to the next refresh, anchored to the stage that will run next. Reliability When the BFF is unreachable the UI now shows a clear \u0026ldquo;Cannot reach the server\u0026rdquo; message instead of the cryptic \u0026ldquo;body stream already read\u0026rdquo; — the API client reads each error response body once and surfaces the real status/text (or a wrapped network error). Server-global service-by-layer catalog. One singleton on the BFF (60s TTL + single-flight) now owns the listLayers + aliased listServices(layer) fan-out. The sidebar menu\u0026rsquo;s per-layer counts and the alarms layer-tagger share this one cache instead of each running their own poll, so OAP sees at most one fan-out per minute regardless of how many routes are polling — and the two views can no longer drift by 60s relative to each other. The brand link and the post-login redirect no longer resolve to an empty address — both now land on the operator\u0026rsquo;s actual landing route. Trace span detail now labels the span direction as Kind (the noun) rather than the mistranslated verb form, and long tag keys wrap inside the panel instead of overflowing their column. ","excerpt":"\u003ch1 id=\"060\"\u003e0.6.0\u003c/h1\u003e\n\u003cp\u003eThis release is the production-readiness pass for Horizon UI: every page now renders correctly …\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/changelog/0.6.0/","title":"0.6.0"},{"body":"0.6.0 Features Add the Satellite CRD, webhooks and controller Bugs Update release images to set numeric user id Fix the satellite config not support number error Use env JAVA_TOOL_OPTIONS to replace AGENT_OPTS Chores Add stabilization windows feature in satellite HPA documentation ","excerpt":"\u003ch2 id=\"060\"\u003e0.6.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd the Satellite CRD, webhooks and controller\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"bugs\"\u003eBugs\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eUpdate release images to set …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/latest/en/changes/changes-0.6.0/","title":"0.6.0"},{"body":"0.6.0 Features Add the Satellite CRD, webhooks and controller Bugs Update release images to set numeric user id Fix the satellite config not support number error Use env JAVA_TOOL_OPTIONS to replace AGENT_OPTS Chores Add stabilization windows feature in satellite HPA documentation ","excerpt":"\u003ch2 id=\"060\"\u003e0.6.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd the Satellite CRD, webhooks and controller\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"bugs\"\u003eBugs\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eUpdate release images to set …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/next/en/changes/changes-0.6.0/","title":"0.6.0"},{"body":"0.6.0 Features Add the Satellite CRD, webhooks and controller Bugs Update release images to set numeric user id Fix the satellite config not support number error Use env JAVA_TOOL_OPTIONS to replace AGENT_OPTS Chores Add stabilization windows feature in satellite HPA documentation ","excerpt":"\u003ch2 id=\"060\"\u003e0.6.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd the Satellite CRD, webhooks and controller\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"bugs\"\u003eBugs\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eUpdate release images to set …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/v0.11.0/en/changes/changes-0.6.0/","title":"0.6.0"},{"body":"0.6.1 Bugs Fix could not deploy metrics adapter to GKE ","excerpt":"\u003ch2 id=\"061\"\u003e0.6.1\u003c/h2\u003e\n\u003ch4 id=\"bugs\"\u003eBugs\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix could not deploy metrics adapter to GKE\u003c/li\u003e\n\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/latest/en/changes/changes-0.6.1/","title":"0.6.1"},{"body":"0.6.1 Bugs Fix could not deploy metrics adapter to GKE ","excerpt":"\u003ch2 id=\"061\"\u003e0.6.1\u003c/h2\u003e\n\u003ch4 id=\"bugs\"\u003eBugs\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix could not deploy metrics adapter to GKE\u003c/li\u003e\n\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/next/en/changes/changes-0.6.1/","title":"0.6.1"},{"body":"0.6.1 Bugs Fix could not deploy metrics adapter to GKE ","excerpt":"\u003ch2 id=\"061\"\u003e0.6.1\u003c/h2\u003e\n\u003ch4 id=\"bugs\"\u003eBugs\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix could not deploy metrics adapter to GKE\u003c/li\u003e\n\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/v0.11.0/en/changes/changes-0.6.1/","title":"0.6.1"},{"body":"0.7.0 Browser errors \u0026amp; source maps New \u0026ldquo;Browser Logs\u0026rdquo; tab on the BROWSER layer — lists the JS error logs the browser agent reports (message, category, page, app version, time, and the minified line:col), filterable by category and time window. Expanding a row shows the raw stack alongside a de-obfuscated view. Source-map de-obfuscation (issue #6784). Upload a .map file from the tab and resolve any error\u0026rsquo;s minified stack back to the original source — file, line, column, symbol name, and a source snippet — by picking which map to apply. Maps are held in the BFF\u0026rsquo;s memory only (no backend storage): they\u0026rsquo;re surfaced as temporary, evicted least-recently-used when the configured budget is hit, and lost on restart. For durable provisioning, mount .map files into the server\u0026rsquo;s static source-map directory (HORIZON_SOURCEMAPS_DIR, /app/sourcemaps in the image) — those reload automatically and can\u0026rsquo;t be deleted from the UI. Budgets are configurable via the new sourceMaps block in horizon.yaml (per-file and total in-memory caps; defaults 64 MiB / 512 MiB). Upload/delete require the new source-map:write permission; viewing + resolving ride on browser-errors:read. Layers Split a layer\u0026rsquo;s menu by service group. A new per-layer Split menu by service group toggle (Layer dashboards admin, right after Alias; default off) fans the layer out into one level-0 sidebar entry per OAP Service.group — the \u0026lt;group\u0026gt;:: prefix. The entry\u0026rsquo;s display name leads with the group so it reads everywhere (sidebar, page header, KPI tile) and survives narrow-sidebar truncation — e.g. agent · General Service. Each entry is scoped to its group: the service header + its picker, the topology map + its in-box selector, the dashboards, and the service roster all show only that group\u0026rsquo;s services. A layer\u0026rsquo;s group entries stay contiguous in the sidebar (sorted by group), and the cross-group view returns by turning the toggle back off (off = one combined entry holding all groups). The group value is OAP-supplied data and is shown verbatim (not translated). Travels with template export/import like every other layer setting. The navigation sidebar is now resizable — drag the divider between the sidebar and the page to widen or narrow it (double-click the divider to reset to the default width); the chosen width persists per browser. Useful when long entries — like group-split names (agent · General Service) or deep namespaces — would otherwise truncate. The per-layer service picker shows each service\u0026rsquo;s group. When a layer has no topology-cluster naming rule, the service-list rows now surface the OAP \u0026lt;group\u0026gt;:: prefix (e.g. agent) as the group chip — so the group is visible there as it already is on the topology map. Every layer OAP reports now appears in the sidebar, including ones with no Horizon template (they render with default capabilities — a plain Service page). The previous hard-coded hidden-layer list (which dropped BanyanDB) is gone; a layer is hidden only when an admin explicitly disables its template, or when it is listed in the new config-driven layers.excluded block in horizon.yaml (defaults: FAAS and VIRTUAL_GATEWAY; clear the list to surface every reported layer). The admin Layer dashboards page is now layer-list-oriented. It lists every available layer — not just the ones shipping a bundled JSON or living on OAP. A layer with no template yet opens on a blank default you can configure (components, metric columns, widgets, topology) and Save, which publishes the template to OAP on first save. No per-layer JSON has to be shipped for a layer to be configurable. The picker gains a Not configured filter (beside Diverged / Local) and the sync banner spells out \u0026ldquo;N templates match bundled defaults · M layers not configured yet\u0026rdquo;. Removed the legacy per-layer overview block from every bundled layer template (and its translation overlays). It no longer rendered anything — the standalone Overview Dashboards replaced the old per-layer Overview tile — so it was dead config; the per-layer KPI strip is driven by the layer-header columns. Airflow monitoring layer (SWIP-7) New Airflow layer under Workflow Scheduler — service dashboard (scheduler / executor / pool KPIs and trends), Components dashboard (per-host scheduler and triggerer metrics for Airflow 3.x native OTel), and a 3D Infra Map load ring for Tasks Executable. Pairs with OAP backend SWIP-7 (meter_airflow_* / meter_airflow_instance_*). BanyanDB self-observability layer (SWIP-15) New BanyanDB layer under Self-Observability, modeling a clustered, role- and tier-aware BanyanDB deployment scraped through its FODC proxy. The cluster is one Cluster (service), each container is one Container (instance, carrying its container_name role and node_type tier as attributes), and each storage Group is an endpoint: Cluster dashboard — write / query / error-rate KPIs, CPU / memory / disk capacity, a throughput + errors trend, and a Containers by Role table. Container dashboard adapts to the selected container\u0026rsquo;s role: every container shows CPU / memory / Go-runtime resources (and, where the system collector runs, uptime / disk / network); a liaison adds ingestion, query, gRPC errors, the tier-2 publish pipeline and write-queue depth; a data node adds storage totals, merge/compaction, inverted index, subscribe queue and retention; a lifecycle sidecar shows migration cycles and last-run time / status. Liaison and data panels are gated on the container\u0026rsquo;s role attribute; resource panels the lifecycle sidecar doesn\u0026rsquo;t emit, and the lifecycle migration panels themselves, self-gate on data presence so they surface only once that container actually reports them (the lifecycle panels stay hidden until the first migration cycle runs). This template targets the clustered model; a single-process standalone BanyanDB (container_name=standalone) shows the shared resource / Go panels but not the role-specific ingestion / storage panels — those extend to standalone once the entity-gate membership operator (SWIP-15 §6) lands. Group dashboard — metrics split per data-model (measure / stream / trace / property): each model gets write rate, query latency, stored data, merge rate / latency / partitions, series write + term-search and total series, plus the type-agnostic subscribe / publish queue (throughput, p99, batch + message rate, publish bytes). Because a BanyanDB group stores one catalog, only the matching model\u0026rsquo;s panels render for a given group — gated by the model\u0026rsquo;s series-count flag — so a measure group shows the measure panels, a property group its index-write / merge / term-search / series panels, and so on. A Deployment tab renders the cluster\u0026rsquo;s container inventory — every container grouped into its node\u0026rsquo;s role/tier box (liaison, data hot/warm/cold) and its pod, with per-role health metrics (liaison query rate + gRPC errors, data ingest rate + disk usage, lifecycle migration cycles + last-run status). The node health-ring legend names each role\u0026rsquo;s own ring metric and its colour-band thresholds (driven by the layer template) instead of a single shared, hard-coded one. Container-to-container call edges carry role-pair-specific metrics off the SWIP-15 instance-relation families: a liaison → data edge shows write / query / part-sync throughput + p99 (one per queue operation), a liaison → liaison edge shows write-forward + control, and a lifecycle → data edge shows tier-migration volume / rate / p99. The edge prints up to 3 of the pair\u0026rsquo;s metrics inline (short aliases like W / R, flowing onto one line or stacking by edge length); the selected-edge panel keeps the full client | server breakdown, and the Flows sub-tab tables every edge per role-pair. Edges render once the OAP build includes the SERVICE_INSTANCE_RELATION scope (the migration_* family also needs the lifecycle sidecar reporting); until then the tab shows the inventory without edges. The whole deployment model — clustering / grouping rules, per-role node metrics, and role-pair edge metrics — is editable from the Layer dashboards admin → Deployment scope. Pairs with OAP backend SWIP-15 (meter_banyandb_* cluster / meter_banyandb_instance_* container / meter_banyandb_endpoint_* group). Queue-batch and lifecycle last-run panels appear once the cluster runs a BanyanDB build that emits those metrics. Dashboard widget value formatting Card widgets gain a enum format with a value→label map: a coded metric (e.g. a 1/0 success gauge) renders a readable label (1 → OK, 0 → Failed) instead of the raw number. Labels are translatable per locale (BFF-side template i18n overlay) and the map is editable in the Layer dashboards admin. BanyanDB\u0026rsquo;s lifecycle Last Sync card uses it. New duration format renders a SECONDS metric as a human time-ago (5m 20s ago; compact 5m / 2h on axes) — used by BanyanDB\u0026rsquo;s Time Since Last Sync card. Record widgets — jump to trace \u0026amp; copy Record widgets now drill into the originating trace. Each sampled row gets a jump-to-trace icon at the row head — shown only when the sample actually carries a trace id (these are sampled, so it can be absent) — that opens the trace waterfall in the global popout. It resolves the trace by id, not by layer, so it works even though the trace belongs to the calling service on a different layer (a virtual-target layer has no traces tab of its own). The statement text itself is click-to-copy. For example, the Slow Statements record widget on a Virtual Database / Cache / MQ service. Instance-list badge The badge on each row of the instance list (Containers / Pods / Nodes / …) is now configurable per layer (instances.badge on the layer template) — it can show any instance attribute instead of the fixed agent language. BanyanDB shows container_name (liaison / data / lifecycle), the role that actually distinguishes a container; agent-traced layers keep language (Java / Go / …). The badge is now hidden when the value is empty or UNKNOWN, so OpenTelemetry-scraped layers (which report no agent language) drop the meaningless UNKNOWN chip across the board. Dashboard widget visibility Layer-dashboard widgets gain a structured Visible when gate (Layer dashboards admin → widget drawer) so a widget only renders when it\u0026rsquo;s relevant to the selected entity. Two kinds: MQE metric — show the widget only when an expression has value, or when any value is \u0026gt; / \u0026lt; a threshold. Naming the widget\u0026rsquo;s own metric self-gates it (the JVM widgets appear only on JVM instances, the MQ widgets only on MQ producers, …); naming a different metric gates a whole group on one shared signal — that metric is checked once and the entire group\u0026rsquo;s queries are skipped when it\u0026rsquo;s empty, so e.g. a non-JVM instance no longer runs the JVM widget queries at all. Entity attribute — on the Instance scope, gate on the selected instance\u0026rsquo;s attributes, e.g. language equals JAVA (case-insensitive) or an attribute simply being present. Service / Endpoint entities carry no attributes, so entity gates are ignored on those scopes. Gates are evaluated server-side; gated-out widgets just don\u0026rsquo;t appear in the grid. Note: a layer dashboard saved before this release that used the old free-text predicate loses its gate (the widget renders ungated) until you re-set the gate in the new editor and save the dashboard. Topology node filter \u0026amp; component icons The per-layer Topology map (and the embedded topology widget on the Services / Mesh overview dashboards) gains a Filter control to hide the conjectured peers that clutter a dense map. One auto-derived facet — by layer, presented exactly as the sidebar shows it: each row carries the layer\u0026rsquo;s own icon and its localized display name (General Service, Virtual Database, Java Agent, …), plus an Others bucket for nodes OAP couldn\u0026rsquo;t resolve, alongside a standalone User toggle. The layer rows self-populate from whatever the map currently shows and re-derive on every refresh / depth / time change. Unchecking a row hides those nodes and their now-dangling edges; the Others bucket is where uninstrumented \u0026ldquo;undefined\u0026rdquo; peers (e.g. a bare rcmd:80) land, so one click clears them, while your real databases / queues / caches — separated by their own VIRTUAL_* layer rows — stay on the map. Filtering is client-side and defaults to showing everything. Technology component icons on the nodes. Service-map nodes now render the icon for their detected component — the same icon set the trace waterfall uses, so a PostgreSQL node looks like PostgreSQL — falling back to the generic service / external / user glyph when the component ships no icon or couldn\u0026rsquo;t be resolved. The topology\u0026rsquo;s service selector (the \u0026ldquo;All services\u0026rdquo; picker) now groups its list by service group — OAP\u0026rsquo;s Service.group (the \u0026lt;group\u0026gt;:: prefix, e.g. agent) shown under a value-first \u0026lt;name\u0026gt; [GROUP] header — so a layer whose services share a group reads grouped instead of as one flat list. This is a per-service attribute and needs no per-layer naming-rule setup; services with no group stay in a single header-less section. Clicking a group header batch-selects or unselects every service in that group — the header carries a filled / half / hollow marker for all / some / none of its services focused. Instance topology The per-layer Topology map gains an instance map drill-down on layers that enable instance topology. Click a call between two services and then Instance map → to open it: the instances of each service as two columns (left = client, right = server) with the instance-level calls between them — pan/zoom, animated client→server flow, the same node health-ring + per-call client/server metric sidebar the service map uses, and a node popover with Open instance dashboard. A back button returns to the service map; a toolbar pair-picker swaps the two services. The two service pickers are relationship-aware, drawn from the service-topology call graph (including conjectured / cross-layer callees like rcmd:80, named the same as on the service map): the server list is the chosen client\u0026rsquo;s callees and the client list is the chosen server\u0026rsquo;s callers, each re-deriving when the other changes without resetting your current pick. A side the graph leaves no real choice for (e.g. a single caller) shows as plain text instead of a one-option dropdown. Each service\u0026rsquo;s instances sit inside a labelled grouping box — named with the service, using the same \u0026lt;group\u0026gt;:: prefix handling as the service map so a name reads identically on both — and a ring-colour legend explains what the node health bands (green → red) mean for the configured ring metric. Labels follow the layer\u0026rsquo;s own terms (e.g. Pods on Kubernetes, Sidecars on the data plane). Configurable like the service map. The Layer-dashboards admin → Topology scope now has an Enable instance topology toggle and its own node / server-edge / client-edge metric editors, kept visually separate from the service-topology metrics so the two are never confused. Enabled out of the box on General, Service Mesh, Kubernetes Service, and Cilium Service; the config rides each layer\u0026rsquo;s topology template (so it travels with template export/import). When OAP\u0026rsquo;s template store is unreachable, the instance map now shows the same empty + connectivity-banner state as the service map, rather than a misleading \u0026ldquo;not supported\u0026rdquo; — block and unsupported are no longer conflated. Localized across all eight UI languages. The instance-map UI, the template-store-unreachable banner, and the remaining alarm / live-debugger strings are now translated in zh-CN, ja, ko, es, pt, de and fr (English stays the source) — no feature renders English-only for non-English operators. Lock \u0026amp; compare entities on a layer dashboard Lock several services, instances, or endpoints — including ones from different services — and compare them in place. Compare is standard on every service / instance / endpoint layer dashboard — no flag, nothing to enable. Pin entities from the service picker or the instance / endpoint list; instance and endpoint pins are cross-service, so instances belonging to different services can be compared side by side. A persistent, scope-aware comparison bar shows the cohort regardless of how the underlying list paginates or which entity is currently selected. The entity you\u0026rsquo;re viewing is always part of the comparison — it appears first, tagged CURRENT in the accent color (and still drives the header KPIs); pinned entities add to it, each in its own stable hue (up to six pins). The comparison-bar chips are display-only: clicking a chip never changes what you\u0026rsquo;re viewing (no disruptive reload) and × unpins — switch the focused entity from the top selector / list as usual. Each widget compares inline in its own tile — line widgets overlay one hued series per entity; card widgets show one row per entity; top-N and record widgets get per-entity tabs plus a merged \u0026ldquo;All\u0026rdquo; tab; table widgets gain an Entity column that groups rows by entity and folds each entity\u0026rsquo;s long tail into one (others) row (summed for counts, count-only for latencies / percentiles where a sum would mislead). With nothing locked, every page renders exactly as before. Labeled series lead with the meaningful dimension — a multi-label series reads \u0026lt;label\u0026gt; · \u0026lt;service\u0026gt; · \u0026lt;instance\u0026gt; so the label (e.g. a JVM thread state) survives when a long instance / endpoint id has to truncate; entities stay distinguishable by color. A widget that only some compared entities expose (e.g. JVM widgets when a Java service is pinned alongside a non-JVM one) shows whenever any compared entity has it. Progressive, per-entity loading — each entity loads as its own request; tiles fill in as entities arrive and one slow or failed entity never blanks the others. The Topology, Deployment, trace, and log pages are unaffected — comparison applies to the service / instance / endpoint dashboards only. Widget tooltips and legends show the widget title, never the raw MQE expression, for un-labeled single-series line widgets; the multi-series tooltip is a fixed, aligned table — the entity name truncates and the values form one clean right-aligned column with the unit in the header. Charts \u0026amp; bundled metrics Large numbers on chart axes and tooltips now use compact SI suffixes (45.1k, 1.34M, 2.5G) instead of scientific notation (4.51e4), which operators found hard to read. Go runtime \u0026ldquo;Metadata Mspan\u0026rdquo; and \u0026ldquo;Metadata Mcache\u0026rdquo; widgets now report KB, fixing values that were displayed as a mislabeled \u0026ldquo;MB\u0026rdquo; of raw bytes (≈1000× too large) — they now read in the same KB scale as the other Go metadata-size widgets. Deployment New per-layer Deployment tab — the deployment topology of all of a service\u0026rsquo;s instances: the instance-to-instance call graph within a single service. Where the instance map drills into the instances between two services, this shows how one service\u0026rsquo;s own instances are deployed and talk to each other (e.g. a clustered store\u0026rsquo;s nodes calling each other). Pick a service from the layer\u0026rsquo;s Service header and the tab draws its instances as health-ring nodes with the intra-service calls between them — pan/zoom, animated edge flow, the per-call client/server metric sidebar, and a node popover that shows the instance\u0026rsquo;s attributes and an Open instance dashboard link. Self-calls and back-and-forth pairs are drawn distinctly. Node clustering. Instances can group into labelled boxes by a single instance attribute (e.g. role / tier), by several attributes combined into one key (e.g. node_role + node_type, where an attribute absent on a node drops out — so a BanyanDB cluster splits its data nodes into hot / warm / cold boxes while the liaison nodes, which carry no tier, stay one box), or by a name regex run on the instance name — so a fleet of mixed-role nodes reads as one box per role instead of a flat cloud. The boxes lay out left→right along the calls between them, so an upstream→downstream chain reads in order. Optional + configurable. Off by default for every layer; a layer opts in from the Layer-dashboards admin → Deployment scope, which has its own node / server-edge / client-edge metric editors (instance scope) plus the clustering-rule picker. The config is a self-contained block on the layer template, so it travels with template export/import and is independent of the service-map topology config. Pod / sibling model. Instances render as hexagons and can bundle into pods: a pod\u0026rsquo;s main container is a full hex with its sibling containers attached as smaller hexes around its edges. Three independent rules drive it — cluster (the dashed boxes), sibling (which containers form one pod), and role (per-container-type metrics + which container is the main). Edges resolve to the exact container, so cross-pod sidecar links (e.g. a lifecycle agent calling its peer in another pod) connect the small hexes. The model can be previewed before real data exists via the admin\u0026rsquo;s draft Preview flow — edit the Deployment scope and preview the live page without publishing. Tiered layout + draggable pods. Each cluster box lays its pods out by call depth — sources on the left, the pods they call to the right — so a hot → warm → cold lifecycle chain reads as left-to-right tiers. Pods stack vertically within a tier, and a tier with more than four pods wraps into additional stacked columns of four. Drag any pod to rearrange; its cluster box re-flows to keep every node enclosed. Role-to-role edge metrics + a Flows view. Deployment edge metrics are keyed by the (source-role → target-role) pair, so each kind of link shows its own metrics rather than one flat set — a liaison → data edge can surface write / query / part-sync throughput while a lifecycle → data edge surfaces migration volume. Pairs match most-specific-first with a * wildcard fallback. Each pair names a primary metric that prints inline on the edge in the map, so the headline number reads at a glance without opening anything; the selected-edge sidebar shows that pair\u0026rsquo;s full metric set. A new Flows sub-tab (next to Topology) lays the edges out as one aligned table per role-pair — click a row to jump to that edge in the graph. API dependency The per-layer API dependency tab renders an endpoint\u0026rsquo;s caller → callee chain as a graph. Pick an endpoint and it lays out in columns by direction — callers on the left, the focus endpoint in the centre, callees on the right — with the same node health-ring border, SLA-coloured RPM, and latency you read on the service map; edges animate the call direction and label the heaviest by RPM. Expand to walk the chain. A selected endpoint shows a single + handle that pulls in its own callers and callees in one click (new callers land left, callees right). The handle spins while the dependency query is in flight; when an endpoint is a leaf with nothing further to load it fades and a brief banner says so — a silent \u0026ldquo;nothing happened\u0026rdquo; never reads as a bug. Rearrange freely. Drag any node box to pull a dense graph apart — edges follow live. Pan, wheel-zoom, and a fit button act on the whole canvas, and a node holds a steady on-screen size whether or not the detail sidebar is open. Drill straight out, in a new tab. The node detail\u0026rsquo;s Open endpoint and Service →, and the service-map node/edge jumps (Open service, API map →, Instance map →), now open in a new browser tab — so you keep the graph you\u0026rsquo;re exploring while the drill-down opens alongside it. Nodes share the service-map\u0026rsquo;s visual vocabulary (SLA-band border, an agent badge on instrumented endpoints, the focus star), and the tab is localized across all eight UI languages. Dashboard template portability Every template admin page — Overview templates, Layer dashboards, and the 3D-map config — now has Export and Import actions. Export downloads the in-use version (what end users render: the version live on OAP, or the bundled default when OAP has none) as a JSON file, for backup, sharing, or moving a dashboard to another OAP. Import reads a JSON file, validates it, and loads it as a local draft in this browser — preview it, then publish with “Check diff \u0026amp; push” as usual. Importing never writes OAP directly. Overview import can recreate a deleted dashboard or seed a brand-new one; layer import targets a layer already present on this deployment. The Translations page has matching Export / Import, scoped to the current language: export the in-use translation for a template + locale as a JSON file, or import one as a local draft to review and push. (Source templates and their translations are edited on separate pages, so their import/export are separate too — each on its own page.) Template store reliability Runtime config is strictly what\u0026rsquo;s on OAP. Layer dashboards, overviews, and topology now render only the version published to OAP\u0026rsquo;s UI-template store (or the in-code minimal default for a layer that has none). The disk-bundled templates reach a running UI only by being synced to OAP (first boot / admin reset) or through the admin Preview button — they are never a silent live fallback. So an operator always sees the live published config, not a stale bundled copy masquerading as current. Unreachable template store is a visible block, not a quiet fallback. When OAP\u0026rsquo;s UI-template host can\u0026rsquo;t be reached, a banner (same red treatment as the OAP-query-unreachable strip) reports it, and the dashboard / overview / topology surfaces stay empty rather than back-filling bundled defaults that could be read as real. The sidebar still navigates so the rest of the app is reachable. The admin Preview button now drives every template-rendered page — the overview detail view and the per-layer topology (incl. the instance map), API dependency, traces, and network-profiling pages — not just the layer dashboards. Previewing renders the draft\u0026rsquo;s metrics/config against live OAP, so an edit to topology or dependency metrics is visible before you publish. Preview and the absent-remote path stay strictly separate: a draft renders only in ?mode=preview; normal reads never carry one. Editors no longer silently fall back to the bundled default. When a layer / overview / translation has no version published to OAP, the editor shows a \u0026ldquo;No published version on OAP\u0026rdquo; panel instead of quietly loading the shipped bundled copy as if it were live. Bundled now reaches the editor only when you click Reset to bundled — matching the runtime, which renders the published version or blocks, never the bundle. Layer landing \u0026amp; service list The layer landing now shows your services, not just an arbitrary 25. It used to cap the metric fan-out at the first 25 services by list order — so larger layers hid the rest, and the \u0026ldquo;top\u0026rdquo; services weren\u0026rsquo;t even the true top (the cap happened before the ranking). Now it probes all services up to a configurable cap and, when a layer exceeds it, runs a cheap single- metric ranking pass to pick the true top-N by the landing\u0026rsquo;s order-by column. The service picker surfaces \u0026ldquo;top N of M\u0026rdquo; so the trim is never silent. Queries drain through a bounded-concurrency pool, so a big layer fans out in controlled waves rather than a thundering herd. New query.landingServiceCap in horizon.yaml (default 100) tunes how many services a landing probes per request — raise it if your OAP + storage can take the larger fan-out, lower it to protect a modest deployment. The service picker now lists the whole layer, not only the metric-probed top-N. Services that ranked below the metric cap on the order-by column now appear as their own rows with low in that column (and — for the others, which were never probed) instead of being hidden — every service stays browsable, searchable, and selectable regardless of the cap. The header chip reads \u0026ldquo;metrics: top N\u0026rdquo; to make the metric trim explicit. Removed the stale \u0026ldquo;Landing KPI tile\u0026rdquo; controls (Headline / Trend line) from the Layer-dashboards admin. They no longer matched the rendered layer header — which shows every configured metric column as its own KPI with its own trend line — so editing them changed nothing on screen. The header is driven entirely by the service-list columns + default sort; the preview now reflects that. Selecting a low-traffic (below-cap) service now works on every tab, not just the dashboard. Logs, traces, and endpoint-dependency resolved the picked service\u0026rsquo;s name from the landing sample only — so a tail service queried as blank (and Logs even snapped the pick back to the top service). All per-layer tabs now resolve the name from the full roster, so a low service drills in everywhere. Profiling scopes no longer show an editor grid that goes nowhere. Trace / eBPF / async profiling are built-in runtime views with nothing to author, so the admin now shows a \u0026ldquo;configured at runtime\u0026rdquo; note for them instead of a widget grid whose widgets never rendered. Access control \u0026amp; permissions The Roles \u0026amp; Permissions board now lists infra-3d:read — the permission to view the 3D Infrastructure Map — under the data-catalog group, with a matching \u0026ldquo;3D infrastructure map\u0026rdquo; row in the menu-visibility matrix. It was already enforced and granted to every built-in role (viewer and up), but it never appeared on the board, so an admin couldn\u0026rsquo;t see who held it. Editing a layer dashboard template is now gated on the dashboard:write permission the editor already advertises; publishing overview, alert, and 3D-map configs stays on overview:write. The required permission is resolved per template kind at save time. Built-in roles are unaffected (operator and admin hold both), but a custom role granted only dashboard:write can now save layer dashboards. The Cluster Status debug view (/api/debug/status) now requires only live-debug:read. It previously also demanded cluster:read, so a role granted live-debug access but not cluster-read was wrongly blocked. Saving a local draft of a template (the \u0026ldquo;Save local\u0026rdquo; action) now enforces the same per-kind permission as publishing — a layer draft needs dashboard:write, other kinds overview:write — instead of a blanket overview:write. Performance hardening Layer dashboards skip a redundant service lookup on every load. The dashboard route used to issue its own listServices to auto-pick the service and carry its entity-scope flags; it now reads the shared per-layer service catalog the sidebar already keeps warm, so a dashboard\u0026rsquo;s first paint costs one fewer OAP round-trip in the common case. It still falls back to a live lookup for a just-registered service, a cold snapshot, or to surface an OAP outage (the \u0026ldquo;OAP unreachable\u0026rdquo; state now follows the actual widget fetch, so a warm cache can\u0026rsquo;t mask a backend that\u0026rsquo;s gone away). The alarms list and count fire their two startup probes in parallel. The server-time offset and backend-capability probes that precede every alarms query now run concurrently instead of one-after-the-other. The 3D Infra Map loads its metrics in parallel. Per-node metric values used to fetch one batch at a time; they now load in bounded-concurrency batches, so the load rings and traffic values fill in sooner on large layers. A new metricConcurrency setting in the Infra Map config (default 4) caps how many metric batches run at once. Oversized layer topologies fail with a clear message instead of an unreadable map. When a layer\u0026rsquo;s service graph exceeds the render ceiling (5,000 services or 15,000 calls), the service map shows a \u0026ldquo;Topology too large to render\u0026rdquo; notice with the live counts and a hint to pick a specific service or lower the depth, rather than attempting to lay out a graph too dense to read. Partial metric-load failures are now surfaced on every topology map. If some metric batches fail to load (a transient OAP error) on the service map, instance topology, deployment, or endpoint-dependency map, a banner now explains that blank values may be unavailable rather than zero — and on the endpoint-dependency map, that some endpoints or links may be missing — so a backend hiccup isn\u0026rsquo;t misread as real \u0026ldquo;no traffic\u0026rdquo; data. Fixes Metrics Inspect — the crosshair value tooltip is no longer clipped behind the navigation sidebar when you hover near a widget\u0026rsquo;s left edge; it now renders above the page chrome. 3D Infrastructure Map config — \u0026ldquo;Check diff \u0026amp; push\u0026rdquo; now requires saving a local draft first (it was selectable while edits were still unsaved), and the push dialog renders the side-by-side before/after JSON diff instead of an empty panel. The API-dependency tab now honors the topbar time picker. It was pinned to the last hour regardless of the selected range; changing the range (and expanding a node) now re-queries the chosen window, like the service map and instance map already did. A dashboard no longer blanks entirely when one metric group fails. A transient backend error (timeout / 5xx / query-complexity limit) on a single batch of widgets now marks only those widgets as failed; the rest of the dashboard renders normally instead of every cell going blank. Trace list rows pick the correct root span on BanyanDB. A multi-service trace could surface a downstream span\u0026rsquo;s endpoint / duration / start time in the list; the row now reliably reflects the trace\u0026rsquo;s true entry span. Correct timestamps right after repointing OAP. The server-timezone offset is now cached per OAP URL, so a configuration reload that switches to a different-timezone OAP re-probes immediately instead of serving the previous server\u0026rsquo;s offset for up to a minute. Baseline security response headers (X-Content-Type-Options: nosniff, X-Frame-Options: DENY, Referrer-Policy: no-referrer) are now sent on every response. Removed an internal ?mockTop= debug query parameter that padded top-N widgets with synthetic rows; it no longer ships in release builds. The profiling pages now use more of the page height. The Trace / eBPF / Async / pprof / network profiling layouts were sized off a viewport offset that over-counted the chrome above them, leaving dead space at the bottom on taller screens; they now extend closer to the bottom of the view. Overview dashboard templates are validated before save. A malformed overview (missing a required field, an unknown widget type) is now rejected with a clear field-level error instead of being written to OAP — restoring a guard that was lost when overview editing moved to the OAP-backed save path. Documentation \u0026amp; release tooling A further accuracy pass corrected the Cluster Status page (three panes — Query, Admin, Zipkin/OTLP — and no per-node member list), the Kubernetes readiness-probe guidance (point it at the public /api/health, not the authenticated /api/oap/info), the layer-template components default (only the service dashboard is on when a key is omitted) and the aliases authoring key, the removed visibleWhen free-text and embedded-i18n template shapes, and the data-retention cold-stage controls. The website docs were brought current with the 0.6.0 build and the configuration pages restructured around the admin UI — the JSON shape is now a reference appendix, not an authoring surface (these admin pages are structured editors, not raw-JSON editors). Accuracy fixes span the RBAC verbs (incl. infra-3d:read), the audit-log action set, the Metrics Inspect API paths, the layer-template component flags, and the redesigned 3D-map config + loading stages. A new docs/CLAUDE.md records the doc-writing principles, and the i18n docs gain a language × scope coverage matrix plus a translation step in the add-a-layer recipe. The container image is published to Docker Hub by CI on every v* tag; the post-vote finalize script now only verifies the published tags (the manual local-push fallback and Docker Hub login preflight were removed). Layer drill-down fixes The per-layer Instance and Endpoint pages now honor the layer\u0026rsquo;s configured aliases in their section headers and in the service-picker\u0026rsquo;s name column — e.g. ActiveMQ reads Brokers / Destinations and Virtual MQ reads Topics / MQ clusters, matching the sidebar — instead of the generic \u0026ldquo;Instance\u0026rdquo; / \u0026ldquo;Endpoint\u0026rdquo; / \u0026ldquo;Service\u0026rdquo; labels. Layers that define no alias still read the generic words. A layer\u0026rsquo;s Instance or Endpoint page no longer hangs on a perpetual \u0026ldquo;Reading data…\u0026rdquo; when the selected service reports no instances or endpoints (or a search matches nothing). It now shows the empty picker and renders the metric widgets in their normal \u0026ldquo;no data\u0026rdquo; state, so the layout stays visible and ready for services that do report them. Clearer cluster boundaries on every topology view. The dashed grouping boxes — namespaces on the service map, per-service boxes on the instance map, role/tier clusters on the Deployment tab — now draw with a bolder, brighter dashed border and a fully transparent background, so the boundary reads clearly on every theme (light themes included) instead of fading into the canvas. The Deployment tab also packs its cluster boxes evenly: boxes sit at a uniform spacing with no dead corridor between tiers and no blank strip before the first box. Live debugger MAL sample groups. A captured step that fans out to many samples no longer dumps every label set on screen: the samples are grouped by metric name into a one-line summary — \u0026lt;metric\u0026gt; · N samples · values=… — and you expand only the groups you care about to see each sample\u0026rsquo;s full labels. Groups are collapsed by default. Diff is the default when a group is expanded. A multi-sample group opens straight into diff view: the labels shared by every sample collapse into a dimmed \u0026ldquo;common\u0026rdquo; block and only the labels that differ are highlighted per sample — so it is immediate which label distinguishes each one (e.g. node_role / pod_name) and what value it maps to. A diff toggle beside the group\u0026rsquo;s header switches back to the full per-sample label list. The \u0026ldquo;common\u0026rdquo; set is computed across the whole group, not just the rendered rows. Multiple output entities collapse the same way. When a record materialises one metric for several entities (e.g. a per-endpoint write rate over sw_metricsMinute / sw_metricsHour / sw_metricsDay), the repeated meter cards fold into one block: a shared header (metric / function / time bucket), a N outputs · values=… summary, and a diff that surfaces only the entity fields that actually differ — whichever they are, not a fixed field — with each output\u0026rsquo;s value beside it. Readable sample values. Long fractional values from rate() / avg() (e.g. 57.0333333333…) are trimmed to a few significant digits for display so they stop overflowing the value column; integer counters still render exact, and the precise value stays available on hover. DSL management — live apply progress \u0026amp; recovery A structural rule change now shows live apply progress. Saving an edit that moves a metric\u0026rsquo;s storage shape (scope, downsampling, or the metric set) no longer just flashes \u0026ldquo;submitted\u0026rdquo; — the editor tracks the apply across the cluster through a phase stepper (Compiled → Confirming across the cluster → Committing → Done) and reports success only once OAP confirms the change is durable. Revert to bundled (also a schema change) goes through the same stepper. Body- and filter-only edits still apply instantly with no stepper. You can navigate away mid-apply; reloading the editor resumes the progress. \u0026ldquo;Applied — cluster propagation unconfirmed\u0026rdquo; is a warning, not an error. When a structural change is committed and durable but one or more nodes hadn\u0026rsquo;t confirmed the new schema within OAP\u0026rsquo;s fence budget, the editor names the lagging nodes and explains they self-converge on their next scan — the rule is applied, not rolled back. Reloading the editor reads it back as applied (from the stored rule). A failed apply is called out as rolled back — the cluster stays on the previous rule, the failure reason is shown inline, and your edit is kept in the editor so you can fix and save again. A compile error now surfaces as an inline diagnostic under the editor instead of a transient toast. Force re-apply to recover. A degraded or transiently-failed apply offers a one-click Force re-apply (recover) that re-runs the apply across the cluster to re-confirm the schema and un-stick any waiting node — gated behind a confirm that spells out it briefly pauses collection for that rule\u0026rsquo;s metrics, even when the content is unchanged. This subsumes the old Advanced force toggle for the recovery case. ","excerpt":"\u003ch1 id=\"070\"\u003e0.7.0\u003c/h1\u003e\n\u003ch3 id=\"browser-errors--source-maps\"\u003eBrowser errors \u0026amp; source maps\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eNew \u0026ldquo;Browser Logs\u0026rdquo; tab on the BROWSER layer\u003c/strong\u003e — …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/changelog/0.7.0/","title":"0.7.0"},{"body":"0.7.0 Browser errors \u0026amp; source maps New \u0026ldquo;Browser Logs\u0026rdquo; tab on the BROWSER layer — lists the JS error logs the browser agent reports (message, category, page, app version, time, and the minified line:col), filterable by category and time window. Expanding a row shows the raw stack alongside a de-obfuscated view. Source-map de-obfuscation (issue #6784). Upload a .map file from the tab and resolve any error\u0026rsquo;s minified stack back to the original source — file, line, column, symbol name, and a source snippet — by picking which map to apply. Maps are held in the BFF\u0026rsquo;s memory only (no backend storage): they\u0026rsquo;re surfaced as temporary, evicted least-recently-used when the configured budget is hit, and lost on restart. For durable provisioning, mount .map files into the server\u0026rsquo;s static source-map directory (HORIZON_SOURCEMAPS_DIR, /app/sourcemaps in the image) — those reload automatically and can\u0026rsquo;t be deleted from the UI. Budgets are configurable via the new sourceMaps block in horizon.yaml (per-file and total in-memory caps; defaults 64 MiB / 512 MiB). Upload/delete require the new source-map:write permission; viewing + resolving ride on browser-errors:read. Layers Split a layer\u0026rsquo;s menu by service group. A new per-layer Split menu by service group toggle (Layer dashboards admin, right after Alias; default off) fans the layer out into one level-0 sidebar entry per OAP Service.group — the \u0026lt;group\u0026gt;:: prefix. The entry\u0026rsquo;s display name leads with the group so it reads everywhere (sidebar, page header, KPI tile) and survives narrow-sidebar truncation — e.g. agent · General Service. Each entry is scoped to its group: the service header + its picker, the topology map + its in-box selector, the dashboards, and the service roster all show only that group\u0026rsquo;s services. A layer\u0026rsquo;s group entries stay contiguous in the sidebar (sorted by group), and the cross-group view returns by turning the toggle back off (off = one combined entry holding all groups). The group value is OAP-supplied data and is shown verbatim (not translated). Travels with template export/import like every other layer setting. The navigation sidebar is now resizable — drag the divider between the sidebar and the page to widen or narrow it (double-click the divider to reset to the default width); the chosen width persists per browser. Useful when long entries — like group-split names (agent · General Service) or deep namespaces — would otherwise truncate. The per-layer service picker shows each service\u0026rsquo;s group. When a layer has no topology-cluster naming rule, the service-list rows now surface the OAP \u0026lt;group\u0026gt;:: prefix (e.g. agent) as the group chip — so the group is visible there as it already is on the topology map. Every layer OAP reports now appears in the sidebar, including ones with no Horizon template (they render with default capabilities — a plain Service page). The previous hard-coded hidden-layer list (which dropped BanyanDB) is gone; a layer is hidden only when an admin explicitly disables its template, or when it is listed in the new config-driven layers.excluded block in horizon.yaml (defaults: FAAS and VIRTUAL_GATEWAY; clear the list to surface every reported layer). The admin Layer dashboards page is now layer-list-oriented. It lists every available layer — not just the ones shipping a bundled JSON or living on OAP. A layer with no template yet opens on a blank default you can configure (components, metric columns, widgets, topology) and Save, which publishes the template to OAP on first save. No per-layer JSON has to be shipped for a layer to be configurable. The picker gains a Not configured filter (beside Diverged / Local) and the sync banner spells out \u0026ldquo;N templates match bundled defaults · M layers not configured yet\u0026rdquo;. Removed the legacy per-layer overview block from every bundled layer template (and its translation overlays). It no longer rendered anything — the standalone Overview Dashboards replaced the old per-layer Overview tile — so it was dead config; the per-layer KPI strip is driven by the layer-header columns. Airflow monitoring layer (SWIP-7) New Airflow layer under Workflow Scheduler — service dashboard (scheduler / executor / pool KPIs and trends), Components dashboard (per-host scheduler and triggerer metrics for Airflow 3.x native OTel), and a 3D Infra Map load ring for Tasks Executable. Pairs with OAP backend SWIP-7 (meter_airflow_* / meter_airflow_instance_*). BanyanDB self-observability layer (SWIP-15) New BanyanDB layer under Self-Observability, modeling a clustered, role- and tier-aware BanyanDB deployment scraped through its FODC proxy. The cluster is one Cluster (service), each container is one Container (instance, carrying its container_name role and node_type tier as attributes), and each storage Group is an endpoint: Cluster dashboard — write / query / error-rate KPIs, CPU / memory / disk capacity, a throughput + errors trend, and a Containers by Role table. Container dashboard adapts to the selected container\u0026rsquo;s role: every container shows CPU / memory / Go-runtime resources (and, where the system collector runs, uptime / disk / network); a liaison adds ingestion, query, gRPC errors, the tier-2 publish pipeline and write-queue depth; a data node adds storage totals, merge/compaction, inverted index, subscribe queue and retention; a lifecycle sidecar shows migration cycles and last-run time / status. Liaison and data panels are gated on the container\u0026rsquo;s role attribute; resource panels the lifecycle sidecar doesn\u0026rsquo;t emit, and the lifecycle migration panels themselves, self-gate on data presence so they surface only once that container actually reports them (the lifecycle panels stay hidden until the first migration cycle runs). This template targets the clustered model; a single-process standalone BanyanDB (container_name=standalone) shows the shared resource / Go panels but not the role-specific ingestion / storage panels — those extend to standalone once the entity-gate membership operator (SWIP-15 §6) lands. Group dashboard — metrics split per data-model (measure / stream / trace / property): each model gets write rate, query latency, stored data, merge rate / latency / partitions, series write + term-search and total series, plus the type-agnostic subscribe / publish queue (throughput, p99, batch + message rate, publish bytes). Because a BanyanDB group stores one catalog, only the matching model\u0026rsquo;s panels render for a given group — gated by the model\u0026rsquo;s series-count flag — so a measure group shows the measure panels, a property group its index-write / merge / term-search / series panels, and so on. A Deployment tab renders the cluster\u0026rsquo;s container inventory — every container grouped into its node\u0026rsquo;s role/tier box (liaison, data hot/warm/cold) and its pod, with per-role health metrics (liaison query rate + gRPC errors, data ingest rate + disk usage, lifecycle migration cycles + last-run status). The node health-ring legend names each role\u0026rsquo;s own ring metric and its colour-band thresholds (driven by the layer template) instead of a single shared, hard-coded one. Container-to-container call edges carry role-pair-specific metrics off the SWIP-15 instance-relation families: a liaison → data edge shows write / query / part-sync throughput + p99 (one per queue operation), a liaison → liaison edge shows write-forward + control, and a lifecycle → data edge shows tier-migration volume / rate / p99. The edge prints up to 3 of the pair\u0026rsquo;s metrics inline (short aliases like W / R, flowing onto one line or stacking by edge length); the selected-edge panel keeps the full client | server breakdown, and the Flows sub-tab tables every edge per role-pair. Edges render once the OAP build includes the SERVICE_INSTANCE_RELATION scope (the migration_* family also needs the lifecycle sidecar reporting); until then the tab shows the inventory without edges. The whole deployment model — clustering / grouping rules, per-role node metrics, and role-pair edge metrics — is editable from the Layer dashboards admin → Deployment scope. Pairs with OAP backend SWIP-15 (meter_banyandb_* cluster / meter_banyandb_instance_* container / meter_banyandb_endpoint_* group). Queue-batch and lifecycle last-run panels appear once the cluster runs a BanyanDB build that emits those metrics. Dashboard widget value formatting Card widgets gain a enum format with a value→label map: a coded metric (e.g. a 1/0 success gauge) renders a readable label (1 → OK, 0 → Failed) instead of the raw number. Labels are translatable per locale (BFF-side template i18n overlay) and the map is editable in the Layer dashboards admin. BanyanDB\u0026rsquo;s lifecycle Last Sync card uses it. New duration format renders a SECONDS metric as a human time-ago (5m 20s ago; compact 5m / 2h on axes) — used by BanyanDB\u0026rsquo;s Time Since Last Sync card. Record widgets — jump to trace \u0026amp; copy Record widgets now drill into the originating trace. Each sampled row gets a jump-to-trace icon at the row head — shown only when the sample actually carries a trace id (these are sampled, so it can be absent) — that opens the trace waterfall in the global popout. It resolves the trace by id, not by layer, so it works even though the trace belongs to the calling service on a different layer (a virtual-target layer has no traces tab of its own). The statement text itself is click-to-copy. For example, the Slow Statements record widget on a Virtual Database / Cache / MQ service. Instance-list badge The badge on each row of the instance list (Containers / Pods / Nodes / …) is now configurable per layer (instances.badge on the layer template) — it can show any instance attribute instead of the fixed agent language. BanyanDB shows container_name (liaison / data / lifecycle), the role that actually distinguishes a container; agent-traced layers keep language (Java / Go / …). The badge is now hidden when the value is empty or UNKNOWN, so OpenTelemetry-scraped layers (which report no agent language) drop the meaningless UNKNOWN chip across the board. Dashboard widget visibility Layer-dashboard widgets gain a structured Visible when gate (Layer dashboards admin → widget drawer) so a widget only renders when it\u0026rsquo;s relevant to the selected entity. Two kinds: MQE metric — show the widget only when an expression has value, or when any value is \u0026gt; / \u0026lt; a threshold. Naming the widget\u0026rsquo;s own metric self-gates it (the JVM widgets appear only on JVM instances, the MQ widgets only on MQ producers, …); naming a different metric gates a whole group on one shared signal — that metric is checked once and the entire group\u0026rsquo;s queries are skipped when it\u0026rsquo;s empty, so e.g. a non-JVM instance no longer runs the JVM widget queries at all. Entity attribute — on the Instance scope, gate on the selected instance\u0026rsquo;s attributes, e.g. language equals JAVA (case-insensitive) or an attribute simply being present. Service / Endpoint entities carry no attributes, so entity gates are ignored on those scopes. Gates are evaluated server-side; gated-out widgets just don\u0026rsquo;t appear in the grid. Note: a layer dashboard saved before this release that used the old free-text predicate loses its gate (the widget renders ungated) until you re-set the gate in the new editor and save the dashboard. Topology node filter \u0026amp; component icons The per-layer Topology map (and the embedded topology widget on the Services / Mesh overview dashboards) gains a Filter control to hide the conjectured peers that clutter a dense map. One auto-derived facet — by layer, presented exactly as the sidebar shows it: each row carries the layer\u0026rsquo;s own icon and its localized display name (General Service, Virtual Database, Java Agent, …), plus an Others bucket for nodes OAP couldn\u0026rsquo;t resolve, alongside a standalone User toggle. The layer rows self-populate from whatever the map currently shows and re-derive on every refresh / depth / time change. Unchecking a row hides those nodes and their now-dangling edges; the Others bucket is where uninstrumented \u0026ldquo;undefined\u0026rdquo; peers (e.g. a bare rcmd:80) land, so one click clears them, while your real databases / queues / caches — separated by their own VIRTUAL_* layer rows — stay on the map. Filtering is client-side and defaults to showing everything. Technology component icons on the nodes. Service-map nodes now render the icon for their detected component — the same icon set the trace waterfall uses, so a PostgreSQL node looks like PostgreSQL — falling back to the generic service / external / user glyph when the component ships no icon or couldn\u0026rsquo;t be resolved. The topology\u0026rsquo;s service selector (the \u0026ldquo;All services\u0026rdquo; picker) now groups its list by service group — OAP\u0026rsquo;s Service.group (the \u0026lt;group\u0026gt;:: prefix, e.g. agent) shown under a value-first \u0026lt;name\u0026gt; [GROUP] header — so a layer whose services share a group reads grouped instead of as one flat list. This is a per-service attribute and needs no per-layer naming-rule setup; services with no group stay in a single header-less section. Clicking a group header batch-selects or unselects every service in that group — the header carries a filled / half / hollow marker for all / some / none of its services focused. Instance topology The per-layer Topology map gains an instance map drill-down on layers that enable instance topology. Click a call between two services and then Instance map → to open it: the instances of each service as two columns (left = client, right = server) with the instance-level calls between them — pan/zoom, animated client→server flow, the same node health-ring + per-call client/server metric sidebar the service map uses, and a node popover with Open instance dashboard. A back button returns to the service map; a toolbar pair-picker swaps the two services. The two service pickers are relationship-aware, drawn from the service-topology call graph (including conjectured / cross-layer callees like rcmd:80, named the same as on the service map): the server list is the chosen client\u0026rsquo;s callees and the client list is the chosen server\u0026rsquo;s callers, each re-deriving when the other changes without resetting your current pick. A side the graph leaves no real choice for (e.g. a single caller) shows as plain text instead of a one-option dropdown. Each service\u0026rsquo;s instances sit inside a labelled grouping box — named with the service, using the same \u0026lt;group\u0026gt;:: prefix handling as the service map so a name reads identically on both — and a ring-colour legend explains what the node health bands (green → red) mean for the configured ring metric. Labels follow the layer\u0026rsquo;s own terms (e.g. Pods on Kubernetes, Sidecars on the data plane). Configurable like the service map. The Layer-dashboards admin → Topology scope now has an Enable instance topology toggle and its own node / server-edge / client-edge metric editors, kept visually separate from the service-topology metrics so the two are never confused. Enabled out of the box on General, Service Mesh, Kubernetes Service, and Cilium Service; the config rides each layer\u0026rsquo;s topology template (so it travels with template export/import). When OAP\u0026rsquo;s template store is unreachable, the instance map now shows the same empty + connectivity-banner state as the service map, rather than a misleading \u0026ldquo;not supported\u0026rdquo; — block and unsupported are no longer conflated. Localized across all eight UI languages. The instance-map UI, the template-store-unreachable banner, and the remaining alarm / live-debugger strings are now translated in zh-CN, ja, ko, es, pt, de and fr (English stays the source) — no feature renders English-only for non-English operators. Lock \u0026amp; compare entities on a layer dashboard Lock several services, instances, or endpoints — including ones from different services — and compare them in place. Compare is standard on every service / instance / endpoint layer dashboard — no flag, nothing to enable. Pin entities from the service picker or the instance / endpoint list; instance and endpoint pins are cross-service, so instances belonging to different services can be compared side by side. A persistent, scope-aware comparison bar shows the cohort regardless of how the underlying list paginates or which entity is currently selected. The entity you\u0026rsquo;re viewing is always part of the comparison — it appears first, tagged CURRENT in the accent color (and still drives the header KPIs); pinned entities add to it, each in its own stable hue (up to six pins). The comparison-bar chips are display-only: clicking a chip never changes what you\u0026rsquo;re viewing (no disruptive reload) and × unpins — switch the focused entity from the top selector / list as usual. Each widget compares inline in its own tile — line widgets overlay one hued series per entity; card widgets show one row per entity; top-N and record widgets get per-entity tabs plus a merged \u0026ldquo;All\u0026rdquo; tab; table widgets gain an Entity column that groups rows by entity and folds each entity\u0026rsquo;s long tail into one (others) row (summed for counts, count-only for latencies / percentiles where a sum would mislead). With nothing locked, every page renders exactly as before. Labeled series lead with the meaningful dimension — a multi-label series reads \u0026lt;label\u0026gt; · \u0026lt;service\u0026gt; · \u0026lt;instance\u0026gt; so the label (e.g. a JVM thread state) survives when a long instance / endpoint id has to truncate; entities stay distinguishable by color. A widget that only some compared entities expose (e.g. JVM widgets when a Java service is pinned alongside a non-JVM one) shows whenever any compared entity has it. Progressive, per-entity loading — each entity loads as its own request; tiles fill in as entities arrive and one slow or failed entity never blanks the others. The Topology, Deployment, trace, and log pages are unaffected — comparison applies to the service / instance / endpoint dashboards only. Widget tooltips and legends show the widget title, never the raw MQE expression, for un-labeled single-series line widgets; the multi-series tooltip is a fixed, aligned table — the entity name truncates and the values form one clean right-aligned column with the unit in the header. Charts \u0026amp; bundled metrics Large numbers on chart axes and tooltips now use compact SI suffixes (45.1k, 1.34M, 2.5G) instead of scientific notation (4.51e4), which operators found hard to read. Go runtime \u0026ldquo;Metadata Mspan\u0026rdquo; and \u0026ldquo;Metadata Mcache\u0026rdquo; widgets now report KB, fixing values that were displayed as a mislabeled \u0026ldquo;MB\u0026rdquo; of raw bytes (≈1000× too large) — they now read in the same KB scale as the other Go metadata-size widgets. Deployment New per-layer Deployment tab — the deployment topology of all of a service\u0026rsquo;s instances: the instance-to-instance call graph within a single service. Where the instance map drills into the instances between two services, this shows how one service\u0026rsquo;s own instances are deployed and talk to each other (e.g. a clustered store\u0026rsquo;s nodes calling each other). Pick a service from the layer\u0026rsquo;s Service header and the tab draws its instances as health-ring nodes with the intra-service calls between them — pan/zoom, animated edge flow, the per-call client/server metric sidebar, and a node popover that shows the instance\u0026rsquo;s attributes and an Open instance dashboard link. Self-calls and back-and-forth pairs are drawn distinctly. Node clustering. Instances can group into labelled boxes by a single instance attribute (e.g. role / tier), by several attributes combined into one key (e.g. node_role + node_type, where an attribute absent on a node drops out — so a BanyanDB cluster splits its data nodes into hot / warm / cold boxes while the liaison nodes, which carry no tier, stay one box), or by a name regex run on the instance name — so a fleet of mixed-role nodes reads as one box per role instead of a flat cloud. The boxes lay out left→right along the calls between them, so an upstream→downstream chain reads in order. Optional + configurable. Off by default for every layer; a layer opts in from the Layer-dashboards admin → Deployment scope, which has its own node / server-edge / client-edge metric editors (instance scope) plus the clustering-rule picker. The config is a self-contained block on the layer template, so it travels with template export/import and is independent of the service-map topology config. Pod / sibling model. Instances render as hexagons and can bundle into pods: a pod\u0026rsquo;s main container is a full hex with its sibling containers attached as smaller hexes around its edges. Three independent rules drive it — cluster (the dashed boxes), sibling (which containers form one pod), and role (per-container-type metrics + which container is the main). Edges resolve to the exact container, so cross-pod sidecar links (e.g. a lifecycle agent calling its peer in another pod) connect the small hexes. The model can be previewed before real data exists via the admin\u0026rsquo;s draft Preview flow — edit the Deployment scope and preview the live page without publishing. Tiered layout + draggable pods. Each cluster box lays its pods out by call depth — sources on the left, the pods they call to the right — so a hot → warm → cold lifecycle chain reads as left-to-right tiers. Pods stack vertically within a tier, and a tier with more than four pods wraps into additional stacked columns of four. Drag any pod to rearrange; its cluster box re-flows to keep every node enclosed. Role-to-role edge metrics + a Flows view. Deployment edge metrics are keyed by the (source-role → target-role) pair, so each kind of link shows its own metrics rather than one flat set — a liaison → data edge can surface write / query / part-sync throughput while a lifecycle → data edge surfaces migration volume. Pairs match most-specific-first with a * wildcard fallback. Each pair names a primary metric that prints inline on the edge in the map, so the headline number reads at a glance without opening anything; the selected-edge sidebar shows that pair\u0026rsquo;s full metric set. A new Flows sub-tab (next to Topology) lays the edges out as one aligned table per role-pair — click a row to jump to that edge in the graph. API dependency The per-layer API dependency tab renders an endpoint\u0026rsquo;s caller → callee chain as a graph. Pick an endpoint and it lays out in columns by direction — callers on the left, the focus endpoint in the centre, callees on the right — with the same node health-ring border, SLA-coloured RPM, and latency you read on the service map; edges animate the call direction and label the heaviest by RPM. Expand to walk the chain. A selected endpoint shows a single + handle that pulls in its own callers and callees in one click (new callers land left, callees right). The handle spins while the dependency query is in flight; when an endpoint is a leaf with nothing further to load it fades and a brief banner says so — a silent \u0026ldquo;nothing happened\u0026rdquo; never reads as a bug. Rearrange freely. Drag any node box to pull a dense graph apart — edges follow live. Pan, wheel-zoom, and a fit button act on the whole canvas, and a node holds a steady on-screen size whether or not the detail sidebar is open. Drill straight out, in a new tab. The node detail\u0026rsquo;s Open endpoint and Service →, and the service-map node/edge jumps (Open service, API map →, Instance map →), now open in a new browser tab — so you keep the graph you\u0026rsquo;re exploring while the drill-down opens alongside it. Nodes share the service-map\u0026rsquo;s visual vocabulary (SLA-band border, an agent badge on instrumented endpoints, the focus star), and the tab is localized across all eight UI languages. Dashboard template portability Every template admin page — Overview templates, Layer dashboards, and the 3D-map config — now has Export and Import actions. Export downloads the in-use version (what end users render: the version live on OAP, or the bundled default when OAP has none) as a JSON file, for backup, sharing, or moving a dashboard to another OAP. Import reads a JSON file, validates it, and loads it as a local draft in this browser — preview it, then publish with “Check diff \u0026amp; push” as usual. Importing never writes OAP directly. Overview import can recreate a deleted dashboard or seed a brand-new one; layer import targets a layer already present on this deployment. The Translations page has matching Export / Import, scoped to the current language: export the in-use translation for a template + locale as a JSON file, or import one as a local draft to review and push. (Source templates and their translations are edited on separate pages, so their import/export are separate too — each on its own page.) Template store reliability Runtime config is strictly what\u0026rsquo;s on OAP. Layer dashboards, overviews, and topology now render only the version published to OAP\u0026rsquo;s UI-template store (or the in-code minimal default for a layer that has none). The disk-bundled templates reach a running UI only by being synced to OAP (first boot / admin reset) or through the admin Preview button — they are never a silent live fallback. So an operator always sees the live published config, not a stale bundled copy masquerading as current. Unreachable template store is a visible block, not a quiet fallback. When OAP\u0026rsquo;s UI-template host can\u0026rsquo;t be reached, a banner (same red treatment as the OAP-query-unreachable strip) reports it, and the dashboard / overview / topology surfaces stay empty rather than back-filling bundled defaults that could be read as real. The sidebar still navigates so the rest of the app is reachable. The admin Preview button now drives every template-rendered page — the overview detail view and the per-layer topology (incl. the instance map), API dependency, traces, and network-profiling pages — not just the layer dashboards. Previewing renders the draft\u0026rsquo;s metrics/config against live OAP, so an edit to topology or dependency metrics is visible before you publish. Preview and the absent-remote path stay strictly separate: a draft renders only in ?mode=preview; normal reads never carry one. Editors no longer silently fall back to the bundled default. When a layer / overview / translation has no version published to OAP, the editor shows a \u0026ldquo;No published version on OAP\u0026rdquo; panel instead of quietly loading the shipped bundled copy as if it were live. Bundled now reaches the editor only when you click Reset to bundled — matching the runtime, which renders the published version or blocks, never the bundle. Layer landing \u0026amp; service list The layer landing now shows your services, not just an arbitrary 25. It used to cap the metric fan-out at the first 25 services by list order — so larger layers hid the rest, and the \u0026ldquo;top\u0026rdquo; services weren\u0026rsquo;t even the true top (the cap happened before the ranking). Now it probes all services up to a configurable cap and, when a layer exceeds it, runs a cheap single- metric ranking pass to pick the true top-N by the landing\u0026rsquo;s order-by column. The service picker surfaces \u0026ldquo;top N of M\u0026rdquo; so the trim is never silent. Queries drain through a bounded-concurrency pool, so a big layer fans out in controlled waves rather than a thundering herd. New query.landingServiceCap in horizon.yaml (default 100) tunes how many services a landing probes per request — raise it if your OAP + storage can take the larger fan-out, lower it to protect a modest deployment. The service picker now lists the whole layer, not only the metric-probed top-N. Services that ranked below the metric cap on the order-by column now appear as their own rows with low in that column (and — for the others, which were never probed) instead of being hidden — every service stays browsable, searchable, and selectable regardless of the cap. The header chip reads \u0026ldquo;metrics: top N\u0026rdquo; to make the metric trim explicit. Removed the stale \u0026ldquo;Landing KPI tile\u0026rdquo; controls (Headline / Trend line) from the Layer-dashboards admin. They no longer matched the rendered layer header — which shows every configured metric column as its own KPI with its own trend line — so editing them changed nothing on screen. The header is driven entirely by the service-list columns + default sort; the preview now reflects that. Selecting a low-traffic (below-cap) service now works on every tab, not just the dashboard. Logs, traces, and endpoint-dependency resolved the picked service\u0026rsquo;s name from the landing sample only — so a tail service queried as blank (and Logs even snapped the pick back to the top service). All per-layer tabs now resolve the name from the full roster, so a low service drills in everywhere. Profiling scopes no longer show an editor grid that goes nowhere. Trace / eBPF / async profiling are built-in runtime views with nothing to author, so the admin now shows a \u0026ldquo;configured at runtime\u0026rdquo; note for them instead of a widget grid whose widgets never rendered. Access control \u0026amp; permissions The Roles \u0026amp; Permissions board now lists infra-3d:read — the permission to view the 3D Infrastructure Map — under the data-catalog group, with a matching \u0026ldquo;3D infrastructure map\u0026rdquo; row in the menu-visibility matrix. It was already enforced and granted to every built-in role (viewer and up), but it never appeared on the board, so an admin couldn\u0026rsquo;t see who held it. Editing a layer dashboard template is now gated on the dashboard:write permission the editor already advertises; publishing overview, alert, and 3D-map configs stays on overview:write. The required permission is resolved per template kind at save time. Built-in roles are unaffected (operator and admin hold both), but a custom role granted only dashboard:write can now save layer dashboards. The Cluster Status debug view (/api/debug/status) now requires only live-debug:read. It previously also demanded cluster:read, so a role granted live-debug access but not cluster-read was wrongly blocked. Saving a local draft of a template (the \u0026ldquo;Save local\u0026rdquo; action) now enforces the same per-kind permission as publishing — a layer draft needs dashboard:write, other kinds overview:write — instead of a blanket overview:write. Performance hardening Layer dashboards skip a redundant service lookup on every load. The dashboard route used to issue its own listServices to auto-pick the service and carry its entity-scope flags; it now reads the shared per-layer service catalog the sidebar already keeps warm, so a dashboard\u0026rsquo;s first paint costs one fewer OAP round-trip in the common case. It still falls back to a live lookup for a just-registered service, a cold snapshot, or to surface an OAP outage (the \u0026ldquo;OAP unreachable\u0026rdquo; state now follows the actual widget fetch, so a warm cache can\u0026rsquo;t mask a backend that\u0026rsquo;s gone away). The alarms list and count fire their two startup probes in parallel. The server-time offset and backend-capability probes that precede every alarms query now run concurrently instead of one-after-the-other. The 3D Infra Map loads its metrics in parallel. Per-node metric values used to fetch one batch at a time; they now load in bounded-concurrency batches, so the load rings and traffic values fill in sooner on large layers. A new metricConcurrency setting in the Infra Map config (default 4) caps how many metric batches run at once. Oversized layer topologies fail with a clear message instead of an unreadable map. When a layer\u0026rsquo;s service graph exceeds the render ceiling (5,000 services or 15,000 calls), the service map shows a \u0026ldquo;Topology too large to render\u0026rdquo; notice with the live counts and a hint to pick a specific service or lower the depth, rather than attempting to lay out a graph too dense to read. Partial metric-load failures are now surfaced on every topology map. If some metric batches fail to load (a transient OAP error) on the service map, instance topology, deployment, or endpoint-dependency map, a banner now explains that blank values may be unavailable rather than zero — and on the endpoint-dependency map, that some endpoints or links may be missing — so a backend hiccup isn\u0026rsquo;t misread as real \u0026ldquo;no traffic\u0026rdquo; data. Fixes Metrics Inspect — the crosshair value tooltip is no longer clipped behind the navigation sidebar when you hover near a widget\u0026rsquo;s left edge; it now renders above the page chrome. 3D Infrastructure Map config — \u0026ldquo;Check diff \u0026amp; push\u0026rdquo; now requires saving a local draft first (it was selectable while edits were still unsaved), and the push dialog renders the side-by-side before/after JSON diff instead of an empty panel. The API-dependency tab now honors the topbar time picker. It was pinned to the last hour regardless of the selected range; changing the range (and expanding a node) now re-queries the chosen window, like the service map and instance map already did. A dashboard no longer blanks entirely when one metric group fails. A transient backend error (timeout / 5xx / query-complexity limit) on a single batch of widgets now marks only those widgets as failed; the rest of the dashboard renders normally instead of every cell going blank. Trace list rows pick the correct root span on BanyanDB. A multi-service trace could surface a downstream span\u0026rsquo;s endpoint / duration / start time in the list; the row now reliably reflects the trace\u0026rsquo;s true entry span. Correct timestamps right after repointing OAP. The server-timezone offset is now cached per OAP URL, so a configuration reload that switches to a different-timezone OAP re-probes immediately instead of serving the previous server\u0026rsquo;s offset for up to a minute. Baseline security response headers (X-Content-Type-Options: nosniff, X-Frame-Options: DENY, Referrer-Policy: no-referrer) are now sent on every response. Removed an internal ?mockTop= debug query parameter that padded top-N widgets with synthetic rows; it no longer ships in release builds. The profiling pages now use more of the page height. The Trace / eBPF / Async / pprof / network profiling layouts were sized off a viewport offset that over-counted the chrome above them, leaving dead space at the bottom on taller screens; they now extend closer to the bottom of the view. Overview dashboard templates are validated before save. A malformed overview (missing a required field, an unknown widget type) is now rejected with a clear field-level error instead of being written to OAP — restoring a guard that was lost when overview editing moved to the OAP-backed save path. Documentation \u0026amp; release tooling A further accuracy pass corrected the Cluster Status page (three panes — Query, Admin, Zipkin/OTLP — and no per-node member list), the Kubernetes readiness-probe guidance (point it at the public /api/health, not the authenticated /api/oap/info), the layer-template components default (only the service dashboard is on when a key is omitted) and the aliases authoring key, the removed visibleWhen free-text and embedded-i18n template shapes, and the data-retention cold-stage controls. The website docs were brought current with the 0.6.0 build and the configuration pages restructured around the admin UI — the JSON shape is now a reference appendix, not an authoring surface (these admin pages are structured editors, not raw-JSON editors). Accuracy fixes span the RBAC verbs (incl. infra-3d:read), the audit-log action set, the Metrics Inspect API paths, the layer-template component flags, and the redesigned 3D-map config + loading stages. A new docs/CLAUDE.md records the doc-writing principles, and the i18n docs gain a language × scope coverage matrix plus a translation step in the add-a-layer recipe. The container image is published to Docker Hub by CI on every v* tag; the post-vote finalize script now only verifies the published tags (the manual local-push fallback and Docker Hub login preflight were removed). Layer drill-down fixes The per-layer Instance and Endpoint pages now honor the layer\u0026rsquo;s configured aliases in their section headers and in the service-picker\u0026rsquo;s name column — e.g. ActiveMQ reads Brokers / Destinations and Virtual MQ reads Topics / MQ clusters, matching the sidebar — instead of the generic \u0026ldquo;Instance\u0026rdquo; / \u0026ldquo;Endpoint\u0026rdquo; / \u0026ldquo;Service\u0026rdquo; labels. Layers that define no alias still read the generic words. A layer\u0026rsquo;s Instance or Endpoint page no longer hangs on a perpetual \u0026ldquo;Reading data…\u0026rdquo; when the selected service reports no instances or endpoints (or a search matches nothing). It now shows the empty picker and renders the metric widgets in their normal \u0026ldquo;no data\u0026rdquo; state, so the layout stays visible and ready for services that do report them. Clearer cluster boundaries on every topology view. The dashed grouping boxes — namespaces on the service map, per-service boxes on the instance map, role/tier clusters on the Deployment tab — now draw with a bolder, brighter dashed border and a fully transparent background, so the boundary reads clearly on every theme (light themes included) instead of fading into the canvas. The Deployment tab also packs its cluster boxes evenly: boxes sit at a uniform spacing with no dead corridor between tiers and no blank strip before the first box. Live debugger MAL sample groups. A captured step that fans out to many samples no longer dumps every label set on screen: the samples are grouped by metric name into a one-line summary — \u0026lt;metric\u0026gt; · N samples · values=… — and you expand only the groups you care about to see each sample\u0026rsquo;s full labels. Groups are collapsed by default. Diff is the default when a group is expanded. A multi-sample group opens straight into diff view: the labels shared by every sample collapse into a dimmed \u0026ldquo;common\u0026rdquo; block and only the labels that differ are highlighted per sample — so it is immediate which label distinguishes each one (e.g. node_role / pod_name) and what value it maps to. A diff toggle beside the group\u0026rsquo;s header switches back to the full per-sample label list. The \u0026ldquo;common\u0026rdquo; set is computed across the whole group, not just the rendered rows. Multiple output entities collapse the same way. When a record materialises one metric for several entities (e.g. a per-endpoint write rate over sw_metricsMinute / sw_metricsHour / sw_metricsDay), the repeated meter cards fold into one block: a shared header (metric / function / time bucket), a N outputs · values=… summary, and a diff that surfaces only the entity fields that actually differ — whichever they are, not a fixed field — with each output\u0026rsquo;s value beside it. Readable sample values. Long fractional values from rate() / avg() (e.g. 57.0333333333…) are trimmed to a few significant digits for display so they stop overflowing the value column; integer counters still render exact, and the precise value stays available on hover. DSL management — live apply progress \u0026amp; recovery A structural rule change now shows live apply progress. Saving an edit that moves a metric\u0026rsquo;s storage shape (scope, downsampling, or the metric set) no longer just flashes \u0026ldquo;submitted\u0026rdquo; — the editor tracks the apply across the cluster through a phase stepper (Compiled → Confirming across the cluster → Committing → Done) and reports success only once OAP confirms the change is durable. Revert to bundled (also a schema change) goes through the same stepper. Body- and filter-only edits still apply instantly with no stepper. You can navigate away mid-apply; reloading the editor resumes the progress. \u0026ldquo;Applied — cluster propagation unconfirmed\u0026rdquo; is a warning, not an error. When a structural change is committed and durable but one or more nodes hadn\u0026rsquo;t confirmed the new schema within OAP\u0026rsquo;s fence budget, the editor names the lagging nodes and explains they self-converge on their next scan — the rule is applied, not rolled back. Reloading the editor reads it back as applied (from the stored rule). A failed apply is called out as rolled back — the cluster stays on the previous rule, the failure reason is shown inline, and your edit is kept in the editor so you can fix and save again. A compile error now surfaces as an inline diagnostic under the editor instead of a transient toast. Force re-apply to recover. A degraded or transiently-failed apply offers a one-click Force re-apply (recover) that re-runs the apply across the cluster to re-confirm the schema and un-stick any waiting node — gated behind a confirm that spells out it briefly pauses collection for that rule\u0026rsquo;s metrics, even when the content is unchanged. This subsumes the old Advanced force toggle for the recovery case. ","excerpt":"\u003ch1 id=\"070\"\u003e0.7.0\u003c/h1\u003e\n\u003ch3 id=\"browser-errors--source-maps\"\u003eBrowser errors \u0026amp; source maps\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eNew \u0026ldquo;Browser Logs\u0026rdquo; tab on the BROWSER layer\u003c/strong\u003e — …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/changelog/0.7.0/","title":"0.7.0"},{"body":"0.7.0 Browser errors \u0026amp; source maps New \u0026ldquo;Browser Logs\u0026rdquo; tab on the BROWSER layer — lists the JS error logs the browser agent reports (message, category, page, app version, time, and the minified line:col), filterable by category and time window. Expanding a row shows the raw stack alongside a de-obfuscated view. Source-map de-obfuscation (issue #6784). Upload a .map file from the tab and resolve any error\u0026rsquo;s minified stack back to the original source — file, line, column, symbol name, and a source snippet — by picking which map to apply. Maps are held in the BFF\u0026rsquo;s memory only (no backend storage): they\u0026rsquo;re surfaced as temporary, evicted least-recently-used when the configured budget is hit, and lost on restart. For durable provisioning, mount .map files into the server\u0026rsquo;s static source-map directory (HORIZON_SOURCEMAPS_DIR, /app/sourcemaps in the image) — those reload automatically and can\u0026rsquo;t be deleted from the UI. Budgets are configurable via the new sourceMaps block in horizon.yaml (per-file and total in-memory caps; defaults 64 MiB / 512 MiB). Upload/delete require the new source-map:write permission; viewing + resolving ride on browser-errors:read. Layers Split a layer\u0026rsquo;s menu by service group. A new per-layer Split menu by service group toggle (Layer dashboards admin, right after Alias; default off) fans the layer out into one level-0 sidebar entry per OAP Service.group — the \u0026lt;group\u0026gt;:: prefix. The entry\u0026rsquo;s display name leads with the group so it reads everywhere (sidebar, page header, KPI tile) and survives narrow-sidebar truncation — e.g. agent · General Service. Each entry is scoped to its group: the service header + its picker, the topology map + its in-box selector, the dashboards, and the service roster all show only that group\u0026rsquo;s services. A layer\u0026rsquo;s group entries stay contiguous in the sidebar (sorted by group), and the cross-group view returns by turning the toggle back off (off = one combined entry holding all groups). The group value is OAP-supplied data and is shown verbatim (not translated). Travels with template export/import like every other layer setting. The navigation sidebar is now resizable — drag the divider between the sidebar and the page to widen or narrow it (double-click the divider to reset to the default width); the chosen width persists per browser. Useful when long entries — like group-split names (agent · General Service) or deep namespaces — would otherwise truncate. The per-layer service picker shows each service\u0026rsquo;s group. When a layer has no topology-cluster naming rule, the service-list rows now surface the OAP \u0026lt;group\u0026gt;:: prefix (e.g. agent) as the group chip — so the group is visible there as it already is on the topology map. Every layer OAP reports now appears in the sidebar, including ones with no Horizon template (they render with default capabilities — a plain Service page). The previous hard-coded hidden-layer list (which dropped BanyanDB) is gone; a layer is hidden only when an admin explicitly disables its template, or when it is listed in the new config-driven layers.excluded block in horizon.yaml (defaults: FAAS and VIRTUAL_GATEWAY; clear the list to surface every reported layer). The admin Layer dashboards page is now layer-list-oriented. It lists every available layer — not just the ones shipping a bundled JSON or living on OAP. A layer with no template yet opens on a blank default you can configure (components, metric columns, widgets, topology) and Save, which publishes the template to OAP on first save. No per-layer JSON has to be shipped for a layer to be configurable. The picker gains a Not configured filter (beside Diverged / Local) and the sync banner spells out \u0026ldquo;N templates match bundled defaults · M layers not configured yet\u0026rdquo;. Removed the legacy per-layer overview block from every bundled layer template (and its translation overlays). It no longer rendered anything — the standalone Overview Dashboards replaced the old per-layer Overview tile — so it was dead config; the per-layer KPI strip is driven by the layer-header columns. Airflow monitoring layer (SWIP-7) New Airflow layer under Workflow Scheduler — service dashboard (scheduler / executor / pool KPIs and trends), Components dashboard (per-host scheduler and triggerer metrics for Airflow 3.x native OTel), and a 3D Infra Map load ring for Tasks Executable. Pairs with OAP backend SWIP-7 (meter_airflow_* / meter_airflow_instance_*). BanyanDB self-observability layer (SWIP-15) New BanyanDB layer under Self-Observability, modeling a clustered, role- and tier-aware BanyanDB deployment scraped through its FODC proxy. The cluster is one Cluster (service), each container is one Container (instance, carrying its container_name role and node_type tier as attributes), and each storage Group is an endpoint: Cluster dashboard — write / query / error-rate KPIs, CPU / memory / disk capacity, a throughput + errors trend, and a Containers by Role table. Container dashboard adapts to the selected container\u0026rsquo;s role: every container shows CPU / memory / Go-runtime resources (and, where the system collector runs, uptime / disk / network); a liaison adds ingestion, query, gRPC errors, the tier-2 publish pipeline and write-queue depth; a data node adds storage totals, merge/compaction, inverted index, subscribe queue and retention; a lifecycle sidecar shows migration cycles and last-run time / status. Liaison and data panels are gated on the container\u0026rsquo;s role attribute; resource panels the lifecycle sidecar doesn\u0026rsquo;t emit, and the lifecycle migration panels themselves, self-gate on data presence so they surface only once that container actually reports them (the lifecycle panels stay hidden until the first migration cycle runs). This template targets the clustered model; a single-process standalone BanyanDB (container_name=standalone) shows the shared resource / Go panels but not the role-specific ingestion / storage panels — those extend to standalone once the entity-gate membership operator (SWIP-15 §6) lands. Group dashboard — metrics split per data-model (measure / stream / trace / property): each model gets write rate, query latency, stored data, merge rate / latency / partitions, series write + term-search and total series, plus the type-agnostic subscribe / publish queue (throughput, p99, batch + message rate, publish bytes). Because a BanyanDB group stores one catalog, only the matching model\u0026rsquo;s panels render for a given group — gated by the model\u0026rsquo;s series-count flag — so a measure group shows the measure panels, a property group its index-write / merge / term-search / series panels, and so on. A Deployment tab renders the cluster\u0026rsquo;s container inventory — every container grouped into its node\u0026rsquo;s role/tier box (liaison, data hot/warm/cold) and its pod, with per-role health metrics (liaison query rate + gRPC errors, data ingest rate + disk usage, lifecycle migration cycles + last-run status). The node health-ring legend names each role\u0026rsquo;s own ring metric and its colour-band thresholds (driven by the layer template) instead of a single shared, hard-coded one. Container-to-container call edges carry role-pair-specific metrics off the SWIP-15 instance-relation families: a liaison → data edge shows write / query / part-sync throughput + p99 (one per queue operation), a liaison → liaison edge shows write-forward + control, and a lifecycle → data edge shows tier-migration volume / rate / p99. The edge prints up to 3 of the pair\u0026rsquo;s metrics inline (short aliases like W / R, flowing onto one line or stacking by edge length); the selected-edge panel keeps the full client | server breakdown, and the Flows sub-tab tables every edge per role-pair. Edges render once the OAP build includes the SERVICE_INSTANCE_RELATION scope (the migration_* family also needs the lifecycle sidecar reporting); until then the tab shows the inventory without edges. The whole deployment model — clustering / grouping rules, per-role node metrics, and role-pair edge metrics — is editable from the Layer dashboards admin → Deployment scope. Pairs with OAP backend SWIP-15 (meter_banyandb_* cluster / meter_banyandb_instance_* container / meter_banyandb_endpoint_* group). Queue-batch and lifecycle last-run panels appear once the cluster runs a BanyanDB build that emits those metrics. Dashboard widget value formatting Card widgets gain a enum format with a value→label map: a coded metric (e.g. a 1/0 success gauge) renders a readable label (1 → OK, 0 → Failed) instead of the raw number. Labels are translatable per locale (BFF-side template i18n overlay) and the map is editable in the Layer dashboards admin. BanyanDB\u0026rsquo;s lifecycle Last Sync card uses it. New duration format renders a SECONDS metric as a human time-ago (5m 20s ago; compact 5m / 2h on axes) — used by BanyanDB\u0026rsquo;s Time Since Last Sync card. Record widgets — jump to trace \u0026amp; copy Record widgets now drill into the originating trace. Each sampled row gets a jump-to-trace icon at the row head — shown only when the sample actually carries a trace id (these are sampled, so it can be absent) — that opens the trace waterfall in the global popout. It resolves the trace by id, not by layer, so it works even though the trace belongs to the calling service on a different layer (a virtual-target layer has no traces tab of its own). The statement text itself is click-to-copy. For example, the Slow Statements record widget on a Virtual Database / Cache / MQ service. Instance-list badge The badge on each row of the instance list (Containers / Pods / Nodes / …) is now configurable per layer (instances.badge on the layer template) — it can show any instance attribute instead of the fixed agent language. BanyanDB shows container_name (liaison / data / lifecycle), the role that actually distinguishes a container; agent-traced layers keep language (Java / Go / …). The badge is now hidden when the value is empty or UNKNOWN, so OpenTelemetry-scraped layers (which report no agent language) drop the meaningless UNKNOWN chip across the board. Dashboard widget visibility Layer-dashboard widgets gain a structured Visible when gate (Layer dashboards admin → widget drawer) so a widget only renders when it\u0026rsquo;s relevant to the selected entity. Two kinds: MQE metric — show the widget only when an expression has value, or when any value is \u0026gt; / \u0026lt; a threshold. Naming the widget\u0026rsquo;s own metric self-gates it (the JVM widgets appear only on JVM instances, the MQ widgets only on MQ producers, …); naming a different metric gates a whole group on one shared signal — that metric is checked once and the entire group\u0026rsquo;s queries are skipped when it\u0026rsquo;s empty, so e.g. a non-JVM instance no longer runs the JVM widget queries at all. Entity attribute — on the Instance scope, gate on the selected instance\u0026rsquo;s attributes, e.g. language equals JAVA (case-insensitive) or an attribute simply being present. Service / Endpoint entities carry no attributes, so entity gates are ignored on those scopes. Gates are evaluated server-side; gated-out widgets just don\u0026rsquo;t appear in the grid. Note: a layer dashboard saved before this release that used the old free-text predicate loses its gate (the widget renders ungated) until you re-set the gate in the new editor and save the dashboard. Topology node filter \u0026amp; component icons The per-layer Topology map (and the embedded topology widget on the Services / Mesh overview dashboards) gains a Filter control to hide the conjectured peers that clutter a dense map. One auto-derived facet — by layer, presented exactly as the sidebar shows it: each row carries the layer\u0026rsquo;s own icon and its localized display name (General Service, Virtual Database, Java Agent, …), plus an Others bucket for nodes OAP couldn\u0026rsquo;t resolve, alongside a standalone User toggle. The layer rows self-populate from whatever the map currently shows and re-derive on every refresh / depth / time change. Unchecking a row hides those nodes and their now-dangling edges; the Others bucket is where uninstrumented \u0026ldquo;undefined\u0026rdquo; peers (e.g. a bare rcmd:80) land, so one click clears them, while your real databases / queues / caches — separated by their own VIRTUAL_* layer rows — stay on the map. Filtering is client-side and defaults to showing everything. Technology component icons on the nodes. Service-map nodes now render the icon for their detected component — the same icon set the trace waterfall uses, so a PostgreSQL node looks like PostgreSQL — falling back to the generic service / external / user glyph when the component ships no icon or couldn\u0026rsquo;t be resolved. The topology\u0026rsquo;s service selector (the \u0026ldquo;All services\u0026rdquo; picker) now groups its list by service group — OAP\u0026rsquo;s Service.group (the \u0026lt;group\u0026gt;:: prefix, e.g. agent) shown under a value-first \u0026lt;name\u0026gt; [GROUP] header — so a layer whose services share a group reads grouped instead of as one flat list. This is a per-service attribute and needs no per-layer naming-rule setup; services with no group stay in a single header-less section. Clicking a group header batch-selects or unselects every service in that group — the header carries a filled / half / hollow marker for all / some / none of its services focused. Instance topology The per-layer Topology map gains an instance map drill-down on layers that enable instance topology. Click a call between two services and then Instance map → to open it: the instances of each service as two columns (left = client, right = server) with the instance-level calls between them — pan/zoom, animated client→server flow, the same node health-ring + per-call client/server metric sidebar the service map uses, and a node popover with Open instance dashboard. A back button returns to the service map; a toolbar pair-picker swaps the two services. The two service pickers are relationship-aware, drawn from the service-topology call graph (including conjectured / cross-layer callees like rcmd:80, named the same as on the service map): the server list is the chosen client\u0026rsquo;s callees and the client list is the chosen server\u0026rsquo;s callers, each re-deriving when the other changes without resetting your current pick. A side the graph leaves no real choice for (e.g. a single caller) shows as plain text instead of a one-option dropdown. Each service\u0026rsquo;s instances sit inside a labelled grouping box — named with the service, using the same \u0026lt;group\u0026gt;:: prefix handling as the service map so a name reads identically on both — and a ring-colour legend explains what the node health bands (green → red) mean for the configured ring metric. Labels follow the layer\u0026rsquo;s own terms (e.g. Pods on Kubernetes, Sidecars on the data plane). Configurable like the service map. The Layer-dashboards admin → Topology scope now has an Enable instance topology toggle and its own node / server-edge / client-edge metric editors, kept visually separate from the service-topology metrics so the two are never confused. Enabled out of the box on General, Service Mesh, Kubernetes Service, and Cilium Service; the config rides each layer\u0026rsquo;s topology template (so it travels with template export/import). When OAP\u0026rsquo;s template store is unreachable, the instance map now shows the same empty + connectivity-banner state as the service map, rather than a misleading \u0026ldquo;not supported\u0026rdquo; — block and unsupported are no longer conflated. Localized across all eight UI languages. The instance-map UI, the template-store-unreachable banner, and the remaining alarm / live-debugger strings are now translated in zh-CN, ja, ko, es, pt, de and fr (English stays the source) — no feature renders English-only for non-English operators. Lock \u0026amp; compare entities on a layer dashboard Lock several services, instances, or endpoints — including ones from different services — and compare them in place. Compare is standard on every service / instance / endpoint layer dashboard — no flag, nothing to enable. Pin entities from the service picker or the instance / endpoint list; instance and endpoint pins are cross-service, so instances belonging to different services can be compared side by side. A persistent, scope-aware comparison bar shows the cohort regardless of how the underlying list paginates or which entity is currently selected. The entity you\u0026rsquo;re viewing is always part of the comparison — it appears first, tagged CURRENT in the accent color (and still drives the header KPIs); pinned entities add to it, each in its own stable hue (up to six pins). The comparison-bar chips are display-only: clicking a chip never changes what you\u0026rsquo;re viewing (no disruptive reload) and × unpins — switch the focused entity from the top selector / list as usual. Each widget compares inline in its own tile — line widgets overlay one hued series per entity; card widgets show one row per entity; top-N and record widgets get per-entity tabs plus a merged \u0026ldquo;All\u0026rdquo; tab; table widgets gain an Entity column that groups rows by entity and folds each entity\u0026rsquo;s long tail into one (others) row (summed for counts, count-only for latencies / percentiles where a sum would mislead). With nothing locked, every page renders exactly as before. Labeled series lead with the meaningful dimension — a multi-label series reads \u0026lt;label\u0026gt; · \u0026lt;service\u0026gt; · \u0026lt;instance\u0026gt; so the label (e.g. a JVM thread state) survives when a long instance / endpoint id has to truncate; entities stay distinguishable by color. A widget that only some compared entities expose (e.g. JVM widgets when a Java service is pinned alongside a non-JVM one) shows whenever any compared entity has it. Progressive, per-entity loading — each entity loads as its own request; tiles fill in as entities arrive and one slow or failed entity never blanks the others. The Topology, Deployment, trace, and log pages are unaffected — comparison applies to the service / instance / endpoint dashboards only. Widget tooltips and legends show the widget title, never the raw MQE expression, for un-labeled single-series line widgets; the multi-series tooltip is a fixed, aligned table — the entity name truncates and the values form one clean right-aligned column with the unit in the header. Charts \u0026amp; bundled metrics Large numbers on chart axes and tooltips now use compact SI suffixes (45.1k, 1.34M, 2.5G) instead of scientific notation (4.51e4), which operators found hard to read. Go runtime \u0026ldquo;Metadata Mspan\u0026rdquo; and \u0026ldquo;Metadata Mcache\u0026rdquo; widgets now report KB, fixing values that were displayed as a mislabeled \u0026ldquo;MB\u0026rdquo; of raw bytes (≈1000× too large) — they now read in the same KB scale as the other Go metadata-size widgets. Deployment New per-layer Deployment tab — the deployment topology of all of a service\u0026rsquo;s instances: the instance-to-instance call graph within a single service. Where the instance map drills into the instances between two services, this shows how one service\u0026rsquo;s own instances are deployed and talk to each other (e.g. a clustered store\u0026rsquo;s nodes calling each other). Pick a service from the layer\u0026rsquo;s Service header and the tab draws its instances as health-ring nodes with the intra-service calls between them — pan/zoom, animated edge flow, the per-call client/server metric sidebar, and a node popover that shows the instance\u0026rsquo;s attributes and an Open instance dashboard link. Self-calls and back-and-forth pairs are drawn distinctly. Node clustering. Instances can group into labelled boxes by a single instance attribute (e.g. role / tier), by several attributes combined into one key (e.g. node_role + node_type, where an attribute absent on a node drops out — so a BanyanDB cluster splits its data nodes into hot / warm / cold boxes while the liaison nodes, which carry no tier, stay one box), or by a name regex run on the instance name — so a fleet of mixed-role nodes reads as one box per role instead of a flat cloud. The boxes lay out left→right along the calls between them, so an upstream→downstream chain reads in order. Optional + configurable. Off by default for every layer; a layer opts in from the Layer-dashboards admin → Deployment scope, which has its own node / server-edge / client-edge metric editors (instance scope) plus the clustering-rule picker. The config is a self-contained block on the layer template, so it travels with template export/import and is independent of the service-map topology config. Pod / sibling model. Instances render as hexagons and can bundle into pods: a pod\u0026rsquo;s main container is a full hex with its sibling containers attached as smaller hexes around its edges. Three independent rules drive it — cluster (the dashed boxes), sibling (which containers form one pod), and role (per-container-type metrics + which container is the main). Edges resolve to the exact container, so cross-pod sidecar links (e.g. a lifecycle agent calling its peer in another pod) connect the small hexes. The model can be previewed before real data exists via the admin\u0026rsquo;s draft Preview flow — edit the Deployment scope and preview the live page without publishing. Tiered layout + draggable pods. Each cluster box lays its pods out by call depth — sources on the left, the pods they call to the right — so a hot → warm → cold lifecycle chain reads as left-to-right tiers. Pods stack vertically within a tier, and a tier with more than four pods wraps into additional stacked columns of four. Drag any pod to rearrange; its cluster box re-flows to keep every node enclosed. Role-to-role edge metrics + a Flows view. Deployment edge metrics are keyed by the (source-role → target-role) pair, so each kind of link shows its own metrics rather than one flat set — a liaison → data edge can surface write / query / part-sync throughput while a lifecycle → data edge surfaces migration volume. Pairs match most-specific-first with a * wildcard fallback. Each pair names a primary metric that prints inline on the edge in the map, so the headline number reads at a glance without opening anything; the selected-edge sidebar shows that pair\u0026rsquo;s full metric set. A new Flows sub-tab (next to Topology) lays the edges out as one aligned table per role-pair — click a row to jump to that edge in the graph. API dependency The per-layer API dependency tab renders an endpoint\u0026rsquo;s caller → callee chain as a graph. Pick an endpoint and it lays out in columns by direction — callers on the left, the focus endpoint in the centre, callees on the right — with the same node health-ring border, SLA-coloured RPM, and latency you read on the service map; edges animate the call direction and label the heaviest by RPM. Expand to walk the chain. A selected endpoint shows a single + handle that pulls in its own callers and callees in one click (new callers land left, callees right). The handle spins while the dependency query is in flight; when an endpoint is a leaf with nothing further to load it fades and a brief banner says so — a silent \u0026ldquo;nothing happened\u0026rdquo; never reads as a bug. Rearrange freely. Drag any node box to pull a dense graph apart — edges follow live. Pan, wheel-zoom, and a fit button act on the whole canvas, and a node holds a steady on-screen size whether or not the detail sidebar is open. Drill straight out, in a new tab. The node detail\u0026rsquo;s Open endpoint and Service →, and the service-map node/edge jumps (Open service, API map →, Instance map →), now open in a new browser tab — so you keep the graph you\u0026rsquo;re exploring while the drill-down opens alongside it. Nodes share the service-map\u0026rsquo;s visual vocabulary (SLA-band border, an agent badge on instrumented endpoints, the focus star), and the tab is localized across all eight UI languages. Dashboard template portability Every template admin page — Overview templates, Layer dashboards, and the 3D-map config — now has Export and Import actions. Export downloads the in-use version (what end users render: the version live on OAP, or the bundled default when OAP has none) as a JSON file, for backup, sharing, or moving a dashboard to another OAP. Import reads a JSON file, validates it, and loads it as a local draft in this browser — preview it, then publish with “Check diff \u0026amp; push” as usual. Importing never writes OAP directly. Overview import can recreate a deleted dashboard or seed a brand-new one; layer import targets a layer already present on this deployment. The Translations page has matching Export / Import, scoped to the current language: export the in-use translation for a template + locale as a JSON file, or import one as a local draft to review and push. (Source templates and their translations are edited on separate pages, so their import/export are separate too — each on its own page.) Template store reliability Runtime config is strictly what\u0026rsquo;s on OAP. Layer dashboards, overviews, and topology now render only the version published to OAP\u0026rsquo;s UI-template store (or the in-code minimal default for a layer that has none). The disk-bundled templates reach a running UI only by being synced to OAP (first boot / admin reset) or through the admin Preview button — they are never a silent live fallback. So an operator always sees the live published config, not a stale bundled copy masquerading as current. Unreachable template store is a visible block, not a quiet fallback. When OAP\u0026rsquo;s UI-template host can\u0026rsquo;t be reached, a banner (same red treatment as the OAP-query-unreachable strip) reports it, and the dashboard / overview / topology surfaces stay empty rather than back-filling bundled defaults that could be read as real. The sidebar still navigates so the rest of the app is reachable. The admin Preview button now drives every template-rendered page — the overview detail view and the per-layer topology (incl. the instance map), API dependency, traces, and network-profiling pages — not just the layer dashboards. Previewing renders the draft\u0026rsquo;s metrics/config against live OAP, so an edit to topology or dependency metrics is visible before you publish. Preview and the absent-remote path stay strictly separate: a draft renders only in ?mode=preview; normal reads never carry one. Editors no longer silently fall back to the bundled default. When a layer / overview / translation has no version published to OAP, the editor shows a \u0026ldquo;No published version on OAP\u0026rdquo; panel instead of quietly loading the shipped bundled copy as if it were live. Bundled now reaches the editor only when you click Reset to bundled — matching the runtime, which renders the published version or blocks, never the bundle. Layer landing \u0026amp; service list The layer landing now shows your services, not just an arbitrary 25. It used to cap the metric fan-out at the first 25 services by list order — so larger layers hid the rest, and the \u0026ldquo;top\u0026rdquo; services weren\u0026rsquo;t even the true top (the cap happened before the ranking). Now it probes all services up to a configurable cap and, when a layer exceeds it, runs a cheap single- metric ranking pass to pick the true top-N by the landing\u0026rsquo;s order-by column. The service picker surfaces \u0026ldquo;top N of M\u0026rdquo; so the trim is never silent. Queries drain through a bounded-concurrency pool, so a big layer fans out in controlled waves rather than a thundering herd. New query.landingServiceCap in horizon.yaml (default 100) tunes how many services a landing probes per request — raise it if your OAP + storage can take the larger fan-out, lower it to protect a modest deployment. The service picker now lists the whole layer, not only the metric-probed top-N. Services that ranked below the metric cap on the order-by column now appear as their own rows with low in that column (and — for the others, which were never probed) instead of being hidden — every service stays browsable, searchable, and selectable regardless of the cap. The header chip reads \u0026ldquo;metrics: top N\u0026rdquo; to make the metric trim explicit. Removed the stale \u0026ldquo;Landing KPI tile\u0026rdquo; controls (Headline / Trend line) from the Layer-dashboards admin. They no longer matched the rendered layer header — which shows every configured metric column as its own KPI with its own trend line — so editing them changed nothing on screen. The header is driven entirely by the service-list columns + default sort; the preview now reflects that. Selecting a low-traffic (below-cap) service now works on every tab, not just the dashboard. Logs, traces, and endpoint-dependency resolved the picked service\u0026rsquo;s name from the landing sample only — so a tail service queried as blank (and Logs even snapped the pick back to the top service). All per-layer tabs now resolve the name from the full roster, so a low service drills in everywhere. Profiling scopes no longer show an editor grid that goes nowhere. Trace / eBPF / async profiling are built-in runtime views with nothing to author, so the admin now shows a \u0026ldquo;configured at runtime\u0026rdquo; note for them instead of a widget grid whose widgets never rendered. Access control \u0026amp; permissions The Roles \u0026amp; Permissions board now lists infra-3d:read — the permission to view the 3D Infrastructure Map — under the data-catalog group, with a matching \u0026ldquo;3D infrastructure map\u0026rdquo; row in the menu-visibility matrix. It was already enforced and granted to every built-in role (viewer and up), but it never appeared on the board, so an admin couldn\u0026rsquo;t see who held it. Editing a layer dashboard template is now gated on the dashboard:write permission the editor already advertises; publishing overview, alert, and 3D-map configs stays on overview:write. The required permission is resolved per template kind at save time. Built-in roles are unaffected (operator and admin hold both), but a custom role granted only dashboard:write can now save layer dashboards. The Cluster Status debug view (/api/debug/status) now requires only live-debug:read. It previously also demanded cluster:read, so a role granted live-debug access but not cluster-read was wrongly blocked. Saving a local draft of a template (the \u0026ldquo;Save local\u0026rdquo; action) now enforces the same per-kind permission as publishing — a layer draft needs dashboard:write, other kinds overview:write — instead of a blanket overview:write. Performance hardening Layer dashboards skip a redundant service lookup on every load. The dashboard route used to issue its own listServices to auto-pick the service and carry its entity-scope flags; it now reads the shared per-layer service catalog the sidebar already keeps warm, so a dashboard\u0026rsquo;s first paint costs one fewer OAP round-trip in the common case. It still falls back to a live lookup for a just-registered service, a cold snapshot, or to surface an OAP outage (the \u0026ldquo;OAP unreachable\u0026rdquo; state now follows the actual widget fetch, so a warm cache can\u0026rsquo;t mask a backend that\u0026rsquo;s gone away). The alarms list and count fire their two startup probes in parallel. The server-time offset and backend-capability probes that precede every alarms query now run concurrently instead of one-after-the-other. The 3D Infra Map loads its metrics in parallel. Per-node metric values used to fetch one batch at a time; they now load in bounded-concurrency batches, so the load rings and traffic values fill in sooner on large layers. A new metricConcurrency setting in the Infra Map config (default 4) caps how many metric batches run at once. Oversized layer topologies fail with a clear message instead of an unreadable map. When a layer\u0026rsquo;s service graph exceeds the render ceiling (5,000 services or 15,000 calls), the service map shows a \u0026ldquo;Topology too large to render\u0026rdquo; notice with the live counts and a hint to pick a specific service or lower the depth, rather than attempting to lay out a graph too dense to read. Partial metric-load failures are now surfaced on every topology map. If some metric batches fail to load (a transient OAP error) on the service map, instance topology, deployment, or endpoint-dependency map, a banner now explains that blank values may be unavailable rather than zero — and on the endpoint-dependency map, that some endpoints or links may be missing — so a backend hiccup isn\u0026rsquo;t misread as real \u0026ldquo;no traffic\u0026rdquo; data. Fixes Metrics Inspect — the crosshair value tooltip is no longer clipped behind the navigation sidebar when you hover near a widget\u0026rsquo;s left edge; it now renders above the page chrome. 3D Infrastructure Map config — \u0026ldquo;Check diff \u0026amp; push\u0026rdquo; now requires saving a local draft first (it was selectable while edits were still unsaved), and the push dialog renders the side-by-side before/after JSON diff instead of an empty panel. The API-dependency tab now honors the topbar time picker. It was pinned to the last hour regardless of the selected range; changing the range (and expanding a node) now re-queries the chosen window, like the service map and instance map already did. A dashboard no longer blanks entirely when one metric group fails. A transient backend error (timeout / 5xx / query-complexity limit) on a single batch of widgets now marks only those widgets as failed; the rest of the dashboard renders normally instead of every cell going blank. Trace list rows pick the correct root span on BanyanDB. A multi-service trace could surface a downstream span\u0026rsquo;s endpoint / duration / start time in the list; the row now reliably reflects the trace\u0026rsquo;s true entry span. Correct timestamps right after repointing OAP. The server-timezone offset is now cached per OAP URL, so a configuration reload that switches to a different-timezone OAP re-probes immediately instead of serving the previous server\u0026rsquo;s offset for up to a minute. Baseline security response headers (X-Content-Type-Options: nosniff, X-Frame-Options: DENY, Referrer-Policy: no-referrer) are now sent on every response. Removed an internal ?mockTop= debug query parameter that padded top-N widgets with synthetic rows; it no longer ships in release builds. The profiling pages now use more of the page height. The Trace / eBPF / Async / pprof / network profiling layouts were sized off a viewport offset that over-counted the chrome above them, leaving dead space at the bottom on taller screens; they now extend closer to the bottom of the view. Overview dashboard templates are validated before save. A malformed overview (missing a required field, an unknown widget type) is now rejected with a clear field-level error instead of being written to OAP — restoring a guard that was lost when overview editing moved to the OAP-backed save path. Documentation \u0026amp; release tooling A further accuracy pass corrected the Cluster Status page (three panes — Query, Admin, Zipkin/OTLP — and no per-node member list), the Kubernetes readiness-probe guidance (point it at the public /api/health, not the authenticated /api/oap/info), the layer-template components default (only the service dashboard is on when a key is omitted) and the aliases authoring key, the removed visibleWhen free-text and embedded-i18n template shapes, and the data-retention cold-stage controls. The website docs were brought current with the 0.6.0 build and the configuration pages restructured around the admin UI — the JSON shape is now a reference appendix, not an authoring surface (these admin pages are structured editors, not raw-JSON editors). Accuracy fixes span the RBAC verbs (incl. infra-3d:read), the audit-log action set, the Metrics Inspect API paths, the layer-template component flags, and the redesigned 3D-map config + loading stages. A new docs/CLAUDE.md records the doc-writing principles, and the i18n docs gain a language × scope coverage matrix plus a translation step in the add-a-layer recipe. The container image is published to Docker Hub by CI on every v* tag; the post-vote finalize script now only verifies the published tags (the manual local-push fallback and Docker Hub login preflight were removed). Layer drill-down fixes The per-layer Instance and Endpoint pages now honor the layer\u0026rsquo;s configured aliases in their section headers and in the service-picker\u0026rsquo;s name column — e.g. ActiveMQ reads Brokers / Destinations and Virtual MQ reads Topics / MQ clusters, matching the sidebar — instead of the generic \u0026ldquo;Instance\u0026rdquo; / \u0026ldquo;Endpoint\u0026rdquo; / \u0026ldquo;Service\u0026rdquo; labels. Layers that define no alias still read the generic words. A layer\u0026rsquo;s Instance or Endpoint page no longer hangs on a perpetual \u0026ldquo;Reading data…\u0026rdquo; when the selected service reports no instances or endpoints (or a search matches nothing). It now shows the empty picker and renders the metric widgets in their normal \u0026ldquo;no data\u0026rdquo; state, so the layout stays visible and ready for services that do report them. Clearer cluster boundaries on every topology view. The dashed grouping boxes — namespaces on the service map, per-service boxes on the instance map, role/tier clusters on the Deployment tab — now draw with a bolder, brighter dashed border and a fully transparent background, so the boundary reads clearly on every theme (light themes included) instead of fading into the canvas. The Deployment tab also packs its cluster boxes evenly: boxes sit at a uniform spacing with no dead corridor between tiers and no blank strip before the first box. Live debugger MAL sample groups. A captured step that fans out to many samples no longer dumps every label set on screen: the samples are grouped by metric name into a one-line summary — \u0026lt;metric\u0026gt; · N samples · values=… — and you expand only the groups you care about to see each sample\u0026rsquo;s full labels. Groups are collapsed by default. Diff is the default when a group is expanded. A multi-sample group opens straight into diff view: the labels shared by every sample collapse into a dimmed \u0026ldquo;common\u0026rdquo; block and only the labels that differ are highlighted per sample — so it is immediate which label distinguishes each one (e.g. node_role / pod_name) and what value it maps to. A diff toggle beside the group\u0026rsquo;s header switches back to the full per-sample label list. The \u0026ldquo;common\u0026rdquo; set is computed across the whole group, not just the rendered rows. Multiple output entities collapse the same way. When a record materialises one metric for several entities (e.g. a per-endpoint write rate over sw_metricsMinute / sw_metricsHour / sw_metricsDay), the repeated meter cards fold into one block: a shared header (metric / function / time bucket), a N outputs · values=… summary, and a diff that surfaces only the entity fields that actually differ — whichever they are, not a fixed field — with each output\u0026rsquo;s value beside it. Readable sample values. Long fractional values from rate() / avg() (e.g. 57.0333333333…) are trimmed to a few significant digits for display so they stop overflowing the value column; integer counters still render exact, and the precise value stays available on hover. DSL management — live apply progress \u0026amp; recovery A structural rule change now shows live apply progress. Saving an edit that moves a metric\u0026rsquo;s storage shape (scope, downsampling, or the metric set) no longer just flashes \u0026ldquo;submitted\u0026rdquo; — the editor tracks the apply across the cluster through a phase stepper (Compiled → Confirming across the cluster → Committing → Done) and reports success only once OAP confirms the change is durable. Revert to bundled (also a schema change) goes through the same stepper. Body- and filter-only edits still apply instantly with no stepper. You can navigate away mid-apply; reloading the editor resumes the progress. \u0026ldquo;Applied — cluster propagation unconfirmed\u0026rdquo; is a warning, not an error. When a structural change is committed and durable but one or more nodes hadn\u0026rsquo;t confirmed the new schema within OAP\u0026rsquo;s fence budget, the editor names the lagging nodes and explains they self-converge on their next scan — the rule is applied, not rolled back. Reloading the editor reads it back as applied (from the stored rule). A failed apply is called out as rolled back — the cluster stays on the previous rule, the failure reason is shown inline, and your edit is kept in the editor so you can fix and save again. A compile error now surfaces as an inline diagnostic under the editor instead of a transient toast. Force re-apply to recover. A degraded or transiently-failed apply offers a one-click Force re-apply (recover) that re-runs the apply across the cluster to re-confirm the schema and un-stick any waiting node — gated behind a confirm that spells out it briefly pauses collection for that rule\u0026rsquo;s metrics, even when the content is unchanged. This subsumes the old Advanced force toggle for the recovery case. ","excerpt":"\u003ch1 id=\"070\"\u003e0.7.0\u003c/h1\u003e\n\u003ch3 id=\"browser-errors--source-maps\"\u003eBrowser errors \u0026amp; source maps\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eNew \u0026ldquo;Browser Logs\u0026rdquo; tab on the BROWSER layer\u003c/strong\u003e — …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/changelog/0.7.0/","title":"0.7.0"},{"body":"0.7.0 Features Replace go-bindata with embed lib. Add the OAPServerConfig CRD, webhooks and controller. Add the OAPServerDynamicConfig CRD, webhooks and controller. Add the SwAgent CRD, webhooks and controller. [Breaking Change] Remove the way to configure the agent through Configmap. Bugs Fix the error in e2e testing. Fix status inconsistent with CI. Bump up prometheus client version to fix cve. Chores Bump several dependencies of adapter. Update license eye version. Bump up SkyWalking OAP to 9.0.0. Bump up the k8s api of the e2e environment to v1.21.10. ","excerpt":"\u003ch2 id=\"070\"\u003e0.7.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eReplace go-bindata with embed lib.\u003c/li\u003e\n\u003cli\u003eAdd the OAPServerConfig CRD, webhooks and …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/latest/en/changes/changes-0.7.0/","title":"0.7.0"},{"body":"0.7.0 Features Replace go-bindata with embed lib. Add the OAPServerConfig CRD, webhooks and controller. Add the OAPServerDynamicConfig CRD, webhooks and controller. Add the SwAgent CRD, webhooks and controller. [Breaking Change] Remove the way to configure the agent through Configmap. Bugs Fix the error in e2e testing. Fix status inconsistent with CI. Bump up prometheus client version to fix cve. Chores Bump several dependencies of adapter. Update license eye version. Bump up SkyWalking OAP to 9.0.0. Bump up the k8s api of the e2e environment to v1.21.10. ","excerpt":"\u003ch2 id=\"070\"\u003e0.7.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eReplace go-bindata with embed lib.\u003c/li\u003e\n\u003cli\u003eAdd the OAPServerConfig CRD, webhooks and …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/next/en/changes/changes-0.7.0/","title":"0.7.0"},{"body":"0.7.0 Features Replace go-bindata with embed lib. Add the OAPServerConfig CRD, webhooks and controller. Add the OAPServerDynamicConfig CRD, webhooks and controller. Add the SwAgent CRD, webhooks and controller. [Breaking Change] Remove the way to configure the agent through Configmap. Bugs Fix the error in e2e testing. Fix status inconsistent with CI. Bump up prometheus client version to fix cve. Chores Bump several dependencies of adapter. Update license eye version. Bump up SkyWalking OAP to 9.0.0. Bump up the k8s api of the e2e environment to v1.21.10. ","excerpt":"\u003ch2 id=\"070\"\u003e0.7.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eReplace go-bindata with embed lib.\u003c/li\u003e\n\u003cli\u003eAdd the OAPServerConfig CRD, webhooks and …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/v0.11.0/en/changes/changes-0.7.0/","title":"0.7.0"},{"body":"0.8.0 Features [Breaking Change] Remove the way to configure the agent through Configmap. Bugs Fix errors in banyandb e2e test. Chores Bump up golang to v1.20. Bump up golangci-lint to v1.53.3. Bump up skywalking-java-agent to v8.16.0. Bump up kustomize to v4.5.6. Bump up SkyWalking OAP to 9.5.0. ","excerpt":"\u003ch2 id=\"080\"\u003e0.8.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e[Breaking Change] Remove the way to configure the agent through Configmap.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"bugs\"\u003eBugs …\u003c/h4\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/latest/en/changes/changes-0.8.0/","title":"0.8.0"},{"body":"0.8.0 Features [Breaking Change] Remove the way to configure the agent through Configmap. Bugs Fix errors in banyandb e2e test. Chores Bump up golang to v1.20. Bump up golangci-lint to v1.53.3. Bump up skywalking-java-agent to v8.16.0. Bump up kustomize to v4.5.6. Bump up SkyWalking OAP to 9.5.0. ","excerpt":"\u003ch2 id=\"080\"\u003e0.8.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e[Breaking Change] Remove the way to configure the agent through Configmap.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"bugs\"\u003eBugs …\u003c/h4\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/next/en/changes/changes-0.8.0/","title":"0.8.0"},{"body":"0.8.0 Features [Breaking Change] Remove the way to configure the agent through Configmap. Bugs Fix errors in banyandb e2e test. Chores Bump up golang to v1.20. Bump up golangci-lint to v1.53.3. Bump up skywalking-java-agent to v8.16.0. Bump up kustomize to v4.5.6. Bump up SkyWalking OAP to 9.5.0. ","excerpt":"\u003ch2 id=\"080\"\u003e0.8.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e[Breaking Change] Remove the way to configure the agent through Configmap.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"bugs\"\u003eBugs …\u003c/h4\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/v0.11.0/en/changes/changes-0.8.0/","title":"0.8.0"},{"body":"0.9.0 Features Add a getting started document about how to deploy swck on the kubernetes cluster. Bugs Fix the bug that the java agent is duplicated injected when update the pod. Chores Bump up custom-metrics-apiserver Bump up golang to v1.22 Bump up controller-gen to v0.14.0 ","excerpt":"\u003ch2 id=\"090\"\u003e0.9.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd a getting started document about how to deploy swck on the kubernetes cluster. …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/latest/en/changes/changes-0.9.0/","title":"0.9.0"},{"body":"0.9.0 Features Add a getting started document about how to deploy swck on the kubernetes cluster. Bugs Fix the bug that the java agent is duplicated injected when update the pod. Chores Bump up custom-metrics-apiserver Bump up golang to v1.22 Bump up controller-gen to v0.14.0 ","excerpt":"\u003ch2 id=\"090\"\u003e0.9.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd a getting started document about how to deploy swck on the kubernetes cluster. …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/next/en/changes/changes-0.9.0/","title":"0.9.0"},{"body":"0.9.0 Features Add a getting started document about how to deploy swck on the kubernetes cluster. Bugs Fix the bug that the java agent is duplicated injected when update the pod. Chores Bump up custom-metrics-apiserver Bump up golang to v1.22 Bump up controller-gen to v0.14.0 ","excerpt":"\u003ch2 id=\"090\"\u003e0.9.0\u003c/h2\u003e\n\u003ch4 id=\"features\"\u003eFeatures\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd a getting started document about how to deploy swck on the kubernetes cluster. …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/skywalking-swck/v0.11.0/en/changes/changes-0.9.0/","title":"0.9.0"},{"body":"1.0.0 The first release of Horizon UI — the next-generation web console for Apache SkyWalking. A dark, dense, information-first interface over the same OAP query protocol and MQE the previous console used, with layer-driven dashboards you configure rather than code, an AI assistant that reads your live data, and an MCP endpoint so the agent you already use can read it too.\nAI assistant Features Ask about your system in plain language and get answers built from real dashboard widgets, not just text. A launcher on the right edge opens a chat: describe what you want to know and the assistant reads live data, then streams back an ordered narrative with inline charts, top-N lists and tables drawn by the same components the dashboards use. Open it as a side drawer, expand it to a full page, or put it in its own tab. It is read-only and inherits your permissions. It can list services, read active alarms, browse each layer\u0026rsquo;s metric catalog, drill a service down to its instances and endpoints, and chart any of it — never seeing more than you can, and never changing configuration, rules or dashboards. It embeds the real product views, scoped to the service you asked about. Ask for topology, traces, logs, browser errors, deployment, API dependencies, an instance map or a cross-layer hierarchy and the actual view mounts inside the chat, interactions intact — click a trace and its span waterfall opens. Both native SkyWalking and Zipkin tracing are covered. Everything it shows is a snapshot, and says so. Each block carries a replay badge and the time it was captured, and re-renders identically when you reopen the conversation — offline, with its edge sparklines and its detail views — rather than quietly re-querying and showing today\u0026rsquo;s data under yesterday\u0026rsquo;s question. It can read Kubernetes pod logs in the chat, as a result rather than a console. When a filter was applied it says so, so an empty result reads as \u0026ldquo;nothing matched\u0026rdquo; rather than a silent pod. It can propose profiling, and only you start it. When metrics and traces cannot localise a cause it presents a decision card explaining what it found and what profiling would reveal; nothing runs until you approve it, and only if you hold the permission. It picks the flavour that fits the target and renders the result — a flame graph, a profiled trace\u0026rsquo;s waterfall beside its flame, or a network conversation graph — inline once collected. Guided root-cause analysis. Ask what the root cause is and it follows built-in investigation playbooks — a master method plus latency, error-rate, saturation, middleware, Kubernetes-workload and service-mesh specialisations — including following a service down into the infrastructure layer behind it, where memory, disk and connection causes live. It answers in each layer\u0026rsquo;s own vocabulary, calling a Kubernetes instance a Pod and a mesh instance a Sidecar, and reads your configured warning thresholds rather than guessing what \u0026ldquo;healthy\u0026rdquo; means. An outage is reported as an outage. When Horizon cannot reach the backend, it says so and stops, instead of reporting every layer as having no metrics — which reads as \u0026ldquo;your services aren\u0026rsquo;t reporting\u0026rdquo; and sends you looking for a problem in your own system. Your question stays in view while the answer streams, pinned to the top of the chat, with a timestamp for when you asked and when the answer finished. Conversations are kept per user in your browser, with a usage meter, a save toggle and a clear-all. History is stored unencrypted, and the page says so — turn it off on shared workstations. Two tabs cannot overwrite each other\u0026rsquo;s chats. Bring your own model. Off by default, vendor-neutral, and configured by an administrator: any OpenAI-compatible endpoint — hosted, local or a gateway — or Amazon Bedrock. The assistant\u0026rsquo;s instructions and the starter prompts shown in an empty chat both ship with defaults you can replace entirely. Until it is pointed at a model the panel opens read-only with a short note. Agent access over MCP Features The agent you already use can read Horizon. Point Claude Code, Codex, Claude Desktop or any Model Context Protocol client at Horizon and it gets the same tools the AI assistant uses — the metric catalog, figures, topology, traces, logs, Kubernetes, profiling proposals and the root-cause playbooks. The model stays on the caller\u0026rsquo;s side, so no provider and no API key are configured here. It is not a new exposure. The endpoint needs the same login as every other route, a permission gates the connection, and each tool re-checks the permission its own screen needs. An agent sees exactly what the operator it authenticated as sees. An agent can log in through your browser instead of being handed a token. It opens Horizon\u0026rsquo;s own login page — backed by whatever your organisation configured — you approve once on a consent screen, and it keeps its token from there. The screen shows the permissions the grant would really carry, filtered by what you actually hold, so it never promises access you cannot delegate. Off by default. A client that can draw gets the real widgets, not a picture of them: the same charts, topology graphs and trace lists the console draws, fed the captured snapshot. A terminal client reads the data itself and presents it in its own way. Two deployments look like two deployments. A Horizon can name itself, and that name reaches the agent — so an operator watching production and staging is never told which is which by guesswork. Nothing served over MCP writes anything, and every tool declares itself read-only, so a host stops asking you to approve each step of a single investigation. Sign-in and access control Features Sign in with your identity provider. Google, Okta, Entra, Keycloak or anything else speaking OpenID Connect — each configured provider becomes a button on the login page. Providers that issue only an access token are supported too. It is additive by design: password login keeps working alongside it, so a misconfigured provider never locks you out during an incident. The login page fits what you configured. With password login present, providers fold into one picker; where single sign-on is the only way in, up to four get their own button and the rest fold into a picker. A deployment with no password backend hides the username and password boxes entirely, rather than showing a form that cannot succeed. An identity provider says who you are; it does not decide what you may do here. New sign-ins are viewers unless you say otherwise, you decide which domains may sign in at all, and per-address and per-domain overrides raise individuals. Horizon calls you what your directory calls you, showing your display name rather than a raw address or an account id. The verified address stays the identity behind every permission check, one hover away. An account page, reached by clicking your own name. Who you are, how you proved it — local account, directory, single sign-on with the provider that vouched for you, break-glass, or a token — and which roles you hold and what they grant, so \u0026ldquo;why can I not see this page\u0026rdquo; has an answer that does not need an administrator. API tokens for callers with no browser. Scripts, CI jobs and agents authenticate on every route under exactly the permissions that route requires. A token names a user and can never carry more than that user currently holds; removing the user revokes it. What one session read never reaches the next person to sign in on that browser. Signing out, signing in, or having a session end mid-use discards everything cached, so a shared workstation cannot serve the previous operator\u0026rsquo;s services, alarms, traces or dashboards to the next one. Shortening the session timeout now shortens the sessions you already have, not just new ones — which is what you want right after an incident. The roles board tells the truth about what each role sees. Every navigation entry is listed, permissions that gate nothing are marked as reserved, and an entry a role cannot open is not offered to it. Fixes Sign-in accepts up to 64 characters for the username and 64 for the password, and the form stops at the same limit rather than letting a longer value reach the server and come back as a generic failure. A directory that issues credentials longer than this cannot be used to sign in — see Local backend.\nThe sign-in card no longer runs off the edge of a phone-width window — it was cut off below roughly 410px.\nSource-map upload and removal are disabled, with the reason on hover, for operators who lack source-map:write — rather than looking available and then failing. Removing a map now asks first: it un-symbolicates stacks for everyone reading that layer.\nLogin audit Features A durable record of who signed in, when, and from where. Optional and off by default, backed by a shared database rather than a file. An hourly summary stacked by how people signed in, then filters, then the list — statistics first, because the first question is whether anything unusual is happening and the second is which. It records only what a valid credential produced. Successful sign-ins, plus the two refusals that happen after authentication already succeeded. A wrong password or an unknown user stays in the application log, because those are what an anonymous caller can produce at will. Nothing that could resume a session is ever recorded. Signing in never waits for the database, and cannot be blocked by it. Records are written in the background; an unreachable database is invisible to the person signing in, and the page says it cannot be reached rather than showing an empty table. Token traffic is counted on its own tab, at its own grain. A sign-in is a person arriving; token traffic is a machine at work, and stacking them let a busy script outweigh every human sign-in beside it. One row per token per hour, grouped by hour, with each hour\u0026rsquo;s totals and its busiest credentials — and a line that always says whether you are seeing all of them or the top ten. Reading it needs its own permission that a wildcard does not grant, because the log holds verified email addresses and client addresses. There is no write and no delete. Fixes A failing statistics write no longer reports the audit store as healthy. Sign-in records, token counts and hourly statistics are written on separate schedules; one of them succeeding used to clear another\u0026rsquo;s failure, and the periodic reachability check cleared a statistics failure it does not actually test. Each is now tracked on its own, so the store reads unhealthy for as long as anything is failing to write. An audit write that fails is no longer retried or held. Sign-in batches and hourly statistics are dropped when the database refuses them, so memory does not grow for the length of an outage and nothing is replayed afterwards — the sign-ins from that window are counted as unrecorded rather than reconstructed. Batching itself is unchanged: writes are still grouped for efficiency. Dashboards Features Layer dashboards are configuration, not code. Every layer\u0026rsquo;s screens are defined by a template you edit in the console — widgets, scopes, service-list columns, thresholds and labels — and published to your backend. Forty-six bundled dashboards ship ready to use, catalogued the way the sidebar groups them. A layer\u0026rsquo;s Service, Instance and Endpoint views can each carry more than one page. Each becomes its own row under the layer with its own URL and its own widgets, so a layer\u0026rsquo;s metrics need not share one screen. A page can name the entity it lists — \u0026ldquo;Brokers\u0026rdquo; rather than \u0026ldquo;Instances\u0026rdquo; — and can narrow which services or instances it is about. A new tab widget packs related views into one slot. A grid tile can hold any number of named tabs, each its own small dashboard, edited right where it sits. Only the active tab is queried, so an unopened tab costs nothing. The dashboard editor works beside the canvas. Picking a widget kind is a menu with descriptions rather than always dropping in a card you retype; the editor pins next to the board and opens complete, wherever on the board you clicked; adding a widget scrolls it into view. Rows under a layer can be put in your own order, dragged in a live preview of the real menu, with a reset to the built-in order. Publishing refuses a template that would break the layer, naming the field at fault and writing nothing — rather than storing it and emptying that layer\u0026rsquo;s screen for everyone. Work in progress still publishes: an empty expression or a half-filled section is a normal state of an unfinished draft. Cards can render values as coloured status chips rather than bare numbers — so the Kubernetes node status reads as Ready in green and its pressure conditions in amber or red, instead of a raw 1. Click a latency or error point on a chart to open the matching traces, pre-filtered to that service and centred on the bucket you clicked, opening slowest-first or error-only depending on the metric. Dashboard authors turn it on per widget. Compare several entities on one dashboard, with one-click exit from comparison. Overview dashboards roll up a whole layer, with per-widget control over how that aggregation is done and how the top services are ranked. Traces, logs and events Features A trace explorer with a duration-distribution scatter, a time-positioned waterfall and a span detail modal — and Zipkin traces render with the same experience as native ones, including plain-language hints for Zipkin\u0026rsquo;s annotation codes. One shareable link opens either kind. Logs and browser errors query on demand. Conditions stage until you press Run query, so a fresh tab prompts you rather than firing a broad query, and switching service resets rather than leaving the previous service\u0026rsquo;s rows under the new name. Stored logs can be searched by their content where the backend supports it — the field appears only on a backend that can actually answer it, rather than silently ignoring what you typed. Clicking a log row opens a full payload popout with format-aware pretty-printing, the tag table, and a link to the trace. Cross-layer inspection for raw logs, browser errors and Kubernetes pod logs. Browser errors carry source-map upload and de-obfuscation, resolving a minified stack back to the original frames with a source snippet. Pod logs tail a container on demand and are never stored. A per-service events popout on every layer\u0026rsquo;s service banner — agent restarts, Kubernetes events and other lifecycle records — laid out as one row per instance on a time axis, with a search box for services running hundreds of them. Result lists say when they were capped, and offer a next page only when there is one with rows on it. A pager reports the page and what is on it, rather than a total the backend does not provide. Tag fields autocomplete on theme, suggesting keys and then per-key values in a dense dropdown instead of the browser\u0026rsquo;s native popup. Fixes A custom time range that cannot be read is now refused, with the reason under the control — Traces, Zipkin traces, Logs, Browser errors and both Inspect pages. A reversed, half-filled or over-wide range used to be swapped silently for a default window, so the results answered a question nobody asked. Ranges longer than six hours also carry a note that they can be slow on a large deployment; that one is advice, and the query still runs. A request made directly against the API rather than through a page is trimmed to the most recent week instead of being refused, so it still answers with the part that matters.\nSwitching service no longer leaves the previous service\u0026rsquo;s endpoints, instances or profiling segments on screen. The dependent lists clear immediately and say Reading… while the new ones load, and a slow reply for a selection you have already moved off is discarded instead of overwriting the current one. This affects the Inspect pages\u0026rsquo; Service/Instance/Endpoint pickers, all five profiling tabs\u0026rsquo; task and segment lists, the network-profiling process graph, and the Zipkin span/remote autocomplete.\nProfiling Features Five kinds of profiling in one place — trace sampling, async-profiler for JVM services, pprof for Go services, eBPF on/off-CPU, and network profiling — each with a task list, a create dialog that tells you upfront what it needs, and a flame graph or conversation graph for the result. Continuous profiling has a home. Arm a policy once and the task starts itself when a process crosses a threshold, with nobody present — which is how you catch the problem that only appears at 3 a.m. Each target lists the instances and processes actually being evaluated and how often each has fired: the difference between a policy that is stored and one that is working. A task a policy started is visible beside the ones you started by hand, newest first, instead of appearing nowhere. A profiling request that cannot be honoured is refused with the reason, rather than quietly repaired into something more expensive or trimmed to a fraction of the fleet you asked for. Kubernetes services gain network profiling — pick a pod and capture process-to-process conversations as a topology, the same capability the mesh layer offers. Look and feel Features Dark, dense and information-first. Horizon is built for an operator watching a system, not for a marketing page: tight tables, small type, and as much signal per screen as stays legible. Escape closes any dismissible panel — modals, row popouts and the topology dropdowns alike. Searchable, on-theme dropdowns everywhere a picker lists more than a handful of entries, replacing the browser\u0026rsquo;s native controls. Denser Kubernetes tables, more rows without scrolling. The live debugger reads cleanly on tall and wide captures — the frozen first column stays pinned as you scroll sideways, clicking a source line flashes the whole matching step, and long captures scroll as one page instead of trapping the result in a fixed-height box. A step that dropped a record says why, in the backend\u0026rsquo;s own words, instead of leaving you to reconstruct the cause from the payload. Serving Horizon under a path prefix is a first-class option, for a reverse proxy that strips it before forwarding. Languages Features The whole console speaks eight languages — English plus German, Spanish, French, Japanese, Korean, Portuguese and Simplified Chinese. Product, protocol and metric names stay in their original form, because those are what operators read across the docs, the source and every other SkyWalking surface. Dashboard text is translated in the console, per language, on a page that shows exactly what the site renders rather than the shipped defaults — with staged drafts, a diff before you publish, and a reset to bundled. A translation belongs to its widget, not to the widget\u0026rsquo;s position, so rearranging a dashboard leaves every other widget\u0026rsquo;s translation where it was — a deleted widget takes its translation with it, and a new one starts out English until you fill it in. Text left behind by a template edit is found and cleaned up deliberately, rather than being discarded by the next unrelated save. Operating Horizon Features The container image runs on environment variables alone — no mounted configuration file, no repackaging. The shipped configuration file doubles as the complete, self-documenting reference for every variable. Run against a backend whose template store you cannot write, rendering every dashboard from the bundled templates and never calling the template API. The configuration surface becomes honestly read-only, while metrics, traces, logs and topology work exactly as before. This is the supported way to run against an OAP release that has no template management endpoint. Cluster Status reports what is actually reachable, testing the real path each feature calls rather than inferring health from configuration being present — so a module that is loaded but broken reads as unreachable instead of a misleading green. Configuration hot-reloads, and a rejected reload says so out loud, naming the field at fault and continuing to serve the last valid configuration rather than silently ignoring the edit or falling back to defaults. Query fan-out is tunable per deployment — batch sizes, concurrency and protective caps — so a beefy backend can be pushed harder and a modest one protected. Defaults match the built-in behaviour, so the whole thing is optional. Responses are not cached by the browser, so metrics, traces, logs and configuration are not left behind on a shared workstation. The console\u0026rsquo;s own files stay cacheable, so nothing gets slower. A strict content policy ships by default, permitting scripts only from Horizon\u0026rsquo;s own origin, forbidding inline script, and refusing to be framed. It needs no configuration. Outbound documentation links are restricted to hosts you trust, and anything that is not a real web address is refused outright — both when a template is published and when one is read back. Warnings reach standard output by default. Break-glass logins, directory failures and rejected configuration reloads are the things you want to see without having raised the log level first. A duplicated dashboard record is reported, never resolved behind your back. A dashboard whose definition is ambiguous is hidden rather than rendered from whichever copy happened to win, and opening it by URL explains why and points at where to fix it. Deciding which copy survives is a deliberate cleanup, not something a restart does for you. Fixes The DSL editor asks before deleting a rule that has no bundled version, and warns before you navigate away or reload with unsaved YAML. Deleting such a rule removes the only copy.\nThe alarms and events Custom range no longer applies as soon as you open it. It takes effect on Apply, and Cancel or Escape leaves the range you were looking at alone.\nThe OAL file viewer no longer strands you on an expired session, and switching files quickly cannot leave one file\u0026rsquo;s contents under another\u0026rsquo;s name.\ncli:hash completes when you press Enter. Typing a password interactively used to hang, because the command waited for end-of-input; the argument and piped forms were unaffected.\n","excerpt":"\u003ch1 id=\"100\"\u003e1.0.0\u003c/h1\u003e\n\u003cp\u003eThe first release of Horizon UI — the next-generation web console for Apache SkyWalking. A …\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/latest/changelog/1.0.0/","title":"1.0.0"},{"body":"1.0.0 The first release of Horizon UI — the next-generation web console for Apache SkyWalking. A dark, dense, information-first interface over the same OAP query protocol and MQE the previous console used, with layer-driven dashboards you configure rather than code, an AI assistant that reads your live data, and an MCP endpoint so the agent you already use can read it too.\nAI assistant Features Ask about your system in plain language and get answers built from real dashboard widgets, not just text. A launcher on the right edge opens a chat: describe what you want to know and the assistant reads live data, then streams back an ordered narrative with inline charts, top-N lists and tables drawn by the same components the dashboards use. Open it as a side drawer, expand it to a full page, or put it in its own tab. It is read-only and inherits your permissions. It can list services, read active alarms, browse each layer\u0026rsquo;s metric catalog, drill a service down to its instances and endpoints, and chart any of it — never seeing more than you can, and never changing configuration, rules or dashboards. It embeds the real product views, scoped to the service you asked about. Ask for topology, traces, logs, browser errors, deployment, API dependencies, an instance map or a cross-layer hierarchy and the actual view mounts inside the chat, interactions intact — click a trace and its span waterfall opens. Both native SkyWalking and Zipkin tracing are covered. Everything it shows is a snapshot, and says so. Each block carries a replay badge and the time it was captured, and re-renders identically when you reopen the conversation — offline, with its edge sparklines and its detail views — rather than quietly re-querying and showing today\u0026rsquo;s data under yesterday\u0026rsquo;s question. It can read Kubernetes pod logs in the chat, as a result rather than a console. When a filter was applied it says so, so an empty result reads as \u0026ldquo;nothing matched\u0026rdquo; rather than a silent pod. It can propose profiling, and only you start it. When metrics and traces cannot localise a cause it presents a decision card explaining what it found and what profiling would reveal; nothing runs until you approve it, and only if you hold the permission. It picks the flavour that fits the target and renders the result — a flame graph, a profiled trace\u0026rsquo;s waterfall beside its flame, or a network conversation graph — inline once collected. Guided root-cause analysis. Ask what the root cause is and it follows built-in investigation playbooks — a master method plus latency, error-rate, saturation, middleware, Kubernetes-workload and service-mesh specialisations — including following a service down into the infrastructure layer behind it, where memory, disk and connection causes live. It answers in each layer\u0026rsquo;s own vocabulary, calling a Kubernetes instance a Pod and a mesh instance a Sidecar, and reads your configured warning thresholds rather than guessing what \u0026ldquo;healthy\u0026rdquo; means. An outage is reported as an outage. When Horizon cannot reach the backend, it says so and stops, instead of reporting every layer as having no metrics — which reads as \u0026ldquo;your services aren\u0026rsquo;t reporting\u0026rdquo; and sends you looking for a problem in your own system. Your question stays in view while the answer streams, pinned to the top of the chat, with a timestamp for when you asked and when the answer finished. Conversations are kept per user in your browser, with a usage meter, a save toggle and a clear-all. History is stored unencrypted, and the page says so — turn it off on shared workstations. Two tabs cannot overwrite each other\u0026rsquo;s chats. Bring your own model. Off by default, vendor-neutral, and configured by an administrator: any OpenAI-compatible endpoint — hosted, local or a gateway — or Amazon Bedrock. The assistant\u0026rsquo;s instructions and the starter prompts shown in an empty chat both ship with defaults you can replace entirely. Until it is pointed at a model the panel opens read-only with a short note. Agent access over MCP Features The agent you already use can read Horizon. Point Claude Code, Codex, Claude Desktop or any Model Context Protocol client at Horizon and it gets the same tools the AI assistant uses — the metric catalog, figures, topology, traces, logs, Kubernetes, profiling proposals and the root-cause playbooks. The model stays on the caller\u0026rsquo;s side, so no provider and no API key are configured here. It is not a new exposure. The endpoint needs the same login as every other route, a permission gates the connection, and each tool re-checks the permission its own screen needs. An agent sees exactly what the operator it authenticated as sees. An agent can log in through your browser instead of being handed a token. It opens Horizon\u0026rsquo;s own login page — backed by whatever your organisation configured — you approve once on a consent screen, and it keeps its token from there. The screen shows the permissions the grant would really carry, filtered by what you actually hold, so it never promises access you cannot delegate. Off by default. A client that can draw gets the real widgets, not a picture of them: the same charts, topology graphs and trace lists the console draws, fed the captured snapshot. A terminal client reads the data itself and presents it in its own way. Two deployments look like two deployments. A Horizon can name itself, and that name reaches the agent — so an operator watching production and staging is never told which is which by guesswork. Nothing served over MCP writes anything, and every tool declares itself read-only, so a host stops asking you to approve each step of a single investigation. Sign-in and access control Features Sign in with your identity provider. Google, Okta, Entra, Keycloak or anything else speaking OpenID Connect — each configured provider becomes a button on the login page. Providers that issue only an access token are supported too. It is additive by design: password login keeps working alongside it, so a misconfigured provider never locks you out during an incident. The login page fits what you configured. With password login present, providers fold into one picker; where single sign-on is the only way in, up to four get their own button and the rest fold into a picker. A deployment with no password backend hides the username and password boxes entirely, rather than showing a form that cannot succeed. An identity provider says who you are; it does not decide what you may do here. New sign-ins are viewers unless you say otherwise, you decide which domains may sign in at all, and per-address and per-domain overrides raise individuals. Horizon calls you what your directory calls you, showing your display name rather than a raw address or an account id. The verified address stays the identity behind every permission check, one hover away. An account page, reached by clicking your own name. Who you are, how you proved it — local account, directory, single sign-on with the provider that vouched for you, break-glass, or a token — and which roles you hold and what they grant, so \u0026ldquo;why can I not see this page\u0026rdquo; has an answer that does not need an administrator. API tokens for callers with no browser. Scripts, CI jobs and agents authenticate on every route under exactly the permissions that route requires. A token names a user and can never carry more than that user currently holds; removing the user revokes it. What one session read never reaches the next person to sign in on that browser. Signing out, signing in, or having a session end mid-use discards everything cached, so a shared workstation cannot serve the previous operator\u0026rsquo;s services, alarms, traces or dashboards to the next one. Shortening the session timeout now shortens the sessions you already have, not just new ones — which is what you want right after an incident. The roles board tells the truth about what each role sees. Every navigation entry is listed, permissions that gate nothing are marked as reserved, and an entry a role cannot open is not offered to it. Fixes Sign-in accepts up to 64 characters for the username and 64 for the password, and the form stops at the same limit rather than letting a longer value reach the server and come back as a generic failure. A directory that issues credentials longer than this cannot be used to sign in — see Local backend.\nThe sign-in card no longer runs off the edge of a phone-width window — it was cut off below roughly 410px.\nSource-map upload and removal are disabled, with the reason on hover, for operators who lack source-map:write — rather than looking available and then failing. Removing a map now asks first: it un-symbolicates stacks for everyone reading that layer.\nLogin audit Features A durable record of who signed in, when, and from where. Optional and off by default, backed by a shared database rather than a file. An hourly summary stacked by how people signed in, then filters, then the list — statistics first, because the first question is whether anything unusual is happening and the second is which. It records only what a valid credential produced. Successful sign-ins, plus the two refusals that happen after authentication already succeeded. A wrong password or an unknown user stays in the application log, because those are what an anonymous caller can produce at will. Nothing that could resume a session is ever recorded. Signing in never waits for the database, and cannot be blocked by it. Records are written in the background; an unreachable database is invisible to the person signing in, and the page says it cannot be reached rather than showing an empty table. Token traffic is counted on its own tab, at its own grain. A sign-in is a person arriving; token traffic is a machine at work, and stacking them let a busy script outweigh every human sign-in beside it. One row per token per hour, grouped by hour, with each hour\u0026rsquo;s totals and its busiest credentials — and a line that always says whether you are seeing all of them or the top ten. Reading it needs its own permission that a wildcard does not grant, because the log holds verified email addresses and client addresses. There is no write and no delete. Fixes A failing statistics write no longer reports the audit store as healthy. Sign-in records, token counts and hourly statistics are written on separate schedules; one of them succeeding used to clear another\u0026rsquo;s failure, and the periodic reachability check cleared a statistics failure it does not actually test. Each is now tracked on its own, so the store reads unhealthy for as long as anything is failing to write. An audit write that fails is no longer retried or held. Sign-in batches and hourly statistics are dropped when the database refuses them, so memory does not grow for the length of an outage and nothing is replayed afterwards — the sign-ins from that window are counted as unrecorded rather than reconstructed. Batching itself is unchanged: writes are still grouped for efficiency. Dashboards Features Layer dashboards are configuration, not code. Every layer\u0026rsquo;s screens are defined by a template you edit in the console — widgets, scopes, service-list columns, thresholds and labels — and published to your backend. Forty-six bundled dashboards ship ready to use, catalogued the way the sidebar groups them. A layer\u0026rsquo;s Service, Instance and Endpoint views can each carry more than one page. Each becomes its own row under the layer with its own URL and its own widgets, so a layer\u0026rsquo;s metrics need not share one screen. A page can name the entity it lists — \u0026ldquo;Brokers\u0026rdquo; rather than \u0026ldquo;Instances\u0026rdquo; — and can narrow which services or instances it is about. A new tab widget packs related views into one slot. A grid tile can hold any number of named tabs, each its own small dashboard, edited right where it sits. Only the active tab is queried, so an unopened tab costs nothing. The dashboard editor works beside the canvas. Picking a widget kind is a menu with descriptions rather than always dropping in a card you retype; the editor pins next to the board and opens complete, wherever on the board you clicked; adding a widget scrolls it into view. Rows under a layer can be put in your own order, dragged in a live preview of the real menu, with a reset to the built-in order. Publishing refuses a template that would break the layer, naming the field at fault and writing nothing — rather than storing it and emptying that layer\u0026rsquo;s screen for everyone. Work in progress still publishes: an empty expression or a half-filled section is a normal state of an unfinished draft. Cards can render values as coloured status chips rather than bare numbers — so the Kubernetes node status reads as Ready in green and its pressure conditions in amber or red, instead of a raw 1. Click a latency or error point on a chart to open the matching traces, pre-filtered to that service and centred on the bucket you clicked, opening slowest-first or error-only depending on the metric. Dashboard authors turn it on per widget. Compare several entities on one dashboard, with one-click exit from comparison. Overview dashboards roll up a whole layer, with per-widget control over how that aggregation is done and how the top services are ranked. Traces, logs and events Features A trace explorer with a duration-distribution scatter, a time-positioned waterfall and a span detail modal — and Zipkin traces render with the same experience as native ones, including plain-language hints for Zipkin\u0026rsquo;s annotation codes. One shareable link opens either kind. Logs and browser errors query on demand. Conditions stage until you press Run query, so a fresh tab prompts you rather than firing a broad query, and switching service resets rather than leaving the previous service\u0026rsquo;s rows under the new name. Stored logs can be searched by their content where the backend supports it — the field appears only on a backend that can actually answer it, rather than silently ignoring what you typed. Clicking a log row opens a full payload popout with format-aware pretty-printing, the tag table, and a link to the trace. Cross-layer inspection for raw logs, browser errors and Kubernetes pod logs. Browser errors carry source-map upload and de-obfuscation, resolving a minified stack back to the original frames with a source snippet. Pod logs tail a container on demand and are never stored. A per-service events popout on every layer\u0026rsquo;s service banner — agent restarts, Kubernetes events and other lifecycle records — laid out as one row per instance on a time axis, with a search box for services running hundreds of them. Result lists say when they were capped, and offer a next page only when there is one with rows on it. A pager reports the page and what is on it, rather than a total the backend does not provide. Tag fields autocomplete on theme, suggesting keys and then per-key values in a dense dropdown instead of the browser\u0026rsquo;s native popup. Fixes A custom time range that cannot be read is now refused, with the reason under the control — Traces, Zipkin traces, Logs, Browser errors and both Inspect pages. A reversed, half-filled or over-wide range used to be swapped silently for a default window, so the results answered a question nobody asked. Ranges longer than six hours also carry a note that they can be slow on a large deployment; that one is advice, and the query still runs. A request made directly against the API rather than through a page is trimmed to the most recent week instead of being refused, so it still answers with the part that matters.\nSwitching service no longer leaves the previous service\u0026rsquo;s endpoints, instances or profiling segments on screen. The dependent lists clear immediately and say Reading… while the new ones load, and a slow reply for a selection you have already moved off is discarded instead of overwriting the current one. This affects the Inspect pages\u0026rsquo; Service/Instance/Endpoint pickers, all five profiling tabs\u0026rsquo; task and segment lists, the network-profiling process graph, and the Zipkin span/remote autocomplete.\nProfiling Features Five kinds of profiling in one place — trace sampling, async-profiler for JVM services, pprof for Go services, eBPF on/off-CPU, and network profiling — each with a task list, a create dialog that tells you upfront what it needs, and a flame graph or conversation graph for the result. Continuous profiling has a home. Arm a policy once and the task starts itself when a process crosses a threshold, with nobody present — which is how you catch the problem that only appears at 3 a.m. Each target lists the instances and processes actually being evaluated and how often each has fired: the difference between a policy that is stored and one that is working. A task a policy started is visible beside the ones you started by hand, newest first, instead of appearing nowhere. A profiling request that cannot be honoured is refused with the reason, rather than quietly repaired into something more expensive or trimmed to a fraction of the fleet you asked for. Kubernetes services gain network profiling — pick a pod and capture process-to-process conversations as a topology, the same capability the mesh layer offers. Look and feel Features Dark, dense and information-first. Horizon is built for an operator watching a system, not for a marketing page: tight tables, small type, and as much signal per screen as stays legible. Escape closes any dismissible panel — modals, row popouts and the topology dropdowns alike. Searchable, on-theme dropdowns everywhere a picker lists more than a handful of entries, replacing the browser\u0026rsquo;s native controls. Denser Kubernetes tables, more rows without scrolling. The live debugger reads cleanly on tall and wide captures — the frozen first column stays pinned as you scroll sideways, clicking a source line flashes the whole matching step, and long captures scroll as one page instead of trapping the result in a fixed-height box. A step that dropped a record says why, in the backend\u0026rsquo;s own words, instead of leaving you to reconstruct the cause from the payload. Serving Horizon under a path prefix is a first-class option, for a reverse proxy that strips it before forwarding. Languages Features The whole console speaks eight languages — English plus German, Spanish, French, Japanese, Korean, Portuguese and Simplified Chinese. Product, protocol and metric names stay in their original form, because those are what operators read across the docs, the source and every other SkyWalking surface. Dashboard text is translated in the console, per language, on a page that shows exactly what the site renders rather than the shipped defaults — with staged drafts, a diff before you publish, and a reset to bundled. A translation belongs to its widget, not to the widget\u0026rsquo;s position, so rearranging a dashboard leaves every other widget\u0026rsquo;s translation where it was — a deleted widget takes its translation with it, and a new one starts out English until you fill it in. Text left behind by a template edit is found and cleaned up deliberately, rather than being discarded by the next unrelated save. Operating Horizon Features The container image runs on environment variables alone — no mounted configuration file, no repackaging. The shipped configuration file doubles as the complete, self-documenting reference for every variable. Run against a backend whose template store you cannot write, rendering every dashboard from the bundled templates and never calling the template API. The configuration surface becomes honestly read-only, while metrics, traces, logs and topology work exactly as before. This is the supported way to run against an OAP release that has no template management endpoint. Cluster Status reports what is actually reachable, testing the real path each feature calls rather than inferring health from configuration being present — so a module that is loaded but broken reads as unreachable instead of a misleading green. Configuration hot-reloads, and a rejected reload says so out loud, naming the field at fault and continuing to serve the last valid configuration rather than silently ignoring the edit or falling back to defaults. Query fan-out is tunable per deployment — batch sizes, concurrency and protective caps — so a beefy backend can be pushed harder and a modest one protected. Defaults match the built-in behaviour, so the whole thing is optional. Responses are not cached by the browser, so metrics, traces, logs and configuration are not left behind on a shared workstation. The console\u0026rsquo;s own files stay cacheable, so nothing gets slower. A strict content policy ships by default, permitting scripts only from Horizon\u0026rsquo;s own origin, forbidding inline script, and refusing to be framed. It needs no configuration. Outbound documentation links are restricted to hosts you trust, and anything that is not a real web address is refused outright — both when a template is published and when one is read back. Warnings reach standard output by default. Break-glass logins, directory failures and rejected configuration reloads are the things you want to see without having raised the log level first. A duplicated dashboard record is reported, never resolved behind your back. A dashboard whose definition is ambiguous is hidden rather than rendered from whichever copy happened to win, and opening it by URL explains why and points at where to fix it. Deciding which copy survives is a deliberate cleanup, not something a restart does for you. Fixes The DSL editor asks before deleting a rule that has no bundled version, and warns before you navigate away or reload with unsaved YAML. Deleting such a rule removes the only copy.\nThe alarms and events Custom range no longer applies as soon as you open it. It takes effect on Apply, and Cancel or Escape leaves the range you were looking at alone.\nThe OAL file viewer no longer strands you on an expired session, and switching files quickly cannot leave one file\u0026rsquo;s contents under another\u0026rsquo;s name.\ncli:hash completes when you press Enter. Typing a password interactively used to hang, because the command waited for end-of-input; the argument and piped forms were unaffected.\n","excerpt":"\u003ch1 id=\"100\"\u003e1.0.0\u003c/h1\u003e\n\u003cp\u003eThe first release of Horizon UI — the next-generation web console for Apache SkyWalking. A …\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/changelog/1.0.0/","title":"1.0.0"},{"body":"1.0.0 The first release of Horizon UI — the next-generation web console for Apache SkyWalking. A dark, dense, information-first interface over the same OAP query protocol and MQE the previous console used, with layer-driven dashboards you configure rather than code, an AI assistant that reads your live data, and an MCP endpoint so the agent you already use can read it too.\nAI assistant Features Ask about your system in plain language and get answers built from real dashboard widgets, not just text. A launcher on the right edge opens a chat: describe what you want to know and the assistant reads live data, then streams back an ordered narrative with inline charts, top-N lists and tables drawn by the same components the dashboards use. Open it as a side drawer, expand it to a full page, or put it in its own tab. It is read-only and inherits your permissions. It can list services, read active alarms, browse each layer\u0026rsquo;s metric catalog, drill a service down to its instances and endpoints, and chart any of it — never seeing more than you can, and never changing configuration, rules or dashboards. It embeds the real product views, scoped to the service you asked about. Ask for topology, traces, logs, browser errors, deployment, API dependencies, an instance map or a cross-layer hierarchy and the actual view mounts inside the chat, interactions intact — click a trace and its span waterfall opens. Both native SkyWalking and Zipkin tracing are covered. Everything it shows is a snapshot, and says so. Each block carries a replay badge and the time it was captured, and re-renders identically when you reopen the conversation — offline, with its edge sparklines and its detail views — rather than quietly re-querying and showing today\u0026rsquo;s data under yesterday\u0026rsquo;s question. It can read Kubernetes pod logs in the chat, as a result rather than a console. When a filter was applied it says so, so an empty result reads as \u0026ldquo;nothing matched\u0026rdquo; rather than a silent pod. It can propose profiling, and only you start it. When metrics and traces cannot localise a cause it presents a decision card explaining what it found and what profiling would reveal; nothing runs until you approve it, and only if you hold the permission. It picks the flavour that fits the target and renders the result — a flame graph, a profiled trace\u0026rsquo;s waterfall beside its flame, or a network conversation graph — inline once collected. Guided root-cause analysis. Ask what the root cause is and it follows built-in investigation playbooks — a master method plus latency, error-rate, saturation, middleware, Kubernetes-workload and service-mesh specialisations — including following a service down into the infrastructure layer behind it, where memory, disk and connection causes live. It answers in each layer\u0026rsquo;s own vocabulary, calling a Kubernetes instance a Pod and a mesh instance a Sidecar, and reads your configured warning thresholds rather than guessing what \u0026ldquo;healthy\u0026rdquo; means. An outage is reported as an outage. When Horizon cannot reach the backend, it says so and stops, instead of reporting every layer as having no metrics — which reads as \u0026ldquo;your services aren\u0026rsquo;t reporting\u0026rdquo; and sends you looking for a problem in your own system. Your question stays in view while the answer streams, pinned to the top of the chat, with a timestamp for when you asked and when the answer finished. Conversations are kept per user in your browser, with a usage meter, a save toggle and a clear-all. History is stored unencrypted, and the page says so — turn it off on shared workstations. Two tabs cannot overwrite each other\u0026rsquo;s chats. Bring your own model. Off by default, vendor-neutral, and configured by an administrator: any OpenAI-compatible endpoint — hosted, local or a gateway — or Amazon Bedrock. The assistant\u0026rsquo;s instructions and the starter prompts shown in an empty chat both ship with defaults you can replace entirely. Until it is pointed at a model the panel opens read-only with a short note. Agent access over MCP Features The agent you already use can read Horizon. Point Claude Code, Codex, Claude Desktop or any Model Context Protocol client at Horizon and it gets the same tools the AI assistant uses — the metric catalog, figures, topology, traces, logs, Kubernetes, profiling proposals and the root-cause playbooks. The model stays on the caller\u0026rsquo;s side, so no provider and no API key are configured here. It is not a new exposure. The endpoint needs the same login as every other route, a permission gates the connection, and each tool re-checks the permission its own screen needs. An agent sees exactly what the operator it authenticated as sees. An agent can log in through your browser instead of being handed a token. It opens Horizon\u0026rsquo;s own login page — backed by whatever your organisation configured — you approve once on a consent screen, and it keeps its token from there. The screen shows the permissions the grant would really carry, filtered by what you actually hold, so it never promises access you cannot delegate. Off by default. A client that can draw gets the real widgets, not a picture of them: the same charts, topology graphs and trace lists the console draws, fed the captured snapshot. A terminal client reads the data itself and presents it in its own way. Two deployments look like two deployments. A Horizon can name itself, and that name reaches the agent — so an operator watching production and staging is never told which is which by guesswork. Nothing served over MCP writes anything, and every tool declares itself read-only, so a host stops asking you to approve each step of a single investigation. Sign-in and access control Features Sign in with your identity provider. Google, Okta, Entra, Keycloak or anything else speaking OpenID Connect — each configured provider becomes a button on the login page. Providers that issue only an access token are supported too. It is additive by design: password login keeps working alongside it, so a misconfigured provider never locks you out during an incident. The login page fits what you configured. With password login present, providers fold into one picker; where single sign-on is the only way in, up to four get their own button and the rest fold into a picker. A deployment with no password backend hides the username and password boxes entirely, rather than showing a form that cannot succeed. An identity provider says who you are; it does not decide what you may do here. New sign-ins are viewers unless you say otherwise, you decide which domains may sign in at all, and per-address and per-domain overrides raise individuals. Horizon calls you what your directory calls you, showing your display name rather than a raw address or an account id. The verified address stays the identity behind every permission check, one hover away. An account page, reached by clicking your own name. Who you are, how you proved it — local account, directory, single sign-on with the provider that vouched for you, break-glass, or a token — and which roles you hold and what they grant, so \u0026ldquo;why can I not see this page\u0026rdquo; has an answer that does not need an administrator. API tokens for callers with no browser. Scripts, CI jobs and agents authenticate on every route under exactly the permissions that route requires. A token names a user and can never carry more than that user currently holds; removing the user revokes it. What one session read never reaches the next person to sign in on that browser. Signing out, signing in, or having a session end mid-use discards everything cached, so a shared workstation cannot serve the previous operator\u0026rsquo;s services, alarms, traces or dashboards to the next one. Shortening the session timeout now shortens the sessions you already have, not just new ones — which is what you want right after an incident. The roles board tells the truth about what each role sees. Every navigation entry is listed, permissions that gate nothing are marked as reserved, and an entry a role cannot open is not offered to it. Fixes Sign-in accepts up to 64 characters for the username and 64 for the password, and the form stops at the same limit rather than letting a longer value reach the server and come back as a generic failure. A directory that issues credentials longer than this cannot be used to sign in — see Local backend.\nThe sign-in card no longer runs off the edge of a phone-width window — it was cut off below roughly 410px.\nSource-map upload and removal are disabled, with the reason on hover, for operators who lack source-map:write — rather than looking available and then failing. Removing a map now asks first: it un-symbolicates stacks for everyone reading that layer.\nLogin audit Features A durable record of who signed in, when, and from where. Optional and off by default, backed by a shared database rather than a file. An hourly summary stacked by how people signed in, then filters, then the list — statistics first, because the first question is whether anything unusual is happening and the second is which. It records only what a valid credential produced. Successful sign-ins, plus the two refusals that happen after authentication already succeeded. A wrong password or an unknown user stays in the application log, because those are what an anonymous caller can produce at will. Nothing that could resume a session is ever recorded. Signing in never waits for the database, and cannot be blocked by it. Records are written in the background; an unreachable database is invisible to the person signing in, and the page says it cannot be reached rather than showing an empty table. Token traffic is counted on its own tab, at its own grain. A sign-in is a person arriving; token traffic is a machine at work, and stacking them let a busy script outweigh every human sign-in beside it. One row per token per hour, grouped by hour, with each hour\u0026rsquo;s totals and its busiest credentials — and a line that always says whether you are seeing all of them or the top ten. Reading it needs its own permission that a wildcard does not grant, because the log holds verified email addresses and client addresses. There is no write and no delete. Fixes A failing statistics write no longer reports the audit store as healthy. Sign-in records, token counts and hourly statistics are written on separate schedules; one of them succeeding used to clear another\u0026rsquo;s failure, and the periodic reachability check cleared a statistics failure it does not actually test. Each is now tracked on its own, so the store reads unhealthy for as long as anything is failing to write. An audit write that fails is no longer retried or held. Sign-in batches and hourly statistics are dropped when the database refuses them, so memory does not grow for the length of an outage and nothing is replayed afterwards — the sign-ins from that window are counted as unrecorded rather than reconstructed. Batching itself is unchanged: writes are still grouped for efficiency. Dashboards Features Layer dashboards are configuration, not code. Every layer\u0026rsquo;s screens are defined by a template you edit in the console — widgets, scopes, service-list columns, thresholds and labels — and published to your backend. Forty-six bundled dashboards ship ready to use, catalogued the way the sidebar groups them. A layer\u0026rsquo;s Service, Instance and Endpoint views can each carry more than one page. Each becomes its own row under the layer with its own URL and its own widgets, so a layer\u0026rsquo;s metrics need not share one screen. A page can name the entity it lists — \u0026ldquo;Brokers\u0026rdquo; rather than \u0026ldquo;Instances\u0026rdquo; — and can narrow which services or instances it is about. A new tab widget packs related views into one slot. A grid tile can hold any number of named tabs, each its own small dashboard, edited right where it sits. Only the active tab is queried, so an unopened tab costs nothing. The dashboard editor works beside the canvas. Picking a widget kind is a menu with descriptions rather than always dropping in a card you retype; the editor pins next to the board and opens complete, wherever on the board you clicked; adding a widget scrolls it into view. Rows under a layer can be put in your own order, dragged in a live preview of the real menu, with a reset to the built-in order. Publishing refuses a template that would break the layer, naming the field at fault and writing nothing — rather than storing it and emptying that layer\u0026rsquo;s screen for everyone. Work in progress still publishes: an empty expression or a half-filled section is a normal state of an unfinished draft. Cards can render values as coloured status chips rather than bare numbers — so the Kubernetes node status reads as Ready in green and its pressure conditions in amber or red, instead of a raw 1. Click a latency or error point on a chart to open the matching traces, pre-filtered to that service and centred on the bucket you clicked, opening slowest-first or error-only depending on the metric. Dashboard authors turn it on per widget. Compare several entities on one dashboard, with one-click exit from comparison. Overview dashboards roll up a whole layer, with per-widget control over how that aggregation is done and how the top services are ranked. Traces, logs and events Features A trace explorer with a duration-distribution scatter, a time-positioned waterfall and a span detail modal — and Zipkin traces render with the same experience as native ones, including plain-language hints for Zipkin\u0026rsquo;s annotation codes. One shareable link opens either kind. Logs and browser errors query on demand. Conditions stage until you press Run query, so a fresh tab prompts you rather than firing a broad query, and switching service resets rather than leaving the previous service\u0026rsquo;s rows under the new name. Stored logs can be searched by their content where the backend supports it — the field appears only on a backend that can actually answer it, rather than silently ignoring what you typed. Clicking a log row opens a full payload popout with format-aware pretty-printing, the tag table, and a link to the trace. Cross-layer inspection for raw logs, browser errors and Kubernetes pod logs. Browser errors carry source-map upload and de-obfuscation, resolving a minified stack back to the original frames with a source snippet. Pod logs tail a container on demand and are never stored. A per-service events popout on every layer\u0026rsquo;s service banner — agent restarts, Kubernetes events and other lifecycle records — laid out as one row per instance on a time axis, with a search box for services running hundreds of them. Result lists say when they were capped, and offer a next page only when there is one with rows on it. A pager reports the page and what is on it, rather than a total the backend does not provide. Tag fields autocomplete on theme, suggesting keys and then per-key values in a dense dropdown instead of the browser\u0026rsquo;s native popup. Fixes A custom time range that cannot be read is now refused, with the reason under the control — Traces, Zipkin traces, Logs, Browser errors and both Inspect pages. A reversed, half-filled or over-wide range used to be swapped silently for a default window, so the results answered a question nobody asked. Ranges longer than six hours also carry a note that they can be slow on a large deployment; that one is advice, and the query still runs. A request made directly against the API rather than through a page is trimmed to the most recent week instead of being refused, so it still answers with the part that matters.\nSwitching service no longer leaves the previous service\u0026rsquo;s endpoints, instances or profiling segments on screen. The dependent lists clear immediately and say Reading… while the new ones load, and a slow reply for a selection you have already moved off is discarded instead of overwriting the current one. This affects the Inspect pages\u0026rsquo; Service/Instance/Endpoint pickers, all five profiling tabs\u0026rsquo; task and segment lists, the network-profiling process graph, and the Zipkin span/remote autocomplete.\nProfiling Features Five kinds of profiling in one place — trace sampling, async-profiler for JVM services, pprof for Go services, eBPF on/off-CPU, and network profiling — each with a task list, a create dialog that tells you upfront what it needs, and a flame graph or conversation graph for the result. Continuous profiling has a home. Arm a policy once and the task starts itself when a process crosses a threshold, with nobody present — which is how you catch the problem that only appears at 3 a.m. Each target lists the instances and processes actually being evaluated and how often each has fired: the difference between a policy that is stored and one that is working. A task a policy started is visible beside the ones you started by hand, newest first, instead of appearing nowhere. A profiling request that cannot be honoured is refused with the reason, rather than quietly repaired into something more expensive or trimmed to a fraction of the fleet you asked for. Kubernetes services gain network profiling — pick a pod and capture process-to-process conversations as a topology, the same capability the mesh layer offers. Look and feel Features Dark, dense and information-first. Horizon is built for an operator watching a system, not for a marketing page: tight tables, small type, and as much signal per screen as stays legible. Escape closes any dismissible panel — modals, row popouts and the topology dropdowns alike. Searchable, on-theme dropdowns everywhere a picker lists more than a handful of entries, replacing the browser\u0026rsquo;s native controls. Denser Kubernetes tables, more rows without scrolling. The live debugger reads cleanly on tall and wide captures — the frozen first column stays pinned as you scroll sideways, clicking a source line flashes the whole matching step, and long captures scroll as one page instead of trapping the result in a fixed-height box. A step that dropped a record says why, in the backend\u0026rsquo;s own words, instead of leaving you to reconstruct the cause from the payload. Serving Horizon under a path prefix is a first-class option, for a reverse proxy that strips it before forwarding. Languages Features The whole console speaks eight languages — English plus German, Spanish, French, Japanese, Korean, Portuguese and Simplified Chinese. Product, protocol and metric names stay in their original form, because those are what operators read across the docs, the source and every other SkyWalking surface. Dashboard text is translated in the console, per language, on a page that shows exactly what the site renders rather than the shipped defaults — with staged drafts, a diff before you publish, and a reset to bundled. A translation belongs to its widget, not to the widget\u0026rsquo;s position, so rearranging a dashboard leaves every other widget\u0026rsquo;s translation where it was — a deleted widget takes its translation with it, and a new one starts out English until you fill it in. Text left behind by a template edit is found and cleaned up deliberately, rather than being discarded by the next unrelated save. Operating Horizon Features The container image runs on environment variables alone — no mounted configuration file, no repackaging. The shipped configuration file doubles as the complete, self-documenting reference for every variable. Run against a backend whose template store you cannot write, rendering every dashboard from the bundled templates and never calling the template API. The configuration surface becomes honestly read-only, while metrics, traces, logs and topology work exactly as before. This is the supported way to run against an OAP release that has no template management endpoint. Cluster Status reports what is actually reachable, testing the real path each feature calls rather than inferring health from configuration being present — so a module that is loaded but broken reads as unreachable instead of a misleading green. Configuration hot-reloads, and a rejected reload says so out loud, naming the field at fault and continuing to serve the last valid configuration rather than silently ignoring the edit or falling back to defaults. Query fan-out is tunable per deployment — batch sizes, concurrency and protective caps — so a beefy backend can be pushed harder and a modest one protected. Defaults match the built-in behaviour, so the whole thing is optional. Responses are not cached by the browser, so metrics, traces, logs and configuration are not left behind on a shared workstation. The console\u0026rsquo;s own files stay cacheable, so nothing gets slower. A strict content policy ships by default, permitting scripts only from Horizon\u0026rsquo;s own origin, forbidding inline script, and refusing to be framed. It needs no configuration. Outbound documentation links are restricted to hosts you trust, and anything that is not a real web address is refused outright — both when a template is published and when one is read back. Warnings reach standard output by default. Break-glass logins, directory failures and rejected configuration reloads are the things you want to see without having raised the log level first. A duplicated dashboard record is reported, never resolved behind your back. A dashboard whose definition is ambiguous is hidden rather than rendered from whichever copy happened to win, and opening it by URL explains why and points at where to fix it. Deciding which copy survives is a deliberate cleanup, not something a restart does for you. Fixes The DSL editor asks before deleting a rule that has no bundled version, and warns before you navigate away or reload with unsaved YAML. Deleting such a rule removes the only copy.\nThe alarms and events Custom range no longer applies as soon as you open it. It takes effect on Apply, and Cancel or Escape leaves the range you were looking at alone.\nThe OAL file viewer no longer strands you on an expired session, and switching files quickly cannot leave one file\u0026rsquo;s contents under another\u0026rsquo;s name.\ncli:hash completes when you press Enter. Typing a password interactively used to hang, because the command waited for end-of-input; the argument and piped forms were unaffected.\n","excerpt":"\u003ch1 id=\"100\"\u003e1.0.0\u003c/h1\u003e\n\u003cp\u003eThe first release of Horizon UI — the next-generation web console for Apache SkyWalking. A …\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/v1.0.0/changelog/1.0.0/","title":"1.0.0"},{"body":"1.1.0 Breaking changes and migration Breaking: Each of the six Dashboard setup pages now has its own read/write permission pair, and three permissions are retired. dashboard:read / dashboard:write become layer-template:read / layer-template:write; overview:write splits into overview-template:write, translation:write, alarm-setup:write, infra-3d-setup:write and setup:write; overview:read keeps only its original meaning, the rendered overview dashboards. The built-in roles are updated for you, and a role still naming a retired permission keeps working — the old names, and the dashboard:* / overview:* wildcards, are accepted for this release and expand to what they used to gate. Rename them in your horizon.yaml before the next one; Horizon now logs a warning at startup naming any permission it does not recognise, which it previously ignored in silence. Two consequences worth checking: viewer and maintainer no longer read stored template configuration (they keep every dashboard they could see), and a role granted overview:read alone no longer reaches the template-administration pages.\nBreaking: Turning the Cold pill on no longer re-reads the page. It changes what the NEXT read asks for — the following auto-refresh round, or whatever you do first, typically choosing the time range the cold data lives in. Flipping it used to start a round of its own, which on a cold tier routinely ran past the sixty-second cap; the sweep that followed then re-queued exactly the queries the cap had just cancelled, so the cap was at its most useless during the outage it exists for. And because a cold read REPLACES the hot one rather than widening it, an operator who had not yet moved the time range watched the whole page empty on the click. Anything already in flight is cancelled, so no batch is left half hot and half cold — the stage is read when a request goes out, and a call still queued behind the concurrency limiter would otherwise leave under the new stage while belonging to the old one. Nothing to migrate: if you want the page re-read immediately after flipping, press Refresh.\nAI agent conversations Features Horizon can now read the AI agent conversations an OAP stores. OAP 11.1.0 keeps the conversations that the SkyWalking AI Sessionizer pushes under the AI_AGENT layer, one row per conversation and one document per conversation. This release wires Horizon to both reads, behind a new permission, ai-conversation:read, that the built-in viewer, maintainer and operator roles carry. It is its own permission rather than part of ai:read, because that one is the AI assistant, where Horizon sends text to a model provider; this one reads stored transcripts of other agents, and a deployment may allow either without the other. Two settings under performance.aiConversation tune the reads: listLimit, how many of the newest rounds the list folds into rows (default the OAP ceiling, 10,000, because a smaller number lets one long conversation hide the short ones), and viewTimeoutMs, how long to wait for a conversation document, whose first byte arrives only once OAP has folded the whole chain (default 120 s, matching OAP). See Configuration File and Roles and Permissions.\nA new bundled layer, AI Agents, with a Conversations tab. When OAP reports the AI_AGENT layer, the sidebar gains an AI Agents entry whose one tab lists the conversations of one agent runtime: pick the runtime in the layer header and a time range (a day to 90 days), optionally one sender, click Run query, and each row shows the conversation\u0026rsquo;s title, which Sessionizer sent it, its talks, steps, streams, segments and unresolved references, its span and its last activity, newest first. A title text and a conversation id are query conditions too, applied by OAP. The line above the table states the round budget the list was folded from, because OAP builds the list from the newest rounds rather than from conversations and cannot say whether the budget cut anything. Layer templates gain the aiConversations component flag behind this tab. See AI Agent Conversations and AI Agents.\nA conversation opens in its own tab, as one shareable page. Clicking a row on the Conversations tab opens /ai-conversation/\u0026lt;id\u0026gt; in a new browser tab: the conversation\u0026rsquo;s transcript, with the agent\u0026rsquo;s work folded under each input; a flow timeline of one execution stream with lanes for input, responses, context, model calls, tools, agent activity and nested child streams, relation lines for the selected step, and a way to dive into a child agent and back; and an inspector with the selected step\u0026rsquo;s details, relations and landed evidence. The address carries the conversation, its runtime and sender, and the reader\u0026rsquo;s position (talk, step, stream), updated in place as you move, so it can be shared and lands on the same step. The page reads the whole document from OAP and shows the wait in phases — OAP assembling the conversation with the seconds counting, the bytes arriving as n of m MB with a percentage, rate and time left, then the parse and the draw — states the document\u0026rsquo;s integrity (verified, incomplete, mismatch) and the Sessionizer\u0026rsquo;s recorded problems, and follows your Horizon theme, the kinds of step in their own colours per theme. The renderer behind it is a framework-free module the SkyWalking AI Sessionizer\u0026rsquo;s own viewer embeds too, built from a pinned Horizon commit, so both show a conversation identically. See AI Agent Conversations.\nSign-in and access control Features A custom role can now be added without restating the built-in ones. Naming rbac.roles has always replaced the block outright, so adding one role meant copying viewer, maintainer, operator and admin into your config and keeping those copies current forever. Set builtinRoles: keep and the built-ins become the base instead: a role name you list overrides that one role, a new name is added, and everything you did not mention stays as it shipped — landingByRole merges the same way. The default is unchanged (replace), because a deployment that trimmed its block to remove a role must not have it handed back by an upgrade, least of all admin. Under keep a built-in can no longer be dropped by omission; grant it nothing (admin: []) instead. The effective role names are logged at startup so you can confirm what a merge produced. See Access Control Configuration.\nThe Dashboard setup pages open read-only when you may look but not publish. Each page — Overview templates, Layer dashboards, Translations, Alert page, 3D Infra Map, Global defaults — is reached with its own read permission and published with its own write permission. Granting the read half alone shows the page in the sidebar and opens it with the whole configuration visible, every editing control disabled, and a banner naming the permission publishing needs; granting the write half turns the page back on. Previously three of these pages could not be opened at all without edit rights, and the other three rendered every button enabled and failed at the click. Refresh from remote is now treated as the read it is, so it works for a read-only viewer instead of being refused. A read-only page shows what OAP holds (or the shipped default) rather than an unpublished draft saved in your browser: the draft is kept and the page marks it local, but it is not displayed while you have no way to discard or publish it.\nThe login audit can now be stored in BanyanDB instead of PostgreSQL. Set audit.provider: banyandb and point audit.banyandb.address at a liaison\u0026rsquo;s gRPC port — including the BanyanDB SkyWalking already stores its telemetry in, since Horizon keeps the records in groups of its own (horizon_audit and horizon_audit_metrics, prefixed by namespace where two deployments share one server). The page is the same on either backend: the same rows, the same three filters, the same hourly summary and token-usage tab. Retention is the groups\u0026rsquo; own lifetime rather than a job Horizon runs, so there is no sweep interval to tune; on a cluster, set shards and replicas, because the single-node defaults would keep the whole audit log on one data node. As with PostgreSQL, an address that is not loopback must use TLS unless allowCleartext says otherwise. See Login audit.\nFixes A malformed permission no longer grants more than it says. A grant is at most three segments, but a longer one used to be silently truncated to its first three — so rule:*:typo was read as rule:* and handed over every rule permission, and rule:write:structural:extra matched rule:write:structural. Anything outside the documented forms now matches nothing and is named in the startup warning alongside other unrecognised grants.\nThe BanyanDB audit configuration is only what a deployment actually differs on — where the server is, how to authenticate to it, a namespace prefix, and how long to keep records. The groups\u0026rsquo; layout is bundled rather than exposed: one shard, which a sign-in rate never outgrows. That is also the safe default, because BanyanDB fixes a group\u0026rsquo;s shard count at creation and does not move data when it changes — a shard count edited after records exist would route new ones away from the old ones rather than resharding. Horizon now reports such a difference instead of applying it. Replication is left to you: Horizon sets no replica count, so a deployment that wants the audit log replicated can create the two groups with the count it wants and Horizon will keep them that way.\nThe login audit list pages the way every other SkyWalking list does. It asks for a page number rather than carrying a position between requests, which is the arrangement OAP uses for traces, logs, alarms and events alike. The page still shows 50 rows at a time and still reports only whether more exist; paging is bounded at 500 pages deep, and a request past that is refused rather than quietly answered with something else.\nThe hourly summary and token-usage totals now refuse a window they cannot read completely. Both are sums, so a row that did not fit did not look missing — it looked like a smaller number. A window holding more rows than one read may return now reports that it cannot be shown, instead of drawing a total that is quietly low.\nA database outage no longer costs the hourly sign-in counts it spanned. Each Horizon process now records a running total for the hour rather than a count of what happened since its last write, so the first write after the database comes back restores the figures for the hour in progress — and a write that is retried after an uncertain outcome leaves the same number instead of counting twice. The sign-in list is unchanged: rows from an outage are still not recorded.\nRefreshing Features Refreshing is one coordinated round. A page used to refresh in pieces, on clocks that drifted apart: the header and the service roster followed the topbar timer, while a dashboard\u0026rsquo;s widgets and the alarms card each polled on intervals of their own. A refresh is now a single round — header, roster, every widget, the alarms card and the maps — all asking about one time window, and the round is not finished until the last of them lands. They still appear as they arrive, so a large comparison fills in over a second or two; what changed is that they are all answering the same question. The countdown measures the gap between rounds rather than between starts: while the readings are out it reads Refreshing, and it begins again when the last of them lands, so time spent loading is never charged against the interval. On a slow backend the interval genuinely stretches and two rounds can no longer overlap. A round that has started always finishes, even if you switch refreshing off or move to a page that pauses it — the next one simply does not begin. One that runs longer than a minute is given up on, so a single wedged screen cannot stop everything else refreshing.\nAuto-refresh has its own on/off, separate from the interval. Turning it off and back on returns to the interval you last chose rather than forgetting it, and off means off everywhere: passing through a page that pauses refreshing and coming back no longer triggers a refresh you had switched off. Switching it on while a page is paused or the tab is in the background saves the setting without refreshing then and there, and the interval menu stays usable on pages that pause refreshing — how often to refresh outlives the page you happen to be on. Pages and overlays pause refreshing independently of each other, so opening the Smartscape hierarchy overlay freezes the background while you pan through it and changing the time range no longer unfreezes it underneath you.\nThe refresh control says whether anything is actually loading. A download arrow beside the countdown appears while requests are in flight and goes when they land — the icon used to spin whenever auto-refresh was merely enabled, which told you nothing about whether data was arriving. While a round is out the refresh buttons are disabled rather than accepting a click that could only ask for what is already being fetched.\nA refresh that fails is recorded where you can go and read it. Failures from the timer collect beside the refresh control, newest five first, with a count of the ones you have not seen — nobody asked for that round, so it waits rather than interrupting. Each entry names the screen, what it was trying to do, the request and the server\u0026rsquo;s answer; secrets in the URL or the response are removed before it is shown. Failures from something you just did — expanding a node, for instance — appear immediately instead, as a message that pauses while you read it and can be opened for the same detail.\nFixes Overview widget grids and layer dashboards emptied on every refresh. Both treated a moving time window as a new question, so each tick blanked the grid and replaced it with a loading line — over widgets that were usually about to be filled with the same numbers. They keep the previous values while the next reading is out and replace them in one go when it lands.\nThe Alarms card on a dashboard refreshed on a clock of its own. It polled every minute regardless of the page\u0026rsquo;s own cadence, so a dashboard set to refresh every fifteen seconds showed an alarm list up to a minute older than the metrics beside it. It moves with the rest of the page now.\nA read you walked away from went on costing OAP work. Navigating away mid-load, or a refresh that gave up, stopped the browser waiting but not the query behind it — so an abandoned page went on costing OAP the whole fan-out, multiplied by the node count on a cluster. Those reads are now cancelled all the way through. Anything that CHANGES something — creating a profiling task, pushing a template — still runs to completion, so a closed tab cannot leave it half applied.\nClicking Refresh against a dead backend appeared to do nothing. The failure was filed away and the screen said nothing, though you had just pressed the button and were watching. A refresh you asked for now answers on screen when part of it fails — once, however many widgets failed — and points at the list beside the control, where the detail is.\nOne outage was listed several times. Two screens sharing the same reading each recorded it, so the failure list showed one problem as several. Each is recorded once now, and a comparison entity that fails on its own is recorded at all, which it was not before.\nA failed read was drawn as a page of zeroes. When a layer read could not reach OAP the answer came back empty, and an overview\u0026rsquo;s KPIs rendered that as 0 and its service counts as none — indistinguishable on screen from a system that genuinely had nothing running. Overviews and the layer service list now keep the last values they read and report the failure instead. Layer dashboards keep their widgets through the same failure rather than emptying them.\nQuery cold stage could leave hot and cold answers on screen together. Flipping it re-read each screen on its own, so the page showed both stages while it settled, and a screen that had not caught up yet could file its answer under the wrong one. The whole page is re-read as one round now.\nA refresh that gave up left its heaviest requests running. The sixty-second cap cancelled the query but not the request behind it for the layer landing, the dashboard batches and the overview — the browser stopped waiting while OAP finished a fan-out nobody would read, multiplied by the node count on a cluster. Those now stop with the round.\nTwo reads racing could leave every request an hour wrong. The server timezone and the capability probe were each fetched per request, so a round\u0026rsquo;s dozen reads probed in parallel; behind a load balancer a fast success could be overwritten by a slower timeout falling back to UTC, and that answer then served every request for the next minute. Each is now read once per expiry and shared.\nA brief storage hiccup emptied the sidebar for a minute. A failed service-catalog read replaced the known-good roster with an empty one and cached it for the full minute, so service counts fell to zero and group navigation collapsed long after OAP was healthy. The last good roster is kept and marked stale instead, and a failed read is retried within seconds — for as long as the outage lasts, rather than surviving only the first failure.\nA landing metric batch that timed out silently reordered the layer. Its services were left without a value, sorted to the bottom as though idle, and could fall out of the top-N — choosing a different busiest service and a different default — while the response still reported success. Services whose metric could not be READ are now ranked above those that genuinely reported nothing, and the layer header says so on screen — naming how many batches failed — rather than leaving the blanks to be read as zeroes. An overview whose widgets aggregate across services says the same thing, because there a lost batch leaves a total quietly low rather than visibly absent. It keeps saying it for as long as the incomplete reading is on screen, not only on the refresh that hit the timeout — and an incomplete reading is re-read once, behind the values already shown, so a momentary backend failure costs a few seconds of a partial header rather than the rest of the hour.\nTopology, deployment and dependency maps Features Changing what you are looking at says what it is loading. Picking a different service, endpoint, depth or time range names its target — Loading topology for \u0026ldquo;checkout\u0026rdquo;… — instead of leaving the previous answer under the new heading while the next one is fetched.\nFixes The 3D Infra Map follows the theme: its canvas, tier slabs, rims, cube edges, panels and cards took their colours from a fixed dark palette, so a light theme such as Daybreak showed dark panels with unreadable text. They now come from the theme tokens and change with the theme.\nThe 3D Infra Map opens framed on the tiers that hold services. A deployment with one populated tier had its slab at the top edge of the view, because the default pose aimed between empty planes; the default and Reset views now frame what is there.\nAn expansion on the API dependency graph could land in a graph you had left. Nothing stopped you switching endpoints while one was loading, and the branch then arrived in whatever graph had replaced it. Expanding now shows its pending state and holds the endpoint picker and the other expand handles until it lands. An expansion started while an admin preview is open also resolves against the draft being previewed, rather than against the published template the rest of the graph is not showing.\nA map that failed its FIRST read showed a loading line for ever. With nothing cached to fall back on there was no way to tell a read still in progress from one that had already failed, so the graph sat on \u0026ldquo;Reading data…\u0026rdquo; indefinitely. It now says the read failed and offers Retry.\nThe maps blanked on every refresh. The Topology, Deployment, API dependency and instance-relationship graphs dropped to a loading line and came back on each tick, taking your zoom, pan and any nodes you had dragged with them. They redraw in place now, and are re-framed only when the question changes — a different layer, service, endpoint, focus, depth or time range. A service appearing or disappearing no longer re-frames the canvas or discards placements.\nA failed reading erased the map you were looking at. When a round could not reach OAP the graph was replaced by an empty one, so an empty map could mean either that there was nothing to show or that the read had failed. The previous picture stays and the failure is reported beside it. A layer whose template an administrator has disabled is the deliberate exception — that is an answer rather than a failure, so the map is cleared and says so, instead of telling you to check a backend that is fine.\nA failed expansion on the API dependency graph was reported as \u0026ldquo;no further dependencies\u0026rdquo;. A read that could not be completed marked the branch exhausted and faded its handle — a claim about your system made from a failure to read it. It now says the expansion failed and leaves the handle live, so the click can simply be repeated.\nThe API dependency columns re-sorted themselves on every refresh. Rows were ranked by the centre metric, which moves every cycle, so an endpoint you were watching had to be found again after each tick. Which endpoints get a row is still decided live — one that becomes busy enough still appears — but an endpoint already on screen keeps its place and an arrival takes the next free one.\nPods drifted on the Deployment map, and dragging could stop working. A placement was held as an offset from the packed position, so a refresh that added a pod carried yours along with the re-pack; and the map only re-armed its drag handles when a count changed, so a refresh that swapped one instance for another silently left them unbound. Placements are absolute now, and a node on the API dependency graph keeps its place the same way.\nDashboard templates Features Cluster Status shows what the template store has actually loaded. A new Dashboard templates pane reports how many layer templates, overviews, alert pages and translation overlays are being served from OAP, and when they were last read — and, when a read fails, the message it failed with, so \u0026ldquo;unreachable\u0026rdquo; is no longer the only thing you are told about a 404, a 401 and a timeout alike. Its badge separates a store that cannot be read while your dashboards keep rendering from one that has never been read at all.\nA template published elsewhere reaches an open browser on its own. Horizon re-reads the template store on a slow cycle of its own, so a dashboard pushed from another Horizon, from swctl, or from anything else writing the same OAP store appears within about a minute — without anyone reloading the page.\nEvery MQE field in the layer template editor can now be run where you type it. A run button beside each expression opens a panel that picks a service (and, where the metric needs one, an instance, endpoint or process), picks a time range with the same precision-tabbed picker the top of the page uses (custom ranges included), fires the expression against your OAP and shows exactly what came back — a graph of the series it returned, aligned on one time axis with a legend per label, the entity each ranked row belongs to, and OAP\u0026rsquo;s own error message when the expression is wrong. Until now the editor fired no queries at all, so the only way to find out whether an expression worked was to push the template, open the layer and look. Every field is covered: the service-list columns, the topology, deployment, process and API-dependency metrics, the dashboard widget expressions and their visibleWhen gates. Relation metrics ask for both a source and a destination, since that is what they measure. A blank service-list column is runnable too — it reports the catalog default that will run in its place.\nFixes A template store that could not be read emptied every dashboard, overview and map. They are rendered from the last successful read now, so a brief outage of OAP\u0026rsquo;s admin port leaves the console up, with the banner saying how stale it is. Horizon still never substitutes the templates bundled in the release — showing shipped defaults in place of your own configuration would misrepresent what is on screen — so a Horizon that has never read the store still blocks those pages rather than inventing content for them.\nThe unreachable banner never cleared once the store came back. Nothing re-read the store\u0026rsquo;s status after the page had loaded, so the warning stayed up — and the pages behind it stayed blocked — however healthy OAP had become. Both recover on their own now.\nOperating Horizon Fixes Data retention now reads the same on every storage backend, and says which setting governs each figure. The page used to mirror OAP\u0026rsquo;s wire shape — a Records / Metrics split, a Minute / Hour / Day trio, and a class called Normal — which on ElasticSearch, MySQL, PostgreSQL, H2 and TiDB meant nine numbers that were really two, presented as nine knobs you could turn apart. Those backends now show the five things retention is actually decided for — Metadata, Metrics, Logs, Traces and Others — each naming core.recordDataTTL or core.metricsDataTTL beneath it, with a note saying the two settings are all there is. BanyanDB shows the same five in the same order, expanding Metrics into its minute / hour / day groups and adding Zipkin traces and Browser error logs, because there each one is a separate bydb.yml group that genuinely can be tuned apart.\nThe retention chart hid data classes whose retention happened to match. Classes sharing a figure were merged into a single All records (5) bar — which reads as a fact about the backend when it is only a fact about today\u0026rsquo;s configuration, and left an operator unable to see what they could change. Every class now has its own row. Others is the one row named for a group rather than a data type: BanyanDB keeps alarms, events, sampled traces, top-N and every profiling record together in records, so they share one retention and no name for a single one of them would be true.\nQuery cold stage Features Cold is asked for only where cold data is kept. Traces, logs and metrics are the classes a deployment is advised to age into cold storage, and they are now the only ones Horizon sends the flag for. Alarms, the instance and endpoint pickers, events and everything from profiling stay hot — they are small, and they are the first things you reach for during an incident, so a Cold toggle emptying them bought nothing. Those pages keep answering while Cold is on.\nFixes The cold-stage warning told you to pick a window that does not exist. On a deployment with no cold stage configured — the BanyanDB default — it still said \u0026ldquo;pick a window older than N days\u0026rdquo;, which cannot work, and following it moved the range out of hot+warm so the warning hid itself: a blank page with the one sentence explaining it gone. It now says plainly that no cold stage is configured and stays up until you turn Cold off. Where cold IS configured, the suggested window is the one that clears the deepest class rather than the shallowest — it used to name the records boundary (3 days by default) while every metric widget needed 7, so following it brought traces back and left the metrics exactly as empty. Both the warning and its advice are now decided by the classes Horizon actually reads under Cold — traces, logs and metrics. Alarms, events and profiling retention no longer count: they are usually kept for the shortest time of anything, and reading them here hid the warning for exactly the windows an operator turning Cold on is most likely to be looking at.\nDashboards Features The layer header\u0026rsquo;s KPIs describe one completed hour, and say which one. They used to be read over whatever window the time picker held, which meant reading the sort metric for every service in the layer on every refresh — hundreds of requests on a large layer, repeated every thirty seconds, for figures that only move by the hour. The header now reads one completed hour and holds it, and the hour it covers is named above the table (09:00–10:00) so the numbers are never older than they appear. Ten minutes are left for the backend to finish aggregating that hour, so the header moves on shortly after ten past. While the next hour is being read the previous one stays on screen with a star against each figure, instead of the table emptying. A deployment too new to have any completed hour shows the hour in progress instead — a finished hour is preferred to it whenever one is available. Whenever no hour can answer at all, the header reads the time picker\u0026rsquo;s window exactly as it always did: on a deployment young enough that the backend has written no hour-level figures yet (they are aggregated on a longer cycle than minute ones), and on the first visit to a large layer while the hour is still being read. Either way the page opens with real numbers rather than a table of dashes. Everything below the header — the trend lines, the dashboards, the maps — still follows the time picker as before.\nThe BanyanDB layer has a Trace Sampling page. BanyanDB 0.11 can run a chain of sampler plugins during trace merge and finalization, dropping whole traces after their fragments have landed — tail sampling inside the storage tier. The new page is an extension page under Cluster, with its own sidebar row, and it reads the meter_banyandb_trace_sampling_* families OAP collects from the plugin chain: the committed drop ratio and the trace outcomes behind it, per-plugin Decide rate and latency, chain batching, what the first-party samplers proposed by rule, every fail-open guard and drop-set capacity counter, finalization state, and the host\u0026rsquo;s bounds on a plugin\u0026rsquo;s own telemetry. It is cluster-wide with the storage group as a series rather than a per-group dashboard, so one page charts every group at once.\nFixes A compared service that failed to refresh was drawn as though it had. In Compare, each entity keeps its last good reading when its own read fails — deliberately, so one failure does not blank its siblings — but the retained series was then plotted across the CURRENT axis and ran to the right-hand edge beside the fresh ones. Ten-minute-old latency sat next to live latency with nothing to tell them apart. Each series is now placed at the buckets it was actually read for, so a stale one stops where its data does and the gap is visible.\nA layer whose dashboard an administrator removed no longer reports an outage. The page said the template was disabled and pointed at an admin screen to re-enable it — a control most operators cannot reach, on a page that does not exist under that name — while the refresh history recorded \u0026ldquo;OAP could not be reached\u0026rdquo; for what was an administrative decision on a healthy server. The page now simply says it is not available, and the layer leaves the sidebar on the next menu read.\nReactivating a layer brings back its own dashboard, not the shipped default. OAP keeps a disabled template\u0026rsquo;s configuration; Reactivate was re-pushing the bundled default over it, discarding every edit the layer carried. It now restores what was there, falling back to the bundled default only when there is nothing to restore.\n","excerpt":"\u003ch1 id=\"110\"\u003e1.1.0\u003c/h1\u003e\n\u003ch2 id=\"breaking-changes-and-migration\"\u003eBreaking changes and migration\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eBreaking:\u003c/strong\u003e Each of the six \u003cstrong\u003eDashboard setup\u003c/strong\u003e pages now has its own …\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/skywalking-horizon-ui/next/changelog/1.1.0/","title":"1.1.0"},{"body":"10.0.0 Project Support Java 21 runtime. Support oap-java21 image for Java 21 runtime. Upgrade OTEL collector version to 0.92.0 in all e2e tests. Switch CI macOS runner to m1. Upgrade PostgreSQL driver to 42.4.4 to fix CVE-2024-1597. Remove CLI(swctl) from the image. Remove CLI_VERSION variable from Makefile build. Add BanyanDB to docker-compose quickstart. Bump up Armeria, jackson, netty, jetcd and grpc to fix CVEs. Bump up BanyanDB Java Client to 0.6.0. OAP Server Add layer parameter to the global topology graphQL query. Add is_present function in MQE for check if the list metrics has a value or not. Remove unreasonable default configurations for gRPC thread executor. Remove gRPCThreadPoolQueueSize (SW_RECEIVER_GRPC_POOL_QUEUE_SIZE) configuration. Allow excluding ServiceEntries in some namespaces when looking up ServiceEntries as a final resolution method of service metadata. Set up the length of source and dest IDs in relation entities of service, instance, endpoint, and process to 250(was 200). Support build Service/Instance Hierarchy and query. Change the string field in Elasticsearch storage from keyword type to text type if it set more than 32766 length. [Break Change] Change the configuration field of ui_template and ui_menu in Elasticsearch storage from keyword type to text. Support Service Hierarchy auto matching, add auto matching layer relationships (upper -\u0026gt; lower) as following: MESH -\u0026gt; MESH_DP MESH -\u0026gt; K8S_SERVICE MESH_DP -\u0026gt; K8S_SERVICE GENERAL -\u0026gt; K8S_SERVICE Add namespace suffix for K8S_SERVICE_NAME_RULE/ISTIO_SERVICE_NAME_RULE and metadata-service-mapping.yaml as default. Allow using a dedicated port for ALS receiver. Fix log query by traceId in JDBCLogQueryDAO. Support handler eBPF access log protocol. Fix SumPerMinFunctionTest error function. Remove unnecessary annotations and functions from Meter Functions. Add max and min functions for MAL down sampling. Fix critical bug of uncontrolled memory cost of TopN statistics. Change topN group key from StorageId to entityId + timeBucket. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: MYSQL -\u0026gt; K8S_SERVICE POSTGRESQL -\u0026gt; K8S_SERVICE SO11Y_OAP -\u0026gt; K8S_SERVICE VIRTUAL_DATABASE -\u0026gt; MYSQL VIRTUAL_DATABASE -\u0026gt; POSTGRESQL Add Golang as a supported language for AMQP. Support available layers of service in the topology. Add count aggregation function for MAL Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: NGINX -\u0026gt; K8S_SERVICE APISIX -\u0026gt; K8S_SERVICE GENERAL -\u0026gt; APISIX Add Golang as a supported language for RocketMQ. Support Apache RocketMQ server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: ROCKETMQ -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; ROCKETMQ Fix ServiceInstance in query. Mock /api/v1/status/buildinfo for PromQL API. Fix table exists check in the JDBC Storage Plugin. Fix day-based table rolling time range strategy in JDBC storage. Add maxInboundMessageSize (SW_DCS_MAX_INBOUND_MESSAGE_SIZE) configuration to change the max inbound message size of DCS. Fix Service Layer when building Events in the EventHookCallback. Add Golang as a supported language for Pulsar. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: RABBITMQ -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; RABBITMQ Remove Column#function mechanism in the kernel. Make query readMetricValue always return the average value of the duration. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: KAFKA -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; KAFKA Support ClickHouse server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: CLICKHOUSE -\u0026gt; K8S_SERVICE VIRTUAL_DATABASE -\u0026gt; CLICKHOUSE Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: PULSAR -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; PULSAR Add Golang as a supported language for Kafka. Support displaying the port services listen to from OAP and UI during server start. Refactor data-generator to support generating metrics. Fix AvgHistogramPercentileFunction legacy name. [Break Change] Labeled Metrics support multiple labels. Storage: store all label names and values instead of only the values. MQE: Support querying by multiple labels(name and value) instead using _ as the anonymous label name. aggregate_labels function support aggregate by specific labels. relabels function require target label and rename label name and value. PromQL: Support querying by multiple labels(name and value) instead using lables as the anonymous label name. Remove general labels labels/relabels/label function. API /api/v1/labels and /api/v1/label/\u0026lt;label_name\u0026gt;/values support return matched metrics labels. OAL: Deprecate percentile function and introduce percentile2 function instead. Bump up Kafka to fix CVE. Fix NullPointerException in Istio ServiceEntry registry. Remove unnecessary componentIds as series ID in the ServiceRelationClientSideMetrics and ServiceRelationServerSideMetrics entities. Fix not throw error when part of expression not matched any expression node in the MQE and `PromQL. Remove kafka-fetcher/default/createTopicIfNotExist as the creation is automatically since #7326 (v8.7.0). Fix inaccuracy nginx service metrics. Fix/Change Windows metrics name(Swap -\u0026gt; Virtual Memory) memory_swap_free -\u0026gt; memory_virtual_memory_free memory_swap_total -\u0026gt; memory_virtual_memory_total memory_swap_percentage -\u0026gt; memory_virtual_memory_percentage Fix/Change UI init setting for Windows Swap -\u0026gt; Virtual Memory Fix Memory Swap Usage/Virtual Memory Usage display with UI init.(Linux/Windows) Fix inaccurate APISIX metrics. Fix inaccurate MongoDB Metrics. Support Apache ActiveMQ server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: ACTIVEMQ -\u0026gt; K8S_SERVICE Calculate Nginx service HTTP Latency by MQE. MQE query: make metadata not return null. MQE labeled metrics Binary Operation: return empty value if the labels not match rather than report error. Fix inaccurate Hierarchy of RabbitMQ Server monitoring metrics. Fix inaccurate MySQL/MariaDB, Redis, PostgreSQL metrics. Support DoubleValue,IntValue,BoolValue in OTEL metrics attributes. [Break Change] gGRPC metrics exporter unified the metric value type and support labeled metrics. Add component definition(ID=152) for c3p0(JDBC3 Connection and Statement Pooling). Fix MQE top_n global query. Fix inaccurate Pulsar and Bookkeeper metrics. MQE support sort_values and sort_label_values functions. UI Fix the mismatch between the unit and calculation of the \u0026ldquo;Network Bandwidth Usage\u0026rdquo; widget in Linux-Service Dashboard. Add theme change animation. Implement the Service and Instance hierarchy topology. Support Tabs in the widget visible when MQE expressions. Support search on Marketplace. Fix default route. Fix layout on the Log widget. Fix Trace associates with Log widget. Add isDefault to the dashboard configuration. Add expressions to dashboard configurations on the dashboard list page. Update Kubernetes related UI templates for adapt data from eBPF access log. Fix dashboard K8S-Service-Root metrics expression. Add dashboards for Service/Instance Hierarchy. Fix MQE in dashboards when using Card widget. Optimize tooltips style. Fix resizing window causes the trace graph to display incorrectly. Add the not found page(404). Enhance VNode logic and support multiple Trace IDs in span\u0026rsquo;s ref. Add the layers filed and associate layers dashboards for the service topology nodes. Fix Nginx-Instance metrics to instance level. Update tabs of the Kubernetes service page. Add Airflow menu i18n. Add Support for dragging in the trace panel. Add workflow icon. Metrics support multiple labels. Support the SINGLE_VALUE for table widgets. Remove the General metric mode and related logical code. Remove metrics for unreal nodes in the topology. Enhance the Trace widget for batch consuming spans. Clean the unused elements in the UI-templates. Documentation Update the release doc to remove the announcement as the tests are through e2e rather than manually. Update the release notification mail a little. Polish docs structure. Move customization docs separately from the introduction docs. Add webhook/gRPC hooks settings example for backend-alarm.md. Begin the process of SWIP - SkyWalking Improvement Proposal. Add SWIP-1 Create and detect Service Hierarchy Relationship. Add SWIP-2 Collecting and Gathering Kubernetes Monitoring Data. Update the Overview docs to add the Service Hierarchy Relationship section. Fix incorrect words for backend-bookkeeper-monitoring.md and backend-pulsar-monitoring.md Document a new way to load balance OAP. Add SWIP-3 Support RocketMQ monitoring. Add OpenTelemetry SkyWalking Exporter deprecated warning doc. Update i18n for rocketmq monitoring. Fix: remove click event after unmounted. Fix: end loading without query results. Update nanoid version to 3.3.7. Update postcss version to 8.4.33. Fix kafka topic name in exporter doc. Fix query-protocol.md, make it consistent with the GraphQL query protocol. Add SWIP-5 Support ClickHouse Monitoring. Remove OpenTelemetry Exporter support from meter doc, as this has been flagged as unmaintained on OTEL upstream. Add doc of one-line quick start script for different storage types. Add FAQ for Why is Clickhouse or Loki or xxx not supported as a storage option?. Add SWIP-8 Support ActiveMQ Monitoring. Move BanyanDB storage to the recommended storage. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1000\"\u003e10.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Java 21 runtime.\u003c/li\u003e\n\u003cli\u003eSupport oap-java21 image for Java 21 runtime.\u003c/li\u003e\n\u003cli\u003eUpgrade \u003ccode\u003eOTEL …\u003c/code\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/latest/en/changes/changes-10.0.0/","title":"10.0.0"},{"body":"10.0.0 Project Support Java 21 runtime. Support oap-java21 image for Java 21 runtime. Upgrade OTEL collector version to 0.92.0 in all e2e tests. Switch CI macOS runner to m1. Upgrade PostgreSQL driver to 42.4.4 to fix CVE-2024-1597. Remove CLI(swctl) from the image. Remove CLI_VERSION variable from Makefile build. Add BanyanDB to docker-compose quickstart. Bump up Armeria, jackson, netty, jetcd and grpc to fix CVEs. Bump up BanyanDB Java Client to 0.6.0. OAP Server Add layer parameter to the global topology graphQL query. Add is_present function in MQE for check if the list metrics has a value or not. Remove unreasonable default configurations for gRPC thread executor. Remove gRPCThreadPoolQueueSize (SW_RECEIVER_GRPC_POOL_QUEUE_SIZE) configuration. Allow excluding ServiceEntries in some namespaces when looking up ServiceEntries as a final resolution method of service metadata. Set up the length of source and dest IDs in relation entities of service, instance, endpoint, and process to 250(was 200). Support build Service/Instance Hierarchy and query. Change the string field in Elasticsearch storage from keyword type to text type if it set more than 32766 length. [Break Change] Change the configuration field of ui_template and ui_menu in Elasticsearch storage from keyword type to text. Support Service Hierarchy auto matching, add auto matching layer relationships (upper -\u0026gt; lower) as following: MESH -\u0026gt; MESH_DP MESH -\u0026gt; K8S_SERVICE MESH_DP -\u0026gt; K8S_SERVICE GENERAL -\u0026gt; K8S_SERVICE Add namespace suffix for K8S_SERVICE_NAME_RULE/ISTIO_SERVICE_NAME_RULE and metadata-service-mapping.yaml as default. Allow using a dedicated port for ALS receiver. Fix log query by traceId in JDBCLogQueryDAO. Support handler eBPF access log protocol. Fix SumPerMinFunctionTest error function. Remove unnecessary annotations and functions from Meter Functions. Add max and min functions for MAL down sampling. Fix critical bug of uncontrolled memory cost of TopN statistics. Change topN group key from StorageId to entityId + timeBucket. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: MYSQL -\u0026gt; K8S_SERVICE POSTGRESQL -\u0026gt; K8S_SERVICE SO11Y_OAP -\u0026gt; K8S_SERVICE VIRTUAL_DATABASE -\u0026gt; MYSQL VIRTUAL_DATABASE -\u0026gt; POSTGRESQL Add Golang as a supported language for AMQP. Support available layers of service in the topology. Add count aggregation function for MAL Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: NGINX -\u0026gt; K8S_SERVICE APISIX -\u0026gt; K8S_SERVICE GENERAL -\u0026gt; APISIX Add Golang as a supported language for RocketMQ. Support Apache RocketMQ server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: ROCKETMQ -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; ROCKETMQ Fix ServiceInstance in query. Mock /api/v1/status/buildinfo for PromQL API. Fix table exists check in the JDBC Storage Plugin. Fix day-based table rolling time range strategy in JDBC storage. Add maxInboundMessageSize (SW_DCS_MAX_INBOUND_MESSAGE_SIZE) configuration to change the max inbound message size of DCS. Fix Service Layer when building Events in the EventHookCallback. Add Golang as a supported language for Pulsar. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: RABBITMQ -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; RABBITMQ Remove Column#function mechanism in the kernel. Make query readMetricValue always return the average value of the duration. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: KAFKA -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; KAFKA Support ClickHouse server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: CLICKHOUSE -\u0026gt; K8S_SERVICE VIRTUAL_DATABASE -\u0026gt; CLICKHOUSE Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: PULSAR -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; PULSAR Add Golang as a supported language for Kafka. Support displaying the port services listen to from OAP and UI during server start. Refactor data-generator to support generating metrics. Fix AvgHistogramPercentileFunction legacy name. [Break Change] Labeled Metrics support multiple labels. Storage: store all label names and values instead of only the values. MQE: Support querying by multiple labels(name and value) instead using _ as the anonymous label name. aggregate_labels function support aggregate by specific labels. relabels function require target label and rename label name and value. PromQL: Support querying by multiple labels(name and value) instead using lables as the anonymous label name. Remove general labels labels/relabels/label function. API /api/v1/labels and /api/v1/label/\u0026lt;label_name\u0026gt;/values support return matched metrics labels. OAL: Deprecate percentile function and introduce percentile2 function instead. Bump up Kafka to fix CVE. Fix NullPointerException in Istio ServiceEntry registry. Remove unnecessary componentIds as series ID in the ServiceRelationClientSideMetrics and ServiceRelationServerSideMetrics entities. Fix not throw error when part of expression not matched any expression node in the MQE and `PromQL. Remove kafka-fetcher/default/createTopicIfNotExist as the creation is automatically since #7326 (v8.7.0). Fix inaccuracy nginx service metrics. Fix/Change Windows metrics name(Swap -\u0026gt; Virtual Memory) memory_swap_free -\u0026gt; memory_virtual_memory_free memory_swap_total -\u0026gt; memory_virtual_memory_total memory_swap_percentage -\u0026gt; memory_virtual_memory_percentage Fix/Change UI init setting for Windows Swap -\u0026gt; Virtual Memory Fix Memory Swap Usage/Virtual Memory Usage display with UI init.(Linux/Windows) Fix inaccurate APISIX metrics. Fix inaccurate MongoDB Metrics. Support Apache ActiveMQ server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: ACTIVEMQ -\u0026gt; K8S_SERVICE Calculate Nginx service HTTP Latency by MQE. MQE query: make metadata not return null. MQE labeled metrics Binary Operation: return empty value if the labels not match rather than report error. Fix inaccurate Hierarchy of RabbitMQ Server monitoring metrics. Fix inaccurate MySQL/MariaDB, Redis, PostgreSQL metrics. Support DoubleValue,IntValue,BoolValue in OTEL metrics attributes. [Break Change] gGRPC metrics exporter unified the metric value type and support labeled metrics. Add component definition(ID=152) for c3p0(JDBC3 Connection and Statement Pooling). Fix MQE top_n global query. Fix inaccurate Pulsar and Bookkeeper metrics. MQE support sort_values and sort_label_values functions. UI Fix the mismatch between the unit and calculation of the \u0026ldquo;Network Bandwidth Usage\u0026rdquo; widget in Linux-Service Dashboard. Add theme change animation. Implement the Service and Instance hierarchy topology. Support Tabs in the widget visible when MQE expressions. Support search on Marketplace. Fix default route. Fix layout on the Log widget. Fix Trace associates with Log widget. Add isDefault to the dashboard configuration. Add expressions to dashboard configurations on the dashboard list page. Update Kubernetes related UI templates for adapt data from eBPF access log. Fix dashboard K8S-Service-Root metrics expression. Add dashboards for Service/Instance Hierarchy. Fix MQE in dashboards when using Card widget. Optimize tooltips style. Fix resizing window causes the trace graph to display incorrectly. Add the not found page(404). Enhance VNode logic and support multiple Trace IDs in span\u0026rsquo;s ref. Add the layers filed and associate layers dashboards for the service topology nodes. Fix Nginx-Instance metrics to instance level. Update tabs of the Kubernetes service page. Add Airflow menu i18n. Add Support for dragging in the trace panel. Add workflow icon. Metrics support multiple labels. Support the SINGLE_VALUE for table widgets. Remove the General metric mode and related logical code. Remove metrics for unreal nodes in the topology. Enhance the Trace widget for batch consuming spans. Clean the unused elements in the UI-templates. Documentation Update the release doc to remove the announcement as the tests are through e2e rather than manually. Update the release notification mail a little. Polish docs structure. Move customization docs separately from the introduction docs. Add webhook/gRPC hooks settings example for backend-alarm.md. Begin the process of SWIP - SkyWalking Improvement Proposal. Add SWIP-1 Create and detect Service Hierarchy Relationship. Add SWIP-2 Collecting and Gathering Kubernetes Monitoring Data. Update the Overview docs to add the Service Hierarchy Relationship section. Fix incorrect words for backend-bookkeeper-monitoring.md and backend-pulsar-monitoring.md Document a new way to load balance OAP. Add SWIP-3 Support RocketMQ monitoring. Add OpenTelemetry SkyWalking Exporter deprecated warning doc. Update i18n for rocketmq monitoring. Fix: remove click event after unmounted. Fix: end loading without query results. Update nanoid version to 3.3.7. Update postcss version to 8.4.33. Fix kafka topic name in exporter doc. Fix query-protocol.md, make it consistent with the GraphQL query protocol. Add SWIP-5 Support ClickHouse Monitoring. Remove OpenTelemetry Exporter support from meter doc, as this has been flagged as unmaintained on OTEL upstream. Add doc of one-line quick start script for different storage types. Add FAQ for Why is Clickhouse or Loki or xxx not supported as a storage option?. Add SWIP-8 Support ActiveMQ Monitoring. Move BanyanDB storage to the recommended storage. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1000\"\u003e10.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Java 21 runtime.\u003c/li\u003e\n\u003cli\u003eSupport oap-java21 image for Java 21 runtime.\u003c/li\u003e\n\u003cli\u003eUpgrade \u003ccode\u003eOTEL …\u003c/code\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/next/en/changes/changes-10.0.0/","title":"10.0.0"},{"body":"10.0.0 Project Support Java 21 runtime. Support oap-java21 image for Java 21 runtime. Upgrade OTEL collector version to 0.92.0 in all e2e tests. Switch CI macOS runner to m1. Upgrade PostgreSQL driver to 42.4.4 to fix CVE-2024-1597. Remove CLI(swctl) from the image. Remove CLI_VERSION variable from Makefile build. Add BanyanDB to docker-compose quickstart. Bump up Armeria, jackson, netty, jetcd and grpc to fix CVEs. Bump up BanyanDB Java Client to 0.6.0. OAP Server Add layer parameter to the global topology graphQL query. Add is_present function in MQE for check if the list metrics has a value or not. Remove unreasonable default configurations for gRPC thread executor. Remove gRPCThreadPoolQueueSize (SW_RECEIVER_GRPC_POOL_QUEUE_SIZE) configuration. Allow excluding ServiceEntries in some namespaces when looking up ServiceEntries as a final resolution method of service metadata. Set up the length of source and dest IDs in relation entities of service, instance, endpoint, and process to 250(was 200). Support build Service/Instance Hierarchy and query. Change the string field in Elasticsearch storage from keyword type to text type if it set more than 32766 length. [Break Change] Change the configuration field of ui_template and ui_menu in Elasticsearch storage from keyword type to text. Support Service Hierarchy auto matching, add auto matching layer relationships (upper -\u0026gt; lower) as following: MESH -\u0026gt; MESH_DP MESH -\u0026gt; K8S_SERVICE MESH_DP -\u0026gt; K8S_SERVICE GENERAL -\u0026gt; K8S_SERVICE Add namespace suffix for K8S_SERVICE_NAME_RULE/ISTIO_SERVICE_NAME_RULE and metadata-service-mapping.yaml as default. Allow using a dedicated port for ALS receiver. Fix log query by traceId in JDBCLogQueryDAO. Support handler eBPF access log protocol. Fix SumPerMinFunctionTest error function. Remove unnecessary annotations and functions from Meter Functions. Add max and min functions for MAL down sampling. Fix critical bug of uncontrolled memory cost of TopN statistics. Change topN group key from StorageId to entityId + timeBucket. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: MYSQL -\u0026gt; K8S_SERVICE POSTGRESQL -\u0026gt; K8S_SERVICE SO11Y_OAP -\u0026gt; K8S_SERVICE VIRTUAL_DATABASE -\u0026gt; MYSQL VIRTUAL_DATABASE -\u0026gt; POSTGRESQL Add Golang as a supported language for AMQP. Support available layers of service in the topology. Add count aggregation function for MAL Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: NGINX -\u0026gt; K8S_SERVICE APISIX -\u0026gt; K8S_SERVICE GENERAL -\u0026gt; APISIX Add Golang as a supported language for RocketMQ. Support Apache RocketMQ server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: ROCKETMQ -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; ROCKETMQ Fix ServiceInstance in query. Mock /api/v1/status/buildinfo for PromQL API. Fix table exists check in the JDBC Storage Plugin. Fix day-based table rolling time range strategy in JDBC storage. Add maxInboundMessageSize (SW_DCS_MAX_INBOUND_MESSAGE_SIZE) configuration to change the max inbound message size of DCS. Fix Service Layer when building Events in the EventHookCallback. Add Golang as a supported language for Pulsar. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: RABBITMQ -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; RABBITMQ Remove Column#function mechanism in the kernel. Make query readMetricValue always return the average value of the duration. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: KAFKA -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; KAFKA Support ClickHouse server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: CLICKHOUSE -\u0026gt; K8S_SERVICE VIRTUAL_DATABASE -\u0026gt; CLICKHOUSE Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: PULSAR -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; PULSAR Add Golang as a supported language for Kafka. Support displaying the port services listen to from OAP and UI during server start. Refactor data-generator to support generating metrics. Fix AvgHistogramPercentileFunction legacy name. [Break Change] Labeled Metrics support multiple labels. Storage: store all label names and values instead of only the values. MQE: Support querying by multiple labels(name and value) instead using _ as the anonymous label name. aggregate_labels function support aggregate by specific labels. relabels function require target label and rename label name and value. PromQL: Support querying by multiple labels(name and value) instead using lables as the anonymous label name. Remove general labels labels/relabels/label function. API /api/v1/labels and /api/v1/label/\u0026lt;label_name\u0026gt;/values support return matched metrics labels. OAL: Deprecate percentile function and introduce percentile2 function instead. Bump up Kafka to fix CVE. Fix NullPointerException in Istio ServiceEntry registry. Remove unnecessary componentIds as series ID in the ServiceRelationClientSideMetrics and ServiceRelationServerSideMetrics entities. Fix not throw error when part of expression not matched any expression node in the MQE and `PromQL. Remove kafka-fetcher/default/createTopicIfNotExist as the creation is automatically since #7326 (v8.7.0). Fix inaccuracy nginx service metrics. Fix/Change Windows metrics name(Swap -\u0026gt; Virtual Memory) memory_swap_free -\u0026gt; memory_virtual_memory_free memory_swap_total -\u0026gt; memory_virtual_memory_total memory_swap_percentage -\u0026gt; memory_virtual_memory_percentage Fix/Change UI init setting for Windows Swap -\u0026gt; Virtual Memory Fix Memory Swap Usage/Virtual Memory Usage display with UI init.(Linux/Windows) Fix inaccurate APISIX metrics. Fix inaccurate MongoDB Metrics. Support Apache ActiveMQ server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: ACTIVEMQ -\u0026gt; K8S_SERVICE Calculate Nginx service HTTP Latency by MQE. MQE query: make metadata not return null. MQE labeled metrics Binary Operation: return empty value if the labels not match rather than report error. Fix inaccurate Hierarchy of RabbitMQ Server monitoring metrics. Fix inaccurate MySQL/MariaDB, Redis, PostgreSQL metrics. Support DoubleValue,IntValue,BoolValue in OTEL metrics attributes. [Break Change] gGRPC metrics exporter unified the metric value type and support labeled metrics. Add component definition(ID=152) for c3p0(JDBC3 Connection and Statement Pooling). Fix MQE top_n global query. Fix inaccurate Pulsar and Bookkeeper metrics. MQE support sort_values and sort_label_values functions. UI Fix the mismatch between the unit and calculation of the \u0026ldquo;Network Bandwidth Usage\u0026rdquo; widget in Linux-Service Dashboard. Add theme change animation. Implement the Service and Instance hierarchy topology. Support Tabs in the widget visible when MQE expressions. Support search on Marketplace. Fix default route. Fix layout on the Log widget. Fix Trace associates with Log widget. Add isDefault to the dashboard configuration. Add expressions to dashboard configurations on the dashboard list page. Update Kubernetes related UI templates for adapt data from eBPF access log. Fix dashboard K8S-Service-Root metrics expression. Add dashboards for Service/Instance Hierarchy. Fix MQE in dashboards when using Card widget. Optimize tooltips style. Fix resizing window causes the trace graph to display incorrectly. Add the not found page(404). Enhance VNode logic and support multiple Trace IDs in span\u0026rsquo;s ref. Add the layers filed and associate layers dashboards for the service topology nodes. Fix Nginx-Instance metrics to instance level. Update tabs of the Kubernetes service page. Add Airflow menu i18n. Add Support for dragging in the trace panel. Add workflow icon. Metrics support multiple labels. Support the SINGLE_VALUE for table widgets. Remove the General metric mode and related logical code. Remove metrics for unreal nodes in the topology. Enhance the Trace widget for batch consuming spans. Clean the unused elements in the UI-templates. Documentation Update the release doc to remove the announcement as the tests are through e2e rather than manually. Update the release notification mail a little. Polish docs structure. Move customization docs separately from the introduction docs. Add webhook/gRPC hooks settings example for backend-alarm.md. Begin the process of SWIP - SkyWalking Improvement Proposal. Add SWIP-1 Create and detect Service Hierarchy Relationship. Add SWIP-2 Collecting and Gathering Kubernetes Monitoring Data. Update the Overview docs to add the Service Hierarchy Relationship section. Fix incorrect words for backend-bookkeeper-monitoring.md and backend-pulsar-monitoring.md Document a new way to load balance OAP. Add SWIP-3 Support RocketMQ monitoring. Add OpenTelemetry SkyWalking Exporter deprecated warning doc. Update i18n for rocketmq monitoring. Fix: remove click event after unmounted. Fix: end loading without query results. Update nanoid version to 3.3.7. Update postcss version to 8.4.33. Fix kafka topic name in exporter doc. Fix query-protocol.md, make it consistent with the GraphQL query protocol. Add SWIP-5 Support ClickHouse Monitoring. Remove OpenTelemetry Exporter support from meter doc, as this has been flagged as unmaintained on OTEL upstream. Add doc of one-line quick start script for different storage types. Add FAQ for Why is Clickhouse or Loki or xxx not supported as a storage option?. Add SWIP-8 Support ActiveMQ Monitoring. Move BanyanDB storage to the recommended storage. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1000\"\u003e10.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Java 21 runtime.\u003c/li\u003e\n\u003cli\u003eSupport oap-java21 image for Java 21 runtime.\u003c/li\u003e\n\u003cli\u003eUpgrade \u003ccode\u003eOTEL …\u003c/code\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.0.0/en/changes/changes-10.0.0/","title":"10.0.0"},{"body":"10.0.0 Project Support Java 21 runtime. Support oap-java21 image for Java 21 runtime. Upgrade OTEL collector version to 0.92.0 in all e2e tests. Switch CI macOS runner to m1. Upgrade PostgreSQL driver to 42.4.4 to fix CVE-2024-1597. Remove CLI(swctl) from the image. Remove CLI_VERSION variable from Makefile build. Add BanyanDB to docker-compose quickstart. Bump up Armeria, jackson, netty, jetcd and grpc to fix CVEs. Bump up BanyanDB Java Client to 0.6.0. OAP Server Add layer parameter to the global topology graphQL query. Add is_present function in MQE for check if the list metrics has a value or not. Remove unreasonable default configurations for gRPC thread executor. Remove gRPCThreadPoolQueueSize (SW_RECEIVER_GRPC_POOL_QUEUE_SIZE) configuration. Allow excluding ServiceEntries in some namespaces when looking up ServiceEntries as a final resolution method of service metadata. Set up the length of source and dest IDs in relation entities of service, instance, endpoint, and process to 250(was 200). Support build Service/Instance Hierarchy and query. Change the string field in Elasticsearch storage from keyword type to text type if it set more than 32766 length. [Break Change] Change the configuration field of ui_template and ui_menu in Elasticsearch storage from keyword type to text. Support Service Hierarchy auto matching, add auto matching layer relationships (upper -\u0026gt; lower) as following: MESH -\u0026gt; MESH_DP MESH -\u0026gt; K8S_SERVICE MESH_DP -\u0026gt; K8S_SERVICE GENERAL -\u0026gt; K8S_SERVICE Add namespace suffix for K8S_SERVICE_NAME_RULE/ISTIO_SERVICE_NAME_RULE and metadata-service-mapping.yaml as default. Allow using a dedicated port for ALS receiver. Fix log query by traceId in JDBCLogQueryDAO. Support handler eBPF access log protocol. Fix SumPerMinFunctionTest error function. Remove unnecessary annotations and functions from Meter Functions. Add max and min functions for MAL down sampling. Fix critical bug of uncontrolled memory cost of TopN statistics. Change topN group key from StorageId to entityId + timeBucket. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: MYSQL -\u0026gt; K8S_SERVICE POSTGRESQL -\u0026gt; K8S_SERVICE SO11Y_OAP -\u0026gt; K8S_SERVICE VIRTUAL_DATABASE -\u0026gt; MYSQL VIRTUAL_DATABASE -\u0026gt; POSTGRESQL Add Golang as a supported language for AMQP. Support available layers of service in the topology. Add count aggregation function for MAL Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: NGINX -\u0026gt; K8S_SERVICE APISIX -\u0026gt; K8S_SERVICE GENERAL -\u0026gt; APISIX Add Golang as a supported language for RocketMQ. Support Apache RocketMQ server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: ROCKETMQ -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; ROCKETMQ Fix ServiceInstance in query. Mock /api/v1/status/buildinfo for PromQL API. Fix table exists check in the JDBC Storage Plugin. Fix day-based table rolling time range strategy in JDBC storage. Add maxInboundMessageSize (SW_DCS_MAX_INBOUND_MESSAGE_SIZE) configuration to change the max inbound message size of DCS. Fix Service Layer when building Events in the EventHookCallback. Add Golang as a supported language for Pulsar. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: RABBITMQ -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; RABBITMQ Remove Column#function mechanism in the kernel. Make query readMetricValue always return the average value of the duration. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: KAFKA -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; KAFKA Support ClickHouse server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: CLICKHOUSE -\u0026gt; K8S_SERVICE VIRTUAL_DATABASE -\u0026gt; CLICKHOUSE Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: PULSAR -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; PULSAR Add Golang as a supported language for Kafka. Support displaying the port services listen to from OAP and UI during server start. Refactor data-generator to support generating metrics. Fix AvgHistogramPercentileFunction legacy name. [Break Change] Labeled Metrics support multiple labels. Storage: store all label names and values instead of only the values. MQE: Support querying by multiple labels(name and value) instead using _ as the anonymous label name. aggregate_labels function support aggregate by specific labels. relabels function require target label and rename label name and value. PromQL: Support querying by multiple labels(name and value) instead using lables as the anonymous label name. Remove general labels labels/relabels/label function. API /api/v1/labels and /api/v1/label/\u0026lt;label_name\u0026gt;/values support return matched metrics labels. OAL: Deprecate percentile function and introduce percentile2 function instead. Bump up Kafka to fix CVE. Fix NullPointerException in Istio ServiceEntry registry. Remove unnecessary componentIds as series ID in the ServiceRelationClientSideMetrics and ServiceRelationServerSideMetrics entities. Fix not throw error when part of expression not matched any expression node in the MQE and `PromQL. Remove kafka-fetcher/default/createTopicIfNotExist as the creation is automatically since #7326 (v8.7.0). Fix inaccuracy nginx service metrics. Fix/Change Windows metrics name(Swap -\u0026gt; Virtual Memory) memory_swap_free -\u0026gt; memory_virtual_memory_free memory_swap_total -\u0026gt; memory_virtual_memory_total memory_swap_percentage -\u0026gt; memory_virtual_memory_percentage Fix/Change UI init setting for Windows Swap -\u0026gt; Virtual Memory Fix Memory Swap Usage/Virtual Memory Usage display with UI init.(Linux/Windows) Fix inaccurate APISIX metrics. Fix inaccurate MongoDB Metrics. Support Apache ActiveMQ server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: ACTIVEMQ -\u0026gt; K8S_SERVICE Calculate Nginx service HTTP Latency by MQE. MQE query: make metadata not return null. MQE labeled metrics Binary Operation: return empty value if the labels not match rather than report error. Fix inaccurate Hierarchy of RabbitMQ Server monitoring metrics. Fix inaccurate MySQL/MariaDB, Redis, PostgreSQL metrics. Support DoubleValue,IntValue,BoolValue in OTEL metrics attributes. [Break Change] gGRPC metrics exporter unified the metric value type and support labeled metrics. Add component definition(ID=152) for c3p0(JDBC3 Connection and Statement Pooling). Fix MQE top_n global query. Fix inaccurate Pulsar and Bookkeeper metrics. MQE support sort_values and sort_label_values functions. UI Fix the mismatch between the unit and calculation of the \u0026ldquo;Network Bandwidth Usage\u0026rdquo; widget in Linux-Service Dashboard. Add theme change animation. Implement the Service and Instance hierarchy topology. Support Tabs in the widget visible when MQE expressions. Support search on Marketplace. Fix default route. Fix layout on the Log widget. Fix Trace associates with Log widget. Add isDefault to the dashboard configuration. Add expressions to dashboard configurations on the dashboard list page. Update Kubernetes related UI templates for adapt data from eBPF access log. Fix dashboard K8S-Service-Root metrics expression. Add dashboards for Service/Instance Hierarchy. Fix MQE in dashboards when using Card widget. Optimize tooltips style. Fix resizing window causes the trace graph to display incorrectly. Add the not found page(404). Enhance VNode logic and support multiple Trace IDs in span\u0026rsquo;s ref. Add the layers filed and associate layers dashboards for the service topology nodes. Fix Nginx-Instance metrics to instance level. Update tabs of the Kubernetes service page. Add Airflow menu i18n. Add Support for dragging in the trace panel. Add workflow icon. Metrics support multiple labels. Support the SINGLE_VALUE for table widgets. Remove the General metric mode and related logical code. Remove metrics for unreal nodes in the topology. Enhance the Trace widget for batch consuming spans. Clean the unused elements in the UI-templates. Documentation Update the release doc to remove the announcement as the tests are through e2e rather than manually. Update the release notification mail a little. Polish docs structure. Move customization docs separately from the introduction docs. Add webhook/gRPC hooks settings example for backend-alarm.md. Begin the process of SWIP - SkyWalking Improvement Proposal. Add SWIP-1 Create and detect Service Hierarchy Relationship. Add SWIP-2 Collecting and Gathering Kubernetes Monitoring Data. Update the Overview docs to add the Service Hierarchy Relationship section. Fix incorrect words for backend-bookkeeper-monitoring.md and backend-pulsar-monitoring.md Document a new way to load balance OAP. Add SWIP-3 Support RocketMQ monitoring. Add OpenTelemetry SkyWalking Exporter deprecated warning doc. Update i18n for rocketmq monitoring. Fix: remove click event after unmounted. Fix: end loading without query results. Update nanoid version to 3.3.7. Update postcss version to 8.4.33. Fix kafka topic name in exporter doc. Fix query-protocol.md, make it consistent with the GraphQL query protocol. Add SWIP-5 Support ClickHouse Monitoring. Remove OpenTelemetry Exporter support from meter doc, as this has been flagged as unmaintained on OTEL upstream. Add doc of one-line quick start script for different storage types. Add FAQ for Why is Clickhouse or Loki or xxx not supported as a storage option?. Add SWIP-8 Support ActiveMQ Monitoring. Move BanyanDB storage to the recommended storage. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1000\"\u003e10.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Java 21 runtime.\u003c/li\u003e\n\u003cli\u003eSupport oap-java21 image for Java 21 runtime.\u003c/li\u003e\n\u003cli\u003eUpgrade \u003ccode\u003eOTEL …\u003c/code\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.0.1/en/changes/changes-10.0.0/","title":"10.0.0"},{"body":"10.0.0 Project Support Java 21 runtime. Support oap-java21 image for Java 21 runtime. Upgrade OTEL collector version to 0.92.0 in all e2e tests. Switch CI macOS runner to m1. Upgrade PostgreSQL driver to 42.4.4 to fix CVE-2024-1597. Remove CLI(swctl) from the image. Remove CLI_VERSION variable from Makefile build. Add BanyanDB to docker-compose quickstart. Bump up Armeria, jackson, netty, jetcd and grpc to fix CVEs. Bump up BanyanDB Java Client to 0.6.0. OAP Server Add layer parameter to the global topology graphQL query. Add is_present function in MQE for check if the list metrics has a value or not. Remove unreasonable default configurations for gRPC thread executor. Remove gRPCThreadPoolQueueSize (SW_RECEIVER_GRPC_POOL_QUEUE_SIZE) configuration. Allow excluding ServiceEntries in some namespaces when looking up ServiceEntries as a final resolution method of service metadata. Set up the length of source and dest IDs in relation entities of service, instance, endpoint, and process to 250(was 200). Support build Service/Instance Hierarchy and query. Change the string field in Elasticsearch storage from keyword type to text type if it set more than 32766 length. [Break Change] Change the configuration field of ui_template and ui_menu in Elasticsearch storage from keyword type to text. Support Service Hierarchy auto matching, add auto matching layer relationships (upper -\u0026gt; lower) as following: MESH -\u0026gt; MESH_DP MESH -\u0026gt; K8S_SERVICE MESH_DP -\u0026gt; K8S_SERVICE GENERAL -\u0026gt; K8S_SERVICE Add namespace suffix for K8S_SERVICE_NAME_RULE/ISTIO_SERVICE_NAME_RULE and metadata-service-mapping.yaml as default. Allow using a dedicated port for ALS receiver. Fix log query by traceId in JDBCLogQueryDAO. Support handler eBPF access log protocol. Fix SumPerMinFunctionTest error function. Remove unnecessary annotations and functions from Meter Functions. Add max and min functions for MAL down sampling. Fix critical bug of uncontrolled memory cost of TopN statistics. Change topN group key from StorageId to entityId + timeBucket. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: MYSQL -\u0026gt; K8S_SERVICE POSTGRESQL -\u0026gt; K8S_SERVICE SO11Y_OAP -\u0026gt; K8S_SERVICE VIRTUAL_DATABASE -\u0026gt; MYSQL VIRTUAL_DATABASE -\u0026gt; POSTGRESQL Add Golang as a supported language for AMQP. Support available layers of service in the topology. Add count aggregation function for MAL Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: NGINX -\u0026gt; K8S_SERVICE APISIX -\u0026gt; K8S_SERVICE GENERAL -\u0026gt; APISIX Add Golang as a supported language for RocketMQ. Support Apache RocketMQ server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: ROCKETMQ -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; ROCKETMQ Fix ServiceInstance in query. Mock /api/v1/status/buildinfo for PromQL API. Fix table exists check in the JDBC Storage Plugin. Fix day-based table rolling time range strategy in JDBC storage. Add maxInboundMessageSize (SW_DCS_MAX_INBOUND_MESSAGE_SIZE) configuration to change the max inbound message size of DCS. Fix Service Layer when building Events in the EventHookCallback. Add Golang as a supported language for Pulsar. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: RABBITMQ -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; RABBITMQ Remove Column#function mechanism in the kernel. Make query readMetricValue always return the average value of the duration. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: KAFKA -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; KAFKA Support ClickHouse server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: CLICKHOUSE -\u0026gt; K8S_SERVICE VIRTUAL_DATABASE -\u0026gt; CLICKHOUSE Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: PULSAR -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; PULSAR Add Golang as a supported language for Kafka. Support displaying the port services listen to from OAP and UI during server start. Refactor data-generator to support generating metrics. Fix AvgHistogramPercentileFunction legacy name. [Break Change] Labeled Metrics support multiple labels. Storage: store all label names and values instead of only the values. MQE: Support querying by multiple labels(name and value) instead using _ as the anonymous label name. aggregate_labels function support aggregate by specific labels. relabels function require target label and rename label name and value. PromQL: Support querying by multiple labels(name and value) instead using lables as the anonymous label name. Remove general labels labels/relabels/label function. API /api/v1/labels and /api/v1/label/\u0026lt;label_name\u0026gt;/values support return matched metrics labels. OAL: Deprecate percentile function and introduce percentile2 function instead. Bump up Kafka to fix CVE. Fix NullPointerException in Istio ServiceEntry registry. Remove unnecessary componentIds as series ID in the ServiceRelationClientSideMetrics and ServiceRelationServerSideMetrics entities. Fix not throw error when part of expression not matched any expression node in the MQE and `PromQL. Remove kafka-fetcher/default/createTopicIfNotExist as the creation is automatically since #7326 (v8.7.0). Fix inaccuracy nginx service metrics. Fix/Change Windows metrics name(Swap -\u0026gt; Virtual Memory) memory_swap_free -\u0026gt; memory_virtual_memory_free memory_swap_total -\u0026gt; memory_virtual_memory_total memory_swap_percentage -\u0026gt; memory_virtual_memory_percentage Fix/Change UI init setting for Windows Swap -\u0026gt; Virtual Memory Fix Memory Swap Usage/Virtual Memory Usage display with UI init.(Linux/Windows) Fix inaccurate APISIX metrics. Fix inaccurate MongoDB Metrics. Support Apache ActiveMQ server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: ACTIVEMQ -\u0026gt; K8S_SERVICE Calculate Nginx service HTTP Latency by MQE. MQE query: make metadata not return null. MQE labeled metrics Binary Operation: return empty value if the labels not match rather than report error. Fix inaccurate Hierarchy of RabbitMQ Server monitoring metrics. Fix inaccurate MySQL/MariaDB, Redis, PostgreSQL metrics. Support DoubleValue,IntValue,BoolValue in OTEL metrics attributes. [Break Change] gGRPC metrics exporter unified the metric value type and support labeled metrics. Add component definition(ID=152) for c3p0(JDBC3 Connection and Statement Pooling). Fix MQE top_n global query. Fix inaccurate Pulsar and Bookkeeper metrics. MQE support sort_values and sort_label_values functions. UI Fix the mismatch between the unit and calculation of the \u0026ldquo;Network Bandwidth Usage\u0026rdquo; widget in Linux-Service Dashboard. Add theme change animation. Implement the Service and Instance hierarchy topology. Support Tabs in the widget visible when MQE expressions. Support search on Marketplace. Fix default route. Fix layout on the Log widget. Fix Trace associates with Log widget. Add isDefault to the dashboard configuration. Add expressions to dashboard configurations on the dashboard list page. Update Kubernetes related UI templates for adapt data from eBPF access log. Fix dashboard K8S-Service-Root metrics expression. Add dashboards for Service/Instance Hierarchy. Fix MQE in dashboards when using Card widget. Optimize tooltips style. Fix resizing window causes the trace graph to display incorrectly. Add the not found page(404). Enhance VNode logic and support multiple Trace IDs in span\u0026rsquo;s ref. Add the layers filed and associate layers dashboards for the service topology nodes. Fix Nginx-Instance metrics to instance level. Update tabs of the Kubernetes service page. Add Airflow menu i18n. Add Support for dragging in the trace panel. Add workflow icon. Metrics support multiple labels. Support the SINGLE_VALUE for table widgets. Remove the General metric mode and related logical code. Remove metrics for unreal nodes in the topology. Enhance the Trace widget for batch consuming spans. Clean the unused elements in the UI-templates. Documentation Update the release doc to remove the announcement as the tests are through e2e rather than manually. Update the release notification mail a little. Polish docs structure. Move customization docs separately from the introduction docs. Add webhook/gRPC hooks settings example for backend-alarm.md. Begin the process of SWIP - SkyWalking Improvement Proposal. Add SWIP-1 Create and detect Service Hierarchy Relationship. Add SWIP-2 Collecting and Gathering Kubernetes Monitoring Data. Update the Overview docs to add the Service Hierarchy Relationship section. Fix incorrect words for backend-bookkeeper-monitoring.md and backend-pulsar-monitoring.md Document a new way to load balance OAP. Add SWIP-3 Support RocketMQ monitoring. Add OpenTelemetry SkyWalking Exporter deprecated warning doc. Update i18n for rocketmq monitoring. Fix: remove click event after unmounted. Fix: end loading without query results. Update nanoid version to 3.3.7. Update postcss version to 8.4.33. Fix kafka topic name in exporter doc. Fix query-protocol.md, make it consistent with the GraphQL query protocol. Add SWIP-5 Support ClickHouse Monitoring. Remove OpenTelemetry Exporter support from meter doc, as this has been flagged as unmaintained on OTEL upstream. Add doc of one-line quick start script for different storage types. Add FAQ for Why is Clickhouse or Loki or xxx not supported as a storage option?. Add SWIP-8 Support ActiveMQ Monitoring. Move BanyanDB storage to the recommended storage. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1000\"\u003e10.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Java 21 runtime.\u003c/li\u003e\n\u003cli\u003eSupport oap-java21 image for Java 21 runtime.\u003c/li\u003e\n\u003cli\u003eUpgrade \u003ccode\u003eOTEL …\u003c/code\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.1.0/en/changes/changes-10.0.0/","title":"10.0.0"},{"body":"10.0.0 Project Support Java 21 runtime. Support oap-java21 image for Java 21 runtime. Upgrade OTEL collector version to 0.92.0 in all e2e tests. Switch CI macOS runner to m1. Upgrade PostgreSQL driver to 42.4.4 to fix CVE-2024-1597. Remove CLI(swctl) from the image. Remove CLI_VERSION variable from Makefile build. Add BanyanDB to docker-compose quickstart. Bump up Armeria, jackson, netty, jetcd and grpc to fix CVEs. Bump up BanyanDB Java Client to 0.6.0. OAP Server Add layer parameter to the global topology graphQL query. Add is_present function in MQE for check if the list metrics has a value or not. Remove unreasonable default configurations for gRPC thread executor. Remove gRPCThreadPoolQueueSize (SW_RECEIVER_GRPC_POOL_QUEUE_SIZE) configuration. Allow excluding ServiceEntries in some namespaces when looking up ServiceEntries as a final resolution method of service metadata. Set up the length of source and dest IDs in relation entities of service, instance, endpoint, and process to 250(was 200). Support build Service/Instance Hierarchy and query. Change the string field in Elasticsearch storage from keyword type to text type if it set more than 32766 length. [Break Change] Change the configuration field of ui_template and ui_menu in Elasticsearch storage from keyword type to text. Support Service Hierarchy auto matching, add auto matching layer relationships (upper -\u0026gt; lower) as following: MESH -\u0026gt; MESH_DP MESH -\u0026gt; K8S_SERVICE MESH_DP -\u0026gt; K8S_SERVICE GENERAL -\u0026gt; K8S_SERVICE Add namespace suffix for K8S_SERVICE_NAME_RULE/ISTIO_SERVICE_NAME_RULE and metadata-service-mapping.yaml as default. Allow using a dedicated port for ALS receiver. Fix log query by traceId in JDBCLogQueryDAO. Support handler eBPF access log protocol. Fix SumPerMinFunctionTest error function. Remove unnecessary annotations and functions from Meter Functions. Add max and min functions for MAL down sampling. Fix critical bug of uncontrolled memory cost of TopN statistics. Change topN group key from StorageId to entityId + timeBucket. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: MYSQL -\u0026gt; K8S_SERVICE POSTGRESQL -\u0026gt; K8S_SERVICE SO11Y_OAP -\u0026gt; K8S_SERVICE VIRTUAL_DATABASE -\u0026gt; MYSQL VIRTUAL_DATABASE -\u0026gt; POSTGRESQL Add Golang as a supported language for AMQP. Support available layers of service in the topology. Add count aggregation function for MAL Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: NGINX -\u0026gt; K8S_SERVICE APISIX -\u0026gt; K8S_SERVICE GENERAL -\u0026gt; APISIX Add Golang as a supported language for RocketMQ. Support Apache RocketMQ server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: ROCKETMQ -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; ROCKETMQ Fix ServiceInstance in query. Mock /api/v1/status/buildinfo for PromQL API. Fix table exists check in the JDBC Storage Plugin. Fix day-based table rolling time range strategy in JDBC storage. Add maxInboundMessageSize (SW_DCS_MAX_INBOUND_MESSAGE_SIZE) configuration to change the max inbound message size of DCS. Fix Service Layer when building Events in the EventHookCallback. Add Golang as a supported language for Pulsar. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: RABBITMQ -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; RABBITMQ Remove Column#function mechanism in the kernel. Make query readMetricValue always return the average value of the duration. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: KAFKA -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; KAFKA Support ClickHouse server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: CLICKHOUSE -\u0026gt; K8S_SERVICE VIRTUAL_DATABASE -\u0026gt; CLICKHOUSE Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: PULSAR -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; PULSAR Add Golang as a supported language for Kafka. Support displaying the port services listen to from OAP and UI during server start. Refactor data-generator to support generating metrics. Fix AvgHistogramPercentileFunction legacy name. [Break Change] Labeled Metrics support multiple labels. Storage: store all label names and values instead of only the values. MQE: Support querying by multiple labels(name and value) instead using _ as the anonymous label name. aggregate_labels function support aggregate by specific labels. relabels function require target label and rename label name and value. PromQL: Support querying by multiple labels(name and value) instead using lables as the anonymous label name. Remove general labels labels/relabels/label function. API /api/v1/labels and /api/v1/label/\u0026lt;label_name\u0026gt;/values support return matched metrics labels. OAL: Deprecate percentile function and introduce percentile2 function instead. Bump up Kafka to fix CVE. Fix NullPointerException in Istio ServiceEntry registry. Remove unnecessary componentIds as series ID in the ServiceRelationClientSideMetrics and ServiceRelationServerSideMetrics entities. Fix not throw error when part of expression not matched any expression node in the MQE and `PromQL. Remove kafka-fetcher/default/createTopicIfNotExist as the creation is automatically since #7326 (v8.7.0). Fix inaccuracy nginx service metrics. Fix/Change Windows metrics name(Swap -\u0026gt; Virtual Memory) memory_swap_free -\u0026gt; memory_virtual_memory_free memory_swap_total -\u0026gt; memory_virtual_memory_total memory_swap_percentage -\u0026gt; memory_virtual_memory_percentage Fix/Change UI init setting for Windows Swap -\u0026gt; Virtual Memory Fix Memory Swap Usage/Virtual Memory Usage display with UI init.(Linux/Windows) Fix inaccurate APISIX metrics. Fix inaccurate MongoDB Metrics. Support Apache ActiveMQ server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: ACTIVEMQ -\u0026gt; K8S_SERVICE Calculate Nginx service HTTP Latency by MQE. MQE query: make metadata not return null. MQE labeled metrics Binary Operation: return empty value if the labels not match rather than report error. Fix inaccurate Hierarchy of RabbitMQ Server monitoring metrics. Fix inaccurate MySQL/MariaDB, Redis, PostgreSQL metrics. Support DoubleValue,IntValue,BoolValue in OTEL metrics attributes. [Break Change] gGRPC metrics exporter unified the metric value type and support labeled metrics. Add component definition(ID=152) for c3p0(JDBC3 Connection and Statement Pooling). Fix MQE top_n global query. Fix inaccurate Pulsar and Bookkeeper metrics. MQE support sort_values and sort_label_values functions. UI Fix the mismatch between the unit and calculation of the \u0026ldquo;Network Bandwidth Usage\u0026rdquo; widget in Linux-Service Dashboard. Add theme change animation. Implement the Service and Instance hierarchy topology. Support Tabs in the widget visible when MQE expressions. Support search on Marketplace. Fix default route. Fix layout on the Log widget. Fix Trace associates with Log widget. Add isDefault to the dashboard configuration. Add expressions to dashboard configurations on the dashboard list page. Update Kubernetes related UI templates for adapt data from eBPF access log. Fix dashboard K8S-Service-Root metrics expression. Add dashboards for Service/Instance Hierarchy. Fix MQE in dashboards when using Card widget. Optimize tooltips style. Fix resizing window causes the trace graph to display incorrectly. Add the not found page(404). Enhance VNode logic and support multiple Trace IDs in span\u0026rsquo;s ref. Add the layers filed and associate layers dashboards for the service topology nodes. Fix Nginx-Instance metrics to instance level. Update tabs of the Kubernetes service page. Add Airflow menu i18n. Add Support for dragging in the trace panel. Add workflow icon. Metrics support multiple labels. Support the SINGLE_VALUE for table widgets. Remove the General metric mode and related logical code. Remove metrics for unreal nodes in the topology. Enhance the Trace widget for batch consuming spans. Clean the unused elements in the UI-templates. Documentation Update the release doc to remove the announcement as the tests are through e2e rather than manually. Update the release notification mail a little. Polish docs structure. Move customization docs separately from the introduction docs. Add webhook/gRPC hooks settings example for backend-alarm.md. Begin the process of SWIP - SkyWalking Improvement Proposal. Add SWIP-1 Create and detect Service Hierarchy Relationship. Add SWIP-2 Collecting and Gathering Kubernetes Monitoring Data. Update the Overview docs to add the Service Hierarchy Relationship section. Fix incorrect words for backend-bookkeeper-monitoring.md and backend-pulsar-monitoring.md Document a new way to load balance OAP. Add SWIP-3 Support RocketMQ monitoring. Add OpenTelemetry SkyWalking Exporter deprecated warning doc. Update i18n for rocketmq monitoring. Fix: remove click event after unmounted. Fix: end loading without query results. Update nanoid version to 3.3.7. Update postcss version to 8.4.33. Fix kafka topic name in exporter doc. Fix query-protocol.md, make it consistent with the GraphQL query protocol. Add SWIP-5 Support ClickHouse Monitoring. Remove OpenTelemetry Exporter support from meter doc, as this has been flagged as unmaintained on OTEL upstream. Add doc of one-line quick start script for different storage types. Add FAQ for Why is Clickhouse or Loki or xxx not supported as a storage option?. Add SWIP-8 Support ActiveMQ Monitoring. Move BanyanDB storage to the recommended storage. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1000\"\u003e10.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Java 21 runtime.\u003c/li\u003e\n\u003cli\u003eSupport oap-java21 image for Java 21 runtime.\u003c/li\u003e\n\u003cli\u003eUpgrade \u003ccode\u003eOTEL …\u003c/code\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.2.0/en/changes/changes-10.0.0/","title":"10.0.0"},{"body":"10.0.0 Project Support Java 21 runtime. Support oap-java21 image for Java 21 runtime. Upgrade OTEL collector version to 0.92.0 in all e2e tests. Switch CI macOS runner to m1. Upgrade PostgreSQL driver to 42.4.4 to fix CVE-2024-1597. Remove CLI(swctl) from the image. Remove CLI_VERSION variable from Makefile build. Add BanyanDB to docker-compose quickstart. Bump up Armeria, jackson, netty, jetcd and grpc to fix CVEs. Bump up BanyanDB Java Client to 0.6.0. OAP Server Add layer parameter to the global topology graphQL query. Add is_present function in MQE for check if the list metrics has a value or not. Remove unreasonable default configurations for gRPC thread executor. Remove gRPCThreadPoolQueueSize (SW_RECEIVER_GRPC_POOL_QUEUE_SIZE) configuration. Allow excluding ServiceEntries in some namespaces when looking up ServiceEntries as a final resolution method of service metadata. Set up the length of source and dest IDs in relation entities of service, instance, endpoint, and process to 250(was 200). Support build Service/Instance Hierarchy and query. Change the string field in Elasticsearch storage from keyword type to text type if it set more than 32766 length. [Break Change] Change the configuration field of ui_template and ui_menu in Elasticsearch storage from keyword type to text. Support Service Hierarchy auto matching, add auto matching layer relationships (upper -\u0026gt; lower) as following: MESH -\u0026gt; MESH_DP MESH -\u0026gt; K8S_SERVICE MESH_DP -\u0026gt; K8S_SERVICE GENERAL -\u0026gt; K8S_SERVICE Add namespace suffix for K8S_SERVICE_NAME_RULE/ISTIO_SERVICE_NAME_RULE and metadata-service-mapping.yaml as default. Allow using a dedicated port for ALS receiver. Fix log query by traceId in JDBCLogQueryDAO. Support handler eBPF access log protocol. Fix SumPerMinFunctionTest error function. Remove unnecessary annotations and functions from Meter Functions. Add max and min functions for MAL down sampling. Fix critical bug of uncontrolled memory cost of TopN statistics. Change topN group key from StorageId to entityId + timeBucket. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: MYSQL -\u0026gt; K8S_SERVICE POSTGRESQL -\u0026gt; K8S_SERVICE SO11Y_OAP -\u0026gt; K8S_SERVICE VIRTUAL_DATABASE -\u0026gt; MYSQL VIRTUAL_DATABASE -\u0026gt; POSTGRESQL Add Golang as a supported language for AMQP. Support available layers of service in the topology. Add count aggregation function for MAL Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: NGINX -\u0026gt; K8S_SERVICE APISIX -\u0026gt; K8S_SERVICE GENERAL -\u0026gt; APISIX Add Golang as a supported language for RocketMQ. Support Apache RocketMQ server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: ROCKETMQ -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; ROCKETMQ Fix ServiceInstance in query. Mock /api/v1/status/buildinfo for PromQL API. Fix table exists check in the JDBC Storage Plugin. Fix day-based table rolling time range strategy in JDBC storage. Add maxInboundMessageSize (SW_DCS_MAX_INBOUND_MESSAGE_SIZE) configuration to change the max inbound message size of DCS. Fix Service Layer when building Events in the EventHookCallback. Add Golang as a supported language for Pulsar. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: RABBITMQ -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; RABBITMQ Remove Column#function mechanism in the kernel. Make query readMetricValue always return the average value of the duration. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: KAFKA -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; KAFKA Support ClickHouse server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: CLICKHOUSE -\u0026gt; K8S_SERVICE VIRTUAL_DATABASE -\u0026gt; CLICKHOUSE Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: PULSAR -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; PULSAR Add Golang as a supported language for Kafka. Support displaying the port services listen to from OAP and UI during server start. Refactor data-generator to support generating metrics. Fix AvgHistogramPercentileFunction legacy name. [Break Change] Labeled Metrics support multiple labels. Storage: store all label names and values instead of only the values. MQE: Support querying by multiple labels(name and value) instead using _ as the anonymous label name. aggregate_labels function support aggregate by specific labels. relabels function require target label and rename label name and value. PromQL: Support querying by multiple labels(name and value) instead using lables as the anonymous label name. Remove general labels labels/relabels/label function. API /api/v1/labels and /api/v1/label/\u0026lt;label_name\u0026gt;/values support return matched metrics labels. OAL: Deprecate percentile function and introduce percentile2 function instead. Bump up Kafka to fix CVE. Fix NullPointerException in Istio ServiceEntry registry. Remove unnecessary componentIds as series ID in the ServiceRelationClientSideMetrics and ServiceRelationServerSideMetrics entities. Fix not throw error when part of expression not matched any expression node in the MQE and `PromQL. Remove kafka-fetcher/default/createTopicIfNotExist as the creation is automatically since #7326 (v8.7.0). Fix inaccuracy nginx service metrics. Fix/Change Windows metrics name(Swap -\u0026gt; Virtual Memory) memory_swap_free -\u0026gt; memory_virtual_memory_free memory_swap_total -\u0026gt; memory_virtual_memory_total memory_swap_percentage -\u0026gt; memory_virtual_memory_percentage Fix/Change UI init setting for Windows Swap -\u0026gt; Virtual Memory Fix Memory Swap Usage/Virtual Memory Usage display with UI init.(Linux/Windows) Fix inaccurate APISIX metrics. Fix inaccurate MongoDB Metrics. Support Apache ActiveMQ server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: ACTIVEMQ -\u0026gt; K8S_SERVICE Calculate Nginx service HTTP Latency by MQE. MQE query: make metadata not return null. MQE labeled metrics Binary Operation: return empty value if the labels not match rather than report error. Fix inaccurate Hierarchy of RabbitMQ Server monitoring metrics. Fix inaccurate MySQL/MariaDB, Redis, PostgreSQL metrics. Support DoubleValue,IntValue,BoolValue in OTEL metrics attributes. [Break Change] gGRPC metrics exporter unified the metric value type and support labeled metrics. Add component definition(ID=152) for c3p0(JDBC3 Connection and Statement Pooling). Fix MQE top_n global query. Fix inaccurate Pulsar and Bookkeeper metrics. MQE support sort_values and sort_label_values functions. UI Fix the mismatch between the unit and calculation of the \u0026ldquo;Network Bandwidth Usage\u0026rdquo; widget in Linux-Service Dashboard. Add theme change animation. Implement the Service and Instance hierarchy topology. Support Tabs in the widget visible when MQE expressions. Support search on Marketplace. Fix default route. Fix layout on the Log widget. Fix Trace associates with Log widget. Add isDefault to the dashboard configuration. Add expressions to dashboard configurations on the dashboard list page. Update Kubernetes related UI templates for adapt data from eBPF access log. Fix dashboard K8S-Service-Root metrics expression. Add dashboards for Service/Instance Hierarchy. Fix MQE in dashboards when using Card widget. Optimize tooltips style. Fix resizing window causes the trace graph to display incorrectly. Add the not found page(404). Enhance VNode logic and support multiple Trace IDs in span\u0026rsquo;s ref. Add the layers filed and associate layers dashboards for the service topology nodes. Fix Nginx-Instance metrics to instance level. Update tabs of the Kubernetes service page. Add Airflow menu i18n. Add Support for dragging in the trace panel. Add workflow icon. Metrics support multiple labels. Support the SINGLE_VALUE for table widgets. Remove the General metric mode and related logical code. Remove metrics for unreal nodes in the topology. Enhance the Trace widget for batch consuming spans. Clean the unused elements in the UI-templates. Documentation Update the release doc to remove the announcement as the tests are through e2e rather than manually. Update the release notification mail a little. Polish docs structure. Move customization docs separately from the introduction docs. Add webhook/gRPC hooks settings example for backend-alarm.md. Begin the process of SWIP - SkyWalking Improvement Proposal. Add SWIP-1 Create and detect Service Hierarchy Relationship. Add SWIP-2 Collecting and Gathering Kubernetes Monitoring Data. Update the Overview docs to add the Service Hierarchy Relationship section. Fix incorrect words for backend-bookkeeper-monitoring.md and backend-pulsar-monitoring.md Document a new way to load balance OAP. Add SWIP-3 Support RocketMQ monitoring. Add OpenTelemetry SkyWalking Exporter deprecated warning doc. Update i18n for rocketmq monitoring. Fix: remove click event after unmounted. Fix: end loading without query results. Update nanoid version to 3.3.7. Update postcss version to 8.4.33. Fix kafka topic name in exporter doc. Fix query-protocol.md, make it consistent with the GraphQL query protocol. Add SWIP-5 Support ClickHouse Monitoring. Remove OpenTelemetry Exporter support from meter doc, as this has been flagged as unmaintained on OTEL upstream. Add doc of one-line quick start script for different storage types. Add FAQ for Why is Clickhouse or Loki or xxx not supported as a storage option?. Add SWIP-8 Support ActiveMQ Monitoring. Move BanyanDB storage to the recommended storage. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1000\"\u003e10.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Java 21 runtime.\u003c/li\u003e\n\u003cli\u003eSupport oap-java21 image for Java 21 runtime.\u003c/li\u003e\n\u003cli\u003eUpgrade \u003ccode\u003eOTEL …\u003c/code\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.3.0/en/changes/changes-10.0.0/","title":"10.0.0"},{"body":"10.0.0 Project Support Java 21 runtime. Support oap-java21 image for Java 21 runtime. Upgrade OTEL collector version to 0.92.0 in all e2e tests. Switch CI macOS runner to m1. Upgrade PostgreSQL driver to 42.4.4 to fix CVE-2024-1597. Remove CLI(swctl) from the image. Remove CLI_VERSION variable from Makefile build. Add BanyanDB to docker-compose quickstart. Bump up Armeria, jackson, netty, jetcd and grpc to fix CVEs. Bump up BanyanDB Java Client to 0.6.0. OAP Server Add layer parameter to the global topology graphQL query. Add is_present function in MQE for check if the list metrics has a value or not. Remove unreasonable default configurations for gRPC thread executor. Remove gRPCThreadPoolQueueSize (SW_RECEIVER_GRPC_POOL_QUEUE_SIZE) configuration. Allow excluding ServiceEntries in some namespaces when looking up ServiceEntries as a final resolution method of service metadata. Set up the length of source and dest IDs in relation entities of service, instance, endpoint, and process to 250(was 200). Support build Service/Instance Hierarchy and query. Change the string field in Elasticsearch storage from keyword type to text type if it set more than 32766 length. [Break Change] Change the configuration field of ui_template and ui_menu in Elasticsearch storage from keyword type to text. Support Service Hierarchy auto matching, add auto matching layer relationships (upper -\u0026gt; lower) as following: MESH -\u0026gt; MESH_DP MESH -\u0026gt; K8S_SERVICE MESH_DP -\u0026gt; K8S_SERVICE GENERAL -\u0026gt; K8S_SERVICE Add namespace suffix for K8S_SERVICE_NAME_RULE/ISTIO_SERVICE_NAME_RULE and metadata-service-mapping.yaml as default. Allow using a dedicated port for ALS receiver. Fix log query by traceId in JDBCLogQueryDAO. Support handler eBPF access log protocol. Fix SumPerMinFunctionTest error function. Remove unnecessary annotations and functions from Meter Functions. Add max and min functions for MAL down sampling. Fix critical bug of uncontrolled memory cost of TopN statistics. Change topN group key from StorageId to entityId + timeBucket. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: MYSQL -\u0026gt; K8S_SERVICE POSTGRESQL -\u0026gt; K8S_SERVICE SO11Y_OAP -\u0026gt; K8S_SERVICE VIRTUAL_DATABASE -\u0026gt; MYSQL VIRTUAL_DATABASE -\u0026gt; POSTGRESQL Add Golang as a supported language for AMQP. Support available layers of service in the topology. Add count aggregation function for MAL Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: NGINX -\u0026gt; K8S_SERVICE APISIX -\u0026gt; K8S_SERVICE GENERAL -\u0026gt; APISIX Add Golang as a supported language for RocketMQ. Support Apache RocketMQ server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: ROCKETMQ -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; ROCKETMQ Fix ServiceInstance in query. Mock /api/v1/status/buildinfo for PromQL API. Fix table exists check in the JDBC Storage Plugin. Fix day-based table rolling time range strategy in JDBC storage. Add maxInboundMessageSize (SW_DCS_MAX_INBOUND_MESSAGE_SIZE) configuration to change the max inbound message size of DCS. Fix Service Layer when building Events in the EventHookCallback. Add Golang as a supported language for Pulsar. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: RABBITMQ -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; RABBITMQ Remove Column#function mechanism in the kernel. Make query readMetricValue always return the average value of the duration. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: KAFKA -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; KAFKA Support ClickHouse server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: CLICKHOUSE -\u0026gt; K8S_SERVICE VIRTUAL_DATABASE -\u0026gt; CLICKHOUSE Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: PULSAR -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; PULSAR Add Golang as a supported language for Kafka. Support displaying the port services listen to from OAP and UI during server start. Refactor data-generator to support generating metrics. Fix AvgHistogramPercentileFunction legacy name. [Break Change] Labeled Metrics support multiple labels. Storage: store all label names and values instead of only the values. MQE: Support querying by multiple labels(name and value) instead using _ as the anonymous label name. aggregate_labels function support aggregate by specific labels. relabels function require target label and rename label name and value. PromQL: Support querying by multiple labels(name and value) instead using lables as the anonymous label name. Remove general labels labels/relabels/label function. API /api/v1/labels and /api/v1/label/\u0026lt;label_name\u0026gt;/values support return matched metrics labels. OAL: Deprecate percentile function and introduce percentile2 function instead. Bump up Kafka to fix CVE. Fix NullPointerException in Istio ServiceEntry registry. Remove unnecessary componentIds as series ID in the ServiceRelationClientSideMetrics and ServiceRelationServerSideMetrics entities. Fix not throw error when part of expression not matched any expression node in the MQE and `PromQL. Remove kafka-fetcher/default/createTopicIfNotExist as the creation is automatically since #7326 (v8.7.0). Fix inaccuracy nginx service metrics. Fix/Change Windows metrics name(Swap -\u0026gt; Virtual Memory) memory_swap_free -\u0026gt; memory_virtual_memory_free memory_swap_total -\u0026gt; memory_virtual_memory_total memory_swap_percentage -\u0026gt; memory_virtual_memory_percentage Fix/Change UI init setting for Windows Swap -\u0026gt; Virtual Memory Fix Memory Swap Usage/Virtual Memory Usage display with UI init.(Linux/Windows) Fix inaccurate APISIX metrics. Fix inaccurate MongoDB Metrics. Support Apache ActiveMQ server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: ACTIVEMQ -\u0026gt; K8S_SERVICE Calculate Nginx service HTTP Latency by MQE. MQE query: make metadata not return null. MQE labeled metrics Binary Operation: return empty value if the labels not match rather than report error. Fix inaccurate Hierarchy of RabbitMQ Server monitoring metrics. Fix inaccurate MySQL/MariaDB, Redis, PostgreSQL metrics. Support DoubleValue,IntValue,BoolValue in OTEL metrics attributes. [Break Change] gGRPC metrics exporter unified the metric value type and support labeled metrics. Add component definition(ID=152) for c3p0(JDBC3 Connection and Statement Pooling). Fix MQE top_n global query. Fix inaccurate Pulsar and Bookkeeper metrics. MQE support sort_values and sort_label_values functions. UI Fix the mismatch between the unit and calculation of the \u0026ldquo;Network Bandwidth Usage\u0026rdquo; widget in Linux-Service Dashboard. Add theme change animation. Implement the Service and Instance hierarchy topology. Support Tabs in the widget visible when MQE expressions. Support search on Marketplace. Fix default route. Fix layout on the Log widget. Fix Trace associates with Log widget. Add isDefault to the dashboard configuration. Add expressions to dashboard configurations on the dashboard list page. Update Kubernetes related UI templates for adapt data from eBPF access log. Fix dashboard K8S-Service-Root metrics expression. Add dashboards for Service/Instance Hierarchy. Fix MQE in dashboards when using Card widget. Optimize tooltips style. Fix resizing window causes the trace graph to display incorrectly. Add the not found page(404). Enhance VNode logic and support multiple Trace IDs in span\u0026rsquo;s ref. Add the layers filed and associate layers dashboards for the service topology nodes. Fix Nginx-Instance metrics to instance level. Update tabs of the Kubernetes service page. Add Airflow menu i18n. Add Support for dragging in the trace panel. Add workflow icon. Metrics support multiple labels. Support the SINGLE_VALUE for table widgets. Remove the General metric mode and related logical code. Remove metrics for unreal nodes in the topology. Enhance the Trace widget for batch consuming spans. Clean the unused elements in the UI-templates. Documentation Update the release doc to remove the announcement as the tests are through e2e rather than manually. Update the release notification mail a little. Polish docs structure. Move customization docs separately from the introduction docs. Add webhook/gRPC hooks settings example for backend-alarm.md. Begin the process of SWIP - SkyWalking Improvement Proposal. Add SWIP-1 Create and detect Service Hierarchy Relationship. Add SWIP-2 Collecting and Gathering Kubernetes Monitoring Data. Update the Overview docs to add the Service Hierarchy Relationship section. Fix incorrect words for backend-bookkeeper-monitoring.md and backend-pulsar-monitoring.md Document a new way to load balance OAP. Add SWIP-3 Support RocketMQ monitoring. Add OpenTelemetry SkyWalking Exporter deprecated warning doc. Update i18n for rocketmq monitoring. Fix: remove click event after unmounted. Fix: end loading without query results. Update nanoid version to 3.3.7. Update postcss version to 8.4.33. Fix kafka topic name in exporter doc. Fix query-protocol.md, make it consistent with the GraphQL query protocol. Add SWIP-5 Support ClickHouse Monitoring. Remove OpenTelemetry Exporter support from meter doc, as this has been flagged as unmaintained on OTEL upstream. Add doc of one-line quick start script for different storage types. Add FAQ for Why is Clickhouse or Loki or xxx not supported as a storage option?. Add SWIP-8 Support ActiveMQ Monitoring. Move BanyanDB storage to the recommended storage. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1000\"\u003e10.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Java 21 runtime.\u003c/li\u003e\n\u003cli\u003eSupport oap-java21 image for Java 21 runtime.\u003c/li\u003e\n\u003cli\u003eUpgrade \u003ccode\u003eOTEL …\u003c/code\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.4.0/en/changes/changes-10.0.0/","title":"10.0.0"},{"body":"10.0.0 Project Support Java 21 runtime. Support oap-java21 image for Java 21 runtime. Upgrade OTEL collector version to 0.92.0 in all e2e tests. Switch CI macOS runner to m1. Upgrade PostgreSQL driver to 42.4.4 to fix CVE-2024-1597. Remove CLI(swctl) from the image. Remove CLI_VERSION variable from Makefile build. Add BanyanDB to docker-compose quickstart. Bump up Armeria, jackson, netty, jetcd and grpc to fix CVEs. Bump up BanyanDB Java Client to 0.6.0. OAP Server Add layer parameter to the global topology graphQL query. Add is_present function in MQE for check if the list metrics has a value or not. Remove unreasonable default configurations for gRPC thread executor. Remove gRPCThreadPoolQueueSize (SW_RECEIVER_GRPC_POOL_QUEUE_SIZE) configuration. Allow excluding ServiceEntries in some namespaces when looking up ServiceEntries as a final resolution method of service metadata. Set up the length of source and dest IDs in relation entities of service, instance, endpoint, and process to 250(was 200). Support build Service/Instance Hierarchy and query. Change the string field in Elasticsearch storage from keyword type to text type if it set more than 32766 length. [Break Change] Change the configuration field of ui_template and ui_menu in Elasticsearch storage from keyword type to text. Support Service Hierarchy auto matching, add auto matching layer relationships (upper -\u0026gt; lower) as following: MESH -\u0026gt; MESH_DP MESH -\u0026gt; K8S_SERVICE MESH_DP -\u0026gt; K8S_SERVICE GENERAL -\u0026gt; K8S_SERVICE Add namespace suffix for K8S_SERVICE_NAME_RULE/ISTIO_SERVICE_NAME_RULE and metadata-service-mapping.yaml as default. Allow using a dedicated port for ALS receiver. Fix log query by traceId in JDBCLogQueryDAO. Support handler eBPF access log protocol. Fix SumPerMinFunctionTest error function. Remove unnecessary annotations and functions from Meter Functions. Add max and min functions for MAL down sampling. Fix critical bug of uncontrolled memory cost of TopN statistics. Change topN group key from StorageId to entityId + timeBucket. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: MYSQL -\u0026gt; K8S_SERVICE POSTGRESQL -\u0026gt; K8S_SERVICE SO11Y_OAP -\u0026gt; K8S_SERVICE VIRTUAL_DATABASE -\u0026gt; MYSQL VIRTUAL_DATABASE -\u0026gt; POSTGRESQL Add Golang as a supported language for AMQP. Support available layers of service in the topology. Add count aggregation function for MAL Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: NGINX -\u0026gt; K8S_SERVICE APISIX -\u0026gt; K8S_SERVICE GENERAL -\u0026gt; APISIX Add Golang as a supported language for RocketMQ. Support Apache RocketMQ server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: ROCKETMQ -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; ROCKETMQ Fix ServiceInstance in query. Mock /api/v1/status/buildinfo for PromQL API. Fix table exists check in the JDBC Storage Plugin. Fix day-based table rolling time range strategy in JDBC storage. Add maxInboundMessageSize (SW_DCS_MAX_INBOUND_MESSAGE_SIZE) configuration to change the max inbound message size of DCS. Fix Service Layer when building Events in the EventHookCallback. Add Golang as a supported language for Pulsar. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: RABBITMQ -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; RABBITMQ Remove Column#function mechanism in the kernel. Make query readMetricValue always return the average value of the duration. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: KAFKA -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; KAFKA Support ClickHouse server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: CLICKHOUSE -\u0026gt; K8S_SERVICE VIRTUAL_DATABASE -\u0026gt; CLICKHOUSE Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: PULSAR -\u0026gt; K8S_SERVICE VIRTUAL_MQ -\u0026gt; PULSAR Add Golang as a supported language for Kafka. Support displaying the port services listen to from OAP and UI during server start. Refactor data-generator to support generating metrics. Fix AvgHistogramPercentileFunction legacy name. [Break Change] Labeled Metrics support multiple labels. Storage: store all label names and values instead of only the values. MQE: Support querying by multiple labels(name and value) instead using _ as the anonymous label name. aggregate_labels function support aggregate by specific labels. relabels function require target label and rename label name and value. PromQL: Support querying by multiple labels(name and value) instead using lables as the anonymous label name. Remove general labels labels/relabels/label function. API /api/v1/labels and /api/v1/label/\u0026lt;label_name\u0026gt;/values support return matched metrics labels. OAL: Deprecate percentile function and introduce percentile2 function instead. Bump up Kafka to fix CVE. Fix NullPointerException in Istio ServiceEntry registry. Remove unnecessary componentIds as series ID in the ServiceRelationClientSideMetrics and ServiceRelationServerSideMetrics entities. Fix not throw error when part of expression not matched any expression node in the MQE and `PromQL. Remove kafka-fetcher/default/createTopicIfNotExist as the creation is automatically since #7326 (v8.7.0). Fix inaccuracy nginx service metrics. Fix/Change Windows metrics name(Swap -\u0026gt; Virtual Memory) memory_swap_free -\u0026gt; memory_virtual_memory_free memory_swap_total -\u0026gt; memory_virtual_memory_total memory_swap_percentage -\u0026gt; memory_virtual_memory_percentage Fix/Change UI init setting for Windows Swap -\u0026gt; Virtual Memory Fix Memory Swap Usage/Virtual Memory Usage display with UI init.(Linux/Windows) Fix inaccurate APISIX metrics. Fix inaccurate MongoDB Metrics. Support Apache ActiveMQ server monitoring. Add Service Hierarchy auto matching layer relationships (upper -\u0026gt; lower) as following: ACTIVEMQ -\u0026gt; K8S_SERVICE Calculate Nginx service HTTP Latency by MQE. MQE query: make metadata not return null. MQE labeled metrics Binary Operation: return empty value if the labels not match rather than report error. Fix inaccurate Hierarchy of RabbitMQ Server monitoring metrics. Fix inaccurate MySQL/MariaDB, Redis, PostgreSQL metrics. Support DoubleValue,IntValue,BoolValue in OTEL metrics attributes. [Break Change] gGRPC metrics exporter unified the metric value type and support labeled metrics. Add component definition(ID=152) for c3p0(JDBC3 Connection and Statement Pooling). Fix MQE top_n global query. Fix inaccurate Pulsar and Bookkeeper metrics. MQE support sort_values and sort_label_values functions. UI Fix the mismatch between the unit and calculation of the \u0026ldquo;Network Bandwidth Usage\u0026rdquo; widget in Linux-Service Dashboard. Add theme change animation. Implement the Service and Instance hierarchy topology. Support Tabs in the widget visible when MQE expressions. Support search on Marketplace. Fix default route. Fix layout on the Log widget. Fix Trace associates with Log widget. Add isDefault to the dashboard configuration. Add expressions to dashboard configurations on the dashboard list page. Update Kubernetes related UI templates for adapt data from eBPF access log. Fix dashboard K8S-Service-Root metrics expression. Add dashboards for Service/Instance Hierarchy. Fix MQE in dashboards when using Card widget. Optimize tooltips style. Fix resizing window causes the trace graph to display incorrectly. Add the not found page(404). Enhance VNode logic and support multiple Trace IDs in span\u0026rsquo;s ref. Add the layers filed and associate layers dashboards for the service topology nodes. Fix Nginx-Instance metrics to instance level. Update tabs of the Kubernetes service page. Add Airflow menu i18n. Add Support for dragging in the trace panel. Add workflow icon. Metrics support multiple labels. Support the SINGLE_VALUE for table widgets. Remove the General metric mode and related logical code. Remove metrics for unreal nodes in the topology. Enhance the Trace widget for batch consuming spans. Clean the unused elements in the UI-templates. Documentation Update the release doc to remove the announcement as the tests are through e2e rather than manually. Update the release notification mail a little. Polish docs structure. Move customization docs separately from the introduction docs. Add webhook/gRPC hooks settings example for backend-alarm.md. Begin the process of SWIP - SkyWalking Improvement Proposal. Add SWIP-1 Create and detect Service Hierarchy Relationship. Add SWIP-2 Collecting and Gathering Kubernetes Monitoring Data. Update the Overview docs to add the Service Hierarchy Relationship section. Fix incorrect words for backend-bookkeeper-monitoring.md and backend-pulsar-monitoring.md Document a new way to load balance OAP. Add SWIP-3 Support RocketMQ monitoring. Add OpenTelemetry SkyWalking Exporter deprecated warning doc. Update i18n for rocketmq monitoring. Fix: remove click event after unmounted. Fix: end loading without query results. Update nanoid version to 3.3.7. Update postcss version to 8.4.33. Fix kafka topic name in exporter doc. Fix query-protocol.md, make it consistent with the GraphQL query protocol. Add SWIP-5 Support ClickHouse Monitoring. Remove OpenTelemetry Exporter support from meter doc, as this has been flagged as unmaintained on OTEL upstream. Add doc of one-line quick start script for different storage types. Add FAQ for Why is Clickhouse or Loki or xxx not supported as a storage option?. Add SWIP-8 Support ActiveMQ Monitoring. Move BanyanDB storage to the recommended storage. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1000\"\u003e10.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Java 21 runtime.\u003c/li\u003e\n\u003cli\u003eSupport oap-java21 image for Java 21 runtime.\u003c/li\u003e\n\u003cli\u003eUpgrade \u003ccode\u003eOTEL …\u003c/code\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v11.0.0/en/changes/changes-10.0.0/","title":"10.0.0"},{"body":"10.0.1 Project Add SBOM (Software Bill of Materials) to the project. OAP Server Fix LAL test query api. Add component libraries of Derby/Sybase/SQLite/DB2/OceanBase jdbc driver. Fix setting the wrong interval to day level measure schema in BanyanDB installation process. UI Fix widget title and tips. Fix statistics span data. Fix browser log display. Fix the topology layout for there are multiple independent network relationships. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1001\"\u003e10.0.1\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd SBOM (Software Bill of Materials) to the project.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"oap-server\"\u003eOAP Server\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix LAL test …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/latest/en/changes/changes-10.0.1/","title":"10.0.1"},{"body":"10.0.1 Project Add SBOM (Software Bill of Materials) to the project. OAP Server Fix LAL test query api. Add component libraries of Derby/Sybase/SQLite/DB2/OceanBase jdbc driver. Fix setting the wrong interval to day level measure schema in BanyanDB installation process. UI Fix widget title and tips. Fix statistics span data. Fix browser log display. Fix the topology layout for there are multiple independent network relationships. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1001\"\u003e10.0.1\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd SBOM (Software Bill of Materials) to the project.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"oap-server\"\u003eOAP Server\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix LAL test …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/next/en/changes/changes-10.0.1/","title":"10.0.1"},{"body":"10.0.1 Project Add SBOM (Software Bill of Materials) to the project. OAP Server Fix LAL test query api. Add component libraries of Derby/Sybase/SQLite/DB2/OceanBase jdbc driver. Fix setting the wrong interval to day level measure schema in BanyanDB installation process. UI Fix widget title and tips. Fix statistics span data. Fix browser log display. Fix the topology layout for there are multiple independent network relationships. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1001\"\u003e10.0.1\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd SBOM (Software Bill of Materials) to the project.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"oap-server\"\u003eOAP Server\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix LAL test …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.0.0/en/changes/changes-10.0.1/","title":"10.0.1"},{"body":"10.0.1 Project Add SBOM (Software Bill of Materials) to the project. OAP Server Fix LAL test query api. Add component libraries of Derby/Sybase/SQLite/DB2/OceanBase jdbc driver. Fix setting the wrong interval to day level measure schema in BanyanDB installation process. UI Fix widget title and tips. Fix statistics span data. Fix browser log display. Fix the topology layout for there are multiple independent network relationships. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1001\"\u003e10.0.1\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd SBOM (Software Bill of Materials) to the project.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"oap-server\"\u003eOAP Server\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix LAL test …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.0.1/en/changes/changes-10.0.1/","title":"10.0.1"},{"body":"10.0.1 Project Add SBOM (Software Bill of Materials) to the project. OAP Server Fix LAL test query api. Add component libraries of Derby/Sybase/SQLite/DB2/OceanBase jdbc driver. Fix setting the wrong interval to day level measure schema in BanyanDB installation process. UI Fix widget title and tips. Fix statistics span data. Fix browser log display. Fix the topology layout for there are multiple independent network relationships. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1001\"\u003e10.0.1\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd SBOM (Software Bill of Materials) to the project.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"oap-server\"\u003eOAP Server\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix LAL test …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.1.0/en/changes/changes-10.0.1/","title":"10.0.1"},{"body":"10.0.1 Project Add SBOM (Software Bill of Materials) to the project. OAP Server Fix LAL test query api. Add component libraries of Derby/Sybase/SQLite/DB2/OceanBase jdbc driver. Fix setting the wrong interval to day level measure schema in BanyanDB installation process. UI Fix widget title and tips. Fix statistics span data. Fix browser log display. Fix the topology layout for there are multiple independent network relationships. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1001\"\u003e10.0.1\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd SBOM (Software Bill of Materials) to the project.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"oap-server\"\u003eOAP Server\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix LAL test …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.2.0/en/changes/changes-10.0.1/","title":"10.0.1"},{"body":"10.0.1 Project Add SBOM (Software Bill of Materials) to the project. OAP Server Fix LAL test query api. Add component libraries of Derby/Sybase/SQLite/DB2/OceanBase jdbc driver. Fix setting the wrong interval to day level measure schema in BanyanDB installation process. UI Fix widget title and tips. Fix statistics span data. Fix browser log display. Fix the topology layout for there are multiple independent network relationships. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1001\"\u003e10.0.1\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd SBOM (Software Bill of Materials) to the project.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"oap-server\"\u003eOAP Server\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix LAL test …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.3.0/en/changes/changes-10.0.1/","title":"10.0.1"},{"body":"10.0.1 Project Add SBOM (Software Bill of Materials) to the project. OAP Server Fix LAL test query api. Add component libraries of Derby/Sybase/SQLite/DB2/OceanBase jdbc driver. Fix setting the wrong interval to day level measure schema in BanyanDB installation process. UI Fix widget title and tips. Fix statistics span data. Fix browser log display. Fix the topology layout for there are multiple independent network relationships. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1001\"\u003e10.0.1\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd SBOM (Software Bill of Materials) to the project.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"oap-server\"\u003eOAP Server\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix LAL test …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.4.0/en/changes/changes-10.0.1/","title":"10.0.1"},{"body":"10.0.1 Project Add SBOM (Software Bill of Materials) to the project. OAP Server Fix LAL test query api. Add component libraries of Derby/Sybase/SQLite/DB2/OceanBase jdbc driver. Fix setting the wrong interval to day level measure schema in BanyanDB installation process. UI Fix widget title and tips. Fix statistics span data. Fix browser log display. Fix the topology layout for there are multiple independent network relationships. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1001\"\u003e10.0.1\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd SBOM (Software Bill of Materials) to the project.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"oap-server\"\u003eOAP Server\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix LAL test …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v11.0.0/en/changes/changes-10.0.1/","title":"10.0.1"},{"body":"10.1.0 A Version of PERFORMANCE Huge UI Performance Improvement. Metrics widgets queries are bundled by leveraging the GraphQL capabilities. Parallel Queries Support in GraphQL engine. Improve query performance. Significantly improve the performance of OTEL metrics handler. Reduce CPU and GC costs in OTEL metrics processes. With adopting BanyanDB 0.7, native database performance and stability are improved. Project E2E: bump up the version of the opentelemetry-collector to 0.102.1. Push snapshot data-generator docker image to ghcr.io. Bump up skywalking-infra-e2e to work around GHA removing docker-compose v1. Bump up CodeQL GitHub Actions. Fix wrong phase of delombok plugin to reduce build warnings. Use ci-friendly revision to set the project version. OAP Server Fix wrong indices in the eBPF Profiling related models. Support exclude the specific namespaces traffic in the eBPF Access Log receiver. Add Golang as a supported language for Elasticsearch. Remove unnecessary BanyanDB flushing logs(info). Increase SW_CORE_GRPC_MAX_MESSAGE_SIZE to 50MB. Support to query relation metrics through PromQL. Support trace MQE query for debugging. Add Component ID(158) for the Solon framework. Fix metrics tag in HTTP handler of browser receiver plugin. Increase alarm_record#message column length to 2000 from 200. Remove alarm_record#message column indexing. Add Python as a supported language for Pulsar. Make more proper histogram buckets for the persistence_timer_bulk_prepare_latency, persistence_timer_bulk_execute_latency and persistence_timer_bulk_all_latency metrics in PersistenceTimer. [Break Change] Update Nacos version to 2.3.2. Nacos 1.x server can\u0026rsquo;t serve as cluster coordinator and configuration server. Support tracing trace query(SkyWalking and Zipkin) for debugging. Fix BanyanDB metrics query: used the wrong Downsampling type to find the schema. Support fetch cilium flow to monitoring network traffic between cilium services. Support labelCount function in the OAL engine. Support BanyanDB internal measure query execution tracing. BanyanDB client config: rise the default maxBulkSize to 10000, add flushTimeout and set default to 10s. Polish BanyanDB group and schema creation logic to fix the schema creation failure issue in distributed race conditions. Support tracing topology query for debugging. Fix expression of graph Current QPS in MySQL dashboard. Support tracing logs query for debugging. BanyanDB: fix Tag autocomplete data storage and query. Support aggregation operators in PromQL query. Update the kubernetes HTTP latency related metrics source unit from ns to ms. Support BanyanDB internal stream query execution tracing. Fix Elasticsearch, MySQL, RabbitMQ dashboards typos and missing expressions. BanyanDB: Zipkin Module set service as Entity for improving the query performance. MQE: check the metrics value before do binary operation to improve robustness. Replace workaround with Armeria native supported context path. Add an http endpoint wrapper for health check. Bump up Armeria and transitive dependencies. BanyanDB: if the model column is already a @BanyanDB.TimestampColumn, set @BanyanDB.NoIndexing on it to reduce indexes. BanyanDB: stream sort-by time query, use internal time-series rather than index to improve the query performance. Bump up graphql-java to 21.5. Add Unknown Node when receive Kubernetes peer address is not aware in current cluster. Fix CounterWindow concurrent increase cause NPE by PriorityQueue Fix format the endpoint name with empty string. Support async query for the composite GraphQL query. Get endpoint list order by timestamp desc. Support sort queries on metrics generated by eBPF receiver. Fix the compatibility with Grafana 11 when using label_values query variables. Nacos as config server and cluster coordinator supports configuration contextPath. Update the endpoint name format to \u0026lt;Method\u0026gt;:\u0026lt;Path\u0026gt; in eBPF Access Log Receiver. Add self-observability metrics for OpenTelemetry receiver. Support service level metrics aggregate when missing pod context in eBPF Access Log Receiver. Fix query getGlobalTopology throw exception when didn\u0026rsquo;t find any services by the given Layer. Fix the previous analysis result missing in the ALS k8s-mesh analyzer. Fix findEndpoint query requires keyword when using BanyanDB. Support to analysis the ztunnel mapped IP address and mTLS mode in eBPF Access Log Receiver. Adapt BanyanDB Java Client 0.7.0. Add SkyWalking Java Agent self observability dashboard. Add Component ID(5022) for the GoFrame framework. Bump up protobuf java dependencies to 3.25.5. BanyanDB: support using native term searching for keyword in query findEndpoint and getAlarm. BanyanDB: support TLS connection and configuration. PromQL service: query API support RFC3399 time format. Improve the performance of OTEL metrics handler. PromQL service: fix operators result missing rangeExpression flag. BanyanDB: use TimestampRange to improve \u0026ldquo;events\u0026rdquo; query for BanyanDB. Optimize network_address_alias table to reduce the number of the index. PromQL service: support round brackets operator. Support query Alarm message Tag for auto-complete. Add SkyWalking Go Agent self observability dashboard. UI Highlight search log keywords. Add Error URL in the browser log. Add a SolonMVC icon. Adding cilium icon and i18n for menu. Fix the mismatch between the unit and calculation of the \u0026ldquo;Network Bandwidth Usage\u0026rdquo; widget in Windows-Service Dashboard. Make a maximum 20 entities per query in service/instance/endpoint list widgets. Polish error nodes in trace widget. Introduce flame graph to the trace profiling. Correct services and instances when changing page numbers. Improve metric queries to make page opening brisker. Bump up dependencies to fix CVEs. Add a loading view for initialization page. Fix a bug for selectors when clicking the refresh icon. Fix health check to OAP backend. Add Service, ServiceInstance, Endpoint dashboard forwarder to Kubernetes Topologies. Fix pagination for service/instance list widgets. Add queries for alarm tags. Add skywalking java agent self observability menu. Documentation Update the version description supported by zabbix receiver. Move the Official Dashboard docs to marketplace docs. Add marketplace introduction docs under quick start menu to reduce the confusion of finding feature docs. Update Windows Metrics(Swap -\u0026gt; Virtual Memory) All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1010\"\u003e10.1.0\u003c/h2\u003e\n\u003ch4 id=\"a-version-of-performance\"\u003eA Version of PERFORMANCE\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eHuge UI Performance Improvement. Metrics widgets queries are …\u003c/strong\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/latest/en/changes/changes-10.1.0/","title":"10.1.0"},{"body":"10.1.0 A Version of PERFORMANCE Huge UI Performance Improvement. Metrics widgets queries are bundled by leveraging the GraphQL capabilities. Parallel Queries Support in GraphQL engine. Improve query performance. Significantly improve the performance of OTEL metrics handler. Reduce CPU and GC costs in OTEL metrics processes. With adopting BanyanDB 0.7, native database performance and stability are improved. Project E2E: bump up the version of the opentelemetry-collector to 0.102.1. Push snapshot data-generator docker image to ghcr.io. Bump up skywalking-infra-e2e to work around GHA removing docker-compose v1. Bump up CodeQL GitHub Actions. Fix wrong phase of delombok plugin to reduce build warnings. Use ci-friendly revision to set the project version. OAP Server Fix wrong indices in the eBPF Profiling related models. Support exclude the specific namespaces traffic in the eBPF Access Log receiver. Add Golang as a supported language for Elasticsearch. Remove unnecessary BanyanDB flushing logs(info). Increase SW_CORE_GRPC_MAX_MESSAGE_SIZE to 50MB. Support to query relation metrics through PromQL. Support trace MQE query for debugging. Add Component ID(158) for the Solon framework. Fix metrics tag in HTTP handler of browser receiver plugin. Increase alarm_record#message column length to 2000 from 200. Remove alarm_record#message column indexing. Add Python as a supported language for Pulsar. Make more proper histogram buckets for the persistence_timer_bulk_prepare_latency, persistence_timer_bulk_execute_latency and persistence_timer_bulk_all_latency metrics in PersistenceTimer. [Break Change] Update Nacos version to 2.3.2. Nacos 1.x server can\u0026rsquo;t serve as cluster coordinator and configuration server. Support tracing trace query(SkyWalking and Zipkin) for debugging. Fix BanyanDB metrics query: used the wrong Downsampling type to find the schema. Support fetch cilium flow to monitoring network traffic between cilium services. Support labelCount function in the OAL engine. Support BanyanDB internal measure query execution tracing. BanyanDB client config: rise the default maxBulkSize to 10000, add flushTimeout and set default to 10s. Polish BanyanDB group and schema creation logic to fix the schema creation failure issue in distributed race conditions. Support tracing topology query for debugging. Fix expression of graph Current QPS in MySQL dashboard. Support tracing logs query for debugging. BanyanDB: fix Tag autocomplete data storage and query. Support aggregation operators in PromQL query. Update the kubernetes HTTP latency related metrics source unit from ns to ms. Support BanyanDB internal stream query execution tracing. Fix Elasticsearch, MySQL, RabbitMQ dashboards typos and missing expressions. BanyanDB: Zipkin Module set service as Entity for improving the query performance. MQE: check the metrics value before do binary operation to improve robustness. Replace workaround with Armeria native supported context path. Add an http endpoint wrapper for health check. Bump up Armeria and transitive dependencies. BanyanDB: if the model column is already a @BanyanDB.TimestampColumn, set @BanyanDB.NoIndexing on it to reduce indexes. BanyanDB: stream sort-by time query, use internal time-series rather than index to improve the query performance. Bump up graphql-java to 21.5. Add Unknown Node when receive Kubernetes peer address is not aware in current cluster. Fix CounterWindow concurrent increase cause NPE by PriorityQueue Fix format the endpoint name with empty string. Support async query for the composite GraphQL query. Get endpoint list order by timestamp desc. Support sort queries on metrics generated by eBPF receiver. Fix the compatibility with Grafana 11 when using label_values query variables. Nacos as config server and cluster coordinator supports configuration contextPath. Update the endpoint name format to \u0026lt;Method\u0026gt;:\u0026lt;Path\u0026gt; in eBPF Access Log Receiver. Add self-observability metrics for OpenTelemetry receiver. Support service level metrics aggregate when missing pod context in eBPF Access Log Receiver. Fix query getGlobalTopology throw exception when didn\u0026rsquo;t find any services by the given Layer. Fix the previous analysis result missing in the ALS k8s-mesh analyzer. Fix findEndpoint query requires keyword when using BanyanDB. Support to analysis the ztunnel mapped IP address and mTLS mode in eBPF Access Log Receiver. Adapt BanyanDB Java Client 0.7.0. Add SkyWalking Java Agent self observability dashboard. Add Component ID(5022) for the GoFrame framework. Bump up protobuf java dependencies to 3.25.5. BanyanDB: support using native term searching for keyword in query findEndpoint and getAlarm. BanyanDB: support TLS connection and configuration. PromQL service: query API support RFC3399 time format. Improve the performance of OTEL metrics handler. PromQL service: fix operators result missing rangeExpression flag. BanyanDB: use TimestampRange to improve \u0026ldquo;events\u0026rdquo; query for BanyanDB. Optimize network_address_alias table to reduce the number of the index. PromQL service: support round brackets operator. Support query Alarm message Tag for auto-complete. Add SkyWalking Go Agent self observability dashboard. UI Highlight search log keywords. Add Error URL in the browser log. Add a SolonMVC icon. Adding cilium icon and i18n for menu. Fix the mismatch between the unit and calculation of the \u0026ldquo;Network Bandwidth Usage\u0026rdquo; widget in Windows-Service Dashboard. Make a maximum 20 entities per query in service/instance/endpoint list widgets. Polish error nodes in trace widget. Introduce flame graph to the trace profiling. Correct services and instances when changing page numbers. Improve metric queries to make page opening brisker. Bump up dependencies to fix CVEs. Add a loading view for initialization page. Fix a bug for selectors when clicking the refresh icon. Fix health check to OAP backend. Add Service, ServiceInstance, Endpoint dashboard forwarder to Kubernetes Topologies. Fix pagination for service/instance list widgets. Add queries for alarm tags. Add skywalking java agent self observability menu. Documentation Update the version description supported by zabbix receiver. Move the Official Dashboard docs to marketplace docs. Add marketplace introduction docs under quick start menu to reduce the confusion of finding feature docs. Update Windows Metrics(Swap -\u0026gt; Virtual Memory) All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1010\"\u003e10.1.0\u003c/h2\u003e\n\u003ch4 id=\"a-version-of-performance\"\u003eA Version of PERFORMANCE\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eHuge UI Performance Improvement. Metrics widgets queries are …\u003c/strong\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/next/en/changes/changes-10.1.0/","title":"10.1.0"},{"body":"10.1.0 Project OAP Server UI Documentation All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1010\"\u003e10.1.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003ch4 id=\"oap-server\"\u003eOAP Server\u003c/h4\u003e\n\u003ch4 id=\"ui\"\u003eUI\u003c/h4\u003e\n\u003ch4 id=\"documentation\"\u003eDocumentation\u003c/h4\u003e\n\u003cp\u003eAll issues and pull requests are \u003ca href=\"https://github.com/apache/skywalking/milestone/205?closed=1\"\u003ehere\u003c/a\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/main/v10.0.0/en/changes/changes/","title":"10.1.0"},{"body":"10.1.0 Project OAP Server UI Documentation All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1010\"\u003e10.1.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003ch4 id=\"oap-server\"\u003eOAP Server\u003c/h4\u003e\n\u003ch4 id=\"ui\"\u003eUI\u003c/h4\u003e\n\u003ch4 id=\"documentation\"\u003eDocumentation\u003c/h4\u003e\n\u003cp\u003eAll issues and pull requests are \u003ca href=\"https://github.com/apache/skywalking/milestone/205?closed=1\"\u003ehere\u003c/a\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/main/v10.0.1/en/changes/changes/","title":"10.1.0"},{"body":"10.1.0 A Version of PERFORMANCE Huge UI Performance Improvement. Metrics widgets queries are bundled by leveraging the GraphQL capabilities. Parallel Queries Support in GraphQL engine. Improve query performance. Significantly improve the performance of OTEL metrics handler. Reduce CPU and GC costs in OTEL metrics processes. With adopting BanyanDB 0.7, native database performance and stability are improved. Project E2E: bump up the version of the opentelemetry-collector to 0.102.1. Push snapshot data-generator docker image to ghcr.io. Bump up skywalking-infra-e2e to work around GHA removing docker-compose v1. Bump up CodeQL GitHub Actions. Fix wrong phase of delombok plugin to reduce build warnings. Use ci-friendly revision to set the project version. OAP Server Fix wrong indices in the eBPF Profiling related models. Support exclude the specific namespaces traffic in the eBPF Access Log receiver. Add Golang as a supported language for Elasticsearch. Remove unnecessary BanyanDB flushing logs(info). Increase SW_CORE_GRPC_MAX_MESSAGE_SIZE to 50MB. Support to query relation metrics through PromQL. Support trace MQE query for debugging. Add Component ID(158) for the Solon framework. Fix metrics tag in HTTP handler of browser receiver plugin. Increase alarm_record#message column length to 2000 from 200. Remove alarm_record#message column indexing. Add Python as a supported language for Pulsar. Make more proper histogram buckets for the persistence_timer_bulk_prepare_latency, persistence_timer_bulk_execute_latency and persistence_timer_bulk_all_latency metrics in PersistenceTimer. [Break Change] Update Nacos version to 2.3.2. Nacos 1.x server can\u0026rsquo;t serve as cluster coordinator and configuration server. Support tracing trace query(SkyWalking and Zipkin) for debugging. Fix BanyanDB metrics query: used the wrong Downsampling type to find the schema. Support fetch cilium flow to monitoring network traffic between cilium services. Support labelCount function in the OAL engine. Support BanyanDB internal measure query execution tracing. BanyanDB client config: rise the default maxBulkSize to 10000, add flushTimeout and set default to 10s. Polish BanyanDB group and schema creation logic to fix the schema creation failure issue in distributed race conditions. Support tracing topology query for debugging. Fix expression of graph Current QPS in MySQL dashboard. Support tracing logs query for debugging. BanyanDB: fix Tag autocomplete data storage and query. Support aggregation operators in PromQL query. Update the kubernetes HTTP latency related metrics source unit from ns to ms. Support BanyanDB internal stream query execution tracing. Fix Elasticsearch, MySQL, RabbitMQ dashboards typos and missing expressions. BanyanDB: Zipkin Module set service as Entity for improving the query performance. MQE: check the metrics value before do binary operation to improve robustness. Replace workaround with Armeria native supported context path. Add an http endpoint wrapper for health check. Bump up Armeria and transitive dependencies. BanyanDB: if the model column is already a @BanyanDB.TimestampColumn, set @BanyanDB.NoIndexing on it to reduce indexes. BanyanDB: stream sort-by time query, use internal time-series rather than index to improve the query performance. Bump up graphql-java to 21.5. Add Unknown Node when receive Kubernetes peer address is not aware in current cluster. Fix CounterWindow concurrent increase cause NPE by PriorityQueue Fix format the endpoint name with empty string. Support async query for the composite GraphQL query. Get endpoint list order by timestamp desc. Support sort queries on metrics generated by eBPF receiver. Fix the compatibility with Grafana 11 when using label_values query variables. Nacos as config server and cluster coordinator supports configuration contextPath. Update the endpoint name format to \u0026lt;Method\u0026gt;:\u0026lt;Path\u0026gt; in eBPF Access Log Receiver. Add self-observability metrics for OpenTelemetry receiver. Support service level metrics aggregate when missing pod context in eBPF Access Log Receiver. Fix query getGlobalTopology throw exception when didn\u0026rsquo;t find any services by the given Layer. Fix the previous analysis result missing in the ALS k8s-mesh analyzer. Fix findEndpoint query requires keyword when using BanyanDB. Support to analysis the ztunnel mapped IP address and mTLS mode in eBPF Access Log Receiver. Adapt BanyanDB Java Client 0.7.0. Add SkyWalking Java Agent self observability dashboard. Add Component ID(5022) for the GoFrame framework. Bump up protobuf java dependencies to 3.25.5. BanyanDB: support using native term searching for keyword in query findEndpoint and getAlarm. BanyanDB: support TLS connection and configuration. PromQL service: query API support RFC3399 time format. Improve the performance of OTEL metrics handler. PromQL service: fix operators result missing rangeExpression flag. BanyanDB: use TimestampRange to improve \u0026ldquo;events\u0026rdquo; query for BanyanDB. Optimize network_address_alias table to reduce the number of the index. PromQL service: support round brackets operator. Support query Alarm message Tag for auto-complete. UI Highlight search log keywords. Add Error URL in the browser log. Add a SolonMVC icon. Adding cilium icon and i18n for menu. Fix the mismatch between the unit and calculation of the \u0026ldquo;Network Bandwidth Usage\u0026rdquo; widget in Windows-Service Dashboard. Make a maximum 20 entities per query in service/instance/endpoint list widgets. Polish error nodes in trace widget. Introduce flame graph to the trace profiling. Correct services and instances when changing page numbers. Improve metric queries to make page opening brisker. Bump up dependencies to fix CVEs. Add a loading view for initialization page. Fix a bug for selectors when clicking the refresh icon. Fix health check to OAP backend. Add Service, ServiceInstance, Endpoint dashboard forwarder to Kubernetes Topologies. Fix pagination for service/instance list widgets. Add queries for alarm tags. Add skywalking java agent self observability menu. Documentation Update the version description supported by zabbix receiver. Move the Official Dashboard docs to marketplace docs. Add marketplace introduction docs under quick start menu to reduce the confusion of finding feature docs. Update Windows Metrics(Swap -\u0026gt; Virtual Memory) All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1010\"\u003e10.1.0\u003c/h2\u003e\n\u003ch4 id=\"a-version-of-performance\"\u003eA Version of PERFORMANCE\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eHuge UI Performance Improvement. Metrics widgets queries are …\u003c/strong\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.1.0/en/changes/changes/","title":"10.1.0"},{"body":"10.1.0 A Version of PERFORMANCE Huge UI Performance Improvement. Metrics widgets queries are bundled by leveraging the GraphQL capabilities. Parallel Queries Support in GraphQL engine. Improve query performance. Significantly improve the performance of OTEL metrics handler. Reduce CPU and GC costs in OTEL metrics processes. With adopting BanyanDB 0.7, native database performance and stability are improved. Project E2E: bump up the version of the opentelemetry-collector to 0.102.1. Push snapshot data-generator docker image to ghcr.io. Bump up skywalking-infra-e2e to work around GHA removing docker-compose v1. Bump up CodeQL GitHub Actions. Fix wrong phase of delombok plugin to reduce build warnings. Use ci-friendly revision to set the project version. OAP Server Fix wrong indices in the eBPF Profiling related models. Support exclude the specific namespaces traffic in the eBPF Access Log receiver. Add Golang as a supported language for Elasticsearch. Remove unnecessary BanyanDB flushing logs(info). Increase SW_CORE_GRPC_MAX_MESSAGE_SIZE to 50MB. Support to query relation metrics through PromQL. Support trace MQE query for debugging. Add Component ID(158) for the Solon framework. Fix metrics tag in HTTP handler of browser receiver plugin. Increase alarm_record#message column length to 2000 from 200. Remove alarm_record#message column indexing. Add Python as a supported language for Pulsar. Make more proper histogram buckets for the persistence_timer_bulk_prepare_latency, persistence_timer_bulk_execute_latency and persistence_timer_bulk_all_latency metrics in PersistenceTimer. [Break Change] Update Nacos version to 2.3.2. Nacos 1.x server can\u0026rsquo;t serve as cluster coordinator and configuration server. Support tracing trace query(SkyWalking and Zipkin) for debugging. Fix BanyanDB metrics query: used the wrong Downsampling type to find the schema. Support fetch cilium flow to monitoring network traffic between cilium services. Support labelCount function in the OAL engine. Support BanyanDB internal measure query execution tracing. BanyanDB client config: rise the default maxBulkSize to 10000, add flushTimeout and set default to 10s. Polish BanyanDB group and schema creation logic to fix the schema creation failure issue in distributed race conditions. Support tracing topology query for debugging. Fix expression of graph Current QPS in MySQL dashboard. Support tracing logs query for debugging. BanyanDB: fix Tag autocomplete data storage and query. Support aggregation operators in PromQL query. Update the kubernetes HTTP latency related metrics source unit from ns to ms. Support BanyanDB internal stream query execution tracing. Fix Elasticsearch, MySQL, RabbitMQ dashboards typos and missing expressions. BanyanDB: Zipkin Module set service as Entity for improving the query performance. MQE: check the metrics value before do binary operation to improve robustness. Replace workaround with Armeria native supported context path. Add an http endpoint wrapper for health check. Bump up Armeria and transitive dependencies. BanyanDB: if the model column is already a @BanyanDB.TimestampColumn, set @BanyanDB.NoIndexing on it to reduce indexes. BanyanDB: stream sort-by time query, use internal time-series rather than index to improve the query performance. Bump up graphql-java to 21.5. Add Unknown Node when receive Kubernetes peer address is not aware in current cluster. Fix CounterWindow concurrent increase cause NPE by PriorityQueue Fix format the endpoint name with empty string. Support async query for the composite GraphQL query. Get endpoint list order by timestamp desc. Support sort queries on metrics generated by eBPF receiver. Fix the compatibility with Grafana 11 when using label_values query variables. Nacos as config server and cluster coordinator supports configuration contextPath. Update the endpoint name format to \u0026lt;Method\u0026gt;:\u0026lt;Path\u0026gt; in eBPF Access Log Receiver. Add self-observability metrics for OpenTelemetry receiver. Support service level metrics aggregate when missing pod context in eBPF Access Log Receiver. Fix query getGlobalTopology throw exception when didn\u0026rsquo;t find any services by the given Layer. Fix the previous analysis result missing in the ALS k8s-mesh analyzer. Fix findEndpoint query requires keyword when using BanyanDB. Support to analysis the ztunnel mapped IP address and mTLS mode in eBPF Access Log Receiver. Adapt BanyanDB Java Client 0.7.0. Add SkyWalking Java Agent self observability dashboard. Add Component ID(5022) for the GoFrame framework. Bump up protobuf java dependencies to 3.25.5. BanyanDB: support using native term searching for keyword in query findEndpoint and getAlarm. BanyanDB: support TLS connection and configuration. PromQL service: query API support RFC3399 time format. Improve the performance of OTEL metrics handler. PromQL service: fix operators result missing rangeExpression flag. BanyanDB: use TimestampRange to improve \u0026ldquo;events\u0026rdquo; query for BanyanDB. Optimize network_address_alias table to reduce the number of the index. PromQL service: support round brackets operator. Support query Alarm message Tag for auto-complete. Add SkyWalking Go Agent self observability dashboard. UI Highlight search log keywords. Add Error URL in the browser log. Add a SolonMVC icon. Adding cilium icon and i18n for menu. Fix the mismatch between the unit and calculation of the \u0026ldquo;Network Bandwidth Usage\u0026rdquo; widget in Windows-Service Dashboard. Make a maximum 20 entities per query in service/instance/endpoint list widgets. Polish error nodes in trace widget. Introduce flame graph to the trace profiling. Correct services and instances when changing page numbers. Improve metric queries to make page opening brisker. Bump up dependencies to fix CVEs. Add a loading view for initialization page. Fix a bug for selectors when clicking the refresh icon. Fix health check to OAP backend. Add Service, ServiceInstance, Endpoint dashboard forwarder to Kubernetes Topologies. Fix pagination for service/instance list widgets. Add queries for alarm tags. Add skywalking java agent self observability menu. Documentation Update the version description supported by zabbix receiver. Move the Official Dashboard docs to marketplace docs. Add marketplace introduction docs under quick start menu to reduce the confusion of finding feature docs. Update Windows Metrics(Swap -\u0026gt; Virtual Memory) All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1010\"\u003e10.1.0\u003c/h2\u003e\n\u003ch4 id=\"a-version-of-performance\"\u003eA Version of PERFORMANCE\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eHuge UI Performance Improvement. Metrics widgets queries are …\u003c/strong\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.2.0/en/changes/changes-10.1.0/","title":"10.1.0"},{"body":"10.1.0 A Version of PERFORMANCE Huge UI Performance Improvement. Metrics widgets queries are bundled by leveraging the GraphQL capabilities. Parallel Queries Support in GraphQL engine. Improve query performance. Significantly improve the performance of OTEL metrics handler. Reduce CPU and GC costs in OTEL metrics processes. With adopting BanyanDB 0.7, native database performance and stability are improved. Project E2E: bump up the version of the opentelemetry-collector to 0.102.1. Push snapshot data-generator docker image to ghcr.io. Bump up skywalking-infra-e2e to work around GHA removing docker-compose v1. Bump up CodeQL GitHub Actions. Fix wrong phase of delombok plugin to reduce build warnings. Use ci-friendly revision to set the project version. OAP Server Fix wrong indices in the eBPF Profiling related models. Support exclude the specific namespaces traffic in the eBPF Access Log receiver. Add Golang as a supported language for Elasticsearch. Remove unnecessary BanyanDB flushing logs(info). Increase SW_CORE_GRPC_MAX_MESSAGE_SIZE to 50MB. Support to query relation metrics through PromQL. Support trace MQE query for debugging. Add Component ID(158) for the Solon framework. Fix metrics tag in HTTP handler of browser receiver plugin. Increase alarm_record#message column length to 2000 from 200. Remove alarm_record#message column indexing. Add Python as a supported language for Pulsar. Make more proper histogram buckets for the persistence_timer_bulk_prepare_latency, persistence_timer_bulk_execute_latency and persistence_timer_bulk_all_latency metrics in PersistenceTimer. [Break Change] Update Nacos version to 2.3.2. Nacos 1.x server can\u0026rsquo;t serve as cluster coordinator and configuration server. Support tracing trace query(SkyWalking and Zipkin) for debugging. Fix BanyanDB metrics query: used the wrong Downsampling type to find the schema. Support fetch cilium flow to monitoring network traffic between cilium services. Support labelCount function in the OAL engine. Support BanyanDB internal measure query execution tracing. BanyanDB client config: rise the default maxBulkSize to 10000, add flushTimeout and set default to 10s. Polish BanyanDB group and schema creation logic to fix the schema creation failure issue in distributed race conditions. Support tracing topology query for debugging. Fix expression of graph Current QPS in MySQL dashboard. Support tracing logs query for debugging. BanyanDB: fix Tag autocomplete data storage and query. Support aggregation operators in PromQL query. Update the kubernetes HTTP latency related metrics source unit from ns to ms. Support BanyanDB internal stream query execution tracing. Fix Elasticsearch, MySQL, RabbitMQ dashboards typos and missing expressions. BanyanDB: Zipkin Module set service as Entity for improving the query performance. MQE: check the metrics value before do binary operation to improve robustness. Replace workaround with Armeria native supported context path. Add an http endpoint wrapper for health check. Bump up Armeria and transitive dependencies. BanyanDB: if the model column is already a @BanyanDB.TimestampColumn, set @BanyanDB.NoIndexing on it to reduce indexes. BanyanDB: stream sort-by time query, use internal time-series rather than index to improve the query performance. Bump up graphql-java to 21.5. Add Unknown Node when receive Kubernetes peer address is not aware in current cluster. Fix CounterWindow concurrent increase cause NPE by PriorityQueue Fix format the endpoint name with empty string. Support async query for the composite GraphQL query. Get endpoint list order by timestamp desc. Support sort queries on metrics generated by eBPF receiver. Fix the compatibility with Grafana 11 when using label_values query variables. Nacos as config server and cluster coordinator supports configuration contextPath. Update the endpoint name format to \u0026lt;Method\u0026gt;:\u0026lt;Path\u0026gt; in eBPF Access Log Receiver. Add self-observability metrics for OpenTelemetry receiver. Support service level metrics aggregate when missing pod context in eBPF Access Log Receiver. Fix query getGlobalTopology throw exception when didn\u0026rsquo;t find any services by the given Layer. Fix the previous analysis result missing in the ALS k8s-mesh analyzer. Fix findEndpoint query requires keyword when using BanyanDB. Support to analysis the ztunnel mapped IP address and mTLS mode in eBPF Access Log Receiver. Adapt BanyanDB Java Client 0.7.0. Add SkyWalking Java Agent self observability dashboard. Add Component ID(5022) for the GoFrame framework. Bump up protobuf java dependencies to 3.25.5. BanyanDB: support using native term searching for keyword in query findEndpoint and getAlarm. BanyanDB: support TLS connection and configuration. PromQL service: query API support RFC3399 time format. Improve the performance of OTEL metrics handler. PromQL service: fix operators result missing rangeExpression flag. BanyanDB: use TimestampRange to improve \u0026ldquo;events\u0026rdquo; query for BanyanDB. Optimize network_address_alias table to reduce the number of the index. PromQL service: support round brackets operator. Support query Alarm message Tag for auto-complete. Add SkyWalking Go Agent self observability dashboard. UI Highlight search log keywords. Add Error URL in the browser log. Add a SolonMVC icon. Adding cilium icon and i18n for menu. Fix the mismatch between the unit and calculation of the \u0026ldquo;Network Bandwidth Usage\u0026rdquo; widget in Windows-Service Dashboard. Make a maximum 20 entities per query in service/instance/endpoint list widgets. Polish error nodes in trace widget. Introduce flame graph to the trace profiling. Correct services and instances when changing page numbers. Improve metric queries to make page opening brisker. Bump up dependencies to fix CVEs. Add a loading view for initialization page. Fix a bug for selectors when clicking the refresh icon. Fix health check to OAP backend. Add Service, ServiceInstance, Endpoint dashboard forwarder to Kubernetes Topologies. Fix pagination for service/instance list widgets. Add queries for alarm tags. Add skywalking java agent self observability menu. Documentation Update the version description supported by zabbix receiver. Move the Official Dashboard docs to marketplace docs. Add marketplace introduction docs under quick start menu to reduce the confusion of finding feature docs. Update Windows Metrics(Swap -\u0026gt; Virtual Memory) All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1010\"\u003e10.1.0\u003c/h2\u003e\n\u003ch4 id=\"a-version-of-performance\"\u003eA Version of PERFORMANCE\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eHuge UI Performance Improvement. Metrics widgets queries are …\u003c/strong\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.3.0/en/changes/changes-10.1.0/","title":"10.1.0"},{"body":"10.1.0 A Version of PERFORMANCE Huge UI Performance Improvement. Metrics widgets queries are bundled by leveraging the GraphQL capabilities. Parallel Queries Support in GraphQL engine. Improve query performance. Significantly improve the performance of OTEL metrics handler. Reduce CPU and GC costs in OTEL metrics processes. With adopting BanyanDB 0.7, native database performance and stability are improved. Project E2E: bump up the version of the opentelemetry-collector to 0.102.1. Push snapshot data-generator docker image to ghcr.io. Bump up skywalking-infra-e2e to work around GHA removing docker-compose v1. Bump up CodeQL GitHub Actions. Fix wrong phase of delombok plugin to reduce build warnings. Use ci-friendly revision to set the project version. OAP Server Fix wrong indices in the eBPF Profiling related models. Support exclude the specific namespaces traffic in the eBPF Access Log receiver. Add Golang as a supported language for Elasticsearch. Remove unnecessary BanyanDB flushing logs(info). Increase SW_CORE_GRPC_MAX_MESSAGE_SIZE to 50MB. Support to query relation metrics through PromQL. Support trace MQE query for debugging. Add Component ID(158) for the Solon framework. Fix metrics tag in HTTP handler of browser receiver plugin. Increase alarm_record#message column length to 2000 from 200. Remove alarm_record#message column indexing. Add Python as a supported language for Pulsar. Make more proper histogram buckets for the persistence_timer_bulk_prepare_latency, persistence_timer_bulk_execute_latency and persistence_timer_bulk_all_latency metrics in PersistenceTimer. [Break Change] Update Nacos version to 2.3.2. Nacos 1.x server can\u0026rsquo;t serve as cluster coordinator and configuration server. Support tracing trace query(SkyWalking and Zipkin) for debugging. Fix BanyanDB metrics query: used the wrong Downsampling type to find the schema. Support fetch cilium flow to monitoring network traffic between cilium services. Support labelCount function in the OAL engine. Support BanyanDB internal measure query execution tracing. BanyanDB client config: rise the default maxBulkSize to 10000, add flushTimeout and set default to 10s. Polish BanyanDB group and schema creation logic to fix the schema creation failure issue in distributed race conditions. Support tracing topology query for debugging. Fix expression of graph Current QPS in MySQL dashboard. Support tracing logs query for debugging. BanyanDB: fix Tag autocomplete data storage and query. Support aggregation operators in PromQL query. Update the kubernetes HTTP latency related metrics source unit from ns to ms. Support BanyanDB internal stream query execution tracing. Fix Elasticsearch, MySQL, RabbitMQ dashboards typos and missing expressions. BanyanDB: Zipkin Module set service as Entity for improving the query performance. MQE: check the metrics value before do binary operation to improve robustness. Replace workaround with Armeria native supported context path. Add an http endpoint wrapper for health check. Bump up Armeria and transitive dependencies. BanyanDB: if the model column is already a @BanyanDB.TimestampColumn, set @BanyanDB.NoIndexing on it to reduce indexes. BanyanDB: stream sort-by time query, use internal time-series rather than index to improve the query performance. Bump up graphql-java to 21.5. Add Unknown Node when receive Kubernetes peer address is not aware in current cluster. Fix CounterWindow concurrent increase cause NPE by PriorityQueue Fix format the endpoint name with empty string. Support async query for the composite GraphQL query. Get endpoint list order by timestamp desc. Support sort queries on metrics generated by eBPF receiver. Fix the compatibility with Grafana 11 when using label_values query variables. Nacos as config server and cluster coordinator supports configuration contextPath. Update the endpoint name format to \u0026lt;Method\u0026gt;:\u0026lt;Path\u0026gt; in eBPF Access Log Receiver. Add self-observability metrics for OpenTelemetry receiver. Support service level metrics aggregate when missing pod context in eBPF Access Log Receiver. Fix query getGlobalTopology throw exception when didn\u0026rsquo;t find any services by the given Layer. Fix the previous analysis result missing in the ALS k8s-mesh analyzer. Fix findEndpoint query requires keyword when using BanyanDB. Support to analysis the ztunnel mapped IP address and mTLS mode in eBPF Access Log Receiver. Adapt BanyanDB Java Client 0.7.0. Add SkyWalking Java Agent self observability dashboard. Add Component ID(5022) for the GoFrame framework. Bump up protobuf java dependencies to 3.25.5. BanyanDB: support using native term searching for keyword in query findEndpoint and getAlarm. BanyanDB: support TLS connection and configuration. PromQL service: query API support RFC3399 time format. Improve the performance of OTEL metrics handler. PromQL service: fix operators result missing rangeExpression flag. BanyanDB: use TimestampRange to improve \u0026ldquo;events\u0026rdquo; query for BanyanDB. Optimize network_address_alias table to reduce the number of the index. PromQL service: support round brackets operator. Support query Alarm message Tag for auto-complete. Add SkyWalking Go Agent self observability dashboard. UI Highlight search log keywords. Add Error URL in the browser log. Add a SolonMVC icon. Adding cilium icon and i18n for menu. Fix the mismatch between the unit and calculation of the \u0026ldquo;Network Bandwidth Usage\u0026rdquo; widget in Windows-Service Dashboard. Make a maximum 20 entities per query in service/instance/endpoint list widgets. Polish error nodes in trace widget. Introduce flame graph to the trace profiling. Correct services and instances when changing page numbers. Improve metric queries to make page opening brisker. Bump up dependencies to fix CVEs. Add a loading view for initialization page. Fix a bug for selectors when clicking the refresh icon. Fix health check to OAP backend. Add Service, ServiceInstance, Endpoint dashboard forwarder to Kubernetes Topologies. Fix pagination for service/instance list widgets. Add queries for alarm tags. Add skywalking java agent self observability menu. Documentation Update the version description supported by zabbix receiver. Move the Official Dashboard docs to marketplace docs. Add marketplace introduction docs under quick start menu to reduce the confusion of finding feature docs. Update Windows Metrics(Swap -\u0026gt; Virtual Memory) All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1010\"\u003e10.1.0\u003c/h2\u003e\n\u003ch4 id=\"a-version-of-performance\"\u003eA Version of PERFORMANCE\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eHuge UI Performance Improvement. Metrics widgets queries are …\u003c/strong\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.4.0/en/changes/changes-10.1.0/","title":"10.1.0"},{"body":"10.1.0 A Version of PERFORMANCE Huge UI Performance Improvement. Metrics widgets queries are bundled by leveraging the GraphQL capabilities. Parallel Queries Support in GraphQL engine. Improve query performance. Significantly improve the performance of OTEL metrics handler. Reduce CPU and GC costs in OTEL metrics processes. With adopting BanyanDB 0.7, native database performance and stability are improved. Project E2E: bump up the version of the opentelemetry-collector to 0.102.1. Push snapshot data-generator docker image to ghcr.io. Bump up skywalking-infra-e2e to work around GHA removing docker-compose v1. Bump up CodeQL GitHub Actions. Fix wrong phase of delombok plugin to reduce build warnings. Use ci-friendly revision to set the project version. OAP Server Fix wrong indices in the eBPF Profiling related models. Support exclude the specific namespaces traffic in the eBPF Access Log receiver. Add Golang as a supported language for Elasticsearch. Remove unnecessary BanyanDB flushing logs(info). Increase SW_CORE_GRPC_MAX_MESSAGE_SIZE to 50MB. Support to query relation metrics through PromQL. Support trace MQE query for debugging. Add Component ID(158) for the Solon framework. Fix metrics tag in HTTP handler of browser receiver plugin. Increase alarm_record#message column length to 2000 from 200. Remove alarm_record#message column indexing. Add Python as a supported language for Pulsar. Make more proper histogram buckets for the persistence_timer_bulk_prepare_latency, persistence_timer_bulk_execute_latency and persistence_timer_bulk_all_latency metrics in PersistenceTimer. [Break Change] Update Nacos version to 2.3.2. Nacos 1.x server can\u0026rsquo;t serve as cluster coordinator and configuration server. Support tracing trace query(SkyWalking and Zipkin) for debugging. Fix BanyanDB metrics query: used the wrong Downsampling type to find the schema. Support fetch cilium flow to monitoring network traffic between cilium services. Support labelCount function in the OAL engine. Support BanyanDB internal measure query execution tracing. BanyanDB client config: rise the default maxBulkSize to 10000, add flushTimeout and set default to 10s. Polish BanyanDB group and schema creation logic to fix the schema creation failure issue in distributed race conditions. Support tracing topology query for debugging. Fix expression of graph Current QPS in MySQL dashboard. Support tracing logs query for debugging. BanyanDB: fix Tag autocomplete data storage and query. Support aggregation operators in PromQL query. Update the kubernetes HTTP latency related metrics source unit from ns to ms. Support BanyanDB internal stream query execution tracing. Fix Elasticsearch, MySQL, RabbitMQ dashboards typos and missing expressions. BanyanDB: Zipkin Module set service as Entity for improving the query performance. MQE: check the metrics value before do binary operation to improve robustness. Replace workaround with Armeria native supported context path. Add an http endpoint wrapper for health check. Bump up Armeria and transitive dependencies. BanyanDB: if the model column is already a @BanyanDB.TimestampColumn, set @BanyanDB.NoIndexing on it to reduce indexes. BanyanDB: stream sort-by time query, use internal time-series rather than index to improve the query performance. Bump up graphql-java to 21.5. Add Unknown Node when receive Kubernetes peer address is not aware in current cluster. Fix CounterWindow concurrent increase cause NPE by PriorityQueue Fix format the endpoint name with empty string. Support async query for the composite GraphQL query. Get endpoint list order by timestamp desc. Support sort queries on metrics generated by eBPF receiver. Fix the compatibility with Grafana 11 when using label_values query variables. Nacos as config server and cluster coordinator supports configuration contextPath. Update the endpoint name format to \u0026lt;Method\u0026gt;:\u0026lt;Path\u0026gt; in eBPF Access Log Receiver. Add self-observability metrics for OpenTelemetry receiver. Support service level metrics aggregate when missing pod context in eBPF Access Log Receiver. Fix query getGlobalTopology throw exception when didn\u0026rsquo;t find any services by the given Layer. Fix the previous analysis result missing in the ALS k8s-mesh analyzer. Fix findEndpoint query requires keyword when using BanyanDB. Support to analysis the ztunnel mapped IP address and mTLS mode in eBPF Access Log Receiver. Adapt BanyanDB Java Client 0.7.0. Add SkyWalking Java Agent self observability dashboard. Add Component ID(5022) for the GoFrame framework. Bump up protobuf java dependencies to 3.25.5. BanyanDB: support using native term searching for keyword in query findEndpoint and getAlarm. BanyanDB: support TLS connection and configuration. PromQL service: query API support RFC3399 time format. Improve the performance of OTEL metrics handler. PromQL service: fix operators result missing rangeExpression flag. BanyanDB: use TimestampRange to improve \u0026ldquo;events\u0026rdquo; query for BanyanDB. Optimize network_address_alias table to reduce the number of the index. PromQL service: support round brackets operator. Support query Alarm message Tag for auto-complete. Add SkyWalking Go Agent self observability dashboard. UI Highlight search log keywords. Add Error URL in the browser log. Add a SolonMVC icon. Adding cilium icon and i18n for menu. Fix the mismatch between the unit and calculation of the \u0026ldquo;Network Bandwidth Usage\u0026rdquo; widget in Windows-Service Dashboard. Make a maximum 20 entities per query in service/instance/endpoint list widgets. Polish error nodes in trace widget. Introduce flame graph to the trace profiling. Correct services and instances when changing page numbers. Improve metric queries to make page opening brisker. Bump up dependencies to fix CVEs. Add a loading view for initialization page. Fix a bug for selectors when clicking the refresh icon. Fix health check to OAP backend. Add Service, ServiceInstance, Endpoint dashboard forwarder to Kubernetes Topologies. Fix pagination for service/instance list widgets. Add queries for alarm tags. Add skywalking java agent self observability menu. Documentation Update the version description supported by zabbix receiver. Move the Official Dashboard docs to marketplace docs. Add marketplace introduction docs under quick start menu to reduce the confusion of finding feature docs. Update Windows Metrics(Swap -\u0026gt; Virtual Memory) All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1010\"\u003e10.1.0\u003c/h2\u003e\n\u003ch4 id=\"a-version-of-performance\"\u003eA Version of PERFORMANCE\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eHuge UI Performance Improvement. Metrics widgets queries are …\u003c/strong\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v11.0.0/en/changes/changes-10.1.0/","title":"10.1.0"},{"body":"10.2.0 Project Add doc_values for fields that need to be sorted or aggregated in Elasticsearch, and disable all others. This change would not impact the existing deployment and its feature for our official release users. Warning If there are custom query plugins for our Elasticsearch indices, this change could break them as sort queries and aggregation queries which used the unexpected fields are being blocked. [Breaking Change] Rename debugging-query module to status-query module. Relative exposed APIs are UNCHANGED. [Breaking Change] All jars of the skywalking-oap-server are no longer published through maven central. We will only publish the source tar and binary tar to the website download page, and docker images to docker hub. Warning If you are using the skywalking-oap-server as a dependency in your project, you need to download the source tar from the website and publish them to your private maven repository. [Breaking Change] Remove H2 as storage option permanently. BanyanDB 0.8(OAP 10.2 required) is easy, stable and production-ready. Don\u0026rsquo;t need H2 as default storage anymore. [Breaking Change] Bump up BanyanDB server version to 0.8.0. This version is not compatible with the previous versions. Please upgrade the BanyanDB server to 0.8.0 before upgrading OAP to 10.2.0. Bump up nodejs to v22.14.0 for the latest UI(booster-ui) compiling. Migrate tj-actions/changed-files to dorny/paths-filter. OAP Server Skip processing OTLP metrics data points with flag FLAG_NO_RECORDED_VALUE, which causes exceptional result. Add self observability metrics for GraphQL query, graphql_query_latency. Reduce the count of process index and adding time range when query process index. Bump up Apache commons-io to 2.17.0. Polish eBPF so11y metrics and add error count for query metrics. Support query endpoint list with duration parameter(optional). Change the endpoint_traffic to updatable for the additional column last_ping. Add Component ID(5023) for the GoZero framework. Support Kong monitoring. Support adding additional attr[0-5] for service/endpoint level metrics. Support async-profiler feature for performance analysis. Add metrics value owner for metrics topN query result. Add naming control for EndpointDependencyBuilder. The index type BanyanDB.IndexRule.IndexType#TREE is removed. All indices are using IndexType#INVERTED now. Add max query size settings to BanyanDB. Fix \u0026ldquo;BanyanDBTraceQueryDAO.queryBasicTraces\u0026rdquo; doesn\u0026rsquo;t support querying by \u0026ldquo;trace_id\u0026rdquo;. Polish mesh data dispatcher: don\u0026rsquo;t generate Instance/Endpoint metrics if they are empty. Adapt the new metadata standardization in Istio 1.24. Bump up netty to 4.1.115, grpc to 1.68.1, boringssl to 2.0.69. BanyanDB: Support update the Group settings when OAP starting. BanyanDB: Introduce index mode and refactor banyandb group settings. BanyanDB: Introduce the new Progressive TTL feature. BanyanDB: Support update the Schema when OAP starting. BanyanDB: Speed up OAP booting while initializing BanyanDB. BanyanDB: Support @EnableSort on the column to enable sorting for IndexRule and set the default to false. Support Get Effective TTL Configurations API. Fix ServerStatusService.statusWatchers concurrent modification. Add protection for dynamic config change propagate chain. Add Ruby component IDs(HttpClient=2, Redis=7, Memcached=20, Elasticsearch=47, Ruby=12000, Sinatra=12001). Add component ID(160) for Caffeine. Alarm: Support store and query the metrics snapshot when the alarm is triggered. Alarm: Remove unused Alarm Trend query. Fix missing remote endpoint IP address in span query of zipkin query module. Fix hierarchy-definition.yml config file packaged into start.jar wrongly. Add bydb.dependencies.properties config file to define server dependency versions. Fix AvgHistogramPercentileFunction doesn\u0026rsquo;t have proper field definition for ranks. BanyanDB: Support the new Property data module. MQE: Support top_n_of function for merging multiple metrics topn query. Support labelAvg function in the OAL engine. Added maxLabelCount parameter in the labelCount function of OAL to limit the number of labels can be counted. Adapt the new Browser API(/browser/perfData/webVitals, /browser/perfData/webInteractions, /browser/perfData/resources) protocol. Add Circuit Breaking mechanism. BanyanDB: Add support for compatibility checks based on the BanyanDB server\u0026rsquo;s API version. MQE: Support \u0026amp;\u0026amp;(and), ||(or) bool operators. OAP self observability: Add JVM heap and direct memory used metrics. OAP self observability: Add watermark circuit break/recover metrics. AI Pipeline: Support query baseline metrics names and predict metrics value. Add Get Node List in the Cluster API. Add type descriptor when converting Envoy logs to JSON for persistence, to avoid conversion error. Bseline: Support query baseline with MQE and use in the Alarm Rule. Bump up netty to 4.11.118 to fix CVE-2025-24970. Add Get Alarm Runtime Status API. Add lock when query the Alarm metrics window values. Add a fail-safe mechanism to prevent traffic metrics inconsistent between in-memory and database server. Add more clear logs when oap-cluster-internal data(metrics/traffic) format is inconsistent. Optimize metrics cache loading when trace latency greater than cache timeout. Allow calling lang.groovy.GString in DSL. BanyanDB: fix alarm query result without sort. Add a component ID for Virtual thread executor. Add more model installation log info for OAP storage initialization. BanyanDB: Separate the storage configuration to an independent file: bydb.yaml. Bump Armeria to 1.32.0 and some transitive dependencies. Skip persisting metrics/record data that have been expired. Fix the issue of missing Last Ping data. Add HTTP headers configuration for the alarm webhook. Bump up BanyanDB java client to 0.8.0. UI Add support for case-insensitive search in the dashboard list. Add content decorations to Table and Card widgets. Support the endpoint list widget query with duration parameter. Support ranges for Value Mappings. Add service global topN widget on General-Root, Mesh-Root, K8S-Root dashboard. Fix initialization dashboards. Update the Kubernetes metrics for reduce multiple metrics calculate in MQE. Support view data value related dashboards in TopList widgets. Add endpoint global topN widget on General-Root, Mesh-Root. Implement owner option for TopList widgets in related trace options. Hide entrances to unrelated dashboards in topn list. Split topology metric query to avoid exceeding the maximum query complexity. Fix view metrics related trace and metrics query. Add support collapse span. Refactor copy util with Web API. Releases an existing object URL. Optimize Trace Profiling widget. Implement Async Profiling widget. Fix inaccurate data query issue on endpoint topology page. Update browser dashboard for the new metrics. Visualize Snapshot on Alerting page. OAP self observability dashboard: Add JVM heap and direct memory used metrics. OAP self observability dashboard: Add watermark circuit break/recover metrics. Implement the legend selector in metrics charts. Fix repetitive names in router. Bump up dependencies. Fixes tooltips cannot completely display metrics information. Fix time range when generate link. Add skywalking go agent self observability menu. add topN selector for endpoint list. Documentation Update release document to adopt newly added revision-based process. Improve BanyanDB documentation. Improve component-libraries documentation. Improve configuration-vocabulary documentation. Add Get Effective TTL Configurations API documentation. Add Status APIs docs. Simplified the release process with removing maven central publish relative processes. Add Circuit Breaking mechanism doc. Add Get Node List in the Cluster API doc. Remove meter.md doc, because mal.md has covered all the content. Merge browser-http-api-protocol.md doc into browser-protocol.md. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1020\"\u003e10.2.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd \u003ca href=\"https://www.elastic.co/guide/en/elasticsearch/reference/current/doc-values.html\"\u003e\u003ccode\u003edoc_values\u003c/code\u003e\u003c/a\u003e for fields\nthat need to be sorted or aggregated in Elasticsearch, and …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/latest/en/changes/changes-10.2.0/","title":"10.2.0"},{"body":"10.2.0 Project Add doc_values for fields that need to be sorted or aggregated in Elasticsearch, and disable all others. This change would not impact the existing deployment and its feature for our official release users. Warning If there are custom query plugins for our Elasticsearch indices, this change could break them as sort queries and aggregation queries which used the unexpected fields are being blocked. [Breaking Change] Rename debugging-query module to status-query module. Relative exposed APIs are UNCHANGED. [Breaking Change] All jars of the skywalking-oap-server are no longer published through maven central. We will only publish the source tar and binary tar to the website download page, and docker images to docker hub. Warning If you are using the skywalking-oap-server as a dependency in your project, you need to download the source tar from the website and publish them to your private maven repository. [Breaking Change] Remove H2 as storage option permanently. BanyanDB 0.8(OAP 10.2 required) is easy, stable and production-ready. Don\u0026rsquo;t need H2 as default storage anymore. [Breaking Change] Bump up BanyanDB server version to 0.8.0. This version is not compatible with the previous versions. Please upgrade the BanyanDB server to 0.8.0 before upgrading OAP to 10.2.0. Bump up nodejs to v22.14.0 for the latest UI(booster-ui) compiling. Migrate tj-actions/changed-files to dorny/paths-filter. OAP Server Skip processing OTLP metrics data points with flag FLAG_NO_RECORDED_VALUE, which causes exceptional result. Add self observability metrics for GraphQL query, graphql_query_latency. Reduce the count of process index and adding time range when query process index. Bump up Apache commons-io to 2.17.0. Polish eBPF so11y metrics and add error count for query metrics. Support query endpoint list with duration parameter(optional). Change the endpoint_traffic to updatable for the additional column last_ping. Add Component ID(5023) for the GoZero framework. Support Kong monitoring. Support adding additional attr[0-5] for service/endpoint level metrics. Support async-profiler feature for performance analysis. Add metrics value owner for metrics topN query result. Add naming control for EndpointDependencyBuilder. The index type BanyanDB.IndexRule.IndexType#TREE is removed. All indices are using IndexType#INVERTED now. Add max query size settings to BanyanDB. Fix \u0026ldquo;BanyanDBTraceQueryDAO.queryBasicTraces\u0026rdquo; doesn\u0026rsquo;t support querying by \u0026ldquo;trace_id\u0026rdquo;. Polish mesh data dispatcher: don\u0026rsquo;t generate Instance/Endpoint metrics if they are empty. Adapt the new metadata standardization in Istio 1.24. Bump up netty to 4.1.115, grpc to 1.68.1, boringssl to 2.0.69. BanyanDB: Support update the Group settings when OAP starting. BanyanDB: Introduce index mode and refactor banyandb group settings. BanyanDB: Introduce the new Progressive TTL feature. BanyanDB: Support update the Schema when OAP starting. BanyanDB: Speed up OAP booting while initializing BanyanDB. BanyanDB: Support @EnableSort on the column to enable sorting for IndexRule and set the default to false. Support Get Effective TTL Configurations API. Fix ServerStatusService.statusWatchers concurrent modification. Add protection for dynamic config change propagate chain. Add Ruby component IDs(HttpClient=2, Redis=7, Memcached=20, Elasticsearch=47, Ruby=12000, Sinatra=12001). Add component ID(160) for Caffeine. Alarm: Support store and query the metrics snapshot when the alarm is triggered. Alarm: Remove unused Alarm Trend query. Fix missing remote endpoint IP address in span query of zipkin query module. Fix hierarchy-definition.yml config file packaged into start.jar wrongly. Add bydb.dependencies.properties config file to define server dependency versions. Fix AvgHistogramPercentileFunction doesn\u0026rsquo;t have proper field definition for ranks. BanyanDB: Support the new Property data module. MQE: Support top_n_of function for merging multiple metrics topn query. Support labelAvg function in the OAL engine. Added maxLabelCount parameter in the labelCount function of OAL to limit the number of labels can be counted. Adapt the new Browser API(/browser/perfData/webVitals, /browser/perfData/webInteractions, /browser/perfData/resources) protocol. Add Circuit Breaking mechanism. BanyanDB: Add support for compatibility checks based on the BanyanDB server\u0026rsquo;s API version. MQE: Support \u0026amp;\u0026amp;(and), ||(or) bool operators. OAP self observability: Add JVM heap and direct memory used metrics. OAP self observability: Add watermark circuit break/recover metrics. AI Pipeline: Support query baseline metrics names and predict metrics value. Add Get Node List in the Cluster API. Add type descriptor when converting Envoy logs to JSON for persistence, to avoid conversion error. Bseline: Support query baseline with MQE and use in the Alarm Rule. Bump up netty to 4.11.118 to fix CVE-2025-24970. Add Get Alarm Runtime Status API. Add lock when query the Alarm metrics window values. Add a fail-safe mechanism to prevent traffic metrics inconsistent between in-memory and database server. Add more clear logs when oap-cluster-internal data(metrics/traffic) format is inconsistent. Optimize metrics cache loading when trace latency greater than cache timeout. Allow calling lang.groovy.GString in DSL. BanyanDB: fix alarm query result without sort. Add a component ID for Virtual thread executor. Add more model installation log info for OAP storage initialization. BanyanDB: Separate the storage configuration to an independent file: bydb.yaml. Bump Armeria to 1.32.0 and some transitive dependencies. Skip persisting metrics/record data that have been expired. Fix the issue of missing Last Ping data. Add HTTP headers configuration for the alarm webhook. Bump up BanyanDB java client to 0.8.0. UI Add support for case-insensitive search in the dashboard list. Add content decorations to Table and Card widgets. Support the endpoint list widget query with duration parameter. Support ranges for Value Mappings. Add service global topN widget on General-Root, Mesh-Root, K8S-Root dashboard. Fix initialization dashboards. Update the Kubernetes metrics for reduce multiple metrics calculate in MQE. Support view data value related dashboards in TopList widgets. Add endpoint global topN widget on General-Root, Mesh-Root. Implement owner option for TopList widgets in related trace options. Hide entrances to unrelated dashboards in topn list. Split topology metric query to avoid exceeding the maximum query complexity. Fix view metrics related trace and metrics query. Add support collapse span. Refactor copy util with Web API. Releases an existing object URL. Optimize Trace Profiling widget. Implement Async Profiling widget. Fix inaccurate data query issue on endpoint topology page. Update browser dashboard for the new metrics. Visualize Snapshot on Alerting page. OAP self observability dashboard: Add JVM heap and direct memory used metrics. OAP self observability dashboard: Add watermark circuit break/recover metrics. Implement the legend selector in metrics charts. Fix repetitive names in router. Bump up dependencies. Fixes tooltips cannot completely display metrics information. Fix time range when generate link. Add skywalking go agent self observability menu. add topN selector for endpoint list. Documentation Update release document to adopt newly added revision-based process. Improve BanyanDB documentation. Improve component-libraries documentation. Improve configuration-vocabulary documentation. Add Get Effective TTL Configurations API documentation. Add Status APIs docs. Simplified the release process with removing maven central publish relative processes. Add Circuit Breaking mechanism doc. Add Get Node List in the Cluster API doc. Remove meter.md doc, because mal.md has covered all the content. Merge browser-http-api-protocol.md doc into browser-protocol.md. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1020\"\u003e10.2.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd \u003ca href=\"https://www.elastic.co/guide/en/elasticsearch/reference/current/doc-values.html\"\u003e\u003ccode\u003edoc_values\u003c/code\u003e\u003c/a\u003e for fields\nthat need to be sorted or aggregated in Elasticsearch, and …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/next/en/changes/changes-10.2.0/","title":"10.2.0"},{"body":"10.2.0 Project Add doc_values for fields that need to be sorted or aggregated in Elasticsearch, and disable all others. This change would not impact the existing deployment and its feature for our official release users. Warning If there are custom query plugins for our Elasticsearch indices, this change could break them as sort queries and aggregation queries which used the unexpected fields are being blocked. [Breaking Change] Rename debugging-query module to status-query module. Relative exposed APIs are UNCHANGED. [Breaking Change] All jars of the skywalking-oap-server are no longer published through maven central. We will only publish the source tar and binary tar to the website download page, and docker images to docker hub. Warning If you are using the skywalking-oap-server as a dependency in your project, you need to download the source tar from the website and publish them to your private maven repository. [Breaking Change] Remove H2 as storage option permanently. BanyanDB 0.8(OAP 10.2 required) is easy, stable and production-ready. Don\u0026rsquo;t need H2 as default storage anymore. [Breaking Change] Bump up BanyanDB server version to 0.8.0. This version is not compatible with the previous versions. Please upgrade the BanyanDB server to 0.8.0 before upgrading OAP to 10.2.0. Bump up nodejs to v22.14.0 for the latest UI(booster-ui) compiling. Migrate tj-actions/changed-files to dorny/paths-filter. OAP Server Skip processing OTLP metrics data points with flag FLAG_NO_RECORDED_VALUE, which causes exceptional result. Add self observability metrics for GraphQL query, graphql_query_latency. Reduce the count of process index and adding time range when query process index. Bump up Apache commons-io to 2.17.0. Polish eBPF so11y metrics and add error count for query metrics. Support query endpoint list with duration parameter(optional). Change the endpoint_traffic to updatable for the additional column last_ping. Add Component ID(5023) for the GoZero framework. Support Kong monitoring. Support adding additional attr[0-5] for service/endpoint level metrics. Support async-profiler feature for performance analysis. Add metrics value owner for metrics topN query result. Add naming control for EndpointDependencyBuilder. The index type BanyanDB.IndexRule.IndexType#TREE is removed. All indices are using IndexType#INVERTED now. Add max query size settings to BanyanDB. Fix \u0026ldquo;BanyanDBTraceQueryDAO.queryBasicTraces\u0026rdquo; doesn\u0026rsquo;t support querying by \u0026ldquo;trace_id\u0026rdquo;. Polish mesh data dispatcher: don\u0026rsquo;t generate Instance/Endpoint metrics if they are empty. Adapt the new metadata standardization in Istio 1.24. Bump up netty to 4.1.115, grpc to 1.68.1, boringssl to 2.0.69. BanyanDB: Support update the Group settings when OAP starting. BanyanDB: Introduce index mode and refactor banyandb group settings. BanyanDB: Introduce the new Progressive TTL feature. BanyanDB: Support update the Schema when OAP starting. BanyanDB: Speed up OAP booting while initializing BanyanDB. BanyanDB: Support @EnableSort on the column to enable sorting for IndexRule and set the default to false. Support Get Effective TTL Configurations API. Fix ServerStatusService.statusWatchers concurrent modification. Add protection for dynamic config change propagate chain. Add Ruby component IDs(HttpClient=2, Redis=7, Memcached=20, Elasticsearch=47, Ruby=12000, Sinatra=12001). Add component ID(160) for Caffeine. Alarm: Support store and query the metrics snapshot when the alarm is triggered. Alarm: Remove unused Alarm Trend query. Fix missing remote endpoint IP address in span query of zipkin query module. Fix hierarchy-definition.yml config file packaged into start.jar wrongly. Add bydb.dependencies.properties config file to define server dependency versions. Fix AvgHistogramPercentileFunction doesn\u0026rsquo;t have proper field definition for ranks. BanyanDB: Support the new Property data module. MQE: Support top_n_of function for merging multiple metrics topn query. Support labelAvg function in the OAL engine. Added maxLabelCount parameter in the labelCount function of OAL to limit the number of labels can be counted. Adapt the new Browser API(/browser/perfData/webVitals, /browser/perfData/webInteractions, /browser/perfData/resources) protocol. Add Circuit Breaking mechanism. BanyanDB: Add support for compatibility checks based on the BanyanDB server\u0026rsquo;s API version. MQE: Support \u0026amp;\u0026amp;(and), ||(or) bool operators. OAP self observability: Add JVM heap and direct memory used metrics. OAP self observability: Add watermark circuit break/recover metrics. AI Pipeline: Support query baseline metrics names and predict metrics value. Add Get Node List in the Cluster API. Add type descriptor when converting Envoy logs to JSON for persistence, to avoid conversion error. Bseline: Support query baseline with MQE and use in the Alarm Rule. Bump up netty to 4.11.118 to fix CVE-2025-24970. Add Get Alarm Runtime Status API. Add lock when query the Alarm metrics window values. Add a fail-safe mechanism to prevent traffic metrics inconsistent between in-memory and database server. Add more clear logs when oap-cluster-internal data(metrics/traffic) format is inconsistent. Optimize metrics cache loading when trace latency greater than cache timeout. Allow calling lang.groovy.GString in DSL. BanyanDB: fix alarm query result without sort. Add a component ID for Virtual thread executor. Add more model installation log info for OAP storage initialization. BanyanDB: Separate the storage configuration to an independent file: bydb.yaml. Bump Armeria to 1.32.0 and some transitive dependencies. Skip persisting metrics/record data that have been expired. Fix the issue of missing Last Ping data. Add HTTP headers configuration for the alarm webhook. Bump up BanyanDB java client to 0.8.0. UI Add support for case-insensitive search in the dashboard list. Add content decorations to Table and Card widgets. Support the endpoint list widget query with duration parameter. Support ranges for Value Mappings. Add service global topN widget on General-Root, Mesh-Root, K8S-Root dashboard. Fix initialization dashboards. Update the Kubernetes metrics for reduce multiple metrics calculate in MQE. Support view data value related dashboards in TopList widgets. Add endpoint global topN widget on General-Root, Mesh-Root. Implement owner option for TopList widgets in related trace options. Hide entrances to unrelated dashboards in topn list. Split topology metric query to avoid exceeding the maximum query complexity. Fix view metrics related trace and metrics query. Add support collapse span. Refactor copy util with Web API. Releases an existing object URL. Optimize Trace Profiling widget. Implement Async Profiling widget. Fix inaccurate data query issue on endpoint topology page. Update browser dashboard for the new metrics. Visualize Snapshot on Alerting page. OAP self observability dashboard: Add JVM heap and direct memory used metrics. OAP self observability dashboard: Add watermark circuit break/recover metrics. Implement the legend selector in metrics charts. Fix repetitive names in router. Bump up dependencies. Fixes tooltips cannot completely display metrics information. Fix time range when generate link. Add skywalking go agent self observability menu. add topN selector for endpoint list. Documentation Update release document to adopt newly added revision-based process. Improve BanyanDB documentation. Improve component-libraries documentation. Improve configuration-vocabulary documentation. Add Get Effective TTL Configurations API documentation. Add Status APIs docs. Simplified the release process with removing maven central publish relative processes. Add Circuit Breaking mechanism doc. Add Get Node List in the Cluster API doc. Remove meter.md doc, because mal.md has covered all the content. Merge browser-http-api-protocol.md doc into browser-protocol.md. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1020\"\u003e10.2.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd \u003ca href=\"https://www.elastic.co/guide/en/elasticsearch/reference/current/doc-values.html\"\u003e\u003ccode\u003edoc_values\u003c/code\u003e\u003c/a\u003e for fields\nthat need to be sorted or aggregated in Elasticsearch, and …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.2.0/en/changes/changes-10.2.0/","title":"10.2.0"},{"body":"10.2.0 Project Add doc_values for fields that need to be sorted or aggregated in Elasticsearch, and disable all others. This change would not impact the existing deployment and its feature for our official release users. Warning If there are custom query plugins for our Elasticsearch indices, this change could break them as sort queries and aggregation queries which used the unexpected fields are being blocked. [Breaking Change] Rename debugging-query module to status-query module. Relative exposed APIs are UNCHANGED. [Breaking Change] All jars of the skywalking-oap-server are no longer published through maven central. We will only publish the source tar and binary tar to the website download page, and docker images to docker hub. Warning If you are using the skywalking-oap-server as a dependency in your project, you need to download the source tar from the website and publish them to your private maven repository. [Breaking Change] Remove H2 as storage option permanently. BanyanDB 0.8(OAP 10.2 required) is easy, stable and production-ready. Don\u0026rsquo;t need H2 as default storage anymore. [Breaking Change] Bump up BanyanDB server version to 0.8.0. This version is not compatible with the previous versions. Please upgrade the BanyanDB server to 0.8.0 before upgrading OAP to 10.2.0. Bump up nodejs to v22.14.0 for the latest UI(booster-ui) compiling. Migrate tj-actions/changed-files to dorny/paths-filter. OAP Server Skip processing OTLP metrics data points with flag FLAG_NO_RECORDED_VALUE, which causes exceptional result. Add self observability metrics for GraphQL query, graphql_query_latency. Reduce the count of process index and adding time range when query process index. Bump up Apache commons-io to 2.17.0. Polish eBPF so11y metrics and add error count for query metrics. Support query endpoint list with duration parameter(optional). Change the endpoint_traffic to updatable for the additional column last_ping. Add Component ID(5023) for the GoZero framework. Support Kong monitoring. Support adding additional attr[0-5] for service/endpoint level metrics. Support async-profiler feature for performance analysis. Add metrics value owner for metrics topN query result. Add naming control for EndpointDependencyBuilder. The index type BanyanDB.IndexRule.IndexType#TREE is removed. All indices are using IndexType#INVERTED now. Add max query size settings to BanyanDB. Fix \u0026ldquo;BanyanDBTraceQueryDAO.queryBasicTraces\u0026rdquo; doesn\u0026rsquo;t support querying by \u0026ldquo;trace_id\u0026rdquo;. Polish mesh data dispatcher: don\u0026rsquo;t generate Instance/Endpoint metrics if they are empty. Adapt the new metadata standardization in Istio 1.24. Bump up netty to 4.1.115, grpc to 1.68.1, boringssl to 2.0.69. BanyanDB: Support update the Group settings when OAP starting. BanyanDB: Introduce index mode and refactor banyandb group settings. BanyanDB: Introduce the new Progressive TTL feature. BanyanDB: Support update the Schema when OAP starting. BanyanDB: Speed up OAP booting while initializing BanyanDB. BanyanDB: Support @EnableSort on the column to enable sorting for IndexRule and set the default to false. Support Get Effective TTL Configurations API. Fix ServerStatusService.statusWatchers concurrent modification. Add protection for dynamic config change propagate chain. Add Ruby component IDs(HttpClient=2, Redis=7, Memcached=20, Elasticsearch=47, Ruby=12000, Sinatra=12001). Add component ID(160) for Caffeine. Alarm: Support store and query the metrics snapshot when the alarm is triggered. Alarm: Remove unused Alarm Trend query. Fix missing remote endpoint IP address in span query of zipkin query module. Fix hierarchy-definition.yml config file packaged into start.jar wrongly. Add bydb.dependencies.properties config file to define server dependency versions. Fix AvgHistogramPercentileFunction doesn\u0026rsquo;t have proper field definition for ranks. BanyanDB: Support the new Property data module. MQE: Support top_n_of function for merging multiple metrics topn query. Support labelAvg function in the OAL engine. Added maxLabelCount parameter in the labelCount function of OAL to limit the number of labels can be counted. Adapt the new Browser API(/browser/perfData/webVitals, /browser/perfData/webInteractions, /browser/perfData/resources) protocol. Add Circuit Breaking mechanism. BanyanDB: Add support for compatibility checks based on the BanyanDB server\u0026rsquo;s API version. MQE: Support \u0026amp;\u0026amp;(and), ||(or) bool operators. OAP self observability: Add JVM heap and direct memory used metrics. OAP self observability: Add watermark circuit break/recover metrics. AI Pipeline: Support query baseline metrics names and predict metrics value. Add Get Node List in the Cluster API. Add type descriptor when converting Envoy logs to JSON for persistence, to avoid conversion error. Bseline: Support query baseline with MQE and use in the Alarm Rule. Bump up netty to 4.11.118 to fix CVE-2025-24970. Add Get Alarm Runtime Status API. Add lock when query the Alarm metrics window values. Add a fail-safe mechanism to prevent traffic metrics inconsistent between in-memory and database server. Add more clear logs when oap-cluster-internal data(metrics/traffic) format is inconsistent. Optimize metrics cache loading when trace latency greater than cache timeout. Allow calling lang.groovy.GString in DSL. BanyanDB: fix alarm query result without sort. Add a component ID for Virtual thread executor. Add more model installation log info for OAP storage initialization. BanyanDB: Separate the storage configuration to an independent file: bydb.yaml. Bump Armeria to 1.32.0 and some transitive dependencies. Skip persisting metrics/record data that have been expired. Fix the issue of missing Last Ping data. Add HTTP headers configuration for the alarm webhook. Bump up BanyanDB java client to 0.8.0. UI Add support for case-insensitive search in the dashboard list. Add content decorations to Table and Card widgets. Support the endpoint list widget query with duration parameter. Support ranges for Value Mappings. Add service global topN widget on General-Root, Mesh-Root, K8S-Root dashboard. Fix initialization dashboards. Update the Kubernetes metrics for reduce multiple metrics calculate in MQE. Support view data value related dashboards in TopList widgets. Add endpoint global topN widget on General-Root, Mesh-Root. Implement owner option for TopList widgets in related trace options. Hide entrances to unrelated dashboards in topn list. Split topology metric query to avoid exceeding the maximum query complexity. Fix view metrics related trace and metrics query. Add support collapse span. Refactor copy util with Web API. Releases an existing object URL. Optimize Trace Profiling widget. Implement Async Profiling widget. Fix inaccurate data query issue on endpoint topology page. Update browser dashboard for the new metrics. Visualize Snapshot on Alerting page. OAP self observability dashboard: Add JVM heap and direct memory used metrics. OAP self observability dashboard: Add watermark circuit break/recover metrics. Implement the legend selector in metrics charts. Fix repetitive names in router. Bump up dependencies. Fixes tooltips cannot completely display metrics information. Fix time range when generate link. Add skywalking go agent self observability menu. add topN selector for endpoint list. Documentation Update release document to adopt newly added revision-based process. Improve BanyanDB documentation. Improve component-libraries documentation. Improve configuration-vocabulary documentation. Add Get Effective TTL Configurations API documentation. Add Status APIs docs. Simplified the release process with removing maven central publish relative processes. Add Circuit Breaking mechanism doc. Add Get Node List in the Cluster API doc. Remove meter.md doc, because mal.md has covered all the content. Merge browser-http-api-protocol.md doc into browser-protocol.md. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1020\"\u003e10.2.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd \u003ca href=\"https://www.elastic.co/guide/en/elasticsearch/reference/current/doc-values.html\"\u003e\u003ccode\u003edoc_values\u003c/code\u003e\u003c/a\u003e for fields\nthat need to be sorted or aggregated in Elasticsearch, and …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.3.0/en/changes/changes-10.2.0/","title":"10.2.0"},{"body":"10.2.0 Project Add doc_values for fields that need to be sorted or aggregated in Elasticsearch, and disable all others. This change would not impact the existing deployment and its feature for our official release users. Warning If there are custom query plugins for our Elasticsearch indices, this change could break them as sort queries and aggregation queries which used the unexpected fields are being blocked. [Breaking Change] Rename debugging-query module to status-query module. Relative exposed APIs are UNCHANGED. [Breaking Change] All jars of the skywalking-oap-server are no longer published through maven central. We will only publish the source tar and binary tar to the website download page, and docker images to docker hub. Warning If you are using the skywalking-oap-server as a dependency in your project, you need to download the source tar from the website and publish them to your private maven repository. [Breaking Change] Remove H2 as storage option permanently. BanyanDB 0.8(OAP 10.2 required) is easy, stable and production-ready. Don\u0026rsquo;t need H2 as default storage anymore. [Breaking Change] Bump up BanyanDB server version to 0.8.0. This version is not compatible with the previous versions. Please upgrade the BanyanDB server to 0.8.0 before upgrading OAP to 10.2.0. Bump up nodejs to v22.14.0 for the latest UI(booster-ui) compiling. Migrate tj-actions/changed-files to dorny/paths-filter. OAP Server Skip processing OTLP metrics data points with flag FLAG_NO_RECORDED_VALUE, which causes exceptional result. Add self observability metrics for GraphQL query, graphql_query_latency. Reduce the count of process index and adding time range when query process index. Bump up Apache commons-io to 2.17.0. Polish eBPF so11y metrics and add error count for query metrics. Support query endpoint list with duration parameter(optional). Change the endpoint_traffic to updatable for the additional column last_ping. Add Component ID(5023) for the GoZero framework. Support Kong monitoring. Support adding additional attr[0-5] for service/endpoint level metrics. Support async-profiler feature for performance analysis. Add metrics value owner for metrics topN query result. Add naming control for EndpointDependencyBuilder. The index type BanyanDB.IndexRule.IndexType#TREE is removed. All indices are using IndexType#INVERTED now. Add max query size settings to BanyanDB. Fix \u0026ldquo;BanyanDBTraceQueryDAO.queryBasicTraces\u0026rdquo; doesn\u0026rsquo;t support querying by \u0026ldquo;trace_id\u0026rdquo;. Polish mesh data dispatcher: don\u0026rsquo;t generate Instance/Endpoint metrics if they are empty. Adapt the new metadata standardization in Istio 1.24. Bump up netty to 4.1.115, grpc to 1.68.1, boringssl to 2.0.69. BanyanDB: Support update the Group settings when OAP starting. BanyanDB: Introduce index mode and refactor banyandb group settings. BanyanDB: Introduce the new Progressive TTL feature. BanyanDB: Support update the Schema when OAP starting. BanyanDB: Speed up OAP booting while initializing BanyanDB. BanyanDB: Support @EnableSort on the column to enable sorting for IndexRule and set the default to false. Support Get Effective TTL Configurations API. Fix ServerStatusService.statusWatchers concurrent modification. Add protection for dynamic config change propagate chain. Add Ruby component IDs(HttpClient=2, Redis=7, Memcached=20, Elasticsearch=47, Ruby=12000, Sinatra=12001). Add component ID(160) for Caffeine. Alarm: Support store and query the metrics snapshot when the alarm is triggered. Alarm: Remove unused Alarm Trend query. Fix missing remote endpoint IP address in span query of zipkin query module. Fix hierarchy-definition.yml config file packaged into start.jar wrongly. Add bydb.dependencies.properties config file to define server dependency versions. Fix AvgHistogramPercentileFunction doesn\u0026rsquo;t have proper field definition for ranks. BanyanDB: Support the new Property data module. MQE: Support top_n_of function for merging multiple metrics topn query. Support labelAvg function in the OAL engine. Added maxLabelCount parameter in the labelCount function of OAL to limit the number of labels can be counted. Adapt the new Browser API(/browser/perfData/webVitals, /browser/perfData/webInteractions, /browser/perfData/resources) protocol. Add Circuit Breaking mechanism. BanyanDB: Add support for compatibility checks based on the BanyanDB server\u0026rsquo;s API version. MQE: Support \u0026amp;\u0026amp;(and), ||(or) bool operators. OAP self observability: Add JVM heap and direct memory used metrics. OAP self observability: Add watermark circuit break/recover metrics. AI Pipeline: Support query baseline metrics names and predict metrics value. Add Get Node List in the Cluster API. Add type descriptor when converting Envoy logs to JSON for persistence, to avoid conversion error. Bseline: Support query baseline with MQE and use in the Alarm Rule. Bump up netty to 4.11.118 to fix CVE-2025-24970. Add Get Alarm Runtime Status API. Add lock when query the Alarm metrics window values. Add a fail-safe mechanism to prevent traffic metrics inconsistent between in-memory and database server. Add more clear logs when oap-cluster-internal data(metrics/traffic) format is inconsistent. Optimize metrics cache loading when trace latency greater than cache timeout. Allow calling lang.groovy.GString in DSL. BanyanDB: fix alarm query result without sort. Add a component ID for Virtual thread executor. Add more model installation log info for OAP storage initialization. BanyanDB: Separate the storage configuration to an independent file: bydb.yaml. Bump Armeria to 1.32.0 and some transitive dependencies. Skip persisting metrics/record data that have been expired. Fix the issue of missing Last Ping data. Add HTTP headers configuration for the alarm webhook. Bump up BanyanDB java client to 0.8.0. UI Add support for case-insensitive search in the dashboard list. Add content decorations to Table and Card widgets. Support the endpoint list widget query with duration parameter. Support ranges for Value Mappings. Add service global topN widget on General-Root, Mesh-Root, K8S-Root dashboard. Fix initialization dashboards. Update the Kubernetes metrics for reduce multiple metrics calculate in MQE. Support view data value related dashboards in TopList widgets. Add endpoint global topN widget on General-Root, Mesh-Root. Implement owner option for TopList widgets in related trace options. Hide entrances to unrelated dashboards in topn list. Split topology metric query to avoid exceeding the maximum query complexity. Fix view metrics related trace and metrics query. Add support collapse span. Refactor copy util with Web API. Releases an existing object URL. Optimize Trace Profiling widget. Implement Async Profiling widget. Fix inaccurate data query issue on endpoint topology page. Update browser dashboard for the new metrics. Visualize Snapshot on Alerting page. OAP self observability dashboard: Add JVM heap and direct memory used metrics. OAP self observability dashboard: Add watermark circuit break/recover metrics. Implement the legend selector in metrics charts. Fix repetitive names in router. Bump up dependencies. Fixes tooltips cannot completely display metrics information. Fix time range when generate link. Add skywalking go agent self observability menu. add topN selector for endpoint list. Documentation Update release document to adopt newly added revision-based process. Improve BanyanDB documentation. Improve component-libraries documentation. Improve configuration-vocabulary documentation. Add Get Effective TTL Configurations API documentation. Add Status APIs docs. Simplified the release process with removing maven central publish relative processes. Add Circuit Breaking mechanism doc. Add Get Node List in the Cluster API doc. Remove meter.md doc, because mal.md has covered all the content. Merge browser-http-api-protocol.md doc into browser-protocol.md. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1020\"\u003e10.2.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd \u003ca href=\"https://www.elastic.co/guide/en/elasticsearch/reference/current/doc-values.html\"\u003e\u003ccode\u003edoc_values\u003c/code\u003e\u003c/a\u003e for fields\nthat need to be sorted or aggregated in Elasticsearch, and …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.4.0/en/changes/changes-10.2.0/","title":"10.2.0"},{"body":"10.2.0 Project Add doc_values for fields that need to be sorted or aggregated in Elasticsearch, and disable all others. This change would not impact the existing deployment and its feature for our official release users. Warning If there are custom query plugins for our Elasticsearch indices, this change could break them as sort queries and aggregation queries which used the unexpected fields are being blocked. [Breaking Change] Rename debugging-query module to status-query module. Relative exposed APIs are UNCHANGED. [Breaking Change] All jars of the skywalking-oap-server are no longer published through maven central. We will only publish the source tar and binary tar to the website download page, and docker images to docker hub. Warning If you are using the skywalking-oap-server as a dependency in your project, you need to download the source tar from the website and publish them to your private maven repository. [Breaking Change] Remove H2 as storage option permanently. BanyanDB 0.8(OAP 10.2 required) is easy, stable and production-ready. Don\u0026rsquo;t need H2 as default storage anymore. [Breaking Change] Bump up BanyanDB server version to 0.8.0. This version is not compatible with the previous versions. Please upgrade the BanyanDB server to 0.8.0 before upgrading OAP to 10.2.0. Bump up nodejs to v22.14.0 for the latest UI(booster-ui) compiling. Migrate tj-actions/changed-files to dorny/paths-filter. OAP Server Skip processing OTLP metrics data points with flag FLAG_NO_RECORDED_VALUE, which causes exceptional result. Add self observability metrics for GraphQL query, graphql_query_latency. Reduce the count of process index and adding time range when query process index. Bump up Apache commons-io to 2.17.0. Polish eBPF so11y metrics and add error count for query metrics. Support query endpoint list with duration parameter(optional). Change the endpoint_traffic to updatable for the additional column last_ping. Add Component ID(5023) for the GoZero framework. Support Kong monitoring. Support adding additional attr[0-5] for service/endpoint level metrics. Support async-profiler feature for performance analysis. Add metrics value owner for metrics topN query result. Add naming control for EndpointDependencyBuilder. The index type BanyanDB.IndexRule.IndexType#TREE is removed. All indices are using IndexType#INVERTED now. Add max query size settings to BanyanDB. Fix \u0026ldquo;BanyanDBTraceQueryDAO.queryBasicTraces\u0026rdquo; doesn\u0026rsquo;t support querying by \u0026ldquo;trace_id\u0026rdquo;. Polish mesh data dispatcher: don\u0026rsquo;t generate Instance/Endpoint metrics if they are empty. Adapt the new metadata standardization in Istio 1.24. Bump up netty to 4.1.115, grpc to 1.68.1, boringssl to 2.0.69. BanyanDB: Support update the Group settings when OAP starting. BanyanDB: Introduce index mode and refactor banyandb group settings. BanyanDB: Introduce the new Progressive TTL feature. BanyanDB: Support update the Schema when OAP starting. BanyanDB: Speed up OAP booting while initializing BanyanDB. BanyanDB: Support @EnableSort on the column to enable sorting for IndexRule and set the default to false. Support Get Effective TTL Configurations API. Fix ServerStatusService.statusWatchers concurrent modification. Add protection for dynamic config change propagate chain. Add Ruby component IDs(HttpClient=2, Redis=7, Memcached=20, Elasticsearch=47, Ruby=12000, Sinatra=12001). Add component ID(160) for Caffeine. Alarm: Support store and query the metrics snapshot when the alarm is triggered. Alarm: Remove unused Alarm Trend query. Fix missing remote endpoint IP address in span query of zipkin query module. Fix hierarchy-definition.yml config file packaged into start.jar wrongly. Add bydb.dependencies.properties config file to define server dependency versions. Fix AvgHistogramPercentileFunction doesn\u0026rsquo;t have proper field definition for ranks. BanyanDB: Support the new Property data module. MQE: Support top_n_of function for merging multiple metrics topn query. Support labelAvg function in the OAL engine. Added maxLabelCount parameter in the labelCount function of OAL to limit the number of labels can be counted. Adapt the new Browser API(/browser/perfData/webVitals, /browser/perfData/webInteractions, /browser/perfData/resources) protocol. Add Circuit Breaking mechanism. BanyanDB: Add support for compatibility checks based on the BanyanDB server\u0026rsquo;s API version. MQE: Support \u0026amp;\u0026amp;(and), ||(or) bool operators. OAP self observability: Add JVM heap and direct memory used metrics. OAP self observability: Add watermark circuit break/recover metrics. AI Pipeline: Support query baseline metrics names and predict metrics value. Add Get Node List in the Cluster API. Add type descriptor when converting Envoy logs to JSON for persistence, to avoid conversion error. Bseline: Support query baseline with MQE and use in the Alarm Rule. Bump up netty to 4.11.118 to fix CVE-2025-24970. Add Get Alarm Runtime Status API. Add lock when query the Alarm metrics window values. Add a fail-safe mechanism to prevent traffic metrics inconsistent between in-memory and database server. Add more clear logs when oap-cluster-internal data(metrics/traffic) format is inconsistent. Optimize metrics cache loading when trace latency greater than cache timeout. Allow calling lang.groovy.GString in DSL. BanyanDB: fix alarm query result without sort. Add a component ID for Virtual thread executor. Add more model installation log info for OAP storage initialization. BanyanDB: Separate the storage configuration to an independent file: bydb.yaml. Bump Armeria to 1.32.0 and some transitive dependencies. Skip persisting metrics/record data that have been expired. Fix the issue of missing Last Ping data. Add HTTP headers configuration for the alarm webhook. Bump up BanyanDB java client to 0.8.0. UI Add support for case-insensitive search in the dashboard list. Add content decorations to Table and Card widgets. Support the endpoint list widget query with duration parameter. Support ranges for Value Mappings. Add service global topN widget on General-Root, Mesh-Root, K8S-Root dashboard. Fix initialization dashboards. Update the Kubernetes metrics for reduce multiple metrics calculate in MQE. Support view data value related dashboards in TopList widgets. Add endpoint global topN widget on General-Root, Mesh-Root. Implement owner option for TopList widgets in related trace options. Hide entrances to unrelated dashboards in topn list. Split topology metric query to avoid exceeding the maximum query complexity. Fix view metrics related trace and metrics query. Add support collapse span. Refactor copy util with Web API. Releases an existing object URL. Optimize Trace Profiling widget. Implement Async Profiling widget. Fix inaccurate data query issue on endpoint topology page. Update browser dashboard for the new metrics. Visualize Snapshot on Alerting page. OAP self observability dashboard: Add JVM heap and direct memory used metrics. OAP self observability dashboard: Add watermark circuit break/recover metrics. Implement the legend selector in metrics charts. Fix repetitive names in router. Bump up dependencies. Fixes tooltips cannot completely display metrics information. Fix time range when generate link. Add skywalking go agent self observability menu. add topN selector for endpoint list. Documentation Update release document to adopt newly added revision-based process. Improve BanyanDB documentation. Improve component-libraries documentation. Improve configuration-vocabulary documentation. Add Get Effective TTL Configurations API documentation. Add Status APIs docs. Simplified the release process with removing maven central publish relative processes. Add Circuit Breaking mechanism doc. Add Get Node List in the Cluster API doc. Remove meter.md doc, because mal.md has covered all the content. Merge browser-http-api-protocol.md doc into browser-protocol.md. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1020\"\u003e10.2.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd \u003ca href=\"https://www.elastic.co/guide/en/elasticsearch/reference/current/doc-values.html\"\u003e\u003ccode\u003edoc_values\u003c/code\u003e\u003c/a\u003e for fields\nthat need to be sorted or aggregated in Elasticsearch, and …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v11.0.0/en/changes/changes-10.2.0/","title":"10.2.0"},{"body":"10.3.0 Project Bump up BanyanDB dependency version(server and java-client) to 0.9.0. Fix CVE-2025-54057, restrict and validate url for widgets. Fix MetricsPersistentWorker, remove DataCarrier queue from Hour/Day dimensions metrics persistent process. This is important to reduce memory cost and Hour/Day dimensions metrics persistent latency. [Break Change] BanyanDB: support new Trace model. OAP Server Implement self-monitoring for BanyanDB via OAP Server. BanyanDB: Support hot/warm/cold stages configuration. Fix query continues profiling policies error when the policy is already in the cache. Support hot/warm/cold stages TTL query in the status API and graphQL API. PromQL Service: traffic query support limit and regex match. Fix an edge case of HashCodeSelector(Integer#MIN_VALUE causes ArrayIndexOutOfBoundsException). Support Flink monitoring. BanyanDB: Support @ShardingKey for Measure tags. BanyanDB: Support cold stage data query for metrics/traces/logs. Increase the idle check interval of the message queue to 200ms to reduce CPU usage under low load conditions. Limit max attempts of DNS resolution of Istio ServiceEntry to 3, and do not wait for first resolution result in case the DNS is not resolvable at all. Support analysis waypoint metrics in Envoy ALS receiver. Add Ztunnel component in the topology. [Break Change] Change componentId to componentIds in the K8SServiceRelation Scope. Adapt the mesh metrics if detect the ambient mesh in the eBPF access log receiver. Add JSON format support for the /debugging/config/dump status API. Enhance status APIs to support multiple accept header values, e.g. Accept: application/json; charset=utf-8. Storage: separate SpanAttachedEventRecord for SkyWalking trace and Zipkin trace. [Break Change]BanyanDB: Setup new Group policy. Bump up commons-beanutils to 1.11.0. Refactor: simplify the Accept http header process. [Break Change]Storage: Move event from metrics to records. Remove string limitation in Jackson deserializer for ElasticSearch client. Fix disable.oal does not work. Enhance the stability of e2e PHP tests and update the PHP agent version. Add component ID for the dameng JDBC driver. BanyanDB: Support custom TopN pre-aggregation rules configuration in file bydb-topn.yml. refactor: implement OTEL handler with SPI for extensibility. chore: add toString implementation for StorageID. chore: add a warning log when connecting to ES takes too long. Fix the query time range in the metadata API. OAP gRPC-Client support Health Check. [Break Change] health_check_xx metrics make response 1 represents healthy, 0 represents unhealthy. Bump up grpc to 1.70.0. BanyanDB: support new Index rule type SKIPPING/TREE, and update the record log\u0026rsquo;s trace_id indexType to SKIPPING BanyanDB: remove index-only from tag setting. Fix analysis tracing profiling span failure in ES storage. Add UI dashboard for Ruby runtime metrics. Tracing Query Execution HTTP APIs: make the argument service layer optional. GraphQL API: metadata, topology, log and trace support query by name. [Break Change] MQE function sort_values sorts according to the aggregation result and labels rather than the simple time series values. Self Observability: add metrics_aggregation_queue_used_percentage and metrics_persistent_collection_cached_size metrics for the OAP server. Optimize metrics aggregate/persistent worker: separate OAL and MAL workers and consume pools. The dataflow signal drives the new MAL consumer, the following table shows the pool size, driven mode and queue size for each worker. Worker poolSize isSignalDrivenMode queueChannelSize queueBufferSize MetricsAggregateOALWorker Math.ceil(availableProcessors * 2 * 1.5) false 2 10000 MetricsAggregateMALWorker availableProcessors * 2 / 8, at least 1 true 1 1000 MetricsPersistentMinOALWorker availableProcessors * 2 / 8, at least 1 false 1 2000 MetricsPersistentMinMALWorker availableProcessors * 2 / 16, at least 1 true 1 1000 Bump up netty to 4.2.4.Final. Bump up commons-lang to 3.18.0. BanyanDB: support group replicas and user/password for basic authentication. BanyanDB: fix Zipkin query missing tag QUERY. Fix IllegalArgumentException: Incorrect number of labels, tags in the LogReportServiceHTTPHandler and LogReportServiceGrpcHandler inconsistent with LogHandler. BanyanDB: fix Zipkin query by annotationQuery HTTP Server: Use the default shared thread pool rather than creating a new event loop thread pool for each server. Remove the MAX_THREADS from each server config. Optimize all Armeria HTTP Server(s) to share the CommonPools for the whole JVM. In the CommonPools, the max threads for EventLoopGroup is processor * 2, and for BlockingTaskExecutor is 200 and can be recycled if over the keepAliveTimeMillis (60000L by default). Here is a summary of the thread dump without UI query in a simple Kind env deployed by SkyWalking showcase: Thread Type Count Main State Description JVM System Threads 12 RUNNABLE/WAITING Includes Reference Handler, Finalizer, Signal Dispatcher, Service Thread, C2/C1 CompilerThreads, Sweeper thread, Common-Cleaner, etc. Netty I/O Worker Threads 32 RUNNABLE Threads named \u0026ldquo;armeria-common-worker-epoll-*\u0026rdquo;, handling network I/O operations. gRPC Worker Threads 16 RUNNABLE Threads named \u0026ldquo;grpc-default-worker-*\u0026rdquo;. HTTP Client Threads 4 RUNNABLE Threads named \u0026ldquo;HttpClient-*-SelectorManager\u0026rdquo;. Data Consumer Threads 47 TIMED_WAITING (sleeping) Threads named \u0026ldquo;DataCarrier.*\u0026rdquo;, used for metrics data consumption. Scheduled Task Threads 10 TIMED_WAITING (parking) Threads named \u0026ldquo;pool--thread-\u0026rdquo;. ForkJoinPool Worker Threads 2 WAITING (parking) Threads named \u0026ldquo;ForkJoinPool-*\u0026rdquo;. BanyanDB Processor Threads 2 TIMED_WAITING (parking) Threads named \u0026ldquo;BanyanDB BulkProcessor\u0026rdquo;. gRPC Executor Threads 3 TIMED_WAITING (parking) Threads named \u0026ldquo;grpc-default-executor-*\u0026rdquo;. JVM GC Threads 13 RUNNABLE Threads named \u0026ldquo;GC Thread#*\u0026rdquo; for garbage collection. Other JVM Internal Threads 3 RUNNABLE Includes VM Thread, G1 Main Marker, VM Periodic Task Thread. Attach Listener 1 RUNNABLE JVM attach listener thread. Total 158 - - BanyanDB: make BanyanDBMetricsDAO output scan all blocks info log only when the model is not indexModel. BanyanDB: fix the BanyanDBMetricsDAO.multiGet not work properly in IndexMode. BanyanDB: remove @StoreIDAsTag, and automatically create a virtual String tag id for the SeriesID in IndexMode. Remove method appendMutant from StorageID. Fix otlp log handler response error and otlp span convert error. Fix service_relation source layer in mq entry span analyse. Fix metrics comparison in promql with bool modifier. Add rate limiter for Zipkin trace receiver to limit maximum spans per second. Open health-checker module by default due to latest UI changes. Change the default check period to 30s. Refactor Kubernetes coordinator to be more accurate about node readiness. Bump up netty to 4.2.5.Final. BanyanDB: fix log query missing order by condition, and fix missing service id condition when query by instance id or endpoint id. Fix potential NPE in the AlarmStatusQueryHandler. Aggregate TopN Slow SQL by service dimension. BanyanDB: support add group prefix (namespace) for BanyanDB groups. BanyanDB: fix when setting @BanyanDB.TimestampColumn, the column should not be indexed. OAP Self Observability: make Trace analysis metrics separate by label protocol, add Zipkin span dropped metrics. BanyanDB: Move data write logic from BanyanDB Java Client to OAP and support observe metrics for write operations. Self Observability: add write latency metrics for BanyanDB and ElasticSearch. Fix the malfunctioning alarm feature of MAL metrics due to unknown metadata in L2 aggregate worker. Make MAL percentile align with OAL percentile calculation. Update Grafana dashboards for OAP observability. BanyanDB: fix query getInstance by instance ID. Support the go agent(0.7.0 release) bundled pprof profiling feature. Service and TCPService source support analyze TLS mode. Library-pprof-parser: feat: add PprofSegmentParser. Storage: feat: add languageType column to ProfileThreadSnapshotRecord. Feat: add go profile analyzer Get Alarm Runtime Status: support query the running status for the whole cluster. UI Implement self-monitoring for BanyanDB via UI. Enhance the trace List/Tree/Table graph to support displaying multiple refs of spans and distinguishing different parents. Fix: correct the same labels for metrics. Refactor: use the Fetch API to instead of Axios. Support cold stage data for metrics, trace and log. Add route to status API /debugging/config/dump in the UI. Implement the Status API on Settings page. Bump vite from 6.2.6 to 6.3.6. Enhance async profiling by adding shorter and custom duration options. Fix select wrong span to analysis in trace profiling. Correct the service list for legends in trace graphs. Correct endpoint topology data to avoid undefined. Fix the snapshot charts unable to display. Bump vue-i18n from 9.14.3 to 9.14.5. Fix split queries for topology to avoid page crash. Self Observability ui-template: Add new panels for monitor metrics aggregation queue used percentage and metrics persistent collection cached size. test: introduce and set up unit tests in the UI. test: implement comprehensive unit tests for components. refactor: optimize data types for widgets and dashboards. fix: optimize appearing the wrong prompt by pop-up for the HTTP environments in copy function. refactor the configuration view and implement the optional config for displaying timestamp in Log widget. test: implement unit tests for hooks and refactor some types. fix: share OAP proxy services for different endpoints and use health checked endpoints group. Optimize buttons in time picker component. Optimize the router system and implement unit tests for router. Bump element-plus from 2.9.4 to 2.11.0. Adapt new trace protocol and implement new trace view. Implement Trace page. Support collapsing and expanding for the event widget. UI-template: add BanyanDB and Elasticsearch write latency dashboards for OAP self observability. Documentation BanyanDB: Add Data Lifecycle Stages(Hot/Warm/Cold) documentation. Add SWIP-9 Support flink monitoring. Fix Metrics Attributes menu link. Implement the Status API on Settings page. Fix: Add the prefix for http url. Enhance the async-profiling duration options. Enhance the TTL Tab on Setting page. Fix the snapshot charts in alarm page. Fix Fluent Bit dead links. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1030\"\u003e10.3.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eBump up BanyanDB dependency version(server and java-client) to 0.9.0.\u003c/li\u003e\n\u003cli\u003eFix …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/latest/en/changes/changes-10.3.0/","title":"10.3.0"},{"body":"10.3.0 Project Bump up BanyanDB dependency version(server and java-client) to 0.9.0. Fix CVE-2025-54057, restrict and validate url for widgets. Fix MetricsPersistentWorker, remove DataCarrier queue from Hour/Day dimensions metrics persistent process. This is important to reduce memory cost and Hour/Day dimensions metrics persistent latency. [Break Change] BanyanDB: support new Trace model. OAP Server Implement self-monitoring for BanyanDB via OAP Server. BanyanDB: Support hot/warm/cold stages configuration. Fix query continues profiling policies error when the policy is already in the cache. Support hot/warm/cold stages TTL query in the status API and graphQL API. PromQL Service: traffic query support limit and regex match. Fix an edge case of HashCodeSelector(Integer#MIN_VALUE causes ArrayIndexOutOfBoundsException). Support Flink monitoring. BanyanDB: Support @ShardingKey for Measure tags. BanyanDB: Support cold stage data query for metrics/traces/logs. Increase the idle check interval of the message queue to 200ms to reduce CPU usage under low load conditions. Limit max attempts of DNS resolution of Istio ServiceEntry to 3, and do not wait for first resolution result in case the DNS is not resolvable at all. Support analysis waypoint metrics in Envoy ALS receiver. Add Ztunnel component in the topology. [Break Change] Change componentId to componentIds in the K8SServiceRelation Scope. Adapt the mesh metrics if detect the ambient mesh in the eBPF access log receiver. Add JSON format support for the /debugging/config/dump status API. Enhance status APIs to support multiple accept header values, e.g. Accept: application/json; charset=utf-8. Storage: separate SpanAttachedEventRecord for SkyWalking trace and Zipkin trace. [Break Change]BanyanDB: Setup new Group policy. Bump up commons-beanutils to 1.11.0. Refactor: simplify the Accept http header process. [Break Change]Storage: Move event from metrics to records. Remove string limitation in Jackson deserializer for ElasticSearch client. Fix disable.oal does not work. Enhance the stability of e2e PHP tests and update the PHP agent version. Add component ID for the dameng JDBC driver. BanyanDB: Support custom TopN pre-aggregation rules configuration in file bydb-topn.yml. refactor: implement OTEL handler with SPI for extensibility. chore: add toString implementation for StorageID. chore: add a warning log when connecting to ES takes too long. Fix the query time range in the metadata API. OAP gRPC-Client support Health Check. [Break Change] health_check_xx metrics make response 1 represents healthy, 0 represents unhealthy. Bump up grpc to 1.70.0. BanyanDB: support new Index rule type SKIPPING/TREE, and update the record log\u0026rsquo;s trace_id indexType to SKIPPING BanyanDB: remove index-only from tag setting. Fix analysis tracing profiling span failure in ES storage. Add UI dashboard for Ruby runtime metrics. Tracing Query Execution HTTP APIs: make the argument service layer optional. GraphQL API: metadata, topology, log and trace support query by name. [Break Change] MQE function sort_values sorts according to the aggregation result and labels rather than the simple time series values. Self Observability: add metrics_aggregation_queue_used_percentage and metrics_persistent_collection_cached_size metrics for the OAP server. Optimize metrics aggregate/persistent worker: separate OAL and MAL workers and consume pools. The dataflow signal drives the new MAL consumer, the following table shows the pool size, driven mode and queue size for each worker. Worker poolSize isSignalDrivenMode queueChannelSize queueBufferSize MetricsAggregateOALWorker Math.ceil(availableProcessors * 2 * 1.5) false 2 10000 MetricsAggregateMALWorker availableProcessors * 2 / 8, at least 1 true 1 1000 MetricsPersistentMinOALWorker availableProcessors * 2 / 8, at least 1 false 1 2000 MetricsPersistentMinMALWorker availableProcessors * 2 / 16, at least 1 true 1 1000 Bump up netty to 4.2.4.Final. Bump up commons-lang to 3.18.0. BanyanDB: support group replicas and user/password for basic authentication. BanyanDB: fix Zipkin query missing tag QUERY. Fix IllegalArgumentException: Incorrect number of labels, tags in the LogReportServiceHTTPHandler and LogReportServiceGrpcHandler inconsistent with LogHandler. BanyanDB: fix Zipkin query by annotationQuery HTTP Server: Use the default shared thread pool rather than creating a new event loop thread pool for each server. Remove the MAX_THREADS from each server config. Optimize all Armeria HTTP Server(s) to share the CommonPools for the whole JVM. In the CommonPools, the max threads for EventLoopGroup is processor * 2, and for BlockingTaskExecutor is 200 and can be recycled if over the keepAliveTimeMillis (60000L by default). Here is a summary of the thread dump without UI query in a simple Kind env deployed by SkyWalking showcase: Thread Type Count Main State Description JVM System Threads 12 RUNNABLE/WAITING Includes Reference Handler, Finalizer, Signal Dispatcher, Service Thread, C2/C1 CompilerThreads, Sweeper thread, Common-Cleaner, etc. Netty I/O Worker Threads 32 RUNNABLE Threads named \u0026ldquo;armeria-common-worker-epoll-*\u0026rdquo;, handling network I/O operations. gRPC Worker Threads 16 RUNNABLE Threads named \u0026ldquo;grpc-default-worker-*\u0026rdquo;. HTTP Client Threads 4 RUNNABLE Threads named \u0026ldquo;HttpClient-*-SelectorManager\u0026rdquo;. Data Consumer Threads 47 TIMED_WAITING (sleeping) Threads named \u0026ldquo;DataCarrier.*\u0026rdquo;, used for metrics data consumption. Scheduled Task Threads 10 TIMED_WAITING (parking) Threads named \u0026ldquo;pool--thread-\u0026rdquo;. ForkJoinPool Worker Threads 2 WAITING (parking) Threads named \u0026ldquo;ForkJoinPool-*\u0026rdquo;. BanyanDB Processor Threads 2 TIMED_WAITING (parking) Threads named \u0026ldquo;BanyanDB BulkProcessor\u0026rdquo;. gRPC Executor Threads 3 TIMED_WAITING (parking) Threads named \u0026ldquo;grpc-default-executor-*\u0026rdquo;. JVM GC Threads 13 RUNNABLE Threads named \u0026ldquo;GC Thread#*\u0026rdquo; for garbage collection. Other JVM Internal Threads 3 RUNNABLE Includes VM Thread, G1 Main Marker, VM Periodic Task Thread. Attach Listener 1 RUNNABLE JVM attach listener thread. Total 158 - - BanyanDB: make BanyanDBMetricsDAO output scan all blocks info log only when the model is not indexModel. BanyanDB: fix the BanyanDBMetricsDAO.multiGet not work properly in IndexMode. BanyanDB: remove @StoreIDAsTag, and automatically create a virtual String tag id for the SeriesID in IndexMode. Remove method appendMutant from StorageID. Fix otlp log handler response error and otlp span convert error. Fix service_relation source layer in mq entry span analyse. Fix metrics comparison in promql with bool modifier. Add rate limiter for Zipkin trace receiver to limit maximum spans per second. Open health-checker module by default due to latest UI changes. Change the default check period to 30s. Refactor Kubernetes coordinator to be more accurate about node readiness. Bump up netty to 4.2.5.Final. BanyanDB: fix log query missing order by condition, and fix missing service id condition when query by instance id or endpoint id. Fix potential NPE in the AlarmStatusQueryHandler. Aggregate TopN Slow SQL by service dimension. BanyanDB: support add group prefix (namespace) for BanyanDB groups. BanyanDB: fix when setting @BanyanDB.TimestampColumn, the column should not be indexed. OAP Self Observability: make Trace analysis metrics separate by label protocol, add Zipkin span dropped metrics. BanyanDB: Move data write logic from BanyanDB Java Client to OAP and support observe metrics for write operations. Self Observability: add write latency metrics for BanyanDB and ElasticSearch. Fix the malfunctioning alarm feature of MAL metrics due to unknown metadata in L2 aggregate worker. Make MAL percentile align with OAL percentile calculation. Update Grafana dashboards for OAP observability. BanyanDB: fix query getInstance by instance ID. Support the go agent(0.7.0 release) bundled pprof profiling feature. Service and TCPService source support analyze TLS mode. Library-pprof-parser: feat: add PprofSegmentParser. Storage: feat: add languageType column to ProfileThreadSnapshotRecord. Feat: add go profile analyzer Get Alarm Runtime Status: support query the running status for the whole cluster. UI Implement self-monitoring for BanyanDB via UI. Enhance the trace List/Tree/Table graph to support displaying multiple refs of spans and distinguishing different parents. Fix: correct the same labels for metrics. Refactor: use the Fetch API to instead of Axios. Support cold stage data for metrics, trace and log. Add route to status API /debugging/config/dump in the UI. Implement the Status API on Settings page. Bump vite from 6.2.6 to 6.3.6. Enhance async profiling by adding shorter and custom duration options. Fix select wrong span to analysis in trace profiling. Correct the service list for legends in trace graphs. Correct endpoint topology data to avoid undefined. Fix the snapshot charts unable to display. Bump vue-i18n from 9.14.3 to 9.14.5. Fix split queries for topology to avoid page crash. Self Observability ui-template: Add new panels for monitor metrics aggregation queue used percentage and metrics persistent collection cached size. test: introduce and set up unit tests in the UI. test: implement comprehensive unit tests for components. refactor: optimize data types for widgets and dashboards. fix: optimize appearing the wrong prompt by pop-up for the HTTP environments in copy function. refactor the configuration view and implement the optional config for displaying timestamp in Log widget. test: implement unit tests for hooks and refactor some types. fix: share OAP proxy services for different endpoints and use health checked endpoints group. Optimize buttons in time picker component. Optimize the router system and implement unit tests for router. Bump element-plus from 2.9.4 to 2.11.0. Adapt new trace protocol and implement new trace view. Implement Trace page. Support collapsing and expanding for the event widget. UI-template: add BanyanDB and Elasticsearch write latency dashboards for OAP self observability. Documentation BanyanDB: Add Data Lifecycle Stages(Hot/Warm/Cold) documentation. Add SWIP-9 Support flink monitoring. Fix Metrics Attributes menu link. Implement the Status API on Settings page. Fix: Add the prefix for http url. Enhance the async-profiling duration options. Enhance the TTL Tab on Setting page. Fix the snapshot charts in alarm page. Fix Fluent Bit dead links. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1030\"\u003e10.3.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eBump up BanyanDB dependency version(server and java-client) to 0.9.0.\u003c/li\u003e\n\u003cli\u003eFix …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/next/en/changes/changes-10.3.0/","title":"10.3.0"},{"body":"10.3.0 Project OAP Server UI Documentation All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1030\"\u003e10.3.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003ch4 id=\"oap-server\"\u003eOAP Server\u003c/h4\u003e\n\u003ch4 id=\"ui\"\u003eUI\u003c/h4\u003e\n\u003ch4 id=\"documentation\"\u003eDocumentation\u003c/h4\u003e\n\u003cp\u003eAll issues and pull requests are \u003ca href=\"https://github.com/apache/skywalking/milestone/230?closed=1\"\u003ehere\u003c/a\u003e\u003c/p\u003e","ref":"https://skywalking.apache.org/docs/main/v10.2.0/en/changes/changes/","title":"10.3.0"},{"body":"10.3.0 Project Bump up BanyanDB dependency version(server and java-client) to 0.9.0. Fix CVE-2025-54057, restrict and validate url for widgets. Fix MetricsPersistentWorker, remove DataCarrier queue from Hour/Day dimensions metrics persistent process. This is important to reduce memory cost and Hour/Day dimensions metrics persistent latency. [Break Change] BanyanDB: support new Trace model. OAP Server Implement self-monitoring for BanyanDB via OAP Server. BanyanDB: Support hot/warm/cold stages configuration. Fix query continues profiling policies error when the policy is already in the cache. Support hot/warm/cold stages TTL query in the status API and graphQL API. PromQL Service: traffic query support limit and regex match. Fix an edge case of HashCodeSelector(Integer#MIN_VALUE causes ArrayIndexOutOfBoundsException). Support Flink monitoring. BanyanDB: Support @ShardingKey for Measure tags. BanyanDB: Support cold stage data query for metrics/traces/logs. Increase the idle check interval of the message queue to 200ms to reduce CPU usage under low load conditions. Limit max attempts of DNS resolution of Istio ServiceEntry to 3, and do not wait for first resolution result in case the DNS is not resolvable at all. Support analysis waypoint metrics in Envoy ALS receiver. Add Ztunnel component in the topology. [Break Change] Change compomentId to componentIds in the K8SServiceRelation Scope. Adapt the mesh metrics if detect the ambient mesh in the eBPF access log receiver. Add JSON format support for the /debugging/config/dump status API. Enhance status APIs to support multiple accept header values, e.g. Accept: application/json; charset=utf-8. Storage: separate SpanAttachedEventRecord for SkyWalking trace and Zipkin trace. [Break Change]BanyanDB: Setup new Group policy. Bump up commons-beanutils to 1.11.0. Refactor: simplify the Accept http header process. [Break Change]Storage: Move event from metrics to records. Remove string limitation in Jackson deserializer for ElasticSearch client. Fix disable.oal does not work. Enhance the stability of e2e PHP tests and update the PHP agent version. Add component ID for the dameng JDBC driver. BanyanDB: Support custom TopN pre-aggregation rules configuration in file bydb-topn.yml. refactor: implement OTEL handler with SPI for extensibility. chore: add toString implementation for StorageID. chore: add a warning log when connecting to ES takes too long. Fix the query time range in the metadata API. OAP gRPC-Client support Health Check. [Break Change] health_check_xx metrics make response 1 represents healthy, 0 represents unhealthy. Bump up grpc to 1.70.0. BanyanDB: support new Index rule type SKIPPING/TREE, and update the record log\u0026rsquo;s trace_id indexType to SKIPPING BanyanDB: remove index-only from tag setting. Fix analysis tracing profiling span failure in ES storage. Add UI dashboard for Ruby runtime metrics. Tracing Query Execution HTTP APIs: make the argument service layer optional. GraphQL API: metadata, topology, log and trace support query by name. [Break Change] MQE function sort_values sorts according to the aggregation result and labels rather than the simple time series values. Self Observability: add metrics_aggregation_queue_used_percentage and metrics_persistent_collection_cached_size metrics for the OAP server. Optimize metrics aggregate/persistent worker: separate OAL and MAL workers and consume pools. The dataflow signal drives the new MAL consumer, the following table shows the pool size，driven mode and queue size for each worker. Worker poolSize isSignalDrivenMode queueChannelSize queueBufferSize MetricsAggregateOALWorker Math.ceil(availableProcessors * 2 * 1.5) false 2 10000 MetricsAggregateMALWorker availableProcessors * 2 / 8, at least 1 true 1 1000 MetricsPersistentMinOALWorker availableProcessors * 2 / 8, at least 1 false 1 2000 MetricsPersistentMinMALWorker availableProcessors * 2 / 16, at least 1 true 1 1000 Bump up netty to 4.2.4.Final. Bump up commons-lang to 3.18.0. BanyanDB: support group replicas and user/password for basic authentication. BanyanDB: fix Zipkin query missing tag QUERY. Fix IllegalArgumentException: Incorrect number of labels, tags in the LogReportServiceHTTPHandler and LogReportServiceGrpcHandler inconsistent with LogHandler. BanyanDB: fix Zipkin query by annotationQuery HTTP Server: Use the default shared thread pool rather than creating a new event loop thread pool for each server. Remove the MAX_THREADS from each server config. Optimize all Armeria HTTP Server(s) to share the CommonPools for the whole JVM. In the CommonPools, the max threads for EventLoopGroup is processor * 2, and for BlockingTaskExecutor is 200 and can be recycled if over the keepAliveTimeMillis (60000L by default). Here is a summary of the thread dump without UI query in a simple Kind env deployed by SkyWalking showcase: Thread Type Count Main State Description JVM System Threads 12 RUNNABLE/WAITING Includes Reference Handler, Finalizer, Signal Dispatcher, Service Thread, C2/C1 CompilerThreads, Sweeper thread, Common-Cleaner, etc. Netty I/O Worker Threads 32 RUNNABLE Threads named \u0026ldquo;armeria-common-worker-epoll-*\u0026rdquo;, handling network I/O operations. gRPC Worker Threads 16 RUNNABLE Threads named \u0026ldquo;grpc-default-worker-*\u0026rdquo;. HTTP Client Threads 4 RUNNABLE Threads named \u0026ldquo;HttpClient-*-SelectorManager\u0026rdquo;. Data Consumer Threads 47 TIMED_WAITING (sleeping) Threads named \u0026ldquo;DataCarrier.*\u0026rdquo;, used for metrics data consumption. Scheduled Task Threads 10 TIMED_WAITING (parking) Threads named \u0026ldquo;pool--thread-\u0026rdquo;. ForkJoinPool Worker Threads 2 WAITING (parking) Threads named \u0026ldquo;ForkJoinPool-*\u0026rdquo;. BanyanDB Processor Threads 2 TIMED_WAITING (parking) Threads named \u0026ldquo;BanyanDB BulkProcessor\u0026rdquo;. gRPC Executor Threads 3 TIMED_WAITING (parking) Threads named \u0026ldquo;grpc-default-executor-*\u0026rdquo;. JVM GC Threads 13 RUNNABLE Threads named \u0026ldquo;GC Thread#*\u0026rdquo; for garbage collection. Other JVM Internal Threads 3 RUNNABLE Includes VM Thread, G1 Main Marker, VM Periodic Task Thread. Attach Listener 1 RUNNABLE JVM attach listener thread. Total 158 - - BanyanDB: make BanyanDBMetricsDAO output scan all blocks info log only when the model is not indexModel. BanyanDB: fix the BanyanDBMetricsDAO.multiGet not work properly in IndexMode. BanyanDB: remove @StoreIDAsTag, and automatically create a virtual String tag id for the SeriesID in IndexMode. Remove method appendMutant from StorageID. Fix otlp log handler reponse error and otlp span convert error. Fix service_relation source layer in mq entry span analyse. Fix metrics comparison in promql with bool modifier. Add rate limiter for Zipkin trace receiver to limit maximum spans per second. Open health-checker module by default due to latest UI changes. Change the default check period to 30s. Refactor Kubernetes coordinator to be more accurate about node readiness. Bump up netty to 4.2.5.Final. BanyanDB: fix log query missing order by condition, and fix missing service id condition when query by instance id or endpoint id. Fix potential NPE in the AlarmStatusQueryHandler. Aggregate TopN Slow SQL by service dimension. BanyanDB: support add group prefix (namespace) for BanyanDB groups. BanyanDB: fix when setting @BanyanDB.TimestampColumn, the column should not be indexed. OAP Self Observability: make Trace analysis metrics separate by label protocol, add Zipkin span dropped metrics. BanyanDB: Move data write logic from BanyanDB Java Client to OAP and support observe metrics for write operations. Self Observability: add write latency metrics for BanyanDB and ElasticSearch. Fix the malfunctioning alarm feature of MAL metrics due to unknown metadata in L2 aggregate worker. Make MAL percentile align with OAL percentile calculation. Update Grafana dashboards for OAP observability. BanyanDB: fix query getInstance by instance ID. Support the go agent(0.7.0 release) bundled pprof profiling feature. Service and TCPService source support analyze TLS mode. Library-pprof-parser: feat: add PprofSegmentParser. Storage: feat: add languageType column to ProfileThreadSnapshotRecord. Feat: add go profile analyzer Get Alarm Runtime Status: support query the running status for the whole cluster. UI Implement self-monitoring for BanyanDB via UI. Enhance the trace List/Tree/Table graph to support displaying multiple refs of spans and distinguishing different parents. Fix: correct the same labels for metrics. Refactor: use the Fetch API to instead of Axios. Support cold stage data for metrics, trace and log. Add route to status API /debugging/config/dump in the UI. Implement the Status API on Settings page. Bump vite from 6.2.6 to 6.3.6. Enhance async profiling by adding shorter and custom duration options. Fix select wrong span to analysis in trace profiling. Correct the service list for legends in trace graphs. Correct endpoint topology data to avoid undefined. Fix the snapshot charts unable to display. Bump vue-i18n from 9.14.3 to 9.14.5. Fix split queries for topology to avoid page crash. Self Observability ui-template: Add new panels for monitor metrics aggregation queue used percentage and metrics persistent collection cached size. test: introduce and set up unit tests in the UI. test: implement comprehensive unit tests for components. refactor: optimize data types for widgets and dashboards. fix: optimize appearing the wrong prompt by pop-up for the HTTP environments in copy function. refactor the configuration view and implement the optional config for displaying timestamp in Log widget. test: implement unit tests for hooks and refactor some types. fix: share OAP proxy servies for different endpoins and use health checked endpoints group. Optimize buttons in time picker component. Optimize the router system and implement unit tests for router. Bump element-plus from 2.9.4 to 2.11.0. Adapt new trace protocol and implement new trace view. Implement Trace page. Support collapsing and expanding for the event widget. UI-template: add BanyanDB and Elasticsearch write latency dashboards for OAP self observability. Documentation BanyanDB: Add Data Lifecycle Stages(Hot/Warm/Cold) documentation. Add SWIP-9 Support flink monitoring. Fix Metrics Attributes menu link. Implement the Status API on Settings page. Fix: Add the prefix for http url. Enhance the async-profiling duration options. Enhance the TTL Tab on Setting page. Fix the snapshot charts in alarm page. Fix Fluent Bit dead links. Fix progressive TTL doc for banyanDB. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1030\"\u003e10.3.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eBump up BanyanDB dependency version(server and java-client) to 0.9.0.\u003c/li\u003e\n\u003cli\u003eFix …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.3.0/en/changes/changes/","title":"10.3.0"},{"body":"10.3.0 Project Bump up BanyanDB dependency version(server and java-client) to 0.9.0. Fix CVE-2025-54057, restrict and validate url for widgets. Fix MetricsPersistentWorker, remove DataCarrier queue from Hour/Day dimensions metrics persistent process. This is important to reduce memory cost and Hour/Day dimensions metrics persistent latency. [Break Change] BanyanDB: support new Trace model. OAP Server Implement self-monitoring for BanyanDB via OAP Server. BanyanDB: Support hot/warm/cold stages configuration. Fix query continues profiling policies error when the policy is already in the cache. Support hot/warm/cold stages TTL query in the status API and graphQL API. PromQL Service: traffic query support limit and regex match. Fix an edge case of HashCodeSelector(Integer#MIN_VALUE causes ArrayIndexOutOfBoundsException). Support Flink monitoring. BanyanDB: Support @ShardingKey for Measure tags. BanyanDB: Support cold stage data query for metrics/traces/logs. Increase the idle check interval of the message queue to 200ms to reduce CPU usage under low load conditions. Limit max attempts of DNS resolution of Istio ServiceEntry to 3, and do not wait for first resolution result in case the DNS is not resolvable at all. Support analysis waypoint metrics in Envoy ALS receiver. Add Ztunnel component in the topology. [Break Change] Change componentId to componentIds in the K8SServiceRelation Scope. Adapt the mesh metrics if detect the ambient mesh in the eBPF access log receiver. Add JSON format support for the /debugging/config/dump status API. Enhance status APIs to support multiple accept header values, e.g. Accept: application/json; charset=utf-8. Storage: separate SpanAttachedEventRecord for SkyWalking trace and Zipkin trace. [Break Change]BanyanDB: Setup new Group policy. Bump up commons-beanutils to 1.11.0. Refactor: simplify the Accept http header process. [Break Change]Storage: Move event from metrics to records. Remove string limitation in Jackson deserializer for ElasticSearch client. Fix disable.oal does not work. Enhance the stability of e2e PHP tests and update the PHP agent version. Add component ID for the dameng JDBC driver. BanyanDB: Support custom TopN pre-aggregation rules configuration in file bydb-topn.yml. refactor: implement OTEL handler with SPI for extensibility. chore: add toString implementation for StorageID. chore: add a warning log when connecting to ES takes too long. Fix the query time range in the metadata API. OAP gRPC-Client support Health Check. [Break Change] health_check_xx metrics make response 1 represents healthy, 0 represents unhealthy. Bump up grpc to 1.70.0. BanyanDB: support new Index rule type SKIPPING/TREE, and update the record log\u0026rsquo;s trace_id indexType to SKIPPING BanyanDB: remove index-only from tag setting. Fix analysis tracing profiling span failure in ES storage. Add UI dashboard for Ruby runtime metrics. Tracing Query Execution HTTP APIs: make the argument service layer optional. GraphQL API: metadata, topology, log and trace support query by name. [Break Change] MQE function sort_values sorts according to the aggregation result and labels rather than the simple time series values. Self Observability: add metrics_aggregation_queue_used_percentage and metrics_persistent_collection_cached_size metrics for the OAP server. Optimize metrics aggregate/persistent worker: separate OAL and MAL workers and consume pools. The dataflow signal drives the new MAL consumer, the following table shows the pool size, driven mode and queue size for each worker. Worker poolSize isSignalDrivenMode queueChannelSize queueBufferSize MetricsAggregateOALWorker Math.ceil(availableProcessors * 2 * 1.5) false 2 10000 MetricsAggregateMALWorker availableProcessors * 2 / 8, at least 1 true 1 1000 MetricsPersistentMinOALWorker availableProcessors * 2 / 8, at least 1 false 1 2000 MetricsPersistentMinMALWorker availableProcessors * 2 / 16, at least 1 true 1 1000 Bump up netty to 4.2.4.Final. Bump up commons-lang to 3.18.0. BanyanDB: support group replicas and user/password for basic authentication. BanyanDB: fix Zipkin query missing tag QUERY. Fix IllegalArgumentException: Incorrect number of labels, tags in the LogReportServiceHTTPHandler and LogReportServiceGrpcHandler inconsistent with LogHandler. BanyanDB: fix Zipkin query by annotationQuery HTTP Server: Use the default shared thread pool rather than creating a new event loop thread pool for each server. Remove the MAX_THREADS from each server config. Optimize all Armeria HTTP Server(s) to share the CommonPools for the whole JVM. In the CommonPools, the max threads for EventLoopGroup is processor * 2, and for BlockingTaskExecutor is 200 and can be recycled if over the keepAliveTimeMillis (60000L by default). Here is a summary of the thread dump without UI query in a simple Kind env deployed by SkyWalking showcase: Thread Type Count Main State Description JVM System Threads 12 RUNNABLE/WAITING Includes Reference Handler, Finalizer, Signal Dispatcher, Service Thread, C2/C1 CompilerThreads, Sweeper thread, Common-Cleaner, etc. Netty I/O Worker Threads 32 RUNNABLE Threads named \u0026ldquo;armeria-common-worker-epoll-*\u0026rdquo;, handling network I/O operations. gRPC Worker Threads 16 RUNNABLE Threads named \u0026ldquo;grpc-default-worker-*\u0026rdquo;. HTTP Client Threads 4 RUNNABLE Threads named \u0026ldquo;HttpClient-*-SelectorManager\u0026rdquo;. Data Consumer Threads 47 TIMED_WAITING (sleeping) Threads named \u0026ldquo;DataCarrier.*\u0026rdquo;, used for metrics data consumption. Scheduled Task Threads 10 TIMED_WAITING (parking) Threads named \u0026ldquo;pool--thread-\u0026rdquo;. ForkJoinPool Worker Threads 2 WAITING (parking) Threads named \u0026ldquo;ForkJoinPool-*\u0026rdquo;. BanyanDB Processor Threads 2 TIMED_WAITING (parking) Threads named \u0026ldquo;BanyanDB BulkProcessor\u0026rdquo;. gRPC Executor Threads 3 TIMED_WAITING (parking) Threads named \u0026ldquo;grpc-default-executor-*\u0026rdquo;. JVM GC Threads 13 RUNNABLE Threads named \u0026ldquo;GC Thread#*\u0026rdquo; for garbage collection. Other JVM Internal Threads 3 RUNNABLE Includes VM Thread, G1 Main Marker, VM Periodic Task Thread. Attach Listener 1 RUNNABLE JVM attach listener thread. Total 158 - - BanyanDB: make BanyanDBMetricsDAO output scan all blocks info log only when the model is not indexModel. BanyanDB: fix the BanyanDBMetricsDAO.multiGet not work properly in IndexMode. BanyanDB: remove @StoreIDAsTag, and automatically create a virtual String tag id for the SeriesID in IndexMode. Remove method appendMutant from StorageID. Fix otlp log handler response error and otlp span convert error. Fix service_relation source layer in mq entry span analyse. Fix metrics comparison in promql with bool modifier. Add rate limiter for Zipkin trace receiver to limit maximum spans per second. Open health-checker module by default due to latest UI changes. Change the default check period to 30s. Refactor Kubernetes coordinator to be more accurate about node readiness. Bump up netty to 4.2.5.Final. BanyanDB: fix log query missing order by condition, and fix missing service id condition when query by instance id or endpoint id. Fix potential NPE in the AlarmStatusQueryHandler. Aggregate TopN Slow SQL by service dimension. BanyanDB: support add group prefix (namespace) for BanyanDB groups. BanyanDB: fix when setting @BanyanDB.TimestampColumn, the column should not be indexed. OAP Self Observability: make Trace analysis metrics separate by label protocol, add Zipkin span dropped metrics. BanyanDB: Move data write logic from BanyanDB Java Client to OAP and support observe metrics for write operations. Self Observability: add write latency metrics for BanyanDB and ElasticSearch. Fix the malfunctioning alarm feature of MAL metrics due to unknown metadata in L2 aggregate worker. Make MAL percentile align with OAL percentile calculation. Update Grafana dashboards for OAP observability. BanyanDB: fix query getInstance by instance ID. Support the go agent(0.7.0 release) bundled pprof profiling feature. Service and TCPService source support analyze TLS mode. Library-pprof-parser: feat: add PprofSegmentParser. Storage: feat: add languageType column to ProfileThreadSnapshotRecord. Feat: add go profile analyzer Get Alarm Runtime Status: support query the running status for the whole cluster. UI Implement self-monitoring for BanyanDB via UI. Enhance the trace List/Tree/Table graph to support displaying multiple refs of spans and distinguishing different parents. Fix: correct the same labels for metrics. Refactor: use the Fetch API to instead of Axios. Support cold stage data for metrics, trace and log. Add route to status API /debugging/config/dump in the UI. Implement the Status API on Settings page. Bump vite from 6.2.6 to 6.3.6. Enhance async profiling by adding shorter and custom duration options. Fix select wrong span to analysis in trace profiling. Correct the service list for legends in trace graphs. Correct endpoint topology data to avoid undefined. Fix the snapshot charts unable to display. Bump vue-i18n from 9.14.3 to 9.14.5. Fix split queries for topology to avoid page crash. Self Observability ui-template: Add new panels for monitor metrics aggregation queue used percentage and metrics persistent collection cached size. test: introduce and set up unit tests in the UI. test: implement comprehensive unit tests for components. refactor: optimize data types for widgets and dashboards. fix: optimize appearing the wrong prompt by pop-up for the HTTP environments in copy function. refactor the configuration view and implement the optional config for displaying timestamp in Log widget. test: implement unit tests for hooks and refactor some types. fix: share OAP proxy services for different endpoints and use health checked endpoints group. Optimize buttons in time picker component. Optimize the router system and implement unit tests for router. Bump element-plus from 2.9.4 to 2.11.0. Adapt new trace protocol and implement new trace view. Implement Trace page. Support collapsing and expanding for the event widget. UI-template: add BanyanDB and Elasticsearch write latency dashboards for OAP self observability. Documentation BanyanDB: Add Data Lifecycle Stages(Hot/Warm/Cold) documentation. Add SWIP-9 Support flink monitoring. Fix Metrics Attributes menu link. Implement the Status API on Settings page. Fix: Add the prefix for http url. Enhance the async-profiling duration options. Enhance the TTL Tab on Setting page. Fix the snapshot charts in alarm page. Fix Fluent Bit dead links. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1030\"\u003e10.3.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eBump up BanyanDB dependency version(server and java-client) to 0.9.0.\u003c/li\u003e\n\u003cli\u003eFix …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.4.0/en/changes/changes-10.3.0/","title":"10.3.0"},{"body":"10.3.0 Project Bump up BanyanDB dependency version(server and java-client) to 0.9.0. Fix CVE-2025-54057, restrict and validate url for widgets. Fix MetricsPersistentWorker, remove DataCarrier queue from Hour/Day dimensions metrics persistent process. This is important to reduce memory cost and Hour/Day dimensions metrics persistent latency. [Break Change] BanyanDB: support new Trace model. OAP Server Implement self-monitoring for BanyanDB via OAP Server. BanyanDB: Support hot/warm/cold stages configuration. Fix query continues profiling policies error when the policy is already in the cache. Support hot/warm/cold stages TTL query in the status API and graphQL API. PromQL Service: traffic query support limit and regex match. Fix an edge case of HashCodeSelector(Integer#MIN_VALUE causes ArrayIndexOutOfBoundsException). Support Flink monitoring. BanyanDB: Support @ShardingKey for Measure tags. BanyanDB: Support cold stage data query for metrics/traces/logs. Increase the idle check interval of the message queue to 200ms to reduce CPU usage under low load conditions. Limit max attempts of DNS resolution of Istio ServiceEntry to 3, and do not wait for first resolution result in case the DNS is not resolvable at all. Support analysis waypoint metrics in Envoy ALS receiver. Add Ztunnel component in the topology. [Break Change] Change componentId to componentIds in the K8SServiceRelation Scope. Adapt the mesh metrics if detect the ambient mesh in the eBPF access log receiver. Add JSON format support for the /debugging/config/dump status API. Enhance status APIs to support multiple accept header values, e.g. Accept: application/json; charset=utf-8. Storage: separate SpanAttachedEventRecord for SkyWalking trace and Zipkin trace. [Break Change]BanyanDB: Setup new Group policy. Bump up commons-beanutils to 1.11.0. Refactor: simplify the Accept http header process. [Break Change]Storage: Move event from metrics to records. Remove string limitation in Jackson deserializer for ElasticSearch client. Fix disable.oal does not work. Enhance the stability of e2e PHP tests and update the PHP agent version. Add component ID for the dameng JDBC driver. BanyanDB: Support custom TopN pre-aggregation rules configuration in file bydb-topn.yml. refactor: implement OTEL handler with SPI for extensibility. chore: add toString implementation for StorageID. chore: add a warning log when connecting to ES takes too long. Fix the query time range in the metadata API. OAP gRPC-Client support Health Check. [Break Change] health_check_xx metrics make response 1 represents healthy, 0 represents unhealthy. Bump up grpc to 1.70.0. BanyanDB: support new Index rule type SKIPPING/TREE, and update the record log\u0026rsquo;s trace_id indexType to SKIPPING BanyanDB: remove index-only from tag setting. Fix analysis tracing profiling span failure in ES storage. Add UI dashboard for Ruby runtime metrics. Tracing Query Execution HTTP APIs: make the argument service layer optional. GraphQL API: metadata, topology, log and trace support query by name. [Break Change] MQE function sort_values sorts according to the aggregation result and labels rather than the simple time series values. Self Observability: add metrics_aggregation_queue_used_percentage and metrics_persistent_collection_cached_size metrics for the OAP server. Optimize metrics aggregate/persistent worker: separate OAL and MAL workers and consume pools. The dataflow signal drives the new MAL consumer, the following table shows the pool size, driven mode and queue size for each worker. Worker poolSize isSignalDrivenMode queueChannelSize queueBufferSize MetricsAggregateOALWorker Math.ceil(availableProcessors * 2 * 1.5) false 2 10000 MetricsAggregateMALWorker availableProcessors * 2 / 8, at least 1 true 1 1000 MetricsPersistentMinOALWorker availableProcessors * 2 / 8, at least 1 false 1 2000 MetricsPersistentMinMALWorker availableProcessors * 2 / 16, at least 1 true 1 1000 Bump up netty to 4.2.4.Final. Bump up commons-lang to 3.18.0. BanyanDB: support group replicas and user/password for basic authentication. BanyanDB: fix Zipkin query missing tag QUERY. Fix IllegalArgumentException: Incorrect number of labels, tags in the LogReportServiceHTTPHandler and LogReportServiceGrpcHandler inconsistent with LogHandler. BanyanDB: fix Zipkin query by annotationQuery HTTP Server: Use the default shared thread pool rather than creating a new event loop thread pool for each server. Remove the MAX_THREADS from each server config. Optimize all Armeria HTTP Server(s) to share the CommonPools for the whole JVM. In the CommonPools, the max threads for EventLoopGroup is processor * 2, and for BlockingTaskExecutor is 200 and can be recycled if over the keepAliveTimeMillis (60000L by default). Here is a summary of the thread dump without UI query in a simple Kind env deployed by SkyWalking showcase: Thread Type Count Main State Description JVM System Threads 12 RUNNABLE/WAITING Includes Reference Handler, Finalizer, Signal Dispatcher, Service Thread, C2/C1 CompilerThreads, Sweeper thread, Common-Cleaner, etc. Netty I/O Worker Threads 32 RUNNABLE Threads named \u0026ldquo;armeria-common-worker-epoll-*\u0026rdquo;, handling network I/O operations. gRPC Worker Threads 16 RUNNABLE Threads named \u0026ldquo;grpc-default-worker-*\u0026rdquo;. HTTP Client Threads 4 RUNNABLE Threads named \u0026ldquo;HttpClient-*-SelectorManager\u0026rdquo;. Data Consumer Threads 47 TIMED_WAITING (sleeping) Threads named \u0026ldquo;DataCarrier.*\u0026rdquo;, used for metrics data consumption. Scheduled Task Threads 10 TIMED_WAITING (parking) Threads named \u0026ldquo;pool--thread-\u0026rdquo;. ForkJoinPool Worker Threads 2 WAITING (parking) Threads named \u0026ldquo;ForkJoinPool-*\u0026rdquo;. BanyanDB Processor Threads 2 TIMED_WAITING (parking) Threads named \u0026ldquo;BanyanDB BulkProcessor\u0026rdquo;. gRPC Executor Threads 3 TIMED_WAITING (parking) Threads named \u0026ldquo;grpc-default-executor-*\u0026rdquo;. JVM GC Threads 13 RUNNABLE Threads named \u0026ldquo;GC Thread#*\u0026rdquo; for garbage collection. Other JVM Internal Threads 3 RUNNABLE Includes VM Thread, G1 Main Marker, VM Periodic Task Thread. Attach Listener 1 RUNNABLE JVM attach listener thread. Total 158 - - BanyanDB: make BanyanDBMetricsDAO output scan all blocks info log only when the model is not indexModel. BanyanDB: fix the BanyanDBMetricsDAO.multiGet not work properly in IndexMode. BanyanDB: remove @StoreIDAsTag, and automatically create a virtual String tag id for the SeriesID in IndexMode. Remove method appendMutant from StorageID. Fix otlp log handler response error and otlp span convert error. Fix service_relation source layer in mq entry span analyse. Fix metrics comparison in promql with bool modifier. Add rate limiter for Zipkin trace receiver to limit maximum spans per second. Open health-checker module by default due to latest UI changes. Change the default check period to 30s. Refactor Kubernetes coordinator to be more accurate about node readiness. Bump up netty to 4.2.5.Final. BanyanDB: fix log query missing order by condition, and fix missing service id condition when query by instance id or endpoint id. Fix potential NPE in the AlarmStatusQueryHandler. Aggregate TopN Slow SQL by service dimension. BanyanDB: support add group prefix (namespace) for BanyanDB groups. BanyanDB: fix when setting @BanyanDB.TimestampColumn, the column should not be indexed. OAP Self Observability: make Trace analysis metrics separate by label protocol, add Zipkin span dropped metrics. BanyanDB: Move data write logic from BanyanDB Java Client to OAP and support observe metrics for write operations. Self Observability: add write latency metrics for BanyanDB and ElasticSearch. Fix the malfunctioning alarm feature of MAL metrics due to unknown metadata in L2 aggregate worker. Make MAL percentile align with OAL percentile calculation. Update Grafana dashboards for OAP observability. BanyanDB: fix query getInstance by instance ID. Support the go agent(0.7.0 release) bundled pprof profiling feature. Service and TCPService source support analyze TLS mode. Library-pprof-parser: feat: add PprofSegmentParser. Storage: feat: add languageType column to ProfileThreadSnapshotRecord. Feat: add go profile analyzer Get Alarm Runtime Status: support query the running status for the whole cluster. UI Implement self-monitoring for BanyanDB via UI. Enhance the trace List/Tree/Table graph to support displaying multiple refs of spans and distinguishing different parents. Fix: correct the same labels for metrics. Refactor: use the Fetch API to instead of Axios. Support cold stage data for metrics, trace and log. Add route to status API /debugging/config/dump in the UI. Implement the Status API on Settings page. Bump vite from 6.2.6 to 6.3.6. Enhance async profiling by adding shorter and custom duration options. Fix select wrong span to analysis in trace profiling. Correct the service list for legends in trace graphs. Correct endpoint topology data to avoid undefined. Fix the snapshot charts unable to display. Bump vue-i18n from 9.14.3 to 9.14.5. Fix split queries for topology to avoid page crash. Self Observability ui-template: Add new panels for monitor metrics aggregation queue used percentage and metrics persistent collection cached size. test: introduce and set up unit tests in the UI. test: implement comprehensive unit tests for components. refactor: optimize data types for widgets and dashboards. fix: optimize appearing the wrong prompt by pop-up for the HTTP environments in copy function. refactor the configuration view and implement the optional config for displaying timestamp in Log widget. test: implement unit tests for hooks and refactor some types. fix: share OAP proxy services for different endpoints and use health checked endpoints group. Optimize buttons in time picker component. Optimize the router system and implement unit tests for router. Bump element-plus from 2.9.4 to 2.11.0. Adapt new trace protocol and implement new trace view. Implement Trace page. Support collapsing and expanding for the event widget. UI-template: add BanyanDB and Elasticsearch write latency dashboards for OAP self observability. Documentation BanyanDB: Add Data Lifecycle Stages(Hot/Warm/Cold) documentation. Add SWIP-9 Support flink monitoring. Fix Metrics Attributes menu link. Implement the Status API on Settings page. Fix: Add the prefix for http url. Enhance the async-profiling duration options. Enhance the TTL Tab on Setting page. Fix the snapshot charts in alarm page. Fix Fluent Bit dead links. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1030\"\u003e10.3.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eBump up BanyanDB dependency version(server and java-client) to 0.9.0.\u003c/li\u003e\n\u003cli\u003eFix …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v11.0.0/en/changes/changes-10.3.0/","title":"10.3.0"},{"body":"10.4.0 Project Introduce OAL V2 engine:\nImmutable AST models for thread safety and predictable behavior Type-safe enums replacing string-based filter operators Precise error location reporting with file, line, and column numbers Clean separation between parsing and code generation phases Enhanced testability with models that can be constructed without parsing Introduce MAL/LAL/Hierarchy V2 engine — replace Groovy-based DSL runtime with ANTLR4 parser + Javassist bytecode generation:\nRemove Groovy runtime dependency from OAP backend Fail-fast compilation at startup — syntax and type errors are caught immediately instead of at first execution Thread-safe generated classes with no ThreadLocal or shared mutable state Immutable AST models for all three DSLs (MAL, LAL, Hierarchy rules) Explicit context passing replaces Groovy binding/closure capture v1 (Groovy) and v2 (ANTLR4+Javassist) cross-version checker validates behavioral equivalence across 1,290+ expressions JMH benchmarks confirm v2 runtime speedups: MAL execute ~6.8x, LAL compile ~39x / execute ~2.8x, Hierarchy execute ~2.6x faster than Groovy v1 Generated class names follow {yamlFileName}_L{lineNo}_{ruleName} pattern for all DSLs (MAL/LAL/Hierarchy) for stack trace traceability Breaking Change — LAL: remove slowSql {} and sampledTrace {} sub-DSLs from the grammar. These are replaced by the configurable outputType mechanism:\nSet outputType at the rule level in YAML config to specify the output entity class. Use the short name registered by LALOutputBuilder SPI (e.g., outputType: SlowSQL, outputType: SampledTrace), or a fully qualified class name as fallback. LALOutputBuilder implementations are discovered via ServiceLoader and expose a name() method for short name resolution. Built-in types: SlowSQL (DatabaseSlowStatementBuilder), SampledTrace (SampledTraceBuilder). Output fields (e.g., id, statement, latency) are now regular field assignments in the extractor block, no longer wrapped in sub-DSL blocks. Custom output fields are validated against the output type\u0026rsquo;s setters at compile time. An explicit sink {} block is now required for data to be persisted. Without sink {}, no data is saved — this applies to all LAL rules including those using outputType. In v1, slowSql {} and sampledTrace {} dispatched data as a side-effect inside the extractor; in v2, persistence is always handled by the sink pipeline. Output type resolution order: per-rule YAML outputType (short name via SPI or FQCN) \u0026gt; LALSourceTypeProvider SPI default \u0026gt; Log.class. All bundled LAL scripts (mysql-slowsql.yaml, pgsql-slowsql.yaml, redis-slowsql.yaml, envoy-als.yaml, k8s-service.yaml, mesh-dp.yaml) have been updated. Users with custom LAL scripts using slowSql {} or sampledTrace {} must migrate to the new syntax. See LAL documentation. Rename ExtractorSpec to MetricExtractor — now only handles LAL metrics {} blocks. Standard field setters (service, layer, timestamp, etc.) are compiled as direct setter calls on the output builder. Add def local variable support in LAL extractor (and filter level). Supports toJson() and toJsonArray() built-in functions for converting strings, Maps, and protobuf Struct to Gson JSON objects. Variables support null-safe navigation (?.), method chaining with compile-time type inference, and explicit type cast via as (built-in types or fully qualified class names, e.g., def resp = parsed?.response as io.envoyproxy.envoy.data.accesslog.v3.HTTPResponseProperties). Breaking Change — LALOutputBuilder.init() signature changed from init(LogData, NamingControl) to init(LogData, Optional\u0026lt;Object\u0026gt; extraLog, NamingControl). The extraLog parameter carries the typed input object (e.g., HTTPAccessLogEntry for envoy access logs) so that output builders can access protocol-specific fields. Custom LALOutputBuilder implementations must update their init() method signature. Fix E2E test metrics verify: make it failure if the metric values all null.\nSupport building, testing, and publishing with Java 25.\nAdd CLAUDE.md as AI assistant guide for the project.\nUpgrade Byte Buddy to 1.18.7 and configure explicit -javaagent for Mockito/Byte Buddy in Surefire to avoid JDK 25 dynamic agent loading warnings.\nUpgrade Groovy to 5.0.3 in OAP backend.\nBump up nodejs to v24.13.0 for the latest UI(booster-ui) compiling.\nDrop Elasticsearch 7.x (EOL) and OpenSearch 1.x from E2E tests, upgrade all ES tests to 8.18.8, and update skywalking-helm to use ECK 8.18.8.\nAdd library-batch-queue module — a partitioned, self-draining queue with type-based dispatch, adaptive partitioning, idle backoff, and throughput-weighted drain rebalancing (DrainBalancer). Designed to replace DataCarrier in high-fan-out scenarios.\nReplace DataCarrier with BatchQueue for L1 metrics aggregation, L2 metrics persistence, TopN persistence, all three exporters (gRPC metrics, Kafka trace, Kafka log), and gRPC remote client. All metric types (OAL + MAL) now share unified queues instead of separate OAL/MAL pools. Each exporter keeps its own dedicated queue with 1 thread, preserving original buffer strategies. Thread count comparison on an 8-core machine (gRPC remote client excluded — unchanged 1 thread per peer):\nQueue Old threads Old channels Old buffer slots New threads New partitions New buffer slots New policy L1 Aggregation (OAL) 24 ~1,240 ~12.4M 8 (unified) ~330 adaptive ~6.6M cpuCores(1.0) L1 Aggregation (MAL) 2 ~100 ~100K (unified above) L2 Persistence (OAL) 2 ~620 ~1.24M 3 (unified) ~330 adaptive ~660K cpuCoresWithBase(1, 0.25) L2 Persistence (MAL) 1 ~100 ~100K (unified above) TopN Persistence 4 4 4K 1 4 adaptive 4K fixed(1) Exporters (gRPC/Kafka) 3 6 120K 3 (1 per exporter) — 60K fixed(1) each Total 36 ~2,070 ~13.9M 15 ~664 ~7.3M Remove library-datacarrier-queue module. All usages have been replaced by library-batch-queue.\nEnable throughput-weighted drain rebalancing for L1 aggregation and L2 persistence queues (10s interval). Periodically reassigns partitions across drain threads to equalize load when metric types have skewed throughput.\nAdd benchmark framework under benchmarks/ with Kind-based Kubernetes environments, automated thread dump collection and analysis. First case: thread-analysis on istio-cluster_oap-banyandb environment.\nAdd virtual thread support (JDK 25+) for gRPC and Armeria HTTP server handler threads. Set SW_VIRTUAL_THREADS_ENABLED=false to disable.\nPool Threads (JDK \u0026lt; 25) Threads (JDK 25+) gRPC server handler (core-grpc, receiver-grpc, als-grpc, ebpf-grpc) Cached platform (unbounded) Virtual threads HTTP blocking (core-http, receiver-http, promql-http, logql-http, zipkin-query-http, zipkin-http, firehose-http) Cached platform (max 200) Virtual threads VT carrier threads (ForkJoinPool) N/A ~9 shared On JDK 25+, all 11 thread pools above share ~9 carrier threads instead of up to 1,400+ platform threads.\nChange default Docker base image to JDK 25 (eclipse-temurin:25-jre). JDK 11 kept as -java11 variant.\nThread count benchmark comparison — 2-node OAP cluster on JDK 25 with BanyanDB, Istio bookinfo traffic (10-core machine, JVM-internal threads excluded):\nPool v10.3.0 threads v10.4.0 threads Notes L1 Aggregation (OAL + MAL) 26 (DataCarrier) 10 (BatchQueue) Unified OAL + MAL L2 Persistence (OAL + MAL) 3 (DataCarrier) 4 (BatchQueue) Unified OAL + MAL TopN Persistence 4 (DataCarrier) 1 (BatchQueue) gRPC Remote Client 1 (DataCarrier) 1 (BatchQueue) Per peer Armeria HTTP event loop 20 5 min(5, cores) shared group Armeria HTTP handler on-demand platform(increasing with payload) - Virtual threads on JDK 25+ gRPC event loop 10 10 Unchanged gRPC handler on-demand platform(increasing with payload) - Virtual threads on JDK 25+ ForkJoinPool (Virtual Thread carrier) 0 ~10 JDK 25+ virtual thread scheduler HttpClient-SelectorManager 4 2 SharedKubernetesClient Schedulers + others ~24 ~24 Mostly unchanged Total (OAP threads) 150+ ~72 ~50% reduction, stable in high payload. Replace PowerMock Whitebox with standard Java Reflection in server-library, server-core, and server-configuration to support JDK 25+.\nFix /debugging/config/dump may leak sensitive information if there are second level properties in the configuration.\nOAP Server KubernetesCoordinator: make self instance return real pod IP address instead of 127.0.0.1. Fix KubernetesCoordinator self-endpoint race condition: include self in the endpoint list so DynamicEndpointGroup re-fires the listener when the self pod appears in the informer after initial sync. Enhance the alarm kernel with recovered status notification capability Fix BrowserWebVitalsPerfData clsTime to cls and make it double type. Init log-mal-rules at module provider start stage to avoid re-init for every LAL. Fail fast if SampleFamily is empty after MAL filter expression. Fix range matrix and scalar binary operation in PromQL. Add LatestLabeledFunction for meter. MAL Labeled metrics support additional attributes. Bump up netty to 4.2.9.Final. Add support for OpenSearch/ElasticSearch client certificate authentication. Fix BanyanDB logs paging query. Replace BanyanDB Java client with native implementation. Remove bydb.dependencies.properties and set the compatible BanyanDB API version number in ${SW_STORAGE_BANYANDB_COMPATIBLE_SERVER_API_VERSIONS}. Fix trace profiling query time range condition. Add named ThreadFactory to all Executors.newXxx() calls to replace anonymous pool-N-thread-M thread names with meaningful names for easier thread dump analysis. Complete OAP server thread inventory (counts on an 8-core machine, exporters and JDBC are optional): Catalog Thread Name Count Policy Partitions Data Pipeline BatchQueue-METRICS_L1_AGGREGATION-N 8 cpuCores(1.0) ~330 adaptive Data Pipeline BatchQueue-METRICS_L2_PERSISTENCE-N 3 cpuCoresWithBase(1, 0.25) ~330 adaptive Data Pipeline BatchQueue-TOPN_PERSISTENCE-N 1 fixed(1) ~4 adaptive Data Pipeline BatchQueue-GRPC_REMOTE_{host}_{port}-N 1 per peer fixed(1) fixed(1) Data Pipeline BatchQueue-EXPORTER_GRPC_METRICS-N 1 fixed(1) fixed(1) Data Pipeline BatchQueue-EXPORTER_KAFKA_TRACE-N 1 fixed(1) fixed(1) Data Pipeline BatchQueue-EXPORTER_KAFKA_LOG-N 1 fixed(1) fixed(1) Data Pipeline BatchQueue-JDBC_ASYNC_BATCH_PERSISTENT-N 4 (configurable) fixed(N) fixed(N) Scheduler RemoteClientManager 1 scheduled — Scheduler PersistenceTimer 1 scheduled — Scheduler PersistenceTimer-prepare-N 2 (configurable) fixed pool — Scheduler DataTTLKeeper 1 scheduled — Scheduler CacheUpdateTimer 1 scheduled — Scheduler HierarchyAutoMatching 1 scheduled — Scheduler WatermarkWatcher 1 scheduled — Scheduler AlarmCore 1 scheduled — Scheduler HealthChecker 1 scheduled — Scheduler EndpointUriRecognition 1 (conditional) scheduled — Scheduler FileChangeMonitor 1 scheduled — Scheduler BanyanDB-ChannelManager 1 scheduled — Scheduler GRPCClient-HealthCheck-{host}:{port} 1 per client scheduled — Scheduler EBPFProfiling-N configurable fixed pool — Fix BanyanDB time range overflow in profile thread snapshot query. BrowserErrorLog, OAP Server generated UUID to replace the original client side ID, because Browser scripts can\u0026rsquo;t guarantee generated IDs are globally unique. MQE: fix multiple labeled metric query and ensure no results are returned if no label value combinations match. Fix BrowserErrorLog BanyanDB storage query order. BanyanDB Client: Property query support Order By. MQE: trim the label values condition for the labeled metrics query to enhance the readability. PromQL service: fix time parse issue when using RFC3339 time format for querying. Envoy metrics service receiver: support adapter listener metrics. Envoy metrics service receiver: support config MAL rules files. Fix HttpAlarmCallback creating a new HttpClient on every alarm post() call, leaking NIO selector threads. Replace with a shared static singleton. Add SharedKubernetesClient singleton in library-kubernetes-support to replace 9 separate KubernetesClientBuilder().build() calls across 7 files. Fixes KubernetesCoordinator client leak (never closed, NIO selector thread persisted). Uses KubernetesHttpClientFactory with virtual threads on JDK 25+ or a single fixed executor thread on JDK \u0026lt;25. Reduce Armeria HTTP server event loop threads. All 7 HTTP servers now share one event loop group instead of each creating their own (Armeria default: cores * 2 per server = 140 on 10-core). Event loop: min(5, cores) shared — non-blocking I/O multiplexing needs few threads. Blocking executor: JDK 25+ uses virtual threads; JDK \u0026lt;25 keeps Armeria\u0026rsquo;s default cached pool (up to 200 on-demand threads) because HTTP handlers block on long storage/DB queries. Add the spring-ai components and the GenAI layer. Bump up netty to 4.2.10.Final. Bump up log4j to 2.25.3 and jackson to 2.18.5. Remove PowerMock dependency. Replace Whitebox with ReflectUtil (standard Java reflection + sun.misc.Unsafe for final fields) across all modules to support JDK 25+. Support TraceQL and Tempo API for Zipkin and SkyWalking native trace query. Remove initExp from MAL configuration. It was an internal Groovy startup validation mechanism, not an end-user feature. The v2 ANTLR4 compiler performs fail-fast validation at startup natively. Update hierarchy rule documentation: auto-matching-rules in hierarchy-definition.yml no longer use Groovy scripts. Rules now use a dedicated expression grammar supporting property access, String methods, if/else, comparisons, and logical operators. All shipped rules are fully compatible. Activate otlp-traces handler in receiver-otel by default. Update Istio E2E test versions: remove EOL 1.20.0, add 1.25.0–1.29.0 for ALS/Metrics/Ambient tests. Update Rover with Istio Process test from 1.15.0 to 1.28.0 with Kubernetes 1.28. Support Virtual-GenAI monitoring. Fix on-demand pod log parsing failure by replacing invalid DateTimeFormatter pattern with ISO_OFFSET_DATE_TIME. Fix Zipkin receiver compatibility with application/x-protobuf Content-Type. Support Envoy AI Gateway observability (SWIP-10): new ENVOY_AI_GATEWAY layer with MAL/LAL rules for GenAI metrics (token usage, latency, TTFT, TPOT) and access log sampling via OTLP. OTel metric receiver: convert data point attribute dots to underscores (consistent with resource attributes and metric names). Label mappings are now fallback-only — explicit job_name in resource attributes takes precedence over the service.name fallback. OTel log handler: prefer service.instance.id (OTel spec) over service.instance with fallback. Add SampleFamily.debugDump() for MAL debugging. Support virtual GenAI analysis for otlp and zipkin traces. UI Fix the missing icon in new native trace view. Enhance the alert page to show the recovery time of resolved alerts. Implement a common pagination component. Fix validation guard for router. Add the coldStage to the Duration for queries. Optimize the pages theme. Fix incorrect virtual service names. Add the GenAI icon to Topology. Bump up dependencies. Correct active/inactive text for the cold stage. Add the gen-ai menu. Fix: set the step to SECOND in the duration for Log/Trace/Alarm/Tag. Documentation Add benchmark selection into banyanDB storage documentation. Fix progressive TTL doc for banyanDB. Restructure docs/README.md for better navigation with high-level documentation overview. Move Marketplace as a top-level menu section with Overview introduction in menu.yml. Polish marketplace.md as the overview page for all out-of-box monitoring features. Add \u0026ldquo;What\u0026rsquo;s Next\u0026rdquo; section to Quick Start docs guiding users to Marketplace. Restructure agent compatibility page with OAP 10.x focus and clearer format for legacy versions. Remove outdated FAQ docs (v3, v6 upgrade guides and 7.x metrics issue). Remove \u0026ldquo;since 7/8/9.x\u0026rdquo; version statements from documentation as features are standard in 10.x. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1040\"\u003e10.4.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eIntroduce OAL V2 engine:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eImmutable AST models for thread safety and predictable …\u003c/li\u003e\u003c/ul\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/latest/en/changes/changes-10.4.0/","title":"10.4.0"},{"body":"10.4.0 Project Introduce OAL V2 engine:\nImmutable AST models for thread safety and predictable behavior Type-safe enums replacing string-based filter operators Precise error location reporting with file, line, and column numbers Clean separation between parsing and code generation phases Enhanced testability with models that can be constructed without parsing Introduce MAL/LAL/Hierarchy V2 engine — replace Groovy-based DSL runtime with ANTLR4 parser + Javassist bytecode generation:\nRemove Groovy runtime dependency from OAP backend Fail-fast compilation at startup — syntax and type errors are caught immediately instead of at first execution Thread-safe generated classes with no ThreadLocal or shared mutable state Immutable AST models for all three DSLs (MAL, LAL, Hierarchy rules) Explicit context passing replaces Groovy binding/closure capture v1 (Groovy) and v2 (ANTLR4+Javassist) cross-version checker validates behavioral equivalence across 1,290+ expressions JMH benchmarks confirm v2 runtime speedups: MAL execute ~6.8x, LAL compile ~39x / execute ~2.8x, Hierarchy execute ~2.6x faster than Groovy v1 Generated class names follow {yamlFileName}_L{lineNo}_{ruleName} pattern for all DSLs (MAL/LAL/Hierarchy) for stack trace traceability Breaking Change — LAL: remove slowSql {} and sampledTrace {} sub-DSLs from the grammar. These are replaced by the configurable outputType mechanism:\nSet outputType at the rule level in YAML config to specify the output entity class. Use the short name registered by LALOutputBuilder SPI (e.g., outputType: SlowSQL, outputType: SampledTrace), or a fully qualified class name as fallback. LALOutputBuilder implementations are discovered via ServiceLoader and expose a name() method for short name resolution. Built-in types: SlowSQL (DatabaseSlowStatementBuilder), SampledTrace (SampledTraceBuilder). Output fields (e.g., id, statement, latency) are now regular field assignments in the extractor block, no longer wrapped in sub-DSL blocks. Custom output fields are validated against the output type\u0026rsquo;s setters at compile time. An explicit sink {} block is now required for data to be persisted. Without sink {}, no data is saved — this applies to all LAL rules including those using outputType. In v1, slowSql {} and sampledTrace {} dispatched data as a side-effect inside the extractor; in v2, persistence is always handled by the sink pipeline. Output type resolution order: per-rule YAML outputType (short name via SPI or FQCN) \u0026gt; LALSourceTypeProvider SPI default \u0026gt; Log.class. All bundled LAL scripts (mysql-slowsql.yaml, pgsql-slowsql.yaml, redis-slowsql.yaml, envoy-als.yaml, k8s-service.yaml, mesh-dp.yaml) have been updated. Users with custom LAL scripts using slowSql {} or sampledTrace {} must migrate to the new syntax. See LAL documentation. Rename ExtractorSpec to MetricExtractor — now only handles LAL metrics {} blocks. Standard field setters (service, layer, timestamp, etc.) are compiled as direct setter calls on the output builder. Add def local variable support in LAL extractor (and filter level). Supports toJson() and toJsonArray() built-in functions for converting strings, Maps, and protobuf Struct to Gson JSON objects. Variables support null-safe navigation (?.), method chaining with compile-time type inference, and explicit type cast via as (built-in types or fully qualified class names, e.g., def resp = parsed?.response as io.envoyproxy.envoy.data.accesslog.v3.HTTPResponseProperties). Breaking Change — LALOutputBuilder.init() signature changed from init(LogData, NamingControl) to init(LogData, Optional\u0026lt;Object\u0026gt; extraLog, NamingControl). The extraLog parameter carries the typed input object (e.g., HTTPAccessLogEntry for envoy access logs) so that output builders can access protocol-specific fields. Custom LALOutputBuilder implementations must update their init() method signature. Fix E2E test metrics verify: make it failure if the metric values all null.\nSupport building, testing, and publishing with Java 25.\nAdd CLAUDE.md as AI assistant guide for the project.\nUpgrade Byte Buddy to 1.18.7 and configure explicit -javaagent for Mockito/Byte Buddy in Surefire to avoid JDK 25 dynamic agent loading warnings.\nUpgrade Groovy to 5.0.3 in OAP backend.\nBump up nodejs to v24.13.0 for the latest UI(booster-ui) compiling.\nDrop Elasticsearch 7.x (EOL) and OpenSearch 1.x from E2E tests, upgrade all ES tests to 8.18.8, and update skywalking-helm to use ECK 8.18.8.\nAdd library-batch-queue module — a partitioned, self-draining queue with type-based dispatch, adaptive partitioning, idle backoff, and throughput-weighted drain rebalancing (DrainBalancer). Designed to replace DataCarrier in high-fan-out scenarios.\nReplace DataCarrier with BatchQueue for L1 metrics aggregation, L2 metrics persistence, TopN persistence, all three exporters (gRPC metrics, Kafka trace, Kafka log), and gRPC remote client. All metric types (OAL + MAL) now share unified queues instead of separate OAL/MAL pools. Each exporter keeps its own dedicated queue with 1 thread, preserving original buffer strategies. Thread count comparison on an 8-core machine (gRPC remote client excluded — unchanged 1 thread per peer):\nQueue Old threads Old channels Old buffer slots New threads New partitions New buffer slots New policy L1 Aggregation (OAL) 24 ~1,240 ~12.4M 8 (unified) ~330 adaptive ~6.6M cpuCores(1.0) L1 Aggregation (MAL) 2 ~100 ~100K (unified above) L2 Persistence (OAL) 2 ~620 ~1.24M 3 (unified) ~330 adaptive ~660K cpuCoresWithBase(1, 0.25) L2 Persistence (MAL) 1 ~100 ~100K (unified above) TopN Persistence 4 4 4K 1 4 adaptive 4K fixed(1) Exporters (gRPC/Kafka) 3 6 120K 3 (1 per exporter) — 60K fixed(1) each Total 36 ~2,070 ~13.9M 15 ~664 ~7.3M Remove library-datacarrier-queue module. All usages have been replaced by library-batch-queue.\nEnable throughput-weighted drain rebalancing for L1 aggregation and L2 persistence queues (10s interval). Periodically reassigns partitions across drain threads to equalize load when metric types have skewed throughput.\nAdd benchmark framework under benchmarks/ with Kind-based Kubernetes environments, automated thread dump collection and analysis. First case: thread-analysis on istio-cluster_oap-banyandb environment.\nAdd virtual thread support (JDK 25+) for gRPC and Armeria HTTP server handler threads. Set SW_VIRTUAL_THREADS_ENABLED=false to disable.\nPool Threads (JDK \u0026lt; 25) Threads (JDK 25+) gRPC server handler (core-grpc, receiver-grpc, als-grpc, ebpf-grpc) Cached platform (unbounded) Virtual threads HTTP blocking (core-http, receiver-http, promql-http, logql-http, zipkin-query-http, zipkin-http, firehose-http) Cached platform (max 200) Virtual threads VT carrier threads (ForkJoinPool) N/A ~9 shared On JDK 25+, all 11 thread pools above share ~9 carrier threads instead of up to 1,400+ platform threads.\nChange default Docker base image to JDK 25 (eclipse-temurin:25-jre). JDK 11 kept as -java11 variant.\nThread count benchmark comparison — 2-node OAP cluster on JDK 25 with BanyanDB, Istio bookinfo traffic (10-core machine, JVM-internal threads excluded):\nPool v10.3.0 threads v10.4.0 threads Notes L1 Aggregation (OAL + MAL) 26 (DataCarrier) 10 (BatchQueue) Unified OAL + MAL L2 Persistence (OAL + MAL) 3 (DataCarrier) 4 (BatchQueue) Unified OAL + MAL TopN Persistence 4 (DataCarrier) 1 (BatchQueue) gRPC Remote Client 1 (DataCarrier) 1 (BatchQueue) Per peer Armeria HTTP event loop 20 5 min(5, cores) shared group Armeria HTTP handler on-demand platform(increasing with payload) - Virtual threads on JDK 25+ gRPC event loop 10 10 Unchanged gRPC handler on-demand platform(increasing with payload) - Virtual threads on JDK 25+ ForkJoinPool (Virtual Thread carrier) 0 ~10 JDK 25+ virtual thread scheduler HttpClient-SelectorManager 4 2 SharedKubernetesClient Schedulers + others ~24 ~24 Mostly unchanged Total (OAP threads) 150+ ~72 ~50% reduction, stable in high payload. Replace PowerMock Whitebox with standard Java Reflection in server-library, server-core, and server-configuration to support JDK 25+.\nFix /debugging/config/dump may leak sensitive information if there are second level properties in the configuration.\nOAP Server KubernetesCoordinator: make self instance return real pod IP address instead of 127.0.0.1. Fix KubernetesCoordinator self-endpoint race condition: include self in the endpoint list so DynamicEndpointGroup re-fires the listener when the self pod appears in the informer after initial sync. Enhance the alarm kernel with recovered status notification capability Fix BrowserWebVitalsPerfData clsTime to cls and make it double type. Init log-mal-rules at module provider start stage to avoid re-init for every LAL. Fail fast if SampleFamily is empty after MAL filter expression. Fix range matrix and scalar binary operation in PromQL. Add LatestLabeledFunction for meter. MAL Labeled metrics support additional attributes. Bump up netty to 4.2.9.Final. Add support for OpenSearch/ElasticSearch client certificate authentication. Fix BanyanDB logs paging query. Replace BanyanDB Java client with native implementation. Remove bydb.dependencies.properties and set the compatible BanyanDB API version number in ${SW_STORAGE_BANYANDB_COMPATIBLE_SERVER_API_VERSIONS}. Fix trace profiling query time range condition. Add named ThreadFactory to all Executors.newXxx() calls to replace anonymous pool-N-thread-M thread names with meaningful names for easier thread dump analysis. Complete OAP server thread inventory (counts on an 8-core machine, exporters and JDBC are optional): Catalog Thread Name Count Policy Partitions Data Pipeline BatchQueue-METRICS_L1_AGGREGATION-N 8 cpuCores(1.0) ~330 adaptive Data Pipeline BatchQueue-METRICS_L2_PERSISTENCE-N 3 cpuCoresWithBase(1, 0.25) ~330 adaptive Data Pipeline BatchQueue-TOPN_PERSISTENCE-N 1 fixed(1) ~4 adaptive Data Pipeline BatchQueue-GRPC_REMOTE_{host}_{port}-N 1 per peer fixed(1) fixed(1) Data Pipeline BatchQueue-EXPORTER_GRPC_METRICS-N 1 fixed(1) fixed(1) Data Pipeline BatchQueue-EXPORTER_KAFKA_TRACE-N 1 fixed(1) fixed(1) Data Pipeline BatchQueue-EXPORTER_KAFKA_LOG-N 1 fixed(1) fixed(1) Data Pipeline BatchQueue-JDBC_ASYNC_BATCH_PERSISTENT-N 4 (configurable) fixed(N) fixed(N) Scheduler RemoteClientManager 1 scheduled — Scheduler PersistenceTimer 1 scheduled — Scheduler PersistenceTimer-prepare-N 2 (configurable) fixed pool — Scheduler DataTTLKeeper 1 scheduled — Scheduler CacheUpdateTimer 1 scheduled — Scheduler HierarchyAutoMatching 1 scheduled — Scheduler WatermarkWatcher 1 scheduled — Scheduler AlarmCore 1 scheduled — Scheduler HealthChecker 1 scheduled — Scheduler EndpointUriRecognition 1 (conditional) scheduled — Scheduler FileChangeMonitor 1 scheduled — Scheduler BanyanDB-ChannelManager 1 scheduled — Scheduler GRPCClient-HealthCheck-{host}:{port} 1 per client scheduled — Scheduler EBPFProfiling-N configurable fixed pool — Fix BanyanDB time range overflow in profile thread snapshot query. BrowserErrorLog, OAP Server generated UUID to replace the original client side ID, because Browser scripts can\u0026rsquo;t guarantee generated IDs are globally unique. MQE: fix multiple labeled metric query and ensure no results are returned if no label value combinations match. Fix BrowserErrorLog BanyanDB storage query order. BanyanDB Client: Property query support Order By. MQE: trim the label values condition for the labeled metrics query to enhance the readability. PromQL service: fix time parse issue when using RFC3339 time format for querying. Envoy metrics service receiver: support adapter listener metrics. Envoy metrics service receiver: support config MAL rules files. Fix HttpAlarmCallback creating a new HttpClient on every alarm post() call, leaking NIO selector threads. Replace with a shared static singleton. Add SharedKubernetesClient singleton in library-kubernetes-support to replace 9 separate KubernetesClientBuilder().build() calls across 7 files. Fixes KubernetesCoordinator client leak (never closed, NIO selector thread persisted). Uses KubernetesHttpClientFactory with virtual threads on JDK 25+ or a single fixed executor thread on JDK \u0026lt;25. Reduce Armeria HTTP server event loop threads. All 7 HTTP servers now share one event loop group instead of each creating their own (Armeria default: cores * 2 per server = 140 on 10-core). Event loop: min(5, cores) shared — non-blocking I/O multiplexing needs few threads. Blocking executor: JDK 25+ uses virtual threads; JDK \u0026lt;25 keeps Armeria\u0026rsquo;s default cached pool (up to 200 on-demand threads) because HTTP handlers block on long storage/DB queries. Add the spring-ai components and the GenAI layer. Bump up netty to 4.2.10.Final. Bump up log4j to 2.25.3 and jackson to 2.18.5. Remove PowerMock dependency. Replace Whitebox with ReflectUtil (standard Java reflection + sun.misc.Unsafe for final fields) across all modules to support JDK 25+. Support TraceQL and Tempo API for Zipkin and SkyWalking native trace query. Remove initExp from MAL configuration. It was an internal Groovy startup validation mechanism, not an end-user feature. The v2 ANTLR4 compiler performs fail-fast validation at startup natively. Update hierarchy rule documentation: auto-matching-rules in hierarchy-definition.yml no longer use Groovy scripts. Rules now use a dedicated expression grammar supporting property access, String methods, if/else, comparisons, and logical operators. All shipped rules are fully compatible. Activate otlp-traces handler in receiver-otel by default. Update Istio E2E test versions: remove EOL 1.20.0, add 1.25.0–1.29.0 for ALS/Metrics/Ambient tests. Update Rover with Istio Process test from 1.15.0 to 1.28.0 with Kubernetes 1.28. Support Virtual-GenAI monitoring. Fix on-demand pod log parsing failure by replacing invalid DateTimeFormatter pattern with ISO_OFFSET_DATE_TIME. Fix Zipkin receiver compatibility with application/x-protobuf Content-Type. Support Envoy AI Gateway observability (SWIP-10): new ENVOY_AI_GATEWAY layer with MAL/LAL rules for GenAI metrics (token usage, latency, TTFT, TPOT) and access log sampling via OTLP. OTel metric receiver: convert data point attribute dots to underscores (consistent with resource attributes and metric names). Label mappings are now fallback-only — explicit job_name in resource attributes takes precedence over the service.name fallback. OTel log handler: prefer service.instance.id (OTel spec) over service.instance with fallback. Add SampleFamily.debugDump() for MAL debugging. Support virtual GenAI analysis for otlp and zipkin traces. UI Fix the missing icon in new native trace view. Enhance the alert page to show the recovery time of resolved alerts. Implement a common pagination component. Fix validation guard for router. Add the coldStage to the Duration for queries. Optimize the pages theme. Fix incorrect virtual service names. Add the GenAI icon to Topology. Bump up dependencies. Correct active/inactive text for the cold stage. Add the gen-ai menu. Fix: set the step to SECOND in the duration for Log/Trace/Alarm/Tag. Documentation Add benchmark selection into banyanDB storage documentation. Fix progressive TTL doc for banyanDB. Restructure docs/README.md for better navigation with high-level documentation overview. Move Marketplace as a top-level menu section with Overview introduction in menu.yml. Polish marketplace.md as the overview page for all out-of-box monitoring features. Add \u0026ldquo;What\u0026rsquo;s Next\u0026rdquo; section to Quick Start docs guiding users to Marketplace. Restructure agent compatibility page with OAP 10.x focus and clearer format for legacy versions. Remove outdated FAQ docs (v3, v6 upgrade guides and 7.x metrics issue). Remove \u0026ldquo;since 7/8/9.x\u0026rdquo; version statements from documentation as features are standard in 10.x. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1040\"\u003e10.4.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eIntroduce OAL V2 engine:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eImmutable AST models for thread safety and predictable …\u003c/li\u003e\u003c/ul\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/next/en/changes/changes-10.4.0/","title":"10.4.0"},{"body":"10.4.0 Project Introduce OAL V2 engine:\nImmutable AST models for thread safety and predictable behavior Type-safe enums replacing string-based filter operators Precise error location reporting with file, line, and column numbers Clean separation between parsing and code generation phases Enhanced testability with models that can be constructed without parsing Introduce MAL/LAL/Hierarchy V2 engine — replace Groovy-based DSL runtime with ANTLR4 parser + Javassist bytecode generation:\nRemove Groovy runtime dependency from OAP backend Fail-fast compilation at startup — syntax and type errors are caught immediately instead of at first execution Thread-safe generated classes with no ThreadLocal or shared mutable state Immutable AST models for all three DSLs (MAL, LAL, Hierarchy rules) Explicit context passing replaces Groovy binding/closure capture v1 (Groovy) and v2 (ANTLR4+Javassist) cross-version checker validates behavioral equivalence across 1,290+ expressions JMH benchmarks confirm v2 runtime speedups: MAL execute ~6.8x, LAL compile ~39x / execute ~2.8x, Hierarchy execute ~2.6x faster than Groovy v1 Generated class names follow {yamlFileName}_L{lineNo}_{ruleName} pattern for all DSLs (MAL/LAL/Hierarchy) for stack trace traceability Breaking Change — LAL: remove slowSql {} and sampledTrace {} sub-DSLs from the grammar. These are replaced by the configurable outputType mechanism:\nSet outputType at the rule level in YAML config to specify the output entity class. Use the short name registered by LALOutputBuilder SPI (e.g., outputType: SlowSQL, outputType: SampledTrace), or a fully qualified class name as fallback. LALOutputBuilder implementations are discovered via ServiceLoader and expose a name() method for short name resolution. Built-in types: SlowSQL (DatabaseSlowStatementBuilder), SampledTrace (SampledTraceBuilder). Output fields (e.g., id, statement, latency) are now regular field assignments in the extractor block, no longer wrapped in sub-DSL blocks. Custom output fields are validated against the output type\u0026rsquo;s setters at compile time. An explicit sink {} block is now required for data to be persisted. Without sink {}, no data is saved — this applies to all LAL rules including those using outputType. In v1, slowSql {} and sampledTrace {} dispatched data as a side-effect inside the extractor; in v2, persistence is always handled by the sink pipeline. Output type resolution order: per-rule YAML outputType (short name via SPI or FQCN) \u0026gt; LALSourceTypeProvider SPI default \u0026gt; Log.class. All bundled LAL scripts (mysql-slowsql.yaml, pgsql-slowsql.yaml, redis-slowsql.yaml, envoy-als.yaml, k8s-service.yaml, mesh-dp.yaml) have been updated. Users with custom LAL scripts using slowSql {} or sampledTrace {} must migrate to the new syntax. See LAL documentation. Rename ExtractorSpec to MetricExtractor — now only handles LAL metrics {} blocks. Standard field setters (service, layer, timestamp, etc.) are compiled as direct setter calls on the output builder. Add def local variable support in LAL extractor (and filter level). Supports toJson() and toJsonArray() built-in functions for converting strings, Maps, and protobuf Struct to Gson JSON objects. Variables support null-safe navigation (?.), method chaining with compile-time type inference, and explicit type cast via as (built-in types or fully qualified class names, e.g., def resp = parsed?.response as io.envoyproxy.envoy.data.accesslog.v3.HTTPResponseProperties). Breaking Change — LALOutputBuilder.init() signature changed from init(LogData, NamingControl) to init(LogData, Optional\u0026lt;Object\u0026gt; extraLog, NamingControl). The extraLog parameter carries the typed input object (e.g., HTTPAccessLogEntry for envoy access logs) so that output builders can access protocol-specific fields. Custom LALOutputBuilder implementations must update their init() method signature. Fix E2E test metrics verify: make it failure if the metric values all null.\nSupport building, testing, and publishing with Java 25.\nAdd CLAUDE.md as AI assistant guide for the project.\nUpgrade Byte Buddy to 1.18.7 and configure explicit -javaagent for Mockito/Byte Buddy in Surefire to avoid JDK 25 dynamic agent loading warnings.\nUpgrade Groovy to 5.0.3 in OAP backend.\nBump up nodejs to v24.13.0 for the latest UI(booster-ui) compiling.\nDrop Elasticsearch 7.x (EOL) and OpenSearch 1.x from E2E tests, upgrade all ES tests to 8.18.8, and update skywalking-helm to use ECK 8.18.8.\nAdd library-batch-queue module — a partitioned, self-draining queue with type-based dispatch, adaptive partitioning, idle backoff, and throughput-weighted drain rebalancing (DrainBalancer). Designed to replace DataCarrier in high-fan-out scenarios.\nReplace DataCarrier with BatchQueue for L1 metrics aggregation, L2 metrics persistence, TopN persistence, all three exporters (gRPC metrics, Kafka trace, Kafka log), and gRPC remote client. All metric types (OAL + MAL) now share unified queues instead of separate OAL/MAL pools. Each exporter keeps its own dedicated queue with 1 thread, preserving original buffer strategies. Thread count comparison on an 8-core machine (gRPC remote client excluded — unchanged 1 thread per peer):\nQueue Old threads Old channels Old buffer slots New threads New partitions New buffer slots New policy L1 Aggregation (OAL) 24 ~1,240 ~12.4M 8 (unified) ~330 adaptive ~6.6M cpuCores(1.0) L1 Aggregation (MAL) 2 ~100 ~100K (unified above) L2 Persistence (OAL) 2 ~620 ~1.24M 3 (unified) ~330 adaptive ~660K cpuCoresWithBase(1, 0.25) L2 Persistence (MAL) 1 ~100 ~100K (unified above) TopN Persistence 4 4 4K 1 4 adaptive 4K fixed(1) Exporters (gRPC/Kafka) 3 6 120K 3 (1 per exporter) — 60K fixed(1) each Total 36 ~2,070 ~13.9M 15 ~664 ~7.3M Remove library-datacarrier-queue module. All usages have been replaced by library-batch-queue.\nEnable throughput-weighted drain rebalancing for L1 aggregation and L2 persistence queues (10s interval). Periodically reassigns partitions across drain threads to equalize load when metric types have skewed throughput.\nAdd benchmark framework under benchmarks/ with Kind-based Kubernetes environments, automated thread dump collection and analysis. First case: thread-analysis on istio-cluster_oap-banyandb environment.\nAdd virtual thread support (JDK 25+) for gRPC and Armeria HTTP server handler threads. Set SW_VIRTUAL_THREADS_ENABLED=false to disable.\nPool Threads (JDK \u0026lt; 25) Threads (JDK 25+) gRPC server handler (core-grpc, receiver-grpc, als-grpc, ebpf-grpc) Cached platform (unbounded) Virtual threads HTTP blocking (core-http, receiver-http, promql-http, logql-http, zipkin-query-http, zipkin-http, firehose-http) Cached platform (max 200) Virtual threads VT carrier threads (ForkJoinPool) N/A ~9 shared On JDK 25+, all 11 thread pools above share ~9 carrier threads instead of up to 1,400+ platform threads.\nChange default Docker base image to JDK 25 (eclipse-temurin:25-jre). JDK 11 kept as -java11 variant.\nThread count benchmark comparison — 2-node OAP cluster on JDK 25 with BanyanDB, Istio bookinfo traffic (10-core machine, JVM-internal threads excluded):\nPool v10.3.0 threads v10.4.0 threads Notes L1 Aggregation (OAL + MAL) 26 (DataCarrier) 10 (BatchQueue) Unified OAL + MAL L2 Persistence (OAL + MAL) 3 (DataCarrier) 4 (BatchQueue) Unified OAL + MAL TopN Persistence 4 (DataCarrier) 1 (BatchQueue) gRPC Remote Client 1 (DataCarrier) 1 (BatchQueue) Per peer Armeria HTTP event loop 20 5 min(5, cores) shared group Armeria HTTP handler on-demand platform(increasing with payload) - Virtual threads on JDK 25+ gRPC event loop 10 10 Unchanged gRPC handler on-demand platform(increasing with payload) - Virtual threads on JDK 25+ ForkJoinPool (Virtual Thread carrier) 0 ~10 JDK 25+ virtual thread scheduler HttpClient-SelectorManager 4 2 SharedKubernetesClient Schedulers + others ~24 ~24 Mostly unchanged Total (OAP threads) 150+ ~72 ~50% reduction, stable in high payload. Replace PowerMock Whitebox with standard Java Reflection in server-library, server-core, and server-configuration to support JDK 25+.\nFix /debugging/config/dump may leak sensitive information if there are second level properties in the configuration.\nOAP Server KubernetesCoordinator: make self instance return real pod IP address instead of 127.0.0.1. Fix KubernetesCoordinator self-endpoint race condition: include self in the endpoint list so DynamicEndpointGroup re-fires the listener when the self pod appears in the informer after initial sync. Enhance the alarm kernel with recovered status notification capability Fix BrowserWebVitalsPerfData clsTime to cls and make it double type. Init log-mal-rules at module provider start stage to avoid re-init for every LAL. Fail fast if SampleFamily is empty after MAL filter expression. Fix range matrix and scalar binary operation in PromQL. Add LatestLabeledFunction for meter. MAL Labeled metrics support additional attributes. Bump up netty to 4.2.9.Final. Add support for OpenSearch/ElasticSearch client certificate authentication. Fix BanyanDB logs paging query. Replace BanyanDB Java client with native implementation. Remove bydb.dependencies.properties and set the compatible BanyanDB API version number in ${SW_STORAGE_BANYANDB_COMPATIBLE_SERVER_API_VERSIONS}. Fix trace profiling query time range condition. Add named ThreadFactory to all Executors.newXxx() calls to replace anonymous pool-N-thread-M thread names with meaningful names for easier thread dump analysis. Complete OAP server thread inventory (counts on an 8-core machine, exporters and JDBC are optional): Catalog Thread Name Count Policy Partitions Data Pipeline BatchQueue-METRICS_L1_AGGREGATION-N 8 cpuCores(1.0) ~330 adaptive Data Pipeline BatchQueue-METRICS_L2_PERSISTENCE-N 3 cpuCoresWithBase(1, 0.25) ~330 adaptive Data Pipeline BatchQueue-TOPN_PERSISTENCE-N 1 fixed(1) ~4 adaptive Data Pipeline BatchQueue-GRPC_REMOTE_{host}_{port}-N 1 per peer fixed(1) fixed(1) Data Pipeline BatchQueue-EXPORTER_GRPC_METRICS-N 1 fixed(1) fixed(1) Data Pipeline BatchQueue-EXPORTER_KAFKA_TRACE-N 1 fixed(1) fixed(1) Data Pipeline BatchQueue-EXPORTER_KAFKA_LOG-N 1 fixed(1) fixed(1) Data Pipeline BatchQueue-JDBC_ASYNC_BATCH_PERSISTENT-N 4 (configurable) fixed(N) fixed(N) Scheduler RemoteClientManager 1 scheduled — Scheduler PersistenceTimer 1 scheduled — Scheduler PersistenceTimer-prepare-N 2 (configurable) fixed pool — Scheduler DataTTLKeeper 1 scheduled — Scheduler CacheUpdateTimer 1 scheduled — Scheduler HierarchyAutoMatching 1 scheduled — Scheduler WatermarkWatcher 1 scheduled — Scheduler AlarmCore 1 scheduled — Scheduler HealthChecker 1 scheduled — Scheduler EndpointUriRecognition 1 (conditional) scheduled — Scheduler FileChangeMonitor 1 scheduled — Scheduler BanyanDB-ChannelManager 1 scheduled — Scheduler GRPCClient-HealthCheck-{host}:{port} 1 per client scheduled — Scheduler EBPFProfiling-N configurable fixed pool — Fix BanyanDB time range overflow in profile thread snapshot query. BrowserErrorLog, OAP Server generated UUID to replace the original client side ID, because Browser scripts can\u0026rsquo;t guarantee generated IDs are globally unique. MQE: fix multiple labeled metric query and ensure no results are returned if no label value combinations match. Fix BrowserErrorLog BanyanDB storage query order. BanyanDB Client: Property query support Order By. MQE: trim the label values condition for the labeled metrics query to enhance the readability. PromQL service: fix time parse issue when using RFC3339 time format for querying. Envoy metrics service receiver: support adapter listener metrics. Envoy metrics service receiver: support config MAL rules files. Fix HttpAlarmCallback creating a new HttpClient on every alarm post() call, leaking NIO selector threads. Replace with a shared static singleton. Add SharedKubernetesClient singleton in library-kubernetes-support to replace 9 separate KubernetesClientBuilder().build() calls across 7 files. Fixes KubernetesCoordinator client leak (never closed, NIO selector thread persisted). Uses KubernetesHttpClientFactory with virtual threads on JDK 25+ or a single fixed executor thread on JDK \u0026lt;25. Reduce Armeria HTTP server event loop threads. All 7 HTTP servers now share one event loop group instead of each creating their own (Armeria default: cores * 2 per server = 140 on 10-core). Event loop: min(5, cores) shared — non-blocking I/O multiplexing needs few threads. Blocking executor: JDK 25+ uses virtual threads; JDK \u0026lt;25 keeps Armeria\u0026rsquo;s default cached pool (up to 200 on-demand threads) because HTTP handlers block on long storage/DB queries. Add the spring-ai components and the GenAI layer. Bump up netty to 4.2.10.Final. Bump up log4j to 2.25.3 and jackson to 2.18.5. Remove PowerMock dependency. Replace Whitebox with ReflectUtil (standard Java reflection + sun.misc.Unsafe for final fields) across all modules to support JDK 25+. Support TraceQL and Tempo API for Zipkin and SkyWalking native trace query. Remove initExp from MAL configuration. It was an internal Groovy startup validation mechanism, not an end-user feature. The v2 ANTLR4 compiler performs fail-fast validation at startup natively. Update hierarchy rule documentation: auto-matching-rules in hierarchy-definition.yml no longer use Groovy scripts. Rules now use a dedicated expression grammar supporting property access, String methods, if/else, comparisons, and logical operators. All shipped rules are fully compatible. Activate otlp-traces handler in receiver-otel by default. Update Istio E2E test versions: remove EOL 1.20.0, add 1.25.0–1.29.0 for ALS/Metrics/Ambient tests. Update Rover with Istio Process test from 1.15.0 to 1.28.0 with Kubernetes 1.28. Support Virtual-GenAI monitoring. Fix on-demand pod log parsing failure by replacing invalid DateTimeFormatter pattern with ISO_OFFSET_DATE_TIME. Fix Zipkin receiver compatibility with application/x-protobuf Content-Type. Support Envoy AI Gateway observability (SWIP-10): new ENVOY_AI_GATEWAY layer with MAL/LAL rules for GenAI metrics (token usage, latency, TTFT, TPOT) and access log sampling via OTLP. OTel metric receiver: convert data point attribute dots to underscores (consistent with resource attributes and metric names). Label mappings are now fallback-only — explicit job_name in resource attributes takes precedence over the service.name fallback. OTel log handler: prefer service.instance.id (OTel spec) over service.instance with fallback. Add SampleFamily.debugDump() for MAL debugging. Support virtual GenAI analysis for otlp and zipkin traces. UI Fix the missing icon in new native trace view. Enhance the alert page to show the recovery time of resolved alerts. Implement a common pagination component. Fix validation guard for router. Add the coldStage to the Duration for queries. Optimize the pages theme. Fix incorrect virtual service names. Add the GenAI icon to Topology. Bump up dependencies. Correct active/inactive text for the cold stage. Add the gen-ai menu. Fix: set the step to SECOND in the duration for Log/Trace/Alarm/Tag. Documentation Add benchmark selection into banyanDB storage documentation. Fix progressive TTL doc for banyanDB. Restructure docs/README.md for better navigation with high-level documentation overview. Move Marketplace as a top-level menu section with Overview introduction in menu.yml. Polish marketplace.md as the overview page for all out-of-box monitoring features. Add \u0026ldquo;What\u0026rsquo;s Next\u0026rdquo; section to Quick Start docs guiding users to Marketplace. Restructure agent compatibility page with OAP 10.x focus and clearer format for legacy versions. Remove outdated FAQ docs (v3, v6 upgrade guides and 7.x metrics issue). Remove \u0026ldquo;since 7/8/9.x\u0026rdquo; version statements from documentation as features are standard in 10.x. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1040\"\u003e10.4.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eIntroduce OAL V2 engine:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eImmutable AST models for thread safety and predictable …\u003c/li\u003e\u003c/ul\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.4.0/en/changes/changes/","title":"10.4.0"},{"body":"10.4.0 Project Introduce OAL V2 engine:\nImmutable AST models for thread safety and predictable behavior Type-safe enums replacing string-based filter operators Precise error location reporting with file, line, and column numbers Clean separation between parsing and code generation phases Enhanced testability with models that can be constructed without parsing Introduce MAL/LAL/Hierarchy V2 engine — replace Groovy-based DSL runtime with ANTLR4 parser + Javassist bytecode generation:\nRemove Groovy runtime dependency from OAP backend Fail-fast compilation at startup — syntax and type errors are caught immediately instead of at first execution Thread-safe generated classes with no ThreadLocal or shared mutable state Immutable AST models for all three DSLs (MAL, LAL, Hierarchy rules) Explicit context passing replaces Groovy binding/closure capture v1 (Groovy) and v2 (ANTLR4+Javassist) cross-version checker validates behavioral equivalence across 1,290+ expressions JMH benchmarks confirm v2 runtime speedups: MAL execute ~6.8x, LAL compile ~39x / execute ~2.8x, Hierarchy execute ~2.6x faster than Groovy v1 Generated class names follow {yamlFileName}_L{lineNo}_{ruleName} pattern for all DSLs (MAL/LAL/Hierarchy) for stack trace traceability Breaking Change — LAL: remove slowSql {} and sampledTrace {} sub-DSLs from the grammar. These are replaced by the configurable outputType mechanism:\nSet outputType at the rule level in YAML config to specify the output entity class. Use the short name registered by LALOutputBuilder SPI (e.g., outputType: SlowSQL, outputType: SampledTrace), or a fully qualified class name as fallback. LALOutputBuilder implementations are discovered via ServiceLoader and expose a name() method for short name resolution. Built-in types: SlowSQL (DatabaseSlowStatementBuilder), SampledTrace (SampledTraceBuilder). Output fields (e.g., id, statement, latency) are now regular field assignments in the extractor block, no longer wrapped in sub-DSL blocks. Custom output fields are validated against the output type\u0026rsquo;s setters at compile time. An explicit sink {} block is now required for data to be persisted. Without sink {}, no data is saved — this applies to all LAL rules including those using outputType. In v1, slowSql {} and sampledTrace {} dispatched data as a side-effect inside the extractor; in v2, persistence is always handled by the sink pipeline. Output type resolution order: per-rule YAML outputType (short name via SPI or FQCN) \u0026gt; LALSourceTypeProvider SPI default \u0026gt; Log.class. All bundled LAL scripts (mysql-slowsql.yaml, pgsql-slowsql.yaml, redis-slowsql.yaml, envoy-als.yaml, k8s-service.yaml, mesh-dp.yaml) have been updated. Users with custom LAL scripts using slowSql {} or sampledTrace {} must migrate to the new syntax. See LAL documentation. Rename ExtractorSpec to MetricExtractor — now only handles LAL metrics {} blocks. Standard field setters (service, layer, timestamp, etc.) are compiled as direct setter calls on the output builder. Add def local variable support in LAL extractor (and filter level). Supports toJson() and toJsonArray() built-in functions for converting strings, Maps, and protobuf Struct to Gson JSON objects. Variables support null-safe navigation (?.), method chaining with compile-time type inference, and explicit type cast via as (built-in types or fully qualified class names, e.g., def resp = parsed?.response as io.envoyproxy.envoy.data.accesslog.v3.HTTPResponseProperties). Breaking Change — LALOutputBuilder.init() signature changed from init(LogData, NamingControl) to init(LogData, Optional\u0026lt;Object\u0026gt; extraLog, NamingControl). The extraLog parameter carries the typed input object (e.g., HTTPAccessLogEntry for envoy access logs) so that output builders can access protocol-specific fields. Custom LALOutputBuilder implementations must update their init() method signature. Fix E2E test metrics verify: make it failure if the metric values all null.\nSupport building, testing, and publishing with Java 25.\nAdd CLAUDE.md as AI assistant guide for the project.\nUpgrade Byte Buddy to 1.18.7 and configure explicit -javaagent for Mockito/Byte Buddy in Surefire to avoid JDK 25 dynamic agent loading warnings.\nUpgrade Groovy to 5.0.3 in OAP backend.\nBump up nodejs to v24.13.0 for the latest UI(booster-ui) compiling.\nDrop Elasticsearch 7.x (EOL) and OpenSearch 1.x from E2E tests, upgrade all ES tests to 8.18.8, and update skywalking-helm to use ECK 8.18.8.\nAdd library-batch-queue module — a partitioned, self-draining queue with type-based dispatch, adaptive partitioning, idle backoff, and throughput-weighted drain rebalancing (DrainBalancer). Designed to replace DataCarrier in high-fan-out scenarios.\nReplace DataCarrier with BatchQueue for L1 metrics aggregation, L2 metrics persistence, TopN persistence, all three exporters (gRPC metrics, Kafka trace, Kafka log), and gRPC remote client. All metric types (OAL + MAL) now share unified queues instead of separate OAL/MAL pools. Each exporter keeps its own dedicated queue with 1 thread, preserving original buffer strategies. Thread count comparison on an 8-core machine (gRPC remote client excluded — unchanged 1 thread per peer):\nQueue Old threads Old channels Old buffer slots New threads New partitions New buffer slots New policy L1 Aggregation (OAL) 24 ~1,240 ~12.4M 8 (unified) ~330 adaptive ~6.6M cpuCores(1.0) L1 Aggregation (MAL) 2 ~100 ~100K (unified above) L2 Persistence (OAL) 2 ~620 ~1.24M 3 (unified) ~330 adaptive ~660K cpuCoresWithBase(1, 0.25) L2 Persistence (MAL) 1 ~100 ~100K (unified above) TopN Persistence 4 4 4K 1 4 adaptive 4K fixed(1) Exporters (gRPC/Kafka) 3 6 120K 3 (1 per exporter) — 60K fixed(1) each Total 36 ~2,070 ~13.9M 15 ~664 ~7.3M Remove library-datacarrier-queue module. All usages have been replaced by library-batch-queue.\nEnable throughput-weighted drain rebalancing for L1 aggregation and L2 persistence queues (10s interval). Periodically reassigns partitions across drain threads to equalize load when metric types have skewed throughput.\nAdd benchmark framework under benchmarks/ with Kind-based Kubernetes environments, automated thread dump collection and analysis. First case: thread-analysis on istio-cluster_oap-banyandb environment.\nAdd virtual thread support (JDK 25+) for gRPC and Armeria HTTP server handler threads. Set SW_VIRTUAL_THREADS_ENABLED=false to disable.\nPool Threads (JDK \u0026lt; 25) Threads (JDK 25+) gRPC server handler (core-grpc, receiver-grpc, als-grpc, ebpf-grpc) Cached platform (unbounded) Virtual threads HTTP blocking (core-http, receiver-http, promql-http, logql-http, zipkin-query-http, zipkin-http, firehose-http) Cached platform (max 200) Virtual threads VT carrier threads (ForkJoinPool) N/A ~9 shared On JDK 25+, all 11 thread pools above share ~9 carrier threads instead of up to 1,400+ platform threads.\nChange default Docker base image to JDK 25 (eclipse-temurin:25-jre). JDK 11 kept as -java11 variant.\nThread count benchmark comparison — 2-node OAP cluster on JDK 25 with BanyanDB, Istio bookinfo traffic (10-core machine, JVM-internal threads excluded):\nPool v10.3.0 threads v10.4.0 threads Notes L1 Aggregation (OAL + MAL) 26 (DataCarrier) 10 (BatchQueue) Unified OAL + MAL L2 Persistence (OAL + MAL) 3 (DataCarrier) 4 (BatchQueue) Unified OAL + MAL TopN Persistence 4 (DataCarrier) 1 (BatchQueue) gRPC Remote Client 1 (DataCarrier) 1 (BatchQueue) Per peer Armeria HTTP event loop 20 5 min(5, cores) shared group Armeria HTTP handler on-demand platform(increasing with payload) - Virtual threads on JDK 25+ gRPC event loop 10 10 Unchanged gRPC handler on-demand platform(increasing with payload) - Virtual threads on JDK 25+ ForkJoinPool (Virtual Thread carrier) 0 ~10 JDK 25+ virtual thread scheduler HttpClient-SelectorManager 4 2 SharedKubernetesClient Schedulers + others ~24 ~24 Mostly unchanged Total (OAP threads) 150+ ~72 ~50% reduction, stable in high payload. Replace PowerMock Whitebox with standard Java Reflection in server-library, server-core, and server-configuration to support JDK 25+.\nFix /debugging/config/dump may leak sensitive information if there are second level properties in the configuration.\nOAP Server KubernetesCoordinator: make self instance return real pod IP address instead of 127.0.0.1. Fix KubernetesCoordinator self-endpoint race condition: include self in the endpoint list so DynamicEndpointGroup re-fires the listener when the self pod appears in the informer after initial sync. Enhance the alarm kernel with recovered status notification capability Fix BrowserWebVitalsPerfData clsTime to cls and make it double type. Init log-mal-rules at module provider start stage to avoid re-init for every LAL. Fail fast if SampleFamily is empty after MAL filter expression. Fix range matrix and scalar binary operation in PromQL. Add LatestLabeledFunction for meter. MAL Labeled metrics support additional attributes. Bump up netty to 4.2.9.Final. Add support for OpenSearch/ElasticSearch client certificate authentication. Fix BanyanDB logs paging query. Replace BanyanDB Java client with native implementation. Remove bydb.dependencies.properties and set the compatible BanyanDB API version number in ${SW_STORAGE_BANYANDB_COMPATIBLE_SERVER_API_VERSIONS}. Fix trace profiling query time range condition. Add named ThreadFactory to all Executors.newXxx() calls to replace anonymous pool-N-thread-M thread names with meaningful names for easier thread dump analysis. Complete OAP server thread inventory (counts on an 8-core machine, exporters and JDBC are optional): Catalog Thread Name Count Policy Partitions Data Pipeline BatchQueue-METRICS_L1_AGGREGATION-N 8 cpuCores(1.0) ~330 adaptive Data Pipeline BatchQueue-METRICS_L2_PERSISTENCE-N 3 cpuCoresWithBase(1, 0.25) ~330 adaptive Data Pipeline BatchQueue-TOPN_PERSISTENCE-N 1 fixed(1) ~4 adaptive Data Pipeline BatchQueue-GRPC_REMOTE_{host}_{port}-N 1 per peer fixed(1) fixed(1) Data Pipeline BatchQueue-EXPORTER_GRPC_METRICS-N 1 fixed(1) fixed(1) Data Pipeline BatchQueue-EXPORTER_KAFKA_TRACE-N 1 fixed(1) fixed(1) Data Pipeline BatchQueue-EXPORTER_KAFKA_LOG-N 1 fixed(1) fixed(1) Data Pipeline BatchQueue-JDBC_ASYNC_BATCH_PERSISTENT-N 4 (configurable) fixed(N) fixed(N) Scheduler RemoteClientManager 1 scheduled — Scheduler PersistenceTimer 1 scheduled — Scheduler PersistenceTimer-prepare-N 2 (configurable) fixed pool — Scheduler DataTTLKeeper 1 scheduled — Scheduler CacheUpdateTimer 1 scheduled — Scheduler HierarchyAutoMatching 1 scheduled — Scheduler WatermarkWatcher 1 scheduled — Scheduler AlarmCore 1 scheduled — Scheduler HealthChecker 1 scheduled — Scheduler EndpointUriRecognition 1 (conditional) scheduled — Scheduler FileChangeMonitor 1 scheduled — Scheduler BanyanDB-ChannelManager 1 scheduled — Scheduler GRPCClient-HealthCheck-{host}:{port} 1 per client scheduled — Scheduler EBPFProfiling-N configurable fixed pool — Fix BanyanDB time range overflow in profile thread snapshot query. BrowserErrorLog, OAP Server generated UUID to replace the original client side ID, because Browser scripts can\u0026rsquo;t guarantee generated IDs are globally unique. MQE: fix multiple labeled metric query and ensure no results are returned if no label value combinations match. Fix BrowserErrorLog BanyanDB storage query order. BanyanDB Client: Property query support Order By. MQE: trim the label values condition for the labeled metrics query to enhance the readability. PromQL service: fix time parse issue when using RFC3339 time format for querying. Envoy metrics service receiver: support adapter listener metrics. Envoy metrics service receiver: support config MAL rules files. Fix HttpAlarmCallback creating a new HttpClient on every alarm post() call, leaking NIO selector threads. Replace with a shared static singleton. Add SharedKubernetesClient singleton in library-kubernetes-support to replace 9 separate KubernetesClientBuilder().build() calls across 7 files. Fixes KubernetesCoordinator client leak (never closed, NIO selector thread persisted). Uses KubernetesHttpClientFactory with virtual threads on JDK 25+ or a single fixed executor thread on JDK \u0026lt;25. Reduce Armeria HTTP server event loop threads. All 7 HTTP servers now share one event loop group instead of each creating their own (Armeria default: cores * 2 per server = 140 on 10-core). Event loop: min(5, cores) shared — non-blocking I/O multiplexing needs few threads. Blocking executor: JDK 25+ uses virtual threads; JDK \u0026lt;25 keeps Armeria\u0026rsquo;s default cached pool (up to 200 on-demand threads) because HTTP handlers block on long storage/DB queries. Add the spring-ai components and the GenAI layer. Bump up netty to 4.2.10.Final. Bump up log4j to 2.25.3 and jackson to 2.18.5. Remove PowerMock dependency. Replace Whitebox with ReflectUtil (standard Java reflection + sun.misc.Unsafe for final fields) across all modules to support JDK 25+. Support TraceQL and Tempo API for Zipkin and SkyWalking native trace query. Remove initExp from MAL configuration. It was an internal Groovy startup validation mechanism, not an end-user feature. The v2 ANTLR4 compiler performs fail-fast validation at startup natively. Update hierarchy rule documentation: auto-matching-rules in hierarchy-definition.yml no longer use Groovy scripts. Rules now use a dedicated expression grammar supporting property access, String methods, if/else, comparisons, and logical operators. All shipped rules are fully compatible. Activate otlp-traces handler in receiver-otel by default. Update Istio E2E test versions: remove EOL 1.20.0, add 1.25.0–1.29.0 for ALS/Metrics/Ambient tests. Update Rover with Istio Process test from 1.15.0 to 1.28.0 with Kubernetes 1.28. Support Virtual-GenAI monitoring. Fix on-demand pod log parsing failure by replacing invalid DateTimeFormatter pattern with ISO_OFFSET_DATE_TIME. Fix Zipkin receiver compatibility with application/x-protobuf Content-Type. Support Envoy AI Gateway observability (SWIP-10): new ENVOY_AI_GATEWAY layer with MAL/LAL rules for GenAI metrics (token usage, latency, TTFT, TPOT) and access log sampling via OTLP. OTel metric receiver: convert data point attribute dots to underscores (consistent with resource attributes and metric names). Label mappings are now fallback-only — explicit job_name in resource attributes takes precedence over the service.name fallback. OTel log handler: prefer service.instance.id (OTel spec) over service.instance with fallback. Add SampleFamily.debugDump() for MAL debugging. Support virtual GenAI analysis for otlp and zipkin traces. UI Fix the missing icon in new native trace view. Enhance the alert page to show the recovery time of resolved alerts. Implement a common pagination component. Fix validation guard for router. Add the coldStage to the Duration for queries. Optimize the pages theme. Fix incorrect virtual service names. Add the GenAI icon to Topology. Bump up dependencies. Correct active/inactive text for the cold stage. Add the gen-ai menu. Fix: set the step to SECOND in the duration for Log/Trace/Alarm/Tag. Documentation Add benchmark selection into banyanDB storage documentation. Fix progressive TTL doc for banyanDB. Restructure docs/README.md for better navigation with high-level documentation overview. Move Marketplace as a top-level menu section with Overview introduction in menu.yml. Polish marketplace.md as the overview page for all out-of-box monitoring features. Add \u0026ldquo;What\u0026rsquo;s Next\u0026rdquo; section to Quick Start docs guiding users to Marketplace. Restructure agent compatibility page with OAP 10.x focus and clearer format for legacy versions. Remove outdated FAQ docs (v3, v6 upgrade guides and 7.x metrics issue). Remove \u0026ldquo;since 7/8/9.x\u0026rdquo; version statements from documentation as features are standard in 10.x. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1040\"\u003e10.4.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eIntroduce OAL V2 engine:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eImmutable AST models for thread safety and predictable …\u003c/li\u003e\u003c/ul\u003e\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v11.0.0/en/changes/changes-10.4.0/","title":"10.4.0"},{"body":"11.0.0 Project Move the DSL class-loading machinery under core/dsl. core/classloader held only DSL types — RuleClassLoader, DSLClassLoaderManager, ClassLoaderGc, UnloadProbePayload and BytecodeClassDefiner — so it is now core/dsl/classloader, and Catalog moves to core/dsl because a rule-file taxonomy is not a class-loading concern. Three copies of the \u0026ldquo;define a generated class into the right loader\u0026rdquo; dispatch (MAL, LAL, MeterSystem) collapse into a static BytecodeClassDefiner.define, which also gives the JDK 17 --add-opens rationale one home instead of four. Three of the four copies of the generated-class dump-directory lookup become DslGeneratedFileWriter.resolveClassDumpDir; OAL keeps its own, because its debug flag is settable independently of the environment variable and two tests rely on that. No behaviour change. Extend the GET /inspect/entities admin API to inspect a metric persisted by any OAP, even one this node does not define locally. When the metric is unknown to the local registry, the caller supplies valueColumn + valueType and the storage backend resolves the physical index/table/group from its own running config (no DB schema/table-metadata read): ES uses the merged metrics-all index + metric_table discriminator, JDBC probes the node\u0026rsquo;s function tables by the table_name discriminator, and BanyanDB synthesizes a read-only measure schema. Scope is no longer required — the entity_id is decoded structurally (service / 2nd-level / relations) with a generic name leaf. Locally-defined metrics keep the exact field names, scope, and mqeEntity as before. Add the POST /inspect/values admin API — read the value series of a metric persisted by another OAP (one this node does not define locally) by supplying its {valueColumn, valueType}. The real MQE engine runs over a request-scoped InspectQueryContext overlay (provide-if-absent — the local catalog always wins) that makes the foreign metric look registered to every read path: ValueColumnMetadata resolves its value column / type / scope, and the storage location registries resolve where it lives (MetadataRegistry synthesizes a BanyanDB measure schema, IndexController resolves the ES metrics-all index, TableHelper probes the JDBC function tables), so the read returns the native MQE ExpressionResult with no per-DAO special-casing. Admin-only (a forced read this OAP cannot validate); not mirrored onto the public REST / GraphQL surface. See the Inspect API. Remove the always-on alarm-to-event conversion (EventHookCallback). A triggered alarm is no longer synthesized into the events pipeline as an Alarm/AlarmRecovery event; events now originate only from real event sources (agents, SkyWalking CLI, Kubernetes Event Exporter). Alarms remain available through the alarm store (getAlarm/queryAlarms) and the configured alarm hooks. This drops a documented \u0026ldquo;Known Event\u0026rdquo; and removes 1-2 synthetic event records per alarm fire. TLS for all OAP HTTP/REST servers, with cert hot-reload. Adds the restSSLEnabled / restSSLKeyPath / restSSLCertChainPath config structure to every OAP HTTP server — core REST, sharing-server, admin, PromQL, LogQL, TraceQL and Zipkin query/receiver — each with its own dedicated environment variables (SW_CORE_REST_SSL_*, SW_RECEIVER_SHARING_REST_SSL_*, SW_ADMIN_SERVER_REST_SSL_*, SW_PROMQL_REST_SSL_*, SW_LOGQL_REST_SSL_*, SW_TRACEQL_REST_SSL_*, SW_QUERY_ZIPKIN_REST_SSL_*, SW_RECEIVER_ZIPKIN_REST_SSL_*). The shared Armeria HTTPServer reloads the key pair from disk on rotation (via TlsProvider.ofScheduled) so refreshed certificates are picked up without restarting the OAP, matching the existing gRPC SSL hot-reload behavior. HTTP TLS is server-side only (no mTLS). New queryAlarms GraphQL query — entity / layer / rule filters for alarms. Adds a comprehensive alarm query API alongside the legacy getAlarm. The new queryAlarms(condition: AlarmQueryCondition!): Alarms accepts a single input type bundling every filter the alarm record stores: entities: [Entity!] (reuses the MQE Entity shape — pin to specific services / instances / endpoints / processes or their relations, matched against alarm id0 OR id1); layer: String (filter by the alarmed entity\u0026rsquo;s layer — single match, since alarm rows persist one layer); ruleNames: [String!] (filter by which alarm rule fired); plus keyword, tags, duration, paging. Legacy getAlarm is marked @deprecated but still routes to the same DAO — no client breakage. Backend additions: a new layer column on AlarmRecord populated at alarm-mint time via MetadataQueryService.getService(serviceId).getLayers(); the existing id0/id1 columns flipped from storageOnly = true to indexed so the entity filter pushes down to storage. IAlarmQueryDAO.queryAlarms(condition, limit, from) is a new abstract method — 3rd-party storage backends fail at compile if they miss the override (SWIP-14 pattern). All three bundled backends implement it: BanyanDB / Elasticsearch / JDBC. Operator semantics: (1) Relation entities are exact-match. Passing {scope: ServiceRelation, serviceName: A, destServiceName: B} matches only the alarm where id0=serviceId(A) AND id1=serviceId(B), not any alarm that touches A or B on either side. Wider \u0026ldquo;anything involving A\u0026rdquo; queries should pass the individual non-relation entity instead ({scope: Service, serviceName: A} — which expands to id0=A OR id1=A). (2) Single layer per alarm row. The persisted column stores ONE layer (the first entry of the entity\u0026rsquo;s resolved layer list — source-first for relations). A service in [GENERAL, K8S_SERVICE] whose metadata resolves to GENERAL first is filed under GENERAL; querying layer: \u0026quot;K8S_SERVICE\u0026quot; will miss it. Operator migration note: existing pre-upgrade alarm rows continue to be filterable by the legacy getAlarm fields; the new entity / layer / rule filters in queryAlarms apply only to alarms written after the upgrade (existing storage indices don\u0026rsquo;t transition index: false → true in place; new daily-rolled indices pick up the indexed columns). Schema additions are non-blocking — bootstrap silently skips column-attribute changes on existing indices. 🚨 Breaking change: apm-webapp and the skywalking-booster-ui submodule are removed. This OAP distribution no longer ships a bundled web UI. The legacy Armeria reverse proxy in apm-webapp/ (the binary that powered the skywalking/ui Docker image) and the skywalking-ui git submodule (which tracked apache/skywalking-booster-ui) are both deleted along with the docker.ui Maven target, the skywalking/ui Docker image build, the apm-dist/ webapp packaging, and every CI workflow path that built or pushed the UI image. The official UI is now Horizon UI, a SkyWalking sub-project that releases independently of the OAP backend on its own schedule, with released container images on Docker Hub at apache/skywalking-ui (tags latest / horizon-\u0026lt;version\u0026gt;; per-commit development images live on ghcr.io/apache/skywalking-horizon-ui). There is no 1:1 mapping between OAP versions and Horizon UI versions — operators pin the UI image tag in their deployment and upgrade the two on separate cadences. Horizon UI consumes the OAP\u0026rsquo;s public GraphQL/REST surface (default 12800) and the admin host (default 17128). The on-disk dashboard seed files in oap-server/server-starter/src/main/resources/ui-initialized-templates/ are deleted; UITemplateInitializer / UIMenuInitializer are removed from CoreModuleProvider.notifyAfterCompleted(), and Horizon UI ships its own dashboard library and its own sidebar menu. UI templates are now created and updated through the new /ui-management/templates/* REST surface on admin-server (see below). All UI-related GraphQL mutations and queries (UIConfigurationManagement: addTemplate, changeTemplate, disableTemplate, getAllTemplates, getDashboardConfiguration, getMenuItems) are retired from the public GraphQL schema, along with the SW_ENABLE_UPDATE_UI_TEMPLATE flag. The OAP backend also no longer stores or serves the sidebar menu — UIMenuManagementService, UIMenuManagementDAO, UIMenu, MenuItem, and the storage impls are all removed; Horizon UI owns the menu client-side and uses listServices(layer:...) for dynamic \u0026ldquo;layer has services\u0026rdquo; gating. Upgrade path: replace skywalking/ui:\u0026lt;tag\u0026gt; with the Horizon UI image apache/skywalking-ui:latest (or a horizon-\u0026lt;version\u0026gt; tag — pick a version per Horizon UI\u0026rsquo;s OAP-compatibility notes, OAP 11.0+ is supported) in your deployment, expose port 17128 from the OAP container, and migrate any scripts that called the legacy GraphQL UI mutations to the REST endpoints under UI Management API. All status / debug endpoints (/status/*, /debugging/*) also move to admin-only — the public REST dual-bind for status is retired in the same release. New ui-management admin module — REST surface for dashboard templates. Hosts five operations on admin-server (port 17128): GET /ui-management/templates, GET /ui-management/templates/{id}, POST /ui-management/templates, PUT /ui-management/templates, POST /ui-management/templates/{id}/disable. Forwards to the existing UITemplateManagementService (no storage DAO changes). Enabled by default (SW_UI_MANAGEMENT=default, on a default-on admin host). Replaces the retired GraphQL UIConfigurationManagement template resolver. The sidebar menu is intentionally NOT served — see the breaking-change entry above. Operator reference: UI Management API. All admin feature modules default-on. admin-server, status, inspect, ui-management, dsl-debugging, and receiver-runtime-rule all default to enabled. Operators who don\u0026rsquo;t want a particular feature set its SW_* env var to empty. This closes a usability gap from 10.4.0 where the runtime-rule / dsl-debugging surfaces required explicit opt-in even though the admin host was already on. Status API moved to admin-host. Status / debug routes (/status/*, /debugging/*) now register on the admin-server REST host (default 17128); they no longer mirror on core.restPort (default 12800). Aligns status with every other admin feature module (inspect, dsl-debugging, runtime-rule, ui-management). Horizon UI consumes status from the admin host. URIs and payloads are unchanged; only the host moved. One exception: /status/config/ttl is also bound on the public REST host (12800) so ecosystem tools that discover TTL bounds via REST before issuing /graphql don\u0026rsquo;t need to learn the admin port. New admin-server module — shared host for admin / on-demand write APIs. Runs on two ports: an HTTP REST surface (default 17128) for operator-facing endpoints, and an admin-internal gRPC bus (default 17129) for peer-to-peer cluster RPCs (runtime-rule Suspend / Resume / Forward; DSL debug install / collect / stop / stopByClientId). The admin-internal bus is a dedicated transport separate from the public agent / cluster gRPC port (core.gRPCPort, default 11800) so privileged admin RPCs stay out of the agent network\u0026rsquo;s blast radius — operators bind gRPCHost to a private peer-to-peer interface only. Both the runtime-rule plugin and the new DSL Debug API (below) mount onto this shared host. Enabled by default so the status feature module is reachable out of the box; the host binds to 0.0.0.0:17128 and has no built-in authentication and must be gateway-protected with IP allow-lists, never exposed to the public internet (see the Admin API security notice). Set SW_ADMIN_SERVER= (empty) to disable entirely. The runtime-rule config block loses its restHost/restPort/restContextPath/restIdleTimeOut/ restAcceptQueueSize/httpMaxRequestHeaderSize keys (and the matching SW_RECEIVER_RUNTIME_RULE_REST_* env vars); host-level knobs move under the new admin-server block (SW_ADMIN_SERVER_HOST / SW_ADMIN_SERVER_PORT / SW_ADMIN_SERVER_GRPC_HOST / SW_ADMIN_SERVER_GRPC_PORT / SW_ADMIN_SERVER_INTERNAL_COMM_TIMEOUT etc.). Runtime rule hot-update for MAL and LAL. Operators can now ship metric (MAL) and log (LAL) rule changes without restarting OAP. A push to a new admin endpoint persists the rule to the configured storage backend, and every node in the cluster converges to the new content within ~30 seconds. Common workflows: addOrUpdate — create or replace a rule. Body is the raw YAML you would normally ship with OAP\u0026rsquo;s static rule files. Returns 200 once the rule is applied locally and persisted; peers pick it up on their next periodic scan (≤ 30 s). inactivate — soft-pause a rule. The OAP stops emitting metrics for that rule but the backend measure (and its history) is preserved, so a later addOrUpdate to the same (catalog, name) is lossless. The \u0026ldquo;off\u0026rdquo; intent is durable across reboots; bundled rules on disk are not auto-resurrected when an inactivate removes the runtime override. This is the safe way to take a rule offline. delete — removes an INACTIVE row (active rules return 409 requires_inactivate_first). For runtime-only rules with no bundled YAML on disk, the row is dropped; the backend measure (if any) is left in place as an inert artefact, matching bundled-rule deletion semantics (removing a YAML from otel-rules/ on disk doesn\u0026rsquo;t drop its measure either). For rules that have a bundled YAML twin, plain delete returns 409 requires_revert_to_bundled — letting bundled silently take over the (catalog, name) is a meaningful state change that requires an explicit operator decision. Re-issue with ?mode=revertToBundled to fall back to bundled: that path runs the schema-change pipeline (rehydrates the runtime DSL locally, then applies the bundled YAML through the standard apply pipeline so the runtime→bundled delta drops runtime-only metrics, registers bundled-only metrics, and reuses bundled-shared metrics at matching shape) before removing the row. Returns 400 no_bundled_twin when ?mode=revertToBundled is used without a bundled YAML on disk. get / bundled / list / dump — read-side endpoints for fetching a single rule\u0026rsquo;s YAML (with ETag support; ?source=bundled reads the on-disk bundled YAML even when a runtime override is in place), listing the bundled-vs-runtime overlay per catalog, inspecting cluster-wide rule state as a JSON envelope ({generatedAt, loaderStats, rules} — each row carries status/localState/loaderKind/bundled/bundledContentHash so a UI can render override badges without a second roundtrip), and exporting all rules as a tar.gz for backup / DR. Hot-updates survive OAP restart: at boot OAP merges bundled rule files with persisted runtime rules, so the cluster never silently regresses to the bundled defaults. All admin writes for a runtime-rule cluster serialize on a single \u0026ldquo;main\u0026rdquo; OAP (deterministic sorted-first peer, no leader election) — non-main nodes that receive an HTTP write transparently forward it to the main over the admin-internal gRPC bus, so an L7 load balancer in front of the admin port can route any operator request to any OAP. Cluster convergence on the periodic refresh tick is configurable via receiver-runtime-rule.refreshRulesPeriod (default 30 s). The endpoint is disabled by default and listens on port 17128 (HTTP) when enabled. It has no built-in authentication — operators must gateway-protect it with IP allow-lists and never expose it to the public internet. Routes mount on the new admin-server HTTP host, which is on by default; enable the runtime-rule feature with SW_RECEIVER_RUNTIME_RULE=default. Live debugger for MAL / LAL / OAL — implements SWIP-13 Live Debugger for MAL / LAL / OAL. Sample-based runtime debugger that captures per-stage inputs/outputs as the three DSLs process live ingest. Idle-path cost is one volatile-bool read per probe call site that JIT eliminates after warm-up; active sessions fan out to every cluster peer over the admin-internal gRPC bus so each peer captures its own slice. The fan-out is LB-safe: any node can serve any verb (POST mints sessionId on the receiving node, broadcasts install to peers, returns 404 rule_not_found only when no node owns the rule), so an L7 load balancer in front of the admin port routes operator requests freely. Mounts on the shared admin-server host (/dsl-debugging/* for session control plane, /runtime/oal/* for the OAL rule picker). Disabled by default; enable with SW_DSL_DEBUGGING=default (admin-server itself is on by default). injectionEnabled is a boot-time codegen switch defaulting to true — once the module is enabled, probes fire and sessions record samples; set false only if the REST surface is wanted but no codegen-side probe overhead is acceptable. Per-session limits enforce hard caps (recordCap ≤ 10000, retentionMillis ≤ 1 hour) — out-of-range requests return 400 invalid_limits. LAL sessions accept a per-session granularity=block|statement flag — block mode captures the parser/extractor/sink stages; statement mode additionally records one line entry per individual extractor statement, carrying the source-line number and verbatim DSL text so the UI can highlight which statement fired. MAL captures render the file-level filter\u0026rsquo;s surviving SampleFamily map ({\u0026quot;families\u0026quot;: N, \u0026quot;items\u0026quot;: [...]}), so multi-metric expressions show cross-family filter narrowing in the captured payload. Capture payloads include raw log bodies and parsed maps — treat the admin port as authenticated infrastructure per the Admin API security notice. Per-DSL operator references: MAL, OAL, LAL. BanyanDB schema mismatches are now visible at boot, not silent. If BanyanDB already holds a resource whose shape doesn\u0026rsquo;t match what the current rule declares (e.g., a rule was edited on disk while OAP was offline), OAP now skips that resource, logs an ERROR with the declared-vs-backend diff, and continues booting — previously the mismatch was silently accepted and samples for the affected resource were quietly dropped. To re-shape a mismatched metric, push the desired YAML through POST /runtime/rule/addOrUpdate. Bump infra-e2e to testcontainers-go v0.42.0 (apache/skywalking-infra-e2e#146), which uses Docker Compose v2 plugin natively and removes docker-compose v1 dependency. Remove deprecated version field from all docker-compose files for Compose v2 compatibility. Best-effort schema-cutover fence for BanyanDB. After firing a schema install or drop OAP now waits up to a bounded window (default 2s) for every BanyanDB data node to apply the change before resuming dispatch — the typical case gets a clean cutover where samples after 200 OK use the new shape. On laggard timeout, OAP logs a warning and proceeds anyway so a single slow node doesn\u0026rsquo;t wedge the apply. Bump dependencies: gRPC 1.70.0 → 1.80.0, protobuf-java 3.25.5 → 4.33.1, Netty 4.2.10.Final → 4.2.12.Final, Netty-tcnative 2.0.75 → 2.0.77, pgv (protoc-gen-validate) 1.2.1 → 1.3.0. Driven by the new BanyanDB schema-consistency RPCs whose generated validation code requires the protobuf-java 4.x runtime. Inspect API on admin-server. Two new admin-only HTTP endpoints for browsing the live metric catalog and the entities currently emitting values for a given metric. GET /inspect/metrics lists every registered metric with its type / scope / catalog / value-column name / supported downsamplings (pure metadata, no I/O). GET /inspect/entities runs the storage backend\u0026rsquo;s entity scan for a metric over a time range + step (capped at 300 rows) and returns each entity decoded into an MQE-ready payload — the response includes a mqeEntity block the operator pastes verbatim into the public GraphQL execExpression mutation, plus the source service\u0026rsquo;s layer(s) (multi-layer services emit one row per layer). Restricted to REGULAR_VALUE / LABELED_VALUE metrics and to non-Process scopes; HEATMAP / SAMPLED_RECORD / Process / ProcessRelation return 400. Adds IMetricsQueryDAO.listEntityIdsInRange as an abstract method on the interface — any 3rd party storage backend must explicitly override or the build fails. Enabled by default (both SW_INSPECT and SW_ADMIN_SERVER are on by default); set SW_INSPECT= empty to disable. Operator reference: Inspect API. Status feature module relocation, finalized. The legacy status-query-plugin was replaced by a new status feature module under server-admin/; the route set (/status/cluster/nodes, /status/alarm/*, /status/config/ttl, /debugging/config/dump, /debugging/query/*) keeps URIs and payloads unchanged. The selector renames from the QUERY-plugin form (SW_QUERY=…,status-query-plugin) to a top-level SW_STATUS=default (on by default); custom application.yml overrides referencing status-query need to repoint to status. Routes are admin-host only — see the \u0026ldquo;Status API is admin-host only\u0026rdquo; entry above for the public REST retirement. Drop six unused test-scoped dependencies from runtime-rule (library-integration-test, library-banyandb-client, storage-banyandb-plugin, testcontainers, testcontainers:junit-jupiter, grpc-testing). They staged the plugin-side ITs that were retired in favour of e2e; that coverage now lives in test/e2e-v2/cases/runtime-rule/ (MAL over BanyanDB / PostgreSQL / Elasticsearch, LAL, meter, and the two-node cluster case). The module has no ITs today, and JUnit and Mockito are inherited from the root POM. Declare server-testing at test scope everywhere. It ships only test scaffolding (ModuleManagerTesting, MockModuleManager, the MAL/LAL/Hierarchy rule loaders) plus two empty org.junit stubs that let Testcontainers\u0026rsquo; GenericContainer hierarchy resolve without JUnit 4, but four modules declared it at compile scope — including the server-configuration parent, so all eight configuration-* children inherited it — which put those org.junit stubs on the runtime classpath that server-starter copies into oap-libs. Modules whose tests need the stubs now declare the dependency themselves rather than inheriting it transitively, and library-banyandb-client gains the direct library-util dependency its BanyanDBClient always needed (it was resolving StringUtil through server-testing, a test-support module). Add ThreadPolicy.ioBound(N) to library-batch-queue, for queues whose consumers spend most of their time blocked. Such a queue runs its drain loops on virtual threads where the runtime provides them (JDK 25+) and falls back to N platform threads otherwise; the count, and therefore concurrency, batching, back-pressure, drop semantics and per-partition ordering, are identical on both paths. Shutdown latency is the one exception: the platform scheduler drops drain tasks parked on their idle backoff, while the virtual-thread adapter sleeps inside the submitted task and cannot, so an ioBound queue should keep maxIdleMs within shutdownTimeoutMs. There is deliberately no CPU-proportional form: virtual threads are not preemptive, so CPU-bound work would hold its carrier and starve the shared carrier pool, and L1/L2/TopN stay on cpuCores/fixed. Also fixes BatchQueue.shutdown(), which ran its final drain on the caller\u0026rsquo;s thread while drain loops could still be inside consume(), invoking a handler concurrently and breaking the single-drain-thread invariant workers such as MetricsAggregateWorker rely on: it now cancels the periodic rebalance task, waits for in-flight consumers (shutdownTimeoutMs, default 500ms per queue), and serialises its final dispatch behind a read/write dispatch lock so the guarantee holds even when that wait times out or is interrupted. Drain loops hold the read lock for the whole cycle — the running recheck, the partition dequeue, the idle notification and the dispatch — because onIdle() touches the same worker state as consume() and an unlocked dequeue would let shutdown dispatch a newer batch ahead of one a task already holds. Concurrent shutdown callers await the winner\u0026rsquo;s completion rather than returning early. A consumer is never interrupted mid-batch. OAP Server Add component IDs for the Spring LDAP Java agent plugin (spring-ldap: 179) and LDAP server (LDAP: 180), including their server mapping. Fix LAL\u0026rsquo;s segmentId and spanId extractor statements, which the grammar accepted and the parser never implemented. LALParser.g4 declares traceIdStatement, segmentIdStatement and spanIdStatement, and the codegen already carried setSegmentId/setSpanId in its setter table, but LALScriptParser.visitExtractorStatement had a branch for only the first of the three. The remaining alternatives fell through to a line that assumed whatever was left had to be an ifStatement, so a rule writing segmentId ... failed at boot with a NullPointerException naming IfStatementContext — for a rule line containing no if. Both statements now work, and an unhandled extractor statement reports its own rule line instead of throwing. Existing log records are unaffected: LogBuilder copies trace id, segment id and span id straight from the log\u0026rsquo;s metadata, and only skips that copy when a rule has set them — which no shipped rule did, which is why the gap went unnoticed. Dedicated execution tests now cover reading all three fields from log.traceContext.* and writing all three from an extractor. Remove dead code from the DSL subsystem and correct the shared kernel\u0026rsquo;s own documentation. Deleted DslContentHash (a byte-identical, zero-caller twin of the live ContentHash), the unused oal-rt metrics-function registry, LogAnalyzerFactory, LALCodegenHelper.METADATA_GETTER_ALIASES (a permanently empty map whose reader branch could never execute — the DSL-name-to-getter mismatch it existed for no longer exists, since LogMetadata.TraceContext names the field traceSegmentId directly), and three unreferenced members. Three kernel classes carried javadoc asserting consumers that do not exist — DSLClassLoaderManager claimed the MAL and LAL compile paths reach for its singleton, LogDataDebugDump claimed core renders through it, and DslContentHash instructed the reader to consolidate toward the dead copy; a false rationale in a shared kernel is an instruction to the next contributor, so those are now what the call sites actually support. OALDebug, OALDebugRecorder and DebugHolderProvider now record why they sit in core while MAL\u0026rsquo;s and LAL\u0026rsquo;s equivalents do not: dsl-debugging declares no oal-rt dependency, so they cannot move. Unify source attribution for every generated DSL class, so a stack frame from OAL, MAL, LAL or Hierarchy code leads back to the rule that produced it. SourceFile now names the RULE and its line, then the generated class file: (otel-rules/activemq/activemq-broker.yaml:32)otel_rules_activemq_activemq_broker_L32_service_meter.java. The .java generated source file is written only under SW_DYNAMIC_CLASS_ENGINE_DEBUG, so in a default deployment naming it named nothing; the class name cannot substitute because sanitising maps /, - and . all to _ and drops the extension. The _L\u0026lt;n\u0026gt;_ segment was the rules-list index rather than a line, so every file\u0026rsquo;s first rule reported as L0. It now carries the rule\u0026rsquo;s real line, resolved by the loaders themselves — including Zabbix and Hierarchy, which supplied no coordinate at all, and the runtime-rule hot-update paths for MAL and LAL, which disagreed with their own boot loaders. LineNumberTable held statement ordinals, matching neither the YAML nor the generated source. MAL keeps a per-statement table; MAL closure companions, LAL and OAL carry one entry at each method\u0026rsquo;s signature, found by searching the assembled generated source file text for that method\u0026rsquo;s declaration rather than counting a per-method offset. Hierarchy writes no generated source file, so it carries none. Generated class names change. They are now built from the rule file\u0026rsquo;s catalog-qualified path, so a MAL class that was vm_L25_cpu_total_percentage is now otel_rules_vm_L25_cpu_total_percentage. The catalog is kept only where the generated class\u0026rsquo;s package does not already imply it, so LAL — one catalog, its own package — is unchanged at default_L3_default. Nothing addresses these classes by name — they are generated, loaded reflectively and never referenced from configuration — but the names appear in stack traces, in SW_DYNAMIC_CLASS_ENGINE_DEBUG dumps and in dsl-debugging output, so saved greps and dashboards keyed on the old form need updating. The unqualified names were ambiguous: two catalogs can each hold a vm.yaml. A generated method\u0026rsquo;s line is located by searching the assembled source for its declaration, and that search now requires an identifier boundary. serialize is a suffix of deserialize and OAL declares both on every metrics class, so the shorter name resolved to the longer method\u0026rsquo;s line whenever the template order changed; two OAL metrics named cpm and commando_cpm in one scope reach the same shape through their shared dispatcher. The wrong line shipped in production bytecode — the attribute is stamped unconditionally — so a frame reported another rule\u0026rsquo;s location. Fix a hot-updated LAL rule stranding its own dsl-debugging binding instead of replacing it. The RuleKey naming a rule file was spelled default.yaml by the boot loader and default by the runtime-rule engine, so the two never met in the holder registry: the pre-update GateHolder stayed reachable, and an operator addressing it enabled probes on a compiled rule that no longer evaluates anything — indistinguishable from a rule receiving no traffic. RuleKey now drops a trailing .yaml/.yml from that component, so both spellings are one key and the debugging API keeps accepting either. Rule execution was never affected: the maps that decide which rules run are keyed by layer and rule name, not by file name. A class shared by several rules — an OAL dispatcher — names its file without a line rather than borrow the first rule\u0026rsquo;s, which would misreport every other rule routed through it. The generated .java source files are written as UTF-8 with an ASCII header instead of through a platform-default FileWriter, the pairing that produced MalformedInputException on a non-UTF-8 JVM. Breaking change: HierarchyDefinitionService.HierarchyRuleProvider — a ServiceLoader SPI — now declares buildRules(Map\u0026lt;String, String\u0026gt; ruleExpressions, Map\u0026lt;String, Integer\u0026gt; ruleLines) in place of the former one-argument buildRules(Map). A third-party provider compiled against the old signature fails with AbstractMethodError and must be recompiled. The second argument carries each rule\u0026rsquo;s line in hierarchy-definition.yml, which the expression map cannot supply because snakeyaml\u0026rsquo;s bean binding discards positional marks; a provider with no line information should pass an empty map. Without it, generated hierarchy classes are labelled _Lunknown_ and their stack frames lead nowhere. The mechanism is one implementation in org.apache.skywalking.oap.server.core.dsl, shared by all four compilers: the coordinate model, the class-name builder, the SourceFile value, the generated source file writer, the signature-line lookup and the bytecode attributes. Previously each compiler had its own, and they had drifted apart. Support runtime rule hot-update and DSL debugging for the meter-analyzer-config catalog, bringing native meter (MeterReportService) rules to parity with otel-rules. Meter rules now load through the same Rules/Rule pipeline otel-rules uses, so they participate in RuleSetMerger, are recorded in StaticRuleRegistry, support the optional layerDefinitions block, and generate source-named expression classes instead of falling back to MalExpr_\u0026lt;N\u0026gt;. MeterProcessService now implements MalConverterRegistry and publishes debug holders at boot, so a meter rule can be added / overridden / inactivated at runtime, and attached to a DSL debug session, without restarting the OAP. The internal MeterConfig / MeterConfigs model is removed in favour of the shared one. Behaviour change: an entry in meterAnalyzerActiveFiles (SW_METER_ANALYZER_ACTIVE_FILES) with no matching rule file now fails OAP startup instead of being silently ignored, matching how otel-rules has always behaved. Support Elasticsearch 9.x as storage. Add Node.js runtime metrics via the Node.js agent MeterReportService pipeline (meter_instance_nodejs_*, default 20s sample/report). OAP analyzes raw meters through nodejs-runtime.yaml. Node.js E2E asserts twelve meter_instance_nodejs_* metrics (test/e2e-v2/cases/nodejs/e2e.yaml). Add PHP runtime PHM meter analyzer (php-runtime.yaml) for SkyWalking PHP agent process metrics (CPU, memory, virtual memory, thread count, open file descriptors sampled from /proc on Linux). Registers six meter_instance_php_* metrics on the General Service layer; php-runtime is included in the default meterAnalyzerActiveFiles. Batch the BanyanDB schema fence per runtime-rule apply. A runtime-rule file changes dozens of rules at once, but the post-DDL fence (SchemaWatcher.awaitRevisionApplied) ran once per metric/downsampling, so a large file did K×M sequential ≤2s fences — on a laggy cluster that overran the apply\u0026rsquo;s REST budget. The main-node apply path now uses StorageManipulationOpt.withSchemaChangeDeferredFence(): the installer records each resource\u0026rsquo;s mod_revision without fencing and registers a single flush that the apply runs once on the file\u0026rsquo;s max revision, collapsing the whole file to one barrier. The flush is one-shot — a reconciler tick reuses one opt across every rule file, so after a file flushes, the closure and accumulated revision reset and each file fences on its own DDL only. Drops still fence inline on the dropped resource\u0026rsquo;s own delete revision — or, when that delete recorded no tombstone (mod_revision == 0), on a key-based deletion barrier (AwaitSchemaDeleted) — never on the shared opt\u0026rsquo;s cumulative revision, so a tombstone-less delete in a multi-file tick is still confirmed removed. On the operator REST apply the single create/update fence runs on a configurable, generous budget (default 180s) in the background before the rule row is persisted and dispatch resumes — it gates the persist + local commit + peer resume so the durable commit point is only reached once the schema is confirmed cluster-wide, and writes never resume against an un-propagated schema (see the apply-status entry below); the reconciler tick keeps the short inline 2s fence (a background reconcile must not wait minutes per file). Peer / withoutSchemaChange applies are unaffected (no fence). Add a runtime-rule apply-status query. The cluster main now tracks each structural apply through a phase machine (SchemaApplyCoordinator: pending → DDL → fencing → rolling-out → applied, with degraded for a committed-but-unconfirmed apply — the cluster schema fence did not confirm within the timeout, in which case the lagging data-node ids are surfaced as fenceLaggards and dispatch is resumed anyway, or the local commit-tail threw — and failed carrying the specific reason). The schema fence runs on a configurable, generous budget (receiver-runtime-rule.deferredFenceTimeoutSeconds, default 180s) and gates everything durable or visible: because an un-propagated write is silently dropped at the data node, the order after a successful DDL is suspend → DDL → fence → persist → commit → resume. The rule row (the durable commit point) is written only AFTER the fence confirms, so \u0026ldquo;durable\u0026rdquo; implies \u0026ldquo;schema propagated cluster-wide\u0026rdquo; — a main crash before persist leaves no row (peers/crash-recovery stay safely on the old content; the orphaned measure is inert), and any durable row is guaranteed fence-confirmed, so convergence never resumes dispatch against an unpropagated schema. The fence + persist + resume run in the background so they never block the HTTP response — POST /addOrUpdate returns its applyId immediately at fencing (accepted, not yet durable; dispatch for that rule still paused — a clean gap, not dropped writes), and the operator polls GET /runtime/rule/status to watch fencing → rolling-out → applied (or degraded/failed); on a genuine laggard, dispatch resumes after the budget so one stuck node can\u0026rsquo;t park the metric forever. A GetApplyStatus admin-internal gRPC served by the main backs the query — by applyId, or by catalog+name (+ optional contentHash, the durable identity) once the handle is gone after a page refresh. When the live status is gone (apply-id evicted, main restarted, or the main is unreachable), the query degrades to the durable rule row: a matching ACTIVE row reports applied derived from the content hash (a durable row is, by the fence-then-persist order, already propagation-confirmed). Non-main nodes route the read to the deterministic main; status is in-memory by design, with the content hash reconstructing truth after a restart. Push runtime-rule convergence to peers on commit. After a successful structural apply — and on the commit_deferred path, where the DB row is durable but this node\u0026rsquo;s commit-tail threw — the main broadcasts a NotifyApplied admin-internal RPC so peers reconcile against the just-persisted DB row immediately, instead of waiting up to one refresh tick (~30s) to notice it. The fan-out runs off the REST response thread (fire-and-forget on a daemon executor) so an unreachable peer\u0026rsquo;s per-call deadline never adds to the operator\u0026rsquo;s apply latency. On the peer side the notify-triggered reconcile is coalesced: a burst of notifies (a multi-rule file, or several applies) collapses to a single queued full reconcile rather than one redundant dao.getAll() scan per notify. The notify is best-effort and idempotent (the peer runs its normal per-file-locked reconcile; a lost notify is harmless — the peer still self-converges on its next tick), so it tightens the cluster-convergence window without adding a hard dependency on the main being reachable. Fix BanyanDB peer nodes permanently flooding \u0026lt;metric\u0026gt; is not registered, and a follow-on case where a peer kept translating writes with a stale schema shape after a runtime-rule reshape, when a node held a live persist worker but its local MetadataRegistry schema cache was missing or stale for that model — a withoutSchemaChange peer apply or a runtime-rule bundled fall-over rebuilt the dispatch worker but skipped the local-cache populate, and the registry was insert-only (never evicting) while the 30s reconcile only covers runtime-rule rows, so nothing re-derived it. The peer / local-cache-only install path now (re)derives and overwrites the local schema entry from the declared model with zero server RPC — honoring the inspectBackend=false contract so the cache can never lag the worker, including across a reshape — and a model removal now evicts its cache entry so a dropped or reshaped model leaves no stale translation behind; the persist DAOs keep an RPC-free re-derivation as a read-side backstop, and the no-init defer poll loop retries a transient backend probe error instead of escaping and crash-looping the pod. Support LAL json {} parsing JSON content delivered in a plain-text log body. The parser reads the native protocol\u0026rsquo;s JSON body first; when that is empty, it tries the text body as JSON — e.g. the OTLP log receiver maps every OTLP string body to a text body, even JSON-shaped ones, so previously-aborting json {} rules on OTLP-fed layers now work without any receiver or protocol change. On a successful parse from a text body, the matching rule persists the log as a JSON body with content type JSON; the normalization is scoped to that rule\u0026rsquo;s context — other rules analyzing the same log still see the original text body. Surface the drop reason in LAL live-debugging. When a LAL rule stops a log at a parse step (a json {} / yaml {} parse failure, a text { regexp } non-match, or a non-log-body input), the recorder now captures a human-readable reason (e.g. the parse exception) onto the DSL-debug Sample, exposed through the dsl-debugging REST session response and the cluster forward proto. Previously a live-debug watcher could only see continueOn=false — that a step stopped, never why — and had to read the OAP server log. Sample.reason is shared across all DSL debuggers but populated by LAL today. Fix a v2 MAL CounterWindow key collision: rate() / increase() / irate() keyed each counter\u0026rsquo;s sliding window on the rule\u0026rsquo;s output metric name (the same for every input metric of a rule) instead of the counter\u0026rsquo;s own name, so two or more counters that reduce to the same label set after .sum(...) shared one window and computed rates against each other\u0026rsquo;s values — fabricating non-zero rates from unchanged counters (e.g. the BanyanDB liaison gRPC error rate read a steady non-zero off three frozen error counters). The window is now keyed by the counter\u0026rsquo;s own metric name. Fix the v2 MAL Elvis operator ?: to honor Groovy-falsy semantics. It compiled to Optional.ofNullable(primary).orElse(fallback), applying the fallback only when the primary is null, so an empty-string primary kept \u0026quot;\u0026quot; instead — e.g. a BanyanDB liaison ServiceInstance stored node_type=\u0026quot;\u0026quot; rather than n/a, because .sum([...,'node_type']) fills an absent group-by label with \u0026quot;\u0026quot;. The fallback now applies for falsy primaries such as null, false, numeric zero, and empty strings/containers. SWIP-15: rebuild BanyanDB self-observability around the cluster / container / group model (requires BanyanDB 0.11+). A BanyanDB cluster is modeled as one Service, each container as a ServiceInstance (role/tier as attributes), and each storage group as an Endpoint. The otel-rules/banyandb/ rules are category-separated by role (node_* / liaison_* / data_* / lifecycle_*) and by data type (measure_* / stream_* / trace_* / property_*), mirroring the upstream FODC-proxy Grafana boards, and include queue batch/message granularity (apache/skywalking-banyandb#1169). Adds a SERVICE_INSTANCE_RELATION MAL scope and serviceInstanceRelation(...) builder powering a new intra-cluster pod-to-pod deployment topology (banyandb-instance-relation.yaml). The stale single-node host_name model is removed. Runtime MAL/LAL hot-update rules can declare layerDefinitions: to introduce new layers. Ordinals are operator-pinned in the 100_000+ tier; the layer is refcount-tracked and unregistered when the last declaring rule is removed. See runtime-rule-hot-update.md#dynamic-layers for the conflict rules and limitations. Fix: runtime-rule (MAL/LAL hot-update) schema changes now work in no-init mode — the deployment mode every production cluster runs. Previously a runtime addOrUpdate that introduced a new metric blocked forever in the storage installer\u0026rsquo;s init-node poll loop (ModelInstaller.whenCreating) on a no-init OAP, because the gate keyed off RunningMode rather than the operation\u0026rsquo;s intent; the /delete?mode=revertToBundled recreate and BanyanDB in-place shape updates were dead the same way. The poll loop is now gated on a new StorageManipulationOpt.Flags.deferDDLToInitNode bit set only on the static boot-time schemaCreateIfAbsent() opt (DRYed into ModelInstaller.deferDDLToInitNode(opt) and reused by the BanyanDB shape-check / group-DDL gates), so the runtime-rule opts (withSchemaChange / verifySchemaOnly / withoutSchemaChange) are driven by their flags and by cluster main-ness — no-init and default no longer differ for DSL DDL; init mode stays the dedicated initializer. DSLManager.tickStorageOpt is collapsed accordingly (main → withSchemaChange, peer → verifySchemaOnly at boot / withoutSchemaChange on tick). Fix: runtime-rule cross-node writes no longer fail with HTTP 400 forward_self_loop on a multi-replica Kubernetes cluster. Every OAP replica shared the cluster selfNodeId 0.0.0.0_11800 (derived from the 0.0.0.0 agent gRPC bind host via TelemetryRelatedContext), so the main\u0026rsquo;s self-loop guard rejected a legitimate peer-to-peer Forward as if it had looped back. The runtime-rule node identity now prefers the unique per-pod SKYWALKING_COLLECTOR_UID (the pod UID injected by the helm chart / swck operator from metadata.uid), resolved in start() before any apply, and falls back to the telemetry id for non-k8s deployments. Adds a kind-based no-init cluster e2e (test/e2e-v2/cases/runtime-rule/cluster, deployed via skywalking-helm with oap.replicas=2) that drives the apply / STRUCTURAL / inactivate / delete lifecycle and the cross-node Forward path, replacing the prior docker-compose default-mode cluster case. Fix: remove the redundant tags from the envoy-ai-gateway.yaml LAL configuration. Add Zipkin Virtual GenAI e2e test. Use zipkin_json exporter to avoid protobuf dependency conflict between opentelemetry-exporter-zipkin-proto-http (protobuf~=3.12) and opentelemetry-proto (protobuf\u0026gt;=5.0). Fix missing taskId filter and incorrect IN clause parameter binding in JDBCJFRDataQueryDAO and JDBCPprofDataQueryDAO. Remove deprecated GroupBy.field_name from BanyanDB MeasureQuery request building (Phase 1 of staged removal across repos). Push taskId filter down to the storage layer in IAsyncProfilerTaskLogQueryDAO, removing in-memory filtering from AsyncProfilerQueryService. Fix missing parentheses around OR conditions in JDBCZipkinQueryDAO.getTraces(), which caused the table filter to be bypassed for all but the first trace ID. Replaced with a proper IN clause. Fix missing and keyword in JDBCEBPFProfilingTaskDAO.getTaskRecord() SQL query, which caused a syntax error on every invocation. Fix storage layer bugs in profiling DAOs and add unit test coverage for JDBC query DAOs. Bug fixes: duplicate TABLE_COLUMN condition in JDBCMetadataQueryDAO.findEndpoint(), wrong merged table check in JFRDataQueryEsDAO (used incorrect INDEX_NAME due to copy-paste), and missing isMergedTable check in ProfileTaskQueryEsDAO.getById(). Test additions: add unit tests for 21 JDBC query DAOs verifying SQL/WHERE clause construction. Optimize TraceQueryService.sortSpans from O(N^2) to O(N) by pre-indexing spans by segmentSpanId, so trace detail queries scale linearly with span count. Support MCP (Model Context Protocol) observability for Envoy AI Gateway: MCP metrics (request CPM/latency, method breakdown, backend breakdown, initialization latency, capabilities), MCP access log sampling (errors only), ai_route_type searchable log tag, and MCP dashboard tabs. Add weighted handler support to BatchQueue adaptive partitioning. MAL metrics use weight 0.05 at L1 (vs 1.0 for OAL), reducing partition count and memory overhead when many MAL metric types are registered. Fix missing taskId filter in pprof task log query and its JDBC/BanyanDB/Elasticsearch implementations. Fix duplicate calls in EndpointTopologyBuilder — calls were not deduplicated unlike ServiceTopologyBuilder, causing duplicate entries when storage returns multiple records for the same relation. Use containsOnce and noDuplicates for topology dependency e2e expected files to enforce no-duplicate verification. Bump infra-e2e to ef073ad to include noDuplicates pipe function support. PromQL: support querying Zipkin metadata (service name, remote service name, span name). TraceQL: support more tags and variables in Grafana for querying. LAL: add sourceAttribute() function for non-persistent OTLP resource attribute access in LAL scripts. LAL: add layer: auto mode for dynamic layer assignment when service.layer is absent. Add two-phase SpanListener SPI mechanism for extensible trace span processing. Refactor GenAI from hardcoded SpanForward.processGenAILogic() to GenAISpanListener. Add OTLP/HTTP receiver support for traces, logs, and metrics (/v1/traces, /v1/logs, /v1/metrics). Supports both application/x-protobuf and application/json content types. Fix: TTL query add metadata TTL. Fix: PersistentWorker used wrong TTL for metrics cache if the storage is BanyanDB. Add iOS/iPadOS app monitoring via OpenTelemetry Swift SDK (SWIP-11). Includes the IOS layer, IOSHTTPSpanListener for outbound HTTP client metrics (supports OTel Swift .old/.stable/.httpDup semantic-convention modes via stable-then-legacy attribute fallback), IOSMetricKitSpanListener for daily MetricKit metrics (exit counts split by foreground/background, app-launch / hang-time percentile histograms with finite 30 s overflow ceiling), LAL rules for crash/hang diagnostics, Mobile menu, and iOS dashboards. Add Apache Airflow monitoring via native OpenTelemetry metrics (SWIP-7). New AIRFLOW layer with Service (cluster) and Instance (host) dimensions, MAL rules under otel-rules/airflow/ (27 metrics), setup documentation, mock OTLP e2e (cases/airflow/mock/e2e.yaml: 2 entity + 27 metric checks, 29 total), and real Celery-cluster integration smoke (cases/airflow/cluster/e2e.yaml: 2 entity + 14 metric checks, 16 total). See test/e2e-v2/cases/airflow/README.md. Horizon UI dashboards ship separately in apache/skywalking-horizon-ui under the Workflow Scheduler menu group. Fix LAL layer: auto mode dropping logs after extractor set the layer. Codegen now propagates layer \u0026quot;...\u0026quot; assignments to LogMetadata.layer so FilterSpec.doSink() sees the script-decided layer. Fix MetricKit histogram percentile metrics being reported at 1000× their true value — the listener now marks its SampleFamily with defaultHistogramBucketUnit(MILLISECONDS) so MAL\u0026rsquo;s default SECONDS→MS rescale of le labels is not applied. Add WeChat and Alipay Mini Program monitoring via the SkyAPM mini-program-monitor SDK (SWIP-12). Two new layers (WECHAT_MINI_PROGRAM, ALIPAY_MINI_PROGRAM); two new JavaScript componentIds (WeChat-MiniProgram: 10002, AliPay-MiniProgram: 10003). Service / instance / endpoint entities are produced by MAL + LAL, not trace analysis — mini-programs are client-side (exit-only) so RPCAnalysisListener stays unchanged (same pattern as browser and iOS). MAL rules per platform × scope under otel-rules/miniprogram/ with explicit .service(...) / .endpoint(...) chains (empty expSuffix so endpoint-scope rules aren\u0026rsquo;t overridden), histogram percentile via .histogram(\u0026quot;le\u0026quot;, TimeUnit.MILLISECONDS) to keep ms bucket bounds intact, and request-cpm derived from the histogram _count family. LAL layer: auto rule produces both layers via miniprogram.platform dispatch and emits error-count samples consumed by per-platform log-MAL rules. Per-layer menu entries and service / instance / endpoint dashboards with Trace and Log sub-tabs. Fix: remove VirtualServiceAnalysisListener\u0026rsquo;s dependency on GenAIAnalyzerModule if it is disabled. MAL: register TimeUnit in MALCodegenHelper.ENUM_FQCN so rule YAML can write .histogram(\u0026quot;le\u0026quot;, TimeUnit.MILLISECONDS) for SDKs that emit histogram bucket bounds in ms (default SECONDS unit applies a ×1000 rescale that would otherwise inflate stored le labels 1000×). Fix: potential unexpected current directory inclusion in Docker OAP classpath. MAL: add safeDiv(divisor) on SampleFamily that yields 0 when the divisor is 0 instead of Infinity/NaN. Replace / with safeDiv(...) in Envoy AI Gateway latency-average rules so sum / count * 1000 no longer produces dropped or out-of-range samples when a counter is zero in a window. Fix: envoy-ai-gateway metrics rules, make the metrics value return 0 when the divisor is 0. Custom Layers can be declared without modifying the OAP source — via an operator-managed layer-extensions.yml, inline layerDefinitions: block in a MAL or LAL rule file, or a plugin extension. UI dashboard templates for new layers are auto-discovered from the ui-initialized-templates/ directory. Recommended ordinal range for external layers is \u0026gt;= 1000; conflicting names or ordinals are reported at boot. LAL: support full arithmetic (+, -, *, /) on numeric operands and fix the original bug where (tag(\u0026quot;x\u0026quot;) as Integer) + (tag(\u0026quot;y\u0026quot;) as Integer) was treated as string concatenation — expressions like input_tokens + output_tokens \u0026lt; 10000 produced the concatenated string \u0026quot;2589115\u0026quot; rather than the integer sum 2704, so token-threshold conditions never triggered abort {}. Operand types are now inferred from explicit casts (as Integer / as Long / as Float / as Double), typed proto fields, or numeric literal shape (with L / F / D suffix support, e.g. 1000L). The compiler honours JLS-style binary numeric promotion and emits Java arithmetic in the declared primitive type — (x as Integer) + (y as Integer) compiles to int + int (not widened to long). + with any String operand falls back to string concatenation; - / * / / against non-numeric operands produces a compile-time error. The as Double and as Float casts are accepted in typeCast clauses, including in def declarations. Numeric comparisons honour declared casts on both sides (no more universal h.toLong() wrapper). Fix: avgHistogramPercentile / sumHistogramPercentile meter functions reported the smallest finite bucket boundary (e.g. 10 for OTel gen_ai_server_request_duration whose le is rewritten from 0.01s → 10ms) for every rank when no samples were observed in any bucket. The percentile loop\u0026rsquo;s count \u0026gt;= roof check matched on the first sorted bucket because both sides were 0. calculate() now short-circuits to 0 for every rank when the windowed total is 0. Fix: MAL expPrefix now applies to every metric source in exp, not just the leading one. Previously the prefix was spliced after the first ., so secondary metrics inside arguments (e.g. the divisor in a.sum(['s']).safeDiv(b.sum(['s']))) silently skipped the prefix — a rule like envoy-ai-gateway\u0026rsquo;s request_latency_avg (sum / count) would tag-rewrite only the dividend. The injection is now AST-aware: every bare-IDENTIFIER metric source is wrapped, while downsampling-type constants (SUM, AVG, LATEST, SUM_PER_MIN, MAX, MIN) are skipped. Add @Stream(allowBootReshape = true) opt-in for additive boot-time reshape of BanyanDB streams / measures. Code-defined stream classes (e.g. AlarmRecord) can now annotate their schema as eligible for in-place additive update at OAP boot — a new @Column is appended to the live tag-family / fields via client.update instead of being silently rejected with SKIPPED_SHAPE_MISMATCH (which previously forced operators to drop the measure / stream and lose historical rows). Additive includes both new tags / fields and relocating an existing tag between families when a @Column\u0026rsquo;s storageOnly flag flips (e.g. id1 moving from storage-only → searchable when it becomes indexed). The opt-in is per-stream and gated by an isPurelyAdditive shape diff: tag type changes, tag drops, kind flips (tag↔field), entity / interval / sharding-key changes, and field re-typing still skip with SKIPPED_SHAPE_MISMATCH, so identity-breaking edits remain explicit operator actions. Only the init / standalone OAP performs the reshape; non-init peers continue through the existing poll-and-wait loop so a single node drives DDL. When a check* records SKIPPED_SHAPE_MISMATCH the dependent IndexRule / IndexRuleBinding reconciliation is also skipped — preventing the previous gap where the binding silently updated to a tag list that diverged from the live tag-family layout. AlarmRecord is opted in. Default remains false for all other models — boot-time reshape stays off unless the annotation is explicitly set. Operator caveat: BanyanDB does not physically migrate existing rows when a tag\u0026rsquo;s family changes; pre-existing data stays in its original on-disk location while new writes go to the declared family — expect a backfill window for queries that route through new IndexRules on relocated tags. Mask keywords trustStorePass, keyStorePass by default. Bump up dependencies to clear CVE alerts on shipped OAP jars: log4j 2.25.3 → 2.25.4, jackson 2.18.5 → 2.18.6, kafka-clients 3.4.0 → 3.9.2, postgresql 42.4.4 → 42.7.11, commons-compress 1.21 → 1.26.2. Bump up more dependencies to clear CVE alerts on shipped OAP jars: netty 4.2.12.Final → 4.2.15.Final, jackson 2.18.6 → 2.18.8, commons-codec 1.11 → 1.13. Also realign jackson-databind 2.16.0 → 2.18.8 so the whole jackson family is managed at a single version (it had been left behind the other jackson artifacts). Bump Apache Curator 4.3.0 → 5.9.0 and Apache ZooKeeper 3.5.7 → 3.9.5 together to clear CVE-2023-44981 (the bundled ZooKeeper jar carried it; OAP is a ZooKeeper client only, so the server-side bug was never reachable, but the jar tripped Dependabot). The cluster-zookeeper and configuration-zookeeper plugins use only stable Curator APIs, so no source changes were required. Operator-facing change: the supported ZooKeeper server version is now 3.6+ (Curator 5.x uses ZooKeeper persistent watches, added in server 3.6.0); older servers (3.5.x, 3.4.x) are no longer supported. Migrate the Consul cluster and configuration client from the abandoned com.orbitz.consul:consul-client 1.5.3 to the maintained fork org.kiwiproject:consul-client 0.9.0 to clear the okhttp CVE the old client carried (CVE-2021-0341; the old client pinned okhttp 3.14.9, fixed in okhttp 4.9.2+), so the BOM now pins okhttp to 4.12.0. The fork\u0026rsquo;s 0.9.x line is the last one built for JDK 11 (which SkyWalking still targets); 1.0.0+ is compiled to JDK 17 bytecode, so the migration stays on 0.9.0. The cluster-consul and configuration-consul plugins use only stable Consul client APIs, so the change is a package rename (com.orbitz.consul → org.kiwiproject.consul); okhttp is pulled only by the Consul plugins (the fabric8 Kubernetes client excludes its okhttp transport), so no other module is affected. Bump test-scope assertj-core 3.20.2 → 3.27.7 to clear CVE-2026-24400 (XXE in isXmlEqualTo, not used by any test). Clear three security alerts: bump the Airflow e2e mock\u0026rsquo;s pinned protobuf 4.25.8 → 5.29.6 (with opentelemetry-proto 1.24.0 → 1.28.0, whose protobuf\u0026lt;5.0 cap was the blocker, and grpcio 1.62.2 → 1.63.2, required because opentelemetry-proto 1.28.0\u0026rsquo;s gRPC stubs call unary_unary(_registered_method=...)) to clear CVE-2026-0994 — a CI-only test fixture, never shipped; and widen the cumulative count accumulator from int to long in SumHistogramPercentileFunction / AvgHistogramPercentileFunction to clear the CodeQL implicit-cast-in-compound-assignment alerts (count += value silently narrowed a long bucket-count sum back to int, while total was already long). Clear Dependabot CVE alerts in the e2e Go test fixtures (cases/go/service and cases/profiling/ebpf/network, CI-only, never shipped in any OAP artifact): bump golang.org/x/net 0.48.0 → 0.55.0 (CVE-2026-25681, CVE-2026-27136, CVE-2026-33814, CVE-2026-39821) and move the Go toolchain from 1.24 to 1.26.5 (CVE-2026-27145 / CVE-2026-42504 fixed in 1.26.4, CVE-2026-39822 fixed in 1.26.5) by switching the shared skywalking-go base image to the -go1.26 variant and bumping SW_AGENT_GO_COMMIT to 7544822, whose -go1.26 image ships go1.26.5. Fix: continuous profiling policy validation now rejects a threshold / count of 0 to match the error messages and rover\u0026rsquo;s value \u0026gt;= threshold trigger semantics (a 0 threshold would always trigger). CPU percent and HTTP error rate are tightened from [0-100] to (0-100]. Fix wrong BanyanDB resource options in record data. Align the default BanyanDB stage segmentInterval values so each coarser stage is an integer multiple of the finer one (records cold 3 → 4, metricsMinute cold 5 → 6, metricsHour warm 7 → 10 and cold 15 → 20), keeping hot → warm → cold lifecycle migration on the cheap whole-segment fast path. Fix: layer-extensions.yml is now excluded from the skywalking-oap jar and shipped to the distribution config/ directory, so an operator-edited config/layer-extensions.yml is no longer shadowed by the empty template bundled in the jar. Because the OAP launch script puts oap-libs/*.jar ahead of config/ on the classpath, ResourceUtils.read(\u0026quot;layer-extensions.yml\u0026quot;) previously always resolved the jar-bundled layers: [] and silently ignored the operator\u0026rsquo;s file — custom layers declared there never registered. The file now follows the same exclude-from-jar + copy-to-config/ packaging as every other operator-editable config (application.yml, alarm-settings.yml, etc.). Fix: the v2 MAL compiler now resolves custom layers referenced as Layer.NAME in an expression. A custom layer declared through a layerDefinitions: block (or layer-extensions.yml / the LayerExtension SPI) has no generated Layer.* static field, so service(['svc'], Layer.IOT_FLEET) previously failed code generation because Layer has no IOT_FLEET field. The compiler now lowers every Layer.NAME static-field reference to a runtime Layer.nameOf(\u0026quot;NAME\u0026quot;) registry lookup, so a custom layer can be referenced exactly like a built-in one (Layer.GENERAL). For a built-in layer this is equivalent, because Layer.nameOf(\u0026quot;GENERAL\u0026quot;) returns the same instance as the Layer.GENERAL field. The lowering is scoped to Layer only; the other MAL enum types (DetectPoint, DownsamplingType, etc.) are real Java enums and keep their direct static-field reference. Fix Envoy ALS rendering for the LAL live-debugger and the persisted log content: an Istio metadata-exchange peer in common_properties.filter_state_objects (legacy Wasm wasm.*_peer = Any{BytesValue} wrapping a FlatBuffer, or modern *_peer = Any{Struct}) is now decoded into the readable peer metadata (pod / namespace / labels) instead of an opaque jsonformat-failed envelope or base64. The serialization is hardened so a single un-printable field can no longer blank the whole entry — the LalPayloadDebugDump printer carries a well-known-type TypeRegistry and sanitizes every value JsonFormat would reject (an unresolvable, no-slash, or corrupt-bytes Any degrades to an @unresolved placeholder; a non-finite Value double NaN/Infinity is rendered as a string), keeping the rest of the entry readable. Because the LAL output builder\u0026rsquo;s bindInput runs eagerly before the debug capture, this also stops an unregistered filter_state_objects type from throwing and aborting the whole rule (dropping the mesh log). Decoding is wired through a new LalInputDebugRenderer SPI (EnvoyAlsHttpDebugRenderer / EnvoyAlsTcpDebugRenderer) so log-analyzer reaches the receiver-side decoders without depending on the Envoy receiver, and covers both HTTP and TCP access logs. Surface the effective BanyanDB configuration (bydb.yml / bydb-topn.yml) in the /debugging/config/dump admin API. Because the BanyanDB config moved to a separate file in 10.2.0, a BanyanDB deployment previously showed an empty storage.banyandb block in the dump; its post-environment-resolution values are now merged into the same response under storage.banyandb.* (TopN rules under storage.banyandb.topN.*), masked by the same secret-keyword list, via a generic ConfigDumpExtension SPI on ServerStatusService that any module loading config from a secondary file can implement. Fix: an MQE top_n(metric, N, order, attrX='value') query whose attribute is not a column of the target metric now returns a descriptive MQE error instead of a raw storage IOException surfaced as Internal IO exception, query metrics error.. Attribute columns (attr0..attrN) exist only on decorated metrics (service_* / endpoint_* / kubernetes_service_*, set to the layer name via OAL .decorator(...)) and the MAL meter base; metrics such as relations or database / cache / mq access carry none, so passing an attribute condition previously reached the storage engine with a tag it does not define and failed there. MQEVisitor now validates each attribute key against the metric\u0026rsquo;s registered queryable columns before the storage call and raises IllegalExpressionException (naming the attribute and the metric) when it is absent. Migrate all BanyanDB storage read queries from the typed query-builder API to BydbQL. Fix: BanyanDB queries no longer silently truncate at the storage engine\u0026rsquo;s implicit row cap. BanyanDB applies its own default limit to any query that carries none — 100 rows for measures, 20 for streams/traces — and applies it after GROUP BY, so an over-long result set is cut short rather than rejected. OAP never sent a limit on several read paths, so a metrics query returned at most 100 data points regardless of the requested range: a 4-hour minute-step read rendered only its first 100 minutes and the rest showed as empty, even though DurationUtils allows up to 500 steps. The same cap silently shortened topology relation maps, instance/process metadata lists, profiling thread snapshots and eBPF task lists. Every BydbQL query now leaves OAP with an explicit LIMIT: the entity-scoped metrics read sends the exact number of assembled duration points (matching the row set the ES/JDBC DAOs fetch by id), ad-hoc SELECT TOP sends its own N, and anything that does not paginate itself falls back to the configured resultWindowMaxSize (default 10000) instead of the engine default. ES and JDBC storage were never affected. Support BanyanDB\u0026rsquo;s group-scoped trace retention pipeline in bydb.yml. The trace and zipkinTrace groups gain a pipeline block (enabled, enabledEvents, mergeGraceSeconds, finalizeGraceSeconds, and an ordered plugins chain) that OAP pushes onto the BanyanDB group as a TracePipelineConfig, letting a sampler plugin drop traces inside the data node during Hot-phase compaction — after storage, so it reclaims space already written and decides per whole trace, unlike the ingest-side server-side trace sampling. Disabled by default, since it deletes stored traces. Even when enabled it is inert unless the data node runs the plugin-capable BanyanDB image with the sampler .so mounted — a node without that support ignores the config, and one that cannot load the plugin logs an error and merges unfiltered, so nothing is dropped unexpectedly. The two grace windows use -1 for \u0026ldquo;inherit the data node default\u0026rdquo; (30s merge / 5m finalize) because the node treats any non-positive grace as unset. enabledEvents accepts a comma-separated string so it can be set from the environment (SW_STORAGE_BANYANDB_TRACE_PIPELINE_ENABLED_EVENTS and its ZIPKIN_ variant) as well as a YAML block list; an empty value falls back to PIPELINE_EVENT_MERGE, so PIPELINE_EVENT_FINALIZE runs only when named explicitly. Each plugin\u0026rsquo;s config is passed through verbatim as a protobuf Struct: nested lists and objects (e.g. keepTagRules) now survive the config loader and are serialized as real ListValue/Struct rather than being flattened to a string. Note a float written as a ${ENV:default} placeholder still reaches the plugin as a JSON string, because the shared placeholder resolver only preserves String/Integer/Long/Boolean; the first-party samplers accept a quoted number for exactly this reason. See Trace Tail Sampling for how a trace is judged, and BanyanDB storage for the configuration keys. Fix: a blank value in bydb.yml (key: with nothing after it) aborted OAP startup with an opaque NullPointerException from java.util.Properties, which rejects null values. The BanyanDB config loader now skips blank entries and leaves the field at its default, the same outcome as omitting the line. Route LAL rules within a layer by their input type, so a single layer can host rules over different proto inputs. Each compiled rule now carries its effective input type (the proto class its parsed.* getters cast to, or null for parser-based / untyped rules), and LogFilterListener skips any rule whose type doesn\u0026rsquo;t match the incoming log instead of running every rule in the layer. This fixes a latent ClassCastException (caught and logged per log) that fired whenever a MESH log of one shape reached a rule compiled for another — e.g. an Envoy TCP access log or a network-profiling LogData hitting the HTTP envoy-als rule. Adds an envoy-als-tcp rule (inputType: TCPAccessLogEntry) alongside the existing HTTP envoy-als; both share the MESH layer and each now only sees its own entry type. Fix the PagerDuty alarm hook to default its Events API v2 endpoint to https://events.pagerduty.com/v2/enqueue. Fix HttpAlarmCallback logging a successful alarm delivery as a failure. The shared HTTP hook helper treated only 200 and 204 as success, so any other 2xx — notably the 202 Accepted returned by asynchronous intake APIs such as PagerDuty\u0026rsquo;s Events API v2 — produced send to ... failure. Response code: 202 at ERROR level on every delivered alarm. The alarm was still delivered; the log entry was wrong. The check now accepts the whole 2xx range, for all alarm hooks. Make the PagerDuty Events API v2 endpoint configurable through a new optional events-api-url setting on each pagerduty hook, defaulting to the US service region endpoint. An account in PagerDuty\u0026rsquo;s EU service region can now point the hook straight at https://events.eu.pagerduty.com/v2/enqueue rather than relying on PagerDuty forwarding the request — and the routing key and alarm payload from an EU-region account no longer transit the US region. Bump the default BanyanDB compatible server API version (SW_STORAGE_BANYANDB_COMPATIBLE_SERVER_API_VERSIONS) from 0.10 to 0.11. UI Add Airflow layer dashboards and menu i18n under Workflow Scheduler in Horizon UI (SWIP-7). Add mobile menu icon and i18n labels for the iOS layer. Fix metric label rendering in multi-expression dashboard widgets. Add i18n menu labels for WeChat Mini Program and Alipay Mini Program (en / zh / es) — sub-menus rendered as raw keys until this bump. Support trace V1 view in trace single page. Documentation Document the meter-analyzer-config catalog in the runtime-rule hot-update and DSL-debugging references, and add the optional layerDefinitions block, the active-files startup-failure behaviour, and a hot-update / debugging section to the meter setup doc. Update LAL documentation with sourceAttribute() function and layer: auto mode. Add Airflow monitoring setup documentation (SWIP-7). Add iOS app monitoring setup documentation. Add WeChat / Alipay Mini Program monitoring setup documentation, plus a client-side-monitoring section in the security guide covering public-internet ingress (OTLP + /v3/segments) for mobile / browser / mini-program SDKs. Improve downsampling documentation Fix the docker-compose quickstart: OAP healthcheck no longer calls curl (absent from the JRE image) and probes the query port via bash /dev/tcp; the Horizon UI service maps the correct container port (8081) and mounts a horizon.yaml (binding 0.0.0.0, OAP URLs, demo admin/admin login) instead of non-existent SW_*_ADDRESS env vars. Add PHP runtime metrics (PHM) dashboard documentation (agent setup, OAP php-runtime MAL rules, Horizon UI widgets). Add Node.js runtime metrics dashboard documentation (agent setup, OAP nodejs-runtime MAL rules, Horizon UI widgets). Add a BanyanDB trace tail sampling guide under \u0026ldquo;BanyanDB Exclusive Setup\u0026rdquo;, covering how a trace is judged (the OR-ed rule chain, the end-to-end duration envelope rather than a per-span maximum, and the deterministic trace-ID hash behind healthySampleRate), what the two first-party samplers read from each trace schema, the MERGE vs FINALIZE events and their grace windows, the fail-open behaviour when a plugin is absent or unloadable, and the metrics to watch. Also document the Zipkin receiver\u0026rsquo;s previously undocumented sampleRate and maxSpansPerSecond in the server-side trace sampling guide. Correct the APISIX monitoring guide to align its Collector configuration and metric names with the current APISIX MAL rules and Horizon UI Dashboard. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1100\"\u003e11.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eMove the DSL class-loading machinery under \u003ccode\u003ecore/dsl\u003c/code\u003e. \u003ccode\u003ecore/classloader\u003c/code\u003e held only DSL …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/latest/en/changes/changes/","title":"11.0.0"},{"body":"11.0.0 Project Move the DSL class-loading machinery under core/dsl. core/classloader held only DSL types — RuleClassLoader, DSLClassLoaderManager, ClassLoaderGc, UnloadProbePayload and BytecodeClassDefiner — so it is now core/dsl/classloader, and Catalog moves to core/dsl because a rule-file taxonomy is not a class-loading concern. Three copies of the \u0026ldquo;define a generated class into the right loader\u0026rdquo; dispatch (MAL, LAL, MeterSystem) collapse into a static BytecodeClassDefiner.define, which also gives the JDK 17 --add-opens rationale one home instead of four. Three of the four copies of the generated-class dump-directory lookup become DslGeneratedFileWriter.resolveClassDumpDir; OAL keeps its own, because its debug flag is settable independently of the environment variable and two tests rely on that. No behaviour change. Extend the GET /inspect/entities admin API to inspect a metric persisted by any OAP, even one this node does not define locally. When the metric is unknown to the local registry, the caller supplies valueColumn + valueType and the storage backend resolves the physical index/table/group from its own running config (no DB schema/table-metadata read): ES uses the merged metrics-all index + metric_table discriminator, JDBC probes the node\u0026rsquo;s function tables by the table_name discriminator, and BanyanDB synthesizes a read-only measure schema. Scope is no longer required — the entity_id is decoded structurally (service / 2nd-level / relations) with a generic name leaf. Locally-defined metrics keep the exact field names, scope, and mqeEntity as before. Add the POST /inspect/values admin API — read the value series of a metric persisted by another OAP (one this node does not define locally) by supplying its {valueColumn, valueType}. The real MQE engine runs over a request-scoped InspectQueryContext overlay (provide-if-absent — the local catalog always wins) that makes the foreign metric look registered to every read path: ValueColumnMetadata resolves its value column / type / scope, and the storage location registries resolve where it lives (MetadataRegistry synthesizes a BanyanDB measure schema, IndexController resolves the ES metrics-all index, TableHelper probes the JDBC function tables), so the read returns the native MQE ExpressionResult with no per-DAO special-casing. Admin-only (a forced read this OAP cannot validate); not mirrored onto the public REST / GraphQL surface. See the Inspect API. Remove the always-on alarm-to-event conversion (EventHookCallback). A triggered alarm is no longer synthesized into the events pipeline as an Alarm/AlarmRecovery event; events now originate only from real event sources (agents, SkyWalking CLI, Kubernetes Event Exporter). Alarms remain available through the alarm store (getAlarm/queryAlarms) and the configured alarm hooks. This drops a documented \u0026ldquo;Known Event\u0026rdquo; and removes 1-2 synthetic event records per alarm fire. TLS for all OAP HTTP/REST servers, with cert hot-reload. Adds the restSSLEnabled / restSSLKeyPath / restSSLCertChainPath config structure to every OAP HTTP server — core REST, sharing-server, admin, PromQL, LogQL, TraceQL and Zipkin query/receiver — each with its own dedicated environment variables (SW_CORE_REST_SSL_*, SW_RECEIVER_SHARING_REST_SSL_*, SW_ADMIN_SERVER_REST_SSL_*, SW_PROMQL_REST_SSL_*, SW_LOGQL_REST_SSL_*, SW_TRACEQL_REST_SSL_*, SW_QUERY_ZIPKIN_REST_SSL_*, SW_RECEIVER_ZIPKIN_REST_SSL_*). The shared Armeria HTTPServer reloads the key pair from disk on rotation (via TlsProvider.ofScheduled) so refreshed certificates are picked up without restarting the OAP, matching the existing gRPC SSL hot-reload behavior. HTTP TLS is server-side only (no mTLS). New queryAlarms GraphQL query — entity / layer / rule filters for alarms. Adds a comprehensive alarm query API alongside the legacy getAlarm. The new queryAlarms(condition: AlarmQueryCondition!): Alarms accepts a single input type bundling every filter the alarm record stores: entities: [Entity!] (reuses the MQE Entity shape — pin to specific services / instances / endpoints / processes or their relations, matched against alarm id0 OR id1); layer: String (filter by the alarmed entity\u0026rsquo;s layer — single match, since alarm rows persist one layer); ruleNames: [String!] (filter by which alarm rule fired); plus keyword, tags, duration, paging. Legacy getAlarm is marked @deprecated but still routes to the same DAO — no client breakage. Backend additions: a new layer column on AlarmRecord populated at alarm-mint time via MetadataQueryService.getService(serviceId).getLayers(); the existing id0/id1 columns flipped from storageOnly = true to indexed so the entity filter pushes down to storage. IAlarmQueryDAO.queryAlarms(condition, limit, from) is a new abstract method — 3rd-party storage backends fail at compile if they miss the override (SWIP-14 pattern). All three bundled backends implement it: BanyanDB / Elasticsearch / JDBC. Operator semantics: (1) Relation entities are exact-match. Passing {scope: ServiceRelation, serviceName: A, destServiceName: B} matches only the alarm where id0=serviceId(A) AND id1=serviceId(B), not any alarm that touches A or B on either side. Wider \u0026ldquo;anything involving A\u0026rdquo; queries should pass the individual non-relation entity instead ({scope: Service, serviceName: A} — which expands to id0=A OR id1=A). (2) Single layer per alarm row. The persisted column stores ONE layer (the first entry of the entity\u0026rsquo;s resolved layer list — source-first for relations). A service in [GENERAL, K8S_SERVICE] whose metadata resolves to GENERAL first is filed under GENERAL; querying layer: \u0026quot;K8S_SERVICE\u0026quot; will miss it. Operator migration note: existing pre-upgrade alarm rows continue to be filterable by the legacy getAlarm fields; the new entity / layer / rule filters in queryAlarms apply only to alarms written after the upgrade (existing storage indices don\u0026rsquo;t transition index: false → true in place; new daily-rolled indices pick up the indexed columns). Schema additions are non-blocking — bootstrap silently skips column-attribute changes on existing indices. 🚨 Breaking change: apm-webapp and the skywalking-booster-ui submodule are removed. This OAP distribution no longer ships a bundled web UI. The legacy Armeria reverse proxy in apm-webapp/ (the binary that powered the skywalking/ui Docker image) and the skywalking-ui git submodule (which tracked apache/skywalking-booster-ui) are both deleted along with the docker.ui Maven target, the skywalking/ui Docker image build, the apm-dist/ webapp packaging, and every CI workflow path that built or pushed the UI image. The official UI is now Horizon UI, a SkyWalking sub-project that releases independently of the OAP backend on its own schedule, with released container images on Docker Hub at apache/skywalking-ui (tags latest / horizon-\u0026lt;version\u0026gt;; per-commit development images live on ghcr.io/apache/skywalking-horizon-ui). There is no 1:1 mapping between OAP versions and Horizon UI versions — operators pin the UI image tag in their deployment and upgrade the two on separate cadences. Horizon UI consumes the OAP\u0026rsquo;s public GraphQL/REST surface (default 12800) and the admin host (default 17128). The on-disk dashboard seed files in oap-server/server-starter/src/main/resources/ui-initialized-templates/ are deleted; UITemplateInitializer / UIMenuInitializer are removed from CoreModuleProvider.notifyAfterCompleted(), and Horizon UI ships its own dashboard library and its own sidebar menu. UI templates are now created and updated through the new /ui-management/templates/* REST surface on admin-server (see below). All UI-related GraphQL mutations and queries (UIConfigurationManagement: addTemplate, changeTemplate, disableTemplate, getAllTemplates, getDashboardConfiguration, getMenuItems) are retired from the public GraphQL schema, along with the SW_ENABLE_UPDATE_UI_TEMPLATE flag. The OAP backend also no longer stores or serves the sidebar menu — UIMenuManagementService, UIMenuManagementDAO, UIMenu, MenuItem, and the storage impls are all removed; Horizon UI owns the menu client-side and uses listServices(layer:...) for dynamic \u0026ldquo;layer has services\u0026rdquo; gating. Upgrade path: replace skywalking/ui:\u0026lt;tag\u0026gt; with the Horizon UI image apache/skywalking-ui:latest (or a horizon-\u0026lt;version\u0026gt; tag — pick a version per Horizon UI\u0026rsquo;s OAP-compatibility notes, OAP 11.0+ is supported) in your deployment, expose port 17128 from the OAP container, and migrate any scripts that called the legacy GraphQL UI mutations to the REST endpoints under UI Management API. All status / debug endpoints (/status/*, /debugging/*) also move to admin-only — the public REST dual-bind for status is retired in the same release. New ui-management admin module — REST surface for dashboard templates. Hosts five operations on admin-server (port 17128): GET /ui-management/templates, GET /ui-management/templates/{id}, POST /ui-management/templates, PUT /ui-management/templates, POST /ui-management/templates/{id}/disable. Forwards to the existing UITemplateManagementService (no storage DAO changes). Enabled by default (SW_UI_MANAGEMENT=default, on a default-on admin host). Replaces the retired GraphQL UIConfigurationManagement template resolver. The sidebar menu is intentionally NOT served — see the breaking-change entry above. Operator reference: UI Management API. All admin feature modules default-on. admin-server, status, inspect, ui-management, dsl-debugging, and receiver-runtime-rule all default to enabled. Operators who don\u0026rsquo;t want a particular feature set its SW_* env var to empty. This closes a usability gap from 10.4.0 where the runtime-rule / dsl-debugging surfaces required explicit opt-in even though the admin host was already on. Status API moved to admin-host. Status / debug routes (/status/*, /debugging/*) now register on the admin-server REST host (default 17128); they no longer mirror on core.restPort (default 12800). Aligns status with every other admin feature module (inspect, dsl-debugging, runtime-rule, ui-management). Horizon UI consumes status from the admin host. URIs and payloads are unchanged; only the host moved. One exception: /status/config/ttl is also bound on the public REST host (12800) so ecosystem tools that discover TTL bounds via REST before issuing /graphql don\u0026rsquo;t need to learn the admin port. New admin-server module — shared host for admin / on-demand write APIs. Runs on two ports: an HTTP REST surface (default 17128) for operator-facing endpoints, and an admin-internal gRPC bus (default 17129) for peer-to-peer cluster RPCs (runtime-rule Suspend / Resume / Forward; DSL debug install / collect / stop / stopByClientId). The admin-internal bus is a dedicated transport separate from the public agent / cluster gRPC port (core.gRPCPort, default 11800) so privileged admin RPCs stay out of the agent network\u0026rsquo;s blast radius — operators bind gRPCHost to a private peer-to-peer interface only. Both the runtime-rule plugin and the new DSL Debug API (below) mount onto this shared host. Enabled by default so the status feature module is reachable out of the box; the host binds to 0.0.0.0:17128 and has no built-in authentication and must be gateway-protected with IP allow-lists, never exposed to the public internet (see the Admin API security notice). Set SW_ADMIN_SERVER= (empty) to disable entirely. The runtime-rule config block loses its restHost/restPort/restContextPath/restIdleTimeOut/ restAcceptQueueSize/httpMaxRequestHeaderSize keys (and the matching SW_RECEIVER_RUNTIME_RULE_REST_* env vars); host-level knobs move under the new admin-server block (SW_ADMIN_SERVER_HOST / SW_ADMIN_SERVER_PORT / SW_ADMIN_SERVER_GRPC_HOST / SW_ADMIN_SERVER_GRPC_PORT / SW_ADMIN_SERVER_INTERNAL_COMM_TIMEOUT etc.). Runtime rule hot-update for MAL and LAL. Operators can now ship metric (MAL) and log (LAL) rule changes without restarting OAP. A push to a new admin endpoint persists the rule to the configured storage backend, and every node in the cluster converges to the new content within ~30 seconds. Common workflows: addOrUpdate — create or replace a rule. Body is the raw YAML you would normally ship with OAP\u0026rsquo;s static rule files. Returns 200 once the rule is applied locally and persisted; peers pick it up on their next periodic scan (≤ 30 s). inactivate — soft-pause a rule. The OAP stops emitting metrics for that rule but the backend measure (and its history) is preserved, so a later addOrUpdate to the same (catalog, name) is lossless. The \u0026ldquo;off\u0026rdquo; intent is durable across reboots; bundled rules on disk are not auto-resurrected when an inactivate removes the runtime override. This is the safe way to take a rule offline. delete — removes an INACTIVE row (active rules return 409 requires_inactivate_first). For runtime-only rules with no bundled YAML on disk, the row is dropped; the backend measure (if any) is left in place as an inert artefact, matching bundled-rule deletion semantics (removing a YAML from otel-rules/ on disk doesn\u0026rsquo;t drop its measure either). For rules that have a bundled YAML twin, plain delete returns 409 requires_revert_to_bundled — letting bundled silently take over the (catalog, name) is a meaningful state change that requires an explicit operator decision. Re-issue with ?mode=revertToBundled to fall back to bundled: that path runs the schema-change pipeline (rehydrates the runtime DSL locally, then applies the bundled YAML through the standard apply pipeline so the runtime→bundled delta drops runtime-only metrics, registers bundled-only metrics, and reuses bundled-shared metrics at matching shape) before removing the row. Returns 400 no_bundled_twin when ?mode=revertToBundled is used without a bundled YAML on disk. get / bundled / list / dump — read-side endpoints for fetching a single rule\u0026rsquo;s YAML (with ETag support; ?source=bundled reads the on-disk bundled YAML even when a runtime override is in place), listing the bundled-vs-runtime overlay per catalog, inspecting cluster-wide rule state as a JSON envelope ({generatedAt, loaderStats, rules} — each row carries status/localState/loaderKind/bundled/bundledContentHash so a UI can render override badges without a second roundtrip), and exporting all rules as a tar.gz for backup / DR. Hot-updates survive OAP restart: at boot OAP merges bundled rule files with persisted runtime rules, so the cluster never silently regresses to the bundled defaults. All admin writes for a runtime-rule cluster serialize on a single \u0026ldquo;main\u0026rdquo; OAP (deterministic sorted-first peer, no leader election) — non-main nodes that receive an HTTP write transparently forward it to the main over the admin-internal gRPC bus, so an L7 load balancer in front of the admin port can route any operator request to any OAP. Cluster convergence on the periodic refresh tick is configurable via receiver-runtime-rule.refreshRulesPeriod (default 30 s). The endpoint is disabled by default and listens on port 17128 (HTTP) when enabled. It has no built-in authentication — operators must gateway-protect it with IP allow-lists and never expose it to the public internet. Routes mount on the new admin-server HTTP host, which is on by default; enable the runtime-rule feature with SW_RECEIVER_RUNTIME_RULE=default. Live debugger for MAL / LAL / OAL — implements SWIP-13 Live Debugger for MAL / LAL / OAL. Sample-based runtime debugger that captures per-stage inputs/outputs as the three DSLs process live ingest. Idle-path cost is one volatile-bool read per probe call site that JIT eliminates after warm-up; active sessions fan out to every cluster peer over the admin-internal gRPC bus so each peer captures its own slice. The fan-out is LB-safe: any node can serve any verb (POST mints sessionId on the receiving node, broadcasts install to peers, returns 404 rule_not_found only when no node owns the rule), so an L7 load balancer in front of the admin port routes operator requests freely. Mounts on the shared admin-server host (/dsl-debugging/* for session control plane, /runtime/oal/* for the OAL rule picker). Disabled by default; enable with SW_DSL_DEBUGGING=default (admin-server itself is on by default). injectionEnabled is a boot-time codegen switch defaulting to true — once the module is enabled, probes fire and sessions record samples; set false only if the REST surface is wanted but no codegen-side probe overhead is acceptable. Per-session limits enforce hard caps (recordCap ≤ 10000, retentionMillis ≤ 1 hour) — out-of-range requests return 400 invalid_limits. LAL sessions accept a per-session granularity=block|statement flag — block mode captures the parser/extractor/sink stages; statement mode additionally records one line entry per individual extractor statement, carrying the source-line number and verbatim DSL text so the UI can highlight which statement fired. MAL captures render the file-level filter\u0026rsquo;s surviving SampleFamily map ({\u0026quot;families\u0026quot;: N, \u0026quot;items\u0026quot;: [...]}), so multi-metric expressions show cross-family filter narrowing in the captured payload. Capture payloads include raw log bodies and parsed maps — treat the admin port as authenticated infrastructure per the Admin API security notice. Per-DSL operator references: MAL, OAL, LAL. BanyanDB schema mismatches are now visible at boot, not silent. If BanyanDB already holds a resource whose shape doesn\u0026rsquo;t match what the current rule declares (e.g., a rule was edited on disk while OAP was offline), OAP now skips that resource, logs an ERROR with the declared-vs-backend diff, and continues booting — previously the mismatch was silently accepted and samples for the affected resource were quietly dropped. To re-shape a mismatched metric, push the desired YAML through POST /runtime/rule/addOrUpdate. Bump infra-e2e to testcontainers-go v0.42.0 (apache/skywalking-infra-e2e#146), which uses Docker Compose v2 plugin natively and removes docker-compose v1 dependency. Remove deprecated version field from all docker-compose files for Compose v2 compatibility. Best-effort schema-cutover fence for BanyanDB. After firing a schema install or drop OAP now waits up to a bounded window (default 2s) for every BanyanDB data node to apply the change before resuming dispatch — the typical case gets a clean cutover where samples after 200 OK use the new shape. On laggard timeout, OAP logs a warning and proceeds anyway so a single slow node doesn\u0026rsquo;t wedge the apply. Bump dependencies: gRPC 1.70.0 → 1.80.0, protobuf-java 3.25.5 → 4.33.1, Netty 4.2.10.Final → 4.2.12.Final, Netty-tcnative 2.0.75 → 2.0.77, pgv (protoc-gen-validate) 1.2.1 → 1.3.0. Driven by the new BanyanDB schema-consistency RPCs whose generated validation code requires the protobuf-java 4.x runtime. Inspect API on admin-server. Two new admin-only HTTP endpoints for browsing the live metric catalog and the entities currently emitting values for a given metric. GET /inspect/metrics lists every registered metric with its type / scope / catalog / value-column name / supported downsamplings (pure metadata, no I/O). GET /inspect/entities runs the storage backend\u0026rsquo;s entity scan for a metric over a time range + step (capped at 300 rows) and returns each entity decoded into an MQE-ready payload — the response includes a mqeEntity block the operator pastes verbatim into the public GraphQL execExpression mutation, plus the source service\u0026rsquo;s layer(s) (multi-layer services emit one row per layer). Restricted to REGULAR_VALUE / LABELED_VALUE metrics and to non-Process scopes; HEATMAP / SAMPLED_RECORD / Process / ProcessRelation return 400. Adds IMetricsQueryDAO.listEntityIdsInRange as an abstract method on the interface — any 3rd party storage backend must explicitly override or the build fails. Enabled by default (both SW_INSPECT and SW_ADMIN_SERVER are on by default); set SW_INSPECT= empty to disable. Operator reference: Inspect API. Status feature module relocation, finalized. The legacy status-query-plugin was replaced by a new status feature module under server-admin/; the route set (/status/cluster/nodes, /status/alarm/*, /status/config/ttl, /debugging/config/dump, /debugging/query/*) keeps URIs and payloads unchanged. The selector renames from the QUERY-plugin form (SW_QUERY=…,status-query-plugin) to a top-level SW_STATUS=default (on by default); custom application.yml overrides referencing status-query need to repoint to status. Routes are admin-host only — see the \u0026ldquo;Status API is admin-host only\u0026rdquo; entry above for the public REST retirement. Drop six unused test-scoped dependencies from runtime-rule (library-integration-test, library-banyandb-client, storage-banyandb-plugin, testcontainers, testcontainers:junit-jupiter, grpc-testing). They staged the plugin-side ITs that were retired in favour of e2e; that coverage now lives in test/e2e-v2/cases/runtime-rule/ (MAL over BanyanDB / PostgreSQL / Elasticsearch, LAL, meter, and the two-node cluster case). The module has no ITs today, and JUnit and Mockito are inherited from the root POM. Declare server-testing at test scope everywhere. It ships only test scaffolding (ModuleManagerTesting, MockModuleManager, the MAL/LAL/Hierarchy rule loaders) plus two empty org.junit stubs that let Testcontainers\u0026rsquo; GenericContainer hierarchy resolve without JUnit 4, but four modules declared it at compile scope — including the server-configuration parent, so all eight configuration-* children inherited it — which put those org.junit stubs on the runtime classpath that server-starter copies into oap-libs. Modules whose tests need the stubs now declare the dependency themselves rather than inheriting it transitively, and library-banyandb-client gains the direct library-util dependency its BanyanDBClient always needed (it was resolving StringUtil through server-testing, a test-support module). Add ThreadPolicy.ioBound(N) to library-batch-queue, for queues whose consumers spend most of their time blocked. Such a queue runs its drain loops on virtual threads where the runtime provides them (JDK 25+) and falls back to N platform threads otherwise; the count, and therefore concurrency, batching, back-pressure, drop semantics and per-partition ordering, are identical on both paths. Shutdown latency is the one exception: the platform scheduler drops drain tasks parked on their idle backoff, while the virtual-thread adapter sleeps inside the submitted task and cannot, so an ioBound queue should keep maxIdleMs within shutdownTimeoutMs. There is deliberately no CPU-proportional form: virtual threads are not preemptive, so CPU-bound work would hold its carrier and starve the shared carrier pool, and L1/L2/TopN stay on cpuCores/fixed. Also fixes BatchQueue.shutdown(), which ran its final drain on the caller\u0026rsquo;s thread while drain loops could still be inside consume(), invoking a handler concurrently and breaking the single-drain-thread invariant workers such as MetricsAggregateWorker rely on: it now cancels the periodic rebalance task, waits for in-flight consumers (shutdownTimeoutMs, default 500ms per queue), and serialises its final dispatch behind a read/write dispatch lock so the guarantee holds even when that wait times out or is interrupted. Drain loops hold the read lock for the whole cycle — the running recheck, the partition dequeue, the idle notification and the dispatch — because onIdle() touches the same worker state as consume() and an unlocked dequeue would let shutdown dispatch a newer batch ahead of one a task already holds. Concurrent shutdown callers await the winner\u0026rsquo;s completion rather than returning early. A consumer is never interrupted mid-batch. OAP Server Add component IDs for the Spring LDAP Java agent plugin (spring-ldap: 179) and LDAP server (LDAP: 180), including their server mapping. Fix LAL\u0026rsquo;s segmentId and spanId extractor statements, which the grammar accepted and the parser never implemented. LALParser.g4 declares traceIdStatement, segmentIdStatement and spanIdStatement, and the codegen already carried setSegmentId/setSpanId in its setter table, but LALScriptParser.visitExtractorStatement had a branch for only the first of the three. The remaining alternatives fell through to a line that assumed whatever was left had to be an ifStatement, so a rule writing segmentId ... failed at boot with a NullPointerException naming IfStatementContext — for a rule line containing no if. Both statements now work, and an unhandled extractor statement reports its own rule line instead of throwing. Existing log records are unaffected: LogBuilder copies trace id, segment id and span id straight from the log\u0026rsquo;s metadata, and only skips that copy when a rule has set them — which no shipped rule did, which is why the gap went unnoticed. Dedicated execution tests now cover reading all three fields from log.traceContext.* and writing all three from an extractor. Remove dead code from the DSL subsystem and correct the shared kernel\u0026rsquo;s own documentation. Deleted DslContentHash (a byte-identical, zero-caller twin of the live ContentHash), the unused oal-rt metrics-function registry, LogAnalyzerFactory, LALCodegenHelper.METADATA_GETTER_ALIASES (a permanently empty map whose reader branch could never execute — the DSL-name-to-getter mismatch it existed for no longer exists, since LogMetadata.TraceContext names the field traceSegmentId directly), and three unreferenced members. Three kernel classes carried javadoc asserting consumers that do not exist — DSLClassLoaderManager claimed the MAL and LAL compile paths reach for its singleton, LogDataDebugDump claimed core renders through it, and DslContentHash instructed the reader to consolidate toward the dead copy; a false rationale in a shared kernel is an instruction to the next contributor, so those are now what the call sites actually support. OALDebug, OALDebugRecorder and DebugHolderProvider now record why they sit in core while MAL\u0026rsquo;s and LAL\u0026rsquo;s equivalents do not: dsl-debugging declares no oal-rt dependency, so they cannot move. Unify source attribution for every generated DSL class, so a stack frame from OAL, MAL, LAL or Hierarchy code leads back to the rule that produced it. SourceFile now names the RULE and its line, then the generated class file: (otel-rules/activemq/activemq-broker.yaml:32)otel_rules_activemq_activemq_broker_L32_service_meter.java. The .java generated source file is written only under SW_DYNAMIC_CLASS_ENGINE_DEBUG, so in a default deployment naming it named nothing; the class name cannot substitute because sanitising maps /, - and . all to _ and drops the extension. The _L\u0026lt;n\u0026gt;_ segment was the rules-list index rather than a line, so every file\u0026rsquo;s first rule reported as L0. It now carries the rule\u0026rsquo;s real line, resolved by the loaders themselves — including Zabbix and Hierarchy, which supplied no coordinate at all, and the runtime-rule hot-update paths for MAL and LAL, which disagreed with their own boot loaders. LineNumberTable held statement ordinals, matching neither the YAML nor the generated source. MAL keeps a per-statement table; MAL closure companions, LAL and OAL carry one entry at each method\u0026rsquo;s signature, found by searching the assembled generated source file text for that method\u0026rsquo;s declaration rather than counting a per-method offset. Hierarchy writes no generated source file, so it carries none. Generated class names change. They are now built from the rule file\u0026rsquo;s catalog-qualified path, so a MAL class that was vm_L25_cpu_total_percentage is now otel_rules_vm_L25_cpu_total_percentage. The catalog is kept only where the generated class\u0026rsquo;s package does not already imply it, so LAL — one catalog, its own package — is unchanged at default_L3_default. Nothing addresses these classes by name — they are generated, loaded reflectively and never referenced from configuration — but the names appear in stack traces, in SW_DYNAMIC_CLASS_ENGINE_DEBUG dumps and in dsl-debugging output, so saved greps and dashboards keyed on the old form need updating. The unqualified names were ambiguous: two catalogs can each hold a vm.yaml. A generated method\u0026rsquo;s line is located by searching the assembled source for its declaration, and that search now requires an identifier boundary. serialize is a suffix of deserialize and OAL declares both on every metrics class, so the shorter name resolved to the longer method\u0026rsquo;s line whenever the template order changed; two OAL metrics named cpm and commando_cpm in one scope reach the same shape through their shared dispatcher. The wrong line shipped in production bytecode — the attribute is stamped unconditionally — so a frame reported another rule\u0026rsquo;s location. Fix a hot-updated LAL rule stranding its own dsl-debugging binding instead of replacing it. The RuleKey naming a rule file was spelled default.yaml by the boot loader and default by the runtime-rule engine, so the two never met in the holder registry: the pre-update GateHolder stayed reachable, and an operator addressing it enabled probes on a compiled rule that no longer evaluates anything — indistinguishable from a rule receiving no traffic. RuleKey now drops a trailing .yaml/.yml from that component, so both spellings are one key and the debugging API keeps accepting either. Rule execution was never affected: the maps that decide which rules run are keyed by layer and rule name, not by file name. A class shared by several rules — an OAL dispatcher — names its file without a line rather than borrow the first rule\u0026rsquo;s, which would misreport every other rule routed through it. The generated .java source files are written as UTF-8 with an ASCII header instead of through a platform-default FileWriter, the pairing that produced MalformedInputException on a non-UTF-8 JVM. Breaking change: HierarchyDefinitionService.HierarchyRuleProvider — a ServiceLoader SPI — now declares buildRules(Map\u0026lt;String, String\u0026gt; ruleExpressions, Map\u0026lt;String, Integer\u0026gt; ruleLines) in place of the former one-argument buildRules(Map). A third-party provider compiled against the old signature fails with AbstractMethodError and must be recompiled. The second argument carries each rule\u0026rsquo;s line in hierarchy-definition.yml, which the expression map cannot supply because snakeyaml\u0026rsquo;s bean binding discards positional marks; a provider with no line information should pass an empty map. Without it, generated hierarchy classes are labelled _Lunknown_ and their stack frames lead nowhere. The mechanism is one implementation in org.apache.skywalking.oap.server.core.dsl, shared by all four compilers: the coordinate model, the class-name builder, the SourceFile value, the generated source file writer, the signature-line lookup and the bytecode attributes. Previously each compiler had its own, and they had drifted apart. Support runtime rule hot-update and DSL debugging for the meter-analyzer-config catalog, bringing native meter (MeterReportService) rules to parity with otel-rules. Meter rules now load through the same Rules/Rule pipeline otel-rules uses, so they participate in RuleSetMerger, are recorded in StaticRuleRegistry, support the optional layerDefinitions block, and generate source-named expression classes instead of falling back to MalExpr_\u0026lt;N\u0026gt;. MeterProcessService now implements MalConverterRegistry and publishes debug holders at boot, so a meter rule can be added / overridden / inactivated at runtime, and attached to a DSL debug session, without restarting the OAP. The internal MeterConfig / MeterConfigs model is removed in favour of the shared one. Behaviour change: an entry in meterAnalyzerActiveFiles (SW_METER_ANALYZER_ACTIVE_FILES) with no matching rule file now fails OAP startup instead of being silently ignored, matching how otel-rules has always behaved. Support Elasticsearch 9.x as storage. Add Node.js runtime metrics via the Node.js agent MeterReportService pipeline (meter_instance_nodejs_*, default 20s sample/report). OAP analyzes raw meters through nodejs-runtime.yaml. Node.js E2E asserts twelve meter_instance_nodejs_* metrics (test/e2e-v2/cases/nodejs/e2e.yaml). Add PHP runtime PHM meter analyzer (php-runtime.yaml) for SkyWalking PHP agent process metrics (CPU, memory, virtual memory, thread count, open file descriptors sampled from /proc on Linux). Registers six meter_instance_php_* metrics on the General Service layer; php-runtime is included in the default meterAnalyzerActiveFiles. Batch the BanyanDB schema fence per runtime-rule apply. A runtime-rule file changes dozens of rules at once, but the post-DDL fence (SchemaWatcher.awaitRevisionApplied) ran once per metric/downsampling, so a large file did K×M sequential ≤2s fences — on a laggy cluster that overran the apply\u0026rsquo;s REST budget. The main-node apply path now uses StorageManipulationOpt.withSchemaChangeDeferredFence(): the installer records each resource\u0026rsquo;s mod_revision without fencing and registers a single flush that the apply runs once on the file\u0026rsquo;s max revision, collapsing the whole file to one barrier. The flush is one-shot — a reconciler tick reuses one opt across every rule file, so after a file flushes, the closure and accumulated revision reset and each file fences on its own DDL only. Drops still fence inline on the dropped resource\u0026rsquo;s own delete revision — or, when that delete recorded no tombstone (mod_revision == 0), on a key-based deletion barrier (AwaitSchemaDeleted) — never on the shared opt\u0026rsquo;s cumulative revision, so a tombstone-less delete in a multi-file tick is still confirmed removed. On the operator REST apply the single create/update fence runs on a configurable, generous budget (default 180s) in the background before the rule row is persisted and dispatch resumes — it gates the persist + local commit + peer resume so the durable commit point is only reached once the schema is confirmed cluster-wide, and writes never resume against an un-propagated schema (see the apply-status entry below); the reconciler tick keeps the short inline 2s fence (a background reconcile must not wait minutes per file). Peer / withoutSchemaChange applies are unaffected (no fence). Add a runtime-rule apply-status query. The cluster main now tracks each structural apply through a phase machine (SchemaApplyCoordinator: pending → DDL → fencing → rolling-out → applied, with degraded for a committed-but-unconfirmed apply — the cluster schema fence did not confirm within the timeout, in which case the lagging data-node ids are surfaced as fenceLaggards and dispatch is resumed anyway, or the local commit-tail threw — and failed carrying the specific reason). The schema fence runs on a configurable, generous budget (receiver-runtime-rule.deferredFenceTimeoutSeconds, default 180s) and gates everything durable or visible: because an un-propagated write is silently dropped at the data node, the order after a successful DDL is suspend → DDL → fence → persist → commit → resume. The rule row (the durable commit point) is written only AFTER the fence confirms, so \u0026ldquo;durable\u0026rdquo; implies \u0026ldquo;schema propagated cluster-wide\u0026rdquo; — a main crash before persist leaves no row (peers/crash-recovery stay safely on the old content; the orphaned measure is inert), and any durable row is guaranteed fence-confirmed, so convergence never resumes dispatch against an unpropagated schema. The fence + persist + resume run in the background so they never block the HTTP response — POST /addOrUpdate returns its applyId immediately at fencing (accepted, not yet durable; dispatch for that rule still paused — a clean gap, not dropped writes), and the operator polls GET /runtime/rule/status to watch fencing → rolling-out → applied (or degraded/failed); on a genuine laggard, dispatch resumes after the budget so one stuck node can\u0026rsquo;t park the metric forever. A GetApplyStatus admin-internal gRPC served by the main backs the query — by applyId, or by catalog+name (+ optional contentHash, the durable identity) once the handle is gone after a page refresh. When the live status is gone (apply-id evicted, main restarted, or the main is unreachable), the query degrades to the durable rule row: a matching ACTIVE row reports applied derived from the content hash (a durable row is, by the fence-then-persist order, already propagation-confirmed). Non-main nodes route the read to the deterministic main; status is in-memory by design, with the content hash reconstructing truth after a restart. Push runtime-rule convergence to peers on commit. After a successful structural apply — and on the commit_deferred path, where the DB row is durable but this node\u0026rsquo;s commit-tail threw — the main broadcasts a NotifyApplied admin-internal RPC so peers reconcile against the just-persisted DB row immediately, instead of waiting up to one refresh tick (~30s) to notice it. The fan-out runs off the REST response thread (fire-and-forget on a daemon executor) so an unreachable peer\u0026rsquo;s per-call deadline never adds to the operator\u0026rsquo;s apply latency. On the peer side the notify-triggered reconcile is coalesced: a burst of notifies (a multi-rule file, or several applies) collapses to a single queued full reconcile rather than one redundant dao.getAll() scan per notify. The notify is best-effort and idempotent (the peer runs its normal per-file-locked reconcile; a lost notify is harmless — the peer still self-converges on its next tick), so it tightens the cluster-convergence window without adding a hard dependency on the main being reachable. Fix BanyanDB peer nodes permanently flooding \u0026lt;metric\u0026gt; is not registered, and a follow-on case where a peer kept translating writes with a stale schema shape after a runtime-rule reshape, when a node held a live persist worker but its local MetadataRegistry schema cache was missing or stale for that model — a withoutSchemaChange peer apply or a runtime-rule bundled fall-over rebuilt the dispatch worker but skipped the local-cache populate, and the registry was insert-only (never evicting) while the 30s reconcile only covers runtime-rule rows, so nothing re-derived it. The peer / local-cache-only install path now (re)derives and overwrites the local schema entry from the declared model with zero server RPC — honoring the inspectBackend=false contract so the cache can never lag the worker, including across a reshape — and a model removal now evicts its cache entry so a dropped or reshaped model leaves no stale translation behind; the persist DAOs keep an RPC-free re-derivation as a read-side backstop, and the no-init defer poll loop retries a transient backend probe error instead of escaping and crash-looping the pod. Support LAL json {} parsing JSON content delivered in a plain-text log body. The parser reads the native protocol\u0026rsquo;s JSON body first; when that is empty, it tries the text body as JSON — e.g. the OTLP log receiver maps every OTLP string body to a text body, even JSON-shaped ones, so previously-aborting json {} rules on OTLP-fed layers now work without any receiver or protocol change. On a successful parse from a text body, the matching rule persists the log as a JSON body with content type JSON; the normalization is scoped to that rule\u0026rsquo;s context — other rules analyzing the same log still see the original text body. Surface the drop reason in LAL live-debugging. When a LAL rule stops a log at a parse step (a json {} / yaml {} parse failure, a text { regexp } non-match, or a non-log-body input), the recorder now captures a human-readable reason (e.g. the parse exception) onto the DSL-debug Sample, exposed through the dsl-debugging REST session response and the cluster forward proto. Previously a live-debug watcher could only see continueOn=false — that a step stopped, never why — and had to read the OAP server log. Sample.reason is shared across all DSL debuggers but populated by LAL today. Fix a v2 MAL CounterWindow key collision: rate() / increase() / irate() keyed each counter\u0026rsquo;s sliding window on the rule\u0026rsquo;s output metric name (the same for every input metric of a rule) instead of the counter\u0026rsquo;s own name, so two or more counters that reduce to the same label set after .sum(...) shared one window and computed rates against each other\u0026rsquo;s values — fabricating non-zero rates from unchanged counters (e.g. the BanyanDB liaison gRPC error rate read a steady non-zero off three frozen error counters). The window is now keyed by the counter\u0026rsquo;s own metric name. Fix the v2 MAL Elvis operator ?: to honor Groovy-falsy semantics. It compiled to Optional.ofNullable(primary).orElse(fallback), applying the fallback only when the primary is null, so an empty-string primary kept \u0026quot;\u0026quot; instead — e.g. a BanyanDB liaison ServiceInstance stored node_type=\u0026quot;\u0026quot; rather than n/a, because .sum([...,'node_type']) fills an absent group-by label with \u0026quot;\u0026quot;. The fallback now applies for falsy primaries such as null, false, numeric zero, and empty strings/containers. SWIP-15: rebuild BanyanDB self-observability around the cluster / container / group model (requires BanyanDB 0.11+). A BanyanDB cluster is modeled as one Service, each container as a ServiceInstance (role/tier as attributes), and each storage group as an Endpoint. The otel-rules/banyandb/ rules are category-separated by role (node_* / liaison_* / data_* / lifecycle_*) and by data type (measure_* / stream_* / trace_* / property_*), mirroring the upstream FODC-proxy Grafana boards, and include queue batch/message granularity (apache/skywalking-banyandb#1169). Adds a SERVICE_INSTANCE_RELATION MAL scope and serviceInstanceRelation(...) builder powering a new intra-cluster pod-to-pod deployment topology (banyandb-instance-relation.yaml). The stale single-node host_name model is removed. Runtime MAL/LAL hot-update rules can declare layerDefinitions: to introduce new layers. Ordinals are operator-pinned in the 100_000+ tier; the layer is refcount-tracked and unregistered when the last declaring rule is removed. See runtime-rule-hot-update.md#dynamic-layers for the conflict rules and limitations. Fix: runtime-rule (MAL/LAL hot-update) schema changes now work in no-init mode — the deployment mode every production cluster runs. Previously a runtime addOrUpdate that introduced a new metric blocked forever in the storage installer\u0026rsquo;s init-node poll loop (ModelInstaller.whenCreating) on a no-init OAP, because the gate keyed off RunningMode rather than the operation\u0026rsquo;s intent; the /delete?mode=revertToBundled recreate and BanyanDB in-place shape updates were dead the same way. The poll loop is now gated on a new StorageManipulationOpt.Flags.deferDDLToInitNode bit set only on the static boot-time schemaCreateIfAbsent() opt (DRYed into ModelInstaller.deferDDLToInitNode(opt) and reused by the BanyanDB shape-check / group-DDL gates), so the runtime-rule opts (withSchemaChange / verifySchemaOnly / withoutSchemaChange) are driven by their flags and by cluster main-ness — no-init and default no longer differ for DSL DDL; init mode stays the dedicated initializer. DSLManager.tickStorageOpt is collapsed accordingly (main → withSchemaChange, peer → verifySchemaOnly at boot / withoutSchemaChange on tick). Fix: runtime-rule cross-node writes no longer fail with HTTP 400 forward_self_loop on a multi-replica Kubernetes cluster. Every OAP replica shared the cluster selfNodeId 0.0.0.0_11800 (derived from the 0.0.0.0 agent gRPC bind host via TelemetryRelatedContext), so the main\u0026rsquo;s self-loop guard rejected a legitimate peer-to-peer Forward as if it had looped back. The runtime-rule node identity now prefers the unique per-pod SKYWALKING_COLLECTOR_UID (the pod UID injected by the helm chart / swck operator from metadata.uid), resolved in start() before any apply, and falls back to the telemetry id for non-k8s deployments. Adds a kind-based no-init cluster e2e (test/e2e-v2/cases/runtime-rule/cluster, deployed via skywalking-helm with oap.replicas=2) that drives the apply / STRUCTURAL / inactivate / delete lifecycle and the cross-node Forward path, replacing the prior docker-compose default-mode cluster case. Fix: remove the redundant tags from the envoy-ai-gateway.yaml LAL configuration. Add Zipkin Virtual GenAI e2e test. Use zipkin_json exporter to avoid protobuf dependency conflict between opentelemetry-exporter-zipkin-proto-http (protobuf~=3.12) and opentelemetry-proto (protobuf\u0026gt;=5.0). Fix missing taskId filter and incorrect IN clause parameter binding in JDBCJFRDataQueryDAO and JDBCPprofDataQueryDAO. Remove deprecated GroupBy.field_name from BanyanDB MeasureQuery request building (Phase 1 of staged removal across repos). Push taskId filter down to the storage layer in IAsyncProfilerTaskLogQueryDAO, removing in-memory filtering from AsyncProfilerQueryService. Fix missing parentheses around OR conditions in JDBCZipkinQueryDAO.getTraces(), which caused the table filter to be bypassed for all but the first trace ID. Replaced with a proper IN clause. Fix missing and keyword in JDBCEBPFProfilingTaskDAO.getTaskRecord() SQL query, which caused a syntax error on every invocation. Fix storage layer bugs in profiling DAOs and add unit test coverage for JDBC query DAOs. Bug fixes: duplicate TABLE_COLUMN condition in JDBCMetadataQueryDAO.findEndpoint(), wrong merged table check in JFRDataQueryEsDAO (used incorrect INDEX_NAME due to copy-paste), and missing isMergedTable check in ProfileTaskQueryEsDAO.getById(). Test additions: add unit tests for 21 JDBC query DAOs verifying SQL/WHERE clause construction. Optimize TraceQueryService.sortSpans from O(N^2) to O(N) by pre-indexing spans by segmentSpanId, so trace detail queries scale linearly with span count. Support MCP (Model Context Protocol) observability for Envoy AI Gateway: MCP metrics (request CPM/latency, method breakdown, backend breakdown, initialization latency, capabilities), MCP access log sampling (errors only), ai_route_type searchable log tag, and MCP dashboard tabs. Add weighted handler support to BatchQueue adaptive partitioning. MAL metrics use weight 0.05 at L1 (vs 1.0 for OAL), reducing partition count and memory overhead when many MAL metric types are registered. Fix missing taskId filter in pprof task log query and its JDBC/BanyanDB/Elasticsearch implementations. Fix duplicate calls in EndpointTopologyBuilder — calls were not deduplicated unlike ServiceTopologyBuilder, causing duplicate entries when storage returns multiple records for the same relation. Use containsOnce and noDuplicates for topology dependency e2e expected files to enforce no-duplicate verification. Bump infra-e2e to ef073ad to include noDuplicates pipe function support. PromQL: support querying Zipkin metadata (service name, remote service name, span name). TraceQL: support more tags and variables in Grafana for querying. LAL: add sourceAttribute() function for non-persistent OTLP resource attribute access in LAL scripts. LAL: add layer: auto mode for dynamic layer assignment when service.layer is absent. Add two-phase SpanListener SPI mechanism for extensible trace span processing. Refactor GenAI from hardcoded SpanForward.processGenAILogic() to GenAISpanListener. Add OTLP/HTTP receiver support for traces, logs, and metrics (/v1/traces, /v1/logs, /v1/metrics). Supports both application/x-protobuf and application/json content types. Fix: TTL query add metadata TTL. Fix: PersistentWorker used wrong TTL for metrics cache if the storage is BanyanDB. Add iOS/iPadOS app monitoring via OpenTelemetry Swift SDK (SWIP-11). Includes the IOS layer, IOSHTTPSpanListener for outbound HTTP client metrics (supports OTel Swift .old/.stable/.httpDup semantic-convention modes via stable-then-legacy attribute fallback), IOSMetricKitSpanListener for daily MetricKit metrics (exit counts split by foreground/background, app-launch / hang-time percentile histograms with finite 30 s overflow ceiling), LAL rules for crash/hang diagnostics, Mobile menu, and iOS dashboards. Add Apache Airflow monitoring via native OpenTelemetry metrics (SWIP-7). New AIRFLOW layer with Service (cluster) and Instance (host) dimensions, MAL rules under otel-rules/airflow/ (27 metrics), setup documentation, mock OTLP e2e (cases/airflow/mock/e2e.yaml: 2 entity + 27 metric checks, 29 total), and real Celery-cluster integration smoke (cases/airflow/cluster/e2e.yaml: 2 entity + 14 metric checks, 16 total). See test/e2e-v2/cases/airflow/README.md. Horizon UI dashboards ship separately in apache/skywalking-horizon-ui under the Workflow Scheduler menu group. Fix LAL layer: auto mode dropping logs after extractor set the layer. Codegen now propagates layer \u0026quot;...\u0026quot; assignments to LogMetadata.layer so FilterSpec.doSink() sees the script-decided layer. Fix MetricKit histogram percentile metrics being reported at 1000× their true value — the listener now marks its SampleFamily with defaultHistogramBucketUnit(MILLISECONDS) so MAL\u0026rsquo;s default SECONDS→MS rescale of le labels is not applied. Add WeChat and Alipay Mini Program monitoring via the SkyAPM mini-program-monitor SDK (SWIP-12). Two new layers (WECHAT_MINI_PROGRAM, ALIPAY_MINI_PROGRAM); two new JavaScript componentIds (WeChat-MiniProgram: 10002, AliPay-MiniProgram: 10003). Service / instance / endpoint entities are produced by MAL + LAL, not trace analysis — mini-programs are client-side (exit-only) so RPCAnalysisListener stays unchanged (same pattern as browser and iOS). MAL rules per platform × scope under otel-rules/miniprogram/ with explicit .service(...) / .endpoint(...) chains (empty expSuffix so endpoint-scope rules aren\u0026rsquo;t overridden), histogram percentile via .histogram(\u0026quot;le\u0026quot;, TimeUnit.MILLISECONDS) to keep ms bucket bounds intact, and request-cpm derived from the histogram _count family. LAL layer: auto rule produces both layers via miniprogram.platform dispatch and emits error-count samples consumed by per-platform log-MAL rules. Per-layer menu entries and service / instance / endpoint dashboards with Trace and Log sub-tabs. Fix: remove VirtualServiceAnalysisListener\u0026rsquo;s dependency on GenAIAnalyzerModule if it is disabled. MAL: register TimeUnit in MALCodegenHelper.ENUM_FQCN so rule YAML can write .histogram(\u0026quot;le\u0026quot;, TimeUnit.MILLISECONDS) for SDKs that emit histogram bucket bounds in ms (default SECONDS unit applies a ×1000 rescale that would otherwise inflate stored le labels 1000×). Fix: potential unexpected current directory inclusion in Docker OAP classpath. MAL: add safeDiv(divisor) on SampleFamily that yields 0 when the divisor is 0 instead of Infinity/NaN. Replace / with safeDiv(...) in Envoy AI Gateway latency-average rules so sum / count * 1000 no longer produces dropped or out-of-range samples when a counter is zero in a window. Fix: envoy-ai-gateway metrics rules, make the metrics value return 0 when the divisor is 0. Custom Layers can be declared without modifying the OAP source — via an operator-managed layer-extensions.yml, inline layerDefinitions: block in a MAL or LAL rule file, or a plugin extension. UI dashboard templates for new layers are auto-discovered from the ui-initialized-templates/ directory. Recommended ordinal range for external layers is \u0026gt;= 1000; conflicting names or ordinals are reported at boot. LAL: support full arithmetic (+, -, *, /) on numeric operands and fix the original bug where (tag(\u0026quot;x\u0026quot;) as Integer) + (tag(\u0026quot;y\u0026quot;) as Integer) was treated as string concatenation — expressions like input_tokens + output_tokens \u0026lt; 10000 produced the concatenated string \u0026quot;2589115\u0026quot; rather than the integer sum 2704, so token-threshold conditions never triggered abort {}. Operand types are now inferred from explicit casts (as Integer / as Long / as Float / as Double), typed proto fields, or numeric literal shape (with L / F / D suffix support, e.g. 1000L). The compiler honours JLS-style binary numeric promotion and emits Java arithmetic in the declared primitive type — (x as Integer) + (y as Integer) compiles to int + int (not widened to long). + with any String operand falls back to string concatenation; - / * / / against non-numeric operands produces a compile-time error. The as Double and as Float casts are accepted in typeCast clauses, including in def declarations. Numeric comparisons honour declared casts on both sides (no more universal h.toLong() wrapper). Fix: avgHistogramPercentile / sumHistogramPercentile meter functions reported the smallest finite bucket boundary (e.g. 10 for OTel gen_ai_server_request_duration whose le is rewritten from 0.01s → 10ms) for every rank when no samples were observed in any bucket. The percentile loop\u0026rsquo;s count \u0026gt;= roof check matched on the first sorted bucket because both sides were 0. calculate() now short-circuits to 0 for every rank when the windowed total is 0. Fix: MAL expPrefix now applies to every metric source in exp, not just the leading one. Previously the prefix was spliced after the first ., so secondary metrics inside arguments (e.g. the divisor in a.sum(['s']).safeDiv(b.sum(['s']))) silently skipped the prefix — a rule like envoy-ai-gateway\u0026rsquo;s request_latency_avg (sum / count) would tag-rewrite only the dividend. The injection is now AST-aware: every bare-IDENTIFIER metric source is wrapped, while downsampling-type constants (SUM, AVG, LATEST, SUM_PER_MIN, MAX, MIN) are skipped. Add @Stream(allowBootReshape = true) opt-in for additive boot-time reshape of BanyanDB streams / measures. Code-defined stream classes (e.g. AlarmRecord) can now annotate their schema as eligible for in-place additive update at OAP boot — a new @Column is appended to the live tag-family / fields via client.update instead of being silently rejected with SKIPPED_SHAPE_MISMATCH (which previously forced operators to drop the measure / stream and lose historical rows). Additive includes both new tags / fields and relocating an existing tag between families when a @Column\u0026rsquo;s storageOnly flag flips (e.g. id1 moving from storage-only → searchable when it becomes indexed). The opt-in is per-stream and gated by an isPurelyAdditive shape diff: tag type changes, tag drops, kind flips (tag↔field), entity / interval / sharding-key changes, and field re-typing still skip with SKIPPED_SHAPE_MISMATCH, so identity-breaking edits remain explicit operator actions. Only the init / standalone OAP performs the reshape; non-init peers continue through the existing poll-and-wait loop so a single node drives DDL. When a check* records SKIPPED_SHAPE_MISMATCH the dependent IndexRule / IndexRuleBinding reconciliation is also skipped — preventing the previous gap where the binding silently updated to a tag list that diverged from the live tag-family layout. AlarmRecord is opted in. Default remains false for all other models — boot-time reshape stays off unless the annotation is explicitly set. Operator caveat: BanyanDB does not physically migrate existing rows when a tag\u0026rsquo;s family changes; pre-existing data stays in its original on-disk location while new writes go to the declared family — expect a backfill window for queries that route through new IndexRules on relocated tags. Mask keywords trustStorePass, keyStorePass by default. Bump up dependencies to clear CVE alerts on shipped OAP jars: log4j 2.25.3 → 2.25.4, jackson 2.18.5 → 2.18.6, kafka-clients 3.4.0 → 3.9.2, postgresql 42.4.4 → 42.7.11, commons-compress 1.21 → 1.26.2. Bump up more dependencies to clear CVE alerts on shipped OAP jars: netty 4.2.12.Final → 4.2.15.Final, jackson 2.18.6 → 2.18.8, commons-codec 1.11 → 1.13. Also realign jackson-databind 2.16.0 → 2.18.8 so the whole jackson family is managed at a single version (it had been left behind the other jackson artifacts). Bump Apache Curator 4.3.0 → 5.9.0 and Apache ZooKeeper 3.5.7 → 3.9.5 together to clear CVE-2023-44981 (the bundled ZooKeeper jar carried it; OAP is a ZooKeeper client only, so the server-side bug was never reachable, but the jar tripped Dependabot). The cluster-zookeeper and configuration-zookeeper plugins use only stable Curator APIs, so no source changes were required. Operator-facing change: the supported ZooKeeper server version is now 3.6+ (Curator 5.x uses ZooKeeper persistent watches, added in server 3.6.0); older servers (3.5.x, 3.4.x) are no longer supported. Migrate the Consul cluster and configuration client from the abandoned com.orbitz.consul:consul-client 1.5.3 to the maintained fork org.kiwiproject:consul-client 0.9.0 to clear the okhttp CVE the old client carried (CVE-2021-0341; the old client pinned okhttp 3.14.9, fixed in okhttp 4.9.2+), so the BOM now pins okhttp to 4.12.0. The fork\u0026rsquo;s 0.9.x line is the last one built for JDK 11 (which SkyWalking still targets); 1.0.0+ is compiled to JDK 17 bytecode, so the migration stays on 0.9.0. The cluster-consul and configuration-consul plugins use only stable Consul client APIs, so the change is a package rename (com.orbitz.consul → org.kiwiproject.consul); okhttp is pulled only by the Consul plugins (the fabric8 Kubernetes client excludes its okhttp transport), so no other module is affected. Bump test-scope assertj-core 3.20.2 → 3.27.7 to clear CVE-2026-24400 (XXE in isXmlEqualTo, not used by any test). Clear three security alerts: bump the Airflow e2e mock\u0026rsquo;s pinned protobuf 4.25.8 → 5.29.6 (with opentelemetry-proto 1.24.0 → 1.28.0, whose protobuf\u0026lt;5.0 cap was the blocker, and grpcio 1.62.2 → 1.63.2, required because opentelemetry-proto 1.28.0\u0026rsquo;s gRPC stubs call unary_unary(_registered_method=...)) to clear CVE-2026-0994 — a CI-only test fixture, never shipped; and widen the cumulative count accumulator from int to long in SumHistogramPercentileFunction / AvgHistogramPercentileFunction to clear the CodeQL implicit-cast-in-compound-assignment alerts (count += value silently narrowed a long bucket-count sum back to int, while total was already long). Clear Dependabot CVE alerts in the e2e Go test fixtures (cases/go/service and cases/profiling/ebpf/network, CI-only, never shipped in any OAP artifact): bump golang.org/x/net 0.48.0 → 0.55.0 (CVE-2026-25681, CVE-2026-27136, CVE-2026-33814, CVE-2026-39821) and move the Go toolchain from 1.24 to 1.26.5 (CVE-2026-27145 / CVE-2026-42504 fixed in 1.26.4, CVE-2026-39822 fixed in 1.26.5) by switching the shared skywalking-go base image to the -go1.26 variant and bumping SW_AGENT_GO_COMMIT to 7544822, whose -go1.26 image ships go1.26.5. Fix: continuous profiling policy validation now rejects a threshold / count of 0 to match the error messages and rover\u0026rsquo;s value \u0026gt;= threshold trigger semantics (a 0 threshold would always trigger). CPU percent and HTTP error rate are tightened from [0-100] to (0-100]. Fix wrong BanyanDB resource options in record data. Align the default BanyanDB stage segmentInterval values so each coarser stage is an integer multiple of the finer one (records cold 3 → 4, metricsMinute cold 5 → 6, metricsHour warm 7 → 10 and cold 15 → 20), keeping hot → warm → cold lifecycle migration on the cheap whole-segment fast path. Fix: layer-extensions.yml is now excluded from the skywalking-oap jar and shipped to the distribution config/ directory, so an operator-edited config/layer-extensions.yml is no longer shadowed by the empty template bundled in the jar. Because the OAP launch script puts oap-libs/*.jar ahead of config/ on the classpath, ResourceUtils.read(\u0026quot;layer-extensions.yml\u0026quot;) previously always resolved the jar-bundled layers: [] and silently ignored the operator\u0026rsquo;s file — custom layers declared there never registered. The file now follows the same exclude-from-jar + copy-to-config/ packaging as every other operator-editable config (application.yml, alarm-settings.yml, etc.). Fix: the v2 MAL compiler now resolves custom layers referenced as Layer.NAME in an expression. A custom layer declared through a layerDefinitions: block (or layer-extensions.yml / the LayerExtension SPI) has no generated Layer.* static field, so service(['svc'], Layer.IOT_FLEET) previously failed code generation because Layer has no IOT_FLEET field. The compiler now lowers every Layer.NAME static-field reference to a runtime Layer.nameOf(\u0026quot;NAME\u0026quot;) registry lookup, so a custom layer can be referenced exactly like a built-in one (Layer.GENERAL). For a built-in layer this is equivalent, because Layer.nameOf(\u0026quot;GENERAL\u0026quot;) returns the same instance as the Layer.GENERAL field. The lowering is scoped to Layer only; the other MAL enum types (DetectPoint, DownsamplingType, etc.) are real Java enums and keep their direct static-field reference. Fix Envoy ALS rendering for the LAL live-debugger and the persisted log content: an Istio metadata-exchange peer in common_properties.filter_state_objects (legacy Wasm wasm.*_peer = Any{BytesValue} wrapping a FlatBuffer, or modern *_peer = Any{Struct}) is now decoded into the readable peer metadata (pod / namespace / labels) instead of an opaque jsonformat-failed envelope or base64. The serialization is hardened so a single un-printable field can no longer blank the whole entry — the LalPayloadDebugDump printer carries a well-known-type TypeRegistry and sanitizes every value JsonFormat would reject (an unresolvable, no-slash, or corrupt-bytes Any degrades to an @unresolved placeholder; a non-finite Value double NaN/Infinity is rendered as a string), keeping the rest of the entry readable. Because the LAL output builder\u0026rsquo;s bindInput runs eagerly before the debug capture, this also stops an unregistered filter_state_objects type from throwing and aborting the whole rule (dropping the mesh log). Decoding is wired through a new LalInputDebugRenderer SPI (EnvoyAlsHttpDebugRenderer / EnvoyAlsTcpDebugRenderer) so log-analyzer reaches the receiver-side decoders without depending on the Envoy receiver, and covers both HTTP and TCP access logs. Surface the effective BanyanDB configuration (bydb.yml / bydb-topn.yml) in the /debugging/config/dump admin API. Because the BanyanDB config moved to a separate file in 10.2.0, a BanyanDB deployment previously showed an empty storage.banyandb block in the dump; its post-environment-resolution values are now merged into the same response under storage.banyandb.* (TopN rules under storage.banyandb.topN.*), masked by the same secret-keyword list, via a generic ConfigDumpExtension SPI on ServerStatusService that any module loading config from a secondary file can implement. Fix: an MQE top_n(metric, N, order, attrX='value') query whose attribute is not a column of the target metric now returns a descriptive MQE error instead of a raw storage IOException surfaced as Internal IO exception, query metrics error.. Attribute columns (attr0..attrN) exist only on decorated metrics (service_* / endpoint_* / kubernetes_service_*, set to the layer name via OAL .decorator(...)) and the MAL meter base; metrics such as relations or database / cache / mq access carry none, so passing an attribute condition previously reached the storage engine with a tag it does not define and failed there. MQEVisitor now validates each attribute key against the metric\u0026rsquo;s registered queryable columns before the storage call and raises IllegalExpressionException (naming the attribute and the metric) when it is absent. Migrate all BanyanDB storage read queries from the typed query-builder API to BydbQL. Fix: BanyanDB queries no longer silently truncate at the storage engine\u0026rsquo;s implicit row cap. BanyanDB applies its own default limit to any query that carries none — 100 rows for measures, 20 for streams/traces — and applies it after GROUP BY, so an over-long result set is cut short rather than rejected. OAP never sent a limit on several read paths, so a metrics query returned at most 100 data points regardless of the requested range: a 4-hour minute-step read rendered only its first 100 minutes and the rest showed as empty, even though DurationUtils allows up to 500 steps. The same cap silently shortened topology relation maps, instance/process metadata lists, profiling thread snapshots and eBPF task lists. Every BydbQL query now leaves OAP with an explicit LIMIT: the entity-scoped metrics read sends the exact number of assembled duration points (matching the row set the ES/JDBC DAOs fetch by id), ad-hoc SELECT TOP sends its own N, and anything that does not paginate itself falls back to the configured resultWindowMaxSize (default 10000) instead of the engine default. ES and JDBC storage were never affected. Support BanyanDB\u0026rsquo;s group-scoped trace retention pipeline in bydb.yml. The trace and zipkinTrace groups gain a pipeline block (enabled, enabledEvents, mergeGraceSeconds, finalizeGraceSeconds, and an ordered plugins chain) that OAP pushes onto the BanyanDB group as a TracePipelineConfig, letting a sampler plugin drop traces inside the data node during Hot-phase compaction — after storage, so it reclaims space already written and decides per whole trace, unlike the ingest-side server-side trace sampling. Disabled by default, since it deletes stored traces. Even when enabled it is inert unless the data node runs the plugin-capable BanyanDB image with the sampler .so mounted — a node without that support ignores the config, and one that cannot load the plugin logs an error and merges unfiltered, so nothing is dropped unexpectedly. The two grace windows use -1 for \u0026ldquo;inherit the data node default\u0026rdquo; (30s merge / 5m finalize) because the node treats any non-positive grace as unset. enabledEvents accepts a comma-separated string so it can be set from the environment (SW_STORAGE_BANYANDB_TRACE_PIPELINE_ENABLED_EVENTS and its ZIPKIN_ variant) as well as a YAML block list; an empty value falls back to PIPELINE_EVENT_MERGE, so PIPELINE_EVENT_FINALIZE runs only when named explicitly. Each plugin\u0026rsquo;s config is passed through verbatim as a protobuf Struct: nested lists and objects (e.g. keepTagRules) now survive the config loader and are serialized as real ListValue/Struct rather than being flattened to a string. Note a float written as a ${ENV:default} placeholder still reaches the plugin as a JSON string, because the shared placeholder resolver only preserves String/Integer/Long/Boolean; the first-party samplers accept a quoted number for exactly this reason. See Trace Tail Sampling for how a trace is judged, and BanyanDB storage for the configuration keys. Fix: a blank value in bydb.yml (key: with nothing after it) aborted OAP startup with an opaque NullPointerException from java.util.Properties, which rejects null values. The BanyanDB config loader now skips blank entries and leaves the field at its default, the same outcome as omitting the line. Route LAL rules within a layer by their input type, so a single layer can host rules over different proto inputs. Each compiled rule now carries its effective input type (the proto class its parsed.* getters cast to, or null for parser-based / untyped rules), and LogFilterListener skips any rule whose type doesn\u0026rsquo;t match the incoming log instead of running every rule in the layer. This fixes a latent ClassCastException (caught and logged per log) that fired whenever a MESH log of one shape reached a rule compiled for another — e.g. an Envoy TCP access log or a network-profiling LogData hitting the HTTP envoy-als rule. Adds an envoy-als-tcp rule (inputType: TCPAccessLogEntry) alongside the existing HTTP envoy-als; both share the MESH layer and each now only sees its own entry type. Fix the PagerDuty alarm hook to default its Events API v2 endpoint to https://events.pagerduty.com/v2/enqueue. Fix HttpAlarmCallback logging a successful alarm delivery as a failure. The shared HTTP hook helper treated only 200 and 204 as success, so any other 2xx — notably the 202 Accepted returned by asynchronous intake APIs such as PagerDuty\u0026rsquo;s Events API v2 — produced send to ... failure. Response code: 202 at ERROR level on every delivered alarm. The alarm was still delivered; the log entry was wrong. The check now accepts the whole 2xx range, for all alarm hooks. Make the PagerDuty Events API v2 endpoint configurable through a new optional events-api-url setting on each pagerduty hook, defaulting to the US service region endpoint. An account in PagerDuty\u0026rsquo;s EU service region can now point the hook straight at https://events.eu.pagerduty.com/v2/enqueue rather than relying on PagerDuty forwarding the request — and the routing key and alarm payload from an EU-region account no longer transit the US region. Bump the default BanyanDB compatible server API version (SW_STORAGE_BANYANDB_COMPATIBLE_SERVER_API_VERSIONS) from 0.10 to 0.11. UI Add Airflow layer dashboards and menu i18n under Workflow Scheduler in Horizon UI (SWIP-7). Add mobile menu icon and i18n labels for the iOS layer. Fix metric label rendering in multi-expression dashboard widgets. Add i18n menu labels for WeChat Mini Program and Alipay Mini Program (en / zh / es) — sub-menus rendered as raw keys until this bump. Support trace V1 view in trace single page. Documentation Document the meter-analyzer-config catalog in the runtime-rule hot-update and DSL-debugging references, and add the optional layerDefinitions block, the active-files startup-failure behaviour, and a hot-update / debugging section to the meter setup doc. Update LAL documentation with sourceAttribute() function and layer: auto mode. Add Airflow monitoring setup documentation (SWIP-7). Add iOS app monitoring setup documentation. Add WeChat / Alipay Mini Program monitoring setup documentation, plus a client-side-monitoring section in the security guide covering public-internet ingress (OTLP + /v3/segments) for mobile / browser / mini-program SDKs. Improve downsampling documentation Fix the docker-compose quickstart: OAP healthcheck no longer calls curl (absent from the JRE image) and probes the query port via bash /dev/tcp; the Horizon UI service maps the correct container port (8081) and mounts a horizon.yaml (binding 0.0.0.0, OAP URLs, demo admin/admin login) instead of non-existent SW_*_ADDRESS env vars. Add PHP runtime metrics (PHM) dashboard documentation (agent setup, OAP php-runtime MAL rules, Horizon UI widgets). Add Node.js runtime metrics dashboard documentation (agent setup, OAP nodejs-runtime MAL rules, Horizon UI widgets). Add a BanyanDB trace tail sampling guide under \u0026ldquo;BanyanDB Exclusive Setup\u0026rdquo;, covering how a trace is judged (the OR-ed rule chain, the end-to-end duration envelope rather than a per-span maximum, and the deterministic trace-ID hash behind healthySampleRate), what the two first-party samplers read from each trace schema, the MERGE vs FINALIZE events and their grace windows, the fail-open behaviour when a plugin is absent or unloadable, and the metrics to watch. Also document the Zipkin receiver\u0026rsquo;s previously undocumented sampleRate and maxSpansPerSecond in the server-side trace sampling guide. Correct the APISIX monitoring guide to align its Collector configuration and metric names with the current APISIX MAL rules and Horizon UI Dashboard. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1100\"\u003e11.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eMove the DSL class-loading machinery under \u003ccode\u003ecore/dsl\u003c/code\u003e. \u003ccode\u003ecore/classloader\u003c/code\u003e held only DSL …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/next/en/changes/changes-11.0.0/","title":"11.0.0"},{"body":"11.0.0 Project Move the DSL class-loading machinery under core/dsl. core/classloader held only DSL types — RuleClassLoader, DSLClassLoaderManager, ClassLoaderGc, UnloadProbePayload and BytecodeClassDefiner — so it is now core/dsl/classloader, and Catalog moves to core/dsl because a rule-file taxonomy is not a class-loading concern. Three copies of the \u0026ldquo;define a generated class into the right loader\u0026rdquo; dispatch (MAL, LAL, MeterSystem) collapse into a static BytecodeClassDefiner.define, which also gives the JDK 17 --add-opens rationale one home instead of four. Three of the four copies of the generated-class dump-directory lookup become DslGeneratedFileWriter.resolveClassDumpDir; OAL keeps its own, because its debug flag is settable independently of the environment variable and two tests rely on that. No behaviour change. Extend the GET /inspect/entities admin API to inspect a metric persisted by any OAP, even one this node does not define locally. When the metric is unknown to the local registry, the caller supplies valueColumn + valueType and the storage backend resolves the physical index/table/group from its own running config (no DB schema/table-metadata read): ES uses the merged metrics-all index + metric_table discriminator, JDBC probes the node\u0026rsquo;s function tables by the table_name discriminator, and BanyanDB synthesizes a read-only measure schema. Scope is no longer required — the entity_id is decoded structurally (service / 2nd-level / relations) with a generic name leaf. Locally-defined metrics keep the exact field names, scope, and mqeEntity as before. Add the POST /inspect/values admin API — read the value series of a metric persisted by another OAP (one this node does not define locally) by supplying its {valueColumn, valueType}. The real MQE engine runs over a request-scoped InspectQueryContext overlay (provide-if-absent — the local catalog always wins) that makes the foreign metric look registered to every read path: ValueColumnMetadata resolves its value column / type / scope, and the storage location registries resolve where it lives (MetadataRegistry synthesizes a BanyanDB measure schema, IndexController resolves the ES metrics-all index, TableHelper probes the JDBC function tables), so the read returns the native MQE ExpressionResult with no per-DAO special-casing. Admin-only (a forced read this OAP cannot validate); not mirrored onto the public REST / GraphQL surface. See the Inspect API. Remove the always-on alarm-to-event conversion (EventHookCallback). A triggered alarm is no longer synthesized into the events pipeline as an Alarm/AlarmRecovery event; events now originate only from real event sources (agents, SkyWalking CLI, Kubernetes Event Exporter). Alarms remain available through the alarm store (getAlarm/queryAlarms) and the configured alarm hooks. This drops a documented \u0026ldquo;Known Event\u0026rdquo; and removes 1-2 synthetic event records per alarm fire. TLS for all OAP HTTP/REST servers, with cert hot-reload. Adds the restSSLEnabled / restSSLKeyPath / restSSLCertChainPath config structure to every OAP HTTP server — core REST, sharing-server, admin, PromQL, LogQL, TraceQL and Zipkin query/receiver — each with its own dedicated environment variables (SW_CORE_REST_SSL_*, SW_RECEIVER_SHARING_REST_SSL_*, SW_ADMIN_SERVER_REST_SSL_*, SW_PROMQL_REST_SSL_*, SW_LOGQL_REST_SSL_*, SW_TRACEQL_REST_SSL_*, SW_QUERY_ZIPKIN_REST_SSL_*, SW_RECEIVER_ZIPKIN_REST_SSL_*). The shared Armeria HTTPServer reloads the key pair from disk on rotation (via TlsProvider.ofScheduled) so refreshed certificates are picked up without restarting the OAP, matching the existing gRPC SSL hot-reload behavior. HTTP TLS is server-side only (no mTLS). New queryAlarms GraphQL query — entity / layer / rule filters for alarms. Adds a comprehensive alarm query API alongside the legacy getAlarm. The new queryAlarms(condition: AlarmQueryCondition!): Alarms accepts a single input type bundling every filter the alarm record stores: entities: [Entity!] (reuses the MQE Entity shape — pin to specific services / instances / endpoints / processes or their relations, matched against alarm id0 OR id1); layer: String (filter by the alarmed entity\u0026rsquo;s layer — single match, since alarm rows persist one layer); ruleNames: [String!] (filter by which alarm rule fired); plus keyword, tags, duration, paging. Legacy getAlarm is marked @deprecated but still routes to the same DAO — no client breakage. Backend additions: a new layer column on AlarmRecord populated at alarm-mint time via MetadataQueryService.getService(serviceId).getLayers(); the existing id0/id1 columns flipped from storageOnly = true to indexed so the entity filter pushes down to storage. IAlarmQueryDAO.queryAlarms(condition, limit, from) is a new abstract method — 3rd-party storage backends fail at compile if they miss the override (SWIP-14 pattern). All three bundled backends implement it: BanyanDB / Elasticsearch / JDBC. Operator semantics: (1) Relation entities are exact-match. Passing {scope: ServiceRelation, serviceName: A, destServiceName: B} matches only the alarm where id0=serviceId(A) AND id1=serviceId(B), not any alarm that touches A or B on either side. Wider \u0026ldquo;anything involving A\u0026rdquo; queries should pass the individual non-relation entity instead ({scope: Service, serviceName: A} — which expands to id0=A OR id1=A). (2) Single layer per alarm row. The persisted column stores ONE layer (the first entry of the entity\u0026rsquo;s resolved layer list — source-first for relations). A service in [GENERAL, K8S_SERVICE] whose metadata resolves to GENERAL first is filed under GENERAL; querying layer: \u0026quot;K8S_SERVICE\u0026quot; will miss it. Operator migration note: existing pre-upgrade alarm rows continue to be filterable by the legacy getAlarm fields; the new entity / layer / rule filters in queryAlarms apply only to alarms written after the upgrade (existing storage indices don\u0026rsquo;t transition index: false → true in place; new daily-rolled indices pick up the indexed columns). Schema additions are non-blocking — bootstrap silently skips column-attribute changes on existing indices. 🚨 Breaking change: apm-webapp and the skywalking-booster-ui submodule are removed. This OAP distribution no longer ships a bundled web UI. The legacy Armeria reverse proxy in apm-webapp/ (the binary that powered the skywalking/ui Docker image) and the skywalking-ui git submodule (which tracked apache/skywalking-booster-ui) are both deleted along with the docker.ui Maven target, the skywalking/ui Docker image build, the apm-dist/ webapp packaging, and every CI workflow path that built or pushed the UI image. The official UI is now Horizon UI, a SkyWalking sub-project that releases independently of the OAP backend on its own schedule, with released container images on Docker Hub at apache/skywalking-ui (tags latest / horizon-\u0026lt;version\u0026gt;; per-commit development images live on ghcr.io/apache/skywalking-horizon-ui). There is no 1:1 mapping between OAP versions and Horizon UI versions — operators pin the UI image tag in their deployment and upgrade the two on separate cadences. Horizon UI consumes the OAP\u0026rsquo;s public GraphQL/REST surface (default 12800) and the admin host (default 17128). The on-disk dashboard seed files in oap-server/server-starter/src/main/resources/ui-initialized-templates/ are deleted; UITemplateInitializer / UIMenuInitializer are removed from CoreModuleProvider.notifyAfterCompleted(), and Horizon UI ships its own dashboard library and its own sidebar menu. UI templates are now created and updated through the new /ui-management/templates/* REST surface on admin-server (see below). All UI-related GraphQL mutations and queries (UIConfigurationManagement: addTemplate, changeTemplate, disableTemplate, getAllTemplates, getDashboardConfiguration, getMenuItems) are retired from the public GraphQL schema, along with the SW_ENABLE_UPDATE_UI_TEMPLATE flag. The OAP backend also no longer stores or serves the sidebar menu — UIMenuManagementService, UIMenuManagementDAO, UIMenu, MenuItem, and the storage impls are all removed; Horizon UI owns the menu client-side and uses listServices(layer:...) for dynamic \u0026ldquo;layer has services\u0026rdquo; gating. Upgrade path: replace skywalking/ui:\u0026lt;tag\u0026gt; with the Horizon UI image apache/skywalking-ui:latest (or a horizon-\u0026lt;version\u0026gt; tag — pick a version per Horizon UI\u0026rsquo;s OAP-compatibility notes, OAP 11.0+ is supported) in your deployment, expose port 17128 from the OAP container, and migrate any scripts that called the legacy GraphQL UI mutations to the REST endpoints under UI Management API. All status / debug endpoints (/status/*, /debugging/*) also move to admin-only — the public REST dual-bind for status is retired in the same release. New ui-management admin module — REST surface for dashboard templates. Hosts five operations on admin-server (port 17128): GET /ui-management/templates, GET /ui-management/templates/{id}, POST /ui-management/templates, PUT /ui-management/templates, POST /ui-management/templates/{id}/disable. Forwards to the existing UITemplateManagementService (no storage DAO changes). Enabled by default (SW_UI_MANAGEMENT=default, on a default-on admin host). Replaces the retired GraphQL UIConfigurationManagement template resolver. The sidebar menu is intentionally NOT served — see the breaking-change entry above. Operator reference: UI Management API. All admin feature modules default-on. admin-server, status, inspect, ui-management, dsl-debugging, and receiver-runtime-rule all default to enabled. Operators who don\u0026rsquo;t want a particular feature set its SW_* env var to empty. This closes a usability gap from 10.4.0 where the runtime-rule / dsl-debugging surfaces required explicit opt-in even though the admin host was already on. Status API moved to admin-host. Status / debug routes (/status/*, /debugging/*) now register on the admin-server REST host (default 17128); they no longer mirror on core.restPort (default 12800). Aligns status with every other admin feature module (inspect, dsl-debugging, runtime-rule, ui-management). Horizon UI consumes status from the admin host. URIs and payloads are unchanged; only the host moved. One exception: /status/config/ttl is also bound on the public REST host (12800) so ecosystem tools that discover TTL bounds via REST before issuing /graphql don\u0026rsquo;t need to learn the admin port. New admin-server module — shared host for admin / on-demand write APIs. Runs on two ports: an HTTP REST surface (default 17128) for operator-facing endpoints, and an admin-internal gRPC bus (default 17129) for peer-to-peer cluster RPCs (runtime-rule Suspend / Resume / Forward; DSL debug install / collect / stop / stopByClientId). The admin-internal bus is a dedicated transport separate from the public agent / cluster gRPC port (core.gRPCPort, default 11800) so privileged admin RPCs stay out of the agent network\u0026rsquo;s blast radius — operators bind gRPCHost to a private peer-to-peer interface only. Both the runtime-rule plugin and the new DSL Debug API (below) mount onto this shared host. Enabled by default so the status feature module is reachable out of the box; the host binds to 0.0.0.0:17128 and has no built-in authentication and must be gateway-protected with IP allow-lists, never exposed to the public internet (see the Admin API security notice). Set SW_ADMIN_SERVER= (empty) to disable entirely. The runtime-rule config block loses its restHost/restPort/restContextPath/restIdleTimeOut/ restAcceptQueueSize/httpMaxRequestHeaderSize keys (and the matching SW_RECEIVER_RUNTIME_RULE_REST_* env vars); host-level knobs move under the new admin-server block (SW_ADMIN_SERVER_HOST / SW_ADMIN_SERVER_PORT / SW_ADMIN_SERVER_GRPC_HOST / SW_ADMIN_SERVER_GRPC_PORT / SW_ADMIN_SERVER_INTERNAL_COMM_TIMEOUT etc.). Runtime rule hot-update for MAL and LAL. Operators can now ship metric (MAL) and log (LAL) rule changes without restarting OAP. A push to a new admin endpoint persists the rule to the configured storage backend, and every node in the cluster converges to the new content within ~30 seconds. Common workflows: addOrUpdate — create or replace a rule. Body is the raw YAML you would normally ship with OAP\u0026rsquo;s static rule files. Returns 200 once the rule is applied locally and persisted; peers pick it up on their next periodic scan (≤ 30 s). inactivate — soft-pause a rule. The OAP stops emitting metrics for that rule but the backend measure (and its history) is preserved, so a later addOrUpdate to the same (catalog, name) is lossless. The \u0026ldquo;off\u0026rdquo; intent is durable across reboots; bundled rules on disk are not auto-resurrected when an inactivate removes the runtime override. This is the safe way to take a rule offline. delete — removes an INACTIVE row (active rules return 409 requires_inactivate_first). For runtime-only rules with no bundled YAML on disk, the row is dropped; the backend measure (if any) is left in place as an inert artefact, matching bundled-rule deletion semantics (removing a YAML from otel-rules/ on disk doesn\u0026rsquo;t drop its measure either). For rules that have a bundled YAML twin, plain delete returns 409 requires_revert_to_bundled — letting bundled silently take over the (catalog, name) is a meaningful state change that requires an explicit operator decision. Re-issue with ?mode=revertToBundled to fall back to bundled: that path runs the schema-change pipeline (rehydrates the runtime DSL locally, then applies the bundled YAML through the standard apply pipeline so the runtime→bundled delta drops runtime-only metrics, registers bundled-only metrics, and reuses bundled-shared metrics at matching shape) before removing the row. Returns 400 no_bundled_twin when ?mode=revertToBundled is used without a bundled YAML on disk. get / bundled / list / dump — read-side endpoints for fetching a single rule\u0026rsquo;s YAML (with ETag support; ?source=bundled reads the on-disk bundled YAML even when a runtime override is in place), listing the bundled-vs-runtime overlay per catalog, inspecting cluster-wide rule state as a JSON envelope ({generatedAt, loaderStats, rules} — each row carries status/localState/loaderKind/bundled/bundledContentHash so a UI can render override badges without a second roundtrip), and exporting all rules as a tar.gz for backup / DR. Hot-updates survive OAP restart: at boot OAP merges bundled rule files with persisted runtime rules, so the cluster never silently regresses to the bundled defaults. All admin writes for a runtime-rule cluster serialize on a single \u0026ldquo;main\u0026rdquo; OAP (deterministic sorted-first peer, no leader election) — non-main nodes that receive an HTTP write transparently forward it to the main over the admin-internal gRPC bus, so an L7 load balancer in front of the admin port can route any operator request to any OAP. Cluster convergence on the periodic refresh tick is configurable via receiver-runtime-rule.refreshRulesPeriod (default 30 s). The endpoint is disabled by default and listens on port 17128 (HTTP) when enabled. It has no built-in authentication — operators must gateway-protect it with IP allow-lists and never expose it to the public internet. Routes mount on the new admin-server HTTP host, which is on by default; enable the runtime-rule feature with SW_RECEIVER_RUNTIME_RULE=default. Live debugger for MAL / LAL / OAL — implements SWIP-13 Live Debugger for MAL / LAL / OAL. Sample-based runtime debugger that captures per-stage inputs/outputs as the three DSLs process live ingest. Idle-path cost is one volatile-bool read per probe call site that JIT eliminates after warm-up; active sessions fan out to every cluster peer over the admin-internal gRPC bus so each peer captures its own slice. The fan-out is LB-safe: any node can serve any verb (POST mints sessionId on the receiving node, broadcasts install to peers, returns 404 rule_not_found only when no node owns the rule), so an L7 load balancer in front of the admin port routes operator requests freely. Mounts on the shared admin-server host (/dsl-debugging/* for session control plane, /runtime/oal/* for the OAL rule picker). Disabled by default; enable with SW_DSL_DEBUGGING=default (admin-server itself is on by default). injectionEnabled is a boot-time codegen switch defaulting to true — once the module is enabled, probes fire and sessions record samples; set false only if the REST surface is wanted but no codegen-side probe overhead is acceptable. Per-session limits enforce hard caps (recordCap ≤ 10000, retentionMillis ≤ 1 hour) — out-of-range requests return 400 invalid_limits. LAL sessions accept a per-session granularity=block|statement flag — block mode captures the parser/extractor/sink stages; statement mode additionally records one line entry per individual extractor statement, carrying the source-line number and verbatim DSL text so the UI can highlight which statement fired. MAL captures render the file-level filter\u0026rsquo;s surviving SampleFamily map ({\u0026quot;families\u0026quot;: N, \u0026quot;items\u0026quot;: [...]}), so multi-metric expressions show cross-family filter narrowing in the captured payload. Capture payloads include raw log bodies and parsed maps — treat the admin port as authenticated infrastructure per the Admin API security notice. Per-DSL operator references: MAL, OAL, LAL. BanyanDB schema mismatches are now visible at boot, not silent. If BanyanDB already holds a resource whose shape doesn\u0026rsquo;t match what the current rule declares (e.g., a rule was edited on disk while OAP was offline), OAP now skips that resource, logs an ERROR with the declared-vs-backend diff, and continues booting — previously the mismatch was silently accepted and samples for the affected resource were quietly dropped. To re-shape a mismatched metric, push the desired YAML through POST /runtime/rule/addOrUpdate. Bump infra-e2e to testcontainers-go v0.42.0 (apache/skywalking-infra-e2e#146), which uses Docker Compose v2 plugin natively and removes docker-compose v1 dependency. Remove deprecated version field from all docker-compose files for Compose v2 compatibility. Best-effort schema-cutover fence for BanyanDB. After firing a schema install or drop OAP now waits up to a bounded window (default 2s) for every BanyanDB data node to apply the change before resuming dispatch — the typical case gets a clean cutover where samples after 200 OK use the new shape. On laggard timeout, OAP logs a warning and proceeds anyway so a single slow node doesn\u0026rsquo;t wedge the apply. Bump dependencies: gRPC 1.70.0 → 1.80.0, protobuf-java 3.25.5 → 4.33.1, Netty 4.2.10.Final → 4.2.12.Final, Netty-tcnative 2.0.75 → 2.0.77, pgv (protoc-gen-validate) 1.2.1 → 1.3.0. Driven by the new BanyanDB schema-consistency RPCs whose generated validation code requires the protobuf-java 4.x runtime. Inspect API on admin-server. Two new admin-only HTTP endpoints for browsing the live metric catalog and the entities currently emitting values for a given metric. GET /inspect/metrics lists every registered metric with its type / scope / catalog / value-column name / supported downsamplings (pure metadata, no I/O). GET /inspect/entities runs the storage backend\u0026rsquo;s entity scan for a metric over a time range + step (capped at 300 rows) and returns each entity decoded into an MQE-ready payload — the response includes a mqeEntity block the operator pastes verbatim into the public GraphQL execExpression mutation, plus the source service\u0026rsquo;s layer(s) (multi-layer services emit one row per layer). Restricted to REGULAR_VALUE / LABELED_VALUE metrics and to non-Process scopes; HEATMAP / SAMPLED_RECORD / Process / ProcessRelation return 400. Adds IMetricsQueryDAO.listEntityIdsInRange as an abstract method on the interface — any 3rd party storage backend must explicitly override or the build fails. Enabled by default (both SW_INSPECT and SW_ADMIN_SERVER are on by default); set SW_INSPECT= empty to disable. Operator reference: Inspect API. Status feature module relocation, finalized. The legacy status-query-plugin was replaced by a new status feature module under server-admin/; the route set (/status/cluster/nodes, /status/alarm/*, /status/config/ttl, /debugging/config/dump, /debugging/query/*) keeps URIs and payloads unchanged. The selector renames from the QUERY-plugin form (SW_QUERY=…,status-query-plugin) to a top-level SW_STATUS=default (on by default); custom application.yml overrides referencing status-query need to repoint to status. Routes are admin-host only — see the \u0026ldquo;Status API is admin-host only\u0026rdquo; entry above for the public REST retirement. Drop six unused test-scoped dependencies from runtime-rule (library-integration-test, library-banyandb-client, storage-banyandb-plugin, testcontainers, testcontainers:junit-jupiter, grpc-testing). They staged the plugin-side ITs that were retired in favour of e2e; that coverage now lives in test/e2e-v2/cases/runtime-rule/ (MAL over BanyanDB / PostgreSQL / Elasticsearch, LAL, meter, and the two-node cluster case). The module has no ITs today, and JUnit and Mockito are inherited from the root POM. Declare server-testing at test scope everywhere. It ships only test scaffolding (ModuleManagerTesting, MockModuleManager, the MAL/LAL/Hierarchy rule loaders) plus two empty org.junit stubs that let Testcontainers\u0026rsquo; GenericContainer hierarchy resolve without JUnit 4, but four modules declared it at compile scope — including the server-configuration parent, so all eight configuration-* children inherited it — which put those org.junit stubs on the runtime classpath that server-starter copies into oap-libs. Modules whose tests need the stubs now declare the dependency themselves rather than inheriting it transitively, and library-banyandb-client gains the direct library-util dependency its BanyanDBClient always needed (it was resolving StringUtil through server-testing, a test-support module). Add ThreadPolicy.ioBound(N) to library-batch-queue, for queues whose consumers spend most of their time blocked. Such a queue runs its drain loops on virtual threads where the runtime provides them (JDK 25+) and falls back to N platform threads otherwise; the count, and therefore concurrency, batching, back-pressure, drop semantics and per-partition ordering, are identical on both paths. Shutdown latency is the one exception: the platform scheduler drops drain tasks parked on their idle backoff, while the virtual-thread adapter sleeps inside the submitted task and cannot, so an ioBound queue should keep maxIdleMs within shutdownTimeoutMs. There is deliberately no CPU-proportional form: virtual threads are not preemptive, so CPU-bound work would hold its carrier and starve the shared carrier pool, and L1/L2/TopN stay on cpuCores/fixed. Also fixes BatchQueue.shutdown(), which ran its final drain on the caller\u0026rsquo;s thread while drain loops could still be inside consume(), invoking a handler concurrently and breaking the single-drain-thread invariant workers such as MetricsAggregateWorker rely on: it now cancels the periodic rebalance task, waits for in-flight consumers (shutdownTimeoutMs, default 500ms per queue), and serialises its final dispatch behind a read/write dispatch lock so the guarantee holds even when that wait times out or is interrupted. Drain loops hold the read lock for the whole cycle — the running recheck, the partition dequeue, the idle notification and the dispatch — because onIdle() touches the same worker state as consume() and an unlocked dequeue would let shutdown dispatch a newer batch ahead of one a task already holds. Concurrent shutdown callers await the winner\u0026rsquo;s completion rather than returning early. A consumer is never interrupted mid-batch. OAP Server Add component IDs for the Spring LDAP Java agent plugin (spring-ldap: 179) and LDAP server (LDAP: 180), including their server mapping. Fix LAL\u0026rsquo;s segmentId and spanId extractor statements, which the grammar accepted and the parser never implemented. LALParser.g4 declares traceIdStatement, segmentIdStatement and spanIdStatement, and the codegen already carried setSegmentId/setSpanId in its setter table, but LALScriptParser.visitExtractorStatement had a branch for only the first of the three. The remaining alternatives fell through to a line that assumed whatever was left had to be an ifStatement, so a rule writing segmentId ... failed at boot with a NullPointerException naming IfStatementContext — for a rule line containing no if. Both statements now work, and an unhandled extractor statement reports its own rule line instead of throwing. Existing log records are unaffected: LogBuilder copies trace id, segment id and span id straight from the log\u0026rsquo;s metadata, and only skips that copy when a rule has set them — which no shipped rule did, which is why the gap went unnoticed. Dedicated execution tests now cover reading all three fields from log.traceContext.* and writing all three from an extractor. Remove dead code from the DSL subsystem and correct the shared kernel\u0026rsquo;s own documentation. Deleted DslContentHash (a byte-identical, zero-caller twin of the live ContentHash), the unused oal-rt metrics-function registry, LogAnalyzerFactory, LALCodegenHelper.METADATA_GETTER_ALIASES (a permanently empty map whose reader branch could never execute — the DSL-name-to-getter mismatch it existed for no longer exists, since LogMetadata.TraceContext names the field traceSegmentId directly), and three unreferenced members. Three kernel classes carried javadoc asserting consumers that do not exist — DSLClassLoaderManager claimed the MAL and LAL compile paths reach for its singleton, LogDataDebugDump claimed core renders through it, and DslContentHash instructed the reader to consolidate toward the dead copy; a false rationale in a shared kernel is an instruction to the next contributor, so those are now what the call sites actually support. OALDebug, OALDebugRecorder and DebugHolderProvider now record why they sit in core while MAL\u0026rsquo;s and LAL\u0026rsquo;s equivalents do not: dsl-debugging declares no oal-rt dependency, so they cannot move. Unify source attribution for every generated DSL class, so a stack frame from OAL, MAL, LAL or Hierarchy code leads back to the rule that produced it. SourceFile now names the RULE and its line, then the generated class file: (otel-rules/activemq/activemq-broker.yaml:32)otel_rules_activemq_activemq_broker_L32_service_meter.java. The .java generated source file is written only under SW_DYNAMIC_CLASS_ENGINE_DEBUG, so in a default deployment naming it named nothing; the class name cannot substitute because sanitising maps /, - and . all to _ and drops the extension. The _L\u0026lt;n\u0026gt;_ segment was the rules-list index rather than a line, so every file\u0026rsquo;s first rule reported as L0. It now carries the rule\u0026rsquo;s real line, resolved by the loaders themselves — including Zabbix and Hierarchy, which supplied no coordinate at all, and the runtime-rule hot-update paths for MAL and LAL, which disagreed with their own boot loaders. LineNumberTable held statement ordinals, matching neither the YAML nor the generated source. MAL keeps a per-statement table; MAL closure companions, LAL and OAL carry one entry at each method\u0026rsquo;s signature, found by searching the assembled generated source file text for that method\u0026rsquo;s declaration rather than counting a per-method offset. Hierarchy writes no generated source file, so it carries none. Generated class names change. They are now built from the rule file\u0026rsquo;s catalog-qualified path, so a MAL class that was vm_L25_cpu_total_percentage is now otel_rules_vm_L25_cpu_total_percentage. The catalog is kept only where the generated class\u0026rsquo;s package does not already imply it, so LAL — one catalog, its own package — is unchanged at default_L3_default. Nothing addresses these classes by name — they are generated, loaded reflectively and never referenced from configuration — but the names appear in stack traces, in SW_DYNAMIC_CLASS_ENGINE_DEBUG dumps and in dsl-debugging output, so saved greps and dashboards keyed on the old form need updating. The unqualified names were ambiguous: two catalogs can each hold a vm.yaml. A generated method\u0026rsquo;s line is located by searching the assembled source for its declaration, and that search now requires an identifier boundary. serialize is a suffix of deserialize and OAL declares both on every metrics class, so the shorter name resolved to the longer method\u0026rsquo;s line whenever the template order changed; two OAL metrics named cpm and commando_cpm in one scope reach the same shape through their shared dispatcher. The wrong line shipped in production bytecode — the attribute is stamped unconditionally — so a frame reported another rule\u0026rsquo;s location. Fix a hot-updated LAL rule stranding its own dsl-debugging binding instead of replacing it. The RuleKey naming a rule file was spelled default.yaml by the boot loader and default by the runtime-rule engine, so the two never met in the holder registry: the pre-update GateHolder stayed reachable, and an operator addressing it enabled probes on a compiled rule that no longer evaluates anything — indistinguishable from a rule receiving no traffic. RuleKey now drops a trailing .yaml/.yml from that component, so both spellings are one key and the debugging API keeps accepting either. Rule execution was never affected: the maps that decide which rules run are keyed by layer and rule name, not by file name. A class shared by several rules — an OAL dispatcher — names its file without a line rather than borrow the first rule\u0026rsquo;s, which would misreport every other rule routed through it. The generated .java source files are written as UTF-8 with an ASCII header instead of through a platform-default FileWriter, the pairing that produced MalformedInputException on a non-UTF-8 JVM. Breaking change: HierarchyDefinitionService.HierarchyRuleProvider — a ServiceLoader SPI — now declares buildRules(Map\u0026lt;String, String\u0026gt; ruleExpressions, Map\u0026lt;String, Integer\u0026gt; ruleLines) in place of the former one-argument buildRules(Map). A third-party provider compiled against the old signature fails with AbstractMethodError and must be recompiled. The second argument carries each rule\u0026rsquo;s line in hierarchy-definition.yml, which the expression map cannot supply because snakeyaml\u0026rsquo;s bean binding discards positional marks; a provider with no line information should pass an empty map. Without it, generated hierarchy classes are labelled _Lunknown_ and their stack frames lead nowhere. The mechanism is one implementation in org.apache.skywalking.oap.server.core.dsl, shared by all four compilers: the coordinate model, the class-name builder, the SourceFile value, the generated source file writer, the signature-line lookup and the bytecode attributes. Previously each compiler had its own, and they had drifted apart. Support runtime rule hot-update and DSL debugging for the meter-analyzer-config catalog, bringing native meter (MeterReportService) rules to parity with otel-rules. Meter rules now load through the same Rules/Rule pipeline otel-rules uses, so they participate in RuleSetMerger, are recorded in StaticRuleRegistry, support the optional layerDefinitions block, and generate source-named expression classes instead of falling back to MalExpr_\u0026lt;N\u0026gt;. MeterProcessService now implements MalConverterRegistry and publishes debug holders at boot, so a meter rule can be added / overridden / inactivated at runtime, and attached to a DSL debug session, without restarting the OAP. The internal MeterConfig / MeterConfigs model is removed in favour of the shared one. Behaviour change: an entry in meterAnalyzerActiveFiles (SW_METER_ANALYZER_ACTIVE_FILES) with no matching rule file now fails OAP startup instead of being silently ignored, matching how otel-rules has always behaved. Support Elasticsearch 9.x as storage. Add Node.js runtime metrics via the Node.js agent MeterReportService pipeline (meter_instance_nodejs_*, default 20s sample/report). OAP analyzes raw meters through nodejs-runtime.yaml. Node.js E2E asserts twelve meter_instance_nodejs_* metrics (test/e2e-v2/cases/nodejs/e2e.yaml). Add PHP runtime PHM meter analyzer (php-runtime.yaml) for SkyWalking PHP agent process metrics (CPU, memory, virtual memory, thread count, open file descriptors sampled from /proc on Linux). Registers six meter_instance_php_* metrics on the General Service layer; php-runtime is included in the default meterAnalyzerActiveFiles. Batch the BanyanDB schema fence per runtime-rule apply. A runtime-rule file changes dozens of rules at once, but the post-DDL fence (SchemaWatcher.awaitRevisionApplied) ran once per metric/downsampling, so a large file did K×M sequential ≤2s fences — on a laggy cluster that overran the apply\u0026rsquo;s REST budget. The main-node apply path now uses StorageManipulationOpt.withSchemaChangeDeferredFence(): the installer records each resource\u0026rsquo;s mod_revision without fencing and registers a single flush that the apply runs once on the file\u0026rsquo;s max revision, collapsing the whole file to one barrier. The flush is one-shot — a reconciler tick reuses one opt across every rule file, so after a file flushes, the closure and accumulated revision reset and each file fences on its own DDL only. Drops still fence inline on the dropped resource\u0026rsquo;s own delete revision — or, when that delete recorded no tombstone (mod_revision == 0), on a key-based deletion barrier (AwaitSchemaDeleted) — never on the shared opt\u0026rsquo;s cumulative revision, so a tombstone-less delete in a multi-file tick is still confirmed removed. On the operator REST apply the single create/update fence runs on a configurable, generous budget (default 180s) in the background before the rule row is persisted and dispatch resumes — it gates the persist + local commit + peer resume so the durable commit point is only reached once the schema is confirmed cluster-wide, and writes never resume against an un-propagated schema (see the apply-status entry below); the reconciler tick keeps the short inline 2s fence (a background reconcile must not wait minutes per file). Peer / withoutSchemaChange applies are unaffected (no fence). Add a runtime-rule apply-status query. The cluster main now tracks each structural apply through a phase machine (SchemaApplyCoordinator: pending → DDL → fencing → rolling-out → applied, with degraded for a committed-but-unconfirmed apply — the cluster schema fence did not confirm within the timeout, in which case the lagging data-node ids are surfaced as fenceLaggards and dispatch is resumed anyway, or the local commit-tail threw — and failed carrying the specific reason). The schema fence runs on a configurable, generous budget (receiver-runtime-rule.deferredFenceTimeoutSeconds, default 180s) and gates everything durable or visible: because an un-propagated write is silently dropped at the data node, the order after a successful DDL is suspend → DDL → fence → persist → commit → resume. The rule row (the durable commit point) is written only AFTER the fence confirms, so \u0026ldquo;durable\u0026rdquo; implies \u0026ldquo;schema propagated cluster-wide\u0026rdquo; — a main crash before persist leaves no row (peers/crash-recovery stay safely on the old content; the orphaned measure is inert), and any durable row is guaranteed fence-confirmed, so convergence never resumes dispatch against an unpropagated schema. The fence + persist + resume run in the background so they never block the HTTP response — POST /addOrUpdate returns its applyId immediately at fencing (accepted, not yet durable; dispatch for that rule still paused — a clean gap, not dropped writes), and the operator polls GET /runtime/rule/status to watch fencing → rolling-out → applied (or degraded/failed); on a genuine laggard, dispatch resumes after the budget so one stuck node can\u0026rsquo;t park the metric forever. A GetApplyStatus admin-internal gRPC served by the main backs the query — by applyId, or by catalog+name (+ optional contentHash, the durable identity) once the handle is gone after a page refresh. When the live status is gone (apply-id evicted, main restarted, or the main is unreachable), the query degrades to the durable rule row: a matching ACTIVE row reports applied derived from the content hash (a durable row is, by the fence-then-persist order, already propagation-confirmed). Non-main nodes route the read to the deterministic main; status is in-memory by design, with the content hash reconstructing truth after a restart. Push runtime-rule convergence to peers on commit. After a successful structural apply — and on the commit_deferred path, where the DB row is durable but this node\u0026rsquo;s commit-tail threw — the main broadcasts a NotifyApplied admin-internal RPC so peers reconcile against the just-persisted DB row immediately, instead of waiting up to one refresh tick (~30s) to notice it. The fan-out runs off the REST response thread (fire-and-forget on a daemon executor) so an unreachable peer\u0026rsquo;s per-call deadline never adds to the operator\u0026rsquo;s apply latency. On the peer side the notify-triggered reconcile is coalesced: a burst of notifies (a multi-rule file, or several applies) collapses to a single queued full reconcile rather than one redundant dao.getAll() scan per notify. The notify is best-effort and idempotent (the peer runs its normal per-file-locked reconcile; a lost notify is harmless — the peer still self-converges on its next tick), so it tightens the cluster-convergence window without adding a hard dependency on the main being reachable. Fix BanyanDB peer nodes permanently flooding \u0026lt;metric\u0026gt; is not registered, and a follow-on case where a peer kept translating writes with a stale schema shape after a runtime-rule reshape, when a node held a live persist worker but its local MetadataRegistry schema cache was missing or stale for that model — a withoutSchemaChange peer apply or a runtime-rule bundled fall-over rebuilt the dispatch worker but skipped the local-cache populate, and the registry was insert-only (never evicting) while the 30s reconcile only covers runtime-rule rows, so nothing re-derived it. The peer / local-cache-only install path now (re)derives and overwrites the local schema entry from the declared model with zero server RPC — honoring the inspectBackend=false contract so the cache can never lag the worker, including across a reshape — and a model removal now evicts its cache entry so a dropped or reshaped model leaves no stale translation behind; the persist DAOs keep an RPC-free re-derivation as a read-side backstop, and the no-init defer poll loop retries a transient backend probe error instead of escaping and crash-looping the pod. Support LAL json {} parsing JSON content delivered in a plain-text log body. The parser reads the native protocol\u0026rsquo;s JSON body first; when that is empty, it tries the text body as JSON — e.g. the OTLP log receiver maps every OTLP string body to a text body, even JSON-shaped ones, so previously-aborting json {} rules on OTLP-fed layers now work without any receiver or protocol change. On a successful parse from a text body, the matching rule persists the log as a JSON body with content type JSON; the normalization is scoped to that rule\u0026rsquo;s context — other rules analyzing the same log still see the original text body. Surface the drop reason in LAL live-debugging. When a LAL rule stops a log at a parse step (a json {} / yaml {} parse failure, a text { regexp } non-match, or a non-log-body input), the recorder now captures a human-readable reason (e.g. the parse exception) onto the DSL-debug Sample, exposed through the dsl-debugging REST session response and the cluster forward proto. Previously a live-debug watcher could only see continueOn=false — that a step stopped, never why — and had to read the OAP server log. Sample.reason is shared across all DSL debuggers but populated by LAL today. Fix a v2 MAL CounterWindow key collision: rate() / increase() / irate() keyed each counter\u0026rsquo;s sliding window on the rule\u0026rsquo;s output metric name (the same for every input metric of a rule) instead of the counter\u0026rsquo;s own name, so two or more counters that reduce to the same label set after .sum(...) shared one window and computed rates against each other\u0026rsquo;s values — fabricating non-zero rates from unchanged counters (e.g. the BanyanDB liaison gRPC error rate read a steady non-zero off three frozen error counters). The window is now keyed by the counter\u0026rsquo;s own metric name. Fix the v2 MAL Elvis operator ?: to honor Groovy-falsy semantics. It compiled to Optional.ofNullable(primary).orElse(fallback), applying the fallback only when the primary is null, so an empty-string primary kept \u0026quot;\u0026quot; instead — e.g. a BanyanDB liaison ServiceInstance stored node_type=\u0026quot;\u0026quot; rather than n/a, because .sum([...,'node_type']) fills an absent group-by label with \u0026quot;\u0026quot;. The fallback now applies for falsy primaries such as null, false, numeric zero, and empty strings/containers. SWIP-15: rebuild BanyanDB self-observability around the cluster / container / group model (requires BanyanDB 0.11+). A BanyanDB cluster is modeled as one Service, each container as a ServiceInstance (role/tier as attributes), and each storage group as an Endpoint. The otel-rules/banyandb/ rules are category-separated by role (node_* / liaison_* / data_* / lifecycle_*) and by data type (measure_* / stream_* / trace_* / property_*), mirroring the upstream FODC-proxy Grafana boards, and include queue batch/message granularity (apache/skywalking-banyandb#1169). Adds a SERVICE_INSTANCE_RELATION MAL scope and serviceInstanceRelation(...) builder powering a new intra-cluster pod-to-pod deployment topology (banyandb-instance-relation.yaml). The stale single-node host_name model is removed. Runtime MAL/LAL hot-update rules can declare layerDefinitions: to introduce new layers. Ordinals are operator-pinned in the 100_000+ tier; the layer is refcount-tracked and unregistered when the last declaring rule is removed. See runtime-rule-hot-update.md#dynamic-layers for the conflict rules and limitations. Fix: runtime-rule (MAL/LAL hot-update) schema changes now work in no-init mode — the deployment mode every production cluster runs. Previously a runtime addOrUpdate that introduced a new metric blocked forever in the storage installer\u0026rsquo;s init-node poll loop (ModelInstaller.whenCreating) on a no-init OAP, because the gate keyed off RunningMode rather than the operation\u0026rsquo;s intent; the /delete?mode=revertToBundled recreate and BanyanDB in-place shape updates were dead the same way. The poll loop is now gated on a new StorageManipulationOpt.Flags.deferDDLToInitNode bit set only on the static boot-time schemaCreateIfAbsent() opt (DRYed into ModelInstaller.deferDDLToInitNode(opt) and reused by the BanyanDB shape-check / group-DDL gates), so the runtime-rule opts (withSchemaChange / verifySchemaOnly / withoutSchemaChange) are driven by their flags and by cluster main-ness — no-init and default no longer differ for DSL DDL; init mode stays the dedicated initializer. DSLManager.tickStorageOpt is collapsed accordingly (main → withSchemaChange, peer → verifySchemaOnly at boot / withoutSchemaChange on tick). Fix: runtime-rule cross-node writes no longer fail with HTTP 400 forward_self_loop on a multi-replica Kubernetes cluster. Every OAP replica shared the cluster selfNodeId 0.0.0.0_11800 (derived from the 0.0.0.0 agent gRPC bind host via TelemetryRelatedContext), so the main\u0026rsquo;s self-loop guard rejected a legitimate peer-to-peer Forward as if it had looped back. The runtime-rule node identity now prefers the unique per-pod SKYWALKING_COLLECTOR_UID (the pod UID injected by the helm chart / swck operator from metadata.uid), resolved in start() before any apply, and falls back to the telemetry id for non-k8s deployments. Adds a kind-based no-init cluster e2e (test/e2e-v2/cases/runtime-rule/cluster, deployed via skywalking-helm with oap.replicas=2) that drives the apply / STRUCTURAL / inactivate / delete lifecycle and the cross-node Forward path, replacing the prior docker-compose default-mode cluster case. Fix: remove the redundant tags from the envoy-ai-gateway.yaml LAL configuration. Add Zipkin Virtual GenAI e2e test. Use zipkin_json exporter to avoid protobuf dependency conflict between opentelemetry-exporter-zipkin-proto-http (protobuf~=3.12) and opentelemetry-proto (protobuf\u0026gt;=5.0). Fix missing taskId filter and incorrect IN clause parameter binding in JDBCJFRDataQueryDAO and JDBCPprofDataQueryDAO. Remove deprecated GroupBy.field_name from BanyanDB MeasureQuery request building (Phase 1 of staged removal across repos). Push taskId filter down to the storage layer in IAsyncProfilerTaskLogQueryDAO, removing in-memory filtering from AsyncProfilerQueryService. Fix missing parentheses around OR conditions in JDBCZipkinQueryDAO.getTraces(), which caused the table filter to be bypassed for all but the first trace ID. Replaced with a proper IN clause. Fix missing and keyword in JDBCEBPFProfilingTaskDAO.getTaskRecord() SQL query, which caused a syntax error on every invocation. Fix storage layer bugs in profiling DAOs and add unit test coverage for JDBC query DAOs. Bug fixes: duplicate TABLE_COLUMN condition in JDBCMetadataQueryDAO.findEndpoint(), wrong merged table check in JFRDataQueryEsDAO (used incorrect INDEX_NAME due to copy-paste), and missing isMergedTable check in ProfileTaskQueryEsDAO.getById(). Test additions: add unit tests for 21 JDBC query DAOs verifying SQL/WHERE clause construction. Optimize TraceQueryService.sortSpans from O(N^2) to O(N) by pre-indexing spans by segmentSpanId, so trace detail queries scale linearly with span count. Support MCP (Model Context Protocol) observability for Envoy AI Gateway: MCP metrics (request CPM/latency, method breakdown, backend breakdown, initialization latency, capabilities), MCP access log sampling (errors only), ai_route_type searchable log tag, and MCP dashboard tabs. Add weighted handler support to BatchQueue adaptive partitioning. MAL metrics use weight 0.05 at L1 (vs 1.0 for OAL), reducing partition count and memory overhead when many MAL metric types are registered. Fix missing taskId filter in pprof task log query and its JDBC/BanyanDB/Elasticsearch implementations. Fix duplicate calls in EndpointTopologyBuilder — calls were not deduplicated unlike ServiceTopologyBuilder, causing duplicate entries when storage returns multiple records for the same relation. Use containsOnce and noDuplicates for topology dependency e2e expected files to enforce no-duplicate verification. Bump infra-e2e to ef073ad to include noDuplicates pipe function support. PromQL: support querying Zipkin metadata (service name, remote service name, span name). TraceQL: support more tags and variables in Grafana for querying. LAL: add sourceAttribute() function for non-persistent OTLP resource attribute access in LAL scripts. LAL: add layer: auto mode for dynamic layer assignment when service.layer is absent. Add two-phase SpanListener SPI mechanism for extensible trace span processing. Refactor GenAI from hardcoded SpanForward.processGenAILogic() to GenAISpanListener. Add OTLP/HTTP receiver support for traces, logs, and metrics (/v1/traces, /v1/logs, /v1/metrics). Supports both application/x-protobuf and application/json content types. Fix: TTL query add metadata TTL. Fix: PersistentWorker used wrong TTL for metrics cache if the storage is BanyanDB. Add iOS/iPadOS app monitoring via OpenTelemetry Swift SDK (SWIP-11). Includes the IOS layer, IOSHTTPSpanListener for outbound HTTP client metrics (supports OTel Swift .old/.stable/.httpDup semantic-convention modes via stable-then-legacy attribute fallback), IOSMetricKitSpanListener for daily MetricKit metrics (exit counts split by foreground/background, app-launch / hang-time percentile histograms with finite 30 s overflow ceiling), LAL rules for crash/hang diagnostics, Mobile menu, and iOS dashboards. Add Apache Airflow monitoring via native OpenTelemetry metrics (SWIP-7). New AIRFLOW layer with Service (cluster) and Instance (host) dimensions, MAL rules under otel-rules/airflow/ (27 metrics), setup documentation, mock OTLP e2e (cases/airflow/mock/e2e.yaml: 2 entity + 27 metric checks, 29 total), and real Celery-cluster integration smoke (cases/airflow/cluster/e2e.yaml: 2 entity + 14 metric checks, 16 total). See test/e2e-v2/cases/airflow/README.md. Horizon UI dashboards ship separately in apache/skywalking-horizon-ui under the Workflow Scheduler menu group. Fix LAL layer: auto mode dropping logs after extractor set the layer. Codegen now propagates layer \u0026quot;...\u0026quot; assignments to LogMetadata.layer so FilterSpec.doSink() sees the script-decided layer. Fix MetricKit histogram percentile metrics being reported at 1000× their true value — the listener now marks its SampleFamily with defaultHistogramBucketUnit(MILLISECONDS) so MAL\u0026rsquo;s default SECONDS→MS rescale of le labels is not applied. Add WeChat and Alipay Mini Program monitoring via the SkyAPM mini-program-monitor SDK (SWIP-12). Two new layers (WECHAT_MINI_PROGRAM, ALIPAY_MINI_PROGRAM); two new JavaScript componentIds (WeChat-MiniProgram: 10002, AliPay-MiniProgram: 10003). Service / instance / endpoint entities are produced by MAL + LAL, not trace analysis — mini-programs are client-side (exit-only) so RPCAnalysisListener stays unchanged (same pattern as browser and iOS). MAL rules per platform × scope under otel-rules/miniprogram/ with explicit .service(...) / .endpoint(...) chains (empty expSuffix so endpoint-scope rules aren\u0026rsquo;t overridden), histogram percentile via .histogram(\u0026quot;le\u0026quot;, TimeUnit.MILLISECONDS) to keep ms bucket bounds intact, and request-cpm derived from the histogram _count family. LAL layer: auto rule produces both layers via miniprogram.platform dispatch and emits error-count samples consumed by per-platform log-MAL rules. Per-layer menu entries and service / instance / endpoint dashboards with Trace and Log sub-tabs. Fix: remove VirtualServiceAnalysisListener\u0026rsquo;s dependency on GenAIAnalyzerModule if it is disabled. MAL: register TimeUnit in MALCodegenHelper.ENUM_FQCN so rule YAML can write .histogram(\u0026quot;le\u0026quot;, TimeUnit.MILLISECONDS) for SDKs that emit histogram bucket bounds in ms (default SECONDS unit applies a ×1000 rescale that would otherwise inflate stored le labels 1000×). Fix: potential unexpected current directory inclusion in Docker OAP classpath. MAL: add safeDiv(divisor) on SampleFamily that yields 0 when the divisor is 0 instead of Infinity/NaN. Replace / with safeDiv(...) in Envoy AI Gateway latency-average rules so sum / count * 1000 no longer produces dropped or out-of-range samples when a counter is zero in a window. Fix: envoy-ai-gateway metrics rules, make the metrics value return 0 when the divisor is 0. Custom Layers can be declared without modifying the OAP source — via an operator-managed layer-extensions.yml, inline layerDefinitions: block in a MAL or LAL rule file, or a plugin extension. UI dashboard templates for new layers are auto-discovered from the ui-initialized-templates/ directory. Recommended ordinal range for external layers is \u0026gt;= 1000; conflicting names or ordinals are reported at boot. LAL: support full arithmetic (+, -, *, /) on numeric operands and fix the original bug where (tag(\u0026quot;x\u0026quot;) as Integer) + (tag(\u0026quot;y\u0026quot;) as Integer) was treated as string concatenation — expressions like input_tokens + output_tokens \u0026lt; 10000 produced the concatenated string \u0026quot;2589115\u0026quot; rather than the integer sum 2704, so token-threshold conditions never triggered abort {}. Operand types are now inferred from explicit casts (as Integer / as Long / as Float / as Double), typed proto fields, or numeric literal shape (with L / F / D suffix support, e.g. 1000L). The compiler honours JLS-style binary numeric promotion and emits Java arithmetic in the declared primitive type — (x as Integer) + (y as Integer) compiles to int + int (not widened to long). + with any String operand falls back to string concatenation; - / * / / against non-numeric operands produces a compile-time error. The as Double and as Float casts are accepted in typeCast clauses, including in def declarations. Numeric comparisons honour declared casts on both sides (no more universal h.toLong() wrapper). Fix: avgHistogramPercentile / sumHistogramPercentile meter functions reported the smallest finite bucket boundary (e.g. 10 for OTel gen_ai_server_request_duration whose le is rewritten from 0.01s → 10ms) for every rank when no samples were observed in any bucket. The percentile loop\u0026rsquo;s count \u0026gt;= roof check matched on the first sorted bucket because both sides were 0. calculate() now short-circuits to 0 for every rank when the windowed total is 0. Fix: MAL expPrefix now applies to every metric source in exp, not just the leading one. Previously the prefix was spliced after the first ., so secondary metrics inside arguments (e.g. the divisor in a.sum(['s']).safeDiv(b.sum(['s']))) silently skipped the prefix — a rule like envoy-ai-gateway\u0026rsquo;s request_latency_avg (sum / count) would tag-rewrite only the dividend. The injection is now AST-aware: every bare-IDENTIFIER metric source is wrapped, while downsampling-type constants (SUM, AVG, LATEST, SUM_PER_MIN, MAX, MIN) are skipped. Add @Stream(allowBootReshape = true) opt-in for additive boot-time reshape of BanyanDB streams / measures. Code-defined stream classes (e.g. AlarmRecord) can now annotate their schema as eligible for in-place additive update at OAP boot — a new @Column is appended to the live tag-family / fields via client.update instead of being silently rejected with SKIPPED_SHAPE_MISMATCH (which previously forced operators to drop the measure / stream and lose historical rows). Additive includes both new tags / fields and relocating an existing tag between families when a @Column\u0026rsquo;s storageOnly flag flips (e.g. id1 moving from storage-only → searchable when it becomes indexed). The opt-in is per-stream and gated by an isPurelyAdditive shape diff: tag type changes, tag drops, kind flips (tag↔field), entity / interval / sharding-key changes, and field re-typing still skip with SKIPPED_SHAPE_MISMATCH, so identity-breaking edits remain explicit operator actions. Only the init / standalone OAP performs the reshape; non-init peers continue through the existing poll-and-wait loop so a single node drives DDL. When a check* records SKIPPED_SHAPE_MISMATCH the dependent IndexRule / IndexRuleBinding reconciliation is also skipped — preventing the previous gap where the binding silently updated to a tag list that diverged from the live tag-family layout. AlarmRecord is opted in. Default remains false for all other models — boot-time reshape stays off unless the annotation is explicitly set. Operator caveat: BanyanDB does not physically migrate existing rows when a tag\u0026rsquo;s family changes; pre-existing data stays in its original on-disk location while new writes go to the declared family — expect a backfill window for queries that route through new IndexRules on relocated tags. Mask keywords trustStorePass, keyStorePass by default. Bump up dependencies to clear CVE alerts on shipped OAP jars: log4j 2.25.3 → 2.25.4, jackson 2.18.5 → 2.18.6, kafka-clients 3.4.0 → 3.9.2, postgresql 42.4.4 → 42.7.11, commons-compress 1.21 → 1.26.2. Bump up more dependencies to clear CVE alerts on shipped OAP jars: netty 4.2.12.Final → 4.2.15.Final, jackson 2.18.6 → 2.18.8, commons-codec 1.11 → 1.13. Also realign jackson-databind 2.16.0 → 2.18.8 so the whole jackson family is managed at a single version (it had been left behind the other jackson artifacts). Bump Apache Curator 4.3.0 → 5.9.0 and Apache ZooKeeper 3.5.7 → 3.9.5 together to clear CVE-2023-44981 (the bundled ZooKeeper jar carried it; OAP is a ZooKeeper client only, so the server-side bug was never reachable, but the jar tripped Dependabot). The cluster-zookeeper and configuration-zookeeper plugins use only stable Curator APIs, so no source changes were required. Operator-facing change: the supported ZooKeeper server version is now 3.6+ (Curator 5.x uses ZooKeeper persistent watches, added in server 3.6.0); older servers (3.5.x, 3.4.x) are no longer supported. Migrate the Consul cluster and configuration client from the abandoned com.orbitz.consul:consul-client 1.5.3 to the maintained fork org.kiwiproject:consul-client 0.9.0 to clear the okhttp CVE the old client carried (CVE-2021-0341; the old client pinned okhttp 3.14.9, fixed in okhttp 4.9.2+), so the BOM now pins okhttp to 4.12.0. The fork\u0026rsquo;s 0.9.x line is the last one built for JDK 11 (which SkyWalking still targets); 1.0.0+ is compiled to JDK 17 bytecode, so the migration stays on 0.9.0. The cluster-consul and configuration-consul plugins use only stable Consul client APIs, so the change is a package rename (com.orbitz.consul → org.kiwiproject.consul); okhttp is pulled only by the Consul plugins (the fabric8 Kubernetes client excludes its okhttp transport), so no other module is affected. Bump test-scope assertj-core 3.20.2 → 3.27.7 to clear CVE-2026-24400 (XXE in isXmlEqualTo, not used by any test). Clear three security alerts: bump the Airflow e2e mock\u0026rsquo;s pinned protobuf 4.25.8 → 5.29.6 (with opentelemetry-proto 1.24.0 → 1.28.0, whose protobuf\u0026lt;5.0 cap was the blocker, and grpcio 1.62.2 → 1.63.2, required because opentelemetry-proto 1.28.0\u0026rsquo;s gRPC stubs call unary_unary(_registered_method=...)) to clear CVE-2026-0994 — a CI-only test fixture, never shipped; and widen the cumulative count accumulator from int to long in SumHistogramPercentileFunction / AvgHistogramPercentileFunction to clear the CodeQL implicit-cast-in-compound-assignment alerts (count += value silently narrowed a long bucket-count sum back to int, while total was already long). Clear Dependabot CVE alerts in the e2e Go test fixtures (cases/go/service and cases/profiling/ebpf/network, CI-only, never shipped in any OAP artifact): bump golang.org/x/net 0.48.0 → 0.55.0 (CVE-2026-25681, CVE-2026-27136, CVE-2026-33814, CVE-2026-39821) and move the Go toolchain from 1.24 to 1.26.5 (CVE-2026-27145 / CVE-2026-42504 fixed in 1.26.4, CVE-2026-39822 fixed in 1.26.5) by switching the shared skywalking-go base image to the -go1.26 variant and bumping SW_AGENT_GO_COMMIT to 7544822, whose -go1.26 image ships go1.26.5. Fix: continuous profiling policy validation now rejects a threshold / count of 0 to match the error messages and rover\u0026rsquo;s value \u0026gt;= threshold trigger semantics (a 0 threshold would always trigger). CPU percent and HTTP error rate are tightened from [0-100] to (0-100]. Fix wrong BanyanDB resource options in record data. Align the default BanyanDB stage segmentInterval values so each coarser stage is an integer multiple of the finer one (records cold 3 → 4, metricsMinute cold 5 → 6, metricsHour warm 7 → 10 and cold 15 → 20), keeping hot → warm → cold lifecycle migration on the cheap whole-segment fast path. Fix: layer-extensions.yml is now excluded from the skywalking-oap jar and shipped to the distribution config/ directory, so an operator-edited config/layer-extensions.yml is no longer shadowed by the empty template bundled in the jar. Because the OAP launch script puts oap-libs/*.jar ahead of config/ on the classpath, ResourceUtils.read(\u0026quot;layer-extensions.yml\u0026quot;) previously always resolved the jar-bundled layers: [] and silently ignored the operator\u0026rsquo;s file — custom layers declared there never registered. The file now follows the same exclude-from-jar + copy-to-config/ packaging as every other operator-editable config (application.yml, alarm-settings.yml, etc.). Fix: the v2 MAL compiler now resolves custom layers referenced as Layer.NAME in an expression. A custom layer declared through a layerDefinitions: block (or layer-extensions.yml / the LayerExtension SPI) has no generated Layer.* static field, so service(['svc'], Layer.IOT_FLEET) previously failed code generation because Layer has no IOT_FLEET field. The compiler now lowers every Layer.NAME static-field reference to a runtime Layer.nameOf(\u0026quot;NAME\u0026quot;) registry lookup, so a custom layer can be referenced exactly like a built-in one (Layer.GENERAL). For a built-in layer this is equivalent, because Layer.nameOf(\u0026quot;GENERAL\u0026quot;) returns the same instance as the Layer.GENERAL field. The lowering is scoped to Layer only; the other MAL enum types (DetectPoint, DownsamplingType, etc.) are real Java enums and keep their direct static-field reference. Fix Envoy ALS rendering for the LAL live-debugger and the persisted log content: an Istio metadata-exchange peer in common_properties.filter_state_objects (legacy Wasm wasm.*_peer = Any{BytesValue} wrapping a FlatBuffer, or modern *_peer = Any{Struct}) is now decoded into the readable peer metadata (pod / namespace / labels) instead of an opaque jsonformat-failed envelope or base64. The serialization is hardened so a single un-printable field can no longer blank the whole entry — the LalPayloadDebugDump printer carries a well-known-type TypeRegistry and sanitizes every value JsonFormat would reject (an unresolvable, no-slash, or corrupt-bytes Any degrades to an @unresolved placeholder; a non-finite Value double NaN/Infinity is rendered as a string), keeping the rest of the entry readable. Because the LAL output builder\u0026rsquo;s bindInput runs eagerly before the debug capture, this also stops an unregistered filter_state_objects type from throwing and aborting the whole rule (dropping the mesh log). Decoding is wired through a new LalInputDebugRenderer SPI (EnvoyAlsHttpDebugRenderer / EnvoyAlsTcpDebugRenderer) so log-analyzer reaches the receiver-side decoders without depending on the Envoy receiver, and covers both HTTP and TCP access logs. Surface the effective BanyanDB configuration (bydb.yml / bydb-topn.yml) in the /debugging/config/dump admin API. Because the BanyanDB config moved to a separate file in 10.2.0, a BanyanDB deployment previously showed an empty storage.banyandb block in the dump; its post-environment-resolution values are now merged into the same response under storage.banyandb.* (TopN rules under storage.banyandb.topN.*), masked by the same secret-keyword list, via a generic ConfigDumpExtension SPI on ServerStatusService that any module loading config from a secondary file can implement. Fix: an MQE top_n(metric, N, order, attrX='value') query whose attribute is not a column of the target metric now returns a descriptive MQE error instead of a raw storage IOException surfaced as Internal IO exception, query metrics error.. Attribute columns (attr0..attrN) exist only on decorated metrics (service_* / endpoint_* / kubernetes_service_*, set to the layer name via OAL .decorator(...)) and the MAL meter base; metrics such as relations or database / cache / mq access carry none, so passing an attribute condition previously reached the storage engine with a tag it does not define and failed there. MQEVisitor now validates each attribute key against the metric\u0026rsquo;s registered queryable columns before the storage call and raises IllegalExpressionException (naming the attribute and the metric) when it is absent. Migrate all BanyanDB storage read queries from the typed query-builder API to BydbQL. Fix: BanyanDB queries no longer silently truncate at the storage engine\u0026rsquo;s implicit row cap. BanyanDB applies its own default limit to any query that carries none — 100 rows for measures, 20 for streams/traces — and applies it after GROUP BY, so an over-long result set is cut short rather than rejected. OAP never sent a limit on several read paths, so a metrics query returned at most 100 data points regardless of the requested range: a 4-hour minute-step read rendered only its first 100 minutes and the rest showed as empty, even though DurationUtils allows up to 500 steps. The same cap silently shortened topology relation maps, instance/process metadata lists, profiling thread snapshots and eBPF task lists. Every BydbQL query now leaves OAP with an explicit LIMIT: the entity-scoped metrics read sends the exact number of assembled duration points (matching the row set the ES/JDBC DAOs fetch by id), ad-hoc SELECT TOP sends its own N, and anything that does not paginate itself falls back to the configured resultWindowMaxSize (default 10000) instead of the engine default. ES and JDBC storage were never affected. Support BanyanDB\u0026rsquo;s group-scoped trace retention pipeline in bydb.yml. The trace and zipkinTrace groups gain a pipeline block (enabled, enabledEvents, mergeGraceSeconds, finalizeGraceSeconds, and an ordered plugins chain) that OAP pushes onto the BanyanDB group as a TracePipelineConfig, letting a sampler plugin drop traces inside the data node during Hot-phase compaction — after storage, so it reclaims space already written and decides per whole trace, unlike the ingest-side server-side trace sampling. Disabled by default, since it deletes stored traces. Even when enabled it is inert unless the data node runs the plugin-capable BanyanDB image with the sampler .so mounted — a node without that support ignores the config, and one that cannot load the plugin logs an error and merges unfiltered, so nothing is dropped unexpectedly. The two grace windows use -1 for \u0026ldquo;inherit the data node default\u0026rdquo; (30s merge / 5m finalize) because the node treats any non-positive grace as unset. enabledEvents accepts a comma-separated string so it can be set from the environment (SW_STORAGE_BANYANDB_TRACE_PIPELINE_ENABLED_EVENTS and its ZIPKIN_ variant) as well as a YAML block list; an empty value falls back to PIPELINE_EVENT_MERGE, so PIPELINE_EVENT_FINALIZE runs only when named explicitly. Each plugin\u0026rsquo;s config is passed through verbatim as a protobuf Struct: nested lists and objects (e.g. keepTagRules) now survive the config loader and are serialized as real ListValue/Struct rather than being flattened to a string. Note a float written as a ${ENV:default} placeholder still reaches the plugin as a JSON string, because the shared placeholder resolver only preserves String/Integer/Long/Boolean; the first-party samplers accept a quoted number for exactly this reason. See Trace Tail Sampling for how a trace is judged, and BanyanDB storage for the configuration keys. Fix: a blank value in bydb.yml (key: with nothing after it) aborted OAP startup with an opaque NullPointerException from java.util.Properties, which rejects null values. The BanyanDB config loader now skips blank entries and leaves the field at its default, the same outcome as omitting the line. Route LAL rules within a layer by their input type, so a single layer can host rules over different proto inputs. Each compiled rule now carries its effective input type (the proto class its parsed.* getters cast to, or null for parser-based / untyped rules), and LogFilterListener skips any rule whose type doesn\u0026rsquo;t match the incoming log instead of running every rule in the layer. This fixes a latent ClassCastException (caught and logged per log) that fired whenever a MESH log of one shape reached a rule compiled for another — e.g. an Envoy TCP access log or a network-profiling LogData hitting the HTTP envoy-als rule. Adds an envoy-als-tcp rule (inputType: TCPAccessLogEntry) alongside the existing HTTP envoy-als; both share the MESH layer and each now only sees its own entry type. Fix the PagerDuty alarm hook to default its Events API v2 endpoint to https://events.pagerduty.com/v2/enqueue. Fix HttpAlarmCallback logging a successful alarm delivery as a failure. The shared HTTP hook helper treated only 200 and 204 as success, so any other 2xx — notably the 202 Accepted returned by asynchronous intake APIs such as PagerDuty\u0026rsquo;s Events API v2 — produced send to ... failure. Response code: 202 at ERROR level on every delivered alarm. The alarm was still delivered; the log entry was wrong. The check now accepts the whole 2xx range, for all alarm hooks. Make the PagerDuty Events API v2 endpoint configurable through a new optional events-api-url setting on each pagerduty hook, defaulting to the US service region endpoint. An account in PagerDuty\u0026rsquo;s EU service region can now point the hook straight at https://events.eu.pagerduty.com/v2/enqueue rather than relying on PagerDuty forwarding the request — and the routing key and alarm payload from an EU-region account no longer transit the US region. Bump the default BanyanDB compatible server API version (SW_STORAGE_BANYANDB_COMPATIBLE_SERVER_API_VERSIONS) from 0.10 to 0.11. UI Add Airflow layer dashboards and menu i18n under Workflow Scheduler in Horizon UI (SWIP-7). Add mobile menu icon and i18n labels for the iOS layer. Fix metric label rendering in multi-expression dashboard widgets. Add i18n menu labels for WeChat Mini Program and Alipay Mini Program (en / zh / es) — sub-menus rendered as raw keys until this bump. Support trace V1 view in trace single page. Documentation Document the meter-analyzer-config catalog in the runtime-rule hot-update and DSL-debugging references, and add the optional layerDefinitions block, the active-files startup-failure behaviour, and a hot-update / debugging section to the meter setup doc. Update LAL documentation with sourceAttribute() function and layer: auto mode. Add Airflow monitoring setup documentation (SWIP-7). Add iOS app monitoring setup documentation. Add WeChat / Alipay Mini Program monitoring setup documentation, plus a client-side-monitoring section in the security guide covering public-internet ingress (OTLP + /v3/segments) for mobile / browser / mini-program SDKs. Improve downsampling documentation Fix the docker-compose quickstart: OAP healthcheck no longer calls curl (absent from the JRE image) and probes the query port via bash /dev/tcp; the Horizon UI service maps the correct container port (8081) and mounts a horizon.yaml (binding 0.0.0.0, OAP URLs, demo admin/admin login) instead of non-existent SW_*_ADDRESS env vars. Add PHP runtime metrics (PHM) dashboard documentation (agent setup, OAP php-runtime MAL rules, Horizon UI widgets). Add Node.js runtime metrics dashboard documentation (agent setup, OAP nodejs-runtime MAL rules, Horizon UI widgets). Add a BanyanDB trace tail sampling guide under \u0026ldquo;BanyanDB Exclusive Setup\u0026rdquo;, covering how a trace is judged (the OR-ed rule chain, the end-to-end duration envelope rather than a per-span maximum, and the deterministic trace-ID hash behind healthySampleRate), what the two first-party samplers read from each trace schema, the MERGE vs FINALIZE events and their grace windows, the fail-open behaviour when a plugin is absent or unloadable, and the metrics to watch. Also document the Zipkin receiver\u0026rsquo;s previously undocumented sampleRate and maxSpansPerSecond in the server-side trace sampling guide. Correct the APISIX monitoring guide to align its Collector configuration and metric names with the current APISIX MAL rules and Horizon UI Dashboard. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1100\"\u003e11.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eMove the DSL class-loading machinery under \u003ccode\u003ecore/dsl\u003c/code\u003e. \u003ccode\u003ecore/classloader\u003c/code\u003e held only DSL …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v11.0.0/en/changes/changes/","title":"11.0.0"},{"body":"11.1.0 Project OAP Server Add customizable LLM-as-judge support for AI evaluation, with OpenAI-compatible endpoint / model / API key configuration, and persist the evaluation result as queryable GenAIEvaluationRecord rows for later inspection. Support hot-reloading the BanyanDB credentials from a new secretsManagementFile setting in bydb.yml (SW_STORAGE_BANYANDB_SECRETS_MANAGEMENT_FILE), matching the ElasticSearch storage plugin. The properties file carrying user/password is watched, so a rotation performed by a 3rd party tool such as Vault is applied without restarting the OAP. The credentials are read per RPC by the gRPC auth interceptor, so they are swapped without re-establishing the channel and without interrupting in-flight queries or writes. Whatever the file contains is applied, including an incomplete pair, so that a mistake in it fails visibly rather than being masked by credentials that silently keep working; a username and a password are only ever sent together, so an incomplete file means requests carry no credentials and are answered with UNAUTHENTICATED rather than authenticating as the named user. Support hot-reloading the BanyanDB TLS trust CA. The file at sslTrustCAPath is now watched, and a change rebuilds the gRPC channel so the new CA takes effect without restarting the OAP. Previously the CA was only re-read after the certificate in use had already caused requests to fail, which meant a rotation was paid for with an outage. The replacement channel is created before the old one is released, so a failed rebuild leaves the current channel serving; requests already in flight finish on the old channel. The replacement is not health-checked before the swap — a gRPC channel connects lazily — so rotating to a CA that does not validate the server interrupts traffic until valid material is written back; the storage doc describes the old+new overlap procedure that avoids this. Note the replacement re-picks an address from targets, so the OAP may connect to a different node after a rotation. Fix MultipleFilesChangeMonitor being able to silently disable every file watch in the OAP. Its registry of monitors was a plain ArrayList that scanChanges() iterated from the scheduler thread without holding the lock that start() / stop() take, so starting a monitor while a scan was in flight could raise a ConcurrentModificationException from the iterator. That exception escapes past the per-monitor catch, and an uncaught exception cancels a scheduleAtFixedRate task permanently — after which no secrets file, keystore, or TLS certificate is ever reloaded again, with nothing in the log to say so. The registry is now copy-on-write. The failure log in the same scan loop also now names the monitor that failed instead of printing an empty gourp = . Fix MultipleFilesChangeMonitor never honouring its watching period. lastCheckTimestamp was declared and compared against, but never assigned, so the guard always measured against 0 and passed — every registered monitor re-stat\u0026rsquo;d its watched files on each 200ms tick of the shared scheduler thread, and the watchingPeriodInSec constructor argument had no effect at all. This affects every file watch in the OAP: the ElasticSearch storage secrets / truststore / keystore watch, the BanyanDB credentials and trust CA watches, and the TLS certificate watches behind each OAP HTTP and gRPC server, all of which ask for 10 seconds. Change detection is now paced as configured, which also means it is no longer near-instant: a rotated file is picked up within the requested period rather than within ~200ms. Add BanyanDB trace tail sampling metrics to the BanyanDB self-observability layer, in a new otel-rules/banyandb/banyandb-trace-sampling.yaml rule file. It covers the whole banyandb_trace_pipeline_* / banyandb_trace_tst_pipeline_* catalog a sampler plugin chain emits — pipeline reconciliation, per-plugin Decide execution rate and latency, chain batching, the trace-level evaluated / retained / dropped / immature outcomes, every fail-open guard and bounded-retention counter, drop-set capacity and finalization state, the plugin telemetry-host safety bounds, and the first-party sw-trace-sampler / zipkin-trace-sampler decision and row metrics. The plugin chain is optional, and the metrics follow it: on a cluster with no sampler configured the wire families are never registered, so every metric here stays absent rather than reading zero. Modeled at Service scope with group kept as a metric label rather than at Endpoint scope, so one cluster-wide page can render per-group series and cluster totals alike — OAP does no cross-scope rollup, so an Endpoint-scope metric could not have been aggregated back up to the cluster. Fix a second CounterWindow key collision in the v2 MAL engine, this time ACROSS rules. rate() / increase() / irate() resolve their lower bound from a process-wide window keyed on the counter\u0026rsquo;s own name plus its post-.sum(...) label set, with nothing identifying the rule doing the evaluation. Two rules that read one wire family, tell their streams apart with tagEqual(...), and then .sum(...) away the label they filtered on therefore collapse onto one window slot and difference against each other\u0026rsquo;s values. The queue is ordered by (timestamp, value), so the smaller counter wins the lower-bound lookup and still reads correctly while its partner is inflated by the gap between them — which is why this went unnoticed. A collision needs the discriminating label to be DROPPED by the .sum(...): where it survives, the rules\u0026rsquo; label values differ and the window keeps them apart. Auditing the shipped rules on that basis gives 10 colliding keys over ~25 rules — meter_activemq_cluster_gc_parallel_young_collection_count reported ~9000/min of young-gen collections from a completely idle broker (differencing against the old-gen counter); MySQL commands_* / tps rate against each other; so do the GenAI gateway input/output token rates, four Envoy cluster_* counters, APISIX matched/unmatched instance bandwidth, and BanyanDB\u0026rsquo;s own network_recv / network_sent, which drop the kind label that separates bytes-received from bytes-sent on one interface. Measured against two live scrapes of the demo cluster\u0026rsquo;s FODC proxy, that last pair was wrong on every interface: network_sent read a flat 0 B/s and network_recv read large negative values (down to -778 MB/s) from differencing against the sent counter, where both now match the byte delta exactly. No rule changes were needed for any of these \u0026ndash; each rule already reduces to the labels it should; only the window key was wrong. The window is now keyed by (owning rule, counter name, labels). This is the complement of the within-rule collision fixed earlier by keying on the counter\u0026rsquo;s own name: neither name alone is sufficient, because the two collisions are independent. RunningContext.metricName — written on every rule evaluation and read by nobody since that earlier fix — is what supplies the rule identity, so no code generation or MAL syntax changes. Note the whole-rule-set comparison suite could not have caught this: it resets the shared window before every rule, the one condition under which the collision cannot appear. Fix meter_rabbitmq_node_outgoing_messages_total double-counting one of its terms. The rule summed six delivery-rate terms but rabbitmq_global_messages_delivered_get_auto_ack_total appeared twice, so auto-ack basic.get deliveries were counted once more than the other four delivery paths and the reported outgoing rate ran high whenever polling consumers were in use. The duplicate term is removed, leaving the five distinct families (redelivered, consume auto/manual ack, get auto/manual ack). Add AI agent conversations landed by the AI Sessionizer: the AI_AGENT layer, the bundled lal/ai-agent.yaml rule with the ConversationFile output builder that verifies and stores Session Data and Session Flow files, the ai_agent_session_data and ai_agent_session_flow models in a new BanyanDB group recordsAIAgent, the ai-agent-conversation module that folds a conversation into one asz.view document, and the listConversations (with optional conversation and title conditions) / getConversationRawFiles GraphQL queries and the streamed GET /ai-agent/conversations/{conversation}/v1/view route that serves the document. A round from before the list attributes existed lands and lists with zero counts, and the view shows as much as landed: the chain resumes after a missing, unreadable or refused round, and the absent rounds and files are named once as ranges. A file over maxFileBytes, 15 MiB by default, is rejected at ingest and counted under the reason size, because one file over BanyanDB\u0026rsquo;s 16 MiB gRPC message limit fails the bulk write it travels in and every record behind it; on MySQL the body column is LONGTEXT, since a body that size outgrows MEDIUMTEXT as Base64. Each window read is capped at maxResponseBytes, 100 MiB by default, as a per-call option on the BanyanDB client in place of its 50 MB default, so the module\u0026rsquo;s reads are bounded by its own settings and nothing else\u0026rsquo;s read changes. UI Add a Virtual GenAI evaluation-record page and evaluation-score chart in Horizon UI, so operators can inspect evaluation result, level, reason, judge model, timestamp, trace linkage, and the gen_ai_model_evaluation_score_ppm trend for evaluated records. Documentation Document the BanyanDB trace tail sampling metrics in the BanyanDB self-observability dashboard catalog, and point the \u0026ldquo;Operating it\u0026rdquo; section of the trace tail sampling guide at them — the OAP-collected metrics show what the sampler plugins proposed next to what storage committed, which the data node\u0026rsquo;s raw metrics endpoint alone does not. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"1110\"\u003e11.1.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003ch4 id=\"oap-server\"\u003eOAP Server\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eAdd customizable LLM-as-judge support for AI evaluation, with …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/next/en/changes/changes/","title":"11.1.0"},{"body":"5.1.0 Agent Changes Fix spring inherit issue in another way Fix classloader dead lock in jdk7+ - 5.x Support Spring mvc 5.x Support Spring webflux 5.x Collector Changes Fix too many open files. Fix the buffer file cannot delete. 5.0.0-GA Agent Changes Add several package names ignore in agent settings. Classes in these packages would be enhanced, even plugin declared. Support Undertow 2.x plugin. Fix wrong class names of Motan plugin, not a feature related issue, just naming. Collector Changes Make buffer file handler close more safety. Fix NPE in AlarmService Documentation Fix compiling doc link. Update new live demo address. 5.0.0-RC2 Agent Changes Support ActiveMQ 5.x Support RuntimeContext used out of TracingContext. Support Oracle ojdbc8 Plugin. Support ElasticSearch client transport 5.2-5.6 Plugin Support using agent.config with given path through system properties. Add a new way to transmit the Request and Response, to avoid bugs in Hytrix scenarios. Fix HTTPComponent client v4 operation name is empty. Fix 2 possible NPEs in Spring plugin. Fix a possible span leak in SpringMVC plugin. Fix NPE in Spring callback plugin. Collector Changes Add GZip support for Zipkin receiver. Add new component IDs for nodejs. Fix Zipkin span receiver may miss data in request. Optimize codes in heatmap calculation. Reduce unnecessary divide. Fix NPE in Alarm content generation. Fix the precision lost in ServiceNameService#startTimeMillis. Fix GC count is 0. Fix topology breaks when RPC client uses the async thread call. UI Changes Fix UI port can\u0026rsquo;t be set by startup script in Windows. Fix Topology self link error. Fix stack color mismatch label color in gc time chart. Documentation Add users list. Fix several document typo. Sync the Chinese documents. Add OpenAPM badge. Add icon/font documents to NOTICE files. Issues and Pull requests\n5.0.0-beta2 UI -\u0026gt; Collector GraphQL query protocol Add order and status in trace query. Agent Changes Add SOFA plugin. Add witness class for Kafka plugin. Add RuntimeContext in Context. Fix RuntimeContext fail in Tomcat plugin. Fix incompatible for getPropertyDescriptors in Spring core. Fix spymemcached plugin bug. Fix database URL parser bug. Fix StringIndexOutOfBoundsException when mysql jdbc url without databaseName。 Fix duplicate slash in Spring MVC plugin bug. Fix namespace bug. Fix NPE in Okhttp plugin when connect failed. FIx MalformedURLException in httpClientComponent plugin. Remove unused dependencies in Dubbo plugin. Remove gRPC timeout to avoid out of memory leak. Rewrite Async http client plugin. [Incubating] Add trace custom ignore optional plugin. Collector Changes Topology query optimization for more than 100 apps. Error rate alarm is not triggered. Tolerate unsupported segments. Support Integer Array, Long Array, String Array, Double Array in streaming data model. Support multiple entry span and multiple service name in one segment durtaion record. Use BulkProcessor to control the linear writing of data by multiple threads. Determine the log is enabled for the DEBUG level before printing message. Add static modifier to Logger. Add AspNet component. Filter inactive service in query. Support to query service based on Application. Fix RemoteDataMappingIdNotFoundException Exclude component-libaries.xml file in collector-*.jar, make sure it is in /conf only. Separate a single TTL in minute to in minute, hour, day, month metric and trace. Add order and status in trace query. Add folder lock to buffer folder. Modify operationName search from match to match_phrase. [Incubating] Add Zipkin span receiver. Support analysis Zipkin v1/v2 formats. [Incubating] Support sharding-sphere as storage implementor. UI Changes Support login and access control. Add new webapp.yml configuration file. Modify webapp startup script. Link to trace query from Thermodynamic graph Add application selector in service view. Add order and status in trace query. Documentation Add architecture design doc. Reformat deploy document. Adjust Tomcat deploy document. Remove all Apache licenses files in dist release packages. Update user cases. Update UI licenses. Add incubating sections in doc. Issues and Pull requests\n5.0.0-beta UI -\u0026gt; Collector GraphQL query protocol Replace all tps to throughput/cpm(calls per min) Add getThermodynamic service Update version to beta Agent Changes Support TLS. Support namespace. Support direct link. Support token. Add across thread toolkit. Add new plugin extend machenism to override agent core implementations. Fix an agent start up sequence bug. Fix wrong gc count. Remove system env override. Add Spring AOP aspect patch to avoid aop conflicts. Collector Changes Trace query based on timeline. Delete JVM aggregation in second. Support TLS. Support namespace. Support token auth. Group and aggregate requests based on response time and timeline, support Thermodynamic chart query Support component librariy setting through yml file for better extendibility. Optimize performance. Support short column name in ES or other storage implementor. Add a new cache module implementor, based on Caffeine. Support system property override settings. Refactor settings initialization. Provide collector instrumentation agent. Support .NET core component libraries. Fix divide zero in query. Fix Data don't remove as expected in ES implementor. Add some checks in collector modulization core. Add some test cases. UI Changes New trace query UI. New Application UI, merge server tab(removed) into application as sub page. New Topology UI. New response time / throughput TopN list. Add Thermodynamic chart in overview page. Change all tps to cpm(calls per minutes). Fix wrong osName in server view. Fix wrong startTime in trace view. Fix some icons internet requirements. Documentation Add TLS document. Add namespace document. Add direct link document. Add token document. Add across thread toolkit document. Add a FAQ about, Agent or collector version upgrade. Sync all English document to Chinese. Issues and Pull requests\n5.0.0-alpha Agent -\u0026gt; Collector protocol Remove C++ keywords Move Ref into Span from Segment Add span type, when register an operation UI -\u0026gt; Collector GraphQL query protocol First version protocol Agent Changes Support gRPC 1.x plugin Support kafka 0.11 and 1.x plugin Support ServiceComb 0.x plugin Support optional plugin mechanism. Support Spring 3.x and 4.x bean annotation optional plugin Support Apache httpcomponent AsyncClient 4.x plugin Provide automatic agent daily tests, and release reports here. Refactor Postgresql, Oracle, MySQL plugin for compatible. Fix jetty client 9 plugin error Fix async APIs of okhttp plugin error Fix log config didn\u0026rsquo;t work Fix a class loader error in okhttp plugin Collector Changes Support metrics analysis and aggregation for application, application instance and service in minute, hour, day and month. Support new GraphQL query protocol Support alarm Provide a prototype instrument for collector. Support node speculate in cluster and application topology. (Provider Node -\u0026gt; Consumer Node) -\u0026gt; (Provider Node -\u0026gt; MQ Server -\u0026gt; Consumer Node) UI Changes New 5.0.0 UI!!! Issues and Pull requests\n","excerpt":"\u003ch2 id=\"510\"\u003e5.1.0\u003c/h2\u003e\n\u003ch4 id=\"agent-changes\"\u003eAgent Changes\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix spring inherit issue in another way\u003c/li\u003e\n\u003cli\u003eFix classloader dead lock in jdk7+ - …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/latest/en/changes/changes-5.x/","title":"5.1.0"},{"body":"5.1.0 Agent Changes Fix spring inherit issue in another way Fix classloader dead lock in jdk7+ - 5.x Support Spring mvc 5.x Support Spring webflux 5.x Collector Changes Fix too many open files. Fix the buffer file cannot delete. 5.0.0-GA Agent Changes Add several package names ignore in agent settings. Classes in these packages would be enhanced, even plugin declared. Support Undertow 2.x plugin. Fix wrong class names of Motan plugin, not a feature related issue, just naming. Collector Changes Make buffer file handler close more safety. Fix NPE in AlarmService Documentation Fix compiling doc link. Update new live demo address. 5.0.0-RC2 Agent Changes Support ActiveMQ 5.x Support RuntimeContext used out of TracingContext. Support Oracle ojdbc8 Plugin. Support ElasticSearch client transport 5.2-5.6 Plugin Support using agent.config with given path through system properties. Add a new way to transmit the Request and Response, to avoid bugs in Hytrix scenarios. Fix HTTPComponent client v4 operation name is empty. Fix 2 possible NPEs in Spring plugin. Fix a possible span leak in SpringMVC plugin. Fix NPE in Spring callback plugin. Collector Changes Add GZip support for Zipkin receiver. Add new component IDs for nodejs. Fix Zipkin span receiver may miss data in request. Optimize codes in heatmap calculation. Reduce unnecessary divide. Fix NPE in Alarm content generation. Fix the precision lost in ServiceNameService#startTimeMillis. Fix GC count is 0. Fix topology breaks when RPC client uses the async thread call. UI Changes Fix UI port can\u0026rsquo;t be set by startup script in Windows. Fix Topology self link error. Fix stack color mismatch label color in gc time chart. Documentation Add users list. Fix several document typo. Sync the Chinese documents. Add OpenAPM badge. Add icon/font documents to NOTICE files. Issues and Pull requests\n5.0.0-beta2 UI -\u0026gt; Collector GraphQL query protocol Add order and status in trace query. Agent Changes Add SOFA plugin. Add witness class for Kafka plugin. Add RuntimeContext in Context. Fix RuntimeContext fail in Tomcat plugin. Fix incompatible for getPropertyDescriptors in Spring core. Fix spymemcached plugin bug. Fix database URL parser bug. Fix StringIndexOutOfBoundsException when mysql jdbc url without databaseName。 Fix duplicate slash in Spring MVC plugin bug. Fix namespace bug. Fix NPE in Okhttp plugin when connect failed. FIx MalformedURLException in httpClientComponent plugin. Remove unused dependencies in Dubbo plugin. Remove gRPC timeout to avoid out of memory leak. Rewrite Async http client plugin. [Incubating] Add trace custom ignore optional plugin. Collector Changes Topology query optimization for more than 100 apps. Error rate alarm is not triggered. Tolerate unsupported segments. Support Integer Array, Long Array, String Array, Double Array in streaming data model. Support multiple entry span and multiple service name in one segment durtaion record. Use BulkProcessor to control the linear writing of data by multiple threads. Determine the log is enabled for the DEBUG level before printing message. Add static modifier to Logger. Add AspNet component. Filter inactive service in query. Support to query service based on Application. Fix RemoteDataMappingIdNotFoundException Exclude component-libaries.xml file in collector-*.jar, make sure it is in /conf only. Separate a single TTL in minute to in minute, hour, day, month metric and trace. Add order and status in trace query. Add folder lock to buffer folder. Modify operationName search from match to match_phrase. [Incubating] Add Zipkin span receiver. Support analysis Zipkin v1/v2 formats. [Incubating] Support sharding-sphere as storage implementor. UI Changes Support login and access control. Add new webapp.yml configuration file. Modify webapp startup script. Link to trace query from Thermodynamic graph Add application selector in service view. Add order and status in trace query. Documentation Add architecture design doc. Reformat deploy document. Adjust Tomcat deploy document. Remove all Apache licenses files in dist release packages. Update user cases. Update UI licenses. Add incubating sections in doc. Issues and Pull requests\n5.0.0-beta UI -\u0026gt; Collector GraphQL query protocol Replace all tps to throughput/cpm(calls per min) Add getThermodynamic service Update version to beta Agent Changes Support TLS. Support namespace. Support direct link. Support token. Add across thread toolkit. Add new plugin extend machenism to override agent core implementations. Fix an agent start up sequence bug. Fix wrong gc count. Remove system env override. Add Spring AOP aspect patch to avoid aop conflicts. Collector Changes Trace query based on timeline. Delete JVM aggregation in second. Support TLS. Support namespace. Support token auth. Group and aggregate requests based on response time and timeline, support Thermodynamic chart query Support component librariy setting through yml file for better extendibility. Optimize performance. Support short column name in ES or other storage implementor. Add a new cache module implementor, based on Caffeine. Support system property override settings. Refactor settings initialization. Provide collector instrumentation agent. Support .NET core component libraries. Fix divide zero in query. Fix Data don't remove as expected in ES implementor. Add some checks in collector modulization core. Add some test cases. UI Changes New trace query UI. New Application UI, merge server tab(removed) into application as sub page. New Topology UI. New response time / throughput TopN list. Add Thermodynamic chart in overview page. Change all tps to cpm(calls per minutes). Fix wrong osName in server view. Fix wrong startTime in trace view. Fix some icons internet requirements. Documentation Add TLS document. Add namespace document. Add direct link document. Add token document. Add across thread toolkit document. Add a FAQ about, Agent or collector version upgrade. Sync all English document to Chinese. Issues and Pull requests\n5.0.0-alpha Agent -\u0026gt; Collector protocol Remove C++ keywords Move Ref into Span from Segment Add span type, when register an operation UI -\u0026gt; Collector GraphQL query protocol First version protocol Agent Changes Support gRPC 1.x plugin Support kafka 0.11 and 1.x plugin Support ServiceComb 0.x plugin Support optional plugin mechanism. Support Spring 3.x and 4.x bean annotation optional plugin Support Apache httpcomponent AsyncClient 4.x plugin Provide automatic agent daily tests, and release reports here. Refactor Postgresql, Oracle, MySQL plugin for compatible. Fix jetty client 9 plugin error Fix async APIs of okhttp plugin error Fix log config didn\u0026rsquo;t work Fix a class loader error in okhttp plugin Collector Changes Support metrics analysis and aggregation for application, application instance and service in minute, hour, day and month. Support new GraphQL query protocol Support alarm Provide a prototype instrument for collector. Support node speculate in cluster and application topology. (Provider Node -\u0026gt; Consumer Node) -\u0026gt; (Provider Node -\u0026gt; MQ Server -\u0026gt; Consumer Node) UI Changes New 5.0.0 UI!!! Issues and Pull requests\n","excerpt":"\u003ch2 id=\"510\"\u003e5.1.0\u003c/h2\u003e\n\u003ch4 id=\"agent-changes\"\u003eAgent Changes\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix spring inherit issue in another way\u003c/li\u003e\n\u003cli\u003eFix classloader dead lock in jdk7+ - …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/next/en/changes/changes-5.x/","title":"5.1.0"},{"body":"5.1.0 Agent Changes Fix spring inherit issue in another way Fix classloader dead lock in jdk7+ - 5.x Support Spring mvc 5.x Support Spring webflux 5.x Collector Changes Fix too many open files. Fix the buffer file cannot delete. 5.0.0-GA Agent Changes Add several package names ignore in agent settings. Classes in these packages would be enhanced, even plugin declared. Support Undertow 2.x plugin. Fix wrong class names of Motan plugin, not a feature related issue, just naming. Collector Changes Make buffer file handler close more safety. Fix NPE in AlarmService Documentation Fix compiling doc link. Update new live demo address. 5.0.0-RC2 Agent Changes Support ActiveMQ 5.x Support RuntimeContext used out of TracingContext. Support Oracle ojdbc8 Plugin. Support ElasticSearch client transport 5.2-5.6 Plugin Support using agent.config with given path through system properties. Add a new way to transmit the Request and Response, to avoid bugs in Hytrix scenarios. Fix HTTPComponent client v4 operation name is empty. Fix 2 possible NPEs in Spring plugin. Fix a possible span leak in SpringMVC plugin. Fix NPE in Spring callback plugin. Collector Changes Add GZip support for Zipkin receiver. Add new component IDs for nodejs. Fix Zipkin span receiver may miss data in request. Optimize codes in heatmap calculation. Reduce unnecessary divide. Fix NPE in Alarm content generation. Fix the precision lost in ServiceNameService#startTimeMillis. Fix GC count is 0. Fix topology breaks when RPC client uses the async thread call. UI Changes Fix UI port can\u0026rsquo;t be set by startup script in Windows. Fix Topology self link error. Fix stack color mismatch label color in gc time chart. Documentation Add users list. Fix several document typo. Sync the Chinese documents. Add OpenAPM badge. Add icon/font documents to NOTICE files. Issues and Pull requests\n5.0.0-beta2 UI -\u0026gt; Collector GraphQL query protocol Add order and status in trace query. Agent Changes Add SOFA plugin. Add witness class for Kafka plugin. Add RuntimeContext in Context. Fix RuntimeContext fail in Tomcat plugin. Fix incompatible for getPropertyDescriptors in Spring core. Fix spymemcached plugin bug. Fix database URL parser bug. Fix StringIndexOutOfBoundsException when mysql jdbc url without databaseName。 Fix duplicate slash in Spring MVC plugin bug. Fix namespace bug. Fix NPE in Okhttp plugin when connect failed. FIx MalformedURLException in httpClientComponent plugin. Remove unused dependencies in Dubbo plugin. Remove gRPC timeout to avoid out of memory leak. Rewrite Async http client plugin. [Incubating] Add trace custom ignore optional plugin. Collector Changes Topology query optimization for more than 100 apps. Error rate alarm is not triggered. Tolerate unsupported segments. Support Integer Array, Long Array, String Array, Double Array in streaming data model. Support multiple entry span and multiple service name in one segment durtaion record. Use BulkProcessor to control the linear writing of data by multiple threads. Determine the log is enabled for the DEBUG level before printing message. Add static modifier to Logger. Add AspNet component. Filter inactive service in query. Support to query service based on Application. Fix RemoteDataMappingIdNotFoundException Exclude component-libaries.xml file in collector-*.jar, make sure it is in /conf only. Separate a single TTL in minute to in minute, hour, day, month metric and trace. Add order and status in trace query. Add folder lock to buffer folder. Modify operationName search from match to match_phrase. [Incubating] Add Zipkin span receiver. Support analysis Zipkin v1/v2 formats. [Incubating] Support sharding-sphere as storage implementor. UI Changes Support login and access control. Add new webapp.yml configuration file. Modify webapp startup script. Link to trace query from Thermodynamic graph Add application selector in service view. Add order and status in trace query. Documentation Add architecture design doc. Reformat deploy document. Adjust Tomcat deploy document. Remove all Apache licenses files in dist release packages. Update user cases. Update UI licenses. Add incubating sections in doc. Issues and Pull requests\n5.0.0-beta UI -\u0026gt; Collector GraphQL query protocol Replace all tps to throughput/cpm(calls per min) Add getThermodynamic service Update version to beta Agent Changes Support TLS. Support namespace. Support direct link. Support token. Add across thread toolkit. Add new plugin extend machenism to override agent core implementations. Fix an agent start up sequence bug. Fix wrong gc count. Remove system env override. Add Spring AOP aspect patch to avoid aop conflicts. Collector Changes Trace query based on timeline. Delete JVM aggregation in second. Support TLS. Support namespace. Support token auth. Group and aggregate requests based on response time and timeline, support Thermodynamic chart query Support component librariy setting through yml file for better extendibility. Optimize performance. Support short column name in ES or other storage implementor. Add a new cache module implementor, based on Caffeine. Support system property override settings. Refactor settings initialization. Provide collector instrumentation agent. Support .NET core component libraries. Fix divide zero in query. Fix Data don't remove as expected in ES implementor. Add some checks in collector modulization core. Add some test cases. UI Changes New trace query UI. New Application UI, merge server tab(removed) into application as sub page. New Topology UI. New response time / throughput TopN list. Add Thermodynamic chart in overview page. Change all tps to cpm(calls per minutes). Fix wrong osName in server view. Fix wrong startTime in trace view. Fix some icons internet requirements. Documentation Add TLS document. Add namespace document. Add direct link document. Add token document. Add across thread toolkit document. Add a FAQ about, Agent or collector version upgrade. Sync all English document to Chinese. Issues and Pull requests\n5.0.0-alpha Agent -\u0026gt; Collector protocol Remove C++ keywords Move Ref into Span from Segment Add span type, when register an operation UI -\u0026gt; Collector GraphQL query protocol First version protocol Agent Changes Support gRPC 1.x plugin Support kafka 0.11 and 1.x plugin Support ServiceComb 0.x plugin Support optional plugin mechanism. Support Spring 3.x and 4.x bean annotation optional plugin Support Apache httpcomponent AsyncClient 4.x plugin Provide automatic agent daily tests, and release reports here. Refactor Postgresql, Oracle, MySQL plugin for compatible. Fix jetty client 9 plugin error Fix async APIs of okhttp plugin error Fix log config didn\u0026rsquo;t work Fix a class loader error in okhttp plugin Collector Changes Support metrics analysis and aggregation for application, application instance and service in minute, hour, day and month. Support new GraphQL query protocol Support alarm Provide a prototype instrument for collector. Support node speculate in cluster and application topology. (Provider Node -\u0026gt; Consumer Node) -\u0026gt; (Provider Node -\u0026gt; MQ Server -\u0026gt; Consumer Node) UI Changes New 5.0.0 UI!!! Issues and Pull requests\n","excerpt":"\u003ch2 id=\"510\"\u003e5.1.0\u003c/h2\u003e\n\u003ch4 id=\"agent-changes\"\u003eAgent Changes\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix spring inherit issue in another way\u003c/li\u003e\n\u003cli\u003eFix classloader dead lock in jdk7+ - …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.0.0/en/changes/changes-5.x/","title":"5.1.0"},{"body":"5.1.0 Agent Changes Fix spring inherit issue in another way Fix classloader dead lock in jdk7+ - 5.x Support Spring mvc 5.x Support Spring webflux 5.x Collector Changes Fix too many open files. Fix the buffer file cannot delete. 5.0.0-GA Agent Changes Add several package names ignore in agent settings. Classes in these packages would be enhanced, even plugin declared. Support Undertow 2.x plugin. Fix wrong class names of Motan plugin, not a feature related issue, just naming. Collector Changes Make buffer file handler close more safety. Fix NPE in AlarmService Documentation Fix compiling doc link. Update new live demo address. 5.0.0-RC2 Agent Changes Support ActiveMQ 5.x Support RuntimeContext used out of TracingContext. Support Oracle ojdbc8 Plugin. Support ElasticSearch client transport 5.2-5.6 Plugin Support using agent.config with given path through system properties. Add a new way to transmit the Request and Response, to avoid bugs in Hytrix scenarios. Fix HTTPComponent client v4 operation name is empty. Fix 2 possible NPEs in Spring plugin. Fix a possible span leak in SpringMVC plugin. Fix NPE in Spring callback plugin. Collector Changes Add GZip support for Zipkin receiver. Add new component IDs for nodejs. Fix Zipkin span receiver may miss data in request. Optimize codes in heatmap calculation. Reduce unnecessary divide. Fix NPE in Alarm content generation. Fix the precision lost in ServiceNameService#startTimeMillis. Fix GC count is 0. Fix topology breaks when RPC client uses the async thread call. UI Changes Fix UI port can\u0026rsquo;t be set by startup script in Windows. Fix Topology self link error. Fix stack color mismatch label color in gc time chart. Documentation Add users list. Fix several document typo. Sync the Chinese documents. Add OpenAPM badge. Add icon/font documents to NOTICE files. Issues and Pull requests\n5.0.0-beta2 UI -\u0026gt; Collector GraphQL query protocol Add order and status in trace query. Agent Changes Add SOFA plugin. Add witness class for Kafka plugin. Add RuntimeContext in Context. Fix RuntimeContext fail in Tomcat plugin. Fix incompatible for getPropertyDescriptors in Spring core. Fix spymemcached plugin bug. Fix database URL parser bug. Fix StringIndexOutOfBoundsException when mysql jdbc url without databaseName。 Fix duplicate slash in Spring MVC plugin bug. Fix namespace bug. Fix NPE in Okhttp plugin when connect failed. FIx MalformedURLException in httpClientComponent plugin. Remove unused dependencies in Dubbo plugin. Remove gRPC timeout to avoid out of memory leak. Rewrite Async http client plugin. [Incubating] Add trace custom ignore optional plugin. Collector Changes Topology query optimization for more than 100 apps. Error rate alarm is not triggered. Tolerate unsupported segments. Support Integer Array, Long Array, String Array, Double Array in streaming data model. Support multiple entry span and multiple service name in one segment durtaion record. Use BulkProcessor to control the linear writing of data by multiple threads. Determine the log is enabled for the DEBUG level before printing message. Add static modifier to Logger. Add AspNet component. Filter inactive service in query. Support to query service based on Application. Fix RemoteDataMappingIdNotFoundException Exclude component-libaries.xml file in collector-*.jar, make sure it is in /conf only. Separate a single TTL in minute to in minute, hour, day, month metric and trace. Add order and status in trace query. Add folder lock to buffer folder. Modify operationName search from match to match_phrase. [Incubating] Add Zipkin span receiver. Support analysis Zipkin v1/v2 formats. [Incubating] Support sharding-sphere as storage implementor. UI Changes Support login and access control. Add new webapp.yml configuration file. Modify webapp startup script. Link to trace query from Thermodynamic graph Add application selector in service view. Add order and status in trace query. Documentation Add architecture design doc. Reformat deploy document. Adjust Tomcat deploy document. Remove all Apache licenses files in dist release packages. Update user cases. Update UI licenses. Add incubating sections in doc. Issues and Pull requests\n5.0.0-beta UI -\u0026gt; Collector GraphQL query protocol Replace all tps to throughput/cpm(calls per min) Add getThermodynamic service Update version to beta Agent Changes Support TLS. Support namespace. Support direct link. Support token. Add across thread toolkit. Add new plugin extend machenism to override agent core implementations. Fix an agent start up sequence bug. Fix wrong gc count. Remove system env override. Add Spring AOP aspect patch to avoid aop conflicts. Collector Changes Trace query based on timeline. Delete JVM aggregation in second. Support TLS. Support namespace. Support token auth. Group and aggregate requests based on response time and timeline, support Thermodynamic chart query Support component librariy setting through yml file for better extendibility. Optimize performance. Support short column name in ES or other storage implementor. Add a new cache module implementor, based on Caffeine. Support system property override settings. Refactor settings initialization. Provide collector instrumentation agent. Support .NET core component libraries. Fix divide zero in query. Fix Data don't remove as expected in ES implementor. Add some checks in collector modulization core. Add some test cases. UI Changes New trace query UI. New Application UI, merge server tab(removed) into application as sub page. New Topology UI. New response time / throughput TopN list. Add Thermodynamic chart in overview page. Change all tps to cpm(calls per minutes). Fix wrong osName in server view. Fix wrong startTime in trace view. Fix some icons internet requirements. Documentation Add TLS document. Add namespace document. Add direct link document. Add token document. Add across thread toolkit document. Add a FAQ about, Agent or collector version upgrade. Sync all English document to Chinese. Issues and Pull requests\n5.0.0-alpha Agent -\u0026gt; Collector protocol Remove C++ keywords Move Ref into Span from Segment Add span type, when register an operation UI -\u0026gt; Collector GraphQL query protocol First version protocol Agent Changes Support gRPC 1.x plugin Support kafka 0.11 and 1.x plugin Support ServiceComb 0.x plugin Support optional plugin mechanism. Support Spring 3.x and 4.x bean annotation optional plugin Support Apache httpcomponent AsyncClient 4.x plugin Provide automatic agent daily tests, and release reports here. Refactor Postgresql, Oracle, MySQL plugin for compatible. Fix jetty client 9 plugin error Fix async APIs of okhttp plugin error Fix log config didn\u0026rsquo;t work Fix a class loader error in okhttp plugin Collector Changes Support metrics analysis and aggregation for application, application instance and service in minute, hour, day and month. Support new GraphQL query protocol Support alarm Provide a prototype instrument for collector. Support node speculate in cluster and application topology. (Provider Node -\u0026gt; Consumer Node) -\u0026gt; (Provider Node -\u0026gt; MQ Server -\u0026gt; Consumer Node) UI Changes New 5.0.0 UI!!! Issues and Pull requests\n","excerpt":"\u003ch2 id=\"510\"\u003e5.1.0\u003c/h2\u003e\n\u003ch4 id=\"agent-changes\"\u003eAgent Changes\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix spring inherit issue in another way\u003c/li\u003e\n\u003cli\u003eFix classloader dead lock in jdk7+ - …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.0.1/en/changes/changes-5.x/","title":"5.1.0"},{"body":"5.1.0 Agent Changes Fix spring inherit issue in another way Fix classloader dead lock in jdk7+ - 5.x Support Spring mvc 5.x Support Spring webflux 5.x Collector Changes Fix too many open files. Fix the buffer file cannot delete. 5.0.0-GA Agent Changes Add several package names ignore in agent settings. Classes in these packages would be enhanced, even plugin declared. Support Undertow 2.x plugin. Fix wrong class names of Motan plugin, not a feature related issue, just naming. Collector Changes Make buffer file handler close more safety. Fix NPE in AlarmService Documentation Fix compiling doc link. Update new live demo address. 5.0.0-RC2 Agent Changes Support ActiveMQ 5.x Support RuntimeContext used out of TracingContext. Support Oracle ojdbc8 Plugin. Support ElasticSearch client transport 5.2-5.6 Plugin Support using agent.config with given path through system properties. Add a new way to transmit the Request and Response, to avoid bugs in Hytrix scenarios. Fix HTTPComponent client v4 operation name is empty. Fix 2 possible NPEs in Spring plugin. Fix a possible span leak in SpringMVC plugin. Fix NPE in Spring callback plugin. Collector Changes Add GZip support for Zipkin receiver. Add new component IDs for nodejs. Fix Zipkin span receiver may miss data in request. Optimize codes in heatmap calculation. Reduce unnecessary divide. Fix NPE in Alarm content generation. Fix the precision lost in ServiceNameService#startTimeMillis. Fix GC count is 0. Fix topology breaks when RPC client uses the async thread call. UI Changes Fix UI port can\u0026rsquo;t be set by startup script in Windows. Fix Topology self link error. Fix stack color mismatch label color in gc time chart. Documentation Add users list. Fix several document typo. Sync the Chinese documents. Add OpenAPM badge. Add icon/font documents to NOTICE files. Issues and Pull requests\n5.0.0-beta2 UI -\u0026gt; Collector GraphQL query protocol Add order and status in trace query. Agent Changes Add SOFA plugin. Add witness class for Kafka plugin. Add RuntimeContext in Context. Fix RuntimeContext fail in Tomcat plugin. Fix incompatible for getPropertyDescriptors in Spring core. Fix spymemcached plugin bug. Fix database URL parser bug. Fix StringIndexOutOfBoundsException when mysql jdbc url without databaseName。 Fix duplicate slash in Spring MVC plugin bug. Fix namespace bug. Fix NPE in Okhttp plugin when connect failed. FIx MalformedURLException in httpClientComponent plugin. Remove unused dependencies in Dubbo plugin. Remove gRPC timeout to avoid out of memory leak. Rewrite Async http client plugin. [Incubating] Add trace custom ignore optional plugin. Collector Changes Topology query optimization for more than 100 apps. Error rate alarm is not triggered. Tolerate unsupported segments. Support Integer Array, Long Array, String Array, Double Array in streaming data model. Support multiple entry span and multiple service name in one segment durtaion record. Use BulkProcessor to control the linear writing of data by multiple threads. Determine the log is enabled for the DEBUG level before printing message. Add static modifier to Logger. Add AspNet component. Filter inactive service in query. Support to query service based on Application. Fix RemoteDataMappingIdNotFoundException Exclude component-libaries.xml file in collector-*.jar, make sure it is in /conf only. Separate a single TTL in minute to in minute, hour, day, month metric and trace. Add order and status in trace query. Add folder lock to buffer folder. Modify operationName search from match to match_phrase. [Incubating] Add Zipkin span receiver. Support analysis Zipkin v1/v2 formats. [Incubating] Support sharding-sphere as storage implementor. UI Changes Support login and access control. Add new webapp.yml configuration file. Modify webapp startup script. Link to trace query from Thermodynamic graph Add application selector in service view. Add order and status in trace query. Documentation Add architecture design doc. Reformat deploy document. Adjust Tomcat deploy document. Remove all Apache licenses files in dist release packages. Update user cases. Update UI licenses. Add incubating sections in doc. Issues and Pull requests\n5.0.0-beta UI -\u0026gt; Collector GraphQL query protocol Replace all tps to throughput/cpm(calls per min) Add getThermodynamic service Update version to beta Agent Changes Support TLS. Support namespace. Support direct link. Support token. Add across thread toolkit. Add new plugin extend machenism to override agent core implementations. Fix an agent start up sequence bug. Fix wrong gc count. Remove system env override. Add Spring AOP aspect patch to avoid aop conflicts. Collector Changes Trace query based on timeline. Delete JVM aggregation in second. Support TLS. Support namespace. Support token auth. Group and aggregate requests based on response time and timeline, support Thermodynamic chart query Support component librariy setting through yml file for better extendibility. Optimize performance. Support short column name in ES or other storage implementor. Add a new cache module implementor, based on Caffeine. Support system property override settings. Refactor settings initialization. Provide collector instrumentation agent. Support .NET core component libraries. Fix divide zero in query. Fix Data don't remove as expected in ES implementor. Add some checks in collector modulization core. Add some test cases. UI Changes New trace query UI. New Application UI, merge server tab(removed) into application as sub page. New Topology UI. New response time / throughput TopN list. Add Thermodynamic chart in overview page. Change all tps to cpm(calls per minutes). Fix wrong osName in server view. Fix wrong startTime in trace view. Fix some icons internet requirements. Documentation Add TLS document. Add namespace document. Add direct link document. Add token document. Add across thread toolkit document. Add a FAQ about, Agent or collector version upgrade. Sync all English document to Chinese. Issues and Pull requests\n5.0.0-alpha Agent -\u0026gt; Collector protocol Remove C++ keywords Move Ref into Span from Segment Add span type, when register an operation UI -\u0026gt; Collector GraphQL query protocol First version protocol Agent Changes Support gRPC 1.x plugin Support kafka 0.11 and 1.x plugin Support ServiceComb 0.x plugin Support optional plugin mechanism. Support Spring 3.x and 4.x bean annotation optional plugin Support Apache httpcomponent AsyncClient 4.x plugin Provide automatic agent daily tests, and release reports here. Refactor Postgresql, Oracle, MySQL plugin for compatible. Fix jetty client 9 plugin error Fix async APIs of okhttp plugin error Fix log config didn\u0026rsquo;t work Fix a class loader error in okhttp plugin Collector Changes Support metrics analysis and aggregation for application, application instance and service in minute, hour, day and month. Support new GraphQL query protocol Support alarm Provide a prototype instrument for collector. Support node speculate in cluster and application topology. (Provider Node -\u0026gt; Consumer Node) -\u0026gt; (Provider Node -\u0026gt; MQ Server -\u0026gt; Consumer Node) UI Changes New 5.0.0 UI!!! Issues and Pull requests\n","excerpt":"\u003ch2 id=\"510\"\u003e5.1.0\u003c/h2\u003e\n\u003ch4 id=\"agent-changes\"\u003eAgent Changes\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix spring inherit issue in another way\u003c/li\u003e\n\u003cli\u003eFix classloader dead lock in jdk7+ - …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.1.0/en/changes/changes-5.x/","title":"5.1.0"},{"body":"5.1.0 Agent Changes Fix spring inherit issue in another way Fix classloader dead lock in jdk7+ - 5.x Support Spring mvc 5.x Support Spring webflux 5.x Collector Changes Fix too many open files. Fix the buffer file cannot delete. 5.0.0-GA Agent Changes Add several package names ignore in agent settings. Classes in these packages would be enhanced, even plugin declared. Support Undertow 2.x plugin. Fix wrong class names of Motan plugin, not a feature related issue, just naming. Collector Changes Make buffer file handler close more safety. Fix NPE in AlarmService Documentation Fix compiling doc link. Update new live demo address. 5.0.0-RC2 Agent Changes Support ActiveMQ 5.x Support RuntimeContext used out of TracingContext. Support Oracle ojdbc8 Plugin. Support ElasticSearch client transport 5.2-5.6 Plugin Support using agent.config with given path through system properties. Add a new way to transmit the Request and Response, to avoid bugs in Hytrix scenarios. Fix HTTPComponent client v4 operation name is empty. Fix 2 possible NPEs in Spring plugin. Fix a possible span leak in SpringMVC plugin. Fix NPE in Spring callback plugin. Collector Changes Add GZip support for Zipkin receiver. Add new component IDs for nodejs. Fix Zipkin span receiver may miss data in request. Optimize codes in heatmap calculation. Reduce unnecessary divide. Fix NPE in Alarm content generation. Fix the precision lost in ServiceNameService#startTimeMillis. Fix GC count is 0. Fix topology breaks when RPC client uses the async thread call. UI Changes Fix UI port can\u0026rsquo;t be set by startup script in Windows. Fix Topology self link error. Fix stack color mismatch label color in gc time chart. Documentation Add users list. Fix several document typo. Sync the Chinese documents. Add OpenAPM badge. Add icon/font documents to NOTICE files. Issues and Pull requests\n5.0.0-beta2 UI -\u0026gt; Collector GraphQL query protocol Add order and status in trace query. Agent Changes Add SOFA plugin. Add witness class for Kafka plugin. Add RuntimeContext in Context. Fix RuntimeContext fail in Tomcat plugin. Fix incompatible for getPropertyDescriptors in Spring core. Fix spymemcached plugin bug. Fix database URL parser bug. Fix StringIndexOutOfBoundsException when mysql jdbc url without databaseName。 Fix duplicate slash in Spring MVC plugin bug. Fix namespace bug. Fix NPE in Okhttp plugin when connect failed. FIx MalformedURLException in httpClientComponent plugin. Remove unused dependencies in Dubbo plugin. Remove gRPC timeout to avoid out of memory leak. Rewrite Async http client plugin. [Incubating] Add trace custom ignore optional plugin. Collector Changes Topology query optimization for more than 100 apps. Error rate alarm is not triggered. Tolerate unsupported segments. Support Integer Array, Long Array, String Array, Double Array in streaming data model. Support multiple entry span and multiple service name in one segment durtaion record. Use BulkProcessor to control the linear writing of data by multiple threads. Determine the log is enabled for the DEBUG level before printing message. Add static modifier to Logger. Add AspNet component. Filter inactive service in query. Support to query service based on Application. Fix RemoteDataMappingIdNotFoundException Exclude component-libaries.xml file in collector-*.jar, make sure it is in /conf only. Separate a single TTL in minute to in minute, hour, day, month metric and trace. Add order and status in trace query. Add folder lock to buffer folder. Modify operationName search from match to match_phrase. [Incubating] Add Zipkin span receiver. Support analysis Zipkin v1/v2 formats. [Incubating] Support sharding-sphere as storage implementor. UI Changes Support login and access control. Add new webapp.yml configuration file. Modify webapp startup script. Link to trace query from Thermodynamic graph Add application selector in service view. Add order and status in trace query. Documentation Add architecture design doc. Reformat deploy document. Adjust Tomcat deploy document. Remove all Apache licenses files in dist release packages. Update user cases. Update UI licenses. Add incubating sections in doc. Issues and Pull requests\n5.0.0-beta UI -\u0026gt; Collector GraphQL query protocol Replace all tps to throughput/cpm(calls per min) Add getThermodynamic service Update version to beta Agent Changes Support TLS. Support namespace. Support direct link. Support token. Add across thread toolkit. Add new plugin extend machenism to override agent core implementations. Fix an agent start up sequence bug. Fix wrong gc count. Remove system env override. Add Spring AOP aspect patch to avoid aop conflicts. Collector Changes Trace query based on timeline. Delete JVM aggregation in second. Support TLS. Support namespace. Support token auth. Group and aggregate requests based on response time and timeline, support Thermodynamic chart query Support component librariy setting through yml file for better extendibility. Optimize performance. Support short column name in ES or other storage implementor. Add a new cache module implementor, based on Caffeine. Support system property override settings. Refactor settings initialization. Provide collector instrumentation agent. Support .NET core component libraries. Fix divide zero in query. Fix Data don't remove as expected in ES implementor. Add some checks in collector modulization core. Add some test cases. UI Changes New trace query UI. New Application UI, merge server tab(removed) into application as sub page. New Topology UI. New response time / throughput TopN list. Add Thermodynamic chart in overview page. Change all tps to cpm(calls per minutes). Fix wrong osName in server view. Fix wrong startTime in trace view. Fix some icons internet requirements. Documentation Add TLS document. Add namespace document. Add direct link document. Add token document. Add across thread toolkit document. Add a FAQ about, Agent or collector version upgrade. Sync all English document to Chinese. Issues and Pull requests\n5.0.0-alpha Agent -\u0026gt; Collector protocol Remove C++ keywords Move Ref into Span from Segment Add span type, when register an operation UI -\u0026gt; Collector GraphQL query protocol First version protocol Agent Changes Support gRPC 1.x plugin Support kafka 0.11 and 1.x plugin Support ServiceComb 0.x plugin Support optional plugin mechanism. Support Spring 3.x and 4.x bean annotation optional plugin Support Apache httpcomponent AsyncClient 4.x plugin Provide automatic agent daily tests, and release reports here. Refactor Postgresql, Oracle, MySQL plugin for compatible. Fix jetty client 9 plugin error Fix async APIs of okhttp plugin error Fix log config didn\u0026rsquo;t work Fix a class loader error in okhttp plugin Collector Changes Support metrics analysis and aggregation for application, application instance and service in minute, hour, day and month. Support new GraphQL query protocol Support alarm Provide a prototype instrument for collector. Support node speculate in cluster and application topology. (Provider Node -\u0026gt; Consumer Node) -\u0026gt; (Provider Node -\u0026gt; MQ Server -\u0026gt; Consumer Node) UI Changes New 5.0.0 UI!!! Issues and Pull requests\n","excerpt":"\u003ch2 id=\"510\"\u003e5.1.0\u003c/h2\u003e\n\u003ch4 id=\"agent-changes\"\u003eAgent Changes\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix spring inherit issue in another way\u003c/li\u003e\n\u003cli\u003eFix classloader dead lock in jdk7+ - …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.2.0/en/changes/changes-5.x/","title":"5.1.0"},{"body":"5.1.0 Agent Changes Fix spring inherit issue in another way Fix classloader dead lock in jdk7+ - 5.x Support Spring mvc 5.x Support Spring webflux 5.x Collector Changes Fix too many open files. Fix the buffer file cannot delete. 5.0.0-GA Agent Changes Add several package names ignore in agent settings. Classes in these packages would be enhanced, even plugin declared. Support Undertow 2.x plugin. Fix wrong class names of Motan plugin, not a feature related issue, just naming. Collector Changes Make buffer file handler close more safety. Fix NPE in AlarmService Documentation Fix compiling doc link. Update new live demo address. 5.0.0-RC2 Agent Changes Support ActiveMQ 5.x Support RuntimeContext used out of TracingContext. Support Oracle ojdbc8 Plugin. Support ElasticSearch client transport 5.2-5.6 Plugin Support using agent.config with given path through system properties. Add a new way to transmit the Request and Response, to avoid bugs in Hytrix scenarios. Fix HTTPComponent client v4 operation name is empty. Fix 2 possible NPEs in Spring plugin. Fix a possible span leak in SpringMVC plugin. Fix NPE in Spring callback plugin. Collector Changes Add GZip support for Zipkin receiver. Add new component IDs for nodejs. Fix Zipkin span receiver may miss data in request. Optimize codes in heatmap calculation. Reduce unnecessary divide. Fix NPE in Alarm content generation. Fix the precision lost in ServiceNameService#startTimeMillis. Fix GC count is 0. Fix topology breaks when RPC client uses the async thread call. UI Changes Fix UI port can\u0026rsquo;t be set by startup script in Windows. Fix Topology self link error. Fix stack color mismatch label color in gc time chart. Documentation Add users list. Fix several document typo. Sync the Chinese documents. Add OpenAPM badge. Add icon/font documents to NOTICE files. Issues and Pull requests\n5.0.0-beta2 UI -\u0026gt; Collector GraphQL query protocol Add order and status in trace query. Agent Changes Add SOFA plugin. Add witness class for Kafka plugin. Add RuntimeContext in Context. Fix RuntimeContext fail in Tomcat plugin. Fix incompatible for getPropertyDescriptors in Spring core. Fix spymemcached plugin bug. Fix database URL parser bug. Fix StringIndexOutOfBoundsException when mysql jdbc url without databaseName。 Fix duplicate slash in Spring MVC plugin bug. Fix namespace bug. Fix NPE in Okhttp plugin when connect failed. FIx MalformedURLException in httpClientComponent plugin. Remove unused dependencies in Dubbo plugin. Remove gRPC timeout to avoid out of memory leak. Rewrite Async http client plugin. [Incubating] Add trace custom ignore optional plugin. Collector Changes Topology query optimization for more than 100 apps. Error rate alarm is not triggered. Tolerate unsupported segments. Support Integer Array, Long Array, String Array, Double Array in streaming data model. Support multiple entry span and multiple service name in one segment durtaion record. Use BulkProcessor to control the linear writing of data by multiple threads. Determine the log is enabled for the DEBUG level before printing message. Add static modifier to Logger. Add AspNet component. Filter inactive service in query. Support to query service based on Application. Fix RemoteDataMappingIdNotFoundException Exclude component-libaries.xml file in collector-*.jar, make sure it is in /conf only. Separate a single TTL in minute to in minute, hour, day, month metric and trace. Add order and status in trace query. Add folder lock to buffer folder. Modify operationName search from match to match_phrase. [Incubating] Add Zipkin span receiver. Support analysis Zipkin v1/v2 formats. [Incubating] Support sharding-sphere as storage implementor. UI Changes Support login and access control. Add new webapp.yml configuration file. Modify webapp startup script. Link to trace query from Thermodynamic graph Add application selector in service view. Add order and status in trace query. Documentation Add architecture design doc. Reformat deploy document. Adjust Tomcat deploy document. Remove all Apache licenses files in dist release packages. Update user cases. Update UI licenses. Add incubating sections in doc. Issues and Pull requests\n5.0.0-beta UI -\u0026gt; Collector GraphQL query protocol Replace all tps to throughput/cpm(calls per min) Add getThermodynamic service Update version to beta Agent Changes Support TLS. Support namespace. Support direct link. Support token. Add across thread toolkit. Add new plugin extend machenism to override agent core implementations. Fix an agent start up sequence bug. Fix wrong gc count. Remove system env override. Add Spring AOP aspect patch to avoid aop conflicts. Collector Changes Trace query based on timeline. Delete JVM aggregation in second. Support TLS. Support namespace. Support token auth. Group and aggregate requests based on response time and timeline, support Thermodynamic chart query Support component librariy setting through yml file for better extendibility. Optimize performance. Support short column name in ES or other storage implementor. Add a new cache module implementor, based on Caffeine. Support system property override settings. Refactor settings initialization. Provide collector instrumentation agent. Support .NET core component libraries. Fix divide zero in query. Fix Data don't remove as expected in ES implementor. Add some checks in collector modulization core. Add some test cases. UI Changes New trace query UI. New Application UI, merge server tab(removed) into application as sub page. New Topology UI. New response time / throughput TopN list. Add Thermodynamic chart in overview page. Change all tps to cpm(calls per minutes). Fix wrong osName in server view. Fix wrong startTime in trace view. Fix some icons internet requirements. Documentation Add TLS document. Add namespace document. Add direct link document. Add token document. Add across thread toolkit document. Add a FAQ about, Agent or collector version upgrade. Sync all English document to Chinese. Issues and Pull requests\n5.0.0-alpha Agent -\u0026gt; Collector protocol Remove C++ keywords Move Ref into Span from Segment Add span type, when register an operation UI -\u0026gt; Collector GraphQL query protocol First version protocol Agent Changes Support gRPC 1.x plugin Support kafka 0.11 and 1.x plugin Support ServiceComb 0.x plugin Support optional plugin mechanism. Support Spring 3.x and 4.x bean annotation optional plugin Support Apache httpcomponent AsyncClient 4.x plugin Provide automatic agent daily tests, and release reports here. Refactor Postgresql, Oracle, MySQL plugin for compatible. Fix jetty client 9 plugin error Fix async APIs of okhttp plugin error Fix log config didn\u0026rsquo;t work Fix a class loader error in okhttp plugin Collector Changes Support metrics analysis and aggregation for application, application instance and service in minute, hour, day and month. Support new GraphQL query protocol Support alarm Provide a prototype instrument for collector. Support node speculate in cluster and application topology. (Provider Node -\u0026gt; Consumer Node) -\u0026gt; (Provider Node -\u0026gt; MQ Server -\u0026gt; Consumer Node) UI Changes New 5.0.0 UI!!! Issues and Pull requests\n","excerpt":"\u003ch2 id=\"510\"\u003e5.1.0\u003c/h2\u003e\n\u003ch4 id=\"agent-changes\"\u003eAgent Changes\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix spring inherit issue in another way\u003c/li\u003e\n\u003cli\u003eFix classloader dead lock in jdk7+ - …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.3.0/en/changes/changes-5.x/","title":"5.1.0"},{"body":"5.1.0 Agent Changes Fix spring inherit issue in another way Fix classloader dead lock in jdk7+ - 5.x Support Spring mvc 5.x Support Spring webflux 5.x Collector Changes Fix too many open files. Fix the buffer file cannot delete. 5.0.0-GA Agent Changes Add several package names ignore in agent settings. Classes in these packages would be enhanced, even plugin declared. Support Undertow 2.x plugin. Fix wrong class names of Motan plugin, not a feature related issue, just naming. Collector Changes Make buffer file handler close more safety. Fix NPE in AlarmService Documentation Fix compiling doc link. Update new live demo address. 5.0.0-RC2 Agent Changes Support ActiveMQ 5.x Support RuntimeContext used out of TracingContext. Support Oracle ojdbc8 Plugin. Support ElasticSearch client transport 5.2-5.6 Plugin Support using agent.config with given path through system properties. Add a new way to transmit the Request and Response, to avoid bugs in Hytrix scenarios. Fix HTTPComponent client v4 operation name is empty. Fix 2 possible NPEs in Spring plugin. Fix a possible span leak in SpringMVC plugin. Fix NPE in Spring callback plugin. Collector Changes Add GZip support for Zipkin receiver. Add new component IDs for nodejs. Fix Zipkin span receiver may miss data in request. Optimize codes in heatmap calculation. Reduce unnecessary divide. Fix NPE in Alarm content generation. Fix the precision lost in ServiceNameService#startTimeMillis. Fix GC count is 0. Fix topology breaks when RPC client uses the async thread call. UI Changes Fix UI port can\u0026rsquo;t be set by startup script in Windows. Fix Topology self link error. Fix stack color mismatch label color in gc time chart. Documentation Add users list. Fix several document typo. Sync the Chinese documents. Add OpenAPM badge. Add icon/font documents to NOTICE files. Issues and Pull requests\n5.0.0-beta2 UI -\u0026gt; Collector GraphQL query protocol Add order and status in trace query. Agent Changes Add SOFA plugin. Add witness class for Kafka plugin. Add RuntimeContext in Context. Fix RuntimeContext fail in Tomcat plugin. Fix incompatible for getPropertyDescriptors in Spring core. Fix spymemcached plugin bug. Fix database URL parser bug. Fix StringIndexOutOfBoundsException when mysql jdbc url without databaseName。 Fix duplicate slash in Spring MVC plugin bug. Fix namespace bug. Fix NPE in Okhttp plugin when connect failed. FIx MalformedURLException in httpClientComponent plugin. Remove unused dependencies in Dubbo plugin. Remove gRPC timeout to avoid out of memory leak. Rewrite Async http client plugin. [Incubating] Add trace custom ignore optional plugin. Collector Changes Topology query optimization for more than 100 apps. Error rate alarm is not triggered. Tolerate unsupported segments. Support Integer Array, Long Array, String Array, Double Array in streaming data model. Support multiple entry span and multiple service name in one segment durtaion record. Use BulkProcessor to control the linear writing of data by multiple threads. Determine the log is enabled for the DEBUG level before printing message. Add static modifier to Logger. Add AspNet component. Filter inactive service in query. Support to query service based on Application. Fix RemoteDataMappingIdNotFoundException Exclude component-libaries.xml file in collector-*.jar, make sure it is in /conf only. Separate a single TTL in minute to in minute, hour, day, month metric and trace. Add order and status in trace query. Add folder lock to buffer folder. Modify operationName search from match to match_phrase. [Incubating] Add Zipkin span receiver. Support analysis Zipkin v1/v2 formats. [Incubating] Support sharding-sphere as storage implementor. UI Changes Support login and access control. Add new webapp.yml configuration file. Modify webapp startup script. Link to trace query from Thermodynamic graph Add application selector in service view. Add order and status in trace query. Documentation Add architecture design doc. Reformat deploy document. Adjust Tomcat deploy document. Remove all Apache licenses files in dist release packages. Update user cases. Update UI licenses. Add incubating sections in doc. Issues and Pull requests\n5.0.0-beta UI -\u0026gt; Collector GraphQL query protocol Replace all tps to throughput/cpm(calls per min) Add getThermodynamic service Update version to beta Agent Changes Support TLS. Support namespace. Support direct link. Support token. Add across thread toolkit. Add new plugin extend machenism to override agent core implementations. Fix an agent start up sequence bug. Fix wrong gc count. Remove system env override. Add Spring AOP aspect patch to avoid aop conflicts. Collector Changes Trace query based on timeline. Delete JVM aggregation in second. Support TLS. Support namespace. Support token auth. Group and aggregate requests based on response time and timeline, support Thermodynamic chart query Support component librariy setting through yml file for better extendibility. Optimize performance. Support short column name in ES or other storage implementor. Add a new cache module implementor, based on Caffeine. Support system property override settings. Refactor settings initialization. Provide collector instrumentation agent. Support .NET core component libraries. Fix divide zero in query. Fix Data don't remove as expected in ES implementor. Add some checks in collector modulization core. Add some test cases. UI Changes New trace query UI. New Application UI, merge server tab(removed) into application as sub page. New Topology UI. New response time / throughput TopN list. Add Thermodynamic chart in overview page. Change all tps to cpm(calls per minutes). Fix wrong osName in server view. Fix wrong startTime in trace view. Fix some icons internet requirements. Documentation Add TLS document. Add namespace document. Add direct link document. Add token document. Add across thread toolkit document. Add a FAQ about, Agent or collector version upgrade. Sync all English document to Chinese. Issues and Pull requests\n5.0.0-alpha Agent -\u0026gt; Collector protocol Remove C++ keywords Move Ref into Span from Segment Add span type, when register an operation UI -\u0026gt; Collector GraphQL query protocol First version protocol Agent Changes Support gRPC 1.x plugin Support kafka 0.11 and 1.x plugin Support ServiceComb 0.x plugin Support optional plugin mechanism. Support Spring 3.x and 4.x bean annotation optional plugin Support Apache httpcomponent AsyncClient 4.x plugin Provide automatic agent daily tests, and release reports here. Refactor Postgresql, Oracle, MySQL plugin for compatible. Fix jetty client 9 plugin error Fix async APIs of okhttp plugin error Fix log config didn\u0026rsquo;t work Fix a class loader error in okhttp plugin Collector Changes Support metrics analysis and aggregation for application, application instance and service in minute, hour, day and month. Support new GraphQL query protocol Support alarm Provide a prototype instrument for collector. Support node speculate in cluster and application topology. (Provider Node -\u0026gt; Consumer Node) -\u0026gt; (Provider Node -\u0026gt; MQ Server -\u0026gt; Consumer Node) UI Changes New 5.0.0 UI!!! Issues and Pull requests\n","excerpt":"\u003ch2 id=\"510\"\u003e5.1.0\u003c/h2\u003e\n\u003ch4 id=\"agent-changes\"\u003eAgent Changes\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix spring inherit issue in another way\u003c/li\u003e\n\u003cli\u003eFix classloader dead lock in jdk7+ - …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.4.0/en/changes/changes-5.x/","title":"5.1.0"},{"body":"5.1.0 Agent Changes Fix spring inherit issue in another way Fix classloader dead lock in jdk7+ - 5.x Support Spring mvc 5.x Support Spring webflux 5.x Collector Changes Fix too many open files. Fix the buffer file cannot delete. 5.0.0-GA Agent Changes Add several package names ignore in agent settings. Classes in these packages would be enhanced, even plugin declared. Support Undertow 2.x plugin. Fix wrong class names of Motan plugin, not a feature related issue, just naming. Collector Changes Make buffer file handler close more safety. Fix NPE in AlarmService Documentation Fix compiling doc link. Update new live demo address. 5.0.0-RC2 Agent Changes Support ActiveMQ 5.x Support RuntimeContext used out of TracingContext. Support Oracle ojdbc8 Plugin. Support ElasticSearch client transport 5.2-5.6 Plugin Support using agent.config with given path through system properties. Add a new way to transmit the Request and Response, to avoid bugs in Hytrix scenarios. Fix HTTPComponent client v4 operation name is empty. Fix 2 possible NPEs in Spring plugin. Fix a possible span leak in SpringMVC plugin. Fix NPE in Spring callback plugin. Collector Changes Add GZip support for Zipkin receiver. Add new component IDs for nodejs. Fix Zipkin span receiver may miss data in request. Optimize codes in heatmap calculation. Reduce unnecessary divide. Fix NPE in Alarm content generation. Fix the precision lost in ServiceNameService#startTimeMillis. Fix GC count is 0. Fix topology breaks when RPC client uses the async thread call. UI Changes Fix UI port can\u0026rsquo;t be set by startup script in Windows. Fix Topology self link error. Fix stack color mismatch label color in gc time chart. Documentation Add users list. Fix several document typo. Sync the Chinese documents. Add OpenAPM badge. Add icon/font documents to NOTICE files. Issues and Pull requests\n5.0.0-beta2 UI -\u0026gt; Collector GraphQL query protocol Add order and status in trace query. Agent Changes Add SOFA plugin. Add witness class for Kafka plugin. Add RuntimeContext in Context. Fix RuntimeContext fail in Tomcat plugin. Fix incompatible for getPropertyDescriptors in Spring core. Fix spymemcached plugin bug. Fix database URL parser bug. Fix StringIndexOutOfBoundsException when mysql jdbc url without databaseName。 Fix duplicate slash in Spring MVC plugin bug. Fix namespace bug. Fix NPE in Okhttp plugin when connect failed. FIx MalformedURLException in httpClientComponent plugin. Remove unused dependencies in Dubbo plugin. Remove gRPC timeout to avoid out of memory leak. Rewrite Async http client plugin. [Incubating] Add trace custom ignore optional plugin. Collector Changes Topology query optimization for more than 100 apps. Error rate alarm is not triggered. Tolerate unsupported segments. Support Integer Array, Long Array, String Array, Double Array in streaming data model. Support multiple entry span and multiple service name in one segment durtaion record. Use BulkProcessor to control the linear writing of data by multiple threads. Determine the log is enabled for the DEBUG level before printing message. Add static modifier to Logger. Add AspNet component. Filter inactive service in query. Support to query service based on Application. Fix RemoteDataMappingIdNotFoundException Exclude component-libaries.xml file in collector-*.jar, make sure it is in /conf only. Separate a single TTL in minute to in minute, hour, day, month metric and trace. Add order and status in trace query. Add folder lock to buffer folder. Modify operationName search from match to match_phrase. [Incubating] Add Zipkin span receiver. Support analysis Zipkin v1/v2 formats. [Incubating] Support sharding-sphere as storage implementor. UI Changes Support login and access control. Add new webapp.yml configuration file. Modify webapp startup script. Link to trace query from Thermodynamic graph Add application selector in service view. Add order and status in trace query. Documentation Add architecture design doc. Reformat deploy document. Adjust Tomcat deploy document. Remove all Apache licenses files in dist release packages. Update user cases. Update UI licenses. Add incubating sections in doc. Issues and Pull requests\n5.0.0-beta UI -\u0026gt; Collector GraphQL query protocol Replace all tps to throughput/cpm(calls per min) Add getThermodynamic service Update version to beta Agent Changes Support TLS. Support namespace. Support direct link. Support token. Add across thread toolkit. Add new plugin extend machenism to override agent core implementations. Fix an agent start up sequence bug. Fix wrong gc count. Remove system env override. Add Spring AOP aspect patch to avoid aop conflicts. Collector Changes Trace query based on timeline. Delete JVM aggregation in second. Support TLS. Support namespace. Support token auth. Group and aggregate requests based on response time and timeline, support Thermodynamic chart query Support component librariy setting through yml file for better extendibility. Optimize performance. Support short column name in ES or other storage implementor. Add a new cache module implementor, based on Caffeine. Support system property override settings. Refactor settings initialization. Provide collector instrumentation agent. Support .NET core component libraries. Fix divide zero in query. Fix Data don't remove as expected in ES implementor. Add some checks in collector modulization core. Add some test cases. UI Changes New trace query UI. New Application UI, merge server tab(removed) into application as sub page. New Topology UI. New response time / throughput TopN list. Add Thermodynamic chart in overview page. Change all tps to cpm(calls per minutes). Fix wrong osName in server view. Fix wrong startTime in trace view. Fix some icons internet requirements. Documentation Add TLS document. Add namespace document. Add direct link document. Add token document. Add across thread toolkit document. Add a FAQ about, Agent or collector version upgrade. Sync all English document to Chinese. Issues and Pull requests\n5.0.0-alpha Agent -\u0026gt; Collector protocol Remove C++ keywords Move Ref into Span from Segment Add span type, when register an operation UI -\u0026gt; Collector GraphQL query protocol First version protocol Agent Changes Support gRPC 1.x plugin Support kafka 0.11 and 1.x plugin Support ServiceComb 0.x plugin Support optional plugin mechanism. Support Spring 3.x and 4.x bean annotation optional plugin Support Apache httpcomponent AsyncClient 4.x plugin Provide automatic agent daily tests, and release reports here. Refactor Postgresql, Oracle, MySQL plugin for compatible. Fix jetty client 9 plugin error Fix async APIs of okhttp plugin error Fix log config didn\u0026rsquo;t work Fix a class loader error in okhttp plugin Collector Changes Support metrics analysis and aggregation for application, application instance and service in minute, hour, day and month. Support new GraphQL query protocol Support alarm Provide a prototype instrument for collector. Support node speculate in cluster and application topology. (Provider Node -\u0026gt; Consumer Node) -\u0026gt; (Provider Node -\u0026gt; MQ Server -\u0026gt; Consumer Node) UI Changes New 5.0.0 UI!!! Issues and Pull requests\n","excerpt":"\u003ch2 id=\"510\"\u003e5.1.0\u003c/h2\u003e\n\u003ch4 id=\"agent-changes\"\u003eAgent Changes\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix spring inherit issue in another way\u003c/li\u003e\n\u003cli\u003eFix classloader dead lock in jdk7+ - …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v11.0.0/en/changes/changes-5.x/","title":"5.1.0"},{"body":"5.1.0 Agent Changes Fix spring inherit issue in another way Fix classloader dead lock in jdk7+ - 5.x Support Spring mvc 5.x Support Spring webflux 5.x Collector Changes Fix too many open files. Fix the buffer file cannot delete. 5.0.0-GA Agent Changes Add several package names ignore in agent settings. Classes in these packages would be enhanced, even plugin declared. Support Undertow 2.x plugin. Fix wrong class names of Motan plugin, not a feature related issue, just naming. Collector Changes Make buffer file handler close more safety. Fix NPE in AlarmService Documentation Fix compiling doc link. Update new live demo address. 5.0.0-RC2 Agent Changes Support ActiveMQ 5.x Support RuntimeContext used out of TracingContext. Support Oracle ojdbc8 Plugin. Support ElasticSearch client transport 5.2-5.6 Plugin Support using agent.config with given path through system properties. Add a new way to transmit the Request and Response, to avoid bugs in Hytrix scenarios. Fix HTTPComponent client v4 operation name is empty. Fix 2 possible NPEs in Spring plugin. Fix a possible span leak in SpringMVC plugin. Fix NPE in Spring callback plugin. Collector Changes Add GZip support for Zipkin receiver. Add new component IDs for nodejs. Fix Zipkin span receiver may miss data in request. Optimize codes in heatmap calculation. Reduce unnecessary divide. Fix NPE in Alarm content generation. Fix the precision lost in ServiceNameService#startTimeMillis. Fix GC count is 0. Fix topology breaks when RPC client uses the async thread call. UI Changes Fix UI port can\u0026rsquo;t be set by startup script in Windows. Fix Topology self link error. Fix stack color mismatch label color in gc time chart. Documentation Add users list. Fix several document typo. Sync the Chinese documents. Add OpenAPM badge. Add icon/font documents to NOTICE files. Issues and Pull requests\n5.0.0-beta2 UI -\u0026gt; Collector GraphQL query protocol Add order and status in trace query. Agent Changes Add SOFA plugin. Add witness class for Kafka plugin. Add RuntimeContext in Context. Fix RuntimeContext fail in Tomcat plugin. Fix incompatible for getPropertyDescriptors in Spring core. Fix spymemcached plugin bug. Fix database URL parser bug. Fix StringIndexOutOfBoundsException when mysql jdbc url without databaseName。 Fix duplicate slash in Spring MVC plugin bug. Fix namespace bug. Fix NPE in Okhttp plugin when connect failed. FIx MalformedURLException in httpClientComponent plugin. Remove unused dependencies in Dubbo plugin. Remove gRPC timeout to avoid out of memory leak. Rewrite Async http client plugin. [Incubating] Add trace custom ignore optional plugin. Collector Changes Topology query optimization for more than 100 apps. Error rate alarm is not triggered. Tolerate unsupported segments. Support Integer Array, Long Array, String Array, Double Array in streaming data model. Support multiple entry span and multiple service name in one segment durtaion record. Use BulkProcessor to control the linear writing of data by multiple threads. Determine the log is enabled for the DEBUG level before printing message. Add static modifier to Logger. Add AspNet component. Filter inactive service in query. Support to query service based on Application. Fix RemoteDataMappingIdNotFoundException Exclude component-libaries.xml file in collector-*.jar, make sure it is in /conf only. Separate a single TTL in minute to in minute, hour, day, month metric and trace. Add order and status in trace query. Add folder lock to buffer folder. Modify operationName search from match to match_phrase. [Incubating] Add Zipkin span receiver. Support analysis Zipkin v1/v2 formats. [Incubating] Support sharding-sphere as storage implementor. UI Changes Support login and access control. Add new webapp.yml configuration file. Modify webapp startup script. Link to trace query from Thermodynamic graph Add application selector in service view. Add order and status in trace query. Documentation Add architecture design doc. Reformat deploy document. Adjust Tomcat deploy document. Remove all Apache licenses files in dist release packages. Update user cases. Update UI licenses. Add incubating sections in doc. Issues and Pull requests\n5.0.0-beta UI -\u0026gt; Collector GraphQL query protocol Replace all tps to throughput/cpm(calls per min) Add getThermodynamic service Update version to beta Agent Changes Support TLS. Support namespace. Support direct link. Support token. Add across thread toolkit. Add new plugin extend machenism to override agent core implementations. Fix an agent start up sequence bug. Fix wrong gc count. Remove system env override. Add Spring AOP aspect patch to avoid aop conflicts. Collector Changes Trace query based on timeline. Delete JVM aggregation in second. Support TLS. Support namespace. Support token auth. Group and aggregate requests based on response time and timeline, support Thermodynamic chart query Support component librariy setting through yml file for better extendibility. Optimize performance. Support short column name in ES or other storage implementor. Add a new cache module implementor, based on Caffeine. Support system property override settings. Refactor settings initialization. Provide collector instrumentation agent. Support .NET core component libraries. Fix divide zero in query. Fix Data don't remove as expected in ES implementor. Add some checks in collector modulization core. Add some test cases. UI Changes New trace query UI. New Application UI, merge server tab(removed) into application as sub page. New Topology UI. New response time / throughput TopN list. Add Thermodynamic chart in overview page. Change all tps to cpm(calls per minutes). Fix wrong osName in server view. Fix wrong startTime in trace view. Fix some icons internet requirements. Documentation Add TLS document. Add namespace document. Add direct link document. Add token document. Add across thread toolkit document. Add a FAQ about, Agent or collector version upgrade. Sync all English document to Chinese. Issues and Pull requests\n5.0.0-alpha Agent -\u0026gt; Collector protocol Remove C++ keywords Move Ref into Span from Segment Add span type, when register an operation UI -\u0026gt; Collector GraphQL query protocol First version protocol Agent Changes Support gRPC 1.x plugin Support kafka 0.11 and 1.x plugin Support ServiceComb 0.x plugin Support optional plugin mechanism. Support Spring 3.x and 4.x bean annotation optional plugin Support Apache httpcomponent AsyncClient 4.x plugin Provide automatic agent daily tests, and release reports here. Refactor Postgresql, Oracle, MySQL plugin for compatible. Fix jetty client 9 plugin error Fix async APIs of okhttp plugin error Fix log config didn\u0026rsquo;t work Fix a class loader error in okhttp plugin Collector Changes Support metrics analysis and aggregation for application, application instance and service in minute, hour, day and month. Support new GraphQL query protocol Support alarm Provide a prototype instrument for collector. Support node speculate in cluster and application topology. (Provider Node -\u0026gt; Consumer Node) -\u0026gt; (Provider Node -\u0026gt; MQ Server -\u0026gt; Consumer Node) UI Changes New 5.0.0 UI!!! Issues and Pull requests\n","excerpt":"\u003ch2 id=\"510\"\u003e5.1.0\u003c/h2\u003e\n\u003ch4 id=\"agent-changes\"\u003eAgent Changes\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix spring inherit issue in another way\u003c/li\u003e\n\u003cli\u003eFix classloader dead lock in jdk7+ - …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.4.0/en/changes/changes-5.x/","title":"5.1.0"},{"body":"5.1.0 Agent Changes Fix spring inherit issue in another way Fix classloader dead lock in jdk7+ - 5.x Support Spring mvc 5.x Support Spring webflux 5.x Collector Changes Fix too many open files. Fix the buffer file cannot delete. 5.0.0-GA Agent Changes Add several package names ignore in agent settings. Classes in these packages would be enhanced, even plugin declared. Support Undertow 2.x plugin. Fix wrong class names of Motan plugin, not a feature related issue, just naming. Collector Changes Make buffer file handler close more safety. Fix NPE in AlarmService Documentation Fix compiling doc link. Update new live demo address. 5.0.0-RC2 Agent Changes Support ActiveMQ 5.x Support RuntimeContext used out of TracingContext. Support Oracle ojdbc8 Plugin. Support ElasticSearch client transport 5.2-5.6 Plugin Support using agent.config with given path through system properties. Add a new way to transmit the Request and Response, to avoid bugs in Hytrix scenarios. Fix HTTPComponent client v4 operation name is empty. Fix 2 possible NPEs in Spring plugin. Fix a possible span leak in SpringMVC plugin. Fix NPE in Spring callback plugin. Collector Changes Add GZip support for Zipkin receiver. Add new component IDs for nodejs. Fix Zipkin span receiver may miss data in request. Optimize codes in heatmap calculation. Reduce unnecessary divide. Fix NPE in Alarm content generation. Fix the precision lost in ServiceNameService#startTimeMillis. Fix GC count is 0. Fix topology breaks when RPC client uses the async thread call. UI Changes Fix UI port can\u0026rsquo;t be set by startup script in Windows. Fix Topology self link error. Fix stack color mismatch label color in gc time chart. Documentation Add users list. Fix several document typo. Sync the Chinese documents. Add OpenAPM badge. Add icon/font documents to NOTICE files. Issues and Pull requests\n5.0.0-beta2 UI -\u0026gt; Collector GraphQL query protocol Add order and status in trace query. Agent Changes Add SOFA plugin. Add witness class for Kafka plugin. Add RuntimeContext in Context. Fix RuntimeContext fail in Tomcat plugin. Fix incompatible for getPropertyDescriptors in Spring core. Fix spymemcached plugin bug. Fix database URL parser bug. Fix StringIndexOutOfBoundsException when mysql jdbc url without databaseName。 Fix duplicate slash in Spring MVC plugin bug. Fix namespace bug. Fix NPE in Okhttp plugin when connect failed. FIx MalformedURLException in httpClientComponent plugin. Remove unused dependencies in Dubbo plugin. Remove gRPC timeout to avoid out of memory leak. Rewrite Async http client plugin. [Incubating] Add trace custom ignore optional plugin. Collector Changes Topology query optimization for more than 100 apps. Error rate alarm is not triggered. Tolerate unsupported segments. Support Integer Array, Long Array, String Array, Double Array in streaming data model. Support multiple entry span and multiple service name in one segment durtaion record. Use BulkProcessor to control the linear writing of data by multiple threads. Determine the log is enabled for the DEBUG level before printing message. Add static modifier to Logger. Add AspNet component. Filter inactive service in query. Support to query service based on Application. Fix RemoteDataMappingIdNotFoundException Exclude component-libaries.xml file in collector-*.jar, make sure it is in /conf only. Separate a single TTL in minute to in minute, hour, day, month metric and trace. Add order and status in trace query. Add folder lock to buffer folder. Modify operationName search from match to match_phrase. [Incubating] Add Zipkin span receiver. Support analysis Zipkin v1/v2 formats. [Incubating] Support sharding-sphere as storage implementor. UI Changes Support login and access control. Add new webapp.yml configuration file. Modify webapp startup script. Link to trace query from Thermodynamic graph Add application selector in service view. Add order and status in trace query. Documentation Add architecture design doc. Reformat deploy document. Adjust Tomcat deploy document. Remove all Apache licenses files in dist release packages. Update user cases. Update UI licenses. Add incubating sections in doc. Issues and Pull requests\n5.0.0-beta UI -\u0026gt; Collector GraphQL query protocol Replace all tps to throughput/cpm(calls per min) Add getThermodynamic service Update version to beta Agent Changes Support TLS. Support namespace. Support direct link. Support token. Add across thread toolkit. Add new plugin extend machenism to override agent core implementations. Fix an agent start up sequence bug. Fix wrong gc count. Remove system env override. Add Spring AOP aspect patch to avoid aop conflicts. Collector Changes Trace query based on timeline. Delete JVM aggregation in second. Support TLS. Support namespace. Support token auth. Group and aggregate requests based on response time and timeline, support Thermodynamic chart query Support component librariy setting through yml file for better extendibility. Optimize performance. Support short column name in ES or other storage implementor. Add a new cache module implementor, based on Caffeine. Support system property override settings. Refactor settings initialization. Provide collector instrumentation agent. Support .NET core component libraries. Fix divide zero in query. Fix Data don't remove as expected in ES implementor. Add some checks in collector modulization core. Add some test cases. UI Changes New trace query UI. New Application UI, merge server tab(removed) into application as sub page. New Topology UI. New response time / throughput TopN list. Add Thermodynamic chart in overview page. Change all tps to cpm(calls per minutes). Fix wrong osName in server view. Fix wrong startTime in trace view. Fix some icons internet requirements. Documentation Add TLS document. Add namespace document. Add direct link document. Add token document. Add across thread toolkit document. Add a FAQ about, Agent or collector version upgrade. Sync all English document to Chinese. Issues and Pull requests\n5.0.0-alpha Agent -\u0026gt; Collector protocol Remove C++ keywords Move Ref into Span from Segment Add span type, when register an operation UI -\u0026gt; Collector GraphQL query protocol First version protocol Agent Changes Support gRPC 1.x plugin Support kafka 0.11 and 1.x plugin Support ServiceComb 0.x plugin Support optional plugin mechanism. Support Spring 3.x and 4.x bean annotation optional plugin Support Apache httpcomponent AsyncClient 4.x plugin Provide automatic agent daily tests, and release reports here. Refactor Postgresql, Oracle, MySQL plugin for compatible. Fix jetty client 9 plugin error Fix async APIs of okhttp plugin error Fix log config didn\u0026rsquo;t work Fix a class loader error in okhttp plugin Collector Changes Support metrics analysis and aggregation for application, application instance and service in minute, hour, day and month. Support new GraphQL query protocol Support alarm Provide a prototype instrument for collector. Support node speculate in cluster and application topology. (Provider Node -\u0026gt; Consumer Node) -\u0026gt; (Provider Node -\u0026gt; MQ Server -\u0026gt; Consumer Node) UI Changes New 5.0.0 UI!!! Issues and Pull requests\n","excerpt":"\u003ch2 id=\"510\"\u003e5.1.0\u003c/h2\u003e\n\u003ch4 id=\"agent-changes\"\u003eAgent Changes\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix spring inherit issue in another way\u003c/li\u003e\n\u003cli\u003eFix classloader dead lock in jdk7+ - …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.5.0/en/changes/changes-5.x/","title":"5.1.0"},{"body":"5.1.0 Agent Changes Fix spring inherit issue in another way Fix classloader dead lock in jdk7+ - 5.x Support Spring mvc 5.x Support Spring webflux 5.x Collector Changes Fix too many open files. Fix the buffer file cannot delete. 5.0.0-GA Agent Changes Add several package names ignore in agent settings. Classes in these packages would be enhanced, even plugin declared. Support Undertow 2.x plugin. Fix wrong class names of Motan plugin, not a feature related issue, just naming. Collector Changes Make buffer file handler close more safety. Fix NPE in AlarmService Documentation Fix compiling doc link. Update new live demo address. 5.0.0-RC2 Agent Changes Support ActiveMQ 5.x Support RuntimeContext used out of TracingContext. Support Oracle ojdbc8 Plugin. Support ElasticSearch client transport 5.2-5.6 Plugin Support using agent.config with given path through system properties. Add a new way to transmit the Request and Response, to avoid bugs in Hytrix scenarios. Fix HTTPComponent client v4 operation name is empty. Fix 2 possible NPEs in Spring plugin. Fix a possible span leak in SpringMVC plugin. Fix NPE in Spring callback plugin. Collector Changes Add GZip support for Zipkin receiver. Add new component IDs for nodejs. Fix Zipkin span receiver may miss data in request. Optimize codes in heatmap calculation. Reduce unnecessary divide. Fix NPE in Alarm content generation. Fix the precision lost in ServiceNameService#startTimeMillis. Fix GC count is 0. Fix topology breaks when RPC client uses the async thread call. UI Changes Fix UI port can\u0026rsquo;t be set by startup script in Windows. Fix Topology self link error. Fix stack color mismatch label color in gc time chart. Documentation Add users list. Fix several document typo. Sync the Chinese documents. Add OpenAPM badge. Add icon/font documents to NOTICE files. Issues and Pull requests\n5.0.0-beta2 UI -\u0026gt; Collector GraphQL query protocol Add order and status in trace query. Agent Changes Add SOFA plugin. Add witness class for Kafka plugin. Add RuntimeContext in Context. Fix RuntimeContext fail in Tomcat plugin. Fix incompatible for getPropertyDescriptors in Spring core. Fix spymemcached plugin bug. Fix database URL parser bug. Fix StringIndexOutOfBoundsException when mysql jdbc url without databaseName。 Fix duplicate slash in Spring MVC plugin bug. Fix namespace bug. Fix NPE in Okhttp plugin when connect failed. FIx MalformedURLException in httpClientComponent plugin. Remove unused dependencies in Dubbo plugin. Remove gRPC timeout to avoid out of memory leak. Rewrite Async http client plugin. [Incubating] Add trace custom ignore optional plugin. Collector Changes Topology query optimization for more than 100 apps. Error rate alarm is not triggered. Tolerate unsupported segments. Support Integer Array, Long Array, String Array, Double Array in streaming data model. Support multiple entry span and multiple service name in one segment durtaion record. Use BulkProcessor to control the linear writing of data by multiple threads. Determine the log is enabled for the DEBUG level before printing message. Add static modifier to Logger. Add AspNet component. Filter inactive service in query. Support to query service based on Application. Fix RemoteDataMappingIdNotFoundException Exclude component-libaries.xml file in collector-*.jar, make sure it is in /conf only. Separate a single TTL in minute to in minute, hour, day, month metric and trace. Add order and status in trace query. Add folder lock to buffer folder. Modify operationName search from match to match_phrase. [Incubating] Add Zipkin span receiver. Support analysis Zipkin v1/v2 formats. [Incubating] Support sharding-sphere as storage implementor. UI Changes Support login and access control. Add new webapp.yml configuration file. Modify webapp startup script. Link to trace query from Thermodynamic graph Add application selector in service view. Add order and status in trace query. Documentation Add architecture design doc. Reformat deploy document. Adjust Tomcat deploy document. Remove all Apache licenses files in dist release packages. Update user cases. Update UI licenses. Add incubating sections in doc. Issues and Pull requests\n5.0.0-beta UI -\u0026gt; Collector GraphQL query protocol Replace all tps to throughput/cpm(calls per min) Add getThermodynamic service Update version to beta Agent Changes Support TLS. Support namespace. Support direct link. Support token. Add across thread toolkit. Add new plugin extend machenism to override agent core implementations. Fix an agent start up sequence bug. Fix wrong gc count. Remove system env override. Add Spring AOP aspect patch to avoid aop conflicts. Collector Changes Trace query based on timeline. Delete JVM aggregation in second. Support TLS. Support namespace. Support token auth. Group and aggregate requests based on response time and timeline, support Thermodynamic chart query Support component librariy setting through yml file for better extendibility. Optimize performance. Support short column name in ES or other storage implementor. Add a new cache module implementor, based on Caffeine. Support system property override settings. Refactor settings initialization. Provide collector instrumentation agent. Support .NET core component libraries. Fix divide zero in query. Fix Data don't remove as expected in ES implementor. Add some checks in collector modulization core. Add some test cases. UI Changes New trace query UI. New Application UI, merge server tab(removed) into application as sub page. New Topology UI. New response time / throughput TopN list. Add Thermodynamic chart in overview page. Change all tps to cpm(calls per minutes). Fix wrong osName in server view. Fix wrong startTime in trace view. Fix some icons internet requirements. Documentation Add TLS document. Add namespace document. Add direct link document. Add token document. Add across thread toolkit document. Add a FAQ about, Agent or collector version upgrade. Sync all English document to Chinese. Issues and Pull requests\n5.0.0-alpha Agent -\u0026gt; Collector protocol Remove C++ keywords Move Ref into Span from Segment Add span type, when register an operation UI -\u0026gt; Collector GraphQL query protocol First version protocol Agent Changes Support gRPC 1.x plugin Support kafka 0.11 and 1.x plugin Support ServiceComb 0.x plugin Support optional plugin mechanism. Support Spring 3.x and 4.x bean annotation optional plugin Support Apache httpcomponent AsyncClient 4.x plugin Provide automatic agent daily tests, and release reports here. Refactor Postgresql, Oracle, MySQL plugin for compatible. Fix jetty client 9 plugin error Fix async APIs of okhttp plugin error Fix log config didn\u0026rsquo;t work Fix a class loader error in okhttp plugin Collector Changes Support metrics analysis and aggregation for application, application instance and service in minute, hour, day and month. Support new GraphQL query protocol Support alarm Provide a prototype instrument for collector. Support node speculate in cluster and application topology. (Provider Node -\u0026gt; Consumer Node) -\u0026gt; (Provider Node -\u0026gt; MQ Server -\u0026gt; Consumer Node) UI Changes New 5.0.0 UI!!! Issues and Pull requests\n","excerpt":"\u003ch2 id=\"510\"\u003e5.1.0\u003c/h2\u003e\n\u003ch4 id=\"agent-changes\"\u003eAgent Changes\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix spring inherit issue in another way\u003c/li\u003e\n\u003cli\u003eFix classloader dead lock in jdk7+ - …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.6.0/en/changes/changes-5.x/","title":"5.1.0"},{"body":"5.1.0 Agent Changes Fix spring inherit issue in another way Fix classloader dead lock in jdk7+ - 5.x Support Spring mvc 5.x Support Spring webflux 5.x Collector Changes Fix too many open files. Fix the buffer file cannot delete. 5.0.0-GA Agent Changes Add several package names ignore in agent settings. Classes in these packages would be enhanced, even plugin declared. Support Undertow 2.x plugin. Fix wrong class names of Motan plugin, not a feature related issue, just naming. Collector Changes Make buffer file handler close more safety. Fix NPE in AlarmService Documentation Fix compiling doc link. Update new live demo address. 5.0.0-RC2 Agent Changes Support ActiveMQ 5.x Support RuntimeContext used out of TracingContext. Support Oracle ojdbc8 Plugin. Support ElasticSearch client transport 5.2-5.6 Plugin Support using agent.config with given path through system properties. Add a new way to transmit the Request and Response, to avoid bugs in Hytrix scenarios. Fix HTTPComponent client v4 operation name is empty. Fix 2 possible NPEs in Spring plugin. Fix a possible span leak in SpringMVC plugin. Fix NPE in Spring callback plugin. Collector Changes Add GZip support for Zipkin receiver. Add new component IDs for nodejs. Fix Zipkin span receiver may miss data in request. Optimize codes in heatmap calculation. Reduce unnecessary divide. Fix NPE in Alarm content generation. Fix the precision lost in ServiceNameService#startTimeMillis. Fix GC count is 0. Fix topology breaks when RPC client uses the async thread call. UI Changes Fix UI port can\u0026rsquo;t be set by startup script in Windows. Fix Topology self link error. Fix stack color mismatch label color in gc time chart. Documentation Add users list. Fix several document typo. Sync the Chinese documents. Add OpenAPM badge. Add icon/font documents to NOTICE files. Issues and Pull requests\n5.0.0-beta2 UI -\u0026gt; Collector GraphQL query protocol Add order and status in trace query. Agent Changes Add SOFA plugin. Add witness class for Kafka plugin. Add RuntimeContext in Context. Fix RuntimeContext fail in Tomcat plugin. Fix incompatible for getPropertyDescriptors in Spring core. Fix spymemcached plugin bug. Fix database URL parser bug. Fix StringIndexOutOfBoundsException when mysql jdbc url without databaseName。 Fix duplicate slash in Spring MVC plugin bug. Fix namespace bug. Fix NPE in Okhttp plugin when connect failed. FIx MalformedURLException in httpClientComponent plugin. Remove unused dependencies in Dubbo plugin. Remove gRPC timeout to avoid out of memory leak. Rewrite Async http client plugin. [Incubating] Add trace custom ignore optional plugin. Collector Changes Topology query optimization for more than 100 apps. Error rate alarm is not triggered. Tolerate unsupported segments. Support Integer Array, Long Array, String Array, Double Array in streaming data model. Support multiple entry span and multiple service name in one segment durtaion record. Use BulkProcessor to control the linear writing of data by multiple threads. Determine the log is enabled for the DEBUG level before printing message. Add static modifier to Logger. Add AspNet component. Filter inactive service in query. Support to query service based on Application. Fix RemoteDataMappingIdNotFoundException Exclude component-libaries.xml file in collector-*.jar, make sure it is in /conf only. Separate a single TTL in minute to in minute, hour, day, month metric and trace. Add order and status in trace query. Add folder lock to buffer folder. Modify operationName search from match to match_phrase. [Incubating] Add Zipkin span receiver. Support analysis Zipkin v1/v2 formats. [Incubating] Support sharding-sphere as storage implementor. UI Changes Support login and access control. Add new webapp.yml configuration file. Modify webapp startup script. Link to trace query from Thermodynamic graph Add application selector in service view. Add order and status in trace query. Documentation Add architecture design doc. Reformat deploy document. Adjust Tomcat deploy document. Remove all Apache licenses files in dist release packages. Update user cases. Update UI licenses. Add incubating sections in doc. Issues and Pull requests\n5.0.0-beta UI -\u0026gt; Collector GraphQL query protocol Replace all tps to throughput/cpm(calls per min) Add getThermodynamic service Update version to beta Agent Changes Support TLS. Support namespace. Support direct link. Support token. Add across thread toolkit. Add new plugin extend machenism to override agent core implementations. Fix an agent start up sequence bug. Fix wrong gc count. Remove system env override. Add Spring AOP aspect patch to avoid aop conflicts. Collector Changes Trace query based on timeline. Delete JVM aggregation in second. Support TLS. Support namespace. Support token auth. Group and aggregate requests based on response time and timeline, support Thermodynamic chart query Support component librariy setting through yml file for better extendibility. Optimize performance. Support short column name in ES or other storage implementor. Add a new cache module implementor, based on Caffeine. Support system property override settings. Refactor settings initialization. Provide collector instrumentation agent. Support .NET core component libraries. Fix divide zero in query. Fix Data don't remove as expected in ES implementor. Add some checks in collector modulization core. Add some test cases. UI Changes New trace query UI. New Application UI, merge server tab(removed) into application as sub page. New Topology UI. New response time / throughput TopN list. Add Thermodynamic chart in overview page. Change all tps to cpm(calls per minutes). Fix wrong osName in server view. Fix wrong startTime in trace view. Fix some icons internet requirements. Documentation Add TLS document. Add namespace document. Add direct link document. Add token document. Add across thread toolkit document. Add a FAQ about, Agent or collector version upgrade. Sync all English document to Chinese. Issues and Pull requests\n5.0.0-alpha Agent -\u0026gt; Collector protocol Remove C++ keywords Move Ref into Span from Segment Add span type, when register an operation UI -\u0026gt; Collector GraphQL query protocol First version protocol Agent Changes Support gRPC 1.x plugin Support kafka 0.11 and 1.x plugin Support ServiceComb 0.x plugin Support optional plugin mechanism. Support Spring 3.x and 4.x bean annotation optional plugin Support Apache httpcomponent AsyncClient 4.x plugin Provide automatic agent daily tests, and release reports here. Refactor Postgresql, Oracle, MySQL plugin for compatible. Fix jetty client 9 plugin error Fix async APIs of okhttp plugin error Fix log config didn\u0026rsquo;t work Fix a class loader error in okhttp plugin Collector Changes Support metrics analysis and aggregation for application, application instance and service in minute, hour, day and month. Support new GraphQL query protocol Support alarm Provide a prototype instrument for collector. Support node speculate in cluster and application topology. (Provider Node -\u0026gt; Consumer Node) -\u0026gt; (Provider Node -\u0026gt; MQ Server -\u0026gt; Consumer Node) UI Changes New 5.0.0 UI!!! Issues and Pull requests\n","excerpt":"\u003ch2 id=\"510\"\u003e5.1.0\u003c/h2\u003e\n\u003ch4 id=\"agent-changes\"\u003eAgent Changes\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix spring inherit issue in another way\u003c/li\u003e\n\u003cli\u003eFix classloader dead lock in jdk7+ - …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.7.0/en/changes/changes-5.x/","title":"5.1.0"},{"body":"6.6.0 Project [IMPORTANT] Local span and exit span are not treated as endpoint detected at client and local. Only entry span is the endpoint. Reduce the load of register and memory cost. Support MiniKube, Istio and SkyWalking on K8s deployment in CI. Support Windows and MacOS build in GitHub Action CI. Support ElasticSearch 7 in official dist. Hundreds plugin cases have been added in GitHub Action CI process. Java Agent Remove the local/exit span operation name register mechanism. Add plugin for JDK Threading classes. Add plugin for Armeria. Support set operation name in async span. Enhance webflux plugin, related to Spring Gateway plugin. Webflux plugin is in optional, due to JDK8 required. Fix a possible deadlock. Fix NPE when OAL scripts are different in different OAP nodes, mostly in upgrading stage. Fix bug about wrong peer in ES plugin. Fix NPE in Spring plugin. Fix wrong class name in Dubbo 2.7 conflict patch. Fix spring annotation inheritance problem. OAP-Backend Remove the local/exit span operation name register mechanism. Remove client side endpoint register in service mesh. Service instance dependency and related metrics. Support min func in OAL Support apdex func in OAL Support custom ES config setting at the index level. Envoy ALS proto upgraded. Update JODA lib as bugs in UTC +13/+14. Support topN sample period configurable. Ignore no statement DB operations in slow SQL collection. Fix bug in docker-entrypoint.sh when using MySQL as storage UI Service topology enhancement. Dive into service, instance and endpoint metrics on topo map. Service instance dependency view and related metrics. Support using URL parameter in trace query page. Support apdex score in service page. Add service dependency metrics into metrics comparison. Fix alarm search not working. Document Update user list and user wall. Add document link for CLI. Add deployment guide of agent in Jetty case. Modify Consul cluster doc. Add document about injecting traceId into the logback with logstack in JSON format. ElementUI license and dependency added. All issues and pull requests are here\n6.5.0 Project TTL E2E test (#3437) Test coverage is back in pull request check status (#3503) Plugin tests begin to be migrated into main repo, and is in process. (#3528, #3756, #3751, etc.) Switch to SkyWalking CI (exclusive) nodes (#3546) MySQL storage e2e test. (#3648) E2E tests are verified in multiple jdk versions, jdk 8, 9, 11, 12 (#3657) Jenkins build jobs run only when necessary (#3662) OAP-Backend Support dynamically configure alarm settings (#3557) Language of instance could be null (#3485) Make query max window size configurable. (#3765) Remove two max size 500 limit. (#3748) Parameterize the cache size. (#3741) ServiceInstanceRelation set error id (#3683) Makes the scope of alarm message more semantic. (#3680) Add register persistent worker latency metrics (#3677) Fix more reasonable error (#3619) Add GraphQL getServiceInstance instanceUuid field. (#3595) Support namespace in Nacos cluster/configuration (#3578) Instead of datasource-settings.properties, use application.yml for MySQLStorageProvider (#3564) Provide consul dynamic configuration center implementation (#3560) Upgrade guava version to support higher jdk version (#3541) Sync latest als from envoy api (#3507) Set telemetry instanced id for Etcd and Nacos plugin (#3492) Support timeout configuration in agent and backend. (#3491) Make sure the cluster register happens before streaming process. (#3471) Agent supports custom properties. (#3367) Miscellaneous bug fixes (#3567) UI Feature: node detail display in topo circle-chart view. BugFix: the jvm-maxheap \u0026amp; jvm-maxnonheap is -1, free is no value Fix bug: time select operation not in effect Fix bug: language initialization failed Fix bug: not show instance language Feature: support the trace list display export png Feature: Metrics comparison view BugFix: Fix dashboard top throughput copy Java Agent Spring async scenario optimize (#3723) Support log4j2 AsyncLogger (#3715) Add config to collect PostgreSQL sql query params (#3695) Support namespace in Nacos cluster/configuration (#3578) Provide plugin for ehcache 2.x (#3575) Supporting RequestRateLimiterGatewayFilterFactory (#3538) Kafka-plugin compatible with KafkaTemplate (#3505) Add pulsar apm plugin (#3476) Spring-cloud-gateway traceId does not transmit #3411 (#3446) Gateway compatible with downstream loss (#3445) Provide cassandra java driver 3.x plugin (#3410) Fix SpringMVC4 NoSuchMethodError (#3408) BugFix: endpoint grouping rules may be not unique (#3510) Add feature to control the maximum agent log files (#3475) Agent support custom properties. (#3367) Add Light4j plugin (#3323) Document Remove travis badge (#3763) Replace user wall to typical users in readme page (#3719) Update istio docs according latest istio release (#3646) Use chart deploy sw docs (#3573) Reorganize the doc, and provide catalog (#3563) Committer vote and set up document. (#3496) Update als setup doc as istio 1.3 released (#3470) Fill faq reply in official document. (#3450) All issues and pull requests are here\n6.4.0 Project Highly recommend to upgrade due to Pxx metrics calculation bug. Make agent working in JDK9+ Module system. Java Agent Make agent working in JDK9+ Module system. Support Kafka 2.x client libs. Log error in OKHTTP OnFailure callback. Support injecting traceid into logstack appender in logback. Add OperationName(including endpoint name) length max threshold. Support using Regex to group operation name. Support Undertow routing handler. RestTemplate plugin support operation name grouping. Fix ClassCastException in Webflux plugin. Ordering zookeeper server list, to make it better in topology. Fix a Dubbo plugin incompatible issue. Fix MySQL 5 plugin issue. Make log writer cached. Optimize Spring Cloud Gateway plugin Fix and improve gRPC reconnect mechanism. Remove Disruptor dependency from agent. Backend Fix Pxx(p50,p75,p90,p95,p99) metrics func bug.(Critical) Support Gateway in backend analysis, even when it doesn\u0026rsquo;t have suitable language agent. Support using HTTPs SSL accessing ElasticSearch storage. Support Zookeeper ACL. Make alarm records listed in order. Fix Pxx data persistence failure in some cases. Fix some bugs in MySQL storage. Setup slow SQL length threshold. Fix TTL settings is not working as expected. Remove scope-meta file. UI Enhance alarm page layout. Support trace tree chart resize. Support trace auto completion when partial traces abandoned somehow. Fix dashboard endpoint slow chart. Add radial chart in topology page. Add trace table mode. Fix topology page bug. Fix calender js bug. Fix \u0026ldquo;The \u0026ldquo;topo-services\u0026rdquo; component did not update the data in time after modifying the time range on the topology page. Document Restore the broken Istio setup doc. Add etcd config center document. Correct span_limit_per_segment default value in document. Enhance plugin develop doc. Fix error description in build document. All issues and pull requests are here\n6.3.0 Project e2e tests have been added, and verify every pull request. Use ArrayList to replace LinkedList in DataCarrier for much better performance. Add plugin instrumentation definition check in CI. DataCarrier performance improvement by avoiding false-sharing. Java Agent Java agent supports JDK 9 - 12, but don\u0026rsquo;t support Java Module yet. Support JVM class auto instrumentation, cataloged as bootstrap plugin. Support JVM HttpClient and HttpsClient plugin.[Optional] Support backend upgrade without rebooting required. Open Redefine and Retransform by other agents. Support Servlet 2.5 in Jetty, Tomcat and SpringMVC plugins. Support Spring @Async plugin. Add new config item to restrict the length of span#peer. Refactor ContextManager#stopSpan. Add gRPC timeout. Support Logback AsyncAppender print tid Fix gRPC reconnect bug. Fix trace segment service doesn\u0026rsquo;t report onComplete. Fix wrong logger class name. Fix gRPC plugin bug. Fix ContextManager.activeSpan() API usage error. Backend Support agent reset command downstream when the storage is erased, mostly because of backend upgrade. Backend stream flow refactor. High dimensionality metrics(Hour/Day/Month) are changed to lower priority, to ease the storage payload. Add OAP metrics cache to ease the storage query payload and improve performance. Remove DataCarrier in trace persistent of ElasticSearch storage, by leveraging the elasticsearch bulk queue. OAP internal communication protocol changed. Don\u0026rsquo;t be compatible with old releases. Improve ElasticSearch storage bulk performance. Support etcd as dynamic configuration center. Simplify the PxxMetrics and ThermodynamicMetrics functions for better performance and GC. Support JVM metrics self observability. Add the new OAL runtime engine. Add gRPC timeout. Add Charset in the alarm web hook. Fix buffer lost. Fix dirty read in ElasticSearch storage. Fix bug of cluster management plugins in un-Mixed mode. Fix wrong logger class name. Fix delete bug in ElasticSearch when using namespace. Fix MySQL TTL failure. Totally remove IDs can't be null log, to avoid misleading. Fix provider has been initialized repeatedly. Adjust providers conflict log message. Fix using wrong gc time metrics in OAL. UI Fix refresh is not working after endpoint and instance changed. Fix endpoint selector but. Fix wrong copy value in slow traces. Fix can\u0026rsquo;t show trace when it is broken partially(Because of agent sampling or fail safe). Fix database and response time graph bugs. Document Add bootstrap plugin development document. Alarm documentation typo fixed. Clarify the Docker file purpose. Fix a license typo. All issues and pull requests are here\n6.2.0 Project ElasticSearch implementation performance improved, and CHANGED totally. Must delete all existing indexes to do upgrade. CI and Integration tests provided by ASF INFRA. Plan to enhance tests including e2e, plugin tests in all pull requests, powered by ASF INFRA. DataCarrier queue write index controller performance improvement. 3-5 times quicker than before. Add windows compile support in CI. Java Agent Support collect SQL parameter in MySQL plugin.[Optional] Support SolrJ plugin. Support RESTEasy plugin. Support Spring Gateway plugin for 2.1.x[Optional] TracingContext performance improvement. Support Apache ShardingSphere(incubating) plugin. Support span#error in application toolkit. Fix OOM by empty stack of exception. FIx wrong cause exception of stack in span log. Fix unclear the running context in SpringMVC plugin. Fix CPU usage accessor calculation issue. Fix SpringMVC plugin span not stop bug when doing HTTP forward. Fix lettuce plugin async commend bug and NPE. Fix webflux plugin cast exception. [CI]Support import check. Backend Support time serious ElasticSearch storage. Provide dynamic configuration module and implementation. Slow SQL threshold supports dynamic config today. Dynamic Configuration module provide multiple implementations, DCS(gRPC based), Zookeeper, Apollo, Nacos. Provide P99/95/90/75/50 charts in topology edge. New topology query protocol and implementation. Support Envoy ALS in Service Mesh scenario. Support Nacos cluster management. Enhance metric exporter. Run in increment and total modes. Fix module provider is loaded repeatedly. Change TOP slow SQL storage in ES to Text from Keyword, as too long text issue. Fix H2TopologyQuery tiny bug. Fix H2 log query bug.(No feature provided yet) Filtering pods not in \u0026lsquo;Running\u0026rsquo; phase in mesh scenario. Fix query alarm bug in MySQL and H2 storage. Codes refactor. UI Fix some ID is null query(s). Page refactor, especially time-picker, more friendly. Login removed. Trace timestamp visualization issue fixed. Provide P99/95/90/75/50 charts in topology edge. Change all P99/95/90/75/50 charts style. More readable. Fix 404 in trace page. Document Go2Sky project has been donated to SkyAPM, change document link. Add FAQ for ElasticSearch storage, and links from document. Add FAQ fro WebSphere installation. Add several open users. Add alarm webhook document. All issues and pull requests are here\n6.1.0 Project SkyWalking graduated as Apache Top Level Project.\nSupport compiling project agent, backend, UI separately. Java Agent Support Vert.x Core 3.x plugin. Support Apache Dubbo plugin. Support use_qualified_name_as_endpoint_name and use_qualified_name_as_operation_name configs in SpringMVC plugin. Support span async close APIs in core. Used in Vert.x plugin. Support MySQL 5,8 plugins. Support set instance id manually(optional). Support customize enhance trace plugin in optional list. Support to set peer in Entry Span. Support Zookeeper plugin. Fix Webflux plugin created unexpected Entry Span. Fix Kafka plugin NPE in Kafka 1.1+ Fix wrong operation name in postgre 8.x plugin. Fix RabbitMQ plugin NPE. Fix agent can\u0026rsquo;t run in JVM 6/7, remove module-info.class. Fix agent can\u0026rsquo;t work well, if there is whitespace in agent path. Fix Spring annotation bug and inheritance enhance issue. Fix CPU accessor bug. Backend Performance improved, especially in CPU limited environment. 3x improvement in service mesh scenario(no trace) in 8C16G VM. Significantly cost less CPU in low payload.\nSupport database metrics and SLOW SQL detection. Support to set max size of metadata query. And change default to 5000 from 100. Support ElasticSearch template for new feature in the future. Support shutdown Zipkin trace analysis, because it doesn\u0026rsquo;t fit production environment. Support log type, scope HTTP_ACCESS_LOG and query. No feature provided, prepare for future versions. Support .NET clr receiver. Support Jaeger trace format, no analysis. Support group endpoint name by regax rules in mesh receiver. Support disable statement in OAL. Support basic auth in ElasticSearch connection. Support metrics exporter module and gRPC implementor. Support \u0026gt;, \u0026lt;, \u0026gt;=, \u0026lt;= in OAL. Support role mode in backend. Support Envoy metrics. Support query segment by service instance. Support to set host/port manually at cluster coordinator, rather than based on core settings. Make sure OAP shutdown when it faces startup error. Support set separated gRPC/Jetty ip:port for receiver, default still use core settings. Fix JVM receiver bug. Fix wrong dest service in mesh analysis. Fix search doesn\u0026rsquo;t work as expected. Refactor ScopeDeclaration annotation. Refactor register lock mechanism. Add SmartSql component for .NET Add integration tests for ElasticSearch client. Add test cases for exporter. Add test cases for queue consume. UI RocketBot UI has been accepted and bind in this release. Support CLR metrics. Document Documents updated, matching Top Level Project requirement. UI licenses updated, according to RocketBot UI IP clearance. User wall and powered-by list updated. CN documents removed, only consider to provide by volunteer out of Apache. All issues and pull requests are here\n6.0.0-GA Java Agent Support gson plugin(optional). Support canal plugin. Fix missing ojdbc component id. Fix dubbo plugin conflict. Fix OpenTracing tag match bug. Fix a missing check in ignore plugin. Backend Adjust service inventory entity, to add properties. Adjust service instance inventory entity, to add properties. Add nodeType to service inventory entity. Fix when operation name of local and exit spans in ref, the segment lost. Fix the index names don\u0026rsquo;t show right in logs. Fix wrong alarm text. Add test case for span limit mechanism. Add telemetry module and prometheus implementation, with grafana setting. A refactor for register API in storage module. Fix H2 and MySQL endpoint dependency map miss upstream side. Optimize the inventory register and refactor the implementation. Speed up the trace buffer read. Fix and removed unnecessary inventory register operations. UI Add new trace view. Add word-break to tag value. Document Add two startup modes document. Add PHP agent links. Add some cn documents. Update year to 2019 User wall updated. Fix a wrong description in how-to-build doc. All issues and pull requests are here\n6.0.0-beta Protocol Provide Trace Data Protocol v2 Provide SkyWalking Cross Process Propagation Headers Protocol v2. Java Agent Support Trace Data Protocol v2 Support SkyWalking Cross Process Propagation Headers Protocol v2. Support SkyWalking Cross Process Propagation Headers Protocol v1 running in compatible way. Need declare open explicitly. Support SpringMVC 5 Support webflux Support a new way to override agent.config by system env. Span tag can override by explicit way. Fix Spring Controller Inherit issue. Fix ElasticSearch plugin NPE. Fix agent classloader dead lock in certain situation. Fix agent log typo. Fix wrong component id in resettemplete plugin. Fix use transform ignore() in wrong way. Fix H2 query bug. Backend Support Trace Data Protocol v2. And Trace Data Protocol v1 is still supported. Support MySQL as storage. Support TiDB as storage. Support a new way to override application.yml by system env. Support service instance and endpoint alarm. Support namespace in istio receiver. Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Support backend trace sampling. Support Zipkin format again. Support init mode. Support namespace in Zookeeper cluster management. Support consul plugin in cluster module. OAL generate tool has been integrated into main repo, in the maven compile stage. Optimize trace paging query. Fix trace query don\u0026rsquo;t use fuzzy query in ElasticSearch storage. Fix alarm can\u0026rsquo;t be active in right way. Fix unnecessary condition in database and cache number query. Fix wrong namespace bug in ElasticSearch storage. Fix Remote clients selector error: / by zero . Fix segment TTL is not working. UI Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Fix TopN endpoint link doesn\u0026rsquo;t work right. Fix trace stack style. Fix CI. Document Add more agent setting documents. Add more contribution documents. Update user wall and powered-by page. Add RocketBot UI project link in document. All issues and pull requests are here\n6.0.0-alpha SkyWalking 6 is totally new milestone for the project. At this point, we are not just a distributing tracing system with analysis and visualization capabilities. We are an Observability Analysis Platform(OAL).\nThe core and most important features in v6 are\nSupport to collect telemetry data from different sources, such as multiple language agents and service mesh. Extensible stream analysis core. Make SQL and cache analysis available in core level, although haven\u0026rsquo;t provided in this release. Provide Observability Analysis Language(OAL) to make analysis metrics customization available. New GraphQL query protocol. Not binding with UI now. UI topology is better now. New alarm core provided. In alpha, only on service related metrics. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"660\"\u003e6.6.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e[\u003cstrong\u003eIMPORTANT\u003c/strong\u003e] Local span and exit span are not treated as endpoint detected at client …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/latest/en/changes/changes-6.x/","title":"6.6.0"},{"body":"6.6.0 Project [IMPORTANT] Local span and exit span are not treated as endpoint detected at client and local. Only entry span is the endpoint. Reduce the load of register and memory cost. Support MiniKube, Istio and SkyWalking on K8s deployment in CI. Support Windows and MacOS build in GitHub Action CI. Support ElasticSearch 7 in official dist. Hundreds plugin cases have been added in GitHub Action CI process. Java Agent Remove the local/exit span operation name register mechanism. Add plugin for JDK Threading classes. Add plugin for Armeria. Support set operation name in async span. Enhance webflux plugin, related to Spring Gateway plugin. Webflux plugin is in optional, due to JDK8 required. Fix a possible deadlock. Fix NPE when OAL scripts are different in different OAP nodes, mostly in upgrading stage. Fix bug about wrong peer in ES plugin. Fix NPE in Spring plugin. Fix wrong class name in Dubbo 2.7 conflict patch. Fix spring annotation inheritance problem. OAP-Backend Remove the local/exit span operation name register mechanism. Remove client side endpoint register in service mesh. Service instance dependency and related metrics. Support min func in OAL Support apdex func in OAL Support custom ES config setting at the index level. Envoy ALS proto upgraded. Update JODA lib as bugs in UTC +13/+14. Support topN sample period configurable. Ignore no statement DB operations in slow SQL collection. Fix bug in docker-entrypoint.sh when using MySQL as storage UI Service topology enhancement. Dive into service, instance and endpoint metrics on topo map. Service instance dependency view and related metrics. Support using URL parameter in trace query page. Support apdex score in service page. Add service dependency metrics into metrics comparison. Fix alarm search not working. Document Update user list and user wall. Add document link for CLI. Add deployment guide of agent in Jetty case. Modify Consul cluster doc. Add document about injecting traceId into the logback with logstack in JSON format. ElementUI license and dependency added. All issues and pull requests are here\n6.5.0 Project TTL E2E test (#3437) Test coverage is back in pull request check status (#3503) Plugin tests begin to be migrated into main repo, and is in process. (#3528, #3756, #3751, etc.) Switch to SkyWalking CI (exclusive) nodes (#3546) MySQL storage e2e test. (#3648) E2E tests are verified in multiple jdk versions, jdk 8, 9, 11, 12 (#3657) Jenkins build jobs run only when necessary (#3662) OAP-Backend Support dynamically configure alarm settings (#3557) Language of instance could be null (#3485) Make query max window size configurable. (#3765) Remove two max size 500 limit. (#3748) Parameterize the cache size. (#3741) ServiceInstanceRelation set error id (#3683) Makes the scope of alarm message more semantic. (#3680) Add register persistent worker latency metrics (#3677) Fix more reasonable error (#3619) Add GraphQL getServiceInstance instanceUuid field. (#3595) Support namespace in Nacos cluster/configuration (#3578) Instead of datasource-settings.properties, use application.yml for MySQLStorageProvider (#3564) Provide consul dynamic configuration center implementation (#3560) Upgrade guava version to support higher jdk version (#3541) Sync latest als from envoy api (#3507) Set telemetry instanced id for Etcd and Nacos plugin (#3492) Support timeout configuration in agent and backend. (#3491) Make sure the cluster register happens before streaming process. (#3471) Agent supports custom properties. (#3367) Miscellaneous bug fixes (#3567) UI Feature: node detail display in topo circle-chart view. BugFix: the jvm-maxheap \u0026amp; jvm-maxnonheap is -1, free is no value Fix bug: time select operation not in effect Fix bug: language initialization failed Fix bug: not show instance language Feature: support the trace list display export png Feature: Metrics comparison view BugFix: Fix dashboard top throughput copy Java Agent Spring async scenario optimize (#3723) Support log4j2 AsyncLogger (#3715) Add config to collect PostgreSQL sql query params (#3695) Support namespace in Nacos cluster/configuration (#3578) Provide plugin for ehcache 2.x (#3575) Supporting RequestRateLimiterGatewayFilterFactory (#3538) Kafka-plugin compatible with KafkaTemplate (#3505) Add pulsar apm plugin (#3476) Spring-cloud-gateway traceId does not transmit #3411 (#3446) Gateway compatible with downstream loss (#3445) Provide cassandra java driver 3.x plugin (#3410) Fix SpringMVC4 NoSuchMethodError (#3408) BugFix: endpoint grouping rules may be not unique (#3510) Add feature to control the maximum agent log files (#3475) Agent support custom properties. (#3367) Add Light4j plugin (#3323) Document Remove travis badge (#3763) Replace user wall to typical users in readme page (#3719) Update istio docs according latest istio release (#3646) Use chart deploy sw docs (#3573) Reorganize the doc, and provide catalog (#3563) Committer vote and set up document. (#3496) Update als setup doc as istio 1.3 released (#3470) Fill faq reply in official document. (#3450) All issues and pull requests are here\n6.4.0 Project Highly recommend to upgrade due to Pxx metrics calculation bug. Make agent working in JDK9+ Module system. Java Agent Make agent working in JDK9+ Module system. Support Kafka 2.x client libs. Log error in OKHTTP OnFailure callback. Support injecting traceid into logstack appender in logback. Add OperationName(including endpoint name) length max threshold. Support using Regex to group operation name. Support Undertow routing handler. RestTemplate plugin support operation name grouping. Fix ClassCastException in Webflux plugin. Ordering zookeeper server list, to make it better in topology. Fix a Dubbo plugin incompatible issue. Fix MySQL 5 plugin issue. Make log writer cached. Optimize Spring Cloud Gateway plugin Fix and improve gRPC reconnect mechanism. Remove Disruptor dependency from agent. Backend Fix Pxx(p50,p75,p90,p95,p99) metrics func bug.(Critical) Support Gateway in backend analysis, even when it doesn\u0026rsquo;t have suitable language agent. Support using HTTPs SSL accessing ElasticSearch storage. Support Zookeeper ACL. Make alarm records listed in order. Fix Pxx data persistence failure in some cases. Fix some bugs in MySQL storage. Setup slow SQL length threshold. Fix TTL settings is not working as expected. Remove scope-meta file. UI Enhance alarm page layout. Support trace tree chart resize. Support trace auto completion when partial traces abandoned somehow. Fix dashboard endpoint slow chart. Add radial chart in topology page. Add trace table mode. Fix topology page bug. Fix calender js bug. Fix \u0026ldquo;The \u0026ldquo;topo-services\u0026rdquo; component did not update the data in time after modifying the time range on the topology page. Document Restore the broken Istio setup doc. Add etcd config center document. Correct span_limit_per_segment default value in document. Enhance plugin develop doc. Fix error description in build document. All issues and pull requests are here\n6.3.0 Project e2e tests have been added, and verify every pull request. Use ArrayList to replace LinkedList in DataCarrier for much better performance. Add plugin instrumentation definition check in CI. DataCarrier performance improvement by avoiding false-sharing. Java Agent Java agent supports JDK 9 - 12, but don\u0026rsquo;t support Java Module yet. Support JVM class auto instrumentation, cataloged as bootstrap plugin. Support JVM HttpClient and HttpsClient plugin.[Optional] Support backend upgrade without rebooting required. Open Redefine and Retransform by other agents. Support Servlet 2.5 in Jetty, Tomcat and SpringMVC plugins. Support Spring @Async plugin. Add new config item to restrict the length of span#peer. Refactor ContextManager#stopSpan. Add gRPC timeout. Support Logback AsyncAppender print tid Fix gRPC reconnect bug. Fix trace segment service doesn\u0026rsquo;t report onComplete. Fix wrong logger class name. Fix gRPC plugin bug. Fix ContextManager.activeSpan() API usage error. Backend Support agent reset command downstream when the storage is erased, mostly because of backend upgrade. Backend stream flow refactor. High dimensionality metrics(Hour/Day/Month) are changed to lower priority, to ease the storage payload. Add OAP metrics cache to ease the storage query payload and improve performance. Remove DataCarrier in trace persistent of ElasticSearch storage, by leveraging the elasticsearch bulk queue. OAP internal communication protocol changed. Don\u0026rsquo;t be compatible with old releases. Improve ElasticSearch storage bulk performance. Support etcd as dynamic configuration center. Simplify the PxxMetrics and ThermodynamicMetrics functions for better performance and GC. Support JVM metrics self observability. Add the new OAL runtime engine. Add gRPC timeout. Add Charset in the alarm web hook. Fix buffer lost. Fix dirty read in ElasticSearch storage. Fix bug of cluster management plugins in un-Mixed mode. Fix wrong logger class name. Fix delete bug in ElasticSearch when using namespace. Fix MySQL TTL failure. Totally remove IDs can't be null log, to avoid misleading. Fix provider has been initialized repeatedly. Adjust providers conflict log message. Fix using wrong gc time metrics in OAL. UI Fix refresh is not working after endpoint and instance changed. Fix endpoint selector but. Fix wrong copy value in slow traces. Fix can\u0026rsquo;t show trace when it is broken partially(Because of agent sampling or fail safe). Fix database and response time graph bugs. Document Add bootstrap plugin development document. Alarm documentation typo fixed. Clarify the Docker file purpose. Fix a license typo. All issues and pull requests are here\n6.2.0 Project ElasticSearch implementation performance improved, and CHANGED totally. Must delete all existing indexes to do upgrade. CI and Integration tests provided by ASF INFRA. Plan to enhance tests including e2e, plugin tests in all pull requests, powered by ASF INFRA. DataCarrier queue write index controller performance improvement. 3-5 times quicker than before. Add windows compile support in CI. Java Agent Support collect SQL parameter in MySQL plugin.[Optional] Support SolrJ plugin. Support RESTEasy plugin. Support Spring Gateway plugin for 2.1.x[Optional] TracingContext performance improvement. Support Apache ShardingSphere(incubating) plugin. Support span#error in application toolkit. Fix OOM by empty stack of exception. FIx wrong cause exception of stack in span log. Fix unclear the running context in SpringMVC plugin. Fix CPU usage accessor calculation issue. Fix SpringMVC plugin span not stop bug when doing HTTP forward. Fix lettuce plugin async commend bug and NPE. Fix webflux plugin cast exception. [CI]Support import check. Backend Support time serious ElasticSearch storage. Provide dynamic configuration module and implementation. Slow SQL threshold supports dynamic config today. Dynamic Configuration module provide multiple implementations, DCS(gRPC based), Zookeeper, Apollo, Nacos. Provide P99/95/90/75/50 charts in topology edge. New topology query protocol and implementation. Support Envoy ALS in Service Mesh scenario. Support Nacos cluster management. Enhance metric exporter. Run in increment and total modes. Fix module provider is loaded repeatedly. Change TOP slow SQL storage in ES to Text from Keyword, as too long text issue. Fix H2TopologyQuery tiny bug. Fix H2 log query bug.(No feature provided yet) Filtering pods not in \u0026lsquo;Running\u0026rsquo; phase in mesh scenario. Fix query alarm bug in MySQL and H2 storage. Codes refactor. UI Fix some ID is null query(s). Page refactor, especially time-picker, more friendly. Login removed. Trace timestamp visualization issue fixed. Provide P99/95/90/75/50 charts in topology edge. Change all P99/95/90/75/50 charts style. More readable. Fix 404 in trace page. Document Go2Sky project has been donated to SkyAPM, change document link. Add FAQ for ElasticSearch storage, and links from document. Add FAQ fro WebSphere installation. Add several open users. Add alarm webhook document. All issues and pull requests are here\n6.1.0 Project SkyWalking graduated as Apache Top Level Project.\nSupport compiling project agent, backend, UI separately. Java Agent Support Vert.x Core 3.x plugin. Support Apache Dubbo plugin. Support use_qualified_name_as_endpoint_name and use_qualified_name_as_operation_name configs in SpringMVC plugin. Support span async close APIs in core. Used in Vert.x plugin. Support MySQL 5,8 plugins. Support set instance id manually(optional). Support customize enhance trace plugin in optional list. Support to set peer in Entry Span. Support Zookeeper plugin. Fix Webflux plugin created unexpected Entry Span. Fix Kafka plugin NPE in Kafka 1.1+ Fix wrong operation name in postgre 8.x plugin. Fix RabbitMQ plugin NPE. Fix agent can\u0026rsquo;t run in JVM 6/7, remove module-info.class. Fix agent can\u0026rsquo;t work well, if there is whitespace in agent path. Fix Spring annotation bug and inheritance enhance issue. Fix CPU accessor bug. Backend Performance improved, especially in CPU limited environment. 3x improvement in service mesh scenario(no trace) in 8C16G VM. Significantly cost less CPU in low payload.\nSupport database metrics and SLOW SQL detection. Support to set max size of metadata query. And change default to 5000 from 100. Support ElasticSearch template for new feature in the future. Support shutdown Zipkin trace analysis, because it doesn\u0026rsquo;t fit production environment. Support log type, scope HTTP_ACCESS_LOG and query. No feature provided, prepare for future versions. Support .NET clr receiver. Support Jaeger trace format, no analysis. Support group endpoint name by regax rules in mesh receiver. Support disable statement in OAL. Support basic auth in ElasticSearch connection. Support metrics exporter module and gRPC implementor. Support \u0026gt;, \u0026lt;, \u0026gt;=, \u0026lt;= in OAL. Support role mode in backend. Support Envoy metrics. Support query segment by service instance. Support to set host/port manually at cluster coordinator, rather than based on core settings. Make sure OAP shutdown when it faces startup error. Support set separated gRPC/Jetty ip:port for receiver, default still use core settings. Fix JVM receiver bug. Fix wrong dest service in mesh analysis. Fix search doesn\u0026rsquo;t work as expected. Refactor ScopeDeclaration annotation. Refactor register lock mechanism. Add SmartSql component for .NET Add integration tests for ElasticSearch client. Add test cases for exporter. Add test cases for queue consume. UI RocketBot UI has been accepted and bind in this release. Support CLR metrics. Document Documents updated, matching Top Level Project requirement. UI licenses updated, according to RocketBot UI IP clearance. User wall and powered-by list updated. CN documents removed, only consider to provide by volunteer out of Apache. All issues and pull requests are here\n6.0.0-GA Java Agent Support gson plugin(optional). Support canal plugin. Fix missing ojdbc component id. Fix dubbo plugin conflict. Fix OpenTracing tag match bug. Fix a missing check in ignore plugin. Backend Adjust service inventory entity, to add properties. Adjust service instance inventory entity, to add properties. Add nodeType to service inventory entity. Fix when operation name of local and exit spans in ref, the segment lost. Fix the index names don\u0026rsquo;t show right in logs. Fix wrong alarm text. Add test case for span limit mechanism. Add telemetry module and prometheus implementation, with grafana setting. A refactor for register API in storage module. Fix H2 and MySQL endpoint dependency map miss upstream side. Optimize the inventory register and refactor the implementation. Speed up the trace buffer read. Fix and removed unnecessary inventory register operations. UI Add new trace view. Add word-break to tag value. Document Add two startup modes document. Add PHP agent links. Add some cn documents. Update year to 2019 User wall updated. Fix a wrong description in how-to-build doc. All issues and pull requests are here\n6.0.0-beta Protocol Provide Trace Data Protocol v2 Provide SkyWalking Cross Process Propagation Headers Protocol v2. Java Agent Support Trace Data Protocol v2 Support SkyWalking Cross Process Propagation Headers Protocol v2. Support SkyWalking Cross Process Propagation Headers Protocol v1 running in compatible way. Need declare open explicitly. Support SpringMVC 5 Support webflux Support a new way to override agent.config by system env. Span tag can override by explicit way. Fix Spring Controller Inherit issue. Fix ElasticSearch plugin NPE. Fix agent classloader dead lock in certain situation. Fix agent log typo. Fix wrong component id in resettemplete plugin. Fix use transform ignore() in wrong way. Fix H2 query bug. Backend Support Trace Data Protocol v2. And Trace Data Protocol v1 is still supported. Support MySQL as storage. Support TiDB as storage. Support a new way to override application.yml by system env. Support service instance and endpoint alarm. Support namespace in istio receiver. Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Support backend trace sampling. Support Zipkin format again. Support init mode. Support namespace in Zookeeper cluster management. Support consul plugin in cluster module. OAL generate tool has been integrated into main repo, in the maven compile stage. Optimize trace paging query. Fix trace query don\u0026rsquo;t use fuzzy query in ElasticSearch storage. Fix alarm can\u0026rsquo;t be active in right way. Fix unnecessary condition in database and cache number query. Fix wrong namespace bug in ElasticSearch storage. Fix Remote clients selector error: / by zero . Fix segment TTL is not working. UI Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Fix TopN endpoint link doesn\u0026rsquo;t work right. Fix trace stack style. Fix CI. Document Add more agent setting documents. Add more contribution documents. Update user wall and powered-by page. Add RocketBot UI project link in document. All issues and pull requests are here\n6.0.0-alpha SkyWalking 6 is totally new milestone for the project. At this point, we are not just a distributing tracing system with analysis and visualization capabilities. We are an Observability Analysis Platform(OAL).\nThe core and most important features in v6 are\nSupport to collect telemetry data from different sources, such as multiple language agents and service mesh. Extensible stream analysis core. Make SQL and cache analysis available in core level, although haven\u0026rsquo;t provided in this release. Provide Observability Analysis Language(OAL) to make analysis metrics customization available. New GraphQL query protocol. Not binding with UI now. UI topology is better now. New alarm core provided. In alpha, only on service related metrics. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"660\"\u003e6.6.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e[\u003cstrong\u003eIMPORTANT\u003c/strong\u003e] Local span and exit span are not treated as endpoint detected at client …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/next/en/changes/changes-6.x/","title":"6.6.0"},{"body":"6.6.0 Project [IMPORTANT] Local span and exit span are not treated as endpoint detected at client and local. Only entry span is the endpoint. Reduce the load of register and memory cost. Support MiniKube, Istio and SkyWalking on K8s deployment in CI. Support Windows and MacOS build in GitHub Action CI. Support ElasticSearch 7 in official dist. Hundreds plugin cases have been added in GitHub Action CI process. Java Agent Remove the local/exit span operation name register mechanism. Add plugin for JDK Threading classes. Add plugin for Armeria. Support set operation name in async span. Enhance webflux plugin, related to Spring Gateway plugin. Webflux plugin is in optional, due to JDK8 required. Fix a possible deadlock. Fix NPE when OAL scripts are different in different OAP nodes, mostly in upgrading stage. Fix bug about wrong peer in ES plugin. Fix NPE in Spring plugin. Fix wrong class name in Dubbo 2.7 conflict patch. Fix spring annotation inheritance problem. OAP-Backend Remove the local/exit span operation name register mechanism. Remove client side endpoint register in service mesh. Service instance dependency and related metrics. Support min func in OAL Support apdex func in OAL Support custom ES config setting at the index level. Envoy ALS proto upgraded. Update JODA lib as bugs in UTC +13/+14. Support topN sample period configurable. Ignore no statement DB operations in slow SQL collection. Fix bug in docker-entrypoint.sh when using MySQL as storage UI Service topology enhancement. Dive into service, instance and endpoint metrics on topo map. Service instance dependency view and related metrics. Support using URL parameter in trace query page. Support apdex score in service page. Add service dependency metrics into metrics comparison. Fix alarm search not working. Document Update user list and user wall. Add document link for CLI. Add deployment guide of agent in Jetty case. Modify Consul cluster doc. Add document about injecting traceId into the logback with logstack in JSON format. ElementUI license and dependency added. All issues and pull requests are here\n6.5.0 Project TTL E2E test (#3437) Test coverage is back in pull request check status (#3503) Plugin tests begin to be migrated into main repo, and is in process. (#3528, #3756, #3751, etc.) Switch to SkyWalking CI (exclusive) nodes (#3546) MySQL storage e2e test. (#3648) E2E tests are verified in multiple jdk versions, jdk 8, 9, 11, 12 (#3657) Jenkins build jobs run only when necessary (#3662) OAP-Backend Support dynamically configure alarm settings (#3557) Language of instance could be null (#3485) Make query max window size configurable. (#3765) Remove two max size 500 limit. (#3748) Parameterize the cache size. (#3741) ServiceInstanceRelation set error id (#3683) Makes the scope of alarm message more semantic. (#3680) Add register persistent worker latency metrics (#3677) Fix more reasonable error (#3619) Add GraphQL getServiceInstance instanceUuid field. (#3595) Support namespace in Nacos cluster/configuration (#3578) Instead of datasource-settings.properties, use application.yml for MySQLStorageProvider (#3564) Provide consul dynamic configuration center implementation (#3560) Upgrade guava version to support higher jdk version (#3541) Sync latest als from envoy api (#3507) Set telemetry instanced id for Etcd and Nacos plugin (#3492) Support timeout configuration in agent and backend. (#3491) Make sure the cluster register happens before streaming process. (#3471) Agent supports custom properties. (#3367) Miscellaneous bug fixes (#3567) UI Feature: node detail display in topo circle-chart view. BugFix: the jvm-maxheap \u0026amp; jvm-maxnonheap is -1, free is no value Fix bug: time select operation not in effect Fix bug: language initialization failed Fix bug: not show instance language Feature: support the trace list display export png Feature: Metrics comparison view BugFix: Fix dashboard top throughput copy Java Agent Spring async scenario optimize (#3723) Support log4j2 AsyncLogger (#3715) Add config to collect PostgreSQL sql query params (#3695) Support namespace in Nacos cluster/configuration (#3578) Provide plugin for ehcache 2.x (#3575) Supporting RequestRateLimiterGatewayFilterFactory (#3538) Kafka-plugin compatible with KafkaTemplate (#3505) Add pulsar apm plugin (#3476) Spring-cloud-gateway traceId does not transmit #3411 (#3446) Gateway compatible with downstream loss (#3445) Provide cassandra java driver 3.x plugin (#3410) Fix SpringMVC4 NoSuchMethodError (#3408) BugFix: endpoint grouping rules may be not unique (#3510) Add feature to control the maximum agent log files (#3475) Agent support custom properties. (#3367) Add Light4j plugin (#3323) Document Remove travis badge (#3763) Replace user wall to typical users in readme page (#3719) Update istio docs according latest istio release (#3646) Use chart deploy sw docs (#3573) Reorganize the doc, and provide catalog (#3563) Committer vote and set up document. (#3496) Update als setup doc as istio 1.3 released (#3470) Fill faq reply in official document. (#3450) All issues and pull requests are here\n6.4.0 Project Highly recommend to upgrade due to Pxx metrics calculation bug. Make agent working in JDK9+ Module system. Java Agent Make agent working in JDK9+ Module system. Support Kafka 2.x client libs. Log error in OKHTTP OnFailure callback. Support injecting traceid into logstack appender in logback. Add OperationName(including endpoint name) length max threshold. Support using Regex to group operation name. Support Undertow routing handler. RestTemplate plugin support operation name grouping. Fix ClassCastException in Webflux plugin. Ordering zookeeper server list, to make it better in topology. Fix a Dubbo plugin incompatible issue. Fix MySQL 5 plugin issue. Make log writer cached. Optimize Spring Cloud Gateway plugin Fix and improve gRPC reconnect mechanism. Remove Disruptor dependency from agent. Backend Fix Pxx(p50,p75,p90,p95,p99) metrics func bug.(Critical) Support Gateway in backend analysis, even when it doesn\u0026rsquo;t have suitable language agent. Support using HTTPs SSL accessing ElasticSearch storage. Support Zookeeper ACL. Make alarm records listed in order. Fix Pxx data persistence failure in some cases. Fix some bugs in MySQL storage. Setup slow SQL length threshold. Fix TTL settings is not working as expected. Remove scope-meta file. UI Enhance alarm page layout. Support trace tree chart resize. Support trace auto completion when partial traces abandoned somehow. Fix dashboard endpoint slow chart. Add radial chart in topology page. Add trace table mode. Fix topology page bug. Fix calender js bug. Fix \u0026ldquo;The \u0026ldquo;topo-services\u0026rdquo; component did not update the data in time after modifying the time range on the topology page. Document Restore the broken Istio setup doc. Add etcd config center document. Correct span_limit_per_segment default value in document. Enhance plugin develop doc. Fix error description in build document. All issues and pull requests are here\n6.3.0 Project e2e tests have been added, and verify every pull request. Use ArrayList to replace LinkedList in DataCarrier for much better performance. Add plugin instrumentation definition check in CI. DataCarrier performance improvement by avoiding false-sharing. Java Agent Java agent supports JDK 9 - 12, but don\u0026rsquo;t support Java Module yet. Support JVM class auto instrumentation, cataloged as bootstrap plugin. Support JVM HttpClient and HttpsClient plugin.[Optional] Support backend upgrade without rebooting required. Open Redefine and Retransform by other agents. Support Servlet 2.5 in Jetty, Tomcat and SpringMVC plugins. Support Spring @Async plugin. Add new config item to restrict the length of span#peer. Refactor ContextManager#stopSpan. Add gRPC timeout. Support Logback AsyncAppender print tid Fix gRPC reconnect bug. Fix trace segment service doesn\u0026rsquo;t report onComplete. Fix wrong logger class name. Fix gRPC plugin bug. Fix ContextManager.activeSpan() API usage error. Backend Support agent reset command downstream when the storage is erased, mostly because of backend upgrade. Backend stream flow refactor. High dimensionality metrics(Hour/Day/Month) are changed to lower priority, to ease the storage payload. Add OAP metrics cache to ease the storage query payload and improve performance. Remove DataCarrier in trace persistent of ElasticSearch storage, by leveraging the elasticsearch bulk queue. OAP internal communication protocol changed. Don\u0026rsquo;t be compatible with old releases. Improve ElasticSearch storage bulk performance. Support etcd as dynamic configuration center. Simplify the PxxMetrics and ThermodynamicMetrics functions for better performance and GC. Support JVM metrics self observability. Add the new OAL runtime engine. Add gRPC timeout. Add Charset in the alarm web hook. Fix buffer lost. Fix dirty read in ElasticSearch storage. Fix bug of cluster management plugins in un-Mixed mode. Fix wrong logger class name. Fix delete bug in ElasticSearch when using namespace. Fix MySQL TTL failure. Totally remove IDs can't be null log, to avoid misleading. Fix provider has been initialized repeatedly. Adjust providers conflict log message. Fix using wrong gc time metrics in OAL. UI Fix refresh is not working after endpoint and instance changed. Fix endpoint selector but. Fix wrong copy value in slow traces. Fix can\u0026rsquo;t show trace when it is broken partially(Because of agent sampling or fail safe). Fix database and response time graph bugs. Document Add bootstrap plugin development document. Alarm documentation typo fixed. Clarify the Docker file purpose. Fix a license typo. All issues and pull requests are here\n6.2.0 Project ElasticSearch implementation performance improved, and CHANGED totally. Must delete all existing indexes to do upgrade. CI and Integration tests provided by ASF INFRA. Plan to enhance tests including e2e, plugin tests in all pull requests, powered by ASF INFRA. DataCarrier queue write index controller performance improvement. 3-5 times quicker than before. Add windows compile support in CI. Java Agent Support collect SQL parameter in MySQL plugin.[Optional] Support SolrJ plugin. Support RESTEasy plugin. Support Spring Gateway plugin for 2.1.x[Optional] TracingContext performance improvement. Support Apache ShardingSphere(incubating) plugin. Support span#error in application toolkit. Fix OOM by empty stack of exception. FIx wrong cause exception of stack in span log. Fix unclear the running context in SpringMVC plugin. Fix CPU usage accessor calculation issue. Fix SpringMVC plugin span not stop bug when doing HTTP forward. Fix lettuce plugin async commend bug and NPE. Fix webflux plugin cast exception. [CI]Support import check. Backend Support time serious ElasticSearch storage. Provide dynamic configuration module and implementation. Slow SQL threshold supports dynamic config today. Dynamic Configuration module provide multiple implementations, DCS(gRPC based), Zookeeper, Apollo, Nacos. Provide P99/95/90/75/50 charts in topology edge. New topology query protocol and implementation. Support Envoy ALS in Service Mesh scenario. Support Nacos cluster management. Enhance metric exporter. Run in increment and total modes. Fix module provider is loaded repeatedly. Change TOP slow SQL storage in ES to Text from Keyword, as too long text issue. Fix H2TopologyQuery tiny bug. Fix H2 log query bug.(No feature provided yet) Filtering pods not in \u0026lsquo;Running\u0026rsquo; phase in mesh scenario. Fix query alarm bug in MySQL and H2 storage. Codes refactor. UI Fix some ID is null query(s). Page refactor, especially time-picker, more friendly. Login removed. Trace timestamp visualization issue fixed. Provide P99/95/90/75/50 charts in topology edge. Change all P99/95/90/75/50 charts style. More readable. Fix 404 in trace page. Document Go2Sky project has been donated to SkyAPM, change document link. Add FAQ for ElasticSearch storage, and links from document. Add FAQ fro WebSphere installation. Add several open users. Add alarm webhook document. All issues and pull requests are here\n6.1.0 Project SkyWalking graduated as Apache Top Level Project.\nSupport compiling project agent, backend, UI separately. Java Agent Support Vert.x Core 3.x plugin. Support Apache Dubbo plugin. Support use_qualified_name_as_endpoint_name and use_qualified_name_as_operation_name configs in SpringMVC plugin. Support span async close APIs in core. Used in Vert.x plugin. Support MySQL 5,8 plugins. Support set instance id manually(optional). Support customize enhance trace plugin in optional list. Support to set peer in Entry Span. Support Zookeeper plugin. Fix Webflux plugin created unexpected Entry Span. Fix Kafka plugin NPE in Kafka 1.1+ Fix wrong operation name in postgre 8.x plugin. Fix RabbitMQ plugin NPE. Fix agent can\u0026rsquo;t run in JVM 6/7, remove module-info.class. Fix agent can\u0026rsquo;t work well, if there is whitespace in agent path. Fix Spring annotation bug and inheritance enhance issue. Fix CPU accessor bug. Backend Performance improved, especially in CPU limited environment. 3x improvement in service mesh scenario(no trace) in 8C16G VM. Significantly cost less CPU in low payload.\nSupport database metrics and SLOW SQL detection. Support to set max size of metadata query. And change default to 5000 from 100. Support ElasticSearch template for new feature in the future. Support shutdown Zipkin trace analysis, because it doesn\u0026rsquo;t fit production environment. Support log type, scope HTTP_ACCESS_LOG and query. No feature provided, prepare for future versions. Support .NET clr receiver. Support Jaeger trace format, no analysis. Support group endpoint name by regax rules in mesh receiver. Support disable statement in OAL. Support basic auth in ElasticSearch connection. Support metrics exporter module and gRPC implementor. Support \u0026gt;, \u0026lt;, \u0026gt;=, \u0026lt;= in OAL. Support role mode in backend. Support Envoy metrics. Support query segment by service instance. Support to set host/port manually at cluster coordinator, rather than based on core settings. Make sure OAP shutdown when it faces startup error. Support set separated gRPC/Jetty ip:port for receiver, default still use core settings. Fix JVM receiver bug. Fix wrong dest service in mesh analysis. Fix search doesn\u0026rsquo;t work as expected. Refactor ScopeDeclaration annotation. Refactor register lock mechanism. Add SmartSql component for .NET Add integration tests for ElasticSearch client. Add test cases for exporter. Add test cases for queue consume. UI RocketBot UI has been accepted and bind in this release. Support CLR metrics. Document Documents updated, matching Top Level Project requirement. UI licenses updated, according to RocketBot UI IP clearance. User wall and powered-by list updated. CN documents removed, only consider to provide by volunteer out of Apache. All issues and pull requests are here\n6.0.0-GA Java Agent Support gson plugin(optional). Support canal plugin. Fix missing ojdbc component id. Fix dubbo plugin conflict. Fix OpenTracing tag match bug. Fix a missing check in ignore plugin. Backend Adjust service inventory entity, to add properties. Adjust service instance inventory entity, to add properties. Add nodeType to service inventory entity. Fix when operation name of local and exit spans in ref, the segment lost. Fix the index names don\u0026rsquo;t show right in logs. Fix wrong alarm text. Add test case for span limit mechanism. Add telemetry module and prometheus implementation, with grafana setting. A refactor for register API in storage module. Fix H2 and MySQL endpoint dependency map miss upstream side. Optimize the inventory register and refactor the implementation. Speed up the trace buffer read. Fix and removed unnecessary inventory register operations. UI Add new trace view. Add word-break to tag value. Document Add two startup modes document. Add PHP agent links. Add some cn documents. Update year to 2019 User wall updated. Fix a wrong description in how-to-build doc. All issues and pull requests are here\n6.0.0-beta Protocol Provide Trace Data Protocol v2 Provide SkyWalking Cross Process Propagation Headers Protocol v2. Java Agent Support Trace Data Protocol v2 Support SkyWalking Cross Process Propagation Headers Protocol v2. Support SkyWalking Cross Process Propagation Headers Protocol v1 running in compatible way. Need declare open explicitly. Support SpringMVC 5 Support webflux Support a new way to override agent.config by system env. Span tag can override by explicit way. Fix Spring Controller Inherit issue. Fix ElasticSearch plugin NPE. Fix agent classloader dead lock in certain situation. Fix agent log typo. Fix wrong component id in resettemplete plugin. Fix use transform ignore() in wrong way. Fix H2 query bug. Backend Support Trace Data Protocol v2. And Trace Data Protocol v1 is still supported. Support MySQL as storage. Support TiDB as storage. Support a new way to override application.yml by system env. Support service instance and endpoint alarm. Support namespace in istio receiver. Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Support backend trace sampling. Support Zipkin format again. Support init mode. Support namespace in Zookeeper cluster management. Support consul plugin in cluster module. OAL generate tool has been integrated into main repo, in the maven compile stage. Optimize trace paging query. Fix trace query don\u0026rsquo;t use fuzzy query in ElasticSearch storage. Fix alarm can\u0026rsquo;t be active in right way. Fix unnecessary condition in database and cache number query. Fix wrong namespace bug in ElasticSearch storage. Fix Remote clients selector error: / by zero . Fix segment TTL is not working. UI Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Fix TopN endpoint link doesn\u0026rsquo;t work right. Fix trace stack style. Fix CI. Document Add more agent setting documents. Add more contribution documents. Update user wall and powered-by page. Add RocketBot UI project link in document. All issues and pull requests are here\n6.0.0-alpha SkyWalking 6 is totally new milestone for the project. At this point, we are not just a distributing tracing system with analysis and visualization capabilities. We are an Observability Analysis Platform(OAL).\nThe core and most important features in v6 are\nSupport to collect telemetry data from different sources, such as multiple language agents and service mesh. Extensible stream analysis core. Make SQL and cache analysis available in core level, although haven\u0026rsquo;t provided in this release. Provide Observability Analysis Language(OAL) to make analysis metrics customization available. New GraphQL query protocol. Not binding with UI now. UI topology is better now. New alarm core provided. In alpha, only on service related metrics. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"660\"\u003e6.6.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e[\u003cstrong\u003eIMPORTANT\u003c/strong\u003e] Local span and exit span are not treated as endpoint detected at client …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.0.0/en/changes/changes-6.x/","title":"6.6.0"},{"body":"6.6.0 Project [IMPORTANT] Local span and exit span are not treated as endpoint detected at client and local. Only entry span is the endpoint. Reduce the load of register and memory cost. Support MiniKube, Istio and SkyWalking on K8s deployment in CI. Support Windows and MacOS build in GitHub Action CI. Support ElasticSearch 7 in official dist. Hundreds plugin cases have been added in GitHub Action CI process. Java Agent Remove the local/exit span operation name register mechanism. Add plugin for JDK Threading classes. Add plugin for Armeria. Support set operation name in async span. Enhance webflux plugin, related to Spring Gateway plugin. Webflux plugin is in optional, due to JDK8 required. Fix a possible deadlock. Fix NPE when OAL scripts are different in different OAP nodes, mostly in upgrading stage. Fix bug about wrong peer in ES plugin. Fix NPE in Spring plugin. Fix wrong class name in Dubbo 2.7 conflict patch. Fix spring annotation inheritance problem. OAP-Backend Remove the local/exit span operation name register mechanism. Remove client side endpoint register in service mesh. Service instance dependency and related metrics. Support min func in OAL Support apdex func in OAL Support custom ES config setting at the index level. Envoy ALS proto upgraded. Update JODA lib as bugs in UTC +13/+14. Support topN sample period configurable. Ignore no statement DB operations in slow SQL collection. Fix bug in docker-entrypoint.sh when using MySQL as storage UI Service topology enhancement. Dive into service, instance and endpoint metrics on topo map. Service instance dependency view and related metrics. Support using URL parameter in trace query page. Support apdex score in service page. Add service dependency metrics into metrics comparison. Fix alarm search not working. Document Update user list and user wall. Add document link for CLI. Add deployment guide of agent in Jetty case. Modify Consul cluster doc. Add document about injecting traceId into the logback with logstack in JSON format. ElementUI license and dependency added. All issues and pull requests are here\n6.5.0 Project TTL E2E test (#3437) Test coverage is back in pull request check status (#3503) Plugin tests begin to be migrated into main repo, and is in process. (#3528, #3756, #3751, etc.) Switch to SkyWalking CI (exclusive) nodes (#3546) MySQL storage e2e test. (#3648) E2E tests are verified in multiple jdk versions, jdk 8, 9, 11, 12 (#3657) Jenkins build jobs run only when necessary (#3662) OAP-Backend Support dynamically configure alarm settings (#3557) Language of instance could be null (#3485) Make query max window size configurable. (#3765) Remove two max size 500 limit. (#3748) Parameterize the cache size. (#3741) ServiceInstanceRelation set error id (#3683) Makes the scope of alarm message more semantic. (#3680) Add register persistent worker latency metrics (#3677) Fix more reasonable error (#3619) Add GraphQL getServiceInstance instanceUuid field. (#3595) Support namespace in Nacos cluster/configuration (#3578) Instead of datasource-settings.properties, use application.yml for MySQLStorageProvider (#3564) Provide consul dynamic configuration center implementation (#3560) Upgrade guava version to support higher jdk version (#3541) Sync latest als from envoy api (#3507) Set telemetry instanced id for Etcd and Nacos plugin (#3492) Support timeout configuration in agent and backend. (#3491) Make sure the cluster register happens before streaming process. (#3471) Agent supports custom properties. (#3367) Miscellaneous bug fixes (#3567) UI Feature: node detail display in topo circle-chart view. BugFix: the jvm-maxheap \u0026amp; jvm-maxnonheap is -1, free is no value Fix bug: time select operation not in effect Fix bug: language initialization failed Fix bug: not show instance language Feature: support the trace list display export png Feature: Metrics comparison view BugFix: Fix dashboard top throughput copy Java Agent Spring async scenario optimize (#3723) Support log4j2 AsyncLogger (#3715) Add config to collect PostgreSQL sql query params (#3695) Support namespace in Nacos cluster/configuration (#3578) Provide plugin for ehcache 2.x (#3575) Supporting RequestRateLimiterGatewayFilterFactory (#3538) Kafka-plugin compatible with KafkaTemplate (#3505) Add pulsar apm plugin (#3476) Spring-cloud-gateway traceId does not transmit #3411 (#3446) Gateway compatible with downstream loss (#3445) Provide cassandra java driver 3.x plugin (#3410) Fix SpringMVC4 NoSuchMethodError (#3408) BugFix: endpoint grouping rules may be not unique (#3510) Add feature to control the maximum agent log files (#3475) Agent support custom properties. (#3367) Add Light4j plugin (#3323) Document Remove travis badge (#3763) Replace user wall to typical users in readme page (#3719) Update istio docs according latest istio release (#3646) Use chart deploy sw docs (#3573) Reorganize the doc, and provide catalog (#3563) Committer vote and set up document. (#3496) Update als setup doc as istio 1.3 released (#3470) Fill faq reply in official document. (#3450) All issues and pull requests are here\n6.4.0 Project Highly recommend to upgrade due to Pxx metrics calculation bug. Make agent working in JDK9+ Module system. Java Agent Make agent working in JDK9+ Module system. Support Kafka 2.x client libs. Log error in OKHTTP OnFailure callback. Support injecting traceid into logstack appender in logback. Add OperationName(including endpoint name) length max threshold. Support using Regex to group operation name. Support Undertow routing handler. RestTemplate plugin support operation name grouping. Fix ClassCastException in Webflux plugin. Ordering zookeeper server list, to make it better in topology. Fix a Dubbo plugin incompatible issue. Fix MySQL 5 plugin issue. Make log writer cached. Optimize Spring Cloud Gateway plugin Fix and improve gRPC reconnect mechanism. Remove Disruptor dependency from agent. Backend Fix Pxx(p50,p75,p90,p95,p99) metrics func bug.(Critical) Support Gateway in backend analysis, even when it doesn\u0026rsquo;t have suitable language agent. Support using HTTPs SSL accessing ElasticSearch storage. Support Zookeeper ACL. Make alarm records listed in order. Fix Pxx data persistence failure in some cases. Fix some bugs in MySQL storage. Setup slow SQL length threshold. Fix TTL settings is not working as expected. Remove scope-meta file. UI Enhance alarm page layout. Support trace tree chart resize. Support trace auto completion when partial traces abandoned somehow. Fix dashboard endpoint slow chart. Add radial chart in topology page. Add trace table mode. Fix topology page bug. Fix calender js bug. Fix \u0026ldquo;The \u0026ldquo;topo-services\u0026rdquo; component did not update the data in time after modifying the time range on the topology page. Document Restore the broken Istio setup doc. Add etcd config center document. Correct span_limit_per_segment default value in document. Enhance plugin develop doc. Fix error description in build document. All issues and pull requests are here\n6.3.0 Project e2e tests have been added, and verify every pull request. Use ArrayList to replace LinkedList in DataCarrier for much better performance. Add plugin instrumentation definition check in CI. DataCarrier performance improvement by avoiding false-sharing. Java Agent Java agent supports JDK 9 - 12, but don\u0026rsquo;t support Java Module yet. Support JVM class auto instrumentation, cataloged as bootstrap plugin. Support JVM HttpClient and HttpsClient plugin.[Optional] Support backend upgrade without rebooting required. Open Redefine and Retransform by other agents. Support Servlet 2.5 in Jetty, Tomcat and SpringMVC plugins. Support Spring @Async plugin. Add new config item to restrict the length of span#peer. Refactor ContextManager#stopSpan. Add gRPC timeout. Support Logback AsyncAppender print tid Fix gRPC reconnect bug. Fix trace segment service doesn\u0026rsquo;t report onComplete. Fix wrong logger class name. Fix gRPC plugin bug. Fix ContextManager.activeSpan() API usage error. Backend Support agent reset command downstream when the storage is erased, mostly because of backend upgrade. Backend stream flow refactor. High dimensionality metrics(Hour/Day/Month) are changed to lower priority, to ease the storage payload. Add OAP metrics cache to ease the storage query payload and improve performance. Remove DataCarrier in trace persistent of ElasticSearch storage, by leveraging the elasticsearch bulk queue. OAP internal communication protocol changed. Don\u0026rsquo;t be compatible with old releases. Improve ElasticSearch storage bulk performance. Support etcd as dynamic configuration center. Simplify the PxxMetrics and ThermodynamicMetrics functions for better performance and GC. Support JVM metrics self observability. Add the new OAL runtime engine. Add gRPC timeout. Add Charset in the alarm web hook. Fix buffer lost. Fix dirty read in ElasticSearch storage. Fix bug of cluster management plugins in un-Mixed mode. Fix wrong logger class name. Fix delete bug in ElasticSearch when using namespace. Fix MySQL TTL failure. Totally remove IDs can't be null log, to avoid misleading. Fix provider has been initialized repeatedly. Adjust providers conflict log message. Fix using wrong gc time metrics in OAL. UI Fix refresh is not working after endpoint and instance changed. Fix endpoint selector but. Fix wrong copy value in slow traces. Fix can\u0026rsquo;t show trace when it is broken partially(Because of agent sampling or fail safe). Fix database and response time graph bugs. Document Add bootstrap plugin development document. Alarm documentation typo fixed. Clarify the Docker file purpose. Fix a license typo. All issues and pull requests are here\n6.2.0 Project ElasticSearch implementation performance improved, and CHANGED totally. Must delete all existing indexes to do upgrade. CI and Integration tests provided by ASF INFRA. Plan to enhance tests including e2e, plugin tests in all pull requests, powered by ASF INFRA. DataCarrier queue write index controller performance improvement. 3-5 times quicker than before. Add windows compile support in CI. Java Agent Support collect SQL parameter in MySQL plugin.[Optional] Support SolrJ plugin. Support RESTEasy plugin. Support Spring Gateway plugin for 2.1.x[Optional] TracingContext performance improvement. Support Apache ShardingSphere(incubating) plugin. Support span#error in application toolkit. Fix OOM by empty stack of exception. FIx wrong cause exception of stack in span log. Fix unclear the running context in SpringMVC plugin. Fix CPU usage accessor calculation issue. Fix SpringMVC plugin span not stop bug when doing HTTP forward. Fix lettuce plugin async commend bug and NPE. Fix webflux plugin cast exception. [CI]Support import check. Backend Support time serious ElasticSearch storage. Provide dynamic configuration module and implementation. Slow SQL threshold supports dynamic config today. Dynamic Configuration module provide multiple implementations, DCS(gRPC based), Zookeeper, Apollo, Nacos. Provide P99/95/90/75/50 charts in topology edge. New topology query protocol and implementation. Support Envoy ALS in Service Mesh scenario. Support Nacos cluster management. Enhance metric exporter. Run in increment and total modes. Fix module provider is loaded repeatedly. Change TOP slow SQL storage in ES to Text from Keyword, as too long text issue. Fix H2TopologyQuery tiny bug. Fix H2 log query bug.(No feature provided yet) Filtering pods not in \u0026lsquo;Running\u0026rsquo; phase in mesh scenario. Fix query alarm bug in MySQL and H2 storage. Codes refactor. UI Fix some ID is null query(s). Page refactor, especially time-picker, more friendly. Login removed. Trace timestamp visualization issue fixed. Provide P99/95/90/75/50 charts in topology edge. Change all P99/95/90/75/50 charts style. More readable. Fix 404 in trace page. Document Go2Sky project has been donated to SkyAPM, change document link. Add FAQ for ElasticSearch storage, and links from document. Add FAQ fro WebSphere installation. Add several open users. Add alarm webhook document. All issues and pull requests are here\n6.1.0 Project SkyWalking graduated as Apache Top Level Project.\nSupport compiling project agent, backend, UI separately. Java Agent Support Vert.x Core 3.x plugin. Support Apache Dubbo plugin. Support use_qualified_name_as_endpoint_name and use_qualified_name_as_operation_name configs in SpringMVC plugin. Support span async close APIs in core. Used in Vert.x plugin. Support MySQL 5,8 plugins. Support set instance id manually(optional). Support customize enhance trace plugin in optional list. Support to set peer in Entry Span. Support Zookeeper plugin. Fix Webflux plugin created unexpected Entry Span. Fix Kafka plugin NPE in Kafka 1.1+ Fix wrong operation name in postgre 8.x plugin. Fix RabbitMQ plugin NPE. Fix agent can\u0026rsquo;t run in JVM 6/7, remove module-info.class. Fix agent can\u0026rsquo;t work well, if there is whitespace in agent path. Fix Spring annotation bug and inheritance enhance issue. Fix CPU accessor bug. Backend Performance improved, especially in CPU limited environment. 3x improvement in service mesh scenario(no trace) in 8C16G VM. Significantly cost less CPU in low payload.\nSupport database metrics and SLOW SQL detection. Support to set max size of metadata query. And change default to 5000 from 100. Support ElasticSearch template for new feature in the future. Support shutdown Zipkin trace analysis, because it doesn\u0026rsquo;t fit production environment. Support log type, scope HTTP_ACCESS_LOG and query. No feature provided, prepare for future versions. Support .NET clr receiver. Support Jaeger trace format, no analysis. Support group endpoint name by regax rules in mesh receiver. Support disable statement in OAL. Support basic auth in ElasticSearch connection. Support metrics exporter module and gRPC implementor. Support \u0026gt;, \u0026lt;, \u0026gt;=, \u0026lt;= in OAL. Support role mode in backend. Support Envoy metrics. Support query segment by service instance. Support to set host/port manually at cluster coordinator, rather than based on core settings. Make sure OAP shutdown when it faces startup error. Support set separated gRPC/Jetty ip:port for receiver, default still use core settings. Fix JVM receiver bug. Fix wrong dest service in mesh analysis. Fix search doesn\u0026rsquo;t work as expected. Refactor ScopeDeclaration annotation. Refactor register lock mechanism. Add SmartSql component for .NET Add integration tests for ElasticSearch client. Add test cases for exporter. Add test cases for queue consume. UI RocketBot UI has been accepted and bind in this release. Support CLR metrics. Document Documents updated, matching Top Level Project requirement. UI licenses updated, according to RocketBot UI IP clearance. User wall and powered-by list updated. CN documents removed, only consider to provide by volunteer out of Apache. All issues and pull requests are here\n6.0.0-GA Java Agent Support gson plugin(optional). Support canal plugin. Fix missing ojdbc component id. Fix dubbo plugin conflict. Fix OpenTracing tag match bug. Fix a missing check in ignore plugin. Backend Adjust service inventory entity, to add properties. Adjust service instance inventory entity, to add properties. Add nodeType to service inventory entity. Fix when operation name of local and exit spans in ref, the segment lost. Fix the index names don\u0026rsquo;t show right in logs. Fix wrong alarm text. Add test case for span limit mechanism. Add telemetry module and prometheus implementation, with grafana setting. A refactor for register API in storage module. Fix H2 and MySQL endpoint dependency map miss upstream side. Optimize the inventory register and refactor the implementation. Speed up the trace buffer read. Fix and removed unnecessary inventory register operations. UI Add new trace view. Add word-break to tag value. Document Add two startup modes document. Add PHP agent links. Add some cn documents. Update year to 2019 User wall updated. Fix a wrong description in how-to-build doc. All issues and pull requests are here\n6.0.0-beta Protocol Provide Trace Data Protocol v2 Provide SkyWalking Cross Process Propagation Headers Protocol v2. Java Agent Support Trace Data Protocol v2 Support SkyWalking Cross Process Propagation Headers Protocol v2. Support SkyWalking Cross Process Propagation Headers Protocol v1 running in compatible way. Need declare open explicitly. Support SpringMVC 5 Support webflux Support a new way to override agent.config by system env. Span tag can override by explicit way. Fix Spring Controller Inherit issue. Fix ElasticSearch plugin NPE. Fix agent classloader dead lock in certain situation. Fix agent log typo. Fix wrong component id in resettemplete plugin. Fix use transform ignore() in wrong way. Fix H2 query bug. Backend Support Trace Data Protocol v2. And Trace Data Protocol v1 is still supported. Support MySQL as storage. Support TiDB as storage. Support a new way to override application.yml by system env. Support service instance and endpoint alarm. Support namespace in istio receiver. Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Support backend trace sampling. Support Zipkin format again. Support init mode. Support namespace in Zookeeper cluster management. Support consul plugin in cluster module. OAL generate tool has been integrated into main repo, in the maven compile stage. Optimize trace paging query. Fix trace query don\u0026rsquo;t use fuzzy query in ElasticSearch storage. Fix alarm can\u0026rsquo;t be active in right way. Fix unnecessary condition in database and cache number query. Fix wrong namespace bug in ElasticSearch storage. Fix Remote clients selector error: / by zero . Fix segment TTL is not working. UI Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Fix TopN endpoint link doesn\u0026rsquo;t work right. Fix trace stack style. Fix CI. Document Add more agent setting documents. Add more contribution documents. Update user wall and powered-by page. Add RocketBot UI project link in document. All issues and pull requests are here\n6.0.0-alpha SkyWalking 6 is totally new milestone for the project. At this point, we are not just a distributing tracing system with analysis and visualization capabilities. We are an Observability Analysis Platform(OAL).\nThe core and most important features in v6 are\nSupport to collect telemetry data from different sources, such as multiple language agents and service mesh. Extensible stream analysis core. Make SQL and cache analysis available in core level, although haven\u0026rsquo;t provided in this release. Provide Observability Analysis Language(OAL) to make analysis metrics customization available. New GraphQL query protocol. Not binding with UI now. UI topology is better now. New alarm core provided. In alpha, only on service related metrics. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"660\"\u003e6.6.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e[\u003cstrong\u003eIMPORTANT\u003c/strong\u003e] Local span and exit span are not treated as endpoint detected at client …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.0.1/en/changes/changes-6.x/","title":"6.6.0"},{"body":"6.6.0 Project [IMPORTANT] Local span and exit span are not treated as endpoint detected at client and local. Only entry span is the endpoint. Reduce the load of register and memory cost. Support MiniKube, Istio and SkyWalking on K8s deployment in CI. Support Windows and MacOS build in GitHub Action CI. Support ElasticSearch 7 in official dist. Hundreds plugin cases have been added in GitHub Action CI process. Java Agent Remove the local/exit span operation name register mechanism. Add plugin for JDK Threading classes. Add plugin for Armeria. Support set operation name in async span. Enhance webflux plugin, related to Spring Gateway plugin. Webflux plugin is in optional, due to JDK8 required. Fix a possible deadlock. Fix NPE when OAL scripts are different in different OAP nodes, mostly in upgrading stage. Fix bug about wrong peer in ES plugin. Fix NPE in Spring plugin. Fix wrong class name in Dubbo 2.7 conflict patch. Fix spring annotation inheritance problem. OAP-Backend Remove the local/exit span operation name register mechanism. Remove client side endpoint register in service mesh. Service instance dependency and related metrics. Support min func in OAL Support apdex func in OAL Support custom ES config setting at the index level. Envoy ALS proto upgraded. Update JODA lib as bugs in UTC +13/+14. Support topN sample period configurable. Ignore no statement DB operations in slow SQL collection. Fix bug in docker-entrypoint.sh when using MySQL as storage UI Service topology enhancement. Dive into service, instance and endpoint metrics on topo map. Service instance dependency view and related metrics. Support using URL parameter in trace query page. Support apdex score in service page. Add service dependency metrics into metrics comparison. Fix alarm search not working. Document Update user list and user wall. Add document link for CLI. Add deployment guide of agent in Jetty case. Modify Consul cluster doc. Add document about injecting traceId into the logback with logstack in JSON format. ElementUI license and dependency added. All issues and pull requests are here\n6.5.0 Project TTL E2E test (#3437) Test coverage is back in pull request check status (#3503) Plugin tests begin to be migrated into main repo, and is in process. (#3528, #3756, #3751, etc.) Switch to SkyWalking CI (exclusive) nodes (#3546) MySQL storage e2e test. (#3648) E2E tests are verified in multiple jdk versions, jdk 8, 9, 11, 12 (#3657) Jenkins build jobs run only when necessary (#3662) OAP-Backend Support dynamically configure alarm settings (#3557) Language of instance could be null (#3485) Make query max window size configurable. (#3765) Remove two max size 500 limit. (#3748) Parameterize the cache size. (#3741) ServiceInstanceRelation set error id (#3683) Makes the scope of alarm message more semantic. (#3680) Add register persistent worker latency metrics (#3677) Fix more reasonable error (#3619) Add GraphQL getServiceInstance instanceUuid field. (#3595) Support namespace in Nacos cluster/configuration (#3578) Instead of datasource-settings.properties, use application.yml for MySQLStorageProvider (#3564) Provide consul dynamic configuration center implementation (#3560) Upgrade guava version to support higher jdk version (#3541) Sync latest als from envoy api (#3507) Set telemetry instanced id for Etcd and Nacos plugin (#3492) Support timeout configuration in agent and backend. (#3491) Make sure the cluster register happens before streaming process. (#3471) Agent supports custom properties. (#3367) Miscellaneous bug fixes (#3567) UI Feature: node detail display in topo circle-chart view. BugFix: the jvm-maxheap \u0026amp; jvm-maxnonheap is -1, free is no value Fix bug: time select operation not in effect Fix bug: language initialization failed Fix bug: not show instance language Feature: support the trace list display export png Feature: Metrics comparison view BugFix: Fix dashboard top throughput copy Java Agent Spring async scenario optimize (#3723) Support log4j2 AsyncLogger (#3715) Add config to collect PostgreSQL sql query params (#3695) Support namespace in Nacos cluster/configuration (#3578) Provide plugin for ehcache 2.x (#3575) Supporting RequestRateLimiterGatewayFilterFactory (#3538) Kafka-plugin compatible with KafkaTemplate (#3505) Add pulsar apm plugin (#3476) Spring-cloud-gateway traceId does not transmit #3411 (#3446) Gateway compatible with downstream loss (#3445) Provide cassandra java driver 3.x plugin (#3410) Fix SpringMVC4 NoSuchMethodError (#3408) BugFix: endpoint grouping rules may be not unique (#3510) Add feature to control the maximum agent log files (#3475) Agent support custom properties. (#3367) Add Light4j plugin (#3323) Document Remove travis badge (#3763) Replace user wall to typical users in readme page (#3719) Update istio docs according latest istio release (#3646) Use chart deploy sw docs (#3573) Reorganize the doc, and provide catalog (#3563) Committer vote and set up document. (#3496) Update als setup doc as istio 1.3 released (#3470) Fill faq reply in official document. (#3450) All issues and pull requests are here\n6.4.0 Project Highly recommend to upgrade due to Pxx metrics calculation bug. Make agent working in JDK9+ Module system. Java Agent Make agent working in JDK9+ Module system. Support Kafka 2.x client libs. Log error in OKHTTP OnFailure callback. Support injecting traceid into logstack appender in logback. Add OperationName(including endpoint name) length max threshold. Support using Regex to group operation name. Support Undertow routing handler. RestTemplate plugin support operation name grouping. Fix ClassCastException in Webflux plugin. Ordering zookeeper server list, to make it better in topology. Fix a Dubbo plugin incompatible issue. Fix MySQL 5 plugin issue. Make log writer cached. Optimize Spring Cloud Gateway plugin Fix and improve gRPC reconnect mechanism. Remove Disruptor dependency from agent. Backend Fix Pxx(p50,p75,p90,p95,p99) metrics func bug.(Critical) Support Gateway in backend analysis, even when it doesn\u0026rsquo;t have suitable language agent. Support using HTTPs SSL accessing ElasticSearch storage. Support Zookeeper ACL. Make alarm records listed in order. Fix Pxx data persistence failure in some cases. Fix some bugs in MySQL storage. Setup slow SQL length threshold. Fix TTL settings is not working as expected. Remove scope-meta file. UI Enhance alarm page layout. Support trace tree chart resize. Support trace auto completion when partial traces abandoned somehow. Fix dashboard endpoint slow chart. Add radial chart in topology page. Add trace table mode. Fix topology page bug. Fix calender js bug. Fix \u0026ldquo;The \u0026ldquo;topo-services\u0026rdquo; component did not update the data in time after modifying the time range on the topology page. Document Restore the broken Istio setup doc. Add etcd config center document. Correct span_limit_per_segment default value in document. Enhance plugin develop doc. Fix error description in build document. All issues and pull requests are here\n6.3.0 Project e2e tests have been added, and verify every pull request. Use ArrayList to replace LinkedList in DataCarrier for much better performance. Add plugin instrumentation definition check in CI. DataCarrier performance improvement by avoiding false-sharing. Java Agent Java agent supports JDK 9 - 12, but don\u0026rsquo;t support Java Module yet. Support JVM class auto instrumentation, cataloged as bootstrap plugin. Support JVM HttpClient and HttpsClient plugin.[Optional] Support backend upgrade without rebooting required. Open Redefine and Retransform by other agents. Support Servlet 2.5 in Jetty, Tomcat and SpringMVC plugins. Support Spring @Async plugin. Add new config item to restrict the length of span#peer. Refactor ContextManager#stopSpan. Add gRPC timeout. Support Logback AsyncAppender print tid Fix gRPC reconnect bug. Fix trace segment service doesn\u0026rsquo;t report onComplete. Fix wrong logger class name. Fix gRPC plugin bug. Fix ContextManager.activeSpan() API usage error. Backend Support agent reset command downstream when the storage is erased, mostly because of backend upgrade. Backend stream flow refactor. High dimensionality metrics(Hour/Day/Month) are changed to lower priority, to ease the storage payload. Add OAP metrics cache to ease the storage query payload and improve performance. Remove DataCarrier in trace persistent of ElasticSearch storage, by leveraging the elasticsearch bulk queue. OAP internal communication protocol changed. Don\u0026rsquo;t be compatible with old releases. Improve ElasticSearch storage bulk performance. Support etcd as dynamic configuration center. Simplify the PxxMetrics and ThermodynamicMetrics functions for better performance and GC. Support JVM metrics self observability. Add the new OAL runtime engine. Add gRPC timeout. Add Charset in the alarm web hook. Fix buffer lost. Fix dirty read in ElasticSearch storage. Fix bug of cluster management plugins in un-Mixed mode. Fix wrong logger class name. Fix delete bug in ElasticSearch when using namespace. Fix MySQL TTL failure. Totally remove IDs can't be null log, to avoid misleading. Fix provider has been initialized repeatedly. Adjust providers conflict log message. Fix using wrong gc time metrics in OAL. UI Fix refresh is not working after endpoint and instance changed. Fix endpoint selector but. Fix wrong copy value in slow traces. Fix can\u0026rsquo;t show trace when it is broken partially(Because of agent sampling or fail safe). Fix database and response time graph bugs. Document Add bootstrap plugin development document. Alarm documentation typo fixed. Clarify the Docker file purpose. Fix a license typo. All issues and pull requests are here\n6.2.0 Project ElasticSearch implementation performance improved, and CHANGED totally. Must delete all existing indexes to do upgrade. CI and Integration tests provided by ASF INFRA. Plan to enhance tests including e2e, plugin tests in all pull requests, powered by ASF INFRA. DataCarrier queue write index controller performance improvement. 3-5 times quicker than before. Add windows compile support in CI. Java Agent Support collect SQL parameter in MySQL plugin.[Optional] Support SolrJ plugin. Support RESTEasy plugin. Support Spring Gateway plugin for 2.1.x[Optional] TracingContext performance improvement. Support Apache ShardingSphere(incubating) plugin. Support span#error in application toolkit. Fix OOM by empty stack of exception. FIx wrong cause exception of stack in span log. Fix unclear the running context in SpringMVC plugin. Fix CPU usage accessor calculation issue. Fix SpringMVC plugin span not stop bug when doing HTTP forward. Fix lettuce plugin async commend bug and NPE. Fix webflux plugin cast exception. [CI]Support import check. Backend Support time serious ElasticSearch storage. Provide dynamic configuration module and implementation. Slow SQL threshold supports dynamic config today. Dynamic Configuration module provide multiple implementations, DCS(gRPC based), Zookeeper, Apollo, Nacos. Provide P99/95/90/75/50 charts in topology edge. New topology query protocol and implementation. Support Envoy ALS in Service Mesh scenario. Support Nacos cluster management. Enhance metric exporter. Run in increment and total modes. Fix module provider is loaded repeatedly. Change TOP slow SQL storage in ES to Text from Keyword, as too long text issue. Fix H2TopologyQuery tiny bug. Fix H2 log query bug.(No feature provided yet) Filtering pods not in \u0026lsquo;Running\u0026rsquo; phase in mesh scenario. Fix query alarm bug in MySQL and H2 storage. Codes refactor. UI Fix some ID is null query(s). Page refactor, especially time-picker, more friendly. Login removed. Trace timestamp visualization issue fixed. Provide P99/95/90/75/50 charts in topology edge. Change all P99/95/90/75/50 charts style. More readable. Fix 404 in trace page. Document Go2Sky project has been donated to SkyAPM, change document link. Add FAQ for ElasticSearch storage, and links from document. Add FAQ fro WebSphere installation. Add several open users. Add alarm webhook document. All issues and pull requests are here\n6.1.0 Project SkyWalking graduated as Apache Top Level Project.\nSupport compiling project agent, backend, UI separately. Java Agent Support Vert.x Core 3.x plugin. Support Apache Dubbo plugin. Support use_qualified_name_as_endpoint_name and use_qualified_name_as_operation_name configs in SpringMVC plugin. Support span async close APIs in core. Used in Vert.x plugin. Support MySQL 5,8 plugins. Support set instance id manually(optional). Support customize enhance trace plugin in optional list. Support to set peer in Entry Span. Support Zookeeper plugin. Fix Webflux plugin created unexpected Entry Span. Fix Kafka plugin NPE in Kafka 1.1+ Fix wrong operation name in postgre 8.x plugin. Fix RabbitMQ plugin NPE. Fix agent can\u0026rsquo;t run in JVM 6/7, remove module-info.class. Fix agent can\u0026rsquo;t work well, if there is whitespace in agent path. Fix Spring annotation bug and inheritance enhance issue. Fix CPU accessor bug. Backend Performance improved, especially in CPU limited environment. 3x improvement in service mesh scenario(no trace) in 8C16G VM. Significantly cost less CPU in low payload.\nSupport database metrics and SLOW SQL detection. Support to set max size of metadata query. And change default to 5000 from 100. Support ElasticSearch template for new feature in the future. Support shutdown Zipkin trace analysis, because it doesn\u0026rsquo;t fit production environment. Support log type, scope HTTP_ACCESS_LOG and query. No feature provided, prepare for future versions. Support .NET clr receiver. Support Jaeger trace format, no analysis. Support group endpoint name by regax rules in mesh receiver. Support disable statement in OAL. Support basic auth in ElasticSearch connection. Support metrics exporter module and gRPC implementor. Support \u0026gt;, \u0026lt;, \u0026gt;=, \u0026lt;= in OAL. Support role mode in backend. Support Envoy metrics. Support query segment by service instance. Support to set host/port manually at cluster coordinator, rather than based on core settings. Make sure OAP shutdown when it faces startup error. Support set separated gRPC/Jetty ip:port for receiver, default still use core settings. Fix JVM receiver bug. Fix wrong dest service in mesh analysis. Fix search doesn\u0026rsquo;t work as expected. Refactor ScopeDeclaration annotation. Refactor register lock mechanism. Add SmartSql component for .NET Add integration tests for ElasticSearch client. Add test cases for exporter. Add test cases for queue consume. UI RocketBot UI has been accepted and bind in this release. Support CLR metrics. Document Documents updated, matching Top Level Project requirement. UI licenses updated, according to RocketBot UI IP clearance. User wall and powered-by list updated. CN documents removed, only consider to provide by volunteer out of Apache. All issues and pull requests are here\n6.0.0-GA Java Agent Support gson plugin(optional). Support canal plugin. Fix missing ojdbc component id. Fix dubbo plugin conflict. Fix OpenTracing tag match bug. Fix a missing check in ignore plugin. Backend Adjust service inventory entity, to add properties. Adjust service instance inventory entity, to add properties. Add nodeType to service inventory entity. Fix when operation name of local and exit spans in ref, the segment lost. Fix the index names don\u0026rsquo;t show right in logs. Fix wrong alarm text. Add test case for span limit mechanism. Add telemetry module and prometheus implementation, with grafana setting. A refactor for register API in storage module. Fix H2 and MySQL endpoint dependency map miss upstream side. Optimize the inventory register and refactor the implementation. Speed up the trace buffer read. Fix and removed unnecessary inventory register operations. UI Add new trace view. Add word-break to tag value. Document Add two startup modes document. Add PHP agent links. Add some cn documents. Update year to 2019 User wall updated. Fix a wrong description in how-to-build doc. All issues and pull requests are here\n6.0.0-beta Protocol Provide Trace Data Protocol v2 Provide SkyWalking Cross Process Propagation Headers Protocol v2. Java Agent Support Trace Data Protocol v2 Support SkyWalking Cross Process Propagation Headers Protocol v2. Support SkyWalking Cross Process Propagation Headers Protocol v1 running in compatible way. Need declare open explicitly. Support SpringMVC 5 Support webflux Support a new way to override agent.config by system env. Span tag can override by explicit way. Fix Spring Controller Inherit issue. Fix ElasticSearch plugin NPE. Fix agent classloader dead lock in certain situation. Fix agent log typo. Fix wrong component id in resettemplete plugin. Fix use transform ignore() in wrong way. Fix H2 query bug. Backend Support Trace Data Protocol v2. And Trace Data Protocol v1 is still supported. Support MySQL as storage. Support TiDB as storage. Support a new way to override application.yml by system env. Support service instance and endpoint alarm. Support namespace in istio receiver. Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Support backend trace sampling. Support Zipkin format again. Support init mode. Support namespace in Zookeeper cluster management. Support consul plugin in cluster module. OAL generate tool has been integrated into main repo, in the maven compile stage. Optimize trace paging query. Fix trace query don\u0026rsquo;t use fuzzy query in ElasticSearch storage. Fix alarm can\u0026rsquo;t be active in right way. Fix unnecessary condition in database and cache number query. Fix wrong namespace bug in ElasticSearch storage. Fix Remote clients selector error: / by zero . Fix segment TTL is not working. UI Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Fix TopN endpoint link doesn\u0026rsquo;t work right. Fix trace stack style. Fix CI. Document Add more agent setting documents. Add more contribution documents. Update user wall and powered-by page. Add RocketBot UI project link in document. All issues and pull requests are here\n6.0.0-alpha SkyWalking 6 is totally new milestone for the project. At this point, we are not just a distributing tracing system with analysis and visualization capabilities. We are an Observability Analysis Platform(OAL).\nThe core and most important features in v6 are\nSupport to collect telemetry data from different sources, such as multiple language agents and service mesh. Extensible stream analysis core. Make SQL and cache analysis available in core level, although haven\u0026rsquo;t provided in this release. Provide Observability Analysis Language(OAL) to make analysis metrics customization available. New GraphQL query protocol. Not binding with UI now. UI topology is better now. New alarm core provided. In alpha, only on service related metrics. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"660\"\u003e6.6.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e[\u003cstrong\u003eIMPORTANT\u003c/strong\u003e] Local span and exit span are not treated as endpoint detected at client …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.1.0/en/changes/changes-6.x/","title":"6.6.0"},{"body":"6.6.0 Project [IMPORTANT] Local span and exit span are not treated as endpoint detected at client and local. Only entry span is the endpoint. Reduce the load of register and memory cost. Support MiniKube, Istio and SkyWalking on K8s deployment in CI. Support Windows and MacOS build in GitHub Action CI. Support ElasticSearch 7 in official dist. Hundreds plugin cases have been added in GitHub Action CI process. Java Agent Remove the local/exit span operation name register mechanism. Add plugin for JDK Threading classes. Add plugin for Armeria. Support set operation name in async span. Enhance webflux plugin, related to Spring Gateway plugin. Webflux plugin is in optional, due to JDK8 required. Fix a possible deadlock. Fix NPE when OAL scripts are different in different OAP nodes, mostly in upgrading stage. Fix bug about wrong peer in ES plugin. Fix NPE in Spring plugin. Fix wrong class name in Dubbo 2.7 conflict patch. Fix spring annotation inheritance problem. OAP-Backend Remove the local/exit span operation name register mechanism. Remove client side endpoint register in service mesh. Service instance dependency and related metrics. Support min func in OAL Support apdex func in OAL Support custom ES config setting at the index level. Envoy ALS proto upgraded. Update JODA lib as bugs in UTC +13/+14. Support topN sample period configurable. Ignore no statement DB operations in slow SQL collection. Fix bug in docker-entrypoint.sh when using MySQL as storage UI Service topology enhancement. Dive into service, instance and endpoint metrics on topo map. Service instance dependency view and related metrics. Support using URL parameter in trace query page. Support apdex score in service page. Add service dependency metrics into metrics comparison. Fix alarm search not working. Document Update user list and user wall. Add document link for CLI. Add deployment guide of agent in Jetty case. Modify Consul cluster doc. Add document about injecting traceId into the logback with logstack in JSON format. ElementUI license and dependency added. All issues and pull requests are here\n6.5.0 Project TTL E2E test (#3437) Test coverage is back in pull request check status (#3503) Plugin tests begin to be migrated into main repo, and is in process. (#3528, #3756, #3751, etc.) Switch to SkyWalking CI (exclusive) nodes (#3546) MySQL storage e2e test. (#3648) E2E tests are verified in multiple jdk versions, jdk 8, 9, 11, 12 (#3657) Jenkins build jobs run only when necessary (#3662) OAP-Backend Support dynamically configure alarm settings (#3557) Language of instance could be null (#3485) Make query max window size configurable. (#3765) Remove two max size 500 limit. (#3748) Parameterize the cache size. (#3741) ServiceInstanceRelation set error id (#3683) Makes the scope of alarm message more semantic. (#3680) Add register persistent worker latency metrics (#3677) Fix more reasonable error (#3619) Add GraphQL getServiceInstance instanceUuid field. (#3595) Support namespace in Nacos cluster/configuration (#3578) Instead of datasource-settings.properties, use application.yml for MySQLStorageProvider (#3564) Provide consul dynamic configuration center implementation (#3560) Upgrade guava version to support higher jdk version (#3541) Sync latest als from envoy api (#3507) Set telemetry instanced id for Etcd and Nacos plugin (#3492) Support timeout configuration in agent and backend. (#3491) Make sure the cluster register happens before streaming process. (#3471) Agent supports custom properties. (#3367) Miscellaneous bug fixes (#3567) UI Feature: node detail display in topo circle-chart view. BugFix: the jvm-maxheap \u0026amp; jvm-maxnonheap is -1, free is no value Fix bug: time select operation not in effect Fix bug: language initialization failed Fix bug: not show instance language Feature: support the trace list display export png Feature: Metrics comparison view BugFix: Fix dashboard top throughput copy Java Agent Spring async scenario optimize (#3723) Support log4j2 AsyncLogger (#3715) Add config to collect PostgreSQL sql query params (#3695) Support namespace in Nacos cluster/configuration (#3578) Provide plugin for ehcache 2.x (#3575) Supporting RequestRateLimiterGatewayFilterFactory (#3538) Kafka-plugin compatible with KafkaTemplate (#3505) Add pulsar apm plugin (#3476) Spring-cloud-gateway traceId does not transmit #3411 (#3446) Gateway compatible with downstream loss (#3445) Provide cassandra java driver 3.x plugin (#3410) Fix SpringMVC4 NoSuchMethodError (#3408) BugFix: endpoint grouping rules may be not unique (#3510) Add feature to control the maximum agent log files (#3475) Agent support custom properties. (#3367) Add Light4j plugin (#3323) Document Remove travis badge (#3763) Replace user wall to typical users in readme page (#3719) Update istio docs according latest istio release (#3646) Use chart deploy sw docs (#3573) Reorganize the doc, and provide catalog (#3563) Committer vote and set up document. (#3496) Update als setup doc as istio 1.3 released (#3470) Fill faq reply in official document. (#3450) All issues and pull requests are here\n6.4.0 Project Highly recommend to upgrade due to Pxx metrics calculation bug. Make agent working in JDK9+ Module system. Java Agent Make agent working in JDK9+ Module system. Support Kafka 2.x client libs. Log error in OKHTTP OnFailure callback. Support injecting traceid into logstack appender in logback. Add OperationName(including endpoint name) length max threshold. Support using Regex to group operation name. Support Undertow routing handler. RestTemplate plugin support operation name grouping. Fix ClassCastException in Webflux plugin. Ordering zookeeper server list, to make it better in topology. Fix a Dubbo plugin incompatible issue. Fix MySQL 5 plugin issue. Make log writer cached. Optimize Spring Cloud Gateway plugin Fix and improve gRPC reconnect mechanism. Remove Disruptor dependency from agent. Backend Fix Pxx(p50,p75,p90,p95,p99) metrics func bug.(Critical) Support Gateway in backend analysis, even when it doesn\u0026rsquo;t have suitable language agent. Support using HTTPs SSL accessing ElasticSearch storage. Support Zookeeper ACL. Make alarm records listed in order. Fix Pxx data persistence failure in some cases. Fix some bugs in MySQL storage. Setup slow SQL length threshold. Fix TTL settings is not working as expected. Remove scope-meta file. UI Enhance alarm page layout. Support trace tree chart resize. Support trace auto completion when partial traces abandoned somehow. Fix dashboard endpoint slow chart. Add radial chart in topology page. Add trace table mode. Fix topology page bug. Fix calender js bug. Fix \u0026ldquo;The \u0026ldquo;topo-services\u0026rdquo; component did not update the data in time after modifying the time range on the topology page. Document Restore the broken Istio setup doc. Add etcd config center document. Correct span_limit_per_segment default value in document. Enhance plugin develop doc. Fix error description in build document. All issues and pull requests are here\n6.3.0 Project e2e tests have been added, and verify every pull request. Use ArrayList to replace LinkedList in DataCarrier for much better performance. Add plugin instrumentation definition check in CI. DataCarrier performance improvement by avoiding false-sharing. Java Agent Java agent supports JDK 9 - 12, but don\u0026rsquo;t support Java Module yet. Support JVM class auto instrumentation, cataloged as bootstrap plugin. Support JVM HttpClient and HttpsClient plugin.[Optional] Support backend upgrade without rebooting required. Open Redefine and Retransform by other agents. Support Servlet 2.5 in Jetty, Tomcat and SpringMVC plugins. Support Spring @Async plugin. Add new config item to restrict the length of span#peer. Refactor ContextManager#stopSpan. Add gRPC timeout. Support Logback AsyncAppender print tid Fix gRPC reconnect bug. Fix trace segment service doesn\u0026rsquo;t report onComplete. Fix wrong logger class name. Fix gRPC plugin bug. Fix ContextManager.activeSpan() API usage error. Backend Support agent reset command downstream when the storage is erased, mostly because of backend upgrade. Backend stream flow refactor. High dimensionality metrics(Hour/Day/Month) are changed to lower priority, to ease the storage payload. Add OAP metrics cache to ease the storage query payload and improve performance. Remove DataCarrier in trace persistent of ElasticSearch storage, by leveraging the elasticsearch bulk queue. OAP internal communication protocol changed. Don\u0026rsquo;t be compatible with old releases. Improve ElasticSearch storage bulk performance. Support etcd as dynamic configuration center. Simplify the PxxMetrics and ThermodynamicMetrics functions for better performance and GC. Support JVM metrics self observability. Add the new OAL runtime engine. Add gRPC timeout. Add Charset in the alarm web hook. Fix buffer lost. Fix dirty read in ElasticSearch storage. Fix bug of cluster management plugins in un-Mixed mode. Fix wrong logger class name. Fix delete bug in ElasticSearch when using namespace. Fix MySQL TTL failure. Totally remove IDs can't be null log, to avoid misleading. Fix provider has been initialized repeatedly. Adjust providers conflict log message. Fix using wrong gc time metrics in OAL. UI Fix refresh is not working after endpoint and instance changed. Fix endpoint selector but. Fix wrong copy value in slow traces. Fix can\u0026rsquo;t show trace when it is broken partially(Because of agent sampling or fail safe). Fix database and response time graph bugs. Document Add bootstrap plugin development document. Alarm documentation typo fixed. Clarify the Docker file purpose. Fix a license typo. All issues and pull requests are here\n6.2.0 Project ElasticSearch implementation performance improved, and CHANGED totally. Must delete all existing indexes to do upgrade. CI and Integration tests provided by ASF INFRA. Plan to enhance tests including e2e, plugin tests in all pull requests, powered by ASF INFRA. DataCarrier queue write index controller performance improvement. 3-5 times quicker than before. Add windows compile support in CI. Java Agent Support collect SQL parameter in MySQL plugin.[Optional] Support SolrJ plugin. Support RESTEasy plugin. Support Spring Gateway plugin for 2.1.x[Optional] TracingContext performance improvement. Support Apache ShardingSphere(incubating) plugin. Support span#error in application toolkit. Fix OOM by empty stack of exception. FIx wrong cause exception of stack in span log. Fix unclear the running context in SpringMVC plugin. Fix CPU usage accessor calculation issue. Fix SpringMVC plugin span not stop bug when doing HTTP forward. Fix lettuce plugin async commend bug and NPE. Fix webflux plugin cast exception. [CI]Support import check. Backend Support time serious ElasticSearch storage. Provide dynamic configuration module and implementation. Slow SQL threshold supports dynamic config today. Dynamic Configuration module provide multiple implementations, DCS(gRPC based), Zookeeper, Apollo, Nacos. Provide P99/95/90/75/50 charts in topology edge. New topology query protocol and implementation. Support Envoy ALS in Service Mesh scenario. Support Nacos cluster management. Enhance metric exporter. Run in increment and total modes. Fix module provider is loaded repeatedly. Change TOP slow SQL storage in ES to Text from Keyword, as too long text issue. Fix H2TopologyQuery tiny bug. Fix H2 log query bug.(No feature provided yet) Filtering pods not in \u0026lsquo;Running\u0026rsquo; phase in mesh scenario. Fix query alarm bug in MySQL and H2 storage. Codes refactor. UI Fix some ID is null query(s). Page refactor, especially time-picker, more friendly. Login removed. Trace timestamp visualization issue fixed. Provide P99/95/90/75/50 charts in topology edge. Change all P99/95/90/75/50 charts style. More readable. Fix 404 in trace page. Document Go2Sky project has been donated to SkyAPM, change document link. Add FAQ for ElasticSearch storage, and links from document. Add FAQ fro WebSphere installation. Add several open users. Add alarm webhook document. All issues and pull requests are here\n6.1.0 Project SkyWalking graduated as Apache Top Level Project.\nSupport compiling project agent, backend, UI separately. Java Agent Support Vert.x Core 3.x plugin. Support Apache Dubbo plugin. Support use_qualified_name_as_endpoint_name and use_qualified_name_as_operation_name configs in SpringMVC plugin. Support span async close APIs in core. Used in Vert.x plugin. Support MySQL 5,8 plugins. Support set instance id manually(optional). Support customize enhance trace plugin in optional list. Support to set peer in Entry Span. Support Zookeeper plugin. Fix Webflux plugin created unexpected Entry Span. Fix Kafka plugin NPE in Kafka 1.1+ Fix wrong operation name in postgre 8.x plugin. Fix RabbitMQ plugin NPE. Fix agent can\u0026rsquo;t run in JVM 6/7, remove module-info.class. Fix agent can\u0026rsquo;t work well, if there is whitespace in agent path. Fix Spring annotation bug and inheritance enhance issue. Fix CPU accessor bug. Backend Performance improved, especially in CPU limited environment. 3x improvement in service mesh scenario(no trace) in 8C16G VM. Significantly cost less CPU in low payload.\nSupport database metrics and SLOW SQL detection. Support to set max size of metadata query. And change default to 5000 from 100. Support ElasticSearch template for new feature in the future. Support shutdown Zipkin trace analysis, because it doesn\u0026rsquo;t fit production environment. Support log type, scope HTTP_ACCESS_LOG and query. No feature provided, prepare for future versions. Support .NET clr receiver. Support Jaeger trace format, no analysis. Support group endpoint name by regax rules in mesh receiver. Support disable statement in OAL. Support basic auth in ElasticSearch connection. Support metrics exporter module and gRPC implementor. Support \u0026gt;, \u0026lt;, \u0026gt;=, \u0026lt;= in OAL. Support role mode in backend. Support Envoy metrics. Support query segment by service instance. Support to set host/port manually at cluster coordinator, rather than based on core settings. Make sure OAP shutdown when it faces startup error. Support set separated gRPC/Jetty ip:port for receiver, default still use core settings. Fix JVM receiver bug. Fix wrong dest service in mesh analysis. Fix search doesn\u0026rsquo;t work as expected. Refactor ScopeDeclaration annotation. Refactor register lock mechanism. Add SmartSql component for .NET Add integration tests for ElasticSearch client. Add test cases for exporter. Add test cases for queue consume. UI RocketBot UI has been accepted and bind in this release. Support CLR metrics. Document Documents updated, matching Top Level Project requirement. UI licenses updated, according to RocketBot UI IP clearance. User wall and powered-by list updated. CN documents removed, only consider to provide by volunteer out of Apache. All issues and pull requests are here\n6.0.0-GA Java Agent Support gson plugin(optional). Support canal plugin. Fix missing ojdbc component id. Fix dubbo plugin conflict. Fix OpenTracing tag match bug. Fix a missing check in ignore plugin. Backend Adjust service inventory entity, to add properties. Adjust service instance inventory entity, to add properties. Add nodeType to service inventory entity. Fix when operation name of local and exit spans in ref, the segment lost. Fix the index names don\u0026rsquo;t show right in logs. Fix wrong alarm text. Add test case for span limit mechanism. Add telemetry module and prometheus implementation, with grafana setting. A refactor for register API in storage module. Fix H2 and MySQL endpoint dependency map miss upstream side. Optimize the inventory register and refactor the implementation. Speed up the trace buffer read. Fix and removed unnecessary inventory register operations. UI Add new trace view. Add word-break to tag value. Document Add two startup modes document. Add PHP agent links. Add some cn documents. Update year to 2019 User wall updated. Fix a wrong description in how-to-build doc. All issues and pull requests are here\n6.0.0-beta Protocol Provide Trace Data Protocol v2 Provide SkyWalking Cross Process Propagation Headers Protocol v2. Java Agent Support Trace Data Protocol v2 Support SkyWalking Cross Process Propagation Headers Protocol v2. Support SkyWalking Cross Process Propagation Headers Protocol v1 running in compatible way. Need declare open explicitly. Support SpringMVC 5 Support webflux Support a new way to override agent.config by system env. Span tag can override by explicit way. Fix Spring Controller Inherit issue. Fix ElasticSearch plugin NPE. Fix agent classloader dead lock in certain situation. Fix agent log typo. Fix wrong component id in resettemplete plugin. Fix use transform ignore() in wrong way. Fix H2 query bug. Backend Support Trace Data Protocol v2. And Trace Data Protocol v1 is still supported. Support MySQL as storage. Support TiDB as storage. Support a new way to override application.yml by system env. Support service instance and endpoint alarm. Support namespace in istio receiver. Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Support backend trace sampling. Support Zipkin format again. Support init mode. Support namespace in Zookeeper cluster management. Support consul plugin in cluster module. OAL generate tool has been integrated into main repo, in the maven compile stage. Optimize trace paging query. Fix trace query don\u0026rsquo;t use fuzzy query in ElasticSearch storage. Fix alarm can\u0026rsquo;t be active in right way. Fix unnecessary condition in database and cache number query. Fix wrong namespace bug in ElasticSearch storage. Fix Remote clients selector error: / by zero . Fix segment TTL is not working. UI Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Fix TopN endpoint link doesn\u0026rsquo;t work right. Fix trace stack style. Fix CI. Document Add more agent setting documents. Add more contribution documents. Update user wall and powered-by page. Add RocketBot UI project link in document. All issues and pull requests are here\n6.0.0-alpha SkyWalking 6 is totally new milestone for the project. At this point, we are not just a distributing tracing system with analysis and visualization capabilities. We are an Observability Analysis Platform(OAL).\nThe core and most important features in v6 are\nSupport to collect telemetry data from different sources, such as multiple language agents and service mesh. Extensible stream analysis core. Make SQL and cache analysis available in core level, although haven\u0026rsquo;t provided in this release. Provide Observability Analysis Language(OAL) to make analysis metrics customization available. New GraphQL query protocol. Not binding with UI now. UI topology is better now. New alarm core provided. In alpha, only on service related metrics. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"660\"\u003e6.6.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e[\u003cstrong\u003eIMPORTANT\u003c/strong\u003e] Local span and exit span are not treated as endpoint detected at client …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.2.0/en/changes/changes-6.x/","title":"6.6.0"},{"body":"6.6.0 Project [IMPORTANT] Local span and exit span are not treated as endpoint detected at client and local. Only entry span is the endpoint. Reduce the load of register and memory cost. Support MiniKube, Istio and SkyWalking on K8s deployment in CI. Support Windows and MacOS build in GitHub Action CI. Support ElasticSearch 7 in official dist. Hundreds plugin cases have been added in GitHub Action CI process. Java Agent Remove the local/exit span operation name register mechanism. Add plugin for JDK Threading classes. Add plugin for Armeria. Support set operation name in async span. Enhance webflux plugin, related to Spring Gateway plugin. Webflux plugin is in optional, due to JDK8 required. Fix a possible deadlock. Fix NPE when OAL scripts are different in different OAP nodes, mostly in upgrading stage. Fix bug about wrong peer in ES plugin. Fix NPE in Spring plugin. Fix wrong class name in Dubbo 2.7 conflict patch. Fix spring annotation inheritance problem. OAP-Backend Remove the local/exit span operation name register mechanism. Remove client side endpoint register in service mesh. Service instance dependency and related metrics. Support min func in OAL Support apdex func in OAL Support custom ES config setting at the index level. Envoy ALS proto upgraded. Update JODA lib as bugs in UTC +13/+14. Support topN sample period configurable. Ignore no statement DB operations in slow SQL collection. Fix bug in docker-entrypoint.sh when using MySQL as storage UI Service topology enhancement. Dive into service, instance and endpoint metrics on topo map. Service instance dependency view and related metrics. Support using URL parameter in trace query page. Support apdex score in service page. Add service dependency metrics into metrics comparison. Fix alarm search not working. Document Update user list and user wall. Add document link for CLI. Add deployment guide of agent in Jetty case. Modify Consul cluster doc. Add document about injecting traceId into the logback with logstack in JSON format. ElementUI license and dependency added. All issues and pull requests are here\n6.5.0 Project TTL E2E test (#3437) Test coverage is back in pull request check status (#3503) Plugin tests begin to be migrated into main repo, and is in process. (#3528, #3756, #3751, etc.) Switch to SkyWalking CI (exclusive) nodes (#3546) MySQL storage e2e test. (#3648) E2E tests are verified in multiple jdk versions, jdk 8, 9, 11, 12 (#3657) Jenkins build jobs run only when necessary (#3662) OAP-Backend Support dynamically configure alarm settings (#3557) Language of instance could be null (#3485) Make query max window size configurable. (#3765) Remove two max size 500 limit. (#3748) Parameterize the cache size. (#3741) ServiceInstanceRelation set error id (#3683) Makes the scope of alarm message more semantic. (#3680) Add register persistent worker latency metrics (#3677) Fix more reasonable error (#3619) Add GraphQL getServiceInstance instanceUuid field. (#3595) Support namespace in Nacos cluster/configuration (#3578) Instead of datasource-settings.properties, use application.yml for MySQLStorageProvider (#3564) Provide consul dynamic configuration center implementation (#3560) Upgrade guava version to support higher jdk version (#3541) Sync latest als from envoy api (#3507) Set telemetry instanced id for Etcd and Nacos plugin (#3492) Support timeout configuration in agent and backend. (#3491) Make sure the cluster register happens before streaming process. (#3471) Agent supports custom properties. (#3367) Miscellaneous bug fixes (#3567) UI Feature: node detail display in topo circle-chart view. BugFix: the jvm-maxheap \u0026amp; jvm-maxnonheap is -1, free is no value Fix bug: time select operation not in effect Fix bug: language initialization failed Fix bug: not show instance language Feature: support the trace list display export png Feature: Metrics comparison view BugFix: Fix dashboard top throughput copy Java Agent Spring async scenario optimize (#3723) Support log4j2 AsyncLogger (#3715) Add config to collect PostgreSQL sql query params (#3695) Support namespace in Nacos cluster/configuration (#3578) Provide plugin for ehcache 2.x (#3575) Supporting RequestRateLimiterGatewayFilterFactory (#3538) Kafka-plugin compatible with KafkaTemplate (#3505) Add pulsar apm plugin (#3476) Spring-cloud-gateway traceId does not transmit #3411 (#3446) Gateway compatible with downstream loss (#3445) Provide cassandra java driver 3.x plugin (#3410) Fix SpringMVC4 NoSuchMethodError (#3408) BugFix: endpoint grouping rules may be not unique (#3510) Add feature to control the maximum agent log files (#3475) Agent support custom properties. (#3367) Add Light4j plugin (#3323) Document Remove travis badge (#3763) Replace user wall to typical users in readme page (#3719) Update istio docs according latest istio release (#3646) Use chart deploy sw docs (#3573) Reorganize the doc, and provide catalog (#3563) Committer vote and set up document. (#3496) Update als setup doc as istio 1.3 released (#3470) Fill faq reply in official document. (#3450) All issues and pull requests are here\n6.4.0 Project Highly recommend to upgrade due to Pxx metrics calculation bug. Make agent working in JDK9+ Module system. Java Agent Make agent working in JDK9+ Module system. Support Kafka 2.x client libs. Log error in OKHTTP OnFailure callback. Support injecting traceid into logstack appender in logback. Add OperationName(including endpoint name) length max threshold. Support using Regex to group operation name. Support Undertow routing handler. RestTemplate plugin support operation name grouping. Fix ClassCastException in Webflux plugin. Ordering zookeeper server list, to make it better in topology. Fix a Dubbo plugin incompatible issue. Fix MySQL 5 plugin issue. Make log writer cached. Optimize Spring Cloud Gateway plugin Fix and improve gRPC reconnect mechanism. Remove Disruptor dependency from agent. Backend Fix Pxx(p50,p75,p90,p95,p99) metrics func bug.(Critical) Support Gateway in backend analysis, even when it doesn\u0026rsquo;t have suitable language agent. Support using HTTPs SSL accessing ElasticSearch storage. Support Zookeeper ACL. Make alarm records listed in order. Fix Pxx data persistence failure in some cases. Fix some bugs in MySQL storage. Setup slow SQL length threshold. Fix TTL settings is not working as expected. Remove scope-meta file. UI Enhance alarm page layout. Support trace tree chart resize. Support trace auto completion when partial traces abandoned somehow. Fix dashboard endpoint slow chart. Add radial chart in topology page. Add trace table mode. Fix topology page bug. Fix calender js bug. Fix \u0026ldquo;The \u0026ldquo;topo-services\u0026rdquo; component did not update the data in time after modifying the time range on the topology page. Document Restore the broken Istio setup doc. Add etcd config center document. Correct span_limit_per_segment default value in document. Enhance plugin develop doc. Fix error description in build document. All issues and pull requests are here\n6.3.0 Project e2e tests have been added, and verify every pull request. Use ArrayList to replace LinkedList in DataCarrier for much better performance. Add plugin instrumentation definition check in CI. DataCarrier performance improvement by avoiding false-sharing. Java Agent Java agent supports JDK 9 - 12, but don\u0026rsquo;t support Java Module yet. Support JVM class auto instrumentation, cataloged as bootstrap plugin. Support JVM HttpClient and HttpsClient plugin.[Optional] Support backend upgrade without rebooting required. Open Redefine and Retransform by other agents. Support Servlet 2.5 in Jetty, Tomcat and SpringMVC plugins. Support Spring @Async plugin. Add new config item to restrict the length of span#peer. Refactor ContextManager#stopSpan. Add gRPC timeout. Support Logback AsyncAppender print tid Fix gRPC reconnect bug. Fix trace segment service doesn\u0026rsquo;t report onComplete. Fix wrong logger class name. Fix gRPC plugin bug. Fix ContextManager.activeSpan() API usage error. Backend Support agent reset command downstream when the storage is erased, mostly because of backend upgrade. Backend stream flow refactor. High dimensionality metrics(Hour/Day/Month) are changed to lower priority, to ease the storage payload. Add OAP metrics cache to ease the storage query payload and improve performance. Remove DataCarrier in trace persistent of ElasticSearch storage, by leveraging the elasticsearch bulk queue. OAP internal communication protocol changed. Don\u0026rsquo;t be compatible with old releases. Improve ElasticSearch storage bulk performance. Support etcd as dynamic configuration center. Simplify the PxxMetrics and ThermodynamicMetrics functions for better performance and GC. Support JVM metrics self observability. Add the new OAL runtime engine. Add gRPC timeout. Add Charset in the alarm web hook. Fix buffer lost. Fix dirty read in ElasticSearch storage. Fix bug of cluster management plugins in un-Mixed mode. Fix wrong logger class name. Fix delete bug in ElasticSearch when using namespace. Fix MySQL TTL failure. Totally remove IDs can't be null log, to avoid misleading. Fix provider has been initialized repeatedly. Adjust providers conflict log message. Fix using wrong gc time metrics in OAL. UI Fix refresh is not working after endpoint and instance changed. Fix endpoint selector but. Fix wrong copy value in slow traces. Fix can\u0026rsquo;t show trace when it is broken partially(Because of agent sampling or fail safe). Fix database and response time graph bugs. Document Add bootstrap plugin development document. Alarm documentation typo fixed. Clarify the Docker file purpose. Fix a license typo. All issues and pull requests are here\n6.2.0 Project ElasticSearch implementation performance improved, and CHANGED totally. Must delete all existing indexes to do upgrade. CI and Integration tests provided by ASF INFRA. Plan to enhance tests including e2e, plugin tests in all pull requests, powered by ASF INFRA. DataCarrier queue write index controller performance improvement. 3-5 times quicker than before. Add windows compile support in CI. Java Agent Support collect SQL parameter in MySQL plugin.[Optional] Support SolrJ plugin. Support RESTEasy plugin. Support Spring Gateway plugin for 2.1.x[Optional] TracingContext performance improvement. Support Apache ShardingSphere(incubating) plugin. Support span#error in application toolkit. Fix OOM by empty stack of exception. FIx wrong cause exception of stack in span log. Fix unclear the running context in SpringMVC plugin. Fix CPU usage accessor calculation issue. Fix SpringMVC plugin span not stop bug when doing HTTP forward. Fix lettuce plugin async commend bug and NPE. Fix webflux plugin cast exception. [CI]Support import check. Backend Support time serious ElasticSearch storage. Provide dynamic configuration module and implementation. Slow SQL threshold supports dynamic config today. Dynamic Configuration module provide multiple implementations, DCS(gRPC based), Zookeeper, Apollo, Nacos. Provide P99/95/90/75/50 charts in topology edge. New topology query protocol and implementation. Support Envoy ALS in Service Mesh scenario. Support Nacos cluster management. Enhance metric exporter. Run in increment and total modes. Fix module provider is loaded repeatedly. Change TOP slow SQL storage in ES to Text from Keyword, as too long text issue. Fix H2TopologyQuery tiny bug. Fix H2 log query bug.(No feature provided yet) Filtering pods not in \u0026lsquo;Running\u0026rsquo; phase in mesh scenario. Fix query alarm bug in MySQL and H2 storage. Codes refactor. UI Fix some ID is null query(s). Page refactor, especially time-picker, more friendly. Login removed. Trace timestamp visualization issue fixed. Provide P99/95/90/75/50 charts in topology edge. Change all P99/95/90/75/50 charts style. More readable. Fix 404 in trace page. Document Go2Sky project has been donated to SkyAPM, change document link. Add FAQ for ElasticSearch storage, and links from document. Add FAQ fro WebSphere installation. Add several open users. Add alarm webhook document. All issues and pull requests are here\n6.1.0 Project SkyWalking graduated as Apache Top Level Project.\nSupport compiling project agent, backend, UI separately. Java Agent Support Vert.x Core 3.x plugin. Support Apache Dubbo plugin. Support use_qualified_name_as_endpoint_name and use_qualified_name_as_operation_name configs in SpringMVC plugin. Support span async close APIs in core. Used in Vert.x plugin. Support MySQL 5,8 plugins. Support set instance id manually(optional). Support customize enhance trace plugin in optional list. Support to set peer in Entry Span. Support Zookeeper plugin. Fix Webflux plugin created unexpected Entry Span. Fix Kafka plugin NPE in Kafka 1.1+ Fix wrong operation name in postgre 8.x plugin. Fix RabbitMQ plugin NPE. Fix agent can\u0026rsquo;t run in JVM 6/7, remove module-info.class. Fix agent can\u0026rsquo;t work well, if there is whitespace in agent path. Fix Spring annotation bug and inheritance enhance issue. Fix CPU accessor bug. Backend Performance improved, especially in CPU limited environment. 3x improvement in service mesh scenario(no trace) in 8C16G VM. Significantly cost less CPU in low payload.\nSupport database metrics and SLOW SQL detection. Support to set max size of metadata query. And change default to 5000 from 100. Support ElasticSearch template for new feature in the future. Support shutdown Zipkin trace analysis, because it doesn\u0026rsquo;t fit production environment. Support log type, scope HTTP_ACCESS_LOG and query. No feature provided, prepare for future versions. Support .NET clr receiver. Support Jaeger trace format, no analysis. Support group endpoint name by regax rules in mesh receiver. Support disable statement in OAL. Support basic auth in ElasticSearch connection. Support metrics exporter module and gRPC implementor. Support \u0026gt;, \u0026lt;, \u0026gt;=, \u0026lt;= in OAL. Support role mode in backend. Support Envoy metrics. Support query segment by service instance. Support to set host/port manually at cluster coordinator, rather than based on core settings. Make sure OAP shutdown when it faces startup error. Support set separated gRPC/Jetty ip:port for receiver, default still use core settings. Fix JVM receiver bug. Fix wrong dest service in mesh analysis. Fix search doesn\u0026rsquo;t work as expected. Refactor ScopeDeclaration annotation. Refactor register lock mechanism. Add SmartSql component for .NET Add integration tests for ElasticSearch client. Add test cases for exporter. Add test cases for queue consume. UI RocketBot UI has been accepted and bind in this release. Support CLR metrics. Document Documents updated, matching Top Level Project requirement. UI licenses updated, according to RocketBot UI IP clearance. User wall and powered-by list updated. CN documents removed, only consider to provide by volunteer out of Apache. All issues and pull requests are here\n6.0.0-GA Java Agent Support gson plugin(optional). Support canal plugin. Fix missing ojdbc component id. Fix dubbo plugin conflict. Fix OpenTracing tag match bug. Fix a missing check in ignore plugin. Backend Adjust service inventory entity, to add properties. Adjust service instance inventory entity, to add properties. Add nodeType to service inventory entity. Fix when operation name of local and exit spans in ref, the segment lost. Fix the index names don\u0026rsquo;t show right in logs. Fix wrong alarm text. Add test case for span limit mechanism. Add telemetry module and prometheus implementation, with grafana setting. A refactor for register API in storage module. Fix H2 and MySQL endpoint dependency map miss upstream side. Optimize the inventory register and refactor the implementation. Speed up the trace buffer read. Fix and removed unnecessary inventory register operations. UI Add new trace view. Add word-break to tag value. Document Add two startup modes document. Add PHP agent links. Add some cn documents. Update year to 2019 User wall updated. Fix a wrong description in how-to-build doc. All issues and pull requests are here\n6.0.0-beta Protocol Provide Trace Data Protocol v2 Provide SkyWalking Cross Process Propagation Headers Protocol v2. Java Agent Support Trace Data Protocol v2 Support SkyWalking Cross Process Propagation Headers Protocol v2. Support SkyWalking Cross Process Propagation Headers Protocol v1 running in compatible way. Need declare open explicitly. Support SpringMVC 5 Support webflux Support a new way to override agent.config by system env. Span tag can override by explicit way. Fix Spring Controller Inherit issue. Fix ElasticSearch plugin NPE. Fix agent classloader dead lock in certain situation. Fix agent log typo. Fix wrong component id in resettemplete plugin. Fix use transform ignore() in wrong way. Fix H2 query bug. Backend Support Trace Data Protocol v2. And Trace Data Protocol v1 is still supported. Support MySQL as storage. Support TiDB as storage. Support a new way to override application.yml by system env. Support service instance and endpoint alarm. Support namespace in istio receiver. Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Support backend trace sampling. Support Zipkin format again. Support init mode. Support namespace in Zookeeper cluster management. Support consul plugin in cluster module. OAL generate tool has been integrated into main repo, in the maven compile stage. Optimize trace paging query. Fix trace query don\u0026rsquo;t use fuzzy query in ElasticSearch storage. Fix alarm can\u0026rsquo;t be active in right way. Fix unnecessary condition in database and cache number query. Fix wrong namespace bug in ElasticSearch storage. Fix Remote clients selector error: / by zero . Fix segment TTL is not working. UI Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Fix TopN endpoint link doesn\u0026rsquo;t work right. Fix trace stack style. Fix CI. Document Add more agent setting documents. Add more contribution documents. Update user wall and powered-by page. Add RocketBot UI project link in document. All issues and pull requests are here\n6.0.0-alpha SkyWalking 6 is totally new milestone for the project. At this point, we are not just a distributing tracing system with analysis and visualization capabilities. We are an Observability Analysis Platform(OAL).\nThe core and most important features in v6 are\nSupport to collect telemetry data from different sources, such as multiple language agents and service mesh. Extensible stream analysis core. Make SQL and cache analysis available in core level, although haven\u0026rsquo;t provided in this release. Provide Observability Analysis Language(OAL) to make analysis metrics customization available. New GraphQL query protocol. Not binding with UI now. UI topology is better now. New alarm core provided. In alpha, only on service related metrics. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"660\"\u003e6.6.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e[\u003cstrong\u003eIMPORTANT\u003c/strong\u003e] Local span and exit span are not treated as endpoint detected at client …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.3.0/en/changes/changes-6.x/","title":"6.6.0"},{"body":"6.6.0 Project [IMPORTANT] Local span and exit span are not treated as endpoint detected at client and local. Only entry span is the endpoint. Reduce the load of register and memory cost. Support MiniKube, Istio and SkyWalking on K8s deployment in CI. Support Windows and MacOS build in GitHub Action CI. Support ElasticSearch 7 in official dist. Hundreds plugin cases have been added in GitHub Action CI process. Java Agent Remove the local/exit span operation name register mechanism. Add plugin for JDK Threading classes. Add plugin for Armeria. Support set operation name in async span. Enhance webflux plugin, related to Spring Gateway plugin. Webflux plugin is in optional, due to JDK8 required. Fix a possible deadlock. Fix NPE when OAL scripts are different in different OAP nodes, mostly in upgrading stage. Fix bug about wrong peer in ES plugin. Fix NPE in Spring plugin. Fix wrong class name in Dubbo 2.7 conflict patch. Fix spring annotation inheritance problem. OAP-Backend Remove the local/exit span operation name register mechanism. Remove client side endpoint register in service mesh. Service instance dependency and related metrics. Support min func in OAL Support apdex func in OAL Support custom ES config setting at the index level. Envoy ALS proto upgraded. Update JODA lib as bugs in UTC +13/+14. Support topN sample period configurable. Ignore no statement DB operations in slow SQL collection. Fix bug in docker-entrypoint.sh when using MySQL as storage UI Service topology enhancement. Dive into service, instance and endpoint metrics on topo map. Service instance dependency view and related metrics. Support using URL parameter in trace query page. Support apdex score in service page. Add service dependency metrics into metrics comparison. Fix alarm search not working. Document Update user list and user wall. Add document link for CLI. Add deployment guide of agent in Jetty case. Modify Consul cluster doc. Add document about injecting traceId into the logback with logstack in JSON format. ElementUI license and dependency added. All issues and pull requests are here\n6.5.0 Project TTL E2E test (#3437) Test coverage is back in pull request check status (#3503) Plugin tests begin to be migrated into main repo, and is in process. (#3528, #3756, #3751, etc.) Switch to SkyWalking CI (exclusive) nodes (#3546) MySQL storage e2e test. (#3648) E2E tests are verified in multiple jdk versions, jdk 8, 9, 11, 12 (#3657) Jenkins build jobs run only when necessary (#3662) OAP-Backend Support dynamically configure alarm settings (#3557) Language of instance could be null (#3485) Make query max window size configurable. (#3765) Remove two max size 500 limit. (#3748) Parameterize the cache size. (#3741) ServiceInstanceRelation set error id (#3683) Makes the scope of alarm message more semantic. (#3680) Add register persistent worker latency metrics (#3677) Fix more reasonable error (#3619) Add GraphQL getServiceInstance instanceUuid field. (#3595) Support namespace in Nacos cluster/configuration (#3578) Instead of datasource-settings.properties, use application.yml for MySQLStorageProvider (#3564) Provide consul dynamic configuration center implementation (#3560) Upgrade guava version to support higher jdk version (#3541) Sync latest als from envoy api (#3507) Set telemetry instanced id for Etcd and Nacos plugin (#3492) Support timeout configuration in agent and backend. (#3491) Make sure the cluster register happens before streaming process. (#3471) Agent supports custom properties. (#3367) Miscellaneous bug fixes (#3567) UI Feature: node detail display in topo circle-chart view. BugFix: the jvm-maxheap \u0026amp; jvm-maxnonheap is -1, free is no value Fix bug: time select operation not in effect Fix bug: language initialization failed Fix bug: not show instance language Feature: support the trace list display export png Feature: Metrics comparison view BugFix: Fix dashboard top throughput copy Java Agent Spring async scenario optimize (#3723) Support log4j2 AsyncLogger (#3715) Add config to collect PostgreSQL sql query params (#3695) Support namespace in Nacos cluster/configuration (#3578) Provide plugin for ehcache 2.x (#3575) Supporting RequestRateLimiterGatewayFilterFactory (#3538) Kafka-plugin compatible with KafkaTemplate (#3505) Add pulsar apm plugin (#3476) Spring-cloud-gateway traceId does not transmit #3411 (#3446) Gateway compatible with downstream loss (#3445) Provide cassandra java driver 3.x plugin (#3410) Fix SpringMVC4 NoSuchMethodError (#3408) BugFix: endpoint grouping rules may be not unique (#3510) Add feature to control the maximum agent log files (#3475) Agent support custom properties. (#3367) Add Light4j plugin (#3323) Document Remove travis badge (#3763) Replace user wall to typical users in readme page (#3719) Update istio docs according latest istio release (#3646) Use chart deploy sw docs (#3573) Reorganize the doc, and provide catalog (#3563) Committer vote and set up document. (#3496) Update als setup doc as istio 1.3 released (#3470) Fill faq reply in official document. (#3450) All issues and pull requests are here\n6.4.0 Project Highly recommend to upgrade due to Pxx metrics calculation bug. Make agent working in JDK9+ Module system. Java Agent Make agent working in JDK9+ Module system. Support Kafka 2.x client libs. Log error in OKHTTP OnFailure callback. Support injecting traceid into logstack appender in logback. Add OperationName(including endpoint name) length max threshold. Support using Regex to group operation name. Support Undertow routing handler. RestTemplate plugin support operation name grouping. Fix ClassCastException in Webflux plugin. Ordering zookeeper server list, to make it better in topology. Fix a Dubbo plugin incompatible issue. Fix MySQL 5 plugin issue. Make log writer cached. Optimize Spring Cloud Gateway plugin Fix and improve gRPC reconnect mechanism. Remove Disruptor dependency from agent. Backend Fix Pxx(p50,p75,p90,p95,p99) metrics func bug.(Critical) Support Gateway in backend analysis, even when it doesn\u0026rsquo;t have suitable language agent. Support using HTTPs SSL accessing ElasticSearch storage. Support Zookeeper ACL. Make alarm records listed in order. Fix Pxx data persistence failure in some cases. Fix some bugs in MySQL storage. Setup slow SQL length threshold. Fix TTL settings is not working as expected. Remove scope-meta file. UI Enhance alarm page layout. Support trace tree chart resize. Support trace auto completion when partial traces abandoned somehow. Fix dashboard endpoint slow chart. Add radial chart in topology page. Add trace table mode. Fix topology page bug. Fix calender js bug. Fix \u0026ldquo;The \u0026ldquo;topo-services\u0026rdquo; component did not update the data in time after modifying the time range on the topology page. Document Restore the broken Istio setup doc. Add etcd config center document. Correct span_limit_per_segment default value in document. Enhance plugin develop doc. Fix error description in build document. All issues and pull requests are here\n6.3.0 Project e2e tests have been added, and verify every pull request. Use ArrayList to replace LinkedList in DataCarrier for much better performance. Add plugin instrumentation definition check in CI. DataCarrier performance improvement by avoiding false-sharing. Java Agent Java agent supports JDK 9 - 12, but don\u0026rsquo;t support Java Module yet. Support JVM class auto instrumentation, cataloged as bootstrap plugin. Support JVM HttpClient and HttpsClient plugin.[Optional] Support backend upgrade without rebooting required. Open Redefine and Retransform by other agents. Support Servlet 2.5 in Jetty, Tomcat and SpringMVC plugins. Support Spring @Async plugin. Add new config item to restrict the length of span#peer. Refactor ContextManager#stopSpan. Add gRPC timeout. Support Logback AsyncAppender print tid Fix gRPC reconnect bug. Fix trace segment service doesn\u0026rsquo;t report onComplete. Fix wrong logger class name. Fix gRPC plugin bug. Fix ContextManager.activeSpan() API usage error. Backend Support agent reset command downstream when the storage is erased, mostly because of backend upgrade. Backend stream flow refactor. High dimensionality metrics(Hour/Day/Month) are changed to lower priority, to ease the storage payload. Add OAP metrics cache to ease the storage query payload and improve performance. Remove DataCarrier in trace persistent of ElasticSearch storage, by leveraging the elasticsearch bulk queue. OAP internal communication protocol changed. Don\u0026rsquo;t be compatible with old releases. Improve ElasticSearch storage bulk performance. Support etcd as dynamic configuration center. Simplify the PxxMetrics and ThermodynamicMetrics functions for better performance and GC. Support JVM metrics self observability. Add the new OAL runtime engine. Add gRPC timeout. Add Charset in the alarm web hook. Fix buffer lost. Fix dirty read in ElasticSearch storage. Fix bug of cluster management plugins in un-Mixed mode. Fix wrong logger class name. Fix delete bug in ElasticSearch when using namespace. Fix MySQL TTL failure. Totally remove IDs can't be null log, to avoid misleading. Fix provider has been initialized repeatedly. Adjust providers conflict log message. Fix using wrong gc time metrics in OAL. UI Fix refresh is not working after endpoint and instance changed. Fix endpoint selector but. Fix wrong copy value in slow traces. Fix can\u0026rsquo;t show trace when it is broken partially(Because of agent sampling or fail safe). Fix database and response time graph bugs. Document Add bootstrap plugin development document. Alarm documentation typo fixed. Clarify the Docker file purpose. Fix a license typo. All issues and pull requests are here\n6.2.0 Project ElasticSearch implementation performance improved, and CHANGED totally. Must delete all existing indexes to do upgrade. CI and Integration tests provided by ASF INFRA. Plan to enhance tests including e2e, plugin tests in all pull requests, powered by ASF INFRA. DataCarrier queue write index controller performance improvement. 3-5 times quicker than before. Add windows compile support in CI. Java Agent Support collect SQL parameter in MySQL plugin.[Optional] Support SolrJ plugin. Support RESTEasy plugin. Support Spring Gateway plugin for 2.1.x[Optional] TracingContext performance improvement. Support Apache ShardingSphere(incubating) plugin. Support span#error in application toolkit. Fix OOM by empty stack of exception. FIx wrong cause exception of stack in span log. Fix unclear the running context in SpringMVC plugin. Fix CPU usage accessor calculation issue. Fix SpringMVC plugin span not stop bug when doing HTTP forward. Fix lettuce plugin async commend bug and NPE. Fix webflux plugin cast exception. [CI]Support import check. Backend Support time serious ElasticSearch storage. Provide dynamic configuration module and implementation. Slow SQL threshold supports dynamic config today. Dynamic Configuration module provide multiple implementations, DCS(gRPC based), Zookeeper, Apollo, Nacos. Provide P99/95/90/75/50 charts in topology edge. New topology query protocol and implementation. Support Envoy ALS in Service Mesh scenario. Support Nacos cluster management. Enhance metric exporter. Run in increment and total modes. Fix module provider is loaded repeatedly. Change TOP slow SQL storage in ES to Text from Keyword, as too long text issue. Fix H2TopologyQuery tiny bug. Fix H2 log query bug.(No feature provided yet) Filtering pods not in \u0026lsquo;Running\u0026rsquo; phase in mesh scenario. Fix query alarm bug in MySQL and H2 storage. Codes refactor. UI Fix some ID is null query(s). Page refactor, especially time-picker, more friendly. Login removed. Trace timestamp visualization issue fixed. Provide P99/95/90/75/50 charts in topology edge. Change all P99/95/90/75/50 charts style. More readable. Fix 404 in trace page. Document Go2Sky project has been donated to SkyAPM, change document link. Add FAQ for ElasticSearch storage, and links from document. Add FAQ fro WebSphere installation. Add several open users. Add alarm webhook document. All issues and pull requests are here\n6.1.0 Project SkyWalking graduated as Apache Top Level Project.\nSupport compiling project agent, backend, UI separately. Java Agent Support Vert.x Core 3.x plugin. Support Apache Dubbo plugin. Support use_qualified_name_as_endpoint_name and use_qualified_name_as_operation_name configs in SpringMVC plugin. Support span async close APIs in core. Used in Vert.x plugin. Support MySQL 5,8 plugins. Support set instance id manually(optional). Support customize enhance trace plugin in optional list. Support to set peer in Entry Span. Support Zookeeper plugin. Fix Webflux plugin created unexpected Entry Span. Fix Kafka plugin NPE in Kafka 1.1+ Fix wrong operation name in postgre 8.x plugin. Fix RabbitMQ plugin NPE. Fix agent can\u0026rsquo;t run in JVM 6/7, remove module-info.class. Fix agent can\u0026rsquo;t work well, if there is whitespace in agent path. Fix Spring annotation bug and inheritance enhance issue. Fix CPU accessor bug. Backend Performance improved, especially in CPU limited environment. 3x improvement in service mesh scenario(no trace) in 8C16G VM. Significantly cost less CPU in low payload.\nSupport database metrics and SLOW SQL detection. Support to set max size of metadata query. And change default to 5000 from 100. Support ElasticSearch template for new feature in the future. Support shutdown Zipkin trace analysis, because it doesn\u0026rsquo;t fit production environment. Support log type, scope HTTP_ACCESS_LOG and query. No feature provided, prepare for future versions. Support .NET clr receiver. Support Jaeger trace format, no analysis. Support group endpoint name by regax rules in mesh receiver. Support disable statement in OAL. Support basic auth in ElasticSearch connection. Support metrics exporter module and gRPC implementor. Support \u0026gt;, \u0026lt;, \u0026gt;=, \u0026lt;= in OAL. Support role mode in backend. Support Envoy metrics. Support query segment by service instance. Support to set host/port manually at cluster coordinator, rather than based on core settings. Make sure OAP shutdown when it faces startup error. Support set separated gRPC/Jetty ip:port for receiver, default still use core settings. Fix JVM receiver bug. Fix wrong dest service in mesh analysis. Fix search doesn\u0026rsquo;t work as expected. Refactor ScopeDeclaration annotation. Refactor register lock mechanism. Add SmartSql component for .NET Add integration tests for ElasticSearch client. Add test cases for exporter. Add test cases for queue consume. UI RocketBot UI has been accepted and bind in this release. Support CLR metrics. Document Documents updated, matching Top Level Project requirement. UI licenses updated, according to RocketBot UI IP clearance. User wall and powered-by list updated. CN documents removed, only consider to provide by volunteer out of Apache. All issues and pull requests are here\n6.0.0-GA Java Agent Support gson plugin(optional). Support canal plugin. Fix missing ojdbc component id. Fix dubbo plugin conflict. Fix OpenTracing tag match bug. Fix a missing check in ignore plugin. Backend Adjust service inventory entity, to add properties. Adjust service instance inventory entity, to add properties. Add nodeType to service inventory entity. Fix when operation name of local and exit spans in ref, the segment lost. Fix the index names don\u0026rsquo;t show right in logs. Fix wrong alarm text. Add test case for span limit mechanism. Add telemetry module and prometheus implementation, with grafana setting. A refactor for register API in storage module. Fix H2 and MySQL endpoint dependency map miss upstream side. Optimize the inventory register and refactor the implementation. Speed up the trace buffer read. Fix and removed unnecessary inventory register operations. UI Add new trace view. Add word-break to tag value. Document Add two startup modes document. Add PHP agent links. Add some cn documents. Update year to 2019 User wall updated. Fix a wrong description in how-to-build doc. All issues and pull requests are here\n6.0.0-beta Protocol Provide Trace Data Protocol v2 Provide SkyWalking Cross Process Propagation Headers Protocol v2. Java Agent Support Trace Data Protocol v2 Support SkyWalking Cross Process Propagation Headers Protocol v2. Support SkyWalking Cross Process Propagation Headers Protocol v1 running in compatible way. Need declare open explicitly. Support SpringMVC 5 Support webflux Support a new way to override agent.config by system env. Span tag can override by explicit way. Fix Spring Controller Inherit issue. Fix ElasticSearch plugin NPE. Fix agent classloader dead lock in certain situation. Fix agent log typo. Fix wrong component id in resettemplete plugin. Fix use transform ignore() in wrong way. Fix H2 query bug. Backend Support Trace Data Protocol v2. And Trace Data Protocol v1 is still supported. Support MySQL as storage. Support TiDB as storage. Support a new way to override application.yml by system env. Support service instance and endpoint alarm. Support namespace in istio receiver. Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Support backend trace sampling. Support Zipkin format again. Support init mode. Support namespace in Zookeeper cluster management. Support consul plugin in cluster module. OAL generate tool has been integrated into main repo, in the maven compile stage. Optimize trace paging query. Fix trace query don\u0026rsquo;t use fuzzy query in ElasticSearch storage. Fix alarm can\u0026rsquo;t be active in right way. Fix unnecessary condition in database and cache number query. Fix wrong namespace bug in ElasticSearch storage. Fix Remote clients selector error: / by zero . Fix segment TTL is not working. UI Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Fix TopN endpoint link doesn\u0026rsquo;t work right. Fix trace stack style. Fix CI. Document Add more agent setting documents. Add more contribution documents. Update user wall and powered-by page. Add RocketBot UI project link in document. All issues and pull requests are here\n6.0.0-alpha SkyWalking 6 is totally new milestone for the project. At this point, we are not just a distributing tracing system with analysis and visualization capabilities. We are an Observability Analysis Platform(OAL).\nThe core and most important features in v6 are\nSupport to collect telemetry data from different sources, such as multiple language agents and service mesh. Extensible stream analysis core. Make SQL and cache analysis available in core level, although haven\u0026rsquo;t provided in this release. Provide Observability Analysis Language(OAL) to make analysis metrics customization available. New GraphQL query protocol. Not binding with UI now. UI topology is better now. New alarm core provided. In alpha, only on service related metrics. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"660\"\u003e6.6.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e[\u003cstrong\u003eIMPORTANT\u003c/strong\u003e] Local span and exit span are not treated as endpoint detected at client …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.4.0/en/changes/changes-6.x/","title":"6.6.0"},{"body":"6.6.0 Project [IMPORTANT] Local span and exit span are not treated as endpoint detected at client and local. Only entry span is the endpoint. Reduce the load of register and memory cost. Support MiniKube, Istio and SkyWalking on K8s deployment in CI. Support Windows and MacOS build in GitHub Action CI. Support ElasticSearch 7 in official dist. Hundreds plugin cases have been added in GitHub Action CI process. Java Agent Remove the local/exit span operation name register mechanism. Add plugin for JDK Threading classes. Add plugin for Armeria. Support set operation name in async span. Enhance webflux plugin, related to Spring Gateway plugin. Webflux plugin is in optional, due to JDK8 required. Fix a possible deadlock. Fix NPE when OAL scripts are different in different OAP nodes, mostly in upgrading stage. Fix bug about wrong peer in ES plugin. Fix NPE in Spring plugin. Fix wrong class name in Dubbo 2.7 conflict patch. Fix spring annotation inheritance problem. OAP-Backend Remove the local/exit span operation name register mechanism. Remove client side endpoint register in service mesh. Service instance dependency and related metrics. Support min func in OAL Support apdex func in OAL Support custom ES config setting at the index level. Envoy ALS proto upgraded. Update JODA lib as bugs in UTC +13/+14. Support topN sample period configurable. Ignore no statement DB operations in slow SQL collection. Fix bug in docker-entrypoint.sh when using MySQL as storage UI Service topology enhancement. Dive into service, instance and endpoint metrics on topo map. Service instance dependency view and related metrics. Support using URL parameter in trace query page. Support apdex score in service page. Add service dependency metrics into metrics comparison. Fix alarm search not working. Document Update user list and user wall. Add document link for CLI. Add deployment guide of agent in Jetty case. Modify Consul cluster doc. Add document about injecting traceId into the logback with logstack in JSON format. ElementUI license and dependency added. All issues and pull requests are here\n6.5.0 Project TTL E2E test (#3437) Test coverage is back in pull request check status (#3503) Plugin tests begin to be migrated into main repo, and is in process. (#3528, #3756, #3751, etc.) Switch to SkyWalking CI (exclusive) nodes (#3546) MySQL storage e2e test. (#3648) E2E tests are verified in multiple jdk versions, jdk 8, 9, 11, 12 (#3657) Jenkins build jobs run only when necessary (#3662) OAP-Backend Support dynamically configure alarm settings (#3557) Language of instance could be null (#3485) Make query max window size configurable. (#3765) Remove two max size 500 limit. (#3748) Parameterize the cache size. (#3741) ServiceInstanceRelation set error id (#3683) Makes the scope of alarm message more semantic. (#3680) Add register persistent worker latency metrics (#3677) Fix more reasonable error (#3619) Add GraphQL getServiceInstance instanceUuid field. (#3595) Support namespace in Nacos cluster/configuration (#3578) Instead of datasource-settings.properties, use application.yml for MySQLStorageProvider (#3564) Provide consul dynamic configuration center implementation (#3560) Upgrade guava version to support higher jdk version (#3541) Sync latest als from envoy api (#3507) Set telemetry instanced id for Etcd and Nacos plugin (#3492) Support timeout configuration in agent and backend. (#3491) Make sure the cluster register happens before streaming process. (#3471) Agent supports custom properties. (#3367) Miscellaneous bug fixes (#3567) UI Feature: node detail display in topo circle-chart view. BugFix: the jvm-maxheap \u0026amp; jvm-maxnonheap is -1, free is no value Fix bug: time select operation not in effect Fix bug: language initialization failed Fix bug: not show instance language Feature: support the trace list display export png Feature: Metrics comparison view BugFix: Fix dashboard top throughput copy Java Agent Spring async scenario optimize (#3723) Support log4j2 AsyncLogger (#3715) Add config to collect PostgreSQL sql query params (#3695) Support namespace in Nacos cluster/configuration (#3578) Provide plugin for ehcache 2.x (#3575) Supporting RequestRateLimiterGatewayFilterFactory (#3538) Kafka-plugin compatible with KafkaTemplate (#3505) Add pulsar apm plugin (#3476) Spring-cloud-gateway traceId does not transmit #3411 (#3446) Gateway compatible with downstream loss (#3445) Provide cassandra java driver 3.x plugin (#3410) Fix SpringMVC4 NoSuchMethodError (#3408) BugFix: endpoint grouping rules may be not unique (#3510) Add feature to control the maximum agent log files (#3475) Agent support custom properties. (#3367) Add Light4j plugin (#3323) Document Remove travis badge (#3763) Replace user wall to typical users in readme page (#3719) Update istio docs according latest istio release (#3646) Use chart deploy sw docs (#3573) Reorganize the doc, and provide catalog (#3563) Committer vote and set up document. (#3496) Update als setup doc as istio 1.3 released (#3470) Fill faq reply in official document. (#3450) All issues and pull requests are here\n6.4.0 Project Highly recommend to upgrade due to Pxx metrics calculation bug. Make agent working in JDK9+ Module system. Java Agent Make agent working in JDK9+ Module system. Support Kafka 2.x client libs. Log error in OKHTTP OnFailure callback. Support injecting traceid into logstack appender in logback. Add OperationName(including endpoint name) length max threshold. Support using Regex to group operation name. Support Undertow routing handler. RestTemplate plugin support operation name grouping. Fix ClassCastException in Webflux plugin. Ordering zookeeper server list, to make it better in topology. Fix a Dubbo plugin incompatible issue. Fix MySQL 5 plugin issue. Make log writer cached. Optimize Spring Cloud Gateway plugin Fix and improve gRPC reconnect mechanism. Remove Disruptor dependency from agent. Backend Fix Pxx(p50,p75,p90,p95,p99) metrics func bug.(Critical) Support Gateway in backend analysis, even when it doesn\u0026rsquo;t have suitable language agent. Support using HTTPs SSL accessing ElasticSearch storage. Support Zookeeper ACL. Make alarm records listed in order. Fix Pxx data persistence failure in some cases. Fix some bugs in MySQL storage. Setup slow SQL length threshold. Fix TTL settings is not working as expected. Remove scope-meta file. UI Enhance alarm page layout. Support trace tree chart resize. Support trace auto completion when partial traces abandoned somehow. Fix dashboard endpoint slow chart. Add radial chart in topology page. Add trace table mode. Fix topology page bug. Fix calender js bug. Fix \u0026ldquo;The \u0026ldquo;topo-services\u0026rdquo; component did not update the data in time after modifying the time range on the topology page. Document Restore the broken Istio setup doc. Add etcd config center document. Correct span_limit_per_segment default value in document. Enhance plugin develop doc. Fix error description in build document. All issues and pull requests are here\n6.3.0 Project e2e tests have been added, and verify every pull request. Use ArrayList to replace LinkedList in DataCarrier for much better performance. Add plugin instrumentation definition check in CI. DataCarrier performance improvement by avoiding false-sharing. Java Agent Java agent supports JDK 9 - 12, but don\u0026rsquo;t support Java Module yet. Support JVM class auto instrumentation, cataloged as bootstrap plugin. Support JVM HttpClient and HttpsClient plugin.[Optional] Support backend upgrade without rebooting required. Open Redefine and Retransform by other agents. Support Servlet 2.5 in Jetty, Tomcat and SpringMVC plugins. Support Spring @Async plugin. Add new config item to restrict the length of span#peer. Refactor ContextManager#stopSpan. Add gRPC timeout. Support Logback AsyncAppender print tid Fix gRPC reconnect bug. Fix trace segment service doesn\u0026rsquo;t report onComplete. Fix wrong logger class name. Fix gRPC plugin bug. Fix ContextManager.activeSpan() API usage error. Backend Support agent reset command downstream when the storage is erased, mostly because of backend upgrade. Backend stream flow refactor. High dimensionality metrics(Hour/Day/Month) are changed to lower priority, to ease the storage payload. Add OAP metrics cache to ease the storage query payload and improve performance. Remove DataCarrier in trace persistent of ElasticSearch storage, by leveraging the elasticsearch bulk queue. OAP internal communication protocol changed. Don\u0026rsquo;t be compatible with old releases. Improve ElasticSearch storage bulk performance. Support etcd as dynamic configuration center. Simplify the PxxMetrics and ThermodynamicMetrics functions for better performance and GC. Support JVM metrics self observability. Add the new OAL runtime engine. Add gRPC timeout. Add Charset in the alarm web hook. Fix buffer lost. Fix dirty read in ElasticSearch storage. Fix bug of cluster management plugins in un-Mixed mode. Fix wrong logger class name. Fix delete bug in ElasticSearch when using namespace. Fix MySQL TTL failure. Totally remove IDs can't be null log, to avoid misleading. Fix provider has been initialized repeatedly. Adjust providers conflict log message. Fix using wrong gc time metrics in OAL. UI Fix refresh is not working after endpoint and instance changed. Fix endpoint selector but. Fix wrong copy value in slow traces. Fix can\u0026rsquo;t show trace when it is broken partially(Because of agent sampling or fail safe). Fix database and response time graph bugs. Document Add bootstrap plugin development document. Alarm documentation typo fixed. Clarify the Docker file purpose. Fix a license typo. All issues and pull requests are here\n6.2.0 Project ElasticSearch implementation performance improved, and CHANGED totally. Must delete all existing indexes to do upgrade. CI and Integration tests provided by ASF INFRA. Plan to enhance tests including e2e, plugin tests in all pull requests, powered by ASF INFRA. DataCarrier queue write index controller performance improvement. 3-5 times quicker than before. Add windows compile support in CI. Java Agent Support collect SQL parameter in MySQL plugin.[Optional] Support SolrJ plugin. Support RESTEasy plugin. Support Spring Gateway plugin for 2.1.x[Optional] TracingContext performance improvement. Support Apache ShardingSphere(incubating) plugin. Support span#error in application toolkit. Fix OOM by empty stack of exception. FIx wrong cause exception of stack in span log. Fix unclear the running context in SpringMVC plugin. Fix CPU usage accessor calculation issue. Fix SpringMVC plugin span not stop bug when doing HTTP forward. Fix lettuce plugin async commend bug and NPE. Fix webflux plugin cast exception. [CI]Support import check. Backend Support time serious ElasticSearch storage. Provide dynamic configuration module and implementation. Slow SQL threshold supports dynamic config today. Dynamic Configuration module provide multiple implementations, DCS(gRPC based), Zookeeper, Apollo, Nacos. Provide P99/95/90/75/50 charts in topology edge. New topology query protocol and implementation. Support Envoy ALS in Service Mesh scenario. Support Nacos cluster management. Enhance metric exporter. Run in increment and total modes. Fix module provider is loaded repeatedly. Change TOP slow SQL storage in ES to Text from Keyword, as too long text issue. Fix H2TopologyQuery tiny bug. Fix H2 log query bug.(No feature provided yet) Filtering pods not in \u0026lsquo;Running\u0026rsquo; phase in mesh scenario. Fix query alarm bug in MySQL and H2 storage. Codes refactor. UI Fix some ID is null query(s). Page refactor, especially time-picker, more friendly. Login removed. Trace timestamp visualization issue fixed. Provide P99/95/90/75/50 charts in topology edge. Change all P99/95/90/75/50 charts style. More readable. Fix 404 in trace page. Document Go2Sky project has been donated to SkyAPM, change document link. Add FAQ for ElasticSearch storage, and links from document. Add FAQ fro WebSphere installation. Add several open users. Add alarm webhook document. All issues and pull requests are here\n6.1.0 Project SkyWalking graduated as Apache Top Level Project.\nSupport compiling project agent, backend, UI separately. Java Agent Support Vert.x Core 3.x plugin. Support Apache Dubbo plugin. Support use_qualified_name_as_endpoint_name and use_qualified_name_as_operation_name configs in SpringMVC plugin. Support span async close APIs in core. Used in Vert.x plugin. Support MySQL 5,8 plugins. Support set instance id manually(optional). Support customize enhance trace plugin in optional list. Support to set peer in Entry Span. Support Zookeeper plugin. Fix Webflux plugin created unexpected Entry Span. Fix Kafka plugin NPE in Kafka 1.1+ Fix wrong operation name in postgre 8.x plugin. Fix RabbitMQ plugin NPE. Fix agent can\u0026rsquo;t run in JVM 6/7, remove module-info.class. Fix agent can\u0026rsquo;t work well, if there is whitespace in agent path. Fix Spring annotation bug and inheritance enhance issue. Fix CPU accessor bug. Backend Performance improved, especially in CPU limited environment. 3x improvement in service mesh scenario(no trace) in 8C16G VM. Significantly cost less CPU in low payload.\nSupport database metrics and SLOW SQL detection. Support to set max size of metadata query. And change default to 5000 from 100. Support ElasticSearch template for new feature in the future. Support shutdown Zipkin trace analysis, because it doesn\u0026rsquo;t fit production environment. Support log type, scope HTTP_ACCESS_LOG and query. No feature provided, prepare for future versions. Support .NET clr receiver. Support Jaeger trace format, no analysis. Support group endpoint name by regax rules in mesh receiver. Support disable statement in OAL. Support basic auth in ElasticSearch connection. Support metrics exporter module and gRPC implementor. Support \u0026gt;, \u0026lt;, \u0026gt;=, \u0026lt;= in OAL. Support role mode in backend. Support Envoy metrics. Support query segment by service instance. Support to set host/port manually at cluster coordinator, rather than based on core settings. Make sure OAP shutdown when it faces startup error. Support set separated gRPC/Jetty ip:port for receiver, default still use core settings. Fix JVM receiver bug. Fix wrong dest service in mesh analysis. Fix search doesn\u0026rsquo;t work as expected. Refactor ScopeDeclaration annotation. Refactor register lock mechanism. Add SmartSql component for .NET Add integration tests for ElasticSearch client. Add test cases for exporter. Add test cases for queue consume. UI RocketBot UI has been accepted and bind in this release. Support CLR metrics. Document Documents updated, matching Top Level Project requirement. UI licenses updated, according to RocketBot UI IP clearance. User wall and powered-by list updated. CN documents removed, only consider to provide by volunteer out of Apache. All issues and pull requests are here\n6.0.0-GA Java Agent Support gson plugin(optional). Support canal plugin. Fix missing ojdbc component id. Fix dubbo plugin conflict. Fix OpenTracing tag match bug. Fix a missing check in ignore plugin. Backend Adjust service inventory entity, to add properties. Adjust service instance inventory entity, to add properties. Add nodeType to service inventory entity. Fix when operation name of local and exit spans in ref, the segment lost. Fix the index names don\u0026rsquo;t show right in logs. Fix wrong alarm text. Add test case for span limit mechanism. Add telemetry module and prometheus implementation, with grafana setting. A refactor for register API in storage module. Fix H2 and MySQL endpoint dependency map miss upstream side. Optimize the inventory register and refactor the implementation. Speed up the trace buffer read. Fix and removed unnecessary inventory register operations. UI Add new trace view. Add word-break to tag value. Document Add two startup modes document. Add PHP agent links. Add some cn documents. Update year to 2019 User wall updated. Fix a wrong description in how-to-build doc. All issues and pull requests are here\n6.0.0-beta Protocol Provide Trace Data Protocol v2 Provide SkyWalking Cross Process Propagation Headers Protocol v2. Java Agent Support Trace Data Protocol v2 Support SkyWalking Cross Process Propagation Headers Protocol v2. Support SkyWalking Cross Process Propagation Headers Protocol v1 running in compatible way. Need declare open explicitly. Support SpringMVC 5 Support webflux Support a new way to override agent.config by system env. Span tag can override by explicit way. Fix Spring Controller Inherit issue. Fix ElasticSearch plugin NPE. Fix agent classloader dead lock in certain situation. Fix agent log typo. Fix wrong component id in resettemplete plugin. Fix use transform ignore() in wrong way. Fix H2 query bug. Backend Support Trace Data Protocol v2. And Trace Data Protocol v1 is still supported. Support MySQL as storage. Support TiDB as storage. Support a new way to override application.yml by system env. Support service instance and endpoint alarm. Support namespace in istio receiver. Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Support backend trace sampling. Support Zipkin format again. Support init mode. Support namespace in Zookeeper cluster management. Support consul plugin in cluster module. OAL generate tool has been integrated into main repo, in the maven compile stage. Optimize trace paging query. Fix trace query don\u0026rsquo;t use fuzzy query in ElasticSearch storage. Fix alarm can\u0026rsquo;t be active in right way. Fix unnecessary condition in database and cache number query. Fix wrong namespace bug in ElasticSearch storage. Fix Remote clients selector error: / by zero . Fix segment TTL is not working. UI Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Fix TopN endpoint link doesn\u0026rsquo;t work right. Fix trace stack style. Fix CI. Document Add more agent setting documents. Add more contribution documents. Update user wall and powered-by page. Add RocketBot UI project link in document. All issues and pull requests are here\n6.0.0-alpha SkyWalking 6 is totally new milestone for the project. At this point, we are not just a distributing tracing system with analysis and visualization capabilities. We are an Observability Analysis Platform(OAL).\nThe core and most important features in v6 are\nSupport to collect telemetry data from different sources, such as multiple language agents and service mesh. Extensible stream analysis core. Make SQL and cache analysis available in core level, although haven\u0026rsquo;t provided in this release. Provide Observability Analysis Language(OAL) to make analysis metrics customization available. New GraphQL query protocol. Not binding with UI now. UI topology is better now. New alarm core provided. In alpha, only on service related metrics. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"660\"\u003e6.6.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e[\u003cstrong\u003eIMPORTANT\u003c/strong\u003e] Local span and exit span are not treated as endpoint detected at client …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v11.0.0/en/changes/changes-6.x/","title":"6.6.0"},{"body":"6.6.0 Project [IMPORTANT] Local span and exit span are not treated as endpoint detected at client and local. Only entry span is the endpoint. Reduce the load of register and memory cost. Support MiniKube, Istio and SkyWalking on K8s deployment in CI. Support Windows and MacOS build in GitHub Action CI. Support ElasticSearch 7 in official dist. Hundreds plugin cases have been added in GitHub Action CI process. Java Agent Remove the local/exit span operation name register mechanism. Add plugin for JDK Threading classes. Add plugin for Armeria. Support set operation name in async span. Enhance webflux plugin, related to Spring Gateway plugin. Webflux plugin is in optional, due to JDK8 required. Fix a possible deadlock. Fix NPE when OAL scripts are different in different OAP nodes, mostly in upgrading stage. Fix bug about wrong peer in ES plugin. Fix NPE in Spring plugin. Fix wrong class name in Dubbo 2.7 conflict patch. Fix spring annotation inheritance problem. OAP-Backend Remove the local/exit span operation name register mechanism. Remove client side endpoint register in service mesh. Service instance dependency and related metrics. Support min func in OAL Support apdex func in OAL Support custom ES config setting at the index level. Envoy ALS proto upgraded. Update JODA lib as bugs in UTC +13/+14. Support topN sample period configurable. Ignore no statement DB operations in slow SQL collection. Fix bug in docker-entrypoint.sh when using MySQL as storage UI Service topology enhancement. Dive into service, instance and endpoint metrics on topo map. Service instance dependency view and related metrics. Support using URL parameter in trace query page. Support apdex score in service page. Add service dependency metrics into metrics comparison. Fix alarm search not working. Document Update user list and user wall. Add document link for CLI. Add deployment guide of agent in Jetty case. Modify Consul cluster doc. Add document about injecting traceId into the logback with logstack in JSON format. ElementUI license and dependency added. All issues and pull requests are here\n6.5.0 Project TTL E2E test (#3437) Test coverage is back in pull request check status (#3503) Plugin tests begin to be migrated into main repo, and is in process. (#3528, #3756, #3751, etc.) Switch to SkyWalking CI (exclusive) nodes (#3546) MySQL storage e2e test. (#3648) E2E tests are verified in multiple jdk versions, jdk 8, 9, 11, 12 (#3657) Jenkins build jobs run only when necessary (#3662) OAP-Backend Support dynamically configure alarm settings (#3557) Language of instance could be null (#3485) Make query max window size configurable. (#3765) Remove two max size 500 limit. (#3748) Parameterize the cache size. (#3741) ServiceInstanceRelation set error id (#3683) Makes the scope of alarm message more semantic. (#3680) Add register persistent worker latency metrics (#3677) Fix more reasonable error (#3619) Add GraphQL getServiceInstance instanceUuid field. (#3595) Support namespace in Nacos cluster/configuration (#3578) Instead of datasource-settings.properties, use application.yml for MySQLStorageProvider (#3564) Provide consul dynamic configuration center implementation (#3560) Upgrade guava version to support higher jdk version (#3541) Sync latest als from envoy api (#3507) Set telemetry instanced id for Etcd and Nacos plugin (#3492) Support timeout configuration in agent and backend. (#3491) Make sure the cluster register happens before streaming process. (#3471) Agent supports custom properties. (#3367) Miscellaneous bug fixes (#3567) UI Feature: node detail display in topo circle-chart view. BugFix: the jvm-maxheap \u0026amp; jvm-maxnonheap is -1, free is no value Fix bug: time select operation not in effect Fix bug: language initialization failed Fix bug: not show instance language Feature: support the trace list display export png Feature: Metrics comparison view BugFix: Fix dashboard top throughput copy Java Agent Spring async scenario optimize (#3723) Support log4j2 AsyncLogger (#3715) Add config to collect PostgreSQL sql query params (#3695) Support namespace in Nacos cluster/configuration (#3578) Provide plugin for ehcache 2.x (#3575) Supporting RequestRateLimiterGatewayFilterFactory (#3538) Kafka-plugin compatible with KafkaTemplate (#3505) Add pulsar apm plugin (#3476) Spring-cloud-gateway traceId does not transmit #3411 (#3446) Gateway compatible with downstream loss (#3445) Provide cassandra java driver 3.x plugin (#3410) Fix SpringMVC4 NoSuchMethodError (#3408) BugFix: endpoint grouping rules may be not unique (#3510) Add feature to control the maximum agent log files (#3475) Agent support custom properties. (#3367) Add Light4j plugin (#3323) Document Remove travis badge (#3763) Replace user wall to typical users in readme page (#3719) Update istio docs according latest istio release (#3646) Use chart deploy sw docs (#3573) Reorganize the doc, and provide catalog (#3563) Committer vote and set up document. (#3496) Update als setup doc as istio 1.3 released (#3470) Fill faq reply in official document. (#3450) All issues and pull requests are here\n6.4.0 Project Highly recommend to upgrade due to Pxx metrics calculation bug. Make agent working in JDK9+ Module system. Java Agent Make agent working in JDK9+ Module system. Support Kafka 2.x client libs. Log error in OKHTTP OnFailure callback. Support injecting traceid into logstack appender in logback. Add OperationName(including endpoint name) length max threshold. Support using Regex to group operation name. Support Undertow routing handler. RestTemplate plugin support operation name grouping. Fix ClassCastException in Webflux plugin. Ordering zookeeper server list, to make it better in topology. Fix a Dubbo plugin incompatible issue. Fix MySQL 5 plugin issue. Make log writer cached. Optimize Spring Cloud Gateway plugin Fix and improve gRPC reconnect mechanism. Remove Disruptor dependency from agent. Backend Fix Pxx(p50,p75,p90,p95,p99) metrics func bug.(Critical) Support Gateway in backend analysis, even when it doesn\u0026rsquo;t have suitable language agent. Support using HTTPs SSL accessing ElasticSearch storage. Support Zookeeper ACL. Make alarm records listed in order. Fix Pxx data persistence failure in some cases. Fix some bugs in MySQL storage. Setup slow SQL length threshold. Fix TTL settings is not working as expected. Remove scope-meta file. UI Enhance alarm page layout. Support trace tree chart resize. Support trace auto completion when partial traces abandoned somehow. Fix dashboard endpoint slow chart. Add radial chart in topology page. Add trace table mode. Fix topology page bug. Fix calender js bug. Fix \u0026ldquo;The \u0026ldquo;topo-services\u0026rdquo; component did not update the data in time after modifying the time range on the topology page. Document Restore the broken Istio setup doc. Add etcd config center document. Correct span_limit_per_segment default value in document. Enhance plugin develop doc. Fix error description in build document. All issues and pull requests are here\n6.3.0 Project e2e tests have been added, and verify every pull request. Use ArrayList to replace LinkedList in DataCarrier for much better performance. Add plugin instrumentation definition check in CI. DataCarrier performance improvement by avoiding false-sharing. Java Agent Java agent supports JDK 9 - 12, but don\u0026rsquo;t support Java Module yet. Support JVM class auto instrumentation, cataloged as bootstrap plugin. Support JVM HttpClient and HttpsClient plugin.[Optional] Support backend upgrade without rebooting required. Open Redefine and Retransform by other agents. Support Servlet 2.5 in Jetty, Tomcat and SpringMVC plugins. Support Spring @Async plugin. Add new config item to restrict the length of span#peer. Refactor ContextManager#stopSpan. Add gRPC timeout. Support Logback AsyncAppender print tid Fix gRPC reconnect bug. Fix trace segment service doesn\u0026rsquo;t report onComplete. Fix wrong logger class name. Fix gRPC plugin bug. Fix ContextManager.activeSpan() API usage error. Backend Support agent reset command downstream when the storage is erased, mostly because of backend upgrade. Backend stream flow refactor. High dimensionality metrics(Hour/Day/Month) are changed to lower priority, to ease the storage payload. Add OAP metrics cache to ease the storage query payload and improve performance. Remove DataCarrier in trace persistent of ElasticSearch storage, by leveraging the elasticsearch bulk queue. OAP internal communication protocol changed. Don\u0026rsquo;t be compatible with old releases. Improve ElasticSearch storage bulk performance. Support etcd as dynamic configuration center. Simplify the PxxMetrics and ThermodynamicMetrics functions for better performance and GC. Support JVM metrics self observability. Add the new OAL runtime engine. Add gRPC timeout. Add Charset in the alarm web hook. Fix buffer lost. Fix dirty read in ElasticSearch storage. Fix bug of cluster management plugins in un-Mixed mode. Fix wrong logger class name. Fix delete bug in ElasticSearch when using namespace. Fix MySQL TTL failure. Totally remove IDs can't be null log, to avoid misleading. Fix provider has been initialized repeatedly. Adjust providers conflict log message. Fix using wrong gc time metrics in OAL. UI Fix refresh is not working after endpoint and instance changed. Fix endpoint selector but. Fix wrong copy value in slow traces. Fix can\u0026rsquo;t show trace when it is broken partially(Because of agent sampling or fail safe). Fix database and response time graph bugs. Document Add bootstrap plugin development document. Alarm documentation typo fixed. Clarify the Docker file purpose. Fix a license typo. All issues and pull requests are here\n6.2.0 Project ElasticSearch implementation performance improved, and CHANGED totally. Must delete all existing indexes to do upgrade. CI and Integration tests provided by ASF INFRA. Plan to enhance tests including e2e, plugin tests in all pull requests, powered by ASF INFRA. DataCarrier queue write index controller performance improvement. 3-5 times quicker than before. Add windows compile support in CI. Java Agent Support collect SQL parameter in MySQL plugin.[Optional] Support SolrJ plugin. Support RESTEasy plugin. Support Spring Gateway plugin for 2.1.x[Optional] TracingContext performance improvement. Support Apache ShardingSphere(incubating) plugin. Support span#error in application toolkit. Fix OOM by empty stack of exception. FIx wrong cause exception of stack in span log. Fix unclear the running context in SpringMVC plugin. Fix CPU usage accessor calculation issue. Fix SpringMVC plugin span not stop bug when doing HTTP forward. Fix lettuce plugin async commend bug and NPE. Fix webflux plugin cast exception. [CI]Support import check. Backend Support time serious ElasticSearch storage. Provide dynamic configuration module and implementation. Slow SQL threshold supports dynamic config today. Dynamic Configuration module provide multiple implementations, DCS(gRPC based), Zookeeper, Apollo, Nacos. Provide P99/95/90/75/50 charts in topology edge. New topology query protocol and implementation. Support Envoy ALS in Service Mesh scenario. Support Nacos cluster management. Enhance metric exporter. Run in increment and total modes. Fix module provider is loaded repeatedly. Change TOP slow SQL storage in ES to Text from Keyword, as too long text issue. Fix H2TopologyQuery tiny bug. Fix H2 log query bug.(No feature provided yet) Filtering pods not in \u0026lsquo;Running\u0026rsquo; phase in mesh scenario. Fix query alarm bug in MySQL and H2 storage. Codes refactor. UI Fix some ID is null query(s). Page refactor, especially time-picker, more friendly. Login removed. Trace timestamp visualization issue fixed. Provide P99/95/90/75/50 charts in topology edge. Change all P99/95/90/75/50 charts style. More readable. Fix 404 in trace page. Document Go2Sky project has been donated to SkyAPM, change document link. Add FAQ for ElasticSearch storage, and links from document. Add FAQ fro WebSphere installation. Add several open users. Add alarm webhook document. All issues and pull requests are here\n6.1.0 Project SkyWalking graduated as Apache Top Level Project.\nSupport compiling project agent, backend, UI separately. Java Agent Support Vert.x Core 3.x plugin. Support Apache Dubbo plugin. Support use_qualified_name_as_endpoint_name and use_qualified_name_as_operation_name configs in SpringMVC plugin. Support span async close APIs in core. Used in Vert.x plugin. Support MySQL 5,8 plugins. Support set instance id manually(optional). Support customize enhance trace plugin in optional list. Support to set peer in Entry Span. Support Zookeeper plugin. Fix Webflux plugin created unexpected Entry Span. Fix Kafka plugin NPE in Kafka 1.1+ Fix wrong operation name in postgre 8.x plugin. Fix RabbitMQ plugin NPE. Fix agent can\u0026rsquo;t run in JVM 6/7, remove module-info.class. Fix agent can\u0026rsquo;t work well, if there is whitespace in agent path. Fix Spring annotation bug and inheritance enhance issue. Fix CPU accessor bug. Backend Performance improved, especially in CPU limited environment. 3x improvement in service mesh scenario(no trace) in 8C16G VM. Significantly cost less CPU in low payload.\nSupport database metrics and SLOW SQL detection. Support to set max size of metadata query. And change default to 5000 from 100. Support ElasticSearch template for new feature in the future. Support shutdown Zipkin trace analysis, because it doesn\u0026rsquo;t fit production environment. Support log type, scope HTTP_ACCESS_LOG and query. No feature provided, prepare for future versions. Support .NET clr receiver. Support Jaeger trace format, no analysis. Support group endpoint name by regax rules in mesh receiver. Support disable statement in OAL. Support basic auth in ElasticSearch connection. Support metrics exporter module and gRPC implementor. Support \u0026gt;, \u0026lt;, \u0026gt;=, \u0026lt;= in OAL. Support role mode in backend. Support Envoy metrics. Support query segment by service instance. Support to set host/port manually at cluster coordinator, rather than based on core settings. Make sure OAP shutdown when it faces startup error. Support set separated gRPC/Jetty ip:port for receiver, default still use core settings. Fix JVM receiver bug. Fix wrong dest service in mesh analysis. Fix search doesn\u0026rsquo;t work as expected. Refactor ScopeDeclaration annotation. Refactor register lock mechanism. Add SmartSql component for .NET Add integration tests for ElasticSearch client. Add test cases for exporter. Add test cases for queue consume. UI RocketBot UI has been accepted and bind in this release. Support CLR metrics. Document Documents updated, matching Top Level Project requirement. UI licenses updated, according to RocketBot UI IP clearance. User wall and powered-by list updated. CN documents removed, only consider to provide by volunteer out of Apache. All issues and pull requests are here\n6.0.0-GA Java Agent Support gson plugin(optional). Support canal plugin. Fix missing ojdbc component id. Fix dubbo plugin conflict. Fix OpenTracing tag match bug. Fix a missing check in ignore plugin. Backend Adjust service inventory entity, to add properties. Adjust service instance inventory entity, to add properties. Add nodeType to service inventory entity. Fix when operation name of local and exit spans in ref, the segment lost. Fix the index names don\u0026rsquo;t show right in logs. Fix wrong alarm text. Add test case for span limit mechanism. Add telemetry module and prometheus implementation, with grafana setting. A refactor for register API in storage module. Fix H2 and MySQL endpoint dependency map miss upstream side. Optimize the inventory register and refactor the implementation. Speed up the trace buffer read. Fix and removed unnecessary inventory register operations. UI Add new trace view. Add word-break to tag value. Document Add two startup modes document. Add PHP agent links. Add some cn documents. Update year to 2019 User wall updated. Fix a wrong description in how-to-build doc. All issues and pull requests are here\n6.0.0-beta Protocol Provide Trace Data Protocol v2 Provide SkyWalking Cross Process Propagation Headers Protocol v2. Java Agent Support Trace Data Protocol v2 Support SkyWalking Cross Process Propagation Headers Protocol v2. Support SkyWalking Cross Process Propagation Headers Protocol v1 running in compatible way. Need declare open explicitly. Support SpringMVC 5 Support webflux Support a new way to override agent.config by system env. Span tag can override by explicit way. Fix Spring Controller Inherit issue. Fix ElasticSearch plugin NPE. Fix agent classloader dead lock in certain situation. Fix agent log typo. Fix wrong component id in resettemplete plugin. Fix use transform ignore() in wrong way. Fix H2 query bug. Backend Support Trace Data Protocol v2. And Trace Data Protocol v1 is still supported. Support MySQL as storage. Support TiDB as storage. Support a new way to override application.yml by system env. Support service instance and endpoint alarm. Support namespace in istio receiver. Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Support backend trace sampling. Support Zipkin format again. Support init mode. Support namespace in Zookeeper cluster management. Support consul plugin in cluster module. OAL generate tool has been integrated into main repo, in the maven compile stage. Optimize trace paging query. Fix trace query don\u0026rsquo;t use fuzzy query in ElasticSearch storage. Fix alarm can\u0026rsquo;t be active in right way. Fix unnecessary condition in database and cache number query. Fix wrong namespace bug in ElasticSearch storage. Fix Remote clients selector error: / by zero . Fix segment TTL is not working. UI Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Fix TopN endpoint link doesn\u0026rsquo;t work right. Fix trace stack style. Fix CI. Document Add more agent setting documents. Add more contribution documents. Update user wall and powered-by page. Add RocketBot UI project link in document. All issues and pull requests are here\n6.0.0-alpha SkyWalking 6 is totally new milestone for the project. At this point, we are not just a distributing tracing system with analysis and visualization capabilities. We are an Observability Analysis Platform(OAL).\nThe core and most important features in v6 are\nSupport to collect telemetry data from different sources, such as multiple language agents and service mesh. Extensible stream analysis core. Make SQL and cache analysis available in core level, although haven\u0026rsquo;t provided in this release. Provide Observability Analysis Language(OAL) to make analysis metrics customization available. New GraphQL query protocol. Not binding with UI now. UI topology is better now. New alarm core provided. In alpha, only on service related metrics. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"660\"\u003e6.6.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e[\u003cstrong\u003eIMPORTANT\u003c/strong\u003e] Local span and exit span are not treated as endpoint detected at client …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.4.0/en/changes/changes-6.x/","title":"6.6.0"},{"body":"6.6.0 Project [IMPORTANT] Local span and exit span are not treated as endpoint detected at client and local. Only entry span is the endpoint. Reduce the load of register and memory cost. Support MiniKube, Istio and SkyWalking on K8s deployment in CI. Support Windows and MacOS build in GitHub Action CI. Support ElasticSearch 7 in official dist. Hundreds plugin cases have been added in GitHub Action CI process. Java Agent Remove the local/exit span operation name register mechanism. Add plugin for JDK Threading classes. Add plugin for Armeria. Support set operation name in async span. Enhance webflux plugin, related to Spring Gateway plugin. Webflux plugin is in optional, due to JDK8 required. Fix a possible deadlock. Fix NPE when OAL scripts are different in different OAP nodes, mostly in upgrading stage. Fix bug about wrong peer in ES plugin. Fix NPE in Spring plugin. Fix wrong class name in Dubbo 2.7 conflict patch. Fix spring annotation inheritance problem. OAP-Backend Remove the local/exit span operation name register mechanism. Remove client side endpoint register in service mesh. Service instance dependency and related metrics. Support min func in OAL Support apdex func in OAL Support custom ES config setting at the index level. Envoy ALS proto upgraded. Update JODA lib as bugs in UTC +13/+14. Support topN sample period configurable. Ignore no statement DB operations in slow SQL collection. Fix bug in docker-entrypoint.sh when using MySQL as storage UI Service topology enhancement. Dive into service, instance and endpoint metrics on topo map. Service instance dependency view and related metrics. Support using URL parameter in trace query page. Support apdex score in service page. Add service dependency metrics into metrics comparison. Fix alarm search not working. Document Update user list and user wall. Add document link for CLI. Add deployment guide of agent in Jetty case. Modify Consul cluster doc. Add document about injecting traceId into the logback with logstack in JSON format. ElementUI license and dependency added. All issues and pull requests are here\n6.5.0 Project TTL E2E test (#3437) Test coverage is back in pull request check status (#3503) Plugin tests begin to be migrated into main repo, and is in process. (#3528, #3756, #3751, etc.) Switch to SkyWalking CI (exclusive) nodes (#3546) MySQL storage e2e test. (#3648) E2E tests are verified in multiple jdk versions, jdk 8, 9, 11, 12 (#3657) Jenkins build jobs run only when necessary (#3662) OAP-Backend Support dynamically configure alarm settings (#3557) Language of instance could be null (#3485) Make query max window size configurable. (#3765) Remove two max size 500 limit. (#3748) Parameterize the cache size. (#3741) ServiceInstanceRelation set error id (#3683) Makes the scope of alarm message more semantic. (#3680) Add register persistent worker latency metrics (#3677) Fix more reasonable error (#3619) Add GraphQL getServiceInstance instanceUuid field. (#3595) Support namespace in Nacos cluster/configuration (#3578) Instead of datasource-settings.properties, use application.yml for MySQLStorageProvider (#3564) Provide consul dynamic configuration center implementation (#3560) Upgrade guava version to support higher jdk version (#3541) Sync latest als from envoy api (#3507) Set telemetry instanced id for Etcd and Nacos plugin (#3492) Support timeout configuration in agent and backend. (#3491) Make sure the cluster register happens before streaming process. (#3471) Agent supports custom properties. (#3367) Miscellaneous bug fixes (#3567) UI Feature: node detail display in topo circle-chart view. BugFix: the jvm-maxheap \u0026amp; jvm-maxnonheap is -1, free is no value Fix bug: time select operation not in effect Fix bug: language initialization failed Fix bug: not show instance language Feature: support the trace list display export png Feature: Metrics comparison view BugFix: Fix dashboard top throughput copy Java Agent Spring async scenario optimize (#3723) Support log4j2 AsyncLogger (#3715) Add config to collect PostgreSQL sql query params (#3695) Support namespace in Nacos cluster/configuration (#3578) Provide plugin for ehcache 2.x (#3575) Supporting RequestRateLimiterGatewayFilterFactory (#3538) Kafka-plugin compatible with KafkaTemplate (#3505) Add pulsar apm plugin (#3476) Spring-cloud-gateway traceId does not transmit #3411 (#3446) Gateway compatible with downstream loss (#3445) Provide cassandra java driver 3.x plugin (#3410) Fix SpringMVC4 NoSuchMethodError (#3408) BugFix: endpoint grouping rules may be not unique (#3510) Add feature to control the maximum agent log files (#3475) Agent support custom properties. (#3367) Add Light4j plugin (#3323) Document Remove travis badge (#3763) Replace user wall to typical users in readme page (#3719) Update istio docs according latest istio release (#3646) Use chart deploy sw docs (#3573) Reorganize the doc, and provide catalog (#3563) Committer vote and set up document. (#3496) Update als setup doc as istio 1.3 released (#3470) Fill faq reply in official document. (#3450) All issues and pull requests are here\n6.4.0 Project Highly recommend to upgrade due to Pxx metrics calculation bug. Make agent working in JDK9+ Module system. Java Agent Make agent working in JDK9+ Module system. Support Kafka 2.x client libs. Log error in OKHTTP OnFailure callback. Support injecting traceid into logstack appender in logback. Add OperationName(including endpoint name) length max threshold. Support using Regex to group operation name. Support Undertow routing handler. RestTemplate plugin support operation name grouping. Fix ClassCastException in Webflux plugin. Ordering zookeeper server list, to make it better in topology. Fix a Dubbo plugin incompatible issue. Fix MySQL 5 plugin issue. Make log writer cached. Optimize Spring Cloud Gateway plugin Fix and improve gRPC reconnect mechanism. Remove Disruptor dependency from agent. Backend Fix Pxx(p50,p75,p90,p95,p99) metrics func bug.(Critical) Support Gateway in backend analysis, even when it doesn\u0026rsquo;t have suitable language agent. Support using HTTPs SSL accessing ElasticSearch storage. Support Zookeeper ACL. Make alarm records listed in order. Fix Pxx data persistence failure in some cases. Fix some bugs in MySQL storage. Setup slow SQL length threshold. Fix TTL settings is not working as expected. Remove scope-meta file. UI Enhance alarm page layout. Support trace tree chart resize. Support trace auto completion when partial traces abandoned somehow. Fix dashboard endpoint slow chart. Add radial chart in topology page. Add trace table mode. Fix topology page bug. Fix calender js bug. Fix \u0026ldquo;The \u0026ldquo;topo-services\u0026rdquo; component did not update the data in time after modifying the time range on the topology page. Document Restore the broken Istio setup doc. Add etcd config center document. Correct span_limit_per_segment default value in document. Enhance plugin develop doc. Fix error description in build document. All issues and pull requests are here\n6.3.0 Project e2e tests have been added, and verify every pull request. Use ArrayList to replace LinkedList in DataCarrier for much better performance. Add plugin instrumentation definition check in CI. DataCarrier performance improvement by avoiding false-sharing. Java Agent Java agent supports JDK 9 - 12, but don\u0026rsquo;t support Java Module yet. Support JVM class auto instrumentation, cataloged as bootstrap plugin. Support JVM HttpClient and HttpsClient plugin.[Optional] Support backend upgrade without rebooting required. Open Redefine and Retransform by other agents. Support Servlet 2.5 in Jetty, Tomcat and SpringMVC plugins. Support Spring @Async plugin. Add new config item to restrict the length of span#peer. Refactor ContextManager#stopSpan. Add gRPC timeout. Support Logback AsyncAppender print tid Fix gRPC reconnect bug. Fix trace segment service doesn\u0026rsquo;t report onComplete. Fix wrong logger class name. Fix gRPC plugin bug. Fix ContextManager.activeSpan() API usage error. Backend Support agent reset command downstream when the storage is erased, mostly because of backend upgrade. Backend stream flow refactor. High dimensionality metrics(Hour/Day/Month) are changed to lower priority, to ease the storage payload. Add OAP metrics cache to ease the storage query payload and improve performance. Remove DataCarrier in trace persistent of ElasticSearch storage, by leveraging the elasticsearch bulk queue. OAP internal communication protocol changed. Don\u0026rsquo;t be compatible with old releases. Improve ElasticSearch storage bulk performance. Support etcd as dynamic configuration center. Simplify the PxxMetrics and ThermodynamicMetrics functions for better performance and GC. Support JVM metrics self observability. Add the new OAL runtime engine. Add gRPC timeout. Add Charset in the alarm web hook. Fix buffer lost. Fix dirty read in ElasticSearch storage. Fix bug of cluster management plugins in un-Mixed mode. Fix wrong logger class name. Fix delete bug in ElasticSearch when using namespace. Fix MySQL TTL failure. Totally remove IDs can't be null log, to avoid misleading. Fix provider has been initialized repeatedly. Adjust providers conflict log message. Fix using wrong gc time metrics in OAL. UI Fix refresh is not working after endpoint and instance changed. Fix endpoint selector but. Fix wrong copy value in slow traces. Fix can\u0026rsquo;t show trace when it is broken partially(Because of agent sampling or fail safe). Fix database and response time graph bugs. Document Add bootstrap plugin development document. Alarm documentation typo fixed. Clarify the Docker file purpose. Fix a license typo. All issues and pull requests are here\n6.2.0 Project ElasticSearch implementation performance improved, and CHANGED totally. Must delete all existing indexes to do upgrade. CI and Integration tests provided by ASF INFRA. Plan to enhance tests including e2e, plugin tests in all pull requests, powered by ASF INFRA. DataCarrier queue write index controller performance improvement. 3-5 times quicker than before. Add windows compile support in CI. Java Agent Support collect SQL parameter in MySQL plugin.[Optional] Support SolrJ plugin. Support RESTEasy plugin. Support Spring Gateway plugin for 2.1.x[Optional] TracingContext performance improvement. Support Apache ShardingSphere(incubating) plugin. Support span#error in application toolkit. Fix OOM by empty stack of exception. FIx wrong cause exception of stack in span log. Fix unclear the running context in SpringMVC plugin. Fix CPU usage accessor calculation issue. Fix SpringMVC plugin span not stop bug when doing HTTP forward. Fix lettuce plugin async commend bug and NPE. Fix webflux plugin cast exception. [CI]Support import check. Backend Support time serious ElasticSearch storage. Provide dynamic configuration module and implementation. Slow SQL threshold supports dynamic config today. Dynamic Configuration module provide multiple implementations, DCS(gRPC based), Zookeeper, Apollo, Nacos. Provide P99/95/90/75/50 charts in topology edge. New topology query protocol and implementation. Support Envoy ALS in Service Mesh scenario. Support Nacos cluster management. Enhance metric exporter. Run in increment and total modes. Fix module provider is loaded repeatedly. Change TOP slow SQL storage in ES to Text from Keyword, as too long text issue. Fix H2TopologyQuery tiny bug. Fix H2 log query bug.(No feature provided yet) Filtering pods not in \u0026lsquo;Running\u0026rsquo; phase in mesh scenario. Fix query alarm bug in MySQL and H2 storage. Codes refactor. UI Fix some ID is null query(s). Page refactor, especially time-picker, more friendly. Login removed. Trace timestamp visualization issue fixed. Provide P99/95/90/75/50 charts in topology edge. Change all P99/95/90/75/50 charts style. More readable. Fix 404 in trace page. Document Go2Sky project has been donated to SkyAPM, change document link. Add FAQ for ElasticSearch storage, and links from document. Add FAQ fro WebSphere installation. Add several open users. Add alarm webhook document. All issues and pull requests are here\n6.1.0 Project SkyWalking graduated as Apache Top Level Project.\nSupport compiling project agent, backend, UI separately. Java Agent Support Vert.x Core 3.x plugin. Support Apache Dubbo plugin. Support use_qualified_name_as_endpoint_name and use_qualified_name_as_operation_name configs in SpringMVC plugin. Support span async close APIs in core. Used in Vert.x plugin. Support MySQL 5,8 plugins. Support set instance id manually(optional). Support customize enhance trace plugin in optional list. Support to set peer in Entry Span. Support Zookeeper plugin. Fix Webflux plugin created unexpected Entry Span. Fix Kafka plugin NPE in Kafka 1.1+ Fix wrong operation name in postgre 8.x plugin. Fix RabbitMQ plugin NPE. Fix agent can\u0026rsquo;t run in JVM 6/7, remove module-info.class. Fix agent can\u0026rsquo;t work well, if there is whitespace in agent path. Fix Spring annotation bug and inheritance enhance issue. Fix CPU accessor bug. Backend Performance improved, especially in CPU limited environment. 3x improvement in service mesh scenario(no trace) in 8C16G VM. Significantly cost less CPU in low payload.\nSupport database metrics and SLOW SQL detection. Support to set max size of metadata query. And change default to 5000 from 100. Support ElasticSearch template for new feature in the future. Support shutdown Zipkin trace analysis, because it doesn\u0026rsquo;t fit production environment. Support log type, scope HTTP_ACCESS_LOG and query. No feature provided, prepare for future versions. Support .NET clr receiver. Support Jaeger trace format, no analysis. Support group endpoint name by regax rules in mesh receiver. Support disable statement in OAL. Support basic auth in ElasticSearch connection. Support metrics exporter module and gRPC implementor. Support \u0026gt;, \u0026lt;, \u0026gt;=, \u0026lt;= in OAL. Support role mode in backend. Support Envoy metrics. Support query segment by service instance. Support to set host/port manually at cluster coordinator, rather than based on core settings. Make sure OAP shutdown when it faces startup error. Support set separated gRPC/Jetty ip:port for receiver, default still use core settings. Fix JVM receiver bug. Fix wrong dest service in mesh analysis. Fix search doesn\u0026rsquo;t work as expected. Refactor ScopeDeclaration annotation. Refactor register lock mechanism. Add SmartSql component for .NET Add integration tests for ElasticSearch client. Add test cases for exporter. Add test cases for queue consume. UI RocketBot UI has been accepted and bind in this release. Support CLR metrics. Document Documents updated, matching Top Level Project requirement. UI licenses updated, according to RocketBot UI IP clearance. User wall and powered-by list updated. CN documents removed, only consider to provide by volunteer out of Apache. All issues and pull requests are here\n6.0.0-GA Java Agent Support gson plugin(optional). Support canal plugin. Fix missing ojdbc component id. Fix dubbo plugin conflict. Fix OpenTracing tag match bug. Fix a missing check in ignore plugin. Backend Adjust service inventory entity, to add properties. Adjust service instance inventory entity, to add properties. Add nodeType to service inventory entity. Fix when operation name of local and exit spans in ref, the segment lost. Fix the index names don\u0026rsquo;t show right in logs. Fix wrong alarm text. Add test case for span limit mechanism. Add telemetry module and prometheus implementation, with grafana setting. A refactor for register API in storage module. Fix H2 and MySQL endpoint dependency map miss upstream side. Optimize the inventory register and refactor the implementation. Speed up the trace buffer read. Fix and removed unnecessary inventory register operations. UI Add new trace view. Add word-break to tag value. Document Add two startup modes document. Add PHP agent links. Add some cn documents. Update year to 2019 User wall updated. Fix a wrong description in how-to-build doc. All issues and pull requests are here\n6.0.0-beta Protocol Provide Trace Data Protocol v2 Provide SkyWalking Cross Process Propagation Headers Protocol v2. Java Agent Support Trace Data Protocol v2 Support SkyWalking Cross Process Propagation Headers Protocol v2. Support SkyWalking Cross Process Propagation Headers Protocol v1 running in compatible way. Need declare open explicitly. Support SpringMVC 5 Support webflux Support a new way to override agent.config by system env. Span tag can override by explicit way. Fix Spring Controller Inherit issue. Fix ElasticSearch plugin NPE. Fix agent classloader dead lock in certain situation. Fix agent log typo. Fix wrong component id in resettemplete plugin. Fix use transform ignore() in wrong way. Fix H2 query bug. Backend Support Trace Data Protocol v2. And Trace Data Protocol v1 is still supported. Support MySQL as storage. Support TiDB as storage. Support a new way to override application.yml by system env. Support service instance and endpoint alarm. Support namespace in istio receiver. Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Support backend trace sampling. Support Zipkin format again. Support init mode. Support namespace in Zookeeper cluster management. Support consul plugin in cluster module. OAL generate tool has been integrated into main repo, in the maven compile stage. Optimize trace paging query. Fix trace query don\u0026rsquo;t use fuzzy query in ElasticSearch storage. Fix alarm can\u0026rsquo;t be active in right way. Fix unnecessary condition in database and cache number query. Fix wrong namespace bug in ElasticSearch storage. Fix Remote clients selector error: / by zero . Fix segment TTL is not working. UI Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Fix TopN endpoint link doesn\u0026rsquo;t work right. Fix trace stack style. Fix CI. Document Add more agent setting documents. Add more contribution documents. Update user wall and powered-by page. Add RocketBot UI project link in document. All issues and pull requests are here\n6.0.0-alpha SkyWalking 6 is totally new milestone for the project. At this point, we are not just a distributing tracing system with analysis and visualization capabilities. We are an Observability Analysis Platform(OAL).\nThe core and most important features in v6 are\nSupport to collect telemetry data from different sources, such as multiple language agents and service mesh. Extensible stream analysis core. Make SQL and cache analysis available in core level, although haven\u0026rsquo;t provided in this release. Provide Observability Analysis Language(OAL) to make analysis metrics customization available. New GraphQL query protocol. Not binding with UI now. UI topology is better now. New alarm core provided. In alpha, only on service related metrics. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"660\"\u003e6.6.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e[\u003cstrong\u003eIMPORTANT\u003c/strong\u003e] Local span and exit span are not treated as endpoint detected at client …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.5.0/en/changes/changes-6.x/","title":"6.6.0"},{"body":"6.6.0 Project [IMPORTANT] Local span and exit span are not treated as endpoint detected at client and local. Only entry span is the endpoint. Reduce the load of register and memory cost. Support MiniKube, Istio and SkyWalking on K8s deployment in CI. Support Windows and MacOS build in GitHub Action CI. Support ElasticSearch 7 in official dist. Hundreds plugin cases have been added in GitHub Action CI process. Java Agent Remove the local/exit span operation name register mechanism. Add plugin for JDK Threading classes. Add plugin for Armeria. Support set operation name in async span. Enhance webflux plugin, related to Spring Gateway plugin. Webflux plugin is in optional, due to JDK8 required. Fix a possible deadlock. Fix NPE when OAL scripts are different in different OAP nodes, mostly in upgrading stage. Fix bug about wrong peer in ES plugin. Fix NPE in Spring plugin. Fix wrong class name in Dubbo 2.7 conflict patch. Fix spring annotation inheritance problem. OAP-Backend Remove the local/exit span operation name register mechanism. Remove client side endpoint register in service mesh. Service instance dependency and related metrics. Support min func in OAL Support apdex func in OAL Support custom ES config setting at the index level. Envoy ALS proto upgraded. Update JODA lib as bugs in UTC +13/+14. Support topN sample period configurable. Ignore no statement DB operations in slow SQL collection. Fix bug in docker-entrypoint.sh when using MySQL as storage UI Service topology enhancement. Dive into service, instance and endpoint metrics on topo map. Service instance dependency view and related metrics. Support using URL parameter in trace query page. Support apdex score in service page. Add service dependency metrics into metrics comparison. Fix alarm search not working. Document Update user list and user wall. Add document link for CLI. Add deployment guide of agent in Jetty case. Modify Consul cluster doc. Add document about injecting traceId into the logback with logstack in JSON format. ElementUI license and dependency added. All issues and pull requests are here\n6.5.0 Project TTL E2E test (#3437) Test coverage is back in pull request check status (#3503) Plugin tests begin to be migrated into main repo, and is in process. (#3528, #3756, #3751, etc.) Switch to SkyWalking CI (exclusive) nodes (#3546) MySQL storage e2e test. (#3648) E2E tests are verified in multiple jdk versions, jdk 8, 9, 11, 12 (#3657) Jenkins build jobs run only when necessary (#3662) OAP-Backend Support dynamically configure alarm settings (#3557) Language of instance could be null (#3485) Make query max window size configurable. (#3765) Remove two max size 500 limit. (#3748) Parameterize the cache size. (#3741) ServiceInstanceRelation set error id (#3683) Makes the scope of alarm message more semantic. (#3680) Add register persistent worker latency metrics (#3677) Fix more reasonable error (#3619) Add GraphQL getServiceInstance instanceUuid field. (#3595) Support namespace in Nacos cluster/configuration (#3578) Instead of datasource-settings.properties, use application.yml for MySQLStorageProvider (#3564) Provide consul dynamic configuration center implementation (#3560) Upgrade guava version to support higher jdk version (#3541) Sync latest als from envoy api (#3507) Set telemetry instanced id for Etcd and Nacos plugin (#3492) Support timeout configuration in agent and backend. (#3491) Make sure the cluster register happens before streaming process. (#3471) Agent supports custom properties. (#3367) Miscellaneous bug fixes (#3567) UI Feature: node detail display in topo circle-chart view. BugFix: the jvm-maxheap \u0026amp; jvm-maxnonheap is -1, free is no value Fix bug: time select operation not in effect Fix bug: language initialization failed Fix bug: not show instance language Feature: support the trace list display export png Feature: Metrics comparison view BugFix: Fix dashboard top throughput copy Java Agent Spring async scenario optimize (#3723) Support log4j2 AsyncLogger (#3715) Add config to collect PostgreSQL sql query params (#3695) Support namespace in Nacos cluster/configuration (#3578) Provide plugin for ehcache 2.x (#3575) Supporting RequestRateLimiterGatewayFilterFactory (#3538) Kafka-plugin compatible with KafkaTemplate (#3505) Add pulsar apm plugin (#3476) Spring-cloud-gateway traceId does not transmit #3411 (#3446) Gateway compatible with downstream loss (#3445) Provide cassandra java driver 3.x plugin (#3410) Fix SpringMVC4 NoSuchMethodError (#3408) BugFix: endpoint grouping rules may be not unique (#3510) Add feature to control the maximum agent log files (#3475) Agent support custom properties. (#3367) Add Light4j plugin (#3323) Document Remove travis badge (#3763) Replace user wall to typical users in readme page (#3719) Update istio docs according latest istio release (#3646) Use chart deploy sw docs (#3573) Reorganize the doc, and provide catalog (#3563) Committer vote and set up document. (#3496) Update als setup doc as istio 1.3 released (#3470) Fill faq reply in official document. (#3450) All issues and pull requests are here\n6.4.0 Project Highly recommend to upgrade due to Pxx metrics calculation bug. Make agent working in JDK9+ Module system. Java Agent Make agent working in JDK9+ Module system. Support Kafka 2.x client libs. Log error in OKHTTP OnFailure callback. Support injecting traceid into logstack appender in logback. Add OperationName(including endpoint name) length max threshold. Support using Regex to group operation name. Support Undertow routing handler. RestTemplate plugin support operation name grouping. Fix ClassCastException in Webflux plugin. Ordering zookeeper server list, to make it better in topology. Fix a Dubbo plugin incompatible issue. Fix MySQL 5 plugin issue. Make log writer cached. Optimize Spring Cloud Gateway plugin Fix and improve gRPC reconnect mechanism. Remove Disruptor dependency from agent. Backend Fix Pxx(p50,p75,p90,p95,p99) metrics func bug.(Critical) Support Gateway in backend analysis, even when it doesn\u0026rsquo;t have suitable language agent. Support using HTTPs SSL accessing ElasticSearch storage. Support Zookeeper ACL. Make alarm records listed in order. Fix Pxx data persistence failure in some cases. Fix some bugs in MySQL storage. Setup slow SQL length threshold. Fix TTL settings is not working as expected. Remove scope-meta file. UI Enhance alarm page layout. Support trace tree chart resize. Support trace auto completion when partial traces abandoned somehow. Fix dashboard endpoint slow chart. Add radial chart in topology page. Add trace table mode. Fix topology page bug. Fix calender js bug. Fix \u0026ldquo;The \u0026ldquo;topo-services\u0026rdquo; component did not update the data in time after modifying the time range on the topology page. Document Restore the broken Istio setup doc. Add etcd config center document. Correct span_limit_per_segment default value in document. Enhance plugin develop doc. Fix error description in build document. All issues and pull requests are here\n6.3.0 Project e2e tests have been added, and verify every pull request. Use ArrayList to replace LinkedList in DataCarrier for much better performance. Add plugin instrumentation definition check in CI. DataCarrier performance improvement by avoiding false-sharing. Java Agent Java agent supports JDK 9 - 12, but don\u0026rsquo;t support Java Module yet. Support JVM class auto instrumentation, cataloged as bootstrap plugin. Support JVM HttpClient and HttpsClient plugin.[Optional] Support backend upgrade without rebooting required. Open Redefine and Retransform by other agents. Support Servlet 2.5 in Jetty, Tomcat and SpringMVC plugins. Support Spring @Async plugin. Add new config item to restrict the length of span#peer. Refactor ContextManager#stopSpan. Add gRPC timeout. Support Logback AsyncAppender print tid Fix gRPC reconnect bug. Fix trace segment service doesn\u0026rsquo;t report onComplete. Fix wrong logger class name. Fix gRPC plugin bug. Fix ContextManager.activeSpan() API usage error. Backend Support agent reset command downstream when the storage is erased, mostly because of backend upgrade. Backend stream flow refactor. High dimensionality metrics(Hour/Day/Month) are changed to lower priority, to ease the storage payload. Add OAP metrics cache to ease the storage query payload and improve performance. Remove DataCarrier in trace persistent of ElasticSearch storage, by leveraging the elasticsearch bulk queue. OAP internal communication protocol changed. Don\u0026rsquo;t be compatible with old releases. Improve ElasticSearch storage bulk performance. Support etcd as dynamic configuration center. Simplify the PxxMetrics and ThermodynamicMetrics functions for better performance and GC. Support JVM metrics self observability. Add the new OAL runtime engine. Add gRPC timeout. Add Charset in the alarm web hook. Fix buffer lost. Fix dirty read in ElasticSearch storage. Fix bug of cluster management plugins in un-Mixed mode. Fix wrong logger class name. Fix delete bug in ElasticSearch when using namespace. Fix MySQL TTL failure. Totally remove IDs can't be null log, to avoid misleading. Fix provider has been initialized repeatedly. Adjust providers conflict log message. Fix using wrong gc time metrics in OAL. UI Fix refresh is not working after endpoint and instance changed. Fix endpoint selector but. Fix wrong copy value in slow traces. Fix can\u0026rsquo;t show trace when it is broken partially(Because of agent sampling or fail safe). Fix database and response time graph bugs. Document Add bootstrap plugin development document. Alarm documentation typo fixed. Clarify the Docker file purpose. Fix a license typo. All issues and pull requests are here\n6.2.0 Project ElasticSearch implementation performance improved, and CHANGED totally. Must delete all existing indexes to do upgrade. CI and Integration tests provided by ASF INFRA. Plan to enhance tests including e2e, plugin tests in all pull requests, powered by ASF INFRA. DataCarrier queue write index controller performance improvement. 3-5 times quicker than before. Add windows compile support in CI. Java Agent Support collect SQL parameter in MySQL plugin.[Optional] Support SolrJ plugin. Support RESTEasy plugin. Support Spring Gateway plugin for 2.1.x[Optional] TracingContext performance improvement. Support Apache ShardingSphere(incubating) plugin. Support span#error in application toolkit. Fix OOM by empty stack of exception. FIx wrong cause exception of stack in span log. Fix unclear the running context in SpringMVC plugin. Fix CPU usage accessor calculation issue. Fix SpringMVC plugin span not stop bug when doing HTTP forward. Fix lettuce plugin async commend bug and NPE. Fix webflux plugin cast exception. [CI]Support import check. Backend Support time serious ElasticSearch storage. Provide dynamic configuration module and implementation. Slow SQL threshold supports dynamic config today. Dynamic Configuration module provide multiple implementations, DCS(gRPC based), Zookeeper, Apollo, Nacos. Provide P99/95/90/75/50 charts in topology edge. New topology query protocol and implementation. Support Envoy ALS in Service Mesh scenario. Support Nacos cluster management. Enhance metric exporter. Run in increment and total modes. Fix module provider is loaded repeatedly. Change TOP slow SQL storage in ES to Text from Keyword, as too long text issue. Fix H2TopologyQuery tiny bug. Fix H2 log query bug.(No feature provided yet) Filtering pods not in \u0026lsquo;Running\u0026rsquo; phase in mesh scenario. Fix query alarm bug in MySQL and H2 storage. Codes refactor. UI Fix some ID is null query(s). Page refactor, especially time-picker, more friendly. Login removed. Trace timestamp visualization issue fixed. Provide P99/95/90/75/50 charts in topology edge. Change all P99/95/90/75/50 charts style. More readable. Fix 404 in trace page. Document Go2Sky project has been donated to SkyAPM, change document link. Add FAQ for ElasticSearch storage, and links from document. Add FAQ fro WebSphere installation. Add several open users. Add alarm webhook document. All issues and pull requests are here\n6.1.0 Project SkyWalking graduated as Apache Top Level Project.\nSupport compiling project agent, backend, UI separately. Java Agent Support Vert.x Core 3.x plugin. Support Apache Dubbo plugin. Support use_qualified_name_as_endpoint_name and use_qualified_name_as_operation_name configs in SpringMVC plugin. Support span async close APIs in core. Used in Vert.x plugin. Support MySQL 5,8 plugins. Support set instance id manually(optional). Support customize enhance trace plugin in optional list. Support to set peer in Entry Span. Support Zookeeper plugin. Fix Webflux plugin created unexpected Entry Span. Fix Kafka plugin NPE in Kafka 1.1+ Fix wrong operation name in postgre 8.x plugin. Fix RabbitMQ plugin NPE. Fix agent can\u0026rsquo;t run in JVM 6/7, remove module-info.class. Fix agent can\u0026rsquo;t work well, if there is whitespace in agent path. Fix Spring annotation bug and inheritance enhance issue. Fix CPU accessor bug. Backend Performance improved, especially in CPU limited environment. 3x improvement in service mesh scenario(no trace) in 8C16G VM. Significantly cost less CPU in low payload.\nSupport database metrics and SLOW SQL detection. Support to set max size of metadata query. And change default to 5000 from 100. Support ElasticSearch template for new feature in the future. Support shutdown Zipkin trace analysis, because it doesn\u0026rsquo;t fit production environment. Support log type, scope HTTP_ACCESS_LOG and query. No feature provided, prepare for future versions. Support .NET clr receiver. Support Jaeger trace format, no analysis. Support group endpoint name by regax rules in mesh receiver. Support disable statement in OAL. Support basic auth in ElasticSearch connection. Support metrics exporter module and gRPC implementor. Support \u0026gt;, \u0026lt;, \u0026gt;=, \u0026lt;= in OAL. Support role mode in backend. Support Envoy metrics. Support query segment by service instance. Support to set host/port manually at cluster coordinator, rather than based on core settings. Make sure OAP shutdown when it faces startup error. Support set separated gRPC/Jetty ip:port for receiver, default still use core settings. Fix JVM receiver bug. Fix wrong dest service in mesh analysis. Fix search doesn\u0026rsquo;t work as expected. Refactor ScopeDeclaration annotation. Refactor register lock mechanism. Add SmartSql component for .NET Add integration tests for ElasticSearch client. Add test cases for exporter. Add test cases for queue consume. UI RocketBot UI has been accepted and bind in this release. Support CLR metrics. Document Documents updated, matching Top Level Project requirement. UI licenses updated, according to RocketBot UI IP clearance. User wall and powered-by list updated. CN documents removed, only consider to provide by volunteer out of Apache. All issues and pull requests are here\n6.0.0-GA Java Agent Support gson plugin(optional). Support canal plugin. Fix missing ojdbc component id. Fix dubbo plugin conflict. Fix OpenTracing tag match bug. Fix a missing check in ignore plugin. Backend Adjust service inventory entity, to add properties. Adjust service instance inventory entity, to add properties. Add nodeType to service inventory entity. Fix when operation name of local and exit spans in ref, the segment lost. Fix the index names don\u0026rsquo;t show right in logs. Fix wrong alarm text. Add test case for span limit mechanism. Add telemetry module and prometheus implementation, with grafana setting. A refactor for register API in storage module. Fix H2 and MySQL endpoint dependency map miss upstream side. Optimize the inventory register and refactor the implementation. Speed up the trace buffer read. Fix and removed unnecessary inventory register operations. UI Add new trace view. Add word-break to tag value. Document Add two startup modes document. Add PHP agent links. Add some cn documents. Update year to 2019 User wall updated. Fix a wrong description in how-to-build doc. All issues and pull requests are here\n6.0.0-beta Protocol Provide Trace Data Protocol v2 Provide SkyWalking Cross Process Propagation Headers Protocol v2. Java Agent Support Trace Data Protocol v2 Support SkyWalking Cross Process Propagation Headers Protocol v2. Support SkyWalking Cross Process Propagation Headers Protocol v1 running in compatible way. Need declare open explicitly. Support SpringMVC 5 Support webflux Support a new way to override agent.config by system env. Span tag can override by explicit way. Fix Spring Controller Inherit issue. Fix ElasticSearch plugin NPE. Fix agent classloader dead lock in certain situation. Fix agent log typo. Fix wrong component id in resettemplete plugin. Fix use transform ignore() in wrong way. Fix H2 query bug. Backend Support Trace Data Protocol v2. And Trace Data Protocol v1 is still supported. Support MySQL as storage. Support TiDB as storage. Support a new way to override application.yml by system env. Support service instance and endpoint alarm. Support namespace in istio receiver. Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Support backend trace sampling. Support Zipkin format again. Support init mode. Support namespace in Zookeeper cluster management. Support consul plugin in cluster module. OAL generate tool has been integrated into main repo, in the maven compile stage. Optimize trace paging query. Fix trace query don\u0026rsquo;t use fuzzy query in ElasticSearch storage. Fix alarm can\u0026rsquo;t be active in right way. Fix unnecessary condition in database and cache number query. Fix wrong namespace bug in ElasticSearch storage. Fix Remote clients selector error: / by zero . Fix segment TTL is not working. UI Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Fix TopN endpoint link doesn\u0026rsquo;t work right. Fix trace stack style. Fix CI. Document Add more agent setting documents. Add more contribution documents. Update user wall and powered-by page. Add RocketBot UI project link in document. All issues and pull requests are here\n6.0.0-alpha SkyWalking 6 is totally new milestone for the project. At this point, we are not just a distributing tracing system with analysis and visualization capabilities. We are an Observability Analysis Platform(OAL).\nThe core and most important features in v6 are\nSupport to collect telemetry data from different sources, such as multiple language agents and service mesh. Extensible stream analysis core. Make SQL and cache analysis available in core level, although haven\u0026rsquo;t provided in this release. Provide Observability Analysis Language(OAL) to make analysis metrics customization available. New GraphQL query protocol. Not binding with UI now. UI topology is better now. New alarm core provided. In alpha, only on service related metrics. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"660\"\u003e6.6.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e[\u003cstrong\u003eIMPORTANT\u003c/strong\u003e] Local span and exit span are not treated as endpoint detected at client …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.6.0/en/changes/changes-6.x/","title":"6.6.0"},{"body":"6.6.0 Project [IMPORTANT] Local span and exit span are not treated as endpoint detected at client and local. Only entry span is the endpoint. Reduce the load of register and memory cost. Support MiniKube, Istio and SkyWalking on K8s deployment in CI. Support Windows and MacOS build in GitHub Action CI. Support ElasticSearch 7 in official dist. Hundreds plugin cases have been added in GitHub Action CI process. Java Agent Remove the local/exit span operation name register mechanism. Add plugin for JDK Threading classes. Add plugin for Armeria. Support set operation name in async span. Enhance webflux plugin, related to Spring Gateway plugin. Webflux plugin is in optional, due to JDK8 required. Fix a possible deadlock. Fix NPE when OAL scripts are different in different OAP nodes, mostly in upgrading stage. Fix bug about wrong peer in ES plugin. Fix NPE in Spring plugin. Fix wrong class name in Dubbo 2.7 conflict patch. Fix spring annotation inheritance problem. OAP-Backend Remove the local/exit span operation name register mechanism. Remove client side endpoint register in service mesh. Service instance dependency and related metrics. Support min func in OAL Support apdex func in OAL Support custom ES config setting at the index level. Envoy ALS proto upgraded. Update JODA lib as bugs in UTC +13/+14. Support topN sample period configurable. Ignore no statement DB operations in slow SQL collection. Fix bug in docker-entrypoint.sh when using MySQL as storage UI Service topology enhancement. Dive into service, instance and endpoint metrics on topo map. Service instance dependency view and related metrics. Support using URL parameter in trace query page. Support apdex score in service page. Add service dependency metrics into metrics comparison. Fix alarm search not working. Document Update user list and user wall. Add document link for CLI. Add deployment guide of agent in Jetty case. Modify Consul cluster doc. Add document about injecting traceId into the logback with logstack in JSON format. ElementUI license and dependency added. All issues and pull requests are here\n6.5.0 Project TTL E2E test (#3437) Test coverage is back in pull request check status (#3503) Plugin tests begin to be migrated into main repo, and is in process. (#3528, #3756, #3751, etc.) Switch to SkyWalking CI (exclusive) nodes (#3546) MySQL storage e2e test. (#3648) E2E tests are verified in multiple jdk versions, jdk 8, 9, 11, 12 (#3657) Jenkins build jobs run only when necessary (#3662) OAP-Backend Support dynamically configure alarm settings (#3557) Language of instance could be null (#3485) Make query max window size configurable. (#3765) Remove two max size 500 limit. (#3748) Parameterize the cache size. (#3741) ServiceInstanceRelation set error id (#3683) Makes the scope of alarm message more semantic. (#3680) Add register persistent worker latency metrics (#3677) Fix more reasonable error (#3619) Add GraphQL getServiceInstance instanceUuid field. (#3595) Support namespace in Nacos cluster/configuration (#3578) Instead of datasource-settings.properties, use application.yml for MySQLStorageProvider (#3564) Provide consul dynamic configuration center implementation (#3560) Upgrade guava version to support higher jdk version (#3541) Sync latest als from envoy api (#3507) Set telemetry instanced id for Etcd and Nacos plugin (#3492) Support timeout configuration in agent and backend. (#3491) Make sure the cluster register happens before streaming process. (#3471) Agent supports custom properties. (#3367) Miscellaneous bug fixes (#3567) UI Feature: node detail display in topo circle-chart view. BugFix: the jvm-maxheap \u0026amp; jvm-maxnonheap is -1, free is no value Fix bug: time select operation not in effect Fix bug: language initialization failed Fix bug: not show instance language Feature: support the trace list display export png Feature: Metrics comparison view BugFix: Fix dashboard top throughput copy Java Agent Spring async scenario optimize (#3723) Support log4j2 AsyncLogger (#3715) Add config to collect PostgreSQL sql query params (#3695) Support namespace in Nacos cluster/configuration (#3578) Provide plugin for ehcache 2.x (#3575) Supporting RequestRateLimiterGatewayFilterFactory (#3538) Kafka-plugin compatible with KafkaTemplate (#3505) Add pulsar apm plugin (#3476) Spring-cloud-gateway traceId does not transmit #3411 (#3446) Gateway compatible with downstream loss (#3445) Provide cassandra java driver 3.x plugin (#3410) Fix SpringMVC4 NoSuchMethodError (#3408) BugFix: endpoint grouping rules may be not unique (#3510) Add feature to control the maximum agent log files (#3475) Agent support custom properties. (#3367) Add Light4j plugin (#3323) Document Remove travis badge (#3763) Replace user wall to typical users in readme page (#3719) Update istio docs according latest istio release (#3646) Use chart deploy sw docs (#3573) Reorganize the doc, and provide catalog (#3563) Committer vote and set up document. (#3496) Update als setup doc as istio 1.3 released (#3470) Fill faq reply in official document. (#3450) All issues and pull requests are here\n6.4.0 Project Highly recommend to upgrade due to Pxx metrics calculation bug. Make agent working in JDK9+ Module system. Java Agent Make agent working in JDK9+ Module system. Support Kafka 2.x client libs. Log error in OKHTTP OnFailure callback. Support injecting traceid into logstack appender in logback. Add OperationName(including endpoint name) length max threshold. Support using Regex to group operation name. Support Undertow routing handler. RestTemplate plugin support operation name grouping. Fix ClassCastException in Webflux plugin. Ordering zookeeper server list, to make it better in topology. Fix a Dubbo plugin incompatible issue. Fix MySQL 5 plugin issue. Make log writer cached. Optimize Spring Cloud Gateway plugin Fix and improve gRPC reconnect mechanism. Remove Disruptor dependency from agent. Backend Fix Pxx(p50,p75,p90,p95,p99) metrics func bug.(Critical) Support Gateway in backend analysis, even when it doesn\u0026rsquo;t have suitable language agent. Support using HTTPs SSL accessing ElasticSearch storage. Support Zookeeper ACL. Make alarm records listed in order. Fix Pxx data persistence failure in some cases. Fix some bugs in MySQL storage. Setup slow SQL length threshold. Fix TTL settings is not working as expected. Remove scope-meta file. UI Enhance alarm page layout. Support trace tree chart resize. Support trace auto completion when partial traces abandoned somehow. Fix dashboard endpoint slow chart. Add radial chart in topology page. Add trace table mode. Fix topology page bug. Fix calender js bug. Fix \u0026ldquo;The \u0026ldquo;topo-services\u0026rdquo; component did not update the data in time after modifying the time range on the topology page. Document Restore the broken Istio setup doc. Add etcd config center document. Correct span_limit_per_segment default value in document. Enhance plugin develop doc. Fix error description in build document. All issues and pull requests are here\n6.3.0 Project e2e tests have been added, and verify every pull request. Use ArrayList to replace LinkedList in DataCarrier for much better performance. Add plugin instrumentation definition check in CI. DataCarrier performance improvement by avoiding false-sharing. Java Agent Java agent supports JDK 9 - 12, but don\u0026rsquo;t support Java Module yet. Support JVM class auto instrumentation, cataloged as bootstrap plugin. Support JVM HttpClient and HttpsClient plugin.[Optional] Support backend upgrade without rebooting required. Open Redefine and Retransform by other agents. Support Servlet 2.5 in Jetty, Tomcat and SpringMVC plugins. Support Spring @Async plugin. Add new config item to restrict the length of span#peer. Refactor ContextManager#stopSpan. Add gRPC timeout. Support Logback AsyncAppender print tid Fix gRPC reconnect bug. Fix trace segment service doesn\u0026rsquo;t report onComplete. Fix wrong logger class name. Fix gRPC plugin bug. Fix ContextManager.activeSpan() API usage error. Backend Support agent reset command downstream when the storage is erased, mostly because of backend upgrade. Backend stream flow refactor. High dimensionality metrics(Hour/Day/Month) are changed to lower priority, to ease the storage payload. Add OAP metrics cache to ease the storage query payload and improve performance. Remove DataCarrier in trace persistent of ElasticSearch storage, by leveraging the elasticsearch bulk queue. OAP internal communication protocol changed. Don\u0026rsquo;t be compatible with old releases. Improve ElasticSearch storage bulk performance. Support etcd as dynamic configuration center. Simplify the PxxMetrics and ThermodynamicMetrics functions for better performance and GC. Support JVM metrics self observability. Add the new OAL runtime engine. Add gRPC timeout. Add Charset in the alarm web hook. Fix buffer lost. Fix dirty read in ElasticSearch storage. Fix bug of cluster management plugins in un-Mixed mode. Fix wrong logger class name. Fix delete bug in ElasticSearch when using namespace. Fix MySQL TTL failure. Totally remove IDs can't be null log, to avoid misleading. Fix provider has been initialized repeatedly. Adjust providers conflict log message. Fix using wrong gc time metrics in OAL. UI Fix refresh is not working after endpoint and instance changed. Fix endpoint selector but. Fix wrong copy value in slow traces. Fix can\u0026rsquo;t show trace when it is broken partially(Because of agent sampling or fail safe). Fix database and response time graph bugs. Document Add bootstrap plugin development document. Alarm documentation typo fixed. Clarify the Docker file purpose. Fix a license typo. All issues and pull requests are here\n6.2.0 Project ElasticSearch implementation performance improved, and CHANGED totally. Must delete all existing indexes to do upgrade. CI and Integration tests provided by ASF INFRA. Plan to enhance tests including e2e, plugin tests in all pull requests, powered by ASF INFRA. DataCarrier queue write index controller performance improvement. 3-5 times quicker than before. Add windows compile support in CI. Java Agent Support collect SQL parameter in MySQL plugin.[Optional] Support SolrJ plugin. Support RESTEasy plugin. Support Spring Gateway plugin for 2.1.x[Optional] TracingContext performance improvement. Support Apache ShardingSphere(incubating) plugin. Support span#error in application toolkit. Fix OOM by empty stack of exception. FIx wrong cause exception of stack in span log. Fix unclear the running context in SpringMVC plugin. Fix CPU usage accessor calculation issue. Fix SpringMVC plugin span not stop bug when doing HTTP forward. Fix lettuce plugin async commend bug and NPE. Fix webflux plugin cast exception. [CI]Support import check. Backend Support time serious ElasticSearch storage. Provide dynamic configuration module and implementation. Slow SQL threshold supports dynamic config today. Dynamic Configuration module provide multiple implementations, DCS(gRPC based), Zookeeper, Apollo, Nacos. Provide P99/95/90/75/50 charts in topology edge. New topology query protocol and implementation. Support Envoy ALS in Service Mesh scenario. Support Nacos cluster management. Enhance metric exporter. Run in increment and total modes. Fix module provider is loaded repeatedly. Change TOP slow SQL storage in ES to Text from Keyword, as too long text issue. Fix H2TopologyQuery tiny bug. Fix H2 log query bug.(No feature provided yet) Filtering pods not in \u0026lsquo;Running\u0026rsquo; phase in mesh scenario. Fix query alarm bug in MySQL and H2 storage. Codes refactor. UI Fix some ID is null query(s). Page refactor, especially time-picker, more friendly. Login removed. Trace timestamp visualization issue fixed. Provide P99/95/90/75/50 charts in topology edge. Change all P99/95/90/75/50 charts style. More readable. Fix 404 in trace page. Document Go2Sky project has been donated to SkyAPM, change document link. Add FAQ for ElasticSearch storage, and links from document. Add FAQ fro WebSphere installation. Add several open users. Add alarm webhook document. All issues and pull requests are here\n6.1.0 Project SkyWalking graduated as Apache Top Level Project.\nSupport compiling project agent, backend, UI separately. Java Agent Support Vert.x Core 3.x plugin. Support Apache Dubbo plugin. Support use_qualified_name_as_endpoint_name and use_qualified_name_as_operation_name configs in SpringMVC plugin. Support span async close APIs in core. Used in Vert.x plugin. Support MySQL 5,8 plugins. Support set instance id manually(optional). Support customize enhance trace plugin in optional list. Support to set peer in Entry Span. Support Zookeeper plugin. Fix Webflux plugin created unexpected Entry Span. Fix Kafka plugin NPE in Kafka 1.1+ Fix wrong operation name in postgre 8.x plugin. Fix RabbitMQ plugin NPE. Fix agent can\u0026rsquo;t run in JVM 6/7, remove module-info.class. Fix agent can\u0026rsquo;t work well, if there is whitespace in agent path. Fix Spring annotation bug and inheritance enhance issue. Fix CPU accessor bug. Backend Performance improved, especially in CPU limited environment. 3x improvement in service mesh scenario(no trace) in 8C16G VM. Significantly cost less CPU in low payload.\nSupport database metrics and SLOW SQL detection. Support to set max size of metadata query. And change default to 5000 from 100. Support ElasticSearch template for new feature in the future. Support shutdown Zipkin trace analysis, because it doesn\u0026rsquo;t fit production environment. Support log type, scope HTTP_ACCESS_LOG and query. No feature provided, prepare for future versions. Support .NET clr receiver. Support Jaeger trace format, no analysis. Support group endpoint name by regax rules in mesh receiver. Support disable statement in OAL. Support basic auth in ElasticSearch connection. Support metrics exporter module and gRPC implementor. Support \u0026gt;, \u0026lt;, \u0026gt;=, \u0026lt;= in OAL. Support role mode in backend. Support Envoy metrics. Support query segment by service instance. Support to set host/port manually at cluster coordinator, rather than based on core settings. Make sure OAP shutdown when it faces startup error. Support set separated gRPC/Jetty ip:port for receiver, default still use core settings. Fix JVM receiver bug. Fix wrong dest service in mesh analysis. Fix search doesn\u0026rsquo;t work as expected. Refactor ScopeDeclaration annotation. Refactor register lock mechanism. Add SmartSql component for .NET Add integration tests for ElasticSearch client. Add test cases for exporter. Add test cases for queue consume. UI RocketBot UI has been accepted and bind in this release. Support CLR metrics. Document Documents updated, matching Top Level Project requirement. UI licenses updated, according to RocketBot UI IP clearance. User wall and powered-by list updated. CN documents removed, only consider to provide by volunteer out of Apache. All issues and pull requests are here\n6.0.0-GA Java Agent Support gson plugin(optional). Support canal plugin. Fix missing ojdbc component id. Fix dubbo plugin conflict. Fix OpenTracing tag match bug. Fix a missing check in ignore plugin. Backend Adjust service inventory entity, to add properties. Adjust service instance inventory entity, to add properties. Add nodeType to service inventory entity. Fix when operation name of local and exit spans in ref, the segment lost. Fix the index names don\u0026rsquo;t show right in logs. Fix wrong alarm text. Add test case for span limit mechanism. Add telemetry module and prometheus implementation, with grafana setting. A refactor for register API in storage module. Fix H2 and MySQL endpoint dependency map miss upstream side. Optimize the inventory register and refactor the implementation. Speed up the trace buffer read. Fix and removed unnecessary inventory register operations. UI Add new trace view. Add word-break to tag value. Document Add two startup modes document. Add PHP agent links. Add some cn documents. Update year to 2019 User wall updated. Fix a wrong description in how-to-build doc. All issues and pull requests are here\n6.0.0-beta Protocol Provide Trace Data Protocol v2 Provide SkyWalking Cross Process Propagation Headers Protocol v2. Java Agent Support Trace Data Protocol v2 Support SkyWalking Cross Process Propagation Headers Protocol v2. Support SkyWalking Cross Process Propagation Headers Protocol v1 running in compatible way. Need declare open explicitly. Support SpringMVC 5 Support webflux Support a new way to override agent.config by system env. Span tag can override by explicit way. Fix Spring Controller Inherit issue. Fix ElasticSearch plugin NPE. Fix agent classloader dead lock in certain situation. Fix agent log typo. Fix wrong component id in resettemplete plugin. Fix use transform ignore() in wrong way. Fix H2 query bug. Backend Support Trace Data Protocol v2. And Trace Data Protocol v1 is still supported. Support MySQL as storage. Support TiDB as storage. Support a new way to override application.yml by system env. Support service instance and endpoint alarm. Support namespace in istio receiver. Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Support backend trace sampling. Support Zipkin format again. Support init mode. Support namespace in Zookeeper cluster management. Support consul plugin in cluster module. OAL generate tool has been integrated into main repo, in the maven compile stage. Optimize trace paging query. Fix trace query don\u0026rsquo;t use fuzzy query in ElasticSearch storage. Fix alarm can\u0026rsquo;t be active in right way. Fix unnecessary condition in database and cache number query. Fix wrong namespace bug in ElasticSearch storage. Fix Remote clients selector error: / by zero . Fix segment TTL is not working. UI Support service throughput(cpm), successful rate(sla), avg response time and p99/p95/p90/p75/p50 response time. Fix TopN endpoint link doesn\u0026rsquo;t work right. Fix trace stack style. Fix CI. Document Add more agent setting documents. Add more contribution documents. Update user wall and powered-by page. Add RocketBot UI project link in document. All issues and pull requests are here\n6.0.0-alpha SkyWalking 6 is totally new milestone for the project. At this point, we are not just a distributing tracing system with analysis and visualization capabilities. We are an Observability Analysis Platform(OAL).\nThe core and most important features in v6 are\nSupport to collect telemetry data from different sources, such as multiple language agents and service mesh. Extensible stream analysis core. Make SQL and cache analysis available in core level, although haven\u0026rsquo;t provided in this release. Provide Observability Analysis Language(OAL) to make analysis metrics customization available. New GraphQL query protocol. Not binding with UI now. UI topology is better now. New alarm core provided. In alpha, only on service related metrics. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"660\"\u003e6.6.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e[\u003cstrong\u003eIMPORTANT\u003c/strong\u003e] Local span and exit span are not treated as endpoint detected at client …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.7.0/en/changes/changes-6.x/","title":"6.6.0"},{"body":"7.0.0 Project SkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The minimal requirement of JDK is JDK8. Support method performance profile. Provide new E2E test framework. Remove AppVeyor from the CI, use GitHub action only. Provide new plugin test tool. Don\u0026rsquo;t support SkyWalking v5 agent in-wire and out-wire protocol. v6 is required. Java Agent Add lazy injection API in the agent core. Support Servlet 2.5 in the Struts plugin. Fix RestTemplate plugin ClassCastException in the Async call. Add Finagle plugin. Add test cases of H2 and struts. Add Armeria 0.98 plugin. Fix ElasticSearch plugin bug. Fix EHCache plugin bug. Fix a potential I/O leak. Support Oracle SID mode. Update Byte-buddy core. Performance tuning: replace AtomicInteger with AtomicIntegerFieldUpdater. Add AVRO plugin. Update to JDK 1.8 Optimize the ignore plugin. Enhance the gRPC plugin. Add Kotlin Coroutine plugin. Support HTTP parameter collection in Tomcat and SpringMVC plugin. Add @Tag annotation in the application toolkit. Move Lettuce into the default plugin list. Move Webflux into the default plugin list. Add HttpClient 3.x plugin. OAP-Backend Support InfluxDB as a new storage option. Add selector in the application.yml. Make the provider activation more flexible through System ENV. Support sub-topology map query. Support gRPC SSL. Support HTTP protocol for agent. Support Nginx LUA agent. Support skip the instance relationship analysis if some agents doesn\u0026rsquo;t have upstream address, currently for LUA agent. Support metrics entity name in the storage. Optional, default OFF. Merge the HOUR and DAY metrics into MINUTE in the ElasticSearch storage implementation. Reduce the payload for ElasticSearch server. Support change detection mechanism in DCS. Support Daily step in the ElasticSearch storage implementation for low traffic system. Provide profile export tool. Support alarm gRPC hook. Fix PHP language doesn\u0026rsquo;t show up on the instance page. Add more comments in the source codes. Add a new metrics type, multiple linears. Fix thread concurrency issue in the alarm core. UI Support custom topology definition. Document Add FAQ about python2 command required in the compiling. Add doc about new e2e framework. Add doc about the new profile feature. Powered-by page updated. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"700\"\u003e7.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/latest/en/changes/changes-7.0.0/","title":"7.0.0"},{"body":"7.0.0 Project SkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The minimal requirement of JDK is JDK8. Support method performance profile. Provide new E2E test framework. Remove AppVeyor from the CI, use GitHub action only. Provide new plugin test tool. Don\u0026rsquo;t support SkyWalking v5 agent in-wire and out-wire protocol. v6 is required. Java Agent Add lazy injection API in the agent core. Support Servlet 2.5 in the Struts plugin. Fix RestTemplate plugin ClassCastException in the Async call. Add Finagle plugin. Add test cases of H2 and struts. Add Armeria 0.98 plugin. Fix ElasticSearch plugin bug. Fix EHCache plugin bug. Fix a potential I/O leak. Support Oracle SID mode. Update Byte-buddy core. Performance tuning: replace AtomicInteger with AtomicIntegerFieldUpdater. Add AVRO plugin. Update to JDK 1.8 Optimize the ignore plugin. Enhance the gRPC plugin. Add Kotlin Coroutine plugin. Support HTTP parameter collection in Tomcat and SpringMVC plugin. Add @Tag annotation in the application toolkit. Move Lettuce into the default plugin list. Move Webflux into the default plugin list. Add HttpClient 3.x plugin. OAP-Backend Support InfluxDB as a new storage option. Add selector in the application.yml. Make the provider activation more flexible through System ENV. Support sub-topology map query. Support gRPC SSL. Support HTTP protocol for agent. Support Nginx LUA agent. Support skip the instance relationship analysis if some agents doesn\u0026rsquo;t have upstream address, currently for LUA agent. Support metrics entity name in the storage. Optional, default OFF. Merge the HOUR and DAY metrics into MINUTE in the ElasticSearch storage implementation. Reduce the payload for ElasticSearch server. Support change detection mechanism in DCS. Support Daily step in the ElasticSearch storage implementation for low traffic system. Provide profile export tool. Support alarm gRPC hook. Fix PHP language doesn\u0026rsquo;t show up on the instance page. Add more comments in the source codes. Add a new metrics type, multiple linears. Fix thread concurrency issue in the alarm core. UI Support custom topology definition. Document Add FAQ about python2 command required in the compiling. Add doc about new e2e framework. Add doc about the new profile feature. Powered-by page updated. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"700\"\u003e7.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/next/en/changes/changes-7.0.0/","title":"7.0.0"},{"body":"7.0.0 Project SkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The minimal requirement of JDK is JDK8. Support method performance profile. Provide new E2E test framework. Remove AppVeyor from the CI, use GitHub action only. Provide new plugin test tool. Don\u0026rsquo;t support SkyWalking v5 agent in-wire and out-wire protocol. v6 is required. Java Agent Add lazy injection API in the agent core. Support Servlet 2.5 in the Struts plugin. Fix RestTemplate plugin ClassCastException in the Async call. Add Finagle plugin. Add test cases of H2 and struts. Add Armeria 0.98 plugin. Fix ElasticSearch plugin bug. Fix EHCache plugin bug. Fix a potential I/O leak. Support Oracle SID mode. Update Byte-buddy core. Performance tuning: replace AtomicInteger with AtomicIntegerFieldUpdater. Add AVRO plugin. Update to JDK 1.8 Optimize the ignore plugin. Enhance the gRPC plugin. Add Kotlin Coroutine plugin. Support HTTP parameter collection in Tomcat and SpringMVC plugin. Add @Tag annotation in the application toolkit. Move Lettuce into the default plugin list. Move Webflux into the default plugin list. Add HttpClient 3.x plugin. OAP-Backend Support InfluxDB as a new storage option. Add selector in the application.yml. Make the provider activation more flexible through System ENV. Support sub-topology map query. Support gRPC SSL. Support HTTP protocol for agent. Support Nginx LUA agent. Support skip the instance relationship analysis if some agents doesn\u0026rsquo;t have upstream address, currently for LUA agent. Support metrics entity name in the storage. Optional, default OFF. Merge the HOUR and DAY metrics into MINUTE in the ElasticSearch storage implementation. Reduce the payload for ElasticSearch server. Support change detection mechanism in DCS. Support Daily step in the ElasticSearch storage implementation for low traffic system. Provide profile export tool. Support alarm gRPC hook. Fix PHP language doesn\u0026rsquo;t show up on the instance page. Add more comments in the source codes. Add a new metrics type, multiple linears. Fix thread concurrency issue in the alarm core. UI Support custom topology definition. Document Add FAQ about python2 command required in the compiling. Add doc about new e2e framework. Add doc about the new profile feature. Powered-by page updated. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"700\"\u003e7.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.0.0/en/changes/changes-7.0.0/","title":"7.0.0"},{"body":"7.0.0 Project SkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The minimal requirement of JDK is JDK8. Support method performance profile. Provide new E2E test framework. Remove AppVeyor from the CI, use GitHub action only. Provide new plugin test tool. Don\u0026rsquo;t support SkyWalking v5 agent in-wire and out-wire protocol. v6 is required. Java Agent Add lazy injection API in the agent core. Support Servlet 2.5 in the Struts plugin. Fix RestTemplate plugin ClassCastException in the Async call. Add Finagle plugin. Add test cases of H2 and struts. Add Armeria 0.98 plugin. Fix ElasticSearch plugin bug. Fix EHCache plugin bug. Fix a potential I/O leak. Support Oracle SID mode. Update Byte-buddy core. Performance tuning: replace AtomicInteger with AtomicIntegerFieldUpdater. Add AVRO plugin. Update to JDK 1.8 Optimize the ignore plugin. Enhance the gRPC plugin. Add Kotlin Coroutine plugin. Support HTTP parameter collection in Tomcat and SpringMVC plugin. Add @Tag annotation in the application toolkit. Move Lettuce into the default plugin list. Move Webflux into the default plugin list. Add HttpClient 3.x plugin. OAP-Backend Support InfluxDB as a new storage option. Add selector in the application.yml. Make the provider activation more flexible through System ENV. Support sub-topology map query. Support gRPC SSL. Support HTTP protocol for agent. Support Nginx LUA agent. Support skip the instance relationship analysis if some agents doesn\u0026rsquo;t have upstream address, currently for LUA agent. Support metrics entity name in the storage. Optional, default OFF. Merge the HOUR and DAY metrics into MINUTE in the ElasticSearch storage implementation. Reduce the payload for ElasticSearch server. Support change detection mechanism in DCS. Support Daily step in the ElasticSearch storage implementation for low traffic system. Provide profile export tool. Support alarm gRPC hook. Fix PHP language doesn\u0026rsquo;t show up on the instance page. Add more comments in the source codes. Add a new metrics type, multiple linears. Fix thread concurrency issue in the alarm core. UI Support custom topology definition. Document Add FAQ about python2 command required in the compiling. Add doc about new e2e framework. Add doc about the new profile feature. Powered-by page updated. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"700\"\u003e7.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.0.1/en/changes/changes-7.0.0/","title":"7.0.0"},{"body":"7.0.0 Project SkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The minimal requirement of JDK is JDK8. Support method performance profile. Provide new E2E test framework. Remove AppVeyor from the CI, use GitHub action only. Provide new plugin test tool. Don\u0026rsquo;t support SkyWalking v5 agent in-wire and out-wire protocol. v6 is required. Java Agent Add lazy injection API in the agent core. Support Servlet 2.5 in the Struts plugin. Fix RestTemplate plugin ClassCastException in the Async call. Add Finagle plugin. Add test cases of H2 and struts. Add Armeria 0.98 plugin. Fix ElasticSearch plugin bug. Fix EHCache plugin bug. Fix a potential I/O leak. Support Oracle SID mode. Update Byte-buddy core. Performance tuning: replace AtomicInteger with AtomicIntegerFieldUpdater. Add AVRO plugin. Update to JDK 1.8 Optimize the ignore plugin. Enhance the gRPC plugin. Add Kotlin Coroutine plugin. Support HTTP parameter collection in Tomcat and SpringMVC plugin. Add @Tag annotation in the application toolkit. Move Lettuce into the default plugin list. Move Webflux into the default plugin list. Add HttpClient 3.x plugin. OAP-Backend Support InfluxDB as a new storage option. Add selector in the application.yml. Make the provider activation more flexible through System ENV. Support sub-topology map query. Support gRPC SSL. Support HTTP protocol for agent. Support Nginx LUA agent. Support skip the instance relationship analysis if some agents doesn\u0026rsquo;t have upstream address, currently for LUA agent. Support metrics entity name in the storage. Optional, default OFF. Merge the HOUR and DAY metrics into MINUTE in the ElasticSearch storage implementation. Reduce the payload for ElasticSearch server. Support change detection mechanism in DCS. Support Daily step in the ElasticSearch storage implementation for low traffic system. Provide profile export tool. Support alarm gRPC hook. Fix PHP language doesn\u0026rsquo;t show up on the instance page. Add more comments in the source codes. Add a new metrics type, multiple linears. Fix thread concurrency issue in the alarm core. UI Support custom topology definition. Document Add FAQ about python2 command required in the compiling. Add doc about new e2e framework. Add doc about the new profile feature. Powered-by page updated. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"700\"\u003e7.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.1.0/en/changes/changes-7.0.0/","title":"7.0.0"},{"body":"7.0.0 Project SkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The minimal requirement of JDK is JDK8. Support method performance profile. Provide new E2E test framework. Remove AppVeyor from the CI, use GitHub action only. Provide new plugin test tool. Don\u0026rsquo;t support SkyWalking v5 agent in-wire and out-wire protocol. v6 is required. Java Agent Add lazy injection API in the agent core. Support Servlet 2.5 in the Struts plugin. Fix RestTemplate plugin ClassCastException in the Async call. Add Finagle plugin. Add test cases of H2 and struts. Add Armeria 0.98 plugin. Fix ElasticSearch plugin bug. Fix EHCache plugin bug. Fix a potential I/O leak. Support Oracle SID mode. Update Byte-buddy core. Performance tuning: replace AtomicInteger with AtomicIntegerFieldUpdater. Add AVRO plugin. Update to JDK 1.8 Optimize the ignore plugin. Enhance the gRPC plugin. Add Kotlin Coroutine plugin. Support HTTP parameter collection in Tomcat and SpringMVC plugin. Add @Tag annotation in the application toolkit. Move Lettuce into the default plugin list. Move Webflux into the default plugin list. Add HttpClient 3.x plugin. OAP-Backend Support InfluxDB as a new storage option. Add selector in the application.yml. Make the provider activation more flexible through System ENV. Support sub-topology map query. Support gRPC SSL. Support HTTP protocol for agent. Support Nginx LUA agent. Support skip the instance relationship analysis if some agents doesn\u0026rsquo;t have upstream address, currently for LUA agent. Support metrics entity name in the storage. Optional, default OFF. Merge the HOUR and DAY metrics into MINUTE in the ElasticSearch storage implementation. Reduce the payload for ElasticSearch server. Support change detection mechanism in DCS. Support Daily step in the ElasticSearch storage implementation for low traffic system. Provide profile export tool. Support alarm gRPC hook. Fix PHP language doesn\u0026rsquo;t show up on the instance page. Add more comments in the source codes. Add a new metrics type, multiple linears. Fix thread concurrency issue in the alarm core. UI Support custom topology definition. Document Add FAQ about python2 command required in the compiling. Add doc about new e2e framework. Add doc about the new profile feature. Powered-by page updated. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"700\"\u003e7.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.2.0/en/changes/changes-7.0.0/","title":"7.0.0"},{"body":"7.0.0 Project SkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The minimal requirement of JDK is JDK8. Support method performance profile. Provide new E2E test framework. Remove AppVeyor from the CI, use GitHub action only. Provide new plugin test tool. Don\u0026rsquo;t support SkyWalking v5 agent in-wire and out-wire protocol. v6 is required. Java Agent Add lazy injection API in the agent core. Support Servlet 2.5 in the Struts plugin. Fix RestTemplate plugin ClassCastException in the Async call. Add Finagle plugin. Add test cases of H2 and struts. Add Armeria 0.98 plugin. Fix ElasticSearch plugin bug. Fix EHCache plugin bug. Fix a potential I/O leak. Support Oracle SID mode. Update Byte-buddy core. Performance tuning: replace AtomicInteger with AtomicIntegerFieldUpdater. Add AVRO plugin. Update to JDK 1.8 Optimize the ignore plugin. Enhance the gRPC plugin. Add Kotlin Coroutine plugin. Support HTTP parameter collection in Tomcat and SpringMVC plugin. Add @Tag annotation in the application toolkit. Move Lettuce into the default plugin list. Move Webflux into the default plugin list. Add HttpClient 3.x plugin. OAP-Backend Support InfluxDB as a new storage option. Add selector in the application.yml. Make the provider activation more flexible through System ENV. Support sub-topology map query. Support gRPC SSL. Support HTTP protocol for agent. Support Nginx LUA agent. Support skip the instance relationship analysis if some agents doesn\u0026rsquo;t have upstream address, currently for LUA agent. Support metrics entity name in the storage. Optional, default OFF. Merge the HOUR and DAY metrics into MINUTE in the ElasticSearch storage implementation. Reduce the payload for ElasticSearch server. Support change detection mechanism in DCS. Support Daily step in the ElasticSearch storage implementation for low traffic system. Provide profile export tool. Support alarm gRPC hook. Fix PHP language doesn\u0026rsquo;t show up on the instance page. Add more comments in the source codes. Add a new metrics type, multiple linears. Fix thread concurrency issue in the alarm core. UI Support custom topology definition. Document Add FAQ about python2 command required in the compiling. Add doc about new e2e framework. Add doc about the new profile feature. Powered-by page updated. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"700\"\u003e7.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.3.0/en/changes/changes-7.0.0/","title":"7.0.0"},{"body":"7.0.0 Project SkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The minimal requirement of JDK is JDK8. Support method performance profile. Provide new E2E test framework. Remove AppVeyor from the CI, use GitHub action only. Provide new plugin test tool. Don\u0026rsquo;t support SkyWalking v5 agent in-wire and out-wire protocol. v6 is required. Java Agent Add lazy injection API in the agent core. Support Servlet 2.5 in the Struts plugin. Fix RestTemplate plugin ClassCastException in the Async call. Add Finagle plugin. Add test cases of H2 and struts. Add Armeria 0.98 plugin. Fix ElasticSearch plugin bug. Fix EHCache plugin bug. Fix a potential I/O leak. Support Oracle SID mode. Update Byte-buddy core. Performance tuning: replace AtomicInteger with AtomicIntegerFieldUpdater. Add AVRO plugin. Update to JDK 1.8 Optimize the ignore plugin. Enhance the gRPC plugin. Add Kotlin Coroutine plugin. Support HTTP parameter collection in Tomcat and SpringMVC plugin. Add @Tag annotation in the application toolkit. Move Lettuce into the default plugin list. Move Webflux into the default plugin list. Add HttpClient 3.x plugin. OAP-Backend Support InfluxDB as a new storage option. Add selector in the application.yml. Make the provider activation more flexible through System ENV. Support sub-topology map query. Support gRPC SSL. Support HTTP protocol for agent. Support Nginx LUA agent. Support skip the instance relationship analysis if some agents doesn\u0026rsquo;t have upstream address, currently for LUA agent. Support metrics entity name in the storage. Optional, default OFF. Merge the HOUR and DAY metrics into MINUTE in the ElasticSearch storage implementation. Reduce the payload for ElasticSearch server. Support change detection mechanism in DCS. Support Daily step in the ElasticSearch storage implementation for low traffic system. Provide profile export tool. Support alarm gRPC hook. Fix PHP language doesn\u0026rsquo;t show up on the instance page. Add more comments in the source codes. Add a new metrics type, multiple linears. Fix thread concurrency issue in the alarm core. UI Support custom topology definition. Document Add FAQ about python2 command required in the compiling. Add doc about new e2e framework. Add doc about the new profile feature. Powered-by page updated. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"700\"\u003e7.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.4.0/en/changes/changes-7.0.0/","title":"7.0.0"},{"body":"7.0.0 Project SkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The minimal requirement of JDK is JDK8. Support method performance profile. Provide new E2E test framework. Remove AppVeyor from the CI, use GitHub action only. Provide new plugin test tool. Don\u0026rsquo;t support SkyWalking v5 agent in-wire and out-wire protocol. v6 is required. Java Agent Add lazy injection API in the agent core. Support Servlet 2.5 in the Struts plugin. Fix RestTemplate plugin ClassCastException in the Async call. Add Finagle plugin. Add test cases of H2 and struts. Add Armeria 0.98 plugin. Fix ElasticSearch plugin bug. Fix EHCache plugin bug. Fix a potential I/O leak. Support Oracle SID mode. Update Byte-buddy core. Performance tuning: replace AtomicInteger with AtomicIntegerFieldUpdater. Add AVRO plugin. Update to JDK 1.8 Optimize the ignore plugin. Enhance the gRPC plugin. Add Kotlin Coroutine plugin. Support HTTP parameter collection in Tomcat and SpringMVC plugin. Add @Tag annotation in the application toolkit. Move Lettuce into the default plugin list. Move Webflux into the default plugin list. Add HttpClient 3.x plugin. OAP-Backend Support InfluxDB as a new storage option. Add selector in the application.yml. Make the provider activation more flexible through System ENV. Support sub-topology map query. Support gRPC SSL. Support HTTP protocol for agent. Support Nginx LUA agent. Support skip the instance relationship analysis if some agents doesn\u0026rsquo;t have upstream address, currently for LUA agent. Support metrics entity name in the storage. Optional, default OFF. Merge the HOUR and DAY metrics into MINUTE in the ElasticSearch storage implementation. Reduce the payload for ElasticSearch server. Support change detection mechanism in DCS. Support Daily step in the ElasticSearch storage implementation for low traffic system. Provide profile export tool. Support alarm gRPC hook. Fix PHP language doesn\u0026rsquo;t show up on the instance page. Add more comments in the source codes. Add a new metrics type, multiple linears. Fix thread concurrency issue in the alarm core. UI Support custom topology definition. Document Add FAQ about python2 command required in the compiling. Add doc about new e2e framework. Add doc about the new profile feature. Powered-by page updated. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"700\"\u003e7.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v11.0.0/en/changes/changes-7.0.0/","title":"7.0.0"},{"body":"7.0.0 Project SkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The minimal requirement of JDK is JDK8. Support method performance profile. Provide new E2E test framework. Remove AppVeyor from the CI, use GitHub action only. Provide new plugin test tool. Don\u0026rsquo;t support SkyWalking v5 agent in-wire and out-wire protocol. v6 is required. Java Agent Add lazy injection API in the agent core. Support Servlet 2.5 in the Struts plugin. Fix RestTemplate plugin ClassCastException in the Async call. Add Finagle plugin. Add test cases of H2 and struts. Add Armeria 0.98 plugin. Fix ElasticSearch plugin bug. Fix EHCache plugin bug. Fix a potential I/O leak. Support Oracle SID mode. Update Byte-buddy core. Performance tuning: replace AtomicInteger with AtomicIntegerFieldUpdater. Add AVRO plugin. Update to JDK 1.8 Optimize the ignore plugin. Enhance the gRPC plugin. Add Kotlin Coroutine plugin. Support HTTP parameter collection in Tomcat and SpringMVC plugin. Add @Tag annotation in the application toolkit. Move Lettuce into the default plugin list. Move Webflux into the default plugin list. Add HttpClient 3.x plugin. OAP-Backend Support InfluxDB as a new storage option. Add selector in the application.yml. Make the provider activation more flexible through System ENV. Support sub-topology map query. Support gRPC SSL. Support HTTP protocol for agent. Support Nginx LUA agent. Support skip the instance relationship analysis if some agents doesn\u0026rsquo;t have upstream address, currently for LUA agent. Support metrics entity name in the storage. Optional, default OFF. Merge the HOUR and DAY metrics into MINUTE in the ElasticSearch storage implementation. Reduce the payload for ElasticSearch server. Support change detection mechanism in DCS. Support Daily step in the ElasticSearch storage implementation for low traffic system. Provide profile export tool. Support alarm gRPC hook. Fix PHP language doesn\u0026rsquo;t show up on the instance page. Add more comments in the source codes. Add a new metrics type, multiple linears. Fix thread concurrency issue in the alarm core. UI Support custom topology definition. Document Add FAQ about python2 command required in the compiling. Add doc about new e2e framework. Add doc about the new profile feature. Powered-by page updated. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"700\"\u003e7.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.4.0/en/changes/changes-7.0.0/","title":"7.0.0"},{"body":"7.0.0 Project SkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The minimal requirement of JDK is JDK8. Support method performance profile. Provide new E2E test framework. Remove AppVeyor from the CI, use GitHub action only. Provide new plugin test tool. Don\u0026rsquo;t support SkyWalking v5 agent in-wire and out-wire protocol. v6 is required. Java Agent Add lazy injection API in the agent core. Support Servlet 2.5 in the Struts plugin. Fix RestTemplate plugin ClassCastException in the Async call. Add Finagle plugin. Add test cases of H2 and struts. Add Armeria 0.98 plugin. Fix ElasticSearch plugin bug. Fix EHCache plugin bug. Fix a potential I/O leak. Support Oracle SID mode. Update Byte-buddy core. Performance tuning: replace AtomicInteger with AtomicIntegerFieldUpdater. Add AVRO plugin. Update to JDK 1.8 Optimize the ignore plugin. Enhance the gRPC plugin. Add Kotlin Coroutine plugin. Support HTTP parameter collection in Tomcat and SpringMVC plugin. Add @Tag annotation in the application toolkit. Move Lettuce into the default plugin list. Move Webflux into the default plugin list. Add HttpClient 3.x plugin. OAP-Backend Support InfluxDB as a new storage option. Add selector in the application.yml. Make the provider activation more flexible through System ENV. Support sub-topology map query. Support gRPC SSL. Support HTTP protocol for agent. Support Nginx LUA agent. Support skip the instance relationship analysis if some agents doesn\u0026rsquo;t have upstream address, currently for LUA agent. Support metrics entity name in the storage. Optional, default OFF. Merge the HOUR and DAY metrics into MINUTE in the ElasticSearch storage implementation. Reduce the payload for ElasticSearch server. Support change detection mechanism in DCS. Support Daily step in the ElasticSearch storage implementation for low traffic system. Provide profile export tool. Support alarm gRPC hook. Fix PHP language doesn\u0026rsquo;t show up on the instance page. Add more comments in the source codes. Add a new metrics type, multiple linears. Fix thread concurrency issue in the alarm core. UI Support custom topology definition. Document Add FAQ about python2 command required in the compiling. Add doc about new e2e framework. Add doc about the new profile feature. Powered-by page updated. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"700\"\u003e7.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.5.0/en/changes/changes-7.0.0/","title":"7.0.0"},{"body":"7.0.0 Project SkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The minimal requirement of JDK is JDK8. Support method performance profile. Provide new E2E test framework. Remove AppVeyor from the CI, use GitHub action only. Provide new plugin test tool. Don\u0026rsquo;t support SkyWalking v5 agent in-wire and out-wire protocol. v6 is required. Java Agent Add lazy injection API in the agent core. Support Servlet 2.5 in the Struts plugin. Fix RestTemplate plugin ClassCastException in the Async call. Add Finagle plugin. Add test cases of H2 and struts. Add Armeria 0.98 plugin. Fix ElasticSearch plugin bug. Fix EHCache plugin bug. Fix a potential I/O leak. Support Oracle SID mode. Update Byte-buddy core. Performance tuning: replace AtomicInteger with AtomicIntegerFieldUpdater. Add AVRO plugin. Update to JDK 1.8 Optimize the ignore plugin. Enhance the gRPC plugin. Add Kotlin Coroutine plugin. Support HTTP parameter collection in Tomcat and SpringMVC plugin. Add @Tag annotation in the application toolkit. Move Lettuce into the default plugin list. Move Webflux into the default plugin list. Add HttpClient 3.x plugin. OAP-Backend Support InfluxDB as a new storage option. Add selector in the application.yml. Make the provider activation more flexible through System ENV. Support sub-topology map query. Support gRPC SSL. Support HTTP protocol for agent. Support Nginx LUA agent. Support skip the instance relationship analysis if some agents doesn\u0026rsquo;t have upstream address, currently for LUA agent. Support metrics entity name in the storage. Optional, default OFF. Merge the HOUR and DAY metrics into MINUTE in the ElasticSearch storage implementation. Reduce the payload for ElasticSearch server. Support change detection mechanism in DCS. Support Daily step in the ElasticSearch storage implementation for low traffic system. Provide profile export tool. Support alarm gRPC hook. Fix PHP language doesn\u0026rsquo;t show up on the instance page. Add more comments in the source codes. Add a new metrics type, multiple linears. Fix thread concurrency issue in the alarm core. UI Support custom topology definition. Document Add FAQ about python2 command required in the compiling. Add doc about new e2e framework. Add doc about the new profile feature. Powered-by page updated. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"700\"\u003e7.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.6.0/en/changes/changes-7.0.0/","title":"7.0.0"},{"body":"7.0.0 Project SkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The minimal requirement of JDK is JDK8. Support method performance profile. Provide new E2E test framework. Remove AppVeyor from the CI, use GitHub action only. Provide new plugin test tool. Don\u0026rsquo;t support SkyWalking v5 agent in-wire and out-wire protocol. v6 is required. Java Agent Add lazy injection API in the agent core. Support Servlet 2.5 in the Struts plugin. Fix RestTemplate plugin ClassCastException in the Async call. Add Finagle plugin. Add test cases of H2 and struts. Add Armeria 0.98 plugin. Fix ElasticSearch plugin bug. Fix EHCache plugin bug. Fix a potential I/O leak. Support Oracle SID mode. Update Byte-buddy core. Performance tuning: replace AtomicInteger with AtomicIntegerFieldUpdater. Add AVRO plugin. Update to JDK 1.8 Optimize the ignore plugin. Enhance the gRPC plugin. Add Kotlin Coroutine plugin. Support HTTP parameter collection in Tomcat and SpringMVC plugin. Add @Tag annotation in the application toolkit. Move Lettuce into the default plugin list. Move Webflux into the default plugin list. Add HttpClient 3.x plugin. OAP-Backend Support InfluxDB as a new storage option. Add selector in the application.yml. Make the provider activation more flexible through System ENV. Support sub-topology map query. Support gRPC SSL. Support HTTP protocol for agent. Support Nginx LUA agent. Support skip the instance relationship analysis if some agents doesn\u0026rsquo;t have upstream address, currently for LUA agent. Support metrics entity name in the storage. Optional, default OFF. Merge the HOUR and DAY metrics into MINUTE in the ElasticSearch storage implementation. Reduce the payload for ElasticSearch server. Support change detection mechanism in DCS. Support Daily step in the ElasticSearch storage implementation for low traffic system. Provide profile export tool. Support alarm gRPC hook. Fix PHP language doesn\u0026rsquo;t show up on the instance page. Add more comments in the source codes. Add a new metrics type, multiple linears. Fix thread concurrency issue in the alarm core. UI Support custom topology definition. Document Add FAQ about python2 command required in the compiling. Add doc about new e2e framework. Add doc about the new profile feature. Powered-by page updated. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"700\"\u003e7.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSkyWalking discards the supports of JDK 1.6 and 1.7 on the java agent side. The …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.7.0/en/changes/changes-7.0.0/","title":"7.0.0"},{"body":"8.0.0 Project v3 protocol is added and implemented. All previous releases are incompatible with 8.x releases. Service, Instance, Endpoint register mechanism and inventory storage entities are removed. New GraphQL query protocol is provided, the legacy protocol is still supported(plan to remove at the end of this year). Support Prometheus network protocol. Metrics in Prometheus format could be transferred into SkyWalking. Python agent provided. All inventory caches have been removed. Apache ShardingSphere(4.1.0, 4.1.1) agent plugin provided. Java Agent Add MariaDB plugin. Vert.x plugin enhancement. More cases are covered. Support v3 extension header. Fix ElasticSearch 5.x plugin TransportClient error. Support Correlation protocol v1. Fix Finagle plugin bug, in processing Noop Span. Make CommandService daemon to avoid blocking target application shutting down gracefully. Refactor spring cloud gateway plugin and support tracing spring cloud gateway 2.2.x OAP-Backend Support meter system for Prometheus adoption. In future releases, we will add native meter APIs and MicroMeter(Sleuth) system. Support endpoint grouping. Add SuperDataSet annotation for storage entity. Add superDatasetIndexShardsFactor in the ElasticSearch storage, to provide more shards for @SuperDataSet annotated entites. Typically TraceSegment. Support alarm settings for relationship of service, instance, and endpoint level metrics. Support alarm settings for database(conjecture node in tracing scenario). Data Model could be added in the runtime, don\u0026rsquo;t depend on the bootstrap sequence anymore. Reduce the memory cost, due to no inventory caches. No buffer files in tracing and service mesh cases. New ReadWriteSafe cache implementation. Simplify codes. Provide default way for metrics query, even the metrics doesn\u0026rsquo;t exist. New GraphQL query protocol is provided. Support the metrics type query. Set up length rule of service, instance, and endpoint. Adjust the default jks for ElasticSearch to empty. Fix Apdex function integer overflow issue. Fix profile storage issue. Fix TTL issue. Fix H2 column type bug. Add JRE 8-14 test for the backend. UI UI dashboard is 100% configurable to adopt new metrics definited in the backend. Document Add v8 upgrade document. Make the coverage accurate including UT and e2e tests. Add miss doc about collecting parameters in the profiled traces. CVE Fix SQL Injection vulnerability in H2/MySQL implementation. Upgrade Nacos to avoid the FastJson CVE in high frequency. Upgrade jasckson-databind to 2.9.10. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"800\"\u003e8.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003ev3 protocol is added and implemented. All previous releases are incompatible with 8.x …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/latest/en/changes/changes-8.0.0/","title":"8.0.0"},{"body":"8.0.0 Project v3 protocol is added and implemented. All previous releases are incompatible with 8.x releases. Service, Instance, Endpoint register mechanism and inventory storage entities are removed. New GraphQL query protocol is provided, the legacy protocol is still supported(plan to remove at the end of this year). Support Prometheus network protocol. Metrics in Prometheus format could be transferred into SkyWalking. Python agent provided. All inventory caches have been removed. Apache ShardingSphere(4.1.0, 4.1.1) agent plugin provided. Java Agent Add MariaDB plugin. Vert.x plugin enhancement. More cases are covered. Support v3 extension header. Fix ElasticSearch 5.x plugin TransportClient error. Support Correlation protocol v1. Fix Finagle plugin bug, in processing Noop Span. Make CommandService daemon to avoid blocking target application shutting down gracefully. Refactor spring cloud gateway plugin and support tracing spring cloud gateway 2.2.x OAP-Backend Support meter system for Prometheus adoption. In future releases, we will add native meter APIs and MicroMeter(Sleuth) system. Support endpoint grouping. Add SuperDataSet annotation for storage entity. Add superDatasetIndexShardsFactor in the ElasticSearch storage, to provide more shards for @SuperDataSet annotated entites. Typically TraceSegment. Support alarm settings for relationship of service, instance, and endpoint level metrics. Support alarm settings for database(conjecture node in tracing scenario). Data Model could be added in the runtime, don\u0026rsquo;t depend on the bootstrap sequence anymore. Reduce the memory cost, due to no inventory caches. No buffer files in tracing and service mesh cases. New ReadWriteSafe cache implementation. Simplify codes. Provide default way for metrics query, even the metrics doesn\u0026rsquo;t exist. New GraphQL query protocol is provided. Support the metrics type query. Set up length rule of service, instance, and endpoint. Adjust the default jks for ElasticSearch to empty. Fix Apdex function integer overflow issue. Fix profile storage issue. Fix TTL issue. Fix H2 column type bug. Add JRE 8-14 test for the backend. UI UI dashboard is 100% configurable to adopt new metrics definited in the backend. Document Add v8 upgrade document. Make the coverage accurate including UT and e2e tests. Add miss doc about collecting parameters in the profiled traces. CVE Fix SQL Injection vulnerability in H2/MySQL implementation. Upgrade Nacos to avoid the FastJson CVE in high frequency. Upgrade jasckson-databind to 2.9.10. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"800\"\u003e8.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003ev3 protocol is added and implemented. All previous releases are incompatible with 8.x …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/next/en/changes/changes-8.0.0/","title":"8.0.0"},{"body":"8.0.0 Project v3 protocol is added and implemented. All previous releases are incompatible with 8.x releases. Service, Instance, Endpoint register mechanism and inventory storage entities are removed. New GraphQL query protocol is provided, the legacy protocol is still supported(plan to remove at the end of this year). Support Prometheus network protocol. Metrics in Prometheus format could be transferred into SkyWalking. Python agent provided. All inventory caches have been removed. Apache ShardingSphere(4.1.0, 4.1.1) agent plugin provided. Java Agent Add MariaDB plugin. Vert.x plugin enhancement. More cases are covered. Support v3 extension header. Fix ElasticSearch 5.x plugin TransportClient error. Support Correlation protocol v1. Fix Finagle plugin bug, in processing Noop Span. Make CommandService daemon to avoid blocking target application shutting down gracefully. Refactor spring cloud gateway plugin and support tracing spring cloud gateway 2.2.x OAP-Backend Support meter system for Prometheus adoption. In future releases, we will add native meter APIs and MicroMeter(Sleuth) system. Support endpoint grouping. Add SuperDataSet annotation for storage entity. Add superDatasetIndexShardsFactor in the ElasticSearch storage, to provide more shards for @SuperDataSet annotated entites. Typically TraceSegment. Support alarm settings for relationship of service, instance, and endpoint level metrics. Support alarm settings for database(conjecture node in tracing scenario). Data Model could be added in the runtime, don\u0026rsquo;t depend on the bootstrap sequence anymore. Reduce the memory cost, due to no inventory caches. No buffer files in tracing and service mesh cases. New ReadWriteSafe cache implementation. Simplify codes. Provide default way for metrics query, even the metrics doesn\u0026rsquo;t exist. New GraphQL query protocol is provided. Support the metrics type query. Set up length rule of service, instance, and endpoint. Adjust the default jks for ElasticSearch to empty. Fix Apdex function integer overflow issue. Fix profile storage issue. Fix TTL issue. Fix H2 column type bug. Add JRE 8-14 test for the backend. UI UI dashboard is 100% configurable to adopt new metrics definited in the backend. Document Add v8 upgrade document. Make the coverage accurate including UT and e2e tests. Add miss doc about collecting parameters in the profiled traces. CVE Fix SQL Injection vulnerability in H2/MySQL implementation. Upgrade Nacos to avoid the FastJson CVE in high frequency. Upgrade jasckson-databind to 2.9.10. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"800\"\u003e8.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003ev3 protocol is added and implemented. All previous releases are incompatible with 8.x …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.0.0/en/changes/changes-8.0.0/","title":"8.0.0"},{"body":"8.0.0 Project v3 protocol is added and implemented. All previous releases are incompatible with 8.x releases. Service, Instance, Endpoint register mechanism and inventory storage entities are removed. New GraphQL query protocol is provided, the legacy protocol is still supported(plan to remove at the end of this year). Support Prometheus network protocol. Metrics in Prometheus format could be transferred into SkyWalking. Python agent provided. All inventory caches have been removed. Apache ShardingSphere(4.1.0, 4.1.1) agent plugin provided. Java Agent Add MariaDB plugin. Vert.x plugin enhancement. More cases are covered. Support v3 extension header. Fix ElasticSearch 5.x plugin TransportClient error. Support Correlation protocol v1. Fix Finagle plugin bug, in processing Noop Span. Make CommandService daemon to avoid blocking target application shutting down gracefully. Refactor spring cloud gateway plugin and support tracing spring cloud gateway 2.2.x OAP-Backend Support meter system for Prometheus adoption. In future releases, we will add native meter APIs and MicroMeter(Sleuth) system. Support endpoint grouping. Add SuperDataSet annotation for storage entity. Add superDatasetIndexShardsFactor in the ElasticSearch storage, to provide more shards for @SuperDataSet annotated entites. Typically TraceSegment. Support alarm settings for relationship of service, instance, and endpoint level metrics. Support alarm settings for database(conjecture node in tracing scenario). Data Model could be added in the runtime, don\u0026rsquo;t depend on the bootstrap sequence anymore. Reduce the memory cost, due to no inventory caches. No buffer files in tracing and service mesh cases. New ReadWriteSafe cache implementation. Simplify codes. Provide default way for metrics query, even the metrics doesn\u0026rsquo;t exist. New GraphQL query protocol is provided. Support the metrics type query. Set up length rule of service, instance, and endpoint. Adjust the default jks for ElasticSearch to empty. Fix Apdex function integer overflow issue. Fix profile storage issue. Fix TTL issue. Fix H2 column type bug. Add JRE 8-14 test for the backend. UI UI dashboard is 100% configurable to adopt new metrics definited in the backend. Document Add v8 upgrade document. Make the coverage accurate including UT and e2e tests. Add miss doc about collecting parameters in the profiled traces. CVE Fix SQL Injection vulnerability in H2/MySQL implementation. Upgrade Nacos to avoid the FastJson CVE in high frequency. Upgrade jasckson-databind to 2.9.10. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"800\"\u003e8.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003ev3 protocol is added and implemented. All previous releases are incompatible with 8.x …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.0.1/en/changes/changes-8.0.0/","title":"8.0.0"},{"body":"8.0.0 Project v3 protocol is added and implemented. All previous releases are incompatible with 8.x releases. Service, Instance, Endpoint register mechanism and inventory storage entities are removed. New GraphQL query protocol is provided, the legacy protocol is still supported(plan to remove at the end of this year). Support Prometheus network protocol. Metrics in Prometheus format could be transferred into SkyWalking. Python agent provided. All inventory caches have been removed. Apache ShardingSphere(4.1.0, 4.1.1) agent plugin provided. Java Agent Add MariaDB plugin. Vert.x plugin enhancement. More cases are covered. Support v3 extension header. Fix ElasticSearch 5.x plugin TransportClient error. Support Correlation protocol v1. Fix Finagle plugin bug, in processing Noop Span. Make CommandService daemon to avoid blocking target application shutting down gracefully. Refactor spring cloud gateway plugin and support tracing spring cloud gateway 2.2.x OAP-Backend Support meter system for Prometheus adoption. In future releases, we will add native meter APIs and MicroMeter(Sleuth) system. Support endpoint grouping. Add SuperDataSet annotation for storage entity. Add superDatasetIndexShardsFactor in the ElasticSearch storage, to provide more shards for @SuperDataSet annotated entites. Typically TraceSegment. Support alarm settings for relationship of service, instance, and endpoint level metrics. Support alarm settings for database(conjecture node in tracing scenario). Data Model could be added in the runtime, don\u0026rsquo;t depend on the bootstrap sequence anymore. Reduce the memory cost, due to no inventory caches. No buffer files in tracing and service mesh cases. New ReadWriteSafe cache implementation. Simplify codes. Provide default way for metrics query, even the metrics doesn\u0026rsquo;t exist. New GraphQL query protocol is provided. Support the metrics type query. Set up length rule of service, instance, and endpoint. Adjust the default jks for ElasticSearch to empty. Fix Apdex function integer overflow issue. Fix profile storage issue. Fix TTL issue. Fix H2 column type bug. Add JRE 8-14 test for the backend. UI UI dashboard is 100% configurable to adopt new metrics definited in the backend. Document Add v8 upgrade document. Make the coverage accurate including UT and e2e tests. Add miss doc about collecting parameters in the profiled traces. CVE Fix SQL Injection vulnerability in H2/MySQL implementation. Upgrade Nacos to avoid the FastJson CVE in high frequency. Upgrade jasckson-databind to 2.9.10. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"800\"\u003e8.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003ev3 protocol is added and implemented. All previous releases are incompatible with 8.x …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.1.0/en/changes/changes-8.0.0/","title":"8.0.0"},{"body":"8.0.0 Project v3 protocol is added and implemented. All previous releases are incompatible with 8.x releases. Service, Instance, Endpoint register mechanism and inventory storage entities are removed. New GraphQL query protocol is provided, the legacy protocol is still supported(plan to remove at the end of this year). Support Prometheus network protocol. Metrics in Prometheus format could be transferred into SkyWalking. Python agent provided. All inventory caches have been removed. Apache ShardingSphere(4.1.0, 4.1.1) agent plugin provided. Java Agent Add MariaDB plugin. Vert.x plugin enhancement. More cases are covered. Support v3 extension header. Fix ElasticSearch 5.x plugin TransportClient error. Support Correlation protocol v1. Fix Finagle plugin bug, in processing Noop Span. Make CommandService daemon to avoid blocking target application shutting down gracefully. Refactor spring cloud gateway plugin and support tracing spring cloud gateway 2.2.x OAP-Backend Support meter system for Prometheus adoption. In future releases, we will add native meter APIs and MicroMeter(Sleuth) system. Support endpoint grouping. Add SuperDataSet annotation for storage entity. Add superDatasetIndexShardsFactor in the ElasticSearch storage, to provide more shards for @SuperDataSet annotated entites. Typically TraceSegment. Support alarm settings for relationship of service, instance, and endpoint level metrics. Support alarm settings for database(conjecture node in tracing scenario). Data Model could be added in the runtime, don\u0026rsquo;t depend on the bootstrap sequence anymore. Reduce the memory cost, due to no inventory caches. No buffer files in tracing and service mesh cases. New ReadWriteSafe cache implementation. Simplify codes. Provide default way for metrics query, even the metrics doesn\u0026rsquo;t exist. New GraphQL query protocol is provided. Support the metrics type query. Set up length rule of service, instance, and endpoint. Adjust the default jks for ElasticSearch to empty. Fix Apdex function integer overflow issue. Fix profile storage issue. Fix TTL issue. Fix H2 column type bug. Add JRE 8-14 test for the backend. UI UI dashboard is 100% configurable to adopt new metrics definited in the backend. Document Add v8 upgrade document. Make the coverage accurate including UT and e2e tests. Add miss doc about collecting parameters in the profiled traces. CVE Fix SQL Injection vulnerability in H2/MySQL implementation. Upgrade Nacos to avoid the FastJson CVE in high frequency. Upgrade jasckson-databind to 2.9.10. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"800\"\u003e8.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003ev3 protocol is added and implemented. All previous releases are incompatible with 8.x …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.2.0/en/changes/changes-8.0.0/","title":"8.0.0"},{"body":"8.0.0 Project v3 protocol is added and implemented. All previous releases are incompatible with 8.x releases. Service, Instance, Endpoint register mechanism and inventory storage entities are removed. New GraphQL query protocol is provided, the legacy protocol is still supported(plan to remove at the end of this year). Support Prometheus network protocol. Metrics in Prometheus format could be transferred into SkyWalking. Python agent provided. All inventory caches have been removed. Apache ShardingSphere(4.1.0, 4.1.1) agent plugin provided. Java Agent Add MariaDB plugin. Vert.x plugin enhancement. More cases are covered. Support v3 extension header. Fix ElasticSearch 5.x plugin TransportClient error. Support Correlation protocol v1. Fix Finagle plugin bug, in processing Noop Span. Make CommandService daemon to avoid blocking target application shutting down gracefully. Refactor spring cloud gateway plugin and support tracing spring cloud gateway 2.2.x OAP-Backend Support meter system for Prometheus adoption. In future releases, we will add native meter APIs and MicroMeter(Sleuth) system. Support endpoint grouping. Add SuperDataSet annotation for storage entity. Add superDatasetIndexShardsFactor in the ElasticSearch storage, to provide more shards for @SuperDataSet annotated entites. Typically TraceSegment. Support alarm settings for relationship of service, instance, and endpoint level metrics. Support alarm settings for database(conjecture node in tracing scenario). Data Model could be added in the runtime, don\u0026rsquo;t depend on the bootstrap sequence anymore. Reduce the memory cost, due to no inventory caches. No buffer files in tracing and service mesh cases. New ReadWriteSafe cache implementation. Simplify codes. Provide default way for metrics query, even the metrics doesn\u0026rsquo;t exist. New GraphQL query protocol is provided. Support the metrics type query. Set up length rule of service, instance, and endpoint. Adjust the default jks for ElasticSearch to empty. Fix Apdex function integer overflow issue. Fix profile storage issue. Fix TTL issue. Fix H2 column type bug. Add JRE 8-14 test for the backend. UI UI dashboard is 100% configurable to adopt new metrics definited in the backend. Document Add v8 upgrade document. Make the coverage accurate including UT and e2e tests. Add miss doc about collecting parameters in the profiled traces. CVE Fix SQL Injection vulnerability in H2/MySQL implementation. Upgrade Nacos to avoid the FastJson CVE in high frequency. Upgrade jasckson-databind to 2.9.10. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"800\"\u003e8.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003ev3 protocol is added and implemented. All previous releases are incompatible with 8.x …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.3.0/en/changes/changes-8.0.0/","title":"8.0.0"},{"body":"8.0.0 Project v3 protocol is added and implemented. All previous releases are incompatible with 8.x releases. Service, Instance, Endpoint register mechanism and inventory storage entities are removed. New GraphQL query protocol is provided, the legacy protocol is still supported(plan to remove at the end of this year). Support Prometheus network protocol. Metrics in Prometheus format could be transferred into SkyWalking. Python agent provided. All inventory caches have been removed. Apache ShardingSphere(4.1.0, 4.1.1) agent plugin provided. Java Agent Add MariaDB plugin. Vert.x plugin enhancement. More cases are covered. Support v3 extension header. Fix ElasticSearch 5.x plugin TransportClient error. Support Correlation protocol v1. Fix Finagle plugin bug, in processing Noop Span. Make CommandService daemon to avoid blocking target application shutting down gracefully. Refactor spring cloud gateway plugin and support tracing spring cloud gateway 2.2.x OAP-Backend Support meter system for Prometheus adoption. In future releases, we will add native meter APIs and MicroMeter(Sleuth) system. Support endpoint grouping. Add SuperDataSet annotation for storage entity. Add superDatasetIndexShardsFactor in the ElasticSearch storage, to provide more shards for @SuperDataSet annotated entites. Typically TraceSegment. Support alarm settings for relationship of service, instance, and endpoint level metrics. Support alarm settings for database(conjecture node in tracing scenario). Data Model could be added in the runtime, don\u0026rsquo;t depend on the bootstrap sequence anymore. Reduce the memory cost, due to no inventory caches. No buffer files in tracing and service mesh cases. New ReadWriteSafe cache implementation. Simplify codes. Provide default way for metrics query, even the metrics doesn\u0026rsquo;t exist. New GraphQL query protocol is provided. Support the metrics type query. Set up length rule of service, instance, and endpoint. Adjust the default jks for ElasticSearch to empty. Fix Apdex function integer overflow issue. Fix profile storage issue. Fix TTL issue. Fix H2 column type bug. Add JRE 8-14 test for the backend. UI UI dashboard is 100% configurable to adopt new metrics definited in the backend. Document Add v8 upgrade document. Make the coverage accurate including UT and e2e tests. Add miss doc about collecting parameters in the profiled traces. CVE Fix SQL Injection vulnerability in H2/MySQL implementation. Upgrade Nacos to avoid the FastJson CVE in high frequency. Upgrade jasckson-databind to 2.9.10. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"800\"\u003e8.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003ev3 protocol is added and implemented. All previous releases are incompatible with 8.x …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.4.0/en/changes/changes-8.0.0/","title":"8.0.0"},{"body":"8.0.0 Project v3 protocol is added and implemented. All previous releases are incompatible with 8.x releases. Service, Instance, Endpoint register mechanism and inventory storage entities are removed. New GraphQL query protocol is provided, the legacy protocol is still supported(plan to remove at the end of this year). Support Prometheus network protocol. Metrics in Prometheus format could be transferred into SkyWalking. Python agent provided. All inventory caches have been removed. Apache ShardingSphere(4.1.0, 4.1.1) agent plugin provided. Java Agent Add MariaDB plugin. Vert.x plugin enhancement. More cases are covered. Support v3 extension header. Fix ElasticSearch 5.x plugin TransportClient error. Support Correlation protocol v1. Fix Finagle plugin bug, in processing Noop Span. Make CommandService daemon to avoid blocking target application shutting down gracefully. Refactor spring cloud gateway plugin and support tracing spring cloud gateway 2.2.x OAP-Backend Support meter system for Prometheus adoption. In future releases, we will add native meter APIs and MicroMeter(Sleuth) system. Support endpoint grouping. Add SuperDataSet annotation for storage entity. Add superDatasetIndexShardsFactor in the ElasticSearch storage, to provide more shards for @SuperDataSet annotated entites. Typically TraceSegment. Support alarm settings for relationship of service, instance, and endpoint level metrics. Support alarm settings for database(conjecture node in tracing scenario). Data Model could be added in the runtime, don\u0026rsquo;t depend on the bootstrap sequence anymore. Reduce the memory cost, due to no inventory caches. No buffer files in tracing and service mesh cases. New ReadWriteSafe cache implementation. Simplify codes. Provide default way for metrics query, even the metrics doesn\u0026rsquo;t exist. New GraphQL query protocol is provided. Support the metrics type query. Set up length rule of service, instance, and endpoint. Adjust the default jks for ElasticSearch to empty. Fix Apdex function integer overflow issue. Fix profile storage issue. Fix TTL issue. Fix H2 column type bug. Add JRE 8-14 test for the backend. UI UI dashboard is 100% configurable to adopt new metrics definited in the backend. Document Add v8 upgrade document. Make the coverage accurate including UT and e2e tests. Add miss doc about collecting parameters in the profiled traces. CVE Fix SQL Injection vulnerability in H2/MySQL implementation. Upgrade Nacos to avoid the FastJson CVE in high frequency. Upgrade jasckson-databind to 2.9.10. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"800\"\u003e8.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003ev3 protocol is added and implemented. All previous releases are incompatible with 8.x …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v11.0.0/en/changes/changes-8.0.0/","title":"8.0.0"},{"body":"8.0.0 Project v3 protocol is added and implemented. All previous releases are incompatible with 8.x releases. Service, Instance, Endpoint register mechanism and inventory storage entities are removed. New GraphQL query protocol is provided, the legacy protocol is still supported(plan to remove at the end of this year). Support Prometheus network protocol. Metrics in Prometheus format could be transferred into SkyWalking. Python agent provided. All inventory caches have been removed. Apache ShardingSphere(4.1.0, 4.1.1) agent plugin provided. Java Agent Add MariaDB plugin. Vert.x plugin enhancement. More cases are covered. Support v3 extension header. Fix ElasticSearch 5.x plugin TransportClient error. Support Correlation protocol v1. Fix Finagle plugin bug, in processing Noop Span. Make CommandService daemon to avoid blocking target application shutting down gracefully. Refactor spring cloud gateway plugin and support tracing spring cloud gateway 2.2.x OAP-Backend Support meter system for Prometheus adoption. In future releases, we will add native meter APIs and MicroMeter(Sleuth) system. Support endpoint grouping. Add SuperDataSet annotation for storage entity. Add superDatasetIndexShardsFactor in the ElasticSearch storage, to provide more shards for @SuperDataSet annotated entites. Typically TraceSegment. Support alarm settings for relationship of service, instance, and endpoint level metrics. Support alarm settings for database(conjecture node in tracing scenario). Data Model could be added in the runtime, don\u0026rsquo;t depend on the bootstrap sequence anymore. Reduce the memory cost, due to no inventory caches. No buffer files in tracing and service mesh cases. New ReadWriteSafe cache implementation. Simplify codes. Provide default way for metrics query, even the metrics doesn\u0026rsquo;t exist. New GraphQL query protocol is provided. Support the metrics type query. Set up length rule of service, instance, and endpoint. Adjust the default jks for ElasticSearch to empty. Fix Apdex function integer overflow issue. Fix profile storage issue. Fix TTL issue. Fix H2 column type bug. Add JRE 8-14 test for the backend. UI UI dashboard is 100% configurable to adopt new metrics definited in the backend. Document Add v8 upgrade document. Make the coverage accurate including UT and e2e tests. Add miss doc about collecting parameters in the profiled traces. CVE Fix SQL Injection vulnerability in H2/MySQL implementation. Upgrade Nacos to avoid the FastJson CVE in high frequency. Upgrade jasckson-databind to 2.9.10. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"800\"\u003e8.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003ev3 protocol is added and implemented. All previous releases are incompatible with 8.x …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.4.0/en/changes/changes-8.0.0/","title":"8.0.0"},{"body":"8.0.0 Project v3 protocol is added and implemented. All previous releases are incompatible with 8.x releases. Service, Instance, Endpoint register mechanism and inventory storage entities are removed. New GraphQL query protocol is provided, the legacy protocol is still supported(plan to remove at the end of this year). Support Prometheus network protocol. Metrics in Prometheus format could be transferred into SkyWalking. Python agent provided. All inventory caches have been removed. Apache ShardingSphere(4.1.0, 4.1.1) agent plugin provided. Java Agent Add MariaDB plugin. Vert.x plugin enhancement. More cases are covered. Support v3 extension header. Fix ElasticSearch 5.x plugin TransportClient error. Support Correlation protocol v1. Fix Finagle plugin bug, in processing Noop Span. Make CommandService daemon to avoid blocking target application shutting down gracefully. Refactor spring cloud gateway plugin and support tracing spring cloud gateway 2.2.x OAP-Backend Support meter system for Prometheus adoption. In future releases, we will add native meter APIs and MicroMeter(Sleuth) system. Support endpoint grouping. Add SuperDataSet annotation for storage entity. Add superDatasetIndexShardsFactor in the ElasticSearch storage, to provide more shards for @SuperDataSet annotated entites. Typically TraceSegment. Support alarm settings for relationship of service, instance, and endpoint level metrics. Support alarm settings for database(conjecture node in tracing scenario). Data Model could be added in the runtime, don\u0026rsquo;t depend on the bootstrap sequence anymore. Reduce the memory cost, due to no inventory caches. No buffer files in tracing and service mesh cases. New ReadWriteSafe cache implementation. Simplify codes. Provide default way for metrics query, even the metrics doesn\u0026rsquo;t exist. New GraphQL query protocol is provided. Support the metrics type query. Set up length rule of service, instance, and endpoint. Adjust the default jks for ElasticSearch to empty. Fix Apdex function integer overflow issue. Fix profile storage issue. Fix TTL issue. Fix H2 column type bug. Add JRE 8-14 test for the backend. UI UI dashboard is 100% configurable to adopt new metrics definited in the backend. Document Add v8 upgrade document. Make the coverage accurate including UT and e2e tests. Add miss doc about collecting parameters in the profiled traces. CVE Fix SQL Injection vulnerability in H2/MySQL implementation. Upgrade Nacos to avoid the FastJson CVE in high frequency. Upgrade jasckson-databind to 2.9.10. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"800\"\u003e8.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003ev3 protocol is added and implemented. All previous releases are incompatible with 8.x …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.5.0/en/changes/changes-8.0.0/","title":"8.0.0"},{"body":"8.0.0 Project v3 protocol is added and implemented. All previous releases are incompatible with 8.x releases. Service, Instance, Endpoint register mechanism and inventory storage entities are removed. New GraphQL query protocol is provided, the legacy protocol is still supported(plan to remove at the end of this year). Support Prometheus network protocol. Metrics in Prometheus format could be transferred into SkyWalking. Python agent provided. All inventory caches have been removed. Apache ShardingSphere(4.1.0, 4.1.1) agent plugin provided. Java Agent Add MariaDB plugin. Vert.x plugin enhancement. More cases are covered. Support v3 extension header. Fix ElasticSearch 5.x plugin TransportClient error. Support Correlation protocol v1. Fix Finagle plugin bug, in processing Noop Span. Make CommandService daemon to avoid blocking target application shutting down gracefully. Refactor spring cloud gateway plugin and support tracing spring cloud gateway 2.2.x OAP-Backend Support meter system for Prometheus adoption. In future releases, we will add native meter APIs and MicroMeter(Sleuth) system. Support endpoint grouping. Add SuperDataSet annotation for storage entity. Add superDatasetIndexShardsFactor in the ElasticSearch storage, to provide more shards for @SuperDataSet annotated entites. Typically TraceSegment. Support alarm settings for relationship of service, instance, and endpoint level metrics. Support alarm settings for database(conjecture node in tracing scenario). Data Model could be added in the runtime, don\u0026rsquo;t depend on the bootstrap sequence anymore. Reduce the memory cost, due to no inventory caches. No buffer files in tracing and service mesh cases. New ReadWriteSafe cache implementation. Simplify codes. Provide default way for metrics query, even the metrics doesn\u0026rsquo;t exist. New GraphQL query protocol is provided. Support the metrics type query. Set up length rule of service, instance, and endpoint. Adjust the default jks for ElasticSearch to empty. Fix Apdex function integer overflow issue. Fix profile storage issue. Fix TTL issue. Fix H2 column type bug. Add JRE 8-14 test for the backend. UI UI dashboard is 100% configurable to adopt new metrics definited in the backend. Document Add v8 upgrade document. Make the coverage accurate including UT and e2e tests. Add miss doc about collecting parameters in the profiled traces. CVE Fix SQL Injection vulnerability in H2/MySQL implementation. Upgrade Nacos to avoid the FastJson CVE in high frequency. Upgrade jasckson-databind to 2.9.10. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"800\"\u003e8.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003ev3 protocol is added and implemented. All previous releases are incompatible with 8.x …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.6.0/en/changes/changes-8.0.0/","title":"8.0.0"},{"body":"8.0.0 Project v3 protocol is added and implemented. All previous releases are incompatible with 8.x releases. Service, Instance, Endpoint register mechanism and inventory storage entities are removed. New GraphQL query protocol is provided, the legacy protocol is still supported(plan to remove at the end of this year). Support Prometheus network protocol. Metrics in Prometheus format could be transferred into SkyWalking. Python agent provided. All inventory caches have been removed. Apache ShardingSphere(4.1.0, 4.1.1) agent plugin provided. Java Agent Add MariaDB plugin. Vert.x plugin enhancement. More cases are covered. Support v3 extension header. Fix ElasticSearch 5.x plugin TransportClient error. Support Correlation protocol v1. Fix Finagle plugin bug, in processing Noop Span. Make CommandService daemon to avoid blocking target application shutting down gracefully. Refactor spring cloud gateway plugin and support tracing spring cloud gateway 2.2.x OAP-Backend Support meter system for Prometheus adoption. In future releases, we will add native meter APIs and MicroMeter(Sleuth) system. Support endpoint grouping. Add SuperDataSet annotation for storage entity. Add superDatasetIndexShardsFactor in the ElasticSearch storage, to provide more shards for @SuperDataSet annotated entites. Typically TraceSegment. Support alarm settings for relationship of service, instance, and endpoint level metrics. Support alarm settings for database(conjecture node in tracing scenario). Data Model could be added in the runtime, don\u0026rsquo;t depend on the bootstrap sequence anymore. Reduce the memory cost, due to no inventory caches. No buffer files in tracing and service mesh cases. New ReadWriteSafe cache implementation. Simplify codes. Provide default way for metrics query, even the metrics doesn\u0026rsquo;t exist. New GraphQL query protocol is provided. Support the metrics type query. Set up length rule of service, instance, and endpoint. Adjust the default jks for ElasticSearch to empty. Fix Apdex function integer overflow issue. Fix profile storage issue. Fix TTL issue. Fix H2 column type bug. Add JRE 8-14 test for the backend. UI UI dashboard is 100% configurable to adopt new metrics definited in the backend. Document Add v8 upgrade document. Make the coverage accurate including UT and e2e tests. Add miss doc about collecting parameters in the profiled traces. CVE Fix SQL Injection vulnerability in H2/MySQL implementation. Upgrade Nacos to avoid the FastJson CVE in high frequency. Upgrade jasckson-databind to 2.9.10. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"800\"\u003e8.0.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003ev3 protocol is added and implemented. All previous releases are incompatible with 8.x …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.7.0/en/changes/changes-8.0.0/","title":"8.0.0"},{"body":"8.0.1 OAP-Backend Fix no-init mode is not working in ElasticSearch storage. ","excerpt":"\u003ch2 id=\"801\"\u003e8.0.1\u003c/h2\u003e\n\u003ch4 id=\"oap-backend\"\u003eOAP-Backend\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix \u003ccode\u003eno-init\u003c/code\u003e mode is not working in ElasticSearch storage.\u003c/li\u003e\n\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/latest/en/changes/changes-8.0.1/","title":"8.0.1"},{"body":"8.0.1 OAP-Backend Fix no-init mode is not working in ElasticSearch storage. ","excerpt":"\u003ch2 id=\"801\"\u003e8.0.1\u003c/h2\u003e\n\u003ch4 id=\"oap-backend\"\u003eOAP-Backend\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix \u003ccode\u003eno-init\u003c/code\u003e mode is not working in ElasticSearch storage.\u003c/li\u003e\n\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/next/en/changes/changes-8.0.1/","title":"8.0.1"},{"body":"8.0.1 OAP-Backend Fix no-init mode is not working in ElasticSearch storage. ","excerpt":"\u003ch2 id=\"801\"\u003e8.0.1\u003c/h2\u003e\n\u003ch4 id=\"oap-backend\"\u003eOAP-Backend\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix \u003ccode\u003eno-init\u003c/code\u003e mode is not working in ElasticSearch storage.\u003c/li\u003e\n\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.0.0/en/changes/changes-8.0.1/","title":"8.0.1"},{"body":"8.0.1 OAP-Backend Fix no-init mode is not working in ElasticSearch storage. ","excerpt":"\u003ch2 id=\"801\"\u003e8.0.1\u003c/h2\u003e\n\u003ch4 id=\"oap-backend\"\u003eOAP-Backend\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix \u003ccode\u003eno-init\u003c/code\u003e mode is not working in ElasticSearch storage.\u003c/li\u003e\n\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.0.1/en/changes/changes-8.0.1/","title":"8.0.1"},{"body":"8.0.1 OAP-Backend Fix no-init mode is not working in ElasticSearch storage. ","excerpt":"\u003ch2 id=\"801\"\u003e8.0.1\u003c/h2\u003e\n\u003ch4 id=\"oap-backend\"\u003eOAP-Backend\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix \u003ccode\u003eno-init\u003c/code\u003e mode is not working in ElasticSearch storage.\u003c/li\u003e\n\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.1.0/en/changes/changes-8.0.1/","title":"8.0.1"},{"body":"8.0.1 OAP-Backend Fix no-init mode is not working in ElasticSearch storage. ","excerpt":"\u003ch2 id=\"801\"\u003e8.0.1\u003c/h2\u003e\n\u003ch4 id=\"oap-backend\"\u003eOAP-Backend\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix \u003ccode\u003eno-init\u003c/code\u003e mode is not working in ElasticSearch storage.\u003c/li\u003e\n\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.2.0/en/changes/changes-8.0.1/","title":"8.0.1"},{"body":"8.0.1 OAP-Backend Fix no-init mode is not working in ElasticSearch storage. ","excerpt":"\u003ch2 id=\"801\"\u003e8.0.1\u003c/h2\u003e\n\u003ch4 id=\"oap-backend\"\u003eOAP-Backend\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix \u003ccode\u003eno-init\u003c/code\u003e mode is not working in ElasticSearch storage.\u003c/li\u003e\n\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.3.0/en/changes/changes-8.0.1/","title":"8.0.1"},{"body":"8.0.1 OAP-Backend Fix no-init mode is not working in ElasticSearch storage. ","excerpt":"\u003ch2 id=\"801\"\u003e8.0.1\u003c/h2\u003e\n\u003ch4 id=\"oap-backend\"\u003eOAP-Backend\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix \u003ccode\u003eno-init\u003c/code\u003e mode is not working in ElasticSearch storage.\u003c/li\u003e\n\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.4.0/en/changes/changes-8.0.1/","title":"8.0.1"},{"body":"8.0.1 OAP-Backend Fix no-init mode is not working in ElasticSearch storage. ","excerpt":"\u003ch2 id=\"801\"\u003e8.0.1\u003c/h2\u003e\n\u003ch4 id=\"oap-backend\"\u003eOAP-Backend\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix \u003ccode\u003eno-init\u003c/code\u003e mode is not working in ElasticSearch storage.\u003c/li\u003e\n\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v11.0.0/en/changes/changes-8.0.1/","title":"8.0.1"},{"body":"8.0.1 OAP-Backend Fix no-init mode is not working in ElasticSearch storage. ","excerpt":"\u003ch2 id=\"801\"\u003e8.0.1\u003c/h2\u003e\n\u003ch4 id=\"oap-backend\"\u003eOAP-Backend\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix \u003ccode\u003eno-init\u003c/code\u003e mode is not working in ElasticSearch storage.\u003c/li\u003e\n\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.4.0/en/changes/changes-8.0.1/","title":"8.0.1"},{"body":"8.0.1 OAP-Backend Fix no-init mode is not working in ElasticSearch storage. ","excerpt":"\u003ch2 id=\"801\"\u003e8.0.1\u003c/h2\u003e\n\u003ch4 id=\"oap-backend\"\u003eOAP-Backend\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix \u003ccode\u003eno-init\u003c/code\u003e mode is not working in ElasticSearch storage.\u003c/li\u003e\n\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.5.0/en/changes/changes-8.0.1/","title":"8.0.1"},{"body":"8.0.1 OAP-Backend Fix no-init mode is not working in ElasticSearch storage. ","excerpt":"\u003ch2 id=\"801\"\u003e8.0.1\u003c/h2\u003e\n\u003ch4 id=\"oap-backend\"\u003eOAP-Backend\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix \u003ccode\u003eno-init\u003c/code\u003e mode is not working in ElasticSearch storage.\u003c/li\u003e\n\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.6.0/en/changes/changes-8.0.1/","title":"8.0.1"},{"body":"8.0.1 OAP-Backend Fix no-init mode is not working in ElasticSearch storage. ","excerpt":"\u003ch2 id=\"801\"\u003e8.0.1\u003c/h2\u003e\n\u003ch4 id=\"oap-backend\"\u003eOAP-Backend\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eFix \u003ccode\u003eno-init\u003c/code\u003e mode is not working in ElasticSearch storage.\u003c/li\u003e\n\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.7.0/en/changes/changes-8.0.1/","title":"8.0.1"},{"body":"8.1.0 Project Support Kafka as an optional trace, JVM metrics, profiling snapshots and meter system data transport layer. Support Meter system, including the native metrics APIs and the Spring Sleuth adoption. Support JVM thread metrics. Java Agent [Core] Fix the concurrency access bug in the Concurrency ClassLoader Case. [Core] Separate the config of the plugins from the core level. [Core] Support instrumented class cached in memory or file, to be compatible with other agents, such as Arthas. Add logic endpoint concept. Could analysis any span or tags flagged by the logic endpoint. Add Spring annotation component name for UI visualization only. Add support to trace Call procedures in MySQL plugin. Support GraphQL plugin. Support Quasar fiber plugin. Support InfluxDB java client plugin. Support brpc java plugin Support ConsoleAppender in the logback v1 plugin. Enhance vert.x endpoint names. Optimize the code to prevent mongo statements from being too long. Fix WebFlux plugin concurrency access bug. Fix ShardingSphere plugins internal conflicts. Fix duplicated Spring MVC endpoint. Fix lettuce plugin sometimes trace doesn‘t show span layer. Fix @Tag returnedObject bug. OAP-Backend Support Jetty Server advanced configurations. Support label based filter in the prometheus fetcher and OpenCensus receiver. Support using k8s configmap as the configuration center. Support OAP health check, and storage module health check. Support sampling rate in the dynamic configuration. Add endpoint_relation_sla and endpoint_relation_percentile for endpoint relationship metrics. Add components for Python plugins, including Kafka, Tornado, Redis, Django, PyMysql. Add components for Golang SDK. Add Nacos 1.3.1 back as an optional cluster coordinator and dynamic configuration center. Enhance the metrics query for ElasticSearch implementation to increase the stability. Reduce the length of storage entity names in the self-observability for MySQL and TiDB storage. Fix labels are missing in Prometheus analysis context. Fix column length issue in MySQL/TiDB storage. Fix no data in 2nd level aggregation in self-observability. Fix searchService bug in ES implementation. Fix wrong validation of endpoint relation entity query. Fix the bug caused by the OAL debug flag. Fix endpoint dependency bug in MQ and uninstrumented proxy cases. Fix time bucket conversion issue in the InfluxDB storage implementation. Update k8s client to 8.0.0 UI Support endpoint dependency graph. Support x-scroll of trace/profile page Fix database selector issue. Add the bar chart in the UI templates. Document Update the user logo wall. Add backend configuration vocabulary document. Add agent installation doc for Tomcat9 on Windows. Add istioctl ALS commands for the document. Fix TTL documentation. Add FAQ doc about thread instrumentation. CVE Fix fuzzy query sql injection in the MySQL/TiDB storage. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"810\"\u003e8.1.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Kafka as an optional trace, JVM metrics, profiling snapshots and meter system …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/latest/en/changes/changes-8.1.0/","title":"8.1.0"},{"body":"8.1.0 Project Support Kafka as an optional trace, JVM metrics, profiling snapshots and meter system data transport layer. Support Meter system, including the native metrics APIs and the Spring Sleuth adoption. Support JVM thread metrics. Java Agent [Core] Fix the concurrency access bug in the Concurrency ClassLoader Case. [Core] Separate the config of the plugins from the core level. [Core] Support instrumented class cached in memory or file, to be compatible with other agents, such as Arthas. Add logic endpoint concept. Could analysis any span or tags flagged by the logic endpoint. Add Spring annotation component name for UI visualization only. Add support to trace Call procedures in MySQL plugin. Support GraphQL plugin. Support Quasar fiber plugin. Support InfluxDB java client plugin. Support brpc java plugin Support ConsoleAppender in the logback v1 plugin. Enhance vert.x endpoint names. Optimize the code to prevent mongo statements from being too long. Fix WebFlux plugin concurrency access bug. Fix ShardingSphere plugins internal conflicts. Fix duplicated Spring MVC endpoint. Fix lettuce plugin sometimes trace doesn‘t show span layer. Fix @Tag returnedObject bug. OAP-Backend Support Jetty Server advanced configurations. Support label based filter in the prometheus fetcher and OpenCensus receiver. Support using k8s configmap as the configuration center. Support OAP health check, and storage module health check. Support sampling rate in the dynamic configuration. Add endpoint_relation_sla and endpoint_relation_percentile for endpoint relationship metrics. Add components for Python plugins, including Kafka, Tornado, Redis, Django, PyMysql. Add components for Golang SDK. Add Nacos 1.3.1 back as an optional cluster coordinator and dynamic configuration center. Enhance the metrics query for ElasticSearch implementation to increase the stability. Reduce the length of storage entity names in the self-observability for MySQL and TiDB storage. Fix labels are missing in Prometheus analysis context. Fix column length issue in MySQL/TiDB storage. Fix no data in 2nd level aggregation in self-observability. Fix searchService bug in ES implementation. Fix wrong validation of endpoint relation entity query. Fix the bug caused by the OAL debug flag. Fix endpoint dependency bug in MQ and uninstrumented proxy cases. Fix time bucket conversion issue in the InfluxDB storage implementation. Update k8s client to 8.0.0 UI Support endpoint dependency graph. Support x-scroll of trace/profile page Fix database selector issue. Add the bar chart in the UI templates. Document Update the user logo wall. Add backend configuration vocabulary document. Add agent installation doc for Tomcat9 on Windows. Add istioctl ALS commands for the document. Fix TTL documentation. Add FAQ doc about thread instrumentation. CVE Fix fuzzy query sql injection in the MySQL/TiDB storage. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"810\"\u003e8.1.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Kafka as an optional trace, JVM metrics, profiling snapshots and meter system …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/next/en/changes/changes-8.1.0/","title":"8.1.0"},{"body":"8.1.0 Project Support Kafka as an optional trace, JVM metrics, profiling snapshots and meter system data transport layer. Support Meter system, including the native metrics APIs and the Spring Sleuth adoption. Support JVM thread metrics. Java Agent [Core] Fix the concurrency access bug in the Concurrency ClassLoader Case. [Core] Separate the config of the plugins from the core level. [Core] Support instrumented class cached in memory or file, to be compatible with other agents, such as Arthas. Add logic endpoint concept. Could analysis any span or tags flagged by the logic endpoint. Add Spring annotation component name for UI visualization only. Add support to trace Call procedures in MySQL plugin. Support GraphQL plugin. Support Quasar fiber plugin. Support InfluxDB java client plugin. Support brpc java plugin Support ConsoleAppender in the logback v1 plugin. Enhance vert.x endpoint names. Optimize the code to prevent mongo statements from being too long. Fix WebFlux plugin concurrency access bug. Fix ShardingSphere plugins internal conflicts. Fix duplicated Spring MVC endpoint. Fix lettuce plugin sometimes trace doesn‘t show span layer. Fix @Tag returnedObject bug. OAP-Backend Support Jetty Server advanced configurations. Support label based filter in the prometheus fetcher and OpenCensus receiver. Support using k8s configmap as the configuration center. Support OAP health check, and storage module health check. Support sampling rate in the dynamic configuration. Add endpoint_relation_sla and endpoint_relation_percentile for endpoint relationship metrics. Add components for Python plugins, including Kafka, Tornado, Redis, Django, PyMysql. Add components for Golang SDK. Add Nacos 1.3.1 back as an optional cluster coordinator and dynamic configuration center. Enhance the metrics query for ElasticSearch implementation to increase the stability. Reduce the length of storage entity names in the self-observability for MySQL and TiDB storage. Fix labels are missing in Prometheus analysis context. Fix column length issue in MySQL/TiDB storage. Fix no data in 2nd level aggregation in self-observability. Fix searchService bug in ES implementation. Fix wrong validation of endpoint relation entity query. Fix the bug caused by the OAL debug flag. Fix endpoint dependency bug in MQ and uninstrumented proxy cases. Fix time bucket conversion issue in the InfluxDB storage implementation. Update k8s client to 8.0.0 UI Support endpoint dependency graph. Support x-scroll of trace/profile page Fix database selector issue. Add the bar chart in the UI templates. Document Update the user logo wall. Add backend configuration vocabulary document. Add agent installation doc for Tomcat9 on Windows. Add istioctl ALS commands for the document. Fix TTL documentation. Add FAQ doc about thread instrumentation. CVE Fix fuzzy query sql injection in the MySQL/TiDB storage. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"810\"\u003e8.1.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Kafka as an optional trace, JVM metrics, profiling snapshots and meter system …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.0.0/en/changes/changes-8.1.0/","title":"8.1.0"},{"body":"8.1.0 Project Support Kafka as an optional trace, JVM metrics, profiling snapshots and meter system data transport layer. Support Meter system, including the native metrics APIs and the Spring Sleuth adoption. Support JVM thread metrics. Java Agent [Core] Fix the concurrency access bug in the Concurrency ClassLoader Case. [Core] Separate the config of the plugins from the core level. [Core] Support instrumented class cached in memory or file, to be compatible with other agents, such as Arthas. Add logic endpoint concept. Could analysis any span or tags flagged by the logic endpoint. Add Spring annotation component name for UI visualization only. Add support to trace Call procedures in MySQL plugin. Support GraphQL plugin. Support Quasar fiber plugin. Support InfluxDB java client plugin. Support brpc java plugin Support ConsoleAppender in the logback v1 plugin. Enhance vert.x endpoint names. Optimize the code to prevent mongo statements from being too long. Fix WebFlux plugin concurrency access bug. Fix ShardingSphere plugins internal conflicts. Fix duplicated Spring MVC endpoint. Fix lettuce plugin sometimes trace doesn‘t show span layer. Fix @Tag returnedObject bug. OAP-Backend Support Jetty Server advanced configurations. Support label based filter in the prometheus fetcher and OpenCensus receiver. Support using k8s configmap as the configuration center. Support OAP health check, and storage module health check. Support sampling rate in the dynamic configuration. Add endpoint_relation_sla and endpoint_relation_percentile for endpoint relationship metrics. Add components for Python plugins, including Kafka, Tornado, Redis, Django, PyMysql. Add components for Golang SDK. Add Nacos 1.3.1 back as an optional cluster coordinator and dynamic configuration center. Enhance the metrics query for ElasticSearch implementation to increase the stability. Reduce the length of storage entity names in the self-observability for MySQL and TiDB storage. Fix labels are missing in Prometheus analysis context. Fix column length issue in MySQL/TiDB storage. Fix no data in 2nd level aggregation in self-observability. Fix searchService bug in ES implementation. Fix wrong validation of endpoint relation entity query. Fix the bug caused by the OAL debug flag. Fix endpoint dependency bug in MQ and uninstrumented proxy cases. Fix time bucket conversion issue in the InfluxDB storage implementation. Update k8s client to 8.0.0 UI Support endpoint dependency graph. Support x-scroll of trace/profile page Fix database selector issue. Add the bar chart in the UI templates. Document Update the user logo wall. Add backend configuration vocabulary document. Add agent installation doc for Tomcat9 on Windows. Add istioctl ALS commands for the document. Fix TTL documentation. Add FAQ doc about thread instrumentation. CVE Fix fuzzy query sql injection in the MySQL/TiDB storage. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"810\"\u003e8.1.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Kafka as an optional trace, JVM metrics, profiling snapshots and meter system …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.0.1/en/changes/changes-8.1.0/","title":"8.1.0"},{"body":"8.1.0 Project Support Kafka as an optional trace, JVM metrics, profiling snapshots and meter system data transport layer. Support Meter system, including the native metrics APIs and the Spring Sleuth adoption. Support JVM thread metrics. Java Agent [Core] Fix the concurrency access bug in the Concurrency ClassLoader Case. [Core] Separate the config of the plugins from the core level. [Core] Support instrumented class cached in memory or file, to be compatible with other agents, such as Arthas. Add logic endpoint concept. Could analysis any span or tags flagged by the logic endpoint. Add Spring annotation component name for UI visualization only. Add support to trace Call procedures in MySQL plugin. Support GraphQL plugin. Support Quasar fiber plugin. Support InfluxDB java client plugin. Support brpc java plugin Support ConsoleAppender in the logback v1 plugin. Enhance vert.x endpoint names. Optimize the code to prevent mongo statements from being too long. Fix WebFlux plugin concurrency access bug. Fix ShardingSphere plugins internal conflicts. Fix duplicated Spring MVC endpoint. Fix lettuce plugin sometimes trace doesn‘t show span layer. Fix @Tag returnedObject bug. OAP-Backend Support Jetty Server advanced configurations. Support label based filter in the prometheus fetcher and OpenCensus receiver. Support using k8s configmap as the configuration center. Support OAP health check, and storage module health check. Support sampling rate in the dynamic configuration. Add endpoint_relation_sla and endpoint_relation_percentile for endpoint relationship metrics. Add components for Python plugins, including Kafka, Tornado, Redis, Django, PyMysql. Add components for Golang SDK. Add Nacos 1.3.1 back as an optional cluster coordinator and dynamic configuration center. Enhance the metrics query for ElasticSearch implementation to increase the stability. Reduce the length of storage entity names in the self-observability for MySQL and TiDB storage. Fix labels are missing in Prometheus analysis context. Fix column length issue in MySQL/TiDB storage. Fix no data in 2nd level aggregation in self-observability. Fix searchService bug in ES implementation. Fix wrong validation of endpoint relation entity query. Fix the bug caused by the OAL debug flag. Fix endpoint dependency bug in MQ and uninstrumented proxy cases. Fix time bucket conversion issue in the InfluxDB storage implementation. Update k8s client to 8.0.0 UI Support endpoint dependency graph. Support x-scroll of trace/profile page Fix database selector issue. Add the bar chart in the UI templates. Document Update the user logo wall. Add backend configuration vocabulary document. Add agent installation doc for Tomcat9 on Windows. Add istioctl ALS commands for the document. Fix TTL documentation. Add FAQ doc about thread instrumentation. CVE Fix fuzzy query sql injection in the MySQL/TiDB storage. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"810\"\u003e8.1.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Kafka as an optional trace, JVM metrics, profiling snapshots and meter system …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.1.0/en/changes/changes-8.1.0/","title":"8.1.0"},{"body":"8.1.0 Project Support Kafka as an optional trace, JVM metrics, profiling snapshots and meter system data transport layer. Support Meter system, including the native metrics APIs and the Spring Sleuth adoption. Support JVM thread metrics. Java Agent [Core] Fix the concurrency access bug in the Concurrency ClassLoader Case. [Core] Separate the config of the plugins from the core level. [Core] Support instrumented class cached in memory or file, to be compatible with other agents, such as Arthas. Add logic endpoint concept. Could analysis any span or tags flagged by the logic endpoint. Add Spring annotation component name for UI visualization only. Add support to trace Call procedures in MySQL plugin. Support GraphQL plugin. Support Quasar fiber plugin. Support InfluxDB java client plugin. Support brpc java plugin Support ConsoleAppender in the logback v1 plugin. Enhance vert.x endpoint names. Optimize the code to prevent mongo statements from being too long. Fix WebFlux plugin concurrency access bug. Fix ShardingSphere plugins internal conflicts. Fix duplicated Spring MVC endpoint. Fix lettuce plugin sometimes trace doesn‘t show span layer. Fix @Tag returnedObject bug. OAP-Backend Support Jetty Server advanced configurations. Support label based filter in the prometheus fetcher and OpenCensus receiver. Support using k8s configmap as the configuration center. Support OAP health check, and storage module health check. Support sampling rate in the dynamic configuration. Add endpoint_relation_sla and endpoint_relation_percentile for endpoint relationship metrics. Add components for Python plugins, including Kafka, Tornado, Redis, Django, PyMysql. Add components for Golang SDK. Add Nacos 1.3.1 back as an optional cluster coordinator and dynamic configuration center. Enhance the metrics query for ElasticSearch implementation to increase the stability. Reduce the length of storage entity names in the self-observability for MySQL and TiDB storage. Fix labels are missing in Prometheus analysis context. Fix column length issue in MySQL/TiDB storage. Fix no data in 2nd level aggregation in self-observability. Fix searchService bug in ES implementation. Fix wrong validation of endpoint relation entity query. Fix the bug caused by the OAL debug flag. Fix endpoint dependency bug in MQ and uninstrumented proxy cases. Fix time bucket conversion issue in the InfluxDB storage implementation. Update k8s client to 8.0.0 UI Support endpoint dependency graph. Support x-scroll of trace/profile page Fix database selector issue. Add the bar chart in the UI templates. Document Update the user logo wall. Add backend configuration vocabulary document. Add agent installation doc for Tomcat9 on Windows. Add istioctl ALS commands for the document. Fix TTL documentation. Add FAQ doc about thread instrumentation. CVE Fix fuzzy query sql injection in the MySQL/TiDB storage. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"810\"\u003e8.1.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Kafka as an optional trace, JVM metrics, profiling snapshots and meter system …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.2.0/en/changes/changes-8.1.0/","title":"8.1.0"},{"body":"8.1.0 Project Support Kafka as an optional trace, JVM metrics, profiling snapshots and meter system data transport layer. Support Meter system, including the native metrics APIs and the Spring Sleuth adoption. Support JVM thread metrics. Java Agent [Core] Fix the concurrency access bug in the Concurrency ClassLoader Case. [Core] Separate the config of the plugins from the core level. [Core] Support instrumented class cached in memory or file, to be compatible with other agents, such as Arthas. Add logic endpoint concept. Could analysis any span or tags flagged by the logic endpoint. Add Spring annotation component name for UI visualization only. Add support to trace Call procedures in MySQL plugin. Support GraphQL plugin. Support Quasar fiber plugin. Support InfluxDB java client plugin. Support brpc java plugin Support ConsoleAppender in the logback v1 plugin. Enhance vert.x endpoint names. Optimize the code to prevent mongo statements from being too long. Fix WebFlux plugin concurrency access bug. Fix ShardingSphere plugins internal conflicts. Fix duplicated Spring MVC endpoint. Fix lettuce plugin sometimes trace doesn‘t show span layer. Fix @Tag returnedObject bug. OAP-Backend Support Jetty Server advanced configurations. Support label based filter in the prometheus fetcher and OpenCensus receiver. Support using k8s configmap as the configuration center. Support OAP health check, and storage module health check. Support sampling rate in the dynamic configuration. Add endpoint_relation_sla and endpoint_relation_percentile for endpoint relationship metrics. Add components for Python plugins, including Kafka, Tornado, Redis, Django, PyMysql. Add components for Golang SDK. Add Nacos 1.3.1 back as an optional cluster coordinator and dynamic configuration center. Enhance the metrics query for ElasticSearch implementation to increase the stability. Reduce the length of storage entity names in the self-observability for MySQL and TiDB storage. Fix labels are missing in Prometheus analysis context. Fix column length issue in MySQL/TiDB storage. Fix no data in 2nd level aggregation in self-observability. Fix searchService bug in ES implementation. Fix wrong validation of endpoint relation entity query. Fix the bug caused by the OAL debug flag. Fix endpoint dependency bug in MQ and uninstrumented proxy cases. Fix time bucket conversion issue in the InfluxDB storage implementation. Update k8s client to 8.0.0 UI Support endpoint dependency graph. Support x-scroll of trace/profile page Fix database selector issue. Add the bar chart in the UI templates. Document Update the user logo wall. Add backend configuration vocabulary document. Add agent installation doc for Tomcat9 on Windows. Add istioctl ALS commands for the document. Fix TTL documentation. Add FAQ doc about thread instrumentation. CVE Fix fuzzy query sql injection in the MySQL/TiDB storage. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"810\"\u003e8.1.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Kafka as an optional trace, JVM metrics, profiling snapshots and meter system …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.3.0/en/changes/changes-8.1.0/","title":"8.1.0"},{"body":"8.1.0 Project Support Kafka as an optional trace, JVM metrics, profiling snapshots and meter system data transport layer. Support Meter system, including the native metrics APIs and the Spring Sleuth adoption. Support JVM thread metrics. Java Agent [Core] Fix the concurrency access bug in the Concurrency ClassLoader Case. [Core] Separate the config of the plugins from the core level. [Core] Support instrumented class cached in memory or file, to be compatible with other agents, such as Arthas. Add logic endpoint concept. Could analysis any span or tags flagged by the logic endpoint. Add Spring annotation component name for UI visualization only. Add support to trace Call procedures in MySQL plugin. Support GraphQL plugin. Support Quasar fiber plugin. Support InfluxDB java client plugin. Support brpc java plugin Support ConsoleAppender in the logback v1 plugin. Enhance vert.x endpoint names. Optimize the code to prevent mongo statements from being too long. Fix WebFlux plugin concurrency access bug. Fix ShardingSphere plugins internal conflicts. Fix duplicated Spring MVC endpoint. Fix lettuce plugin sometimes trace doesn‘t show span layer. Fix @Tag returnedObject bug. OAP-Backend Support Jetty Server advanced configurations. Support label based filter in the prometheus fetcher and OpenCensus receiver. Support using k8s configmap as the configuration center. Support OAP health check, and storage module health check. Support sampling rate in the dynamic configuration. Add endpoint_relation_sla and endpoint_relation_percentile for endpoint relationship metrics. Add components for Python plugins, including Kafka, Tornado, Redis, Django, PyMysql. Add components for Golang SDK. Add Nacos 1.3.1 back as an optional cluster coordinator and dynamic configuration center. Enhance the metrics query for ElasticSearch implementation to increase the stability. Reduce the length of storage entity names in the self-observability for MySQL and TiDB storage. Fix labels are missing in Prometheus analysis context. Fix column length issue in MySQL/TiDB storage. Fix no data in 2nd level aggregation in self-observability. Fix searchService bug in ES implementation. Fix wrong validation of endpoint relation entity query. Fix the bug caused by the OAL debug flag. Fix endpoint dependency bug in MQ and uninstrumented proxy cases. Fix time bucket conversion issue in the InfluxDB storage implementation. Update k8s client to 8.0.0 UI Support endpoint dependency graph. Support x-scroll of trace/profile page Fix database selector issue. Add the bar chart in the UI templates. Document Update the user logo wall. Add backend configuration vocabulary document. Add agent installation doc for Tomcat9 on Windows. Add istioctl ALS commands for the document. Fix TTL documentation. Add FAQ doc about thread instrumentation. CVE Fix fuzzy query sql injection in the MySQL/TiDB storage. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"810\"\u003e8.1.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Kafka as an optional trace, JVM metrics, profiling snapshots and meter system …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.4.0/en/changes/changes-8.1.0/","title":"8.1.0"},{"body":"8.1.0 Project Support Kafka as an optional trace, JVM metrics, profiling snapshots and meter system data transport layer. Support Meter system, including the native metrics APIs and the Spring Sleuth adoption. Support JVM thread metrics. Java Agent [Core] Fix the concurrency access bug in the Concurrency ClassLoader Case. [Core] Separate the config of the plugins from the core level. [Core] Support instrumented class cached in memory or file, to be compatible with other agents, such as Arthas. Add logic endpoint concept. Could analysis any span or tags flagged by the logic endpoint. Add Spring annotation component name for UI visualization only. Add support to trace Call procedures in MySQL plugin. Support GraphQL plugin. Support Quasar fiber plugin. Support InfluxDB java client plugin. Support brpc java plugin Support ConsoleAppender in the logback v1 plugin. Enhance vert.x endpoint names. Optimize the code to prevent mongo statements from being too long. Fix WebFlux plugin concurrency access bug. Fix ShardingSphere plugins internal conflicts. Fix duplicated Spring MVC endpoint. Fix lettuce plugin sometimes trace doesn‘t show span layer. Fix @Tag returnedObject bug. OAP-Backend Support Jetty Server advanced configurations. Support label based filter in the prometheus fetcher and OpenCensus receiver. Support using k8s configmap as the configuration center. Support OAP health check, and storage module health check. Support sampling rate in the dynamic configuration. Add endpoint_relation_sla and endpoint_relation_percentile for endpoint relationship metrics. Add components for Python plugins, including Kafka, Tornado, Redis, Django, PyMysql. Add components for Golang SDK. Add Nacos 1.3.1 back as an optional cluster coordinator and dynamic configuration center. Enhance the metrics query for ElasticSearch implementation to increase the stability. Reduce the length of storage entity names in the self-observability for MySQL and TiDB storage. Fix labels are missing in Prometheus analysis context. Fix column length issue in MySQL/TiDB storage. Fix no data in 2nd level aggregation in self-observability. Fix searchService bug in ES implementation. Fix wrong validation of endpoint relation entity query. Fix the bug caused by the OAL debug flag. Fix endpoint dependency bug in MQ and uninstrumented proxy cases. Fix time bucket conversion issue in the InfluxDB storage implementation. Update k8s client to 8.0.0 UI Support endpoint dependency graph. Support x-scroll of trace/profile page Fix database selector issue. Add the bar chart in the UI templates. Document Update the user logo wall. Add backend configuration vocabulary document. Add agent installation doc for Tomcat9 on Windows. Add istioctl ALS commands for the document. Fix TTL documentation. Add FAQ doc about thread instrumentation. CVE Fix fuzzy query sql injection in the MySQL/TiDB storage. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"810\"\u003e8.1.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Kafka as an optional trace, JVM metrics, profiling snapshots and meter system …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v11.0.0/en/changes/changes-8.1.0/","title":"8.1.0"},{"body":"8.1.0 Project Support Kafka as an optional trace, JVM metrics, profiling snapshots and meter system data transport layer. Support Meter system, including the native metrics APIs and the Spring Sleuth adoption. Support JVM thread metrics. Java Agent [Core] Fix the concurrency access bug in the Concurrency ClassLoader Case. [Core] Separate the config of the plugins from the core level. [Core] Support instrumented class cached in memory or file, to be compatible with other agents, such as Arthas. Add logic endpoint concept. Could analysis any span or tags flagged by the logic endpoint. Add Spring annotation component name for UI visualization only. Add support to trace Call procedures in MySQL plugin. Support GraphQL plugin. Support Quasar fiber plugin. Support InfluxDB java client plugin. Support brpc java plugin Support ConsoleAppender in the logback v1 plugin. Enhance vert.x endpoint names. Optimize the code to prevent mongo statements from being too long. Fix WebFlux plugin concurrency access bug. Fix ShardingSphere plugins internal conflicts. Fix duplicated Spring MVC endpoint. Fix lettuce plugin sometimes trace doesn‘t show span layer. Fix @Tag returnedObject bug. OAP-Backend Support Jetty Server advanced configurations. Support label based filter in the prometheus fetcher and OpenCensus receiver. Support using k8s configmap as the configuration center. Support OAP health check, and storage module health check. Support sampling rate in the dynamic configuration. Add endpoint_relation_sla and endpoint_relation_percentile for endpoint relationship metrics. Add components for Python plugins, including Kafka, Tornado, Redis, Django, PyMysql. Add components for Golang SDK. Add Nacos 1.3.1 back as an optional cluster coordinator and dynamic configuration center. Enhance the metrics query for ElasticSearch implementation to increase the stability. Reduce the length of storage entity names in the self-observability for MySQL and TiDB storage. Fix labels are missing in Prometheus analysis context. Fix column length issue in MySQL/TiDB storage. Fix no data in 2nd level aggregation in self-observability. Fix searchService bug in ES implementation. Fix wrong validation of endpoint relation entity query. Fix the bug caused by the OAL debug flag. Fix endpoint dependency bug in MQ and uninstrumented proxy cases. Fix time bucket conversion issue in the InfluxDB storage implementation. Update k8s client to 8.0.0 UI Support endpoint dependency graph. Support x-scroll of trace/profile page Fix database selector issue. Add the bar chart in the UI templates. Document Update the user logo wall. Add backend configuration vocabulary document. Add agent installation doc for Tomcat9 on Windows. Add istioctl ALS commands for the document. Fix TTL documentation. Add FAQ doc about thread instrumentation. CVE Fix fuzzy query sql injection in the MySQL/TiDB storage. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"810\"\u003e8.1.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Kafka as an optional trace, JVM metrics, profiling snapshots and meter system …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.4.0/en/changes/changes-8.1.0/","title":"8.1.0"},{"body":"8.1.0 Project Support Kafka as an optional trace, JVM metrics, profiling snapshots and meter system data transport layer. Support Meter system, including the native metrics APIs and the Spring Sleuth adoption. Support JVM thread metrics. Java Agent [Core] Fix the concurrency access bug in the Concurrency ClassLoader Case. [Core] Separate the config of the plugins from the core level. [Core] Support instrumented class cached in memory or file, to be compatible with other agents, such as Arthas. Add logic endpoint concept. Could analysis any span or tags flagged by the logic endpoint. Add Spring annotation component name for UI visualization only. Add support to trace Call procedures in MySQL plugin. Support GraphQL plugin. Support Quasar fiber plugin. Support InfluxDB java client plugin. Support brpc java plugin Support ConsoleAppender in the logback v1 plugin. Enhance vert.x endpoint names. Optimize the code to prevent mongo statements from being too long. Fix WebFlux plugin concurrency access bug. Fix ShardingSphere plugins internal conflicts. Fix duplicated Spring MVC endpoint. Fix lettuce plugin sometimes trace doesn‘t show span layer. Fix @Tag returnedObject bug. OAP-Backend Support Jetty Server advanced configurations. Support label based filter in the prometheus fetcher and OpenCensus receiver. Support using k8s configmap as the configuration center. Support OAP health check, and storage module health check. Support sampling rate in the dynamic configuration. Add endpoint_relation_sla and endpoint_relation_percentile for endpoint relationship metrics. Add components for Python plugins, including Kafka, Tornado, Redis, Django, PyMysql. Add components for Golang SDK. Add Nacos 1.3.1 back as an optional cluster coordinator and dynamic configuration center. Enhance the metrics query for ElasticSearch implementation to increase the stability. Reduce the length of storage entity names in the self-observability for MySQL and TiDB storage. Fix labels are missing in Prometheus analysis context. Fix column length issue in MySQL/TiDB storage. Fix no data in 2nd level aggregation in self-observability. Fix searchService bug in ES implementation. Fix wrong validation of endpoint relation entity query. Fix the bug caused by the OAL debug flag. Fix endpoint dependency bug in MQ and uninstrumented proxy cases. Fix time bucket conversion issue in the InfluxDB storage implementation. Update k8s client to 8.0.0 UI Support endpoint dependency graph. Support x-scroll of trace/profile page Fix database selector issue. Add the bar chart in the UI templates. Document Update the user logo wall. Add backend configuration vocabulary document. Add agent installation doc for Tomcat9 on Windows. Add istioctl ALS commands for the document. Fix TTL documentation. Add FAQ doc about thread instrumentation. CVE Fix fuzzy query sql injection in the MySQL/TiDB storage. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"810\"\u003e8.1.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Kafka as an optional trace, JVM metrics, profiling snapshots and meter system …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.5.0/en/changes/changes-8.1.0/","title":"8.1.0"},{"body":"8.1.0 Project Support Kafka as an optional trace, JVM metrics, profiling snapshots and meter system data transport layer. Support Meter system, including the native metrics APIs and the Spring Sleuth adoption. Support JVM thread metrics. Java Agent [Core] Fix the concurrency access bug in the Concurrency ClassLoader Case. [Core] Separate the config of the plugins from the core level. [Core] Support instrumented class cached in memory or file, to be compatible with other agents, such as Arthas. Add logic endpoint concept. Could analysis any span or tags flagged by the logic endpoint. Add Spring annotation component name for UI visualization only. Add support to trace Call procedures in MySQL plugin. Support GraphQL plugin. Support Quasar fiber plugin. Support InfluxDB java client plugin. Support brpc java plugin Support ConsoleAppender in the logback v1 plugin. Enhance vert.x endpoint names. Optimize the code to prevent mongo statements from being too long. Fix WebFlux plugin concurrency access bug. Fix ShardingSphere plugins internal conflicts. Fix duplicated Spring MVC endpoint. Fix lettuce plugin sometimes trace doesn‘t show span layer. Fix @Tag returnedObject bug. OAP-Backend Support Jetty Server advanced configurations. Support label based filter in the prometheus fetcher and OpenCensus receiver. Support using k8s configmap as the configuration center. Support OAP health check, and storage module health check. Support sampling rate in the dynamic configuration. Add endpoint_relation_sla and endpoint_relation_percentile for endpoint relationship metrics. Add components for Python plugins, including Kafka, Tornado, Redis, Django, PyMysql. Add components for Golang SDK. Add Nacos 1.3.1 back as an optional cluster coordinator and dynamic configuration center. Enhance the metrics query for ElasticSearch implementation to increase the stability. Reduce the length of storage entity names in the self-observability for MySQL and TiDB storage. Fix labels are missing in Prometheus analysis context. Fix column length issue in MySQL/TiDB storage. Fix no data in 2nd level aggregation in self-observability. Fix searchService bug in ES implementation. Fix wrong validation of endpoint relation entity query. Fix the bug caused by the OAL debug flag. Fix endpoint dependency bug in MQ and uninstrumented proxy cases. Fix time bucket conversion issue in the InfluxDB storage implementation. Update k8s client to 8.0.0 UI Support endpoint dependency graph. Support x-scroll of trace/profile page Fix database selector issue. Add the bar chart in the UI templates. Document Update the user logo wall. Add backend configuration vocabulary document. Add agent installation doc for Tomcat9 on Windows. Add istioctl ALS commands for the document. Fix TTL documentation. Add FAQ doc about thread instrumentation. CVE Fix fuzzy query sql injection in the MySQL/TiDB storage. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"810\"\u003e8.1.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Kafka as an optional trace, JVM metrics, profiling snapshots and meter system …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.6.0/en/changes/changes-8.1.0/","title":"8.1.0"},{"body":"8.1.0 Project Support Kafka as an optional trace, JVM metrics, profiling snapshots and meter system data transport layer. Support Meter system, including the native metrics APIs and the Spring Sleuth adoption. Support JVM thread metrics. Java Agent [Core] Fix the concurrency access bug in the Concurrency ClassLoader Case. [Core] Separate the config of the plugins from the core level. [Core] Support instrumented class cached in memory or file, to be compatible with other agents, such as Arthas. Add logic endpoint concept. Could analysis any span or tags flagged by the logic endpoint. Add Spring annotation component name for UI visualization only. Add support to trace Call procedures in MySQL plugin. Support GraphQL plugin. Support Quasar fiber plugin. Support InfluxDB java client plugin. Support brpc java plugin Support ConsoleAppender in the logback v1 plugin. Enhance vert.x endpoint names. Optimize the code to prevent mongo statements from being too long. Fix WebFlux plugin concurrency access bug. Fix ShardingSphere plugins internal conflicts. Fix duplicated Spring MVC endpoint. Fix lettuce plugin sometimes trace doesn‘t show span layer. Fix @Tag returnedObject bug. OAP-Backend Support Jetty Server advanced configurations. Support label based filter in the prometheus fetcher and OpenCensus receiver. Support using k8s configmap as the configuration center. Support OAP health check, and storage module health check. Support sampling rate in the dynamic configuration. Add endpoint_relation_sla and endpoint_relation_percentile for endpoint relationship metrics. Add components for Python plugins, including Kafka, Tornado, Redis, Django, PyMysql. Add components for Golang SDK. Add Nacos 1.3.1 back as an optional cluster coordinator and dynamic configuration center. Enhance the metrics query for ElasticSearch implementation to increase the stability. Reduce the length of storage entity names in the self-observability for MySQL and TiDB storage. Fix labels are missing in Prometheus analysis context. Fix column length issue in MySQL/TiDB storage. Fix no data in 2nd level aggregation in self-observability. Fix searchService bug in ES implementation. Fix wrong validation of endpoint relation entity query. Fix the bug caused by the OAL debug flag. Fix endpoint dependency bug in MQ and uninstrumented proxy cases. Fix time bucket conversion issue in the InfluxDB storage implementation. Update k8s client to 8.0.0 UI Support endpoint dependency graph. Support x-scroll of trace/profile page Fix database selector issue. Add the bar chart in the UI templates. Document Update the user logo wall. Add backend configuration vocabulary document. Add agent installation doc for Tomcat9 on Windows. Add istioctl ALS commands for the document. Fix TTL documentation. Add FAQ doc about thread instrumentation. CVE Fix fuzzy query sql injection in the MySQL/TiDB storage. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"810\"\u003e8.1.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Kafka as an optional trace, JVM metrics, profiling snapshots and meter system …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.7.0/en/changes/changes-8.1.0/","title":"8.1.0"},{"body":"8.2.0 Project Support Browser monitoring. Add e2e test for ALS solution of service mesh observability. Support compiling(include testing) in JDK11. Support build a single module. Java Agent Support metrics plugin. Support slf4j logs of gRPC and Kafka(when agent uses them) into the agent log files. Add PROPERTIES_REPORT_PERIOD_FACTOR config to avoid the properties of instance cleared. Limit the size of traced SQL to avoid OOM. Support mount command to load a new set of plugins. Add plugin selector mechanism. Enhance the witness classes for MongoDB plugin. Enhance the parameter truncate mechanism of SQL plugins. Enhance the SpringMVC plugin in the reactive APIs. Enhance the SpringMVC plugin to collect HTTP headers as the span tags. Enhance the Kafka plugin, about @KafkaPollAndInvoke Enhance the configuration initialization core. Plugin could have its own plugins. Enhance Feign plugin to collect parameters. Enhance Dubbo plugin to collect parameters. Provide Thrift plugin. Provide XXL-job plugin. Provide MongoDB 4.x plugin. Provide Kafka client 2.1+ plugin. Provide WebFlux-WebClient plugin. Provide ignore-exception plugin. Provide quartz scheduler plugin. Provide ElasticJob 2.x plugin. Provide Spring @Scheduled plugin. Provide Spring-Kafka plugin. Provide HBase client plugin. Provide JSON log format. Move Spring WebFlux plugin to the optional plugin. Fix inconsistent logic bug in PrefixMatch Fix duplicate exit spans in Feign LoadBalancer mechanism. Fix the target service blocked by the Kafka reporter. Fix configurations of Kafka report don\u0026rsquo;t work. Fix rest template concurrent conflict. Fix NPE in the ActiveMQ plugin. Fix conflict between Kafka reporter and sampling plugin. Fix NPE in the log formatter. Fix span layer missing in certain cases, in the Kafka plugin. Fix error format of time in serviceTraffic update. Upgrade bytebuddy to 1.10.14 OAP-Backend Support Nacos authentication. Support labeled meter in the meter receiver. Separate UI template into multiple files. Provide support for Envoy tracing. Envoy tracer depends on the Envoy community. Support query trace by tags. Support composite alarm rules. Support alarm messages to DingTalk. Support alarm messages to WeChat. Support alarm messages to Slack. Support SSL for Prometheus fetcher and self telemetry. Support labeled histogram in the prometheus format. Support the status of segment based on entry span or first span only. Support the error segment in the sampling mechanism. Support SSL certs of gRPC server. Support labeled metrics in the alarm rule setting. Support to query all labeled data, if no explicit label in the query condition. Add TLS parameters in the mesh analysis. Add health check for InfluxDB storage. Add super dataset concept for the traces/logs. Add separate replicas configuration for super dataset. Add IN operator in the OAL. Add != operator in the OAL. Add like operator in the OAL. Add latest function in the prometheus analysis. Add more configurations in the gRPC server. Optimize the trace query performance. Optimize the CPU usage rate calculation, at least to be 1. Optimize the length of slow SQL column in the MySQL storage. Optimize the topology query, use client side component name when no server side mapping. Add component IDs for Python component. Add component ID range for C++. Fix Slack notification setting NPE. Fix some module missing check of the module manager core. Fix authentication doesn\u0026rsquo;t work in sharing server. Fix metrics batch persistent size bug. Fix trace sampling bug. Fix CLR receiver bug. Fix end time bug in the query process. Fix Exporter INCREMENT mode is not working. Fix an error when executing startup.bat when the log directory exists Add syncBulkActions configuration to set up the batch size of the metrics persistent. Meter Analysis Language. UI Add browser dashboard. Add browser log query page. Support query trace by tags. Fix JVM configuration. Fix CLR configuration. Document Add the document about SW_NO_UPSTREAM_REAL_ADDRESS. Update ALS setup document. Add Customization Config section for plugin development. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"820\"\u003e8.2.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Browser monitoring.\u003c/li\u003e\n\u003cli\u003eAdd e2e test for ALS solution of service mesh …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/latest/en/changes/changes-8.2.0/","title":"8.2.0"},{"body":"8.2.0 Project Support Browser monitoring. Add e2e test for ALS solution of service mesh observability. Support compiling(include testing) in JDK11. Support build a single module. Java Agent Support metrics plugin. Support slf4j logs of gRPC and Kafka(when agent uses them) into the agent log files. Add PROPERTIES_REPORT_PERIOD_FACTOR config to avoid the properties of instance cleared. Limit the size of traced SQL to avoid OOM. Support mount command to load a new set of plugins. Add plugin selector mechanism. Enhance the witness classes for MongoDB plugin. Enhance the parameter truncate mechanism of SQL plugins. Enhance the SpringMVC plugin in the reactive APIs. Enhance the SpringMVC plugin to collect HTTP headers as the span tags. Enhance the Kafka plugin, about @KafkaPollAndInvoke Enhance the configuration initialization core. Plugin could have its own plugins. Enhance Feign plugin to collect parameters. Enhance Dubbo plugin to collect parameters. Provide Thrift plugin. Provide XXL-job plugin. Provide MongoDB 4.x plugin. Provide Kafka client 2.1+ plugin. Provide WebFlux-WebClient plugin. Provide ignore-exception plugin. Provide quartz scheduler plugin. Provide ElasticJob 2.x plugin. Provide Spring @Scheduled plugin. Provide Spring-Kafka plugin. Provide HBase client plugin. Provide JSON log format. Move Spring WebFlux plugin to the optional plugin. Fix inconsistent logic bug in PrefixMatch Fix duplicate exit spans in Feign LoadBalancer mechanism. Fix the target service blocked by the Kafka reporter. Fix configurations of Kafka report don\u0026rsquo;t work. Fix rest template concurrent conflict. Fix NPE in the ActiveMQ plugin. Fix conflict between Kafka reporter and sampling plugin. Fix NPE in the log formatter. Fix span layer missing in certain cases, in the Kafka plugin. Fix error format of time in serviceTraffic update. Upgrade bytebuddy to 1.10.14 OAP-Backend Support Nacos authentication. Support labeled meter in the meter receiver. Separate UI template into multiple files. Provide support for Envoy tracing. Envoy tracer depends on the Envoy community. Support query trace by tags. Support composite alarm rules. Support alarm messages to DingTalk. Support alarm messages to WeChat. Support alarm messages to Slack. Support SSL for Prometheus fetcher and self telemetry. Support labeled histogram in the prometheus format. Support the status of segment based on entry span or first span only. Support the error segment in the sampling mechanism. Support SSL certs of gRPC server. Support labeled metrics in the alarm rule setting. Support to query all labeled data, if no explicit label in the query condition. Add TLS parameters in the mesh analysis. Add health check for InfluxDB storage. Add super dataset concept for the traces/logs. Add separate replicas configuration for super dataset. Add IN operator in the OAL. Add != operator in the OAL. Add like operator in the OAL. Add latest function in the prometheus analysis. Add more configurations in the gRPC server. Optimize the trace query performance. Optimize the CPU usage rate calculation, at least to be 1. Optimize the length of slow SQL column in the MySQL storage. Optimize the topology query, use client side component name when no server side mapping. Add component IDs for Python component. Add component ID range for C++. Fix Slack notification setting NPE. Fix some module missing check of the module manager core. Fix authentication doesn\u0026rsquo;t work in sharing server. Fix metrics batch persistent size bug. Fix trace sampling bug. Fix CLR receiver bug. Fix end time bug in the query process. Fix Exporter INCREMENT mode is not working. Fix an error when executing startup.bat when the log directory exists Add syncBulkActions configuration to set up the batch size of the metrics persistent. Meter Analysis Language. UI Add browser dashboard. Add browser log query page. Support query trace by tags. Fix JVM configuration. Fix CLR configuration. Document Add the document about SW_NO_UPSTREAM_REAL_ADDRESS. Update ALS setup document. Add Customization Config section for plugin development. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"820\"\u003e8.2.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Browser monitoring.\u003c/li\u003e\n\u003cli\u003eAdd e2e test for ALS solution of service mesh …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/next/en/changes/changes-8.2.0/","title":"8.2.0"},{"body":"8.2.0 Project Support Browser monitoring. Add e2e test for ALS solution of service mesh observability. Support compiling(include testing) in JDK11. Support build a single module. Java Agent Support metrics plugin. Support slf4j logs of gRPC and Kafka(when agent uses them) into the agent log files. Add PROPERTIES_REPORT_PERIOD_FACTOR config to avoid the properties of instance cleared. Limit the size of traced SQL to avoid OOM. Support mount command to load a new set of plugins. Add plugin selector mechanism. Enhance the witness classes for MongoDB plugin. Enhance the parameter truncate mechanism of SQL plugins. Enhance the SpringMVC plugin in the reactive APIs. Enhance the SpringMVC plugin to collect HTTP headers as the span tags. Enhance the Kafka plugin, about @KafkaPollAndInvoke Enhance the configuration initialization core. Plugin could have its own plugins. Enhance Feign plugin to collect parameters. Enhance Dubbo plugin to collect parameters. Provide Thrift plugin. Provide XXL-job plugin. Provide MongoDB 4.x plugin. Provide Kafka client 2.1+ plugin. Provide WebFlux-WebClient plugin. Provide ignore-exception plugin. Provide quartz scheduler plugin. Provide ElasticJob 2.x plugin. Provide Spring @Scheduled plugin. Provide Spring-Kafka plugin. Provide HBase client plugin. Provide JSON log format. Move Spring WebFlux plugin to the optional plugin. Fix inconsistent logic bug in PrefixMatch Fix duplicate exit spans in Feign LoadBalancer mechanism. Fix the target service blocked by the Kafka reporter. Fix configurations of Kafka report don\u0026rsquo;t work. Fix rest template concurrent conflict. Fix NPE in the ActiveMQ plugin. Fix conflict between Kafka reporter and sampling plugin. Fix NPE in the log formatter. Fix span layer missing in certain cases, in the Kafka plugin. Fix error format of time in serviceTraffic update. Upgrade bytebuddy to 1.10.14 OAP-Backend Support Nacos authentication. Support labeled meter in the meter receiver. Separate UI template into multiple files. Provide support for Envoy tracing. Envoy tracer depends on the Envoy community. Support query trace by tags. Support composite alarm rules. Support alarm messages to DingTalk. Support alarm messages to WeChat. Support alarm messages to Slack. Support SSL for Prometheus fetcher and self telemetry. Support labeled histogram in the prometheus format. Support the status of segment based on entry span or first span only. Support the error segment in the sampling mechanism. Support SSL certs of gRPC server. Support labeled metrics in the alarm rule setting. Support to query all labeled data, if no explicit label in the query condition. Add TLS parameters in the mesh analysis. Add health check for InfluxDB storage. Add super dataset concept for the traces/logs. Add separate replicas configuration for super dataset. Add IN operator in the OAL. Add != operator in the OAL. Add like operator in the OAL. Add latest function in the prometheus analysis. Add more configurations in the gRPC server. Optimize the trace query performance. Optimize the CPU usage rate calculation, at least to be 1. Optimize the length of slow SQL column in the MySQL storage. Optimize the topology query, use client side component name when no server side mapping. Add component IDs for Python component. Add component ID range for C++. Fix Slack notification setting NPE. Fix some module missing check of the module manager core. Fix authentication doesn\u0026rsquo;t work in sharing server. Fix metrics batch persistent size bug. Fix trace sampling bug. Fix CLR receiver bug. Fix end time bug in the query process. Fix Exporter INCREMENT mode is not working. Fix an error when executing startup.bat when the log directory exists Add syncBulkActions configuration to set up the batch size of the metrics persistent. Meter Analysis Language. UI Add browser dashboard. Add browser log query page. Support query trace by tags. Fix JVM configuration. Fix CLR configuration. Document Add the document about SW_NO_UPSTREAM_REAL_ADDRESS. Update ALS setup document. Add Customization Config section for plugin development. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"820\"\u003e8.2.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Browser monitoring.\u003c/li\u003e\n\u003cli\u003eAdd e2e test for ALS solution of service mesh …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.0.0/en/changes/changes-8.2.0/","title":"8.2.0"},{"body":"8.2.0 Project Support Browser monitoring. Add e2e test for ALS solution of service mesh observability. Support compiling(include testing) in JDK11. Support build a single module. Java Agent Support metrics plugin. Support slf4j logs of gRPC and Kafka(when agent uses them) into the agent log files. Add PROPERTIES_REPORT_PERIOD_FACTOR config to avoid the properties of instance cleared. Limit the size of traced SQL to avoid OOM. Support mount command to load a new set of plugins. Add plugin selector mechanism. Enhance the witness classes for MongoDB plugin. Enhance the parameter truncate mechanism of SQL plugins. Enhance the SpringMVC plugin in the reactive APIs. Enhance the SpringMVC plugin to collect HTTP headers as the span tags. Enhance the Kafka plugin, about @KafkaPollAndInvoke Enhance the configuration initialization core. Plugin could have its own plugins. Enhance Feign plugin to collect parameters. Enhance Dubbo plugin to collect parameters. Provide Thrift plugin. Provide XXL-job plugin. Provide MongoDB 4.x plugin. Provide Kafka client 2.1+ plugin. Provide WebFlux-WebClient plugin. Provide ignore-exception plugin. Provide quartz scheduler plugin. Provide ElasticJob 2.x plugin. Provide Spring @Scheduled plugin. Provide Spring-Kafka plugin. Provide HBase client plugin. Provide JSON log format. Move Spring WebFlux plugin to the optional plugin. Fix inconsistent logic bug in PrefixMatch Fix duplicate exit spans in Feign LoadBalancer mechanism. Fix the target service blocked by the Kafka reporter. Fix configurations of Kafka report don\u0026rsquo;t work. Fix rest template concurrent conflict. Fix NPE in the ActiveMQ plugin. Fix conflict between Kafka reporter and sampling plugin. Fix NPE in the log formatter. Fix span layer missing in certain cases, in the Kafka plugin. Fix error format of time in serviceTraffic update. Upgrade bytebuddy to 1.10.14 OAP-Backend Support Nacos authentication. Support labeled meter in the meter receiver. Separate UI template into multiple files. Provide support for Envoy tracing. Envoy tracer depends on the Envoy community. Support query trace by tags. Support composite alarm rules. Support alarm messages to DingTalk. Support alarm messages to WeChat. Support alarm messages to Slack. Support SSL for Prometheus fetcher and self telemetry. Support labeled histogram in the prometheus format. Support the status of segment based on entry span or first span only. Support the error segment in the sampling mechanism. Support SSL certs of gRPC server. Support labeled metrics in the alarm rule setting. Support to query all labeled data, if no explicit label in the query condition. Add TLS parameters in the mesh analysis. Add health check for InfluxDB storage. Add super dataset concept for the traces/logs. Add separate replicas configuration for super dataset. Add IN operator in the OAL. Add != operator in the OAL. Add like operator in the OAL. Add latest function in the prometheus analysis. Add more configurations in the gRPC server. Optimize the trace query performance. Optimize the CPU usage rate calculation, at least to be 1. Optimize the length of slow SQL column in the MySQL storage. Optimize the topology query, use client side component name when no server side mapping. Add component IDs for Python component. Add component ID range for C++. Fix Slack notification setting NPE. Fix some module missing check of the module manager core. Fix authentication doesn\u0026rsquo;t work in sharing server. Fix metrics batch persistent size bug. Fix trace sampling bug. Fix CLR receiver bug. Fix end time bug in the query process. Fix Exporter INCREMENT mode is not working. Fix an error when executing startup.bat when the log directory exists Add syncBulkActions configuration to set up the batch size of the metrics persistent. Meter Analysis Language. UI Add browser dashboard. Add browser log query page. Support query trace by tags. Fix JVM configuration. Fix CLR configuration. Document Add the document about SW_NO_UPSTREAM_REAL_ADDRESS. Update ALS setup document. Add Customization Config section for plugin development. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"820\"\u003e8.2.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Browser monitoring.\u003c/li\u003e\n\u003cli\u003eAdd e2e test for ALS solution of service mesh …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.0.1/en/changes/changes-8.2.0/","title":"8.2.0"},{"body":"8.2.0 Project Support Browser monitoring. Add e2e test for ALS solution of service mesh observability. Support compiling(include testing) in JDK11. Support build a single module. Java Agent Support metrics plugin. Support slf4j logs of gRPC and Kafka(when agent uses them) into the agent log files. Add PROPERTIES_REPORT_PERIOD_FACTOR config to avoid the properties of instance cleared. Limit the size of traced SQL to avoid OOM. Support mount command to load a new set of plugins. Add plugin selector mechanism. Enhance the witness classes for MongoDB plugin. Enhance the parameter truncate mechanism of SQL plugins. Enhance the SpringMVC plugin in the reactive APIs. Enhance the SpringMVC plugin to collect HTTP headers as the span tags. Enhance the Kafka plugin, about @KafkaPollAndInvoke Enhance the configuration initialization core. Plugin could have its own plugins. Enhance Feign plugin to collect parameters. Enhance Dubbo plugin to collect parameters. Provide Thrift plugin. Provide XXL-job plugin. Provide MongoDB 4.x plugin. Provide Kafka client 2.1+ plugin. Provide WebFlux-WebClient plugin. Provide ignore-exception plugin. Provide quartz scheduler plugin. Provide ElasticJob 2.x plugin. Provide Spring @Scheduled plugin. Provide Spring-Kafka plugin. Provide HBase client plugin. Provide JSON log format. Move Spring WebFlux plugin to the optional plugin. Fix inconsistent logic bug in PrefixMatch Fix duplicate exit spans in Feign LoadBalancer mechanism. Fix the target service blocked by the Kafka reporter. Fix configurations of Kafka report don\u0026rsquo;t work. Fix rest template concurrent conflict. Fix NPE in the ActiveMQ plugin. Fix conflict between Kafka reporter and sampling plugin. Fix NPE in the log formatter. Fix span layer missing in certain cases, in the Kafka plugin. Fix error format of time in serviceTraffic update. Upgrade bytebuddy to 1.10.14 OAP-Backend Support Nacos authentication. Support labeled meter in the meter receiver. Separate UI template into multiple files. Provide support for Envoy tracing. Envoy tracer depends on the Envoy community. Support query trace by tags. Support composite alarm rules. Support alarm messages to DingTalk. Support alarm messages to WeChat. Support alarm messages to Slack. Support SSL for Prometheus fetcher and self telemetry. Support labeled histogram in the prometheus format. Support the status of segment based on entry span or first span only. Support the error segment in the sampling mechanism. Support SSL certs of gRPC server. Support labeled metrics in the alarm rule setting. Support to query all labeled data, if no explicit label in the query condition. Add TLS parameters in the mesh analysis. Add health check for InfluxDB storage. Add super dataset concept for the traces/logs. Add separate replicas configuration for super dataset. Add IN operator in the OAL. Add != operator in the OAL. Add like operator in the OAL. Add latest function in the prometheus analysis. Add more configurations in the gRPC server. Optimize the trace query performance. Optimize the CPU usage rate calculation, at least to be 1. Optimize the length of slow SQL column in the MySQL storage. Optimize the topology query, use client side component name when no server side mapping. Add component IDs for Python component. Add component ID range for C++. Fix Slack notification setting NPE. Fix some module missing check of the module manager core. Fix authentication doesn\u0026rsquo;t work in sharing server. Fix metrics batch persistent size bug. Fix trace sampling bug. Fix CLR receiver bug. Fix end time bug in the query process. Fix Exporter INCREMENT mode is not working. Fix an error when executing startup.bat when the log directory exists Add syncBulkActions configuration to set up the batch size of the metrics persistent. Meter Analysis Language. UI Add browser dashboard. Add browser log query page. Support query trace by tags. Fix JVM configuration. Fix CLR configuration. Document Add the document about SW_NO_UPSTREAM_REAL_ADDRESS. Update ALS setup document. Add Customization Config section for plugin development. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"820\"\u003e8.2.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Browser monitoring.\u003c/li\u003e\n\u003cli\u003eAdd e2e test for ALS solution of service mesh …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.1.0/en/changes/changes-8.2.0/","title":"8.2.0"},{"body":"8.2.0 Project Support Browser monitoring. Add e2e test for ALS solution of service mesh observability. Support compiling(include testing) in JDK11. Support build a single module. Java Agent Support metrics plugin. Support slf4j logs of gRPC and Kafka(when agent uses them) into the agent log files. Add PROPERTIES_REPORT_PERIOD_FACTOR config to avoid the properties of instance cleared. Limit the size of traced SQL to avoid OOM. Support mount command to load a new set of plugins. Add plugin selector mechanism. Enhance the witness classes for MongoDB plugin. Enhance the parameter truncate mechanism of SQL plugins. Enhance the SpringMVC plugin in the reactive APIs. Enhance the SpringMVC plugin to collect HTTP headers as the span tags. Enhance the Kafka plugin, about @KafkaPollAndInvoke Enhance the configuration initialization core. Plugin could have its own plugins. Enhance Feign plugin to collect parameters. Enhance Dubbo plugin to collect parameters. Provide Thrift plugin. Provide XXL-job plugin. Provide MongoDB 4.x plugin. Provide Kafka client 2.1+ plugin. Provide WebFlux-WebClient plugin. Provide ignore-exception plugin. Provide quartz scheduler plugin. Provide ElasticJob 2.x plugin. Provide Spring @Scheduled plugin. Provide Spring-Kafka plugin. Provide HBase client plugin. Provide JSON log format. Move Spring WebFlux plugin to the optional plugin. Fix inconsistent logic bug in PrefixMatch Fix duplicate exit spans in Feign LoadBalancer mechanism. Fix the target service blocked by the Kafka reporter. Fix configurations of Kafka report don\u0026rsquo;t work. Fix rest template concurrent conflict. Fix NPE in the ActiveMQ plugin. Fix conflict between Kafka reporter and sampling plugin. Fix NPE in the log formatter. Fix span layer missing in certain cases, in the Kafka plugin. Fix error format of time in serviceTraffic update. Upgrade bytebuddy to 1.10.14 OAP-Backend Support Nacos authentication. Support labeled meter in the meter receiver. Separate UI template into multiple files. Provide support for Envoy tracing. Envoy tracer depends on the Envoy community. Support query trace by tags. Support composite alarm rules. Support alarm messages to DingTalk. Support alarm messages to WeChat. Support alarm messages to Slack. Support SSL for Prometheus fetcher and self telemetry. Support labeled histogram in the prometheus format. Support the status of segment based on entry span or first span only. Support the error segment in the sampling mechanism. Support SSL certs of gRPC server. Support labeled metrics in the alarm rule setting. Support to query all labeled data, if no explicit label in the query condition. Add TLS parameters in the mesh analysis. Add health check for InfluxDB storage. Add super dataset concept for the traces/logs. Add separate replicas configuration for super dataset. Add IN operator in the OAL. Add != operator in the OAL. Add like operator in the OAL. Add latest function in the prometheus analysis. Add more configurations in the gRPC server. Optimize the trace query performance. Optimize the CPU usage rate calculation, at least to be 1. Optimize the length of slow SQL column in the MySQL storage. Optimize the topology query, use client side component name when no server side mapping. Add component IDs for Python component. Add component ID range for C++. Fix Slack notification setting NPE. Fix some module missing check of the module manager core. Fix authentication doesn\u0026rsquo;t work in sharing server. Fix metrics batch persistent size bug. Fix trace sampling bug. Fix CLR receiver bug. Fix end time bug in the query process. Fix Exporter INCREMENT mode is not working. Fix an error when executing startup.bat when the log directory exists Add syncBulkActions configuration to set up the batch size of the metrics persistent. Meter Analysis Language. UI Add browser dashboard. Add browser log query page. Support query trace by tags. Fix JVM configuration. Fix CLR configuration. Document Add the document about SW_NO_UPSTREAM_REAL_ADDRESS. Update ALS setup document. Add Customization Config section for plugin development. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"820\"\u003e8.2.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Browser monitoring.\u003c/li\u003e\n\u003cli\u003eAdd e2e test for ALS solution of service mesh …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.2.0/en/changes/changes-8.2.0/","title":"8.2.0"},{"body":"8.2.0 Project Support Browser monitoring. Add e2e test for ALS solution of service mesh observability. Support compiling(include testing) in JDK11. Support build a single module. Java Agent Support metrics plugin. Support slf4j logs of gRPC and Kafka(when agent uses them) into the agent log files. Add PROPERTIES_REPORT_PERIOD_FACTOR config to avoid the properties of instance cleared. Limit the size of traced SQL to avoid OOM. Support mount command to load a new set of plugins. Add plugin selector mechanism. Enhance the witness classes for MongoDB plugin. Enhance the parameter truncate mechanism of SQL plugins. Enhance the SpringMVC plugin in the reactive APIs. Enhance the SpringMVC plugin to collect HTTP headers as the span tags. Enhance the Kafka plugin, about @KafkaPollAndInvoke Enhance the configuration initialization core. Plugin could have its own plugins. Enhance Feign plugin to collect parameters. Enhance Dubbo plugin to collect parameters. Provide Thrift plugin. Provide XXL-job plugin. Provide MongoDB 4.x plugin. Provide Kafka client 2.1+ plugin. Provide WebFlux-WebClient plugin. Provide ignore-exception plugin. Provide quartz scheduler plugin. Provide ElasticJob 2.x plugin. Provide Spring @Scheduled plugin. Provide Spring-Kafka plugin. Provide HBase client plugin. Provide JSON log format. Move Spring WebFlux plugin to the optional plugin. Fix inconsistent logic bug in PrefixMatch Fix duplicate exit spans in Feign LoadBalancer mechanism. Fix the target service blocked by the Kafka reporter. Fix configurations of Kafka report don\u0026rsquo;t work. Fix rest template concurrent conflict. Fix NPE in the ActiveMQ plugin. Fix conflict between Kafka reporter and sampling plugin. Fix NPE in the log formatter. Fix span layer missing in certain cases, in the Kafka plugin. Fix error format of time in serviceTraffic update. Upgrade bytebuddy to 1.10.14 OAP-Backend Support Nacos authentication. Support labeled meter in the meter receiver. Separate UI template into multiple files. Provide support for Envoy tracing. Envoy tracer depends on the Envoy community. Support query trace by tags. Support composite alarm rules. Support alarm messages to DingTalk. Support alarm messages to WeChat. Support alarm messages to Slack. Support SSL for Prometheus fetcher and self telemetry. Support labeled histogram in the prometheus format. Support the status of segment based on entry span or first span only. Support the error segment in the sampling mechanism. Support SSL certs of gRPC server. Support labeled metrics in the alarm rule setting. Support to query all labeled data, if no explicit label in the query condition. Add TLS parameters in the mesh analysis. Add health check for InfluxDB storage. Add super dataset concept for the traces/logs. Add separate replicas configuration for super dataset. Add IN operator in the OAL. Add != operator in the OAL. Add like operator in the OAL. Add latest function in the prometheus analysis. Add more configurations in the gRPC server. Optimize the trace query performance. Optimize the CPU usage rate calculation, at least to be 1. Optimize the length of slow SQL column in the MySQL storage. Optimize the topology query, use client side component name when no server side mapping. Add component IDs for Python component. Add component ID range for C++. Fix Slack notification setting NPE. Fix some module missing check of the module manager core. Fix authentication doesn\u0026rsquo;t work in sharing server. Fix metrics batch persistent size bug. Fix trace sampling bug. Fix CLR receiver bug. Fix end time bug in the query process. Fix Exporter INCREMENT mode is not working. Fix an error when executing startup.bat when the log directory exists Add syncBulkActions configuration to set up the batch size of the metrics persistent. Meter Analysis Language. UI Add browser dashboard. Add browser log query page. Support query trace by tags. Fix JVM configuration. Fix CLR configuration. Document Add the document about SW_NO_UPSTREAM_REAL_ADDRESS. Update ALS setup document. Add Customization Config section for plugin development. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"820\"\u003e8.2.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Browser monitoring.\u003c/li\u003e\n\u003cli\u003eAdd e2e test for ALS solution of service mesh …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.3.0/en/changes/changes-8.2.0/","title":"8.2.0"},{"body":"8.2.0 Project Support Browser monitoring. Add e2e test for ALS solution of service mesh observability. Support compiling(include testing) in JDK11. Support build a single module. Java Agent Support metrics plugin. Support slf4j logs of gRPC and Kafka(when agent uses them) into the agent log files. Add PROPERTIES_REPORT_PERIOD_FACTOR config to avoid the properties of instance cleared. Limit the size of traced SQL to avoid OOM. Support mount command to load a new set of plugins. Add plugin selector mechanism. Enhance the witness classes for MongoDB plugin. Enhance the parameter truncate mechanism of SQL plugins. Enhance the SpringMVC plugin in the reactive APIs. Enhance the SpringMVC plugin to collect HTTP headers as the span tags. Enhance the Kafka plugin, about @KafkaPollAndInvoke Enhance the configuration initialization core. Plugin could have its own plugins. Enhance Feign plugin to collect parameters. Enhance Dubbo plugin to collect parameters. Provide Thrift plugin. Provide XXL-job plugin. Provide MongoDB 4.x plugin. Provide Kafka client 2.1+ plugin. Provide WebFlux-WebClient plugin. Provide ignore-exception plugin. Provide quartz scheduler plugin. Provide ElasticJob 2.x plugin. Provide Spring @Scheduled plugin. Provide Spring-Kafka plugin. Provide HBase client plugin. Provide JSON log format. Move Spring WebFlux plugin to the optional plugin. Fix inconsistent logic bug in PrefixMatch Fix duplicate exit spans in Feign LoadBalancer mechanism. Fix the target service blocked by the Kafka reporter. Fix configurations of Kafka report don\u0026rsquo;t work. Fix rest template concurrent conflict. Fix NPE in the ActiveMQ plugin. Fix conflict between Kafka reporter and sampling plugin. Fix NPE in the log formatter. Fix span layer missing in certain cases, in the Kafka plugin. Fix error format of time in serviceTraffic update. Upgrade bytebuddy to 1.10.14 OAP-Backend Support Nacos authentication. Support labeled meter in the meter receiver. Separate UI template into multiple files. Provide support for Envoy tracing. Envoy tracer depends on the Envoy community. Support query trace by tags. Support composite alarm rules. Support alarm messages to DingTalk. Support alarm messages to WeChat. Support alarm messages to Slack. Support SSL for Prometheus fetcher and self telemetry. Support labeled histogram in the prometheus format. Support the status of segment based on entry span or first span only. Support the error segment in the sampling mechanism. Support SSL certs of gRPC server. Support labeled metrics in the alarm rule setting. Support to query all labeled data, if no explicit label in the query condition. Add TLS parameters in the mesh analysis. Add health check for InfluxDB storage. Add super dataset concept for the traces/logs. Add separate replicas configuration for super dataset. Add IN operator in the OAL. Add != operator in the OAL. Add like operator in the OAL. Add latest function in the prometheus analysis. Add more configurations in the gRPC server. Optimize the trace query performance. Optimize the CPU usage rate calculation, at least to be 1. Optimize the length of slow SQL column in the MySQL storage. Optimize the topology query, use client side component name when no server side mapping. Add component IDs for Python component. Add component ID range for C++. Fix Slack notification setting NPE. Fix some module missing check of the module manager core. Fix authentication doesn\u0026rsquo;t work in sharing server. Fix metrics batch persistent size bug. Fix trace sampling bug. Fix CLR receiver bug. Fix end time bug in the query process. Fix Exporter INCREMENT mode is not working. Fix an error when executing startup.bat when the log directory exists Add syncBulkActions configuration to set up the batch size of the metrics persistent. Meter Analysis Language. UI Add browser dashboard. Add browser log query page. Support query trace by tags. Fix JVM configuration. Fix CLR configuration. Document Add the document about SW_NO_UPSTREAM_REAL_ADDRESS. Update ALS setup document. Add Customization Config section for plugin development. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"820\"\u003e8.2.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Browser monitoring.\u003c/li\u003e\n\u003cli\u003eAdd e2e test for ALS solution of service mesh …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.4.0/en/changes/changes-8.2.0/","title":"8.2.0"},{"body":"8.2.0 Project Support Browser monitoring. Add e2e test for ALS solution of service mesh observability. Support compiling(include testing) in JDK11. Support build a single module. Java Agent Support metrics plugin. Support slf4j logs of gRPC and Kafka(when agent uses them) into the agent log files. Add PROPERTIES_REPORT_PERIOD_FACTOR config to avoid the properties of instance cleared. Limit the size of traced SQL to avoid OOM. Support mount command to load a new set of plugins. Add plugin selector mechanism. Enhance the witness classes for MongoDB plugin. Enhance the parameter truncate mechanism of SQL plugins. Enhance the SpringMVC plugin in the reactive APIs. Enhance the SpringMVC plugin to collect HTTP headers as the span tags. Enhance the Kafka plugin, about @KafkaPollAndInvoke Enhance the configuration initialization core. Plugin could have its own plugins. Enhance Feign plugin to collect parameters. Enhance Dubbo plugin to collect parameters. Provide Thrift plugin. Provide XXL-job plugin. Provide MongoDB 4.x plugin. Provide Kafka client 2.1+ plugin. Provide WebFlux-WebClient plugin. Provide ignore-exception plugin. Provide quartz scheduler plugin. Provide ElasticJob 2.x plugin. Provide Spring @Scheduled plugin. Provide Spring-Kafka plugin. Provide HBase client plugin. Provide JSON log format. Move Spring WebFlux plugin to the optional plugin. Fix inconsistent logic bug in PrefixMatch Fix duplicate exit spans in Feign LoadBalancer mechanism. Fix the target service blocked by the Kafka reporter. Fix configurations of Kafka report don\u0026rsquo;t work. Fix rest template concurrent conflict. Fix NPE in the ActiveMQ plugin. Fix conflict between Kafka reporter and sampling plugin. Fix NPE in the log formatter. Fix span layer missing in certain cases, in the Kafka plugin. Fix error format of time in serviceTraffic update. Upgrade bytebuddy to 1.10.14 OAP-Backend Support Nacos authentication. Support labeled meter in the meter receiver. Separate UI template into multiple files. Provide support for Envoy tracing. Envoy tracer depends on the Envoy community. Support query trace by tags. Support composite alarm rules. Support alarm messages to DingTalk. Support alarm messages to WeChat. Support alarm messages to Slack. Support SSL for Prometheus fetcher and self telemetry. Support labeled histogram in the prometheus format. Support the status of segment based on entry span or first span only. Support the error segment in the sampling mechanism. Support SSL certs of gRPC server. Support labeled metrics in the alarm rule setting. Support to query all labeled data, if no explicit label in the query condition. Add TLS parameters in the mesh analysis. Add health check for InfluxDB storage. Add super dataset concept for the traces/logs. Add separate replicas configuration for super dataset. Add IN operator in the OAL. Add != operator in the OAL. Add like operator in the OAL. Add latest function in the prometheus analysis. Add more configurations in the gRPC server. Optimize the trace query performance. Optimize the CPU usage rate calculation, at least to be 1. Optimize the length of slow SQL column in the MySQL storage. Optimize the topology query, use client side component name when no server side mapping. Add component IDs for Python component. Add component ID range for C++. Fix Slack notification setting NPE. Fix some module missing check of the module manager core. Fix authentication doesn\u0026rsquo;t work in sharing server. Fix metrics batch persistent size bug. Fix trace sampling bug. Fix CLR receiver bug. Fix end time bug in the query process. Fix Exporter INCREMENT mode is not working. Fix an error when executing startup.bat when the log directory exists Add syncBulkActions configuration to set up the batch size of the metrics persistent. Meter Analysis Language. UI Add browser dashboard. Add browser log query page. Support query trace by tags. Fix JVM configuration. Fix CLR configuration. Document Add the document about SW_NO_UPSTREAM_REAL_ADDRESS. Update ALS setup document. Add Customization Config section for plugin development. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"820\"\u003e8.2.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Browser monitoring.\u003c/li\u003e\n\u003cli\u003eAdd e2e test for ALS solution of service mesh …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v11.0.0/en/changes/changes-8.2.0/","title":"8.2.0"},{"body":"8.2.0 Project Support Browser monitoring. Add e2e test for ALS solution of service mesh observability. Support compiling(include testing) in JDK11. Support build a single module. Java Agent Support metrics plugin. Support slf4j logs of gRPC and Kafka(when agent uses them) into the agent log files. Add PROPERTIES_REPORT_PERIOD_FACTOR config to avoid the properties of instance cleared. Limit the size of traced SQL to avoid OOM. Support mount command to load a new set of plugins. Add plugin selector mechanism. Enhance the witness classes for MongoDB plugin. Enhance the parameter truncate mechanism of SQL plugins. Enhance the SpringMVC plugin in the reactive APIs. Enhance the SpringMVC plugin to collect HTTP headers as the span tags. Enhance the Kafka plugin, about @KafkaPollAndInvoke Enhance the configuration initialization core. Plugin could have its own plugins. Enhance Feign plugin to collect parameters. Enhance Dubbo plugin to collect parameters. Provide Thrift plugin. Provide XXL-job plugin. Provide MongoDB 4.x plugin. Provide Kafka client 2.1+ plugin. Provide WebFlux-WebClient plugin. Provide ignore-exception plugin. Provide quartz scheduler plugin. Provide ElasticJob 2.x plugin. Provide Spring @Scheduled plugin. Provide Spring-Kafka plugin. Provide HBase client plugin. Provide JSON log format. Move Spring WebFlux plugin to the optional plugin. Fix inconsistent logic bug in PrefixMatch Fix duplicate exit spans in Feign LoadBalancer mechanism. Fix the target service blocked by the Kafka reporter. Fix configurations of Kafka report don\u0026rsquo;t work. Fix rest template concurrent conflict. Fix NPE in the ActiveMQ plugin. Fix conflict between Kafka reporter and sampling plugin. Fix NPE in the log formatter. Fix span layer missing in certain cases, in the Kafka plugin. Fix error format of time in serviceTraffic update. Upgrade bytebuddy to 1.10.14 OAP-Backend Support Nacos authentication. Support labeled meter in the meter receiver. Separate UI template into multiple files. Provide support for Envoy tracing. Envoy tracer depends on the Envoy community. Support query trace by tags. Support composite alarm rules. Support alarm messages to DingTalk. Support alarm messages to WeChat. Support alarm messages to Slack. Support SSL for Prometheus fetcher and self telemetry. Support labeled histogram in the prometheus format. Support the status of segment based on entry span or first span only. Support the error segment in the sampling mechanism. Support SSL certs of gRPC server. Support labeled metrics in the alarm rule setting. Support to query all labeled data, if no explicit label in the query condition. Add TLS parameters in the mesh analysis. Add health check for InfluxDB storage. Add super dataset concept for the traces/logs. Add separate replicas configuration for super dataset. Add IN operator in the OAL. Add != operator in the OAL. Add like operator in the OAL. Add latest function in the prometheus analysis. Add more configurations in the gRPC server. Optimize the trace query performance. Optimize the CPU usage rate calculation, at least to be 1. Optimize the length of slow SQL column in the MySQL storage. Optimize the topology query, use client side component name when no server side mapping. Add component IDs for Python component. Add component ID range for C++. Fix Slack notification setting NPE. Fix some module missing check of the module manager core. Fix authentication doesn\u0026rsquo;t work in sharing server. Fix metrics batch persistent size bug. Fix trace sampling bug. Fix CLR receiver bug. Fix end time bug in the query process. Fix Exporter INCREMENT mode is not working. Fix an error when executing startup.bat when the log directory exists Add syncBulkActions configuration to set up the batch size of the metrics persistent. Meter Analysis Language. UI Add browser dashboard. Add browser log query page. Support query trace by tags. Fix JVM configuration. Fix CLR configuration. Document Add the document about SW_NO_UPSTREAM_REAL_ADDRESS. Update ALS setup document. Add Customization Config section for plugin development. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"820\"\u003e8.2.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Browser monitoring.\u003c/li\u003e\n\u003cli\u003eAdd e2e test for ALS solution of service mesh …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.4.0/en/changes/changes-8.2.0/","title":"8.2.0"},{"body":"8.2.0 Project Support Browser monitoring. Add e2e test for ALS solution of service mesh observability. Support compiling(include testing) in JDK11. Support build a single module. Java Agent Support metrics plugin. Support slf4j logs of gRPC and Kafka(when agent uses them) into the agent log files. Add PROPERTIES_REPORT_PERIOD_FACTOR config to avoid the properties of instance cleared. Limit the size of traced SQL to avoid OOM. Support mount command to load a new set of plugins. Add plugin selector mechanism. Enhance the witness classes for MongoDB plugin. Enhance the parameter truncate mechanism of SQL plugins. Enhance the SpringMVC plugin in the reactive APIs. Enhance the SpringMVC plugin to collect HTTP headers as the span tags. Enhance the Kafka plugin, about @KafkaPollAndInvoke Enhance the configuration initialization core. Plugin could have its own plugins. Enhance Feign plugin to collect parameters. Enhance Dubbo plugin to collect parameters. Provide Thrift plugin. Provide XXL-job plugin. Provide MongoDB 4.x plugin. Provide Kafka client 2.1+ plugin. Provide WebFlux-WebClient plugin. Provide ignore-exception plugin. Provide quartz scheduler plugin. Provide ElasticJob 2.x plugin. Provide Spring @Scheduled plugin. Provide Spring-Kafka plugin. Provide HBase client plugin. Provide JSON log format. Move Spring WebFlux plugin to the optional plugin. Fix inconsistent logic bug in PrefixMatch Fix duplicate exit spans in Feign LoadBalancer mechanism. Fix the target service blocked by the Kafka reporter. Fix configurations of Kafka report don\u0026rsquo;t work. Fix rest template concurrent conflict. Fix NPE in the ActiveMQ plugin. Fix conflict between Kafka reporter and sampling plugin. Fix NPE in the log formatter. Fix span layer missing in certain cases, in the Kafka plugin. Fix error format of time in serviceTraffic update. Upgrade bytebuddy to 1.10.14 OAP-Backend Support Nacos authentication. Support labeled meter in the meter receiver. Separate UI template into multiple files. Provide support for Envoy tracing. Envoy tracer depends on the Envoy community. Support query trace by tags. Support composite alarm rules. Support alarm messages to DingTalk. Support alarm messages to WeChat. Support alarm messages to Slack. Support SSL for Prometheus fetcher and self telemetry. Support labeled histogram in the prometheus format. Support the status of segment based on entry span or first span only. Support the error segment in the sampling mechanism. Support SSL certs of gRPC server. Support labeled metrics in the alarm rule setting. Support to query all labeled data, if no explicit label in the query condition. Add TLS parameters in the mesh analysis. Add health check for InfluxDB storage. Add super dataset concept for the traces/logs. Add separate replicas configuration for super dataset. Add IN operator in the OAL. Add != operator in the OAL. Add like operator in the OAL. Add latest function in the prometheus analysis. Add more configurations in the gRPC server. Optimize the trace query performance. Optimize the CPU usage rate calculation, at least to be 1. Optimize the length of slow SQL column in the MySQL storage. Optimize the topology query, use client side component name when no server side mapping. Add component IDs for Python component. Add component ID range for C++. Fix Slack notification setting NPE. Fix some module missing check of the module manager core. Fix authentication doesn\u0026rsquo;t work in sharing server. Fix metrics batch persistent size bug. Fix trace sampling bug. Fix CLR receiver bug. Fix end time bug in the query process. Fix Exporter INCREMENT mode is not working. Fix an error when executing startup.bat when the log directory exists Add syncBulkActions configuration to set up the batch size of the metrics persistent. Meter Analysis Language. UI Add browser dashboard. Add browser log query page. Support query trace by tags. Fix JVM configuration. Fix CLR configuration. Document Add the document about SW_NO_UPSTREAM_REAL_ADDRESS. Update ALS setup document. Add Customization Config section for plugin development. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"820\"\u003e8.2.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Browser monitoring.\u003c/li\u003e\n\u003cli\u003eAdd e2e test for ALS solution of service mesh …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.5.0/en/changes/changes-8.2.0/","title":"8.2.0"},{"body":"8.2.0 Project Support Browser monitoring. Add e2e test for ALS solution of service mesh observability. Support compiling(include testing) in JDK11. Support build a single module. Java Agent Support metrics plugin. Support slf4j logs of gRPC and Kafka(when agent uses them) into the agent log files. Add PROPERTIES_REPORT_PERIOD_FACTOR config to avoid the properties of instance cleared. Limit the size of traced SQL to avoid OOM. Support mount command to load a new set of plugins. Add plugin selector mechanism. Enhance the witness classes for MongoDB plugin. Enhance the parameter truncate mechanism of SQL plugins. Enhance the SpringMVC plugin in the reactive APIs. Enhance the SpringMVC plugin to collect HTTP headers as the span tags. Enhance the Kafka plugin, about @KafkaPollAndInvoke Enhance the configuration initialization core. Plugin could have its own plugins. Enhance Feign plugin to collect parameters. Enhance Dubbo plugin to collect parameters. Provide Thrift plugin. Provide XXL-job plugin. Provide MongoDB 4.x plugin. Provide Kafka client 2.1+ plugin. Provide WebFlux-WebClient plugin. Provide ignore-exception plugin. Provide quartz scheduler plugin. Provide ElasticJob 2.x plugin. Provide Spring @Scheduled plugin. Provide Spring-Kafka plugin. Provide HBase client plugin. Provide JSON log format. Move Spring WebFlux plugin to the optional plugin. Fix inconsistent logic bug in PrefixMatch Fix duplicate exit spans in Feign LoadBalancer mechanism. Fix the target service blocked by the Kafka reporter. Fix configurations of Kafka report don\u0026rsquo;t work. Fix rest template concurrent conflict. Fix NPE in the ActiveMQ plugin. Fix conflict between Kafka reporter and sampling plugin. Fix NPE in the log formatter. Fix span layer missing in certain cases, in the Kafka plugin. Fix error format of time in serviceTraffic update. Upgrade bytebuddy to 1.10.14 OAP-Backend Support Nacos authentication. Support labeled meter in the meter receiver. Separate UI template into multiple files. Provide support for Envoy tracing. Envoy tracer depends on the Envoy community. Support query trace by tags. Support composite alarm rules. Support alarm messages to DingTalk. Support alarm messages to WeChat. Support alarm messages to Slack. Support SSL for Prometheus fetcher and self telemetry. Support labeled histogram in the prometheus format. Support the status of segment based on entry span or first span only. Support the error segment in the sampling mechanism. Support SSL certs of gRPC server. Support labeled metrics in the alarm rule setting. Support to query all labeled data, if no explicit label in the query condition. Add TLS parameters in the mesh analysis. Add health check for InfluxDB storage. Add super dataset concept for the traces/logs. Add separate replicas configuration for super dataset. Add IN operator in the OAL. Add != operator in the OAL. Add like operator in the OAL. Add latest function in the prometheus analysis. Add more configurations in the gRPC server. Optimize the trace query performance. Optimize the CPU usage rate calculation, at least to be 1. Optimize the length of slow SQL column in the MySQL storage. Optimize the topology query, use client side component name when no server side mapping. Add component IDs for Python component. Add component ID range for C++. Fix Slack notification setting NPE. Fix some module missing check of the module manager core. Fix authentication doesn\u0026rsquo;t work in sharing server. Fix metrics batch persistent size bug. Fix trace sampling bug. Fix CLR receiver bug. Fix end time bug in the query process. Fix Exporter INCREMENT mode is not working. Fix an error when executing startup.bat when the log directory exists Add syncBulkActions configuration to set up the batch size of the metrics persistent. Meter Analysis Language. UI Add browser dashboard. Add browser log query page. Support query trace by tags. Fix JVM configuration. Fix CLR configuration. Document Add the document about SW_NO_UPSTREAM_REAL_ADDRESS. Update ALS setup document. Add Customization Config section for plugin development. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"820\"\u003e8.2.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Browser monitoring.\u003c/li\u003e\n\u003cli\u003eAdd e2e test for ALS solution of service mesh …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.6.0/en/changes/changes-8.2.0/","title":"8.2.0"},{"body":"8.2.0 Project Support Browser monitoring. Add e2e test for ALS solution of service mesh observability. Support compiling(include testing) in JDK11. Support build a single module. Java Agent Support metrics plugin. Support slf4j logs of gRPC and Kafka(when agent uses them) into the agent log files. Add PROPERTIES_REPORT_PERIOD_FACTOR config to avoid the properties of instance cleared. Limit the size of traced SQL to avoid OOM. Support mount command to load a new set of plugins. Add plugin selector mechanism. Enhance the witness classes for MongoDB plugin. Enhance the parameter truncate mechanism of SQL plugins. Enhance the SpringMVC plugin in the reactive APIs. Enhance the SpringMVC plugin to collect HTTP headers as the span tags. Enhance the Kafka plugin, about @KafkaPollAndInvoke Enhance the configuration initialization core. Plugin could have its own plugins. Enhance Feign plugin to collect parameters. Enhance Dubbo plugin to collect parameters. Provide Thrift plugin. Provide XXL-job plugin. Provide MongoDB 4.x plugin. Provide Kafka client 2.1+ plugin. Provide WebFlux-WebClient plugin. Provide ignore-exception plugin. Provide quartz scheduler plugin. Provide ElasticJob 2.x plugin. Provide Spring @Scheduled plugin. Provide Spring-Kafka plugin. Provide HBase client plugin. Provide JSON log format. Move Spring WebFlux plugin to the optional plugin. Fix inconsistent logic bug in PrefixMatch Fix duplicate exit spans in Feign LoadBalancer mechanism. Fix the target service blocked by the Kafka reporter. Fix configurations of Kafka report don\u0026rsquo;t work. Fix rest template concurrent conflict. Fix NPE in the ActiveMQ plugin. Fix conflict between Kafka reporter and sampling plugin. Fix NPE in the log formatter. Fix span layer missing in certain cases, in the Kafka plugin. Fix error format of time in serviceTraffic update. Upgrade bytebuddy to 1.10.14 OAP-Backend Support Nacos authentication. Support labeled meter in the meter receiver. Separate UI template into multiple files. Provide support for Envoy tracing. Envoy tracer depends on the Envoy community. Support query trace by tags. Support composite alarm rules. Support alarm messages to DingTalk. Support alarm messages to WeChat. Support alarm messages to Slack. Support SSL for Prometheus fetcher and self telemetry. Support labeled histogram in the prometheus format. Support the status of segment based on entry span or first span only. Support the error segment in the sampling mechanism. Support SSL certs of gRPC server. Support labeled metrics in the alarm rule setting. Support to query all labeled data, if no explicit label in the query condition. Add TLS parameters in the mesh analysis. Add health check for InfluxDB storage. Add super dataset concept for the traces/logs. Add separate replicas configuration for super dataset. Add IN operator in the OAL. Add != operator in the OAL. Add like operator in the OAL. Add latest function in the prometheus analysis. Add more configurations in the gRPC server. Optimize the trace query performance. Optimize the CPU usage rate calculation, at least to be 1. Optimize the length of slow SQL column in the MySQL storage. Optimize the topology query, use client side component name when no server side mapping. Add component IDs for Python component. Add component ID range for C++. Fix Slack notification setting NPE. Fix some module missing check of the module manager core. Fix authentication doesn\u0026rsquo;t work in sharing server. Fix metrics batch persistent size bug. Fix trace sampling bug. Fix CLR receiver bug. Fix end time bug in the query process. Fix Exporter INCREMENT mode is not working. Fix an error when executing startup.bat when the log directory exists Add syncBulkActions configuration to set up the batch size of the metrics persistent. Meter Analysis Language. UI Add browser dashboard. Add browser log query page. Support query trace by tags. Fix JVM configuration. Fix CLR configuration. Document Add the document about SW_NO_UPSTREAM_REAL_ADDRESS. Update ALS setup document. Add Customization Config section for plugin development. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"820\"\u003e8.2.0\u003c/h2\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eSupport Browser monitoring.\u003c/li\u003e\n\u003cli\u003eAdd e2e test for ALS solution of service mesh …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.7.0/en/changes/changes-8.2.0/","title":"8.2.0"},{"body":"8.3.0 Project Test: ElasticSearch version 7.0.0 and 7.9.3 as storage are E2E tested. Test: Bump up testcontainers version to work around the Docker bug on MacOS. Java Agent Support propagate the sending timestamp in MQ plugins to calculate the transfer latency in the async MQ scenarios. Support auto-tag with the fixed values propagated in the correlation context. Make HttpClient 3.x, 4.x, and HttpAsyncClient 3.x plugins to support collecting HTTP parameters. Make the Feign plugin to support Java 14 Make the okhttp3 plugin to support Java 14 Polish tracing context related codes. Add the plugin for async-http-client 2.x Fix NPE in the nutz plugin. Provide Apache Commons DBCP 2.x plugin. Add the plugin for mssql-jtds 1.x. Add the plugin for mssql-jdbc 6.x -\u0026gt; 9.x. Fix the default ignore mechanism isn\u0026rsquo;t accurate enough bug. Add the plugin for spring-kafka 1.3.x. Add the plugin for Apache CXF 3.x. Fix okhttp-3.x and async-http-client-2.x did not overwrite the old trace header. OAP-Backend Add the @SuperDataset annotation for BrowserErrorLog. Add the thread pool to the Kafka fetcher to increase the performance. Add contain and not contain OPS in OAL. Add Envoy ALS analyzer based on metadata exchange. Add listMetrics GraphQL query. Add group name into services of so11y and istio relevant metrics Support keeping collecting the slowly segments in the sampling mechanism. Support choose files to active the meter analyzer. Support nested class definition in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Support sideCar.internalErrorCode in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Improve Kubernetes service registry for ALS analysis. Add health checker for cluster management Support the service auto grouping. Support query service list by the group name. Improve the queryable tags generation. Remove the duplicated tags to reduce the storage payload. Fix the threads of the Kafka fetcher exit if some unexpected exceptions happen. Fix the excessive timeout period set by the kubernetes-client. Fix deadlock problem when using elasticsearch-client-7.0.0. Fix storage-jdbc isExists not set dbname. Fix searchService bug in the InfluxDB storage implementation. Fix CVE in the alarm module, when activating the dynamic configuration feature. Fix CVE in the endpoint grouping, when activating the dynamic configuration feature. Fix CVE in the uninstrumented gateways configs, when activating the dynamic configuration feature. Fix CVE in the Apdex threshold configs, when activating the dynamic configuration feature. Make the codes and doc consistent in sharding server and core server. Fix that chunked string is incorrect while the tag contains colon. Fix the incorrect dynamic configuration key bug of endpoint-name-grouping. Remove unused min date timebucket in jdbc deletehistory logical Fix \u0026ldquo;transaction too large error\u0026rdquo; when use TiDB as storage. Fix \u0026ldquo;index not found\u0026rdquo; in trace query when use ES7 storage. Add otel rules to ui template to observe Istio control plane. Remove istio mixer Support close influxdb batch write model. Check SAN in the ALS (m)TLS process. UI Fix incorrect label in radial chart in topology. Replace node-sass with dart-sass. Replace serviceFilter with serviceGroup Removed \u0026ldquo;Les Miserables\u0026rdquo; from radial chart in topology. Add the Promise dropdown option Documentation Add VNode FAQ doc. Add logic endpoint section in the agent setup doc. Adjust configuration names and system environment names of the sharing server module Tweak Istio metrics collection doc. Add otel receiver. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"830\"\u003e8.3.0\u003c/h2\u003e\n\u003chr\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eTest: ElasticSearch version 7.0.0 and 7.9.3 as storage are E2E tested.\u003c/li\u003e\n\u003cli\u003eTest: Bump up …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/latest/en/changes/changes-8.3.0/","title":"8.3.0"},{"body":"8.3.0 Project Test: ElasticSearch version 7.0.0 and 7.9.3 as storage are E2E tested. Test: Bump up testcontainers version to work around the Docker bug on MacOS. Java Agent Support propagate the sending timestamp in MQ plugins to calculate the transfer latency in the async MQ scenarios. Support auto-tag with the fixed values propagated in the correlation context. Make HttpClient 3.x, 4.x, and HttpAsyncClient 3.x plugins to support collecting HTTP parameters. Make the Feign plugin to support Java 14 Make the okhttp3 plugin to support Java 14 Polish tracing context related codes. Add the plugin for async-http-client 2.x Fix NPE in the nutz plugin. Provide Apache Commons DBCP 2.x plugin. Add the plugin for mssql-jtds 1.x. Add the plugin for mssql-jdbc 6.x -\u0026gt; 9.x. Fix the default ignore mechanism isn\u0026rsquo;t accurate enough bug. Add the plugin for spring-kafka 1.3.x. Add the plugin for Apache CXF 3.x. Fix okhttp-3.x and async-http-client-2.x did not overwrite the old trace header. OAP-Backend Add the @SuperDataset annotation for BrowserErrorLog. Add the thread pool to the Kafka fetcher to increase the performance. Add contain and not contain OPS in OAL. Add Envoy ALS analyzer based on metadata exchange. Add listMetrics GraphQL query. Add group name into services of so11y and istio relevant metrics Support keeping collecting the slowly segments in the sampling mechanism. Support choose files to active the meter analyzer. Support nested class definition in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Support sideCar.internalErrorCode in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Improve Kubernetes service registry for ALS analysis. Add health checker for cluster management Support the service auto grouping. Support query service list by the group name. Improve the queryable tags generation. Remove the duplicated tags to reduce the storage payload. Fix the threads of the Kafka fetcher exit if some unexpected exceptions happen. Fix the excessive timeout period set by the kubernetes-client. Fix deadlock problem when using elasticsearch-client-7.0.0. Fix storage-jdbc isExists not set dbname. Fix searchService bug in the InfluxDB storage implementation. Fix CVE in the alarm module, when activating the dynamic configuration feature. Fix CVE in the endpoint grouping, when activating the dynamic configuration feature. Fix CVE in the uninstrumented gateways configs, when activating the dynamic configuration feature. Fix CVE in the Apdex threshold configs, when activating the dynamic configuration feature. Make the codes and doc consistent in sharding server and core server. Fix that chunked string is incorrect while the tag contains colon. Fix the incorrect dynamic configuration key bug of endpoint-name-grouping. Remove unused min date timebucket in jdbc deletehistory logical Fix \u0026ldquo;transaction too large error\u0026rdquo; when use TiDB as storage. Fix \u0026ldquo;index not found\u0026rdquo; in trace query when use ES7 storage. Add otel rules to ui template to observe Istio control plane. Remove istio mixer Support close influxdb batch write model. Check SAN in the ALS (m)TLS process. UI Fix incorrect label in radial chart in topology. Replace node-sass with dart-sass. Replace serviceFilter with serviceGroup Removed \u0026ldquo;Les Miserables\u0026rdquo; from radial chart in topology. Add the Promise dropdown option Documentation Add VNode FAQ doc. Add logic endpoint section in the agent setup doc. Adjust configuration names and system environment names of the sharing server module Tweak Istio metrics collection doc. Add otel receiver. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"830\"\u003e8.3.0\u003c/h2\u003e\n\u003chr\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eTest: ElasticSearch version 7.0.0 and 7.9.3 as storage are E2E tested.\u003c/li\u003e\n\u003cli\u003eTest: Bump up …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/next/en/changes/changes-8.3.0/","title":"8.3.0"},{"body":"8.3.0 Project Test: ElasticSearch version 7.0.0 and 7.9.3 as storage are E2E tested. Test: Bump up testcontainers version to work around the Docker bug on MacOS. Java Agent Support propagate the sending timestamp in MQ plugins to calculate the transfer latency in the async MQ scenarios. Support auto-tag with the fixed values propagated in the correlation context. Make HttpClient 3.x, 4.x, and HttpAsyncClient 3.x plugins to support collecting HTTP parameters. Make the Feign plugin to support Java 14 Make the okhttp3 plugin to support Java 14 Polish tracing context related codes. Add the plugin for async-http-client 2.x Fix NPE in the nutz plugin. Provide Apache Commons DBCP 2.x plugin. Add the plugin for mssql-jtds 1.x. Add the plugin for mssql-jdbc 6.x -\u0026gt; 9.x. Fix the default ignore mechanism isn\u0026rsquo;t accurate enough bug. Add the plugin for spring-kafka 1.3.x. Add the plugin for Apache CXF 3.x. Fix okhttp-3.x and async-http-client-2.x did not overwrite the old trace header. OAP-Backend Add the @SuperDataset annotation for BrowserErrorLog. Add the thread pool to the Kafka fetcher to increase the performance. Add contain and not contain OPS in OAL. Add Envoy ALS analyzer based on metadata exchange. Add listMetrics GraphQL query. Add group name into services of so11y and istio relevant metrics Support keeping collecting the slowly segments in the sampling mechanism. Support choose files to active the meter analyzer. Support nested class definition in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Support sideCar.internalErrorCode in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Improve Kubernetes service registry for ALS analysis. Add health checker for cluster management Support the service auto grouping. Support query service list by the group name. Improve the queryable tags generation. Remove the duplicated tags to reduce the storage payload. Fix the threads of the Kafka fetcher exit if some unexpected exceptions happen. Fix the excessive timeout period set by the kubernetes-client. Fix deadlock problem when using elasticsearch-client-7.0.0. Fix storage-jdbc isExists not set dbname. Fix searchService bug in the InfluxDB storage implementation. Fix CVE in the alarm module, when activating the dynamic configuration feature. Fix CVE in the endpoint grouping, when activating the dynamic configuration feature. Fix CVE in the uninstrumented gateways configs, when activating the dynamic configuration feature. Fix CVE in the Apdex threshold configs, when activating the dynamic configuration feature. Make the codes and doc consistent in sharding server and core server. Fix that chunked string is incorrect while the tag contains colon. Fix the incorrect dynamic configuration key bug of endpoint-name-grouping. Remove unused min date timebucket in jdbc deletehistory logical Fix \u0026ldquo;transaction too large error\u0026rdquo; when use TiDB as storage. Fix \u0026ldquo;index not found\u0026rdquo; in trace query when use ES7 storage. Add otel rules to ui template to observe Istio control plane. Remove istio mixer Support close influxdb batch write model. Check SAN in the ALS (m)TLS process. UI Fix incorrect label in radial chart in topology. Replace node-sass with dart-sass. Replace serviceFilter with serviceGroup Removed \u0026ldquo;Les Miserables\u0026rdquo; from radial chart in topology. Add the Promise dropdown option Documentation Add VNode FAQ doc. Add logic endpoint section in the agent setup doc. Adjust configuration names and system environment names of the sharing server module Tweak Istio metrics collection doc. Add otel receiver. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"830\"\u003e8.3.0\u003c/h2\u003e\n\u003chr\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eTest: ElasticSearch version 7.0.0 and 7.9.3 as storage are E2E tested.\u003c/li\u003e\n\u003cli\u003eTest: Bump up …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.0.0/en/changes/changes-8.3.0/","title":"8.3.0"},{"body":"8.3.0 Project Test: ElasticSearch version 7.0.0 and 7.9.3 as storage are E2E tested. Test: Bump up testcontainers version to work around the Docker bug on MacOS. Java Agent Support propagate the sending timestamp in MQ plugins to calculate the transfer latency in the async MQ scenarios. Support auto-tag with the fixed values propagated in the correlation context. Make HttpClient 3.x, 4.x, and HttpAsyncClient 3.x plugins to support collecting HTTP parameters. Make the Feign plugin to support Java 14 Make the okhttp3 plugin to support Java 14 Polish tracing context related codes. Add the plugin for async-http-client 2.x Fix NPE in the nutz plugin. Provide Apache Commons DBCP 2.x plugin. Add the plugin for mssql-jtds 1.x. Add the plugin for mssql-jdbc 6.x -\u0026gt; 9.x. Fix the default ignore mechanism isn\u0026rsquo;t accurate enough bug. Add the plugin for spring-kafka 1.3.x. Add the plugin for Apache CXF 3.x. Fix okhttp-3.x and async-http-client-2.x did not overwrite the old trace header. OAP-Backend Add the @SuperDataset annotation for BrowserErrorLog. Add the thread pool to the Kafka fetcher to increase the performance. Add contain and not contain OPS in OAL. Add Envoy ALS analyzer based on metadata exchange. Add listMetrics GraphQL query. Add group name into services of so11y and istio relevant metrics Support keeping collecting the slowly segments in the sampling mechanism. Support choose files to active the meter analyzer. Support nested class definition in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Support sideCar.internalErrorCode in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Improve Kubernetes service registry for ALS analysis. Add health checker for cluster management Support the service auto grouping. Support query service list by the group name. Improve the queryable tags generation. Remove the duplicated tags to reduce the storage payload. Fix the threads of the Kafka fetcher exit if some unexpected exceptions happen. Fix the excessive timeout period set by the kubernetes-client. Fix deadlock problem when using elasticsearch-client-7.0.0. Fix storage-jdbc isExists not set dbname. Fix searchService bug in the InfluxDB storage implementation. Fix CVE in the alarm module, when activating the dynamic configuration feature. Fix CVE in the endpoint grouping, when activating the dynamic configuration feature. Fix CVE in the uninstrumented gateways configs, when activating the dynamic configuration feature. Fix CVE in the Apdex threshold configs, when activating the dynamic configuration feature. Make the codes and doc consistent in sharding server and core server. Fix that chunked string is incorrect while the tag contains colon. Fix the incorrect dynamic configuration key bug of endpoint-name-grouping. Remove unused min date timebucket in jdbc deletehistory logical Fix \u0026ldquo;transaction too large error\u0026rdquo; when use TiDB as storage. Fix \u0026ldquo;index not found\u0026rdquo; in trace query when use ES7 storage. Add otel rules to ui template to observe Istio control plane. Remove istio mixer Support close influxdb batch write model. Check SAN in the ALS (m)TLS process. UI Fix incorrect label in radial chart in topology. Replace node-sass with dart-sass. Replace serviceFilter with serviceGroup Removed \u0026ldquo;Les Miserables\u0026rdquo; from radial chart in topology. Add the Promise dropdown option Documentation Add VNode FAQ doc. Add logic endpoint section in the agent setup doc. Adjust configuration names and system environment names of the sharing server module Tweak Istio metrics collection doc. Add otel receiver. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"830\"\u003e8.3.0\u003c/h2\u003e\n\u003chr\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eTest: ElasticSearch version 7.0.0 and 7.9.3 as storage are E2E tested.\u003c/li\u003e\n\u003cli\u003eTest: Bump up …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.0.1/en/changes/changes-8.3.0/","title":"8.3.0"},{"body":"8.3.0 Project Test: ElasticSearch version 7.0.0 and 7.9.3 as storage are E2E tested. Test: Bump up testcontainers version to work around the Docker bug on MacOS. Java Agent Support propagate the sending timestamp in MQ plugins to calculate the transfer latency in the async MQ scenarios. Support auto-tag with the fixed values propagated in the correlation context. Make HttpClient 3.x, 4.x, and HttpAsyncClient 3.x plugins to support collecting HTTP parameters. Make the Feign plugin to support Java 14 Make the okhttp3 plugin to support Java 14 Polish tracing context related codes. Add the plugin for async-http-client 2.x Fix NPE in the nutz plugin. Provide Apache Commons DBCP 2.x plugin. Add the plugin for mssql-jtds 1.x. Add the plugin for mssql-jdbc 6.x -\u0026gt; 9.x. Fix the default ignore mechanism isn\u0026rsquo;t accurate enough bug. Add the plugin for spring-kafka 1.3.x. Add the plugin for Apache CXF 3.x. Fix okhttp-3.x and async-http-client-2.x did not overwrite the old trace header. OAP-Backend Add the @SuperDataset annotation for BrowserErrorLog. Add the thread pool to the Kafka fetcher to increase the performance. Add contain and not contain OPS in OAL. Add Envoy ALS analyzer based on metadata exchange. Add listMetrics GraphQL query. Add group name into services of so11y and istio relevant metrics Support keeping collecting the slowly segments in the sampling mechanism. Support choose files to active the meter analyzer. Support nested class definition in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Support sideCar.internalErrorCode in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Improve Kubernetes service registry for ALS analysis. Add health checker for cluster management Support the service auto grouping. Support query service list by the group name. Improve the queryable tags generation. Remove the duplicated tags to reduce the storage payload. Fix the threads of the Kafka fetcher exit if some unexpected exceptions happen. Fix the excessive timeout period set by the kubernetes-client. Fix deadlock problem when using elasticsearch-client-7.0.0. Fix storage-jdbc isExists not set dbname. Fix searchService bug in the InfluxDB storage implementation. Fix CVE in the alarm module, when activating the dynamic configuration feature. Fix CVE in the endpoint grouping, when activating the dynamic configuration feature. Fix CVE in the uninstrumented gateways configs, when activating the dynamic configuration feature. Fix CVE in the Apdex threshold configs, when activating the dynamic configuration feature. Make the codes and doc consistent in sharding server and core server. Fix that chunked string is incorrect while the tag contains colon. Fix the incorrect dynamic configuration key bug of endpoint-name-grouping. Remove unused min date timebucket in jdbc deletehistory logical Fix \u0026ldquo;transaction too large error\u0026rdquo; when use TiDB as storage. Fix \u0026ldquo;index not found\u0026rdquo; in trace query when use ES7 storage. Add otel rules to ui template to observe Istio control plane. Remove istio mixer Support close influxdb batch write model. Check SAN in the ALS (m)TLS process. UI Fix incorrect label in radial chart in topology. Replace node-sass with dart-sass. Replace serviceFilter with serviceGroup Removed \u0026ldquo;Les Miserables\u0026rdquo; from radial chart in topology. Add the Promise dropdown option Documentation Add VNode FAQ doc. Add logic endpoint section in the agent setup doc. Adjust configuration names and system environment names of the sharing server module Tweak Istio metrics collection doc. Add otel receiver. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"830\"\u003e8.3.0\u003c/h2\u003e\n\u003chr\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eTest: ElasticSearch version 7.0.0 and 7.9.3 as storage are E2E tested.\u003c/li\u003e\n\u003cli\u003eTest: Bump up …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.1.0/en/changes/changes-8.3.0/","title":"8.3.0"},{"body":"8.3.0 Project Test: ElasticSearch version 7.0.0 and 7.9.3 as storage are E2E tested. Test: Bump up testcontainers version to work around the Docker bug on MacOS. Java Agent Support propagate the sending timestamp in MQ plugins to calculate the transfer latency in the async MQ scenarios. Support auto-tag with the fixed values propagated in the correlation context. Make HttpClient 3.x, 4.x, and HttpAsyncClient 3.x plugins to support collecting HTTP parameters. Make the Feign plugin to support Java 14 Make the okhttp3 plugin to support Java 14 Polish tracing context related codes. Add the plugin for async-http-client 2.x Fix NPE in the nutz plugin. Provide Apache Commons DBCP 2.x plugin. Add the plugin for mssql-jtds 1.x. Add the plugin for mssql-jdbc 6.x -\u0026gt; 9.x. Fix the default ignore mechanism isn\u0026rsquo;t accurate enough bug. Add the plugin for spring-kafka 1.3.x. Add the plugin for Apache CXF 3.x. Fix okhttp-3.x and async-http-client-2.x did not overwrite the old trace header. OAP-Backend Add the @SuperDataset annotation for BrowserErrorLog. Add the thread pool to the Kafka fetcher to increase the performance. Add contain and not contain OPS in OAL. Add Envoy ALS analyzer based on metadata exchange. Add listMetrics GraphQL query. Add group name into services of so11y and istio relevant metrics Support keeping collecting the slowly segments in the sampling mechanism. Support choose files to active the meter analyzer. Support nested class definition in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Support sideCar.internalErrorCode in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Improve Kubernetes service registry for ALS analysis. Add health checker for cluster management Support the service auto grouping. Support query service list by the group name. Improve the queryable tags generation. Remove the duplicated tags to reduce the storage payload. Fix the threads of the Kafka fetcher exit if some unexpected exceptions happen. Fix the excessive timeout period set by the kubernetes-client. Fix deadlock problem when using elasticsearch-client-7.0.0. Fix storage-jdbc isExists not set dbname. Fix searchService bug in the InfluxDB storage implementation. Fix CVE in the alarm module, when activating the dynamic configuration feature. Fix CVE in the endpoint grouping, when activating the dynamic configuration feature. Fix CVE in the uninstrumented gateways configs, when activating the dynamic configuration feature. Fix CVE in the Apdex threshold configs, when activating the dynamic configuration feature. Make the codes and doc consistent in sharding server and core server. Fix that chunked string is incorrect while the tag contains colon. Fix the incorrect dynamic configuration key bug of endpoint-name-grouping. Remove unused min date timebucket in jdbc deletehistory logical Fix \u0026ldquo;transaction too large error\u0026rdquo; when use TiDB as storage. Fix \u0026ldquo;index not found\u0026rdquo; in trace query when use ES7 storage. Add otel rules to ui template to observe Istio control plane. Remove istio mixer Support close influxdb batch write model. Check SAN in the ALS (m)TLS process. UI Fix incorrect label in radial chart in topology. Replace node-sass with dart-sass. Replace serviceFilter with serviceGroup Removed \u0026ldquo;Les Miserables\u0026rdquo; from radial chart in topology. Add the Promise dropdown option Documentation Add VNode FAQ doc. Add logic endpoint section in the agent setup doc. Adjust configuration names and system environment names of the sharing server module Tweak Istio metrics collection doc. Add otel receiver. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"830\"\u003e8.3.0\u003c/h2\u003e\n\u003chr\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eTest: ElasticSearch version 7.0.0 and 7.9.3 as storage are E2E tested.\u003c/li\u003e\n\u003cli\u003eTest: Bump up …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.2.0/en/changes/changes-8.3.0/","title":"8.3.0"},{"body":"8.3.0 Project Test: ElasticSearch version 7.0.0 and 7.9.3 as storage are E2E tested. Test: Bump up testcontainers version to work around the Docker bug on MacOS. Java Agent Support propagate the sending timestamp in MQ plugins to calculate the transfer latency in the async MQ scenarios. Support auto-tag with the fixed values propagated in the correlation context. Make HttpClient 3.x, 4.x, and HttpAsyncClient 3.x plugins to support collecting HTTP parameters. Make the Feign plugin to support Java 14 Make the okhttp3 plugin to support Java 14 Polish tracing context related codes. Add the plugin for async-http-client 2.x Fix NPE in the nutz plugin. Provide Apache Commons DBCP 2.x plugin. Add the plugin for mssql-jtds 1.x. Add the plugin for mssql-jdbc 6.x -\u0026gt; 9.x. Fix the default ignore mechanism isn\u0026rsquo;t accurate enough bug. Add the plugin for spring-kafka 1.3.x. Add the plugin for Apache CXF 3.x. Fix okhttp-3.x and async-http-client-2.x did not overwrite the old trace header. OAP-Backend Add the @SuperDataset annotation for BrowserErrorLog. Add the thread pool to the Kafka fetcher to increase the performance. Add contain and not contain OPS in OAL. Add Envoy ALS analyzer based on metadata exchange. Add listMetrics GraphQL query. Add group name into services of so11y and istio relevant metrics Support keeping collecting the slowly segments in the sampling mechanism. Support choose files to active the meter analyzer. Support nested class definition in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Support sideCar.internalErrorCode in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Improve Kubernetes service registry for ALS analysis. Add health checker for cluster management Support the service auto grouping. Support query service list by the group name. Improve the queryable tags generation. Remove the duplicated tags to reduce the storage payload. Fix the threads of the Kafka fetcher exit if some unexpected exceptions happen. Fix the excessive timeout period set by the kubernetes-client. Fix deadlock problem when using elasticsearch-client-7.0.0. Fix storage-jdbc isExists not set dbname. Fix searchService bug in the InfluxDB storage implementation. Fix CVE in the alarm module, when activating the dynamic configuration feature. Fix CVE in the endpoint grouping, when activating the dynamic configuration feature. Fix CVE in the uninstrumented gateways configs, when activating the dynamic configuration feature. Fix CVE in the Apdex threshold configs, when activating the dynamic configuration feature. Make the codes and doc consistent in sharding server and core server. Fix that chunked string is incorrect while the tag contains colon. Fix the incorrect dynamic configuration key bug of endpoint-name-grouping. Remove unused min date timebucket in jdbc deletehistory logical Fix \u0026ldquo;transaction too large error\u0026rdquo; when use TiDB as storage. Fix \u0026ldquo;index not found\u0026rdquo; in trace query when use ES7 storage. Add otel rules to ui template to observe Istio control plane. Remove istio mixer Support close influxdb batch write model. Check SAN in the ALS (m)TLS process. UI Fix incorrect label in radial chart in topology. Replace node-sass with dart-sass. Replace serviceFilter with serviceGroup Removed \u0026ldquo;Les Miserables\u0026rdquo; from radial chart in topology. Add the Promise dropdown option Documentation Add VNode FAQ doc. Add logic endpoint section in the agent setup doc. Adjust configuration names and system environment names of the sharing server module Tweak Istio metrics collection doc. Add otel receiver. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"830\"\u003e8.3.0\u003c/h2\u003e\n\u003chr\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eTest: ElasticSearch version 7.0.0 and 7.9.3 as storage are E2E tested.\u003c/li\u003e\n\u003cli\u003eTest: Bump up …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.3.0/en/changes/changes-8.3.0/","title":"8.3.0"},{"body":"8.3.0 Project Test: ElasticSearch version 7.0.0 and 7.9.3 as storage are E2E tested. Test: Bump up testcontainers version to work around the Docker bug on MacOS. Java Agent Support propagate the sending timestamp in MQ plugins to calculate the transfer latency in the async MQ scenarios. Support auto-tag with the fixed values propagated in the correlation context. Make HttpClient 3.x, 4.x, and HttpAsyncClient 3.x plugins to support collecting HTTP parameters. Make the Feign plugin to support Java 14 Make the okhttp3 plugin to support Java 14 Polish tracing context related codes. Add the plugin for async-http-client 2.x Fix NPE in the nutz plugin. Provide Apache Commons DBCP 2.x plugin. Add the plugin for mssql-jtds 1.x. Add the plugin for mssql-jdbc 6.x -\u0026gt; 9.x. Fix the default ignore mechanism isn\u0026rsquo;t accurate enough bug. Add the plugin for spring-kafka 1.3.x. Add the plugin for Apache CXF 3.x. Fix okhttp-3.x and async-http-client-2.x did not overwrite the old trace header. OAP-Backend Add the @SuperDataset annotation for BrowserErrorLog. Add the thread pool to the Kafka fetcher to increase the performance. Add contain and not contain OPS in OAL. Add Envoy ALS analyzer based on metadata exchange. Add listMetrics GraphQL query. Add group name into services of so11y and istio relevant metrics Support keeping collecting the slowly segments in the sampling mechanism. Support choose files to active the meter analyzer. Support nested class definition in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Support sideCar.internalErrorCode in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Improve Kubernetes service registry for ALS analysis. Add health checker for cluster management Support the service auto grouping. Support query service list by the group name. Improve the queryable tags generation. Remove the duplicated tags to reduce the storage payload. Fix the threads of the Kafka fetcher exit if some unexpected exceptions happen. Fix the excessive timeout period set by the kubernetes-client. Fix deadlock problem when using elasticsearch-client-7.0.0. Fix storage-jdbc isExists not set dbname. Fix searchService bug in the InfluxDB storage implementation. Fix CVE in the alarm module, when activating the dynamic configuration feature. Fix CVE in the endpoint grouping, when activating the dynamic configuration feature. Fix CVE in the uninstrumented gateways configs, when activating the dynamic configuration feature. Fix CVE in the Apdex threshold configs, when activating the dynamic configuration feature. Make the codes and doc consistent in sharding server and core server. Fix that chunked string is incorrect while the tag contains colon. Fix the incorrect dynamic configuration key bug of endpoint-name-grouping. Remove unused min date timebucket in jdbc deletehistory logical Fix \u0026ldquo;transaction too large error\u0026rdquo; when use TiDB as storage. Fix \u0026ldquo;index not found\u0026rdquo; in trace query when use ES7 storage. Add otel rules to ui template to observe Istio control plane. Remove istio mixer Support close influxdb batch write model. Check SAN in the ALS (m)TLS process. UI Fix incorrect label in radial chart in topology. Replace node-sass with dart-sass. Replace serviceFilter with serviceGroup Removed \u0026ldquo;Les Miserables\u0026rdquo; from radial chart in topology. Add the Promise dropdown option Documentation Add VNode FAQ doc. Add logic endpoint section in the agent setup doc. Adjust configuration names and system environment names of the sharing server module Tweak Istio metrics collection doc. Add otel receiver. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"830\"\u003e8.3.0\u003c/h2\u003e\n\u003chr\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eTest: ElasticSearch version 7.0.0 and 7.9.3 as storage are E2E tested.\u003c/li\u003e\n\u003cli\u003eTest: Bump up …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v10.4.0/en/changes/changes-8.3.0/","title":"8.3.0"},{"body":"8.3.0 Project Test: ElasticSearch version 7.0.0 and 7.9.3 as storage are E2E tested. Test: Bump up testcontainers version to work around the Docker bug on MacOS. Java Agent Support propagate the sending timestamp in MQ plugins to calculate the transfer latency in the async MQ scenarios. Support auto-tag with the fixed values propagated in the correlation context. Make HttpClient 3.x, 4.x, and HttpAsyncClient 3.x plugins to support collecting HTTP parameters. Make the Feign plugin to support Java 14 Make the okhttp3 plugin to support Java 14 Polish tracing context related codes. Add the plugin for async-http-client 2.x Fix NPE in the nutz plugin. Provide Apache Commons DBCP 2.x plugin. Add the plugin for mssql-jtds 1.x. Add the plugin for mssql-jdbc 6.x -\u0026gt; 9.x. Fix the default ignore mechanism isn\u0026rsquo;t accurate enough bug. Add the plugin for spring-kafka 1.3.x. Add the plugin for Apache CXF 3.x. Fix okhttp-3.x and async-http-client-2.x did not overwrite the old trace header. OAP-Backend Add the @SuperDataset annotation for BrowserErrorLog. Add the thread pool to the Kafka fetcher to increase the performance. Add contain and not contain OPS in OAL. Add Envoy ALS analyzer based on metadata exchange. Add listMetrics GraphQL query. Add group name into services of so11y and istio relevant metrics Support keeping collecting the slowly segments in the sampling mechanism. Support choose files to active the meter analyzer. Support nested class definition in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Support sideCar.internalErrorCode in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Improve Kubernetes service registry for ALS analysis. Add health checker for cluster management Support the service auto grouping. Support query service list by the group name. Improve the queryable tags generation. Remove the duplicated tags to reduce the storage payload. Fix the threads of the Kafka fetcher exit if some unexpected exceptions happen. Fix the excessive timeout period set by the kubernetes-client. Fix deadlock problem when using elasticsearch-client-7.0.0. Fix storage-jdbc isExists not set dbname. Fix searchService bug in the InfluxDB storage implementation. Fix CVE in the alarm module, when activating the dynamic configuration feature. Fix CVE in the endpoint grouping, when activating the dynamic configuration feature. Fix CVE in the uninstrumented gateways configs, when activating the dynamic configuration feature. Fix CVE in the Apdex threshold configs, when activating the dynamic configuration feature. Make the codes and doc consistent in sharding server and core server. Fix that chunked string is incorrect while the tag contains colon. Fix the incorrect dynamic configuration key bug of endpoint-name-grouping. Remove unused min date timebucket in jdbc deletehistory logical Fix \u0026ldquo;transaction too large error\u0026rdquo; when use TiDB as storage. Fix \u0026ldquo;index not found\u0026rdquo; in trace query when use ES7 storage. Add otel rules to ui template to observe Istio control plane. Remove istio mixer Support close influxdb batch write model. Check SAN in the ALS (m)TLS process. UI Fix incorrect label in radial chart in topology. Replace node-sass with dart-sass. Replace serviceFilter with serviceGroup Removed \u0026ldquo;Les Miserables\u0026rdquo; from radial chart in topology. Add the Promise dropdown option Documentation Add VNode FAQ doc. Add logic endpoint section in the agent setup doc. Adjust configuration names and system environment names of the sharing server module Tweak Istio metrics collection doc. Add otel receiver. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"830\"\u003e8.3.0\u003c/h2\u003e\n\u003chr\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eTest: ElasticSearch version 7.0.0 and 7.9.3 as storage are E2E tested.\u003c/li\u003e\n\u003cli\u003eTest: Bump up …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v11.0.0/en/changes/changes-8.3.0/","title":"8.3.0"},{"body":"8.3.0 Project Test: ElasticSearch version 7.0.0 and 7.9.3 as storage are E2E tested. Test: Bump up testcontainers version to work around the Docker bug on MacOS. Java Agent Support propagate the sending timestamp in MQ plugins to calculate the transfer latency in the async MQ scenarios. Support auto-tag with the fixed values propagated in the correlation context. Make HttpClient 3.x, 4.x, and HttpAsyncClient 3.x plugins to support collecting HTTP parameters. Make the Feign plugin to support Java 14 Make the okhttp3 plugin to support Java 14 Polish tracing context related codes. Add the plugin for async-http-client 2.x Fix NPE in the nutz plugin. Provide Apache Commons DBCP 2.x plugin. Add the plugin for mssql-jtds 1.x. Add the plugin for mssql-jdbc 6.x -\u0026gt; 9.x. Fix the default ignore mechanism isn\u0026rsquo;t accurate enough bug. Add the plugin for spring-kafka 1.3.x. Add the plugin for Apache CXF 3.x. Fix okhttp-3.x and async-http-client-2.x did not overwrite the old trace header. OAP-Backend Add the @SuperDataset annotation for BrowserErrorLog. Add the thread pool to the Kafka fetcher to increase the performance. Add contain and not contain OPS in OAL. Add Envoy ALS analyzer based on metadata exchange. Add listMetrics GraphQL query. Add group name into services of so11y and istio relevant metrics Support keeping collecting the slowly segments in the sampling mechanism. Support choose files to active the meter analyzer. Support nested class definition in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Support sideCar.internalErrorCode in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Improve Kubernetes service registry for ALS analysis. Add health checker for cluster management Support the service auto grouping. Support query service list by the group name. Improve the queryable tags generation. Remove the duplicated tags to reduce the storage payload. Fix the threads of the Kafka fetcher exit if some unexpected exceptions happen. Fix the excessive timeout period set by the kubernetes-client. Fix deadlock problem when using elasticsearch-client-7.0.0. Fix storage-jdbc isExists not set dbname. Fix searchService bug in the InfluxDB storage implementation. Fix CVE in the alarm module, when activating the dynamic configuration feature. Fix CVE in the endpoint grouping, when activating the dynamic configuration feature. Fix CVE in the uninstrumented gateways configs, when activating the dynamic configuration feature. Fix CVE in the Apdex threshold configs, when activating the dynamic configuration feature. Make the codes and doc consistent in sharding server and core server. Fix that chunked string is incorrect while the tag contains colon. Fix the incorrect dynamic configuration key bug of endpoint-name-grouping. Remove unused min date timebucket in jdbc deletehistory logical Fix \u0026ldquo;transaction too large error\u0026rdquo; when use TiDB as storage. Fix \u0026ldquo;index not found\u0026rdquo; in trace query when use ES7 storage. Add otel rules to ui template to observe Istio control plane. Remove istio mixer Support close influxdb batch write model. Check SAN in the ALS (m)TLS process. UI Fix incorrect label in radial chart in topology. Replace node-sass with dart-sass. Replace serviceFilter with serviceGroup Removed \u0026ldquo;Les Miserables\u0026rdquo; from radial chart in topology. Add the Promise dropdown option Documentation Add VNode FAQ doc. Add logic endpoint section in the agent setup doc. Adjust configuration names and system environment names of the sharing server module Tweak Istio metrics collection doc. Add otel receiver. All issues and pull requests are here\n","excerpt":"\u003ch2 id=\"830\"\u003e8.3.0\u003c/h2\u003e\n\u003chr\u003e\n\u003ch4 id=\"project\"\u003eProject\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eTest: ElasticSearch version 7.0.0 and 7.9.3 as storage are E2E tested.\u003c/li\u003e\n\u003cli\u003eTest: Bump up …\u003c/li\u003e\u003c/ul\u003e","ref":"https://skywalking.apache.org/docs/main/v9.4.0/en/changes/changes-8.3.0/","title":"8.3.0"},{"body":"8.3.0 Project Test: ElasticSearch version 7.0.0 and 7.9.3 as storage are E2E tested. Test: Bump up testcontainers version to work around the Docker bug on MacOS. Java Agent Support propagate the sending timestamp in MQ plugins to calculate the transfer latency in the async MQ scenarios. Support auto-tag with the fixed values propagated in the correlation context. Make HttpClient 3.x, 4.x, and HttpAsyncClient 3.x plugins to support collecting HTTP parameters. Make the Feign plugin to support Java 14 Make the okhttp3 plugin to support Java 14 Polish tracing context related codes. Add the plugin for async-http-client 2.x Fix NPE in the nutz plugin. Provide Apache Commons DBCP 2.x plugin. Add the plugin for mssql-jtds 1.x. Add the plugin for mssql-jdbc 6.x -\u0026gt; 9.x. Fix the default ignore mechanism isn\u0026rsquo;t accurate enough bug. Add the plugin for spring-kafka 1.3.x. Add the plugin for Apache CXF 3.x. Fix okhttp-3.x and async-http-client-2.x did not overwrite the old trace header. OAP-Backend Add the @SuperDataset annotation for BrowserErrorLog. Add the thread pool to the Kafka fetcher to increase the performance. Add contain and not contain OPS in OAL. Add Envoy ALS analyzer based on metadata exchange. Add listMetrics GraphQL query. Add group name into services of so11y and istio relevant metrics Support keeping collecting the slowly segments in the sampling mechanism. Support choose files to active the meter analyzer. Support nested class definition in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Support sideCar.internalErrorCode in the Service, ServiceInstance, Endpoint, ServiceRelation, and ServiceInstanceRelation sources. Improve Kubernetes service registry for ALS analysis. Add health checker for cluster management Support the service auto grouping. Support query service list by the group name. Improve the queryable tags generation. Remove the duplicated tags to reduce the storage payload. Fix the threads of the Kafka fetcher exit if some unexpected exceptions happen. Fix the excessive timeout period set by the kubernetes-client. Fix deadlock problem when using elasticsearch-client-7.