Native Trace Analysis

The trace-analyzer module analyzes the segments SkyWalking agents send with TAL rules for native segments. The shipped rules build the services, instances, endpoints, topology, virtual database, cache, MQ and GenAI nodes, slow statements, network address aliases and stored traces of the agent path, the sources core.oal and the other .oal files compute every agent metric from. See SkyWalking Native Traces for how the segments arrive and the modules they need.

Configuration

trace-analyzer:
  selector: ${SW_TRACE_ANALYZER:default}
  default:
    # TAL rule files under `tal/` for spans that SkyWalking agents send, without extension, comma separated, in the
    # order their rules run. The default reproduces the agent path.
    nativeTalFiles: ${SW_TRACE_NATIVE_TAL_FILES:native-rpc,native-endpoint-cross-thread,native-network-alias,native-segment,native-virtual-service,native-virtual-genai,native-ai-evaluation}

The rule files live under config/tal/ of the distribution, beside the OpenTelemetry ones, and are read when the OAP starts. A listed file that does not exist, a rule that does not compile, or two native rules of one name stop the OAP from starting, with a message that names the missing file, the rule that failed to compile, or the repeated rule name. nativeTalFiles is separate from talFiles, so an operator who lists their own OpenTelemetry files keeps the native analysis.

The other settings the rules read belong to agent-analyzer: trace sampling, the slow database statement and slow cache command thresholds, maxSlowSQLLength, segmentStatusAnalysisStrategy, forceSampleErrorSegment, noUpstreamRealAddressAgents and traceAnalysis; the core’s searchableTracesTags, endpoint grouping and name length limits; and the uninstrumented gateways and server clusters files.

The shipped rule files

native-virtual-genai lists gen-ai-analyzer under requiredModules, and native-ai-evaluation lists ai-evaluation: a file whose module is not loaded is not compiled, and its rules do not run. ai-evaluation is off by default, and when it is loaded without a complete judge configuration it samples nothing.

traceAnalysis: false in agent-analyzer turns off native-rpc, native-endpoint-cross-thread and native-network-alias, whose rules abort on every span; the other files ignore it, so the traces are still stored and the virtual nodes still built.

The order of the files

The order of nativeTalFiles is part of the analysis. No rule reads another rule’s output: the one cross-file write, an alias row, reaches native-rpc-exit only through the alias cache, once the cache reloads. The order still decides what reaches an order-sensitive merge first, what a failing segment has already sent, and the order of the stateful naming calls:

Within native-rpc the rules run entry, exit, logic endpoint; within native-virtual-service cache, database, MQ.

Adding a rule file

Put a file of your own under config/tal/, declare inputType: sw-native on each rule, and append its name to nativeTalFiles. Appended at the end, it keeps the shipped rules in their relative order. Its rules still run on every span of every segment, so one of them that throws stops the segment before the build phase, and one that formats endpoint names changes the endpoint grouping’s state that later spans and the shipped build blocks read. Moving or removing a shipped file changes what the agent path produces: a metric whose source no rule sends any more stays empty.

Logging

A rule’s log statements and the functions that log for it write under the script’s logger, org.apache.skywalking.trace.analysis.script.<file>; the segment-level lines, the unknown span type error and the dropped long tags, under the runner’s, org.apache.skywalking.oap.trace.analyzer.v2.provider.nativetrace.SwNativeRuleRunner. The naming, the endpoint grouping, the gateway lookup and the other Java the rules call still log under their own classes. A logging configuration that named one of the Java listeners that analyzed segments before, in org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener, or TraceAnalyzer one package up, now names the logger that writes its lines:

GenAIMeterAnalyzer keeps logging under its own name for OpenTelemetry and Zipkin spans.

Startup

trace-analyzer requires agent-analyzer and every module a listed file names under requiredModules, when they are loaded, so they start before it. It compiles the native rules in its start(), and the trace receiver and the Kafka fetcher, which require it, start after. Removing the module with SW_TRACE_ANALYZER=- while the trace receiver (loaded by default) or the Kafka fetcher is loaded fails the boot, naming the missing module; with neither loaded, removing it still works for OpenTelemetry alone, see Turning the analysis off.