PromQL Service

PromQL(Prometheus Query Language) Service exposes Prometheus Querying HTTP APIs including the bundled PromQL expression system. Third-party systems or visualization platforms that already support PromQL (such as Grafana), could obtain metrics through PromQL Service.

As SkyWalking and Prometheus have fundamental differences in metrics classification, format, storage, etc. The PromQL Service supported will be a subset of the complete PromQL.

Details Of Supported Protocol

The following doc describes the details of the supported protocol and compared it to the PromQL official documentation. If not mentioned, it will not be supported by default.

Time series Selectors

Instant Vector Selectors

Prometheus Docs Reference

For example: select metric service_cpm which the service is $service and the layer is $layer.

service_cpm{service='$service', layer='$layer'}

Note: The label matching operators only support = instead of regular expressions.

Range Vector Selectors

Prometheus Docs Reference

For example: select metric service_cpm which the service is $service and the layer is $layer within the last 5 minutes.

service_cpm{service='$service', layer='$layer'}[5m]

Time Durations

Prometheus Docs Reference

Unit Definition Support
ms milliseconds yes
s seconds yes
m minutes yes
h hours yes
d days yes
w weeks yes
y years no

Binary operators

Arithmetic binary operators

Prometheus Docs Reference

Operator Definition Support
+ addition yes
- subtraction yes
* multiplication yes
/ division yes
% modulo yes
^ power/exponentiation no
Between two scalars

For example:

1 + 2
Between an instant vector and a scalar

For example:

service_cpm{service='$service', layer='$layer'} / 100
Between two instant vectors

For example:

service_cpm{service='$service', layer='$layer'} + service_cpm{service='$service', layer='$layer'}

Note: The operations between vectors require the same metric and labels, and don’t support Vector matching.

Comparison binary operators

Prometheus Docs Reference

Operator Definition Support
== equal yes
!= not-equal yes
> greater-than yes
< less-than yes
>= greater-or-equal yes
<= less-or-equal) yes
Between two scalars

For example:

1 > bool 2
Between an instant vector and a scalar

For example:

service_cpm{service='$service', layer='$layer'} > 1
Between two instant vectors

For example:

service_cpm{service='service_A', layer='$layer'} > service_cpm{service='service_B', layer='$layer'}

Aggregation operators

Prometheus Docs Reference

Operator Definition Support
sum calculate sum over dimensions yes
min select minimum over dimensions yes
max select maximum over dimensions yes
avg calculate the average over dimensions yes

For example:

If the metric http_requests_total had time series that fan out by service, service_instance_id, and group labels, we could calculate the total number of seen HTTP requests per service and group over all service instances via:

sum by (service, group) (http_requests_total{service='$service', layer='$layer'})

Which is equivalent to:

sum without (service_instance_id) (http_requests_total{service='$service', layer='$layer'})

If we are just interested in the total of HTTP requests we have seen in all services, we could simply write:

sum(http_requests_total{service='$service', layer='$layer'})

HTTP API

Expression queries

Instant queries

Prometheus Docs Reference

GET|POST /api/v1/query
Parameter Definition Support Optional
query prometheus expression yes no
time The latest metrics value from current time to this time is returned. If time is empty, the default look-back time is 2 minutes. time format: RFC3399 or unix_timestamp in seconds yes yes
timeout evaluation timeout no ignore

For example:

/api/v1/query?query=service_cpm{service='agent::songs', layer='GENERAL'}

Result:

{
    "status": "success",
    "data": {
        "resultType": "vector",
        "result": [
            {
                "metric": {
                    "__name__": "service_cpm",
                    "layer": "GENERAL",
                    "scope": "Service",
                    "service": "agent::songs"
                },
                "value": [
                    1677548400,
                    "6"
                ]
            }
        ]
    }
}

We can also use Range Vector Selectors in the instant query.

/api/v1/query?query=service_cpm{service='agent::songs', layer='GENERAL'}[5m]

the result is the same as the Range queries.

Range queries

Prometheus Docs Reference

GET|POST /api/v1/query_range
Parameter Definition Support Optional
query prometheus expression yes no
start start timestamp, format: RFC3399 or unix_timestamp in seconds yes no
end end timestamp, format: RFC3399 or unix_timestamp in seconds yes no
step SkyWalking will automatically fit Step(DAY, HOUR, MINUTE) through start and end. no ignore
timeout evaluation timeout no ignore

For example:

/api/v1/query_range?query=service_cpm{service='agent::songs', layer='GENERAL'}&start=1677479336&end=1677479636

Result:

{
    "status": "success",
    "data": {
        "resultType": "matrix",
        "result": [
            {
                "metric": {
                    "__name__": "service_cpm",
                    "layer": "GENERAL",
                    "scope": "Service",
                    "service": "agent::songs"
                },
                "values": [
                    [
                        1677479280,
                        "18"
                    ],
                    [
                        1677479340,
                        "18"
                    ],
                    [
                        1677479400,
                        "18"
                    ],
                    [
                        1677479460,
                        "18"
                    ],
                    [
                        1677479520,
                        "18"
                    ],
                    [
                        1677479580,
                        "18"
                    ]
                ]
            }
        ]
    }
}

Querying metadata

Finding series by label matchers

Prometheus Docs Reference

GET|POST /api/v1/series
Parameter Definition Support Optional
match[] series selector yes no
start start, format: RFC3399 or unix_timestamp in seconds yes no
end end, format: RFC3399 or unix_timestamp in seconds yes no
limit integer, maximum number of returned series yes yes

For metadata metrics: Note: SkyWalking’s metadata exists in the following metrics(traffics):

Name Require Labels Optional Labels Support Label Match
service_traffic layer service, limit =, (only service label support !=, =~, !~)
instance_traffic layer, service service_instance, limit =, (only service_instance label support !=, =~, !~)
endpoint_traffic layer, service endpoint, keyword, limit =, (only endpoint label support !=, =~, !~)
  • =: Label value equals the provided string.
  • !=: Label value does not equal the provided string.
  • =~: Label value regex-match the provided string.
  • !~: Label value does not regex-match the provided string

If the limit is not set by parameter or label, the default value is 100. If the `limit ' is also set in the query parameter, it returns the minimum of the two.

For example:

/api/v1/series?match[]=service_traffic{layer='GENERAL'}&start=1677479336&end=1677479636&limit=5

or

/api/v1/series?match[]=service_traffic{layer='GENERAL', limit='5'}&start=1677479336&end=1677479636

Result:

{
    "status": "success",
    "data": [
        {
            "__name__": "service_traffic",
            "service": "agent::songs",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {
            "__name__": "service_traffic",
            "service": "agent::recommendation",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {
            "__name__": "service_traffic",
            "service": "agent::app",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {
            "__name__": "service_traffic",
            "service": "agent::gateway",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {
            "__name__": "service_traffic",
            "service": "agent::frontend",
            "scope": "Service",
            "layer": "GENERAL"
        }
    ]
}
  • You can use the service label to filter the service_traffic result.
/api/v1/series?match[]=service_traffic{layer='GENERAL', service='agent::songs'}&start=1677479336&end=1677479636

use regex:

/api/v1/series?match[]=service_traffic{layer='GENERAL', service=~'agent::songs|agent::recommendation'}&start=1677479336&end=1677479636
  • You can use the service_instance label to filter the instance_traffic result.
/api/v1/series?match[]=service_traffic{layer='GENERAL', service='agent::songs', service_instance=~'instance1|instance2'}&start=1677479336&end=1677479636
  • You can use the endpoint label to filter the endpoint_traffic result.
/api/v1/series?match[]=service_traffic{layer='GENERAL', service='agent::songs', endpoint=~'endpoint1|endpoint2'}&start=1677479336&end=1677479636

Getting label names

Prometheus Docs Reference

GET|POST /api/v1/labels
Parameter Definition Support Optional
match[] series selector yes yes
start start, format: RFC3399 or unix_timestamp in seconds no yes
end end timestamp, if end time is not present, use current time as default end time yes yes
limit integer, maximum number of returned labels, default 100 yes yes

For example:

/api/v1/labels?match[]=instance_jvm_cpu'

Result:

{
    "status": "success",
    "data": [
        "layer",
        "service",
        "top_n",
        "order",
        "service_instance",
        "parent_service"
    ]
}

Querying label values

Prometheus Docs Reference

GET /api/v1/label/<label_name>/values
Parameter Definition Support Optional
match[] series selector yes yes
start start, format: RFC3399 or unix_timestamp in seconds no yes
end end, format: RFC3399 or unix_timestamp in seconds, if end time is not present, use current time as default end time yes yes
limit integer, maximum number of returned label values, default 100 yes yes

For example:

/api/v1/label/__name__/values

Result:

{
  "status": "success",
  "data": [
    "meter_mysql_instance_qps",
    "service_cpm",
    "envoy_cluster_up_rq_active",
    "instance_jvm_class_loaded_class_count",
    "k8s_cluster_memory_requests",
    "meter_vm_memory_used",
    "meter_apisix_sv_bandwidth_unmatched",
    "meter_vm_memory_total",
    "instance_jvm_thread_live_count",
    "instance_jvm_thread_timed_waiting_state_thread_count",
    "browser_app_page_first_pack_percentile",
    "instance_clr_max_worker_threads",
    ...
  ]
}

For metadata metrics:

Name Require Labels Optional Labels Support Label Match
service_traffic layer service, limit =, (only service label support !=, =~, !~)
instance_traffic layer, service service_instance, limit =, (only service_instance label support !=, =~, !~)
endpoint_traffic layer, service endpoint, keyword, limit =, (only endpoint label support !=, =~, !~)
  • =: Label value equal to the provided string.
  • !=: Label value not equal to the provided string.
  • =~: Label value regex-match the provided string.
  • !~: Label value do not regex-match the provided string

If the limit is not set by parameter or label, the default value is 100. And If the limit also set in the query parameter, will return the min number of the two.

For example:

  • If you want to query the label values of the service label in the service_traffic metric:
/api/v1/label/service/values?match[]=service_traffic{layer='GENERAL', service='agent::songs|agent::recommendation'}&limit=1

or

/api/v1/label/service/values?match[]=service_traffic{layer='GENERAL', service='agent::songs|agent::recommendation',limit='1'}
  • If you want to query the label values of the service_instance label in the instance_traffic metric:
/api/v1/label/service_instance/values?match[]=instance_traffic{layer='GENERAL', service='agent::songs', service_instance='instance1|instance2'}
  • If you want to query the label values of the endpoint label in the endpoint_traffic metric:
/api/v1/label/endpoint/values?match[]=endpoint_traffic{layer='GENERAL', service='agent::songs', endpoint='endpoint1|endpoint2'}

Querying metric metadata

Prometheus Docs Reference

GET /api/v1/metadata
Parameter Definition Support Optional
limit maximum number of metrics to return yes yes
metric metric name, support regular expression yes yes

For example:

/api/v1/metadata?limit=10

Result:

{
  "status": "success",
  "data": {
    "meter_mysql_instance_qps": [
      {
        "type": "gauge",
        "help": "",
        "unit": ""
      }
    ],
    "meter_apisix_sv_bandwidth_unmatched": [
      {
        "type": "gauge",
        "help": "",
        "unit": ""
      }
    ],
    "service_cpm": [
      {
        "type": "gauge",
        "help": "",
        "unit": ""
      }
    ],
    ...
  }
}

Metrics Type For Query

Supported Metrics Scope(Catalog)

Not all scopes are supported for now, please check the following table:

Scope Support
Service yes
ServiceInstance yes
Endpoint yes
ServiceRelation yes
ServiceInstanceRelation yes
EndpointRelation yes
Process no
ProcessRelation no

All Scopes could be found here.

General labels

Each metric contains general labels: layer. Different metrics will have different labels depending on their Scope and metric value type.

Query Labels Scope Expression Example
layer, service Service service_cpm{service='$service', layer='$layer'}
layer, service, service_instance ServiceInstance service_instance_cpm{service='$service', service_instance='$service_instance', layer='$layer'}
layer, service, endpoint Endpoint endpoint_cpm{service='$service', endpoint='$endpoint', layer='$layer'}
layer, service, dest_service, dest_layer ServiceRelation service_relation_metric{service='$service', layer='$layer', dest_layer='$dest_layer', dest_service='$dest_service'}
layer, service, dest_service, dest_layer, service_instance, dest_service_instance ServiceInstanceRelation service_instance_relation_metric{service='$service', layer='$layer', dest_layer='$dest_layer', dest_service='$dest_service', dest_service_instance='$dest_service_instance', service_instance='$service_instance'}
layer, service, dest_service, dest_layer, endpoint, dest_endpoint EndpointRelation endpoint_relation_metric{service='$service', endpoint='$endpoint', layer='$layer', dest_layer='$dest_layer', dest_service='$dest_service', dest_endpoint='$dest_endpoint'}

Common Value Metrics

  • Query Labels:
{General labels}
  • Expression Example:
service_cpm{service='agent::songs', layer='GENERAL'}
  • Result (Instant Query):
{
    "status": "success",
    "data": {
        "resultType": "vector",
        "result": [
            {
                "metric": {
                    "__name__": "service_cpm",
                    "layer": "GENERAL",
                    "scope": "Service",
                    "service": "agent::songs"
                },
                "value": [
                    1677490740,
                    "3"
                ]
            }
        ]
    }
}

Labeled Value Metrics

  • Query Labels:
--{General labels}
--metric labels: Used to filter the value labels to be returned
  • Expression Example:
service_percentile{service='agent::songs', layer='GENERAL', p='50,75,90'}
  • Result (Instant Query):
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "service_percentile",
          "p": "50",
          "layer": "GENERAL",
          "scope": "Service",
          "service": "agent::songs"
        },
        "value": [
          1677493380,
          "0"
        ]
      },
      {
        "metric": {
          "__name__": "service_percentile",
          "p": "75",
          "layer": "GENERAL",
          "scope": "Service",
          "service": "agent::songs"
        },
        "value": [
          1677493380,
          "0"
        ]
      },
      {
        "metric": {
          "__name__": "service_percentile",
          "p": "90",
          "layer": "GENERAL",
          "scope": "Service",
          "service": "agent::songs"
        },
        "value": [
          1677493380,
          "0"
        ]
      }
    ]
  }
}

Sort Metrics

  • Query Labels:
--parent_service: <optional> Name of the parent service.
--top_n: The max number of the selected metric value
--order: ASC/DES
  • Expression Example:
service_instance_cpm{parent_service='agent::songs', layer='GENERAL',  top_n='10', order='DES'}
  • Result (Instant Query):
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "service_instance_cpm",
          "layer": "GENERAL",
          "scope": "ServiceInstance",
          "service_instance": "651db53c0e3843d8b9c4c53a90b4992a@10.4.0.28"
        },
        "value": [
          1677494280,
          "14"
        ]
      },
      {
        "metric": {
          "__name__": "service_instance_cpm",
          "layer": "GENERAL",
          "scope": "ServiceInstance",
          "service_instance": "4c04cf44d6bd408880556aa3c2cfb620@10.4.0.232"
        },
        "value": [
          1677494280,
          "6"
        ]
      },
      {
        "metric": {
          "__name__": "service_instance_cpm",
          "layer": "GENERAL",
          "scope": "ServiceInstance",
          "service_instance": "f5ac8ead31af4e6795cae761729a2742@10.4.0.236"
        },
        "value": [
          1677494280,
          "5"
        ]
      }
    ]
  }
}

Sampled Records

  • Query Labels:
--parent_service: Name of the parent service
--top_n: The max number of the selected records value
--order: ASC/DES
  • Expression Example:
top_n_database_statement{parent_service='localhost:-1', layer='VIRTUAL_DATABASE',  top_n='10', order='DES'}
  • Result (Instant Query):
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "top_n_database_statement",
          "layer": "VIRTUAL_DATABASE",
          "scope": "Service",
          "record": "select song0_.id as id1_0_, song0_.artist as artist2_0_, song0_.genre as genre3_0_, song0_.liked as liked4_0_, song0_.name as name5_0_ from song song0_ where song0_.liked>?"
        },
        "value": [
          1677501360,
          "1"
        ]
      },
      {
        "metric": {
          "__name__": "top_n_database_statement",
          "layer": "VIRTUAL_DATABASE",
          "scope": "Service",
          "record": "select song0_.id as id1_0_, song0_.artist as artist2_0_, song0_.genre as genre3_0_, song0_.liked as liked4_0_, song0_.name as name5_0_ from song song0_ where song0_.liked>?"
        },
        "value": [
          1677501360,
          "1"
        ]
      },
      {
        "metric": {
          "__name__": "top_n_database_statement",
          "layer": "VIRTUAL_DATABASE",
          "scope": "Service",
          "record": "select song0_.id as id1_0_, song0_.artist as artist2_0_, song0_.genre as genre3_0_, song0_.liked as liked4_0_, song0_.name as name5_0_ from song song0_ where song0_.liked>?"
        },
        "value": [
          1677501360,
          "1"
        ]
      }
    ]
  }
}