Nginx monitoring

Nginx performance from nginx-lua-prometheus

The nginx-lua-prometheus is a lua library that can be used with Nginx to collect metrics and expose them on a separate web page. To use this library, you will need Nginx with lua-nginx-module or directly OpenResty.

SkyWalking leverages OpenTelemetry Collector to transfer the metrics to OpenTelemetry receiver and into the Meter System.

Data flow

  1. nginx-lua-prometheus collects metrics from Nginx and expose them to an endpoint.
  2. OpenTelemetry Collector fetches metrics from the endpoint expose above via Prometheus Receiver and pushes metrics to SkyWalking OAP Server via OpenTelemetry gRPC exporter.
  3. The SkyWalking OAP Server parses the expression with MAL to filter/calculate/aggregate and store the results.

Set up

  1. Collect Nginx metrics and expose the following four metrics by nginx-lua-prometheus. For details on metrics definition, refer to here.
  • histogram: nginx_http_latency
  • gauge: nginx_http_connections
  • counter: nginx_http_size_bytes
  • counter: nginx_http_requests_total
  1. Set up OpenTelemetry Collector . For details on Prometheus Receiver in OpenTelemetry Collector, refer to here.
  2. Config SkyWalking OpenTelemetry receiver.

Nginx Monitoring

SkyWalking observes the status, payload, and latency of the Nginx server, which is cataloged as a LAYER: Nginx Service in the OAP and instances would be recognized as LAYER: Nginx instance.

About LAYER: Nginx endpoint, it depends on how precision you want to monitor the nginx. We do not recommend expose every request path metrics, because it will cause explosion of metrics endpoint data.

You can collect host metrics:

http {
  log_by_lua_block {
      metric_bytes:inc(tonumber(ngx.var.request_length), {"request", ngx.var.host})
      metric_bytes:inc(tonumber(ngx.var.bytes_send), {"response", ngx.var.host})
      metric_requests:inc(1, {ngx.var.status, ngx.var.host})
      metric_latency:observe(tonumber(ngx.var.request_time), {ngx.var.host})
  }
}

or grouped urls and upstream metrics:

upstream backend {
  server ip:port;
}

server {

  location /test {
    default_type application/json;
    return 200  '{"code": 200, "message": "success"}';
  
    log_by_lua_block {
      metric_bytes:inc(tonumber(ngx.var.request_length), {"request", "/test/**"})
      metric_bytes:inc(tonumber(ngx.var.bytes_send), {"response", "/test/**"})
      metric_requests:inc(1, {ngx.var.status, "/test/**"})
      metric_latency:observe(tonumber(ngx.var.request_time), {"/test/**"})
    }
  }
  
  location /test_upstream {
  
    proxy_pass http://backend;
  
    log_by_lua_block {
      metric_bytes:inc(tonumber(ngx.var.request_length), {"request", "upstream/backend"})
      metric_bytes:inc(tonumber(ngx.var.bytes_send), {"response", "upstream/backend"})
      metric_requests:inc(1, {ngx.var.status, "upstream/backend"})
      metric_latency:observe(tonumber(ngx.var.request_time), {"upstream/backend"})
    }
  }
}

Nginx Service Supported Metrics

Monitoring Panel Unit Metric Name Catalog Description Data Source
HTTP Request Trend meter_nginx_service_http_requests Service The increment rate of HTTP requests nginx-lua-prometheus
HTTP Latency ms meter_nginx_service_http_latency Service The increment rate of the latency of HTTP requests nginx-lua-prometheus
HTTP Bandwidth KB meter_nginx_service_bandwidth Service The increment rate of the bandwidth of HTTP requests nginx-lua-prometheus
HTTP Connections meter_nginx_service_http_connections Service The avg number of the connections nginx-lua-prometheus
HTTP Status Trend meter_nginx_service_http_status Service The increment rate of the status of HTTP requests nginx-lua-prometheus
HTTP Status 4xx Percent % meter_nginx_service_http_4xx_requests_increment / meter_nginx_service_http_requests_increment Service The percentage of 4xx status of HTTP requests nginx-lua-prometheus
HTTP Status 5xx Percent % meter_nginx_service_http_5xx_requests_increment / meter_nginx_service_http_requests_increment Service The percentage of 4xx status of HTTP requests nginx-lua-prometheus

Nginx Instance Supported Metrics

Monitoring Panel Unit Metric Name Catalog Description Data Source
HTTP Request Trend meter_nginx_instance_http_requests Instance The increment rate of HTTP requests nginx-lua-prometheus
HTTP Latency ms meter_nginx_instance_http_latency Instance The increment rate of the latency of HTTP requests nginx-lua-prometheus
HTTP Bandwidth KB meter_nginx_instance_bandwidth Instance The increment rate of the bandwidth of HTTP requests nginx-lua-prometheus
HTTP Connections meter_nginx_instance_http_connections Instance The avg number of the connections nginx-lua-prometheus
HTTP Status Trend meter_nginx_instance_http_status Instance The increment rate of the status of HTTP requests nginx-lua-prometheus
HTTP Status 4xx Percent % meter_nginx_instance_http_4xx_requests_increment / meter_nginx_instance_http_requests_increment Instance The percentage of 4xx status of HTTP requests nginx-lua-prometheus
HTTP Status 5xx Percent % meter_nginx_instance_http_5xx_requests_increment / meter_nginx_instance_http_requests_increment Instance The percentage of 4xx status of HTTP requests nginx-lua-prometheus

Nginx Endpoint Supported Metrics

Monitoring Panel Unit Metric Name Catalog Description Data Source
HTTP Request Trend meter_nginx_endpoint_http_requests Endpoint The increment rate of HTTP requests nginx-lua-prometheus
HTTP Latency ms meter_nginx_endpoint_http_latency Endpoint The increment rate of the latency of HTTP requests nginx-lua-prometheus
HTTP Bandwidth KB meter_nginx_endpoint_bandwidth Endpoint The increment rate of the bandwidth of HTTP requests nginx-lua-prometheus
HTTP Status Trend meter_nginx_endpoint_http_status Endpoint The increment rate of the status of HTTP requests nginx-lua-prometheus
HTTP Status 4xx Percent % meter_nginx_endpoint_http_4xx_requests_increment / meter_nginx_endpoint_http_requests_increment Endpoint The percentage of 4xx status of HTTP requests nginx-lua-prometheus
HTTP Status 5xx Percent % meter_nginx_endpoint_http_5xx_requests_increment / meter_nginx_endpoint_http_requests_increment Endpoint The percentage of 4xx status of HTTP requests nginx-lua-prometheus

Customizations

You can customize your own metrics/expression/dashboard panel.

The metrics definition and expression rules are found in /config/otel-rules/nginx-service.yaml, /config/otel-rules/nginx-instance.yaml, /config/otel-rules/nginx-endpoint.yaml.

The Nginx dashboard panel configurations are found in /config/ui-initialized-templates/nginx.

Collect nginx access and error log

SkyWalking leverages fluentbit or other log agents for collecting access log and error log of Nginx.

Data flow

  1. fluentbit agent collects access log and error log from Nginx.
  2. fluentbit agent sends data to SkyWalking OAP Server using native meter APIs via HTTP.
  3. The SkyWalking OAP Server parses the expression with LAL to parse/extract and store the results.

Set up

  1. Install fluentbit.
  2. Config fluent bit with fluent-bit.conf, refer to here.

Error Log Monitoring

Error Log monitoring provides monitoring of the error.log of the Nginx server.

Supported Metrics

Monitoring Panel Metric Name Catalog Description Data Source
Service Error Log Count meter_nginx_service_error_log_count Service The count of log level of nginx error.log fluent bit
Instance Error Log Count meter_nginx_instance_error_log_count Instance The count of log level of nginx error.log fluent bit

Customizations

You can customize your own metrics/expression/dashboard panel.

The log collect and analyse rules are found in /config/lal/nginx.yaml, /config/log-mal-rules/nginx.yaml.

The Nginx dashboard panel configurations are found in /config/ui-initialized-templates/nginx.