AWS Firehose receiver

AWS Firehose receiver listens on 0.0.0.0:12801 by default, and provides an HTTP Endpoint /aws/firehose/metrics that follows Amazon Kinesis Data Firehose Delivery Stream HTTP Endpoint Delivery Specifications You could leverage the receiver to collect AWS CloudWatch metrics, and analysis it through MAL as the receiver bases on OpenTelemetry receiver

Setup(S3 example)

  1. Create CloudWatch metrics configuration for S3 (refer to S3 CloudWatch metrics)
  2. Stream CloudWatch metrics to AWS Kinesis Data Firehose delivery stream by CloudWatch metrics stream
  3. Specify AWS Kinesis Data Firehose delivery stream HTTP Endpoint (refer to Choose HTTP Endpoint for Your Destination)

Usually, the AWS CloudWatch metrics process flow with OAP is as follows:

CloudWatch metrics with S3 -->  CloudWatch Metric Stream (OpenTelemetry formart) --> Kinesis Data Firehose Delivery Stream --> AWS Firehose receiver(OAP) --> OpenTelemetry receiver(OAP)

The following blogs demonstrate complete setup process for AWS S3 and API Gateway:

Supported metrics

Description Configuration File Data Source
Metrics of AWS Cloud S3 otel-rules/aws-s3/s3-service.yaml AWS CloudWatcher Metrics Stream -> AWS Firehose delivery stream -> SkyWalking OAP Server with AWS Firehose receiver
Metrics of AWS DynamoDB otel-rules/aws-dynamodb/dynamodb-service.yaml AWS CloudWatcher Metrics Stream -> AWS Firehose delivery stream -> SkyWalking OAP Server with AWS Firehose receiver
Metrics of AWS DynamoDB otel-rules/aws-dynamodb/dynamodb-endpoint.yaml AWS CloudWatcher Metrics Stream -> AWS Firehose delivery stream -> SkyWalking OAP Server with AWS Firehose receiver
Metrics of AWS API Gateway otel-rules/aws-gateway/gateway-service.yaml AWS CloudWatcher Metrics Stream -> AWS Firehose delivery stream -> SkyWalking OAP Server with AWS Firehose receiver
Metrics of AWS API Gateway otel-rules/aws-gateway/gateway-endpoint.yaml AWS CloudWatcher Metrics Stream -> AWS Firehose delivery stream -> SkyWalking OAP Server with AWS Firehose receiver

Notice

  1. Only OpenTelemetry format is supported (refer to Metric streams output formats)
  2. According to HTTPS requirement by AWS Firehose(refer to Amazon Kinesis Data Firehose Delivery Stream HTTP Endpoint Delivery Specifications, users have two options
  • A proxy(e.g. Nginx, Envoy) is required in front of OAP’s Firehose receiver to accept HTTPS requests from AWS Firehose through port 443. (Recommended based on the general security policy)
  • Set aws-firehose/enableTLS=true with suitable cert/key files through aws-firehose/tlsKeyPath and aws-firehose/tlsCertChainPath at OAP side to accept requests from firehose directly.
  1. AWS Firehose receiver support setting accessKey for Kinesis Data Firehose, please refer to configuration vocabulary