Metrics & Monitoring - WunderGraph

Open Telemetry (OTEL) & Prometheus

OTEL metrics are the foundation for both the OTEL and Prometheus metric storage models. Internally, we track metrics using the OTEL instrumentation library and export them via a Prometheus exporter. This process converts the metrics to be compatible with both formats. As a result, we aim to expose the same metrics consistently across both models, with a few exceptions:

Metrics

Synchronous and asynchronous instruments

OpenTelemetry instruments are either synchronous or asynchronous. Synchronous instruments take a measurement when they are called e.g. when a request is made. The measurement is done as another call during program execution, just like any other function call. Periodically, the aggregation of these measurements is exported by a configured exporter.Asynchronous instruments, on the other hand, provide a measurement at an interval, not triggered explicitly through program execution. All measurements on asynchronous instruments are performed once per export cycle.By default, the routers exports OTEL data every 15 seconds.

List of synchronous instruments

We collect the following metrics to get useful insights in the HTTP traffic:

Additionally, we expose the following metric:

Circuit Breaker specific metrics

We currently support two attributes for monitoring circuit breakers. To enable these metrics you need to set one of the following

telemetry:
  metrics:
    otlp:
      circuit_breaker: true
    prometheus:
      circuit_breaker: true

All the below mentioned metrics have the wg.subgraph.name dimensions. Do note that since a circuit breaker can be shared across subgraphs if they have the same routing url, the dimension is a string slice instead of a string.

GraphQL specific metrics

Cost Control metrics

When Cost Control is enabled and the cost_stats toggles are set, the following metrics are emitted:

See Cost Control for more details.Use the following config to enable some of them:

telemetry:
  metrics:
    otlp:
      cost_stats:
        estimated_enabled: true
        actual_enabled: true
    prometheus:
      cost_stats:
        estimated_enabled: true
        actual_enabled: true

Those metrics use the exact same attribute set as other router request-level metrics.Additionally, use response headers (X-WG-Cost-Estimated and X-WG-Cost-Actual) to observe costs.

List of asynchronous instruments

All asynchronous metrics are tracked from the router start. If polling is enabled, the router will serve different metric series when the server has swapped.

We collect the following metrics to gain useful insights into the router’s runtime behavior. Asynchronous metrics are not exposed on the Prometheus endpoint.

If you don’t need runtime metrics, you can disable them in the config:

config.yaml

telemetry:
  metrics:
    router_runtime: false

Dimensions

All metrics are tracked along different dimensions. A dimension is an attribute in Open Telemetry or a label in Prometheus.

Common and known environment dimensions across all metrics

Common GraphQL dimensions across all metrics

Error identificationYou can use the router.http.requests.error metric to track errors across router and subgraph requests. In addition to that, you can also use the router.http.requests metric to identify errors. We attach the following fields:

You have two ways to query for errors. Both metrics can be quite useful depending on the query scenario. In general, we recommend to use the router.http.requests.error metric.

For a detailed breakdown of which scenarios mark spans as ERROR and increment error metrics, see Span Error Status & Error Metrics.

Subgraph specific

Resource attributes

In OTEL, resource attributes are special attributes, identifying metrics based on resource characteristics. Although stored differently, they affect metric cardinality. In Prometheus, this data is labeled on the target_info metric.

Few attributes can configured through the route configuration:

config.yaml

instance_id: my_replica_1 # Unique instance ID e.g. replica-1
cluster:
  name: us-central1-cosmo-cloud
telemetry:
  service_name: my-application # Name of your application e.g. my-router

Please use meaningful names in the Studio to ensure clarity. The instance ID is particularly important because it allows for the identification of a router even after restarts. If not specified, a random unique ID is used.

Engine Metrics

We expose subscription statistics from the router and the internal engine. By utilizing these metrics users can figure out the usage of subscriptions.

config.yaml

telemetry:
  metrics:
    otlp:
      engine_stats:
        subscriptions: true
    prometheus:
      engine_stats:
        subscriptions: true

Metrics

Resolver Metrics

These metrics expose how the GraphQL engine’s resolver concurrency pool is used. The router limits the number of resolvers that can run at the same time (see max_concurrent_resolvers). Use these metrics to detect when operations are queuing because the pool is saturated.

config.yaml

telemetry:
  metrics:
    otlp:
      resolver:
        enabled: true
    prometheus:
      resolver:
        enabled: true

Metrics

Cosmo Streams Metrics

We expose metrics for Cosmo Streams. New data points are emitted when new messages are either sent or received from the messaging backend

config.yaml

telemetry:
  metrics:
    otlp:
      streams: true
    prometheus:
      streams: true

Metrics

Connection Metrics

We also provide lower level metrics which helps track connection and pool metrics. By utilizing these metrics users can figure out when the router’s connection pool is full and when connections are misbehaving by for example observing spikes in time to acquire connections. This can be enabled for Open Telemetry or Prometheus via

config.yaml

telemetry:
metrics:
  otlp:
    connection_stats: true
  prometheus:
    connection_stats: true

Metrics

Network Metrics

These metrics break down each outgoing subgraph request into its individual HTTP phases. They are collected per request using Go’s httptrace, so unlike Connection Metrics they do not depend on the connection pool dialer. Use them to attribute subgraph latency to DNS resolution, connection setup, the TLS handshake, or the subgraph itself. This is useful for diagnosing issues such as slow DNS lookups.

config.yaml

telemetry:
metrics:
  otlp:
    network:
      enabled: true
  prometheus:
    network:
      enabled: true

Metrics

Custom Attributes

You can also add custom attributes to OTEL and Prometheus. Please refer to the Custom Attributes section.

Subgraph errors

When the router sends a request to your subgraphs and encounters a failure, we capture each subgraph error as individual OpenTelemetry (OTEL) events associated with the Engine - Fetch Span. These events include details such as the subgraph name, ID, error message, and extension code (if available). Additionally, we increment the router.http.requests.error metric.

Prometheus

To get a list of all Prometheus metrics, we advise navigating to the Prometheus endpoint http://127.0.0.1:8088/metrics. However, it’s important to initiate a request first; failing to do so will result in no request metrics being displayed.

Exclude certain metrics and labels

Sometimes it is useful to have the flexibility to exclude specific metrics or labels to reduce the load “cardinality” of your metrics server. You can do this easily by excluding them in the router config. We support a Go Regex string. You can test your Regex at https://regex101.com/.

config.yaml

telemetry:
  # OpenTelemetry Metrics
  metrics:
    # Expose OpenTelemetry metrics as Prometheus metrics
    prometheus:
      exclude_metrics:
        - "^router_http_request_duration_milliseconds$" # Exclude the full histogram
      exclude_metric_labels:
        - "^wg_client_version$"

This excludes router_http_request_duration_milliseconds histogram and the label wg_client_version from all metrics.

Exclude scope info

You can exclude OpenTelemetry scope info from Prometheus labels as well, which can be useful in some cases where you want to reduce metrics cardinality.

config.yaml

telemetry:
  # OpenTelemetry Metrics
  metrics:
    # Expose OpenTelemetry metrics as Prometheus metrics
    prometheus:
      exclude_scope_info: true

Make metrics accessible on all networks

In container environments, it is necessary to expose your server on 0.0.0.0 to make the port accessible from outside. You can enable it by setting the following configuration.

config.yaml

telemetry:
  metrics:
    prometheus:
      listen_addr: "0.0.0.0:8088"

Alternatively, you can use the environment variable.

PROMETHEUS_LISTEN_ADDR=0.0.0.0:8088

Example Prometheus Queries

Here you can see a few example queries to query useful information about your client traffic:

Get request rate by operation name

sum by (wg_operation_name) (rate(router_http_requests_total{wg_subgraph_name="",wg_operation_name!=""}[5m]))

Get error rate by operation name

sum by (wg_operation_name) (rate(router_http_requests_error_total{wg_subgraph_name="",wg_operation_name!=""}[5m]))

95th percentile latency by operation name

histogram_quantile(0.95, sum by (wg_operation_name, le) (rate(router_http_request_duration…

A GraphQL query can result in multiple requests to your subgraphs. Every request is annotated with the subgraph name and id.

Get request rate by operation name

sum by (wg_operation_name) (rate(router_http_requests_total{wg_subgraph_name!="",wg_operation_name!=""}[5m]))

Get error rate by operation name

sum by (wg_operation_name) (rate(router_http_requests_error_total{wg_subgraph_name!="",wg_operation_name!=""}[5m]))

95th percentile latency by operation name

histogram_quantile(0.95, sum by (wg_operation_name, le) (rate(router_http_request_duration_milliseconds_bucket{wg_subgraph_id!=""}[5m])))

Get error rate by subgraph name

sum by (wg_subgraph_name, le) (rate(router_http_requests_error_total[5m]))

If you have enabled connection_stats you can use the following queries to get useful information about your connection pool. We use server_address, server_port and subgraph as dimensions. The subgraph dimension is used because it is possible for two subgraphs to have the same host and port in the routing url.

Get Number Of Unused Connections In Pool

An important caveat is that if you have a non global max_conns_per_host for a subgraph, you must apply the correct labels on router_http_client_max_connections to get the correct value.

router_http_client_max_connections - on() group_left router_http_client_active_connections{server_address="192.1.2.5",server_port="4001"}

Check if the connection pool is not exhausted for the last N minutes

(
 max_over_time(router_http_client_max_connections[2m])
 - on() group_left
 min_over_time(router_http_client_active_connections{server_address="192.1.2.5", server_port="4001"}[2m])
) == 0

Identify slow connection acquisitions

You can combine the above metric with outliers in durations to acquire connections. Since this is a histogram, you can filter by buckets (buckets are broken down by “ms”). We also have the reused dimension, which points to connections that already have been established. It is worth noting this dimension when observing timings, as reused true connections will most likely be faster than its reused false connection equivalent. In the below example, we find connections that took less than 25ms to acquire (using the 25ms bucket).

router_http_client_connection_acquire_duration_bucket{wg_http_client_reused_connection="true",le="25.0"}

Cosmo Streams exposes only two core metrics. To get meaningful insights, you’ll need to filter them using the available attributes. The examples below show how to work with these metrics effectively.

Get failed publishes for a message broker

Let’s say we want to see any failed publishes to our kafka broker. We can use the following query,

router_streams_sent_messages_total{wg_provider_type="kafka",wg_provider_id="provider-id",wg_destination_name="employeeUpdated",wg_error_type!=""}

If we are using a provider like nats which could publish to the same destination through different methods, we could also filter using the stream operation name

router_streams_sent_messages_total{wg_provider_type="nats",wg_destination_name="employeeUpdatedMyNats.1",wg_error_type!="",wg_stream_operation_name="publish"}

Look at messaged received throughput

You can use the rate function to understand the per-second average rate of messages received over the last 60 minutes

rate(router_streams_received_messages_total{wg_provider_type="kafka"}[60m])

With proper attribute filtering as demonstrated above, you can make it specific for a messaging backend or a destination.

Summary

By collecting the metrics, you can find answers to the following questions: