Open Telemetry - WunderGraph

OpenTelemetry Configuration

Cosmo router supports exporting tracing and metrics via OpenTelemetry. By default, both are exported to Cosmo Cloud, but these can be configured with additional exporters or disable the default ones. Both http and grpc are supported.

config.yaml

telemetry:
  service_name: "cosmo-router" # Service name for traces and metrics
  tracing:
    enabled: true
    # If no exporters are configured, telemetry is exported to Cosmo Cloud
    exporters:
    - endpoint: https://cosmo-otel.wundergraph.com # The default
      exporter: "http" # or "grpc" with port 4317
      headers: {Authorization: Bearer ${ROUTER_TOKEN}}
  metrics:
    otlp:
      enabled: true
      # If no exporters are configured, telemetry is exported to Cosmo Cloud
      exporters:
      - endpoint: https://cosmo-otel.wundergraph.com # The default
        exporter: "http" # or "grpc" with port 4317
        headers: {Authorization: Bearer ${ROUTER_TOKEN}}

If no exporters are configured, the default one is used instead (set by the DEFAULT_TELEMETRY_EXPORTER environment variable).

The router can also expose Prometheus metrics. It works with the same OTEL metrics we export over OTEL.

Exclude certain metrics and labels

Excluding certain metrics and labels can significantly reduce the cardinality of the collected telemetry data, allowing for a tailored setup that aligns with your specific monitoring needs and minimizes unnecessary data collection. These exclusion options can be easily configured within the otlp section of the router config.

config.yaml

telemetry:
  metrics:
    otlp:
      exclude_metrics:
        # ❌ Incorrect - matches any character, not literal dots
        # - "wg.operation.hash"

# ✅ Correct - double escaping for literal dots
        - "wg\.operation\.hash"

# 💡 Best practice - single quotes, cleaner syntax
        - '^router\.http\.request\.duration_milliseconds$'
      exclude_metric_labels:
        - '^wg\.client\.version$'

Metrics Log Exporter

The metrics log exporter is a standalone OTLP exporter that prints all collected metrics directly to the router’s log output. It operates independently from your configured OTLP exporters and does not wrap them — it simply logs the metrics at each export interval. This gives you direct, local visibility into the metrics pipeline without needing an external collector.

config.yaml

telemetry:
  metrics:
    otlp:
      log_exporter:
        enabled: true
        exclude_metrics:
          - 'process\. '
          - 'server\.uptime'
          - 'router\.info'
      # alternatively use include_metrics to only include certain metrics

exclude_metrics and include_metrics cannot be used together. Use only one to filter which metrics are logged.

Limits

High metric cardinality can lead to performance issues by consuming excessive resources and slowing down data processing. When too many distinct metric labels are generated, the system might struggle to manage the data efficiently. To mitigate this, we have set a default hard cardinality limit of 2000. This limit helps to ensure that the metrics remain manageable and that the performance of our system is not adversely affected. Once the limit is reached, all further datapoints to a metric will be stored without attributes.

Tracing

You can configure multiple exporters. A common case is to forward telemetry data to Cosmo Cloud and e.g. Datadog Agent which has native support to ingest OpenTelemetry data.

config.yaml

telemetry:
  service_name: "cosmo-router" # Service name for traces and metrics

# OpenTelemetry Tracing
  tracing:
    enabled: true
    # If no exporters are defined, the default one is used
    exporters:
    # Cosmo Cloud
    - endpoint: https://cosmo-otel.wundergraph.com
      exporter: "http"
      headers: {Authorization: Bearer ${MY_AUTH_TOKEN}}
    # Datadog Agent
    - endpoint: http://datadog-agent:4318
      exporter: "http"

# OpenTelemetry Metrics
  metrics:
    otlp:
      enabled: true
      # If no exporters are defined, the default one is used
      exporters:
        - exporter: http # or grpc
          disabled: false
          endpoint: https://my-otel-collector.example.com

Trace ID Response header

This configuration allows you to include trace ID in the response headers, where the name of the header would be the value provided.

config.yaml

telemetry:
  tracing:
    response_trace_id:
      enabled: true
      header_name: "my-trace-id" # default: "x-wg-trace-id"

Sanitize UTF-8

OTLP uses Protocol Buffers for serialization, and protobuf’s string type requires valid UTF-8. Enable the following option to avoid issues caused by invalid UTF-8 bytes in string fields.

config.yaml

telemetry:
  tracing:
    sanitize_utf8:
      enabled: true
      # Optional: log a warning for each sanitized attribute (useful for debugging)
      log_sanitizations: false

FAQ

Can I export OTEL data from my application?
Yes, but this is currently limited to private backend applications e.g. subgraphs because we don’t provide a secure mechanism to issue short-lived tokens in public applications like Web-Apps. A possible solution is to run an otelcollector and utilize the export functionality to forward telemetry data to Cosmo Cloud.

Why is my trace incomplete?
In certain conditions, it can happen that spans are not listed in the Studio. You have to ensure that all spans are sent to us so we can reconstruct the full trace.