## Open Telemetry (OTEL) & Prometheus OTEL metrics are the foundation for both the OTEL and Prometheus metric storage models. Internally, we track metrics using the OTEL instrumentation library and export them via a Prometheus exporter. This process converts the metrics to be compatible with both formats. As a result, we aim to expose the same metrics consistently across both models, with a few exceptions: - We do not expose [asynchronous instruments](https://cosmo-docs.wundergraph.com/router/metrics-and-monitoring#asynchronous-instruments) through Prometheus. This decision is pending further feedback collection. - In Prometheus, metrics are formatted in snake_case and conclude with specific suffixes. ## Metrics ### Synchronous and asynchronous instruments OpenTelemetry instruments are either synchronous or asynchronous. Synchronous instruments take a measurement when they are called e.g. when a request is made. The measurement is done as another call during program execution, just like any other function call. Periodically, the aggregation of these measurements is exported by a configured exporter.Asynchronous instruments, on the other hand, provide a measurement at an interval, not triggered explicitly through program execution. All measurements on asynchronous instruments are performed once per export cycle.By default, the routers exports OTEL data **every** 15 seconds. ### List of synchronous instruments We collect the following metrics to get useful insights in the HTTP traffic: - `router.http.requests`: Total count of incoming and outgoing requests. Incoming requests are requests that the router receives from the clients and outgoing requests are requests that the router makes to the subgraphs. The outgoing requests have the `wg_subgraph_id` dimension attached. - `router.http.response.content_length`: Total bytes of incoming requests - `router.http.request.content_length`: Total bytes of outgoing responses - `router.http.request.duration_milliseconds`: End-to-end duration of incoming requests in (histogram). The top boundary of the buckets is 10s. - `router.http.requests.in_flight`: Number of in-flight requests. (Only static and subgraph dimensions are attached) - `router.http.requests.error`: Total number of failed requests. Additionally, we expose the following metric: - `router_info`: Provides information about the router configuration versions currently in use. There will be one entry for the base configuration and additional entries for each feature flag. For the base configuration, no feature flag name will be attached. The metric value is always set to 1. The metric includes the following attributes: - `wg_router_config_version`: The identifier of the router execution configuration - `wg_router_version`: The version of the router that is running - `wg_feature_flag`: (Optional) The name of the feature flag if this is a feature flag configuration #### Circuit Breaker specific metrics We currently support two attributes for monitoring circuit breakers. To enable these metrics you need to set one of the following ``` telemetry: metrics: otlp: circuit_breaker: true prometheus: circuit_breaker: true ``` All the below mentioned metrics have the `wg.subgraph.name` dimensions. Do note that since a circuit breaker can be shared across subgraphs if they have the same routing url, the dimension is a string slice instead of a string. - `router.circuit_breaker.state`: This indicates the current state of a circuit, `0` represents not opened, and `1` represents opened. - `router.circuit_breaker.short_circuits`: This indicates how many requests for this circuit have failed without even being processed, because the circuit was open. #### GraphQL specific metrics - `router.graphql.operation.planning_time`: Time taken to plan the operation. An additional attribute `wg.engine.plan_cache_hit` indicates if the plan was served from the main execution plan cache or the plan fallback cache. #### Cost Control metrics When [Cost Control](https://cosmo-docs.wundergraph.com/router/security/cost-control) is enabled and the `cost_stats` toggles are set, the following metrics are emitted: - `router.graphql.operation.cost.estimated`: Estimated cost calculated before execution, based on `@cost` / `@listSize` directive weights and list size estimates. (Int64 Histogram) - `router.graphql.operation.cost.actual`: Actual cost calculated after execution using real list sizes returned by subgraphs. (Int64 Histogram) See [Cost Control](https://cosmo-docs.wundergraph.com/router/security/cost-control#telemetry) for more details.Use the following config to enable some of them: ``` telemetry: metrics: otlp: cost_stats: estimated_enabled: true actual_enabled: true prometheus: cost_stats: estimated_enabled: true actual_enabled: true ``` Those metrics use the exact same [attribute set](https://cosmo-docs.wundergraph.com/router/metrics-and-monitoring#Dimensions) as other router request-level metrics.Additionally, use [response headers](https://cosmo-docs.wundergraph.com/router/security/cost-control#response-headers) (`X-WG-Cost-Estimated` and `X-WG-Cost-Actual`) to observe costs. ### List of asynchronous instruments All asynchronous metrics are tracked from the router start. If polling is enabled, the router will serve different metric series when the server has swapped. We collect the following metrics to gain useful insights into the router’s runtime behavior. Asynchronous metrics are not exposed on the Prometheus endpoint. - `runtime.uptime`: Seconds since application was initialized - `server.uptime`: Seconds since the server was initialized. Typically, this time represents how long a specific version of the graph has been running when polling from the controlplane is enabled. - `process.cpu.usage`: Total CPU usage of this process in percentage of host total CPU capacity - `process.runtime.go.mem.heap_alloc`: Bytes of allocated heap objects - `process.runtime.go.mem.heap_idle`: Bytes in idle (unused) spans - `process.runtime.go.mem.heap_inuse`: Bytes in in-use spans - `process.runtime.go.mem.heap_objects`: Number of allocated heap objects - `process.runtime.go.mem.heap_released`: Bytes of idle spans whose physical memory has been returned to the OS - `process.runtime.go.mem.heap_sys`: Bytes of heap memory obtained from the OS - `process.runtime.go.mem.live_objects`: Number of live objects is the number of cumulative Mallocs - Frees - `process.runtime.go.gc.count`: Number of completed garbage collection cycles - `process.runtime.go.goroutines.count`: Number of goroutines that currently exist - `process.runtime.go.info`: Information about the Go runtime environment e.g. Go version If you don’t need runtime metrics, you can disable them in the config: config.yaml ``` telemetry: metrics: router_runtime: false ``` ### Dimensions All metrics are tracked along different dimensions. A dimension is an attribute in Open Telemetry or a label in Prometheus. #### Common and known environment dimensions across all metrics - `wg.federated_graph.id`: The ID of the running graph - `wg.router.version`: The current router binary version - `wg.router.config.version`: The current router config version #### Common GraphQL dimensions across all metrics - `wg.operation.protocol`: The used protocol `http`, `ws` - `wg.operation.name`: The name of the operation - `wg.operation.type`: The type of the operation e.g. `query` - `wg.client.name`: The client name - `wg.client.version`: The client version **Error identification**You can use the `router.http.requests.error` metric to track errors across router and subgraph requests. In addition to that, you can also use the `router.http.requests` metric to identify errors. We attach the following fields: - `wg.request.error`: Identify if a server-side error occurred. This applies to a request that didn’t result in a successful HTTP or GraphQL response. Only set when it is `true`. Be aware that a Status-Code `200` can still be an error in GraphQL. Client disconnections are **not** counted as errors. - `http.status_code`: The status code of the response. You have two ways to query for errors. Both metrics can be quite useful depending on the query scenario. In general, we recommend to use the `router.http.requests.error` metric. For a detailed breakdown of which scenarios mark spans as ERROR and increment error metrics, see [Span Error Status & Error Metrics](https://cosmo-docs.wundergraph.com/router/metrics-and-monitoring/span-error-status). #### Subgraph specific - `wg.subgraph.name`: The name of the subgraph - `wg.subgraph.id`: The ID of the subgraph ### Resource attributes In OTEL, resource attributes are special attributes, identifying metrics based on resource characteristics. Although stored differently, they affect metric cardinality. In Prometheus, this data is labeled on the `target_info` metric. - `service.name`: The name of the router. Can be configured through the `telemetry.service_name` option. - `service.version`: The version of the router binary. - `service.instance.id`: The unique instance ID of the router. Can be configured through the `instance_id`option. - `process.pid`: The id of the running process - `host.name`: The hostname of the machine where the router is running. - `telemetry.sdk.version`: The version of instrumentation library. - `telemetry.sdk.language`: The programming language of the instrumented application. Few attributes can configured through the route configuration: config.yaml ``` instance_id: my_replica_1 # Unique instance ID e.g. replica-1 cluster: name: us-central1-cosmo-cloud telemetry: service_name: my-application # Name of your application e.g. my-router ``` Please use meaningful names in the Studio to ensure clarity. The instance ID is particularly important because it allows for the identification of a router even after restarts. If not specified, a random unique ID is used. ### Engine Metrics We expose subscription statistics from the router and the internal engine. By utilizing these metrics users can figure out the usage of subscriptions. config.yaml ``` telemetry: metrics: otlp: engine_stats: subscriptions: true prometheus: engine_stats: subscriptions: true ``` #### Metrics - `router.engine.connections`: This contains the number of connections active from the client to the router. - `router.engine.subscriptions`: The number of currently active subscriptions, this number will differ from `router.engine.connections` if the client reuses connections. - `router.engine.triggers`: The number of currently active triggers. Subscriptions from the router to the subgraphs are grouped by triggers, which includes query, headers, variables and such. If two subscriptions initiated contain the same parameters, it will utilize the same trigger, and as a result the same connection from the router to the subgraph. - `router.engine.messages.sent`: The number of total messages for subscriptions sent over from the subgraph to the router. ### Resolver Metrics These metrics expose how the GraphQL engine’s resolver concurrency pool is used. The router limits the number of resolvers that can run at the same time (see [`max_concurrent_resolvers`](https://cosmo-docs.wundergraph.com/router/configuration#router-engine-configuration)). Use these metrics to detect when operations are queuing because the pool is saturated. config.yaml ``` telemetry: metrics: otlp: resolver: enabled: true prometheus: resolver: enabled: true ``` #### Metrics - `router.engine.resolvers.max_concurrent`: The configured maximum number of concurrent GraphQL resolver slots, as set by `ENGINE_MAX_CONCURRENT_RESOLVERS`. - `router.engine.resolvers.inflight`: The number of GraphQL resolver slots currently in use. When this approaches `router.engine.resolvers.max_concurrent`, operations start waiting for a free slot. - `router.graphql.resolver.acquire_duration`: Histogram (ms) of the time spent waiting to acquire a resolver concurrency slot. A rising value indicates resolver pool saturation. ### Cosmo Streams Metrics We expose metrics for [Cosmo Streams](https://cosmo-docs.wundergraph.com/router/cosmo-streams). New data points are emitted when new messages are either sent or received from the messaging backend config.yaml ``` telemetry: metrics: otlp: streams: true prometheus: streams: true ``` #### Metrics - `router.streams.received.messages`: Counter of messages consumed/received directly by the router from the messaging backend. - `router.streams.sent.messages`: Counter of messages produced/sent to the messaging backend, this happens when a mutation decorated with the `@edfs__xPublish` directive is executed. ### Connection Metrics We also provide lower level metrics which helps track connection and pool metrics. By utilizing these metrics users can figure out when the router’s connection pool is full and when connections are misbehaving by for example observing spikes in time to acquire connections. This can be enabled for Open Telemetry or Prometheus via config.yaml ``` telemetry: metrics: otlp: connection_stats: true prometheus: connection_stats: true ``` #### Metrics - `router.http.client.connection.max`: Static configuration values with the maximum connections allowed per host with a subgraph dimension. - `router.http.client.connection.active`: The number of currently active connections, grouped by both subgraph and host. A connection is considered active once it has completed DNS resolution, TLS handshake, and dialing. While it’s less common, multiple subgraphs can share the same host, which is why both dimensions are included. - `router.http.client.connection.acquire_duration`: The duration in ms that a connection took to be initialized, which includes all of DNS, TLS Handshakes, and Dialing the host. ### Network Metrics These metrics break down each outgoing subgraph request into its individual HTTP phases. They are collected per request using Go’s `httptrace`, so unlike [Connection Metrics](https://cosmo-docs.wundergraph.com/router/metrics-and-monitoring#connection-metrics) they do not depend on the connection pool dialer. Use them to attribute subgraph latency to DNS resolution, connection setup, the TLS handshake, or the subgraph itself. This is useful for diagnosing issues such as slow DNS lookups. config.yaml ``` telemetry: metrics: otlp: network: enabled: true prometheus: network: enabled: true ``` #### Metrics - `router.http.client.dns_lookup_duration`: Histogram (ms) of the DNS lookup duration for outgoing subgraph requests. Recorded as `0` when a cached or reused connection skips the lookup. - `router.http.client.tcp_connect_duration`: Histogram (ms) of the TCP connect (dial) duration for outgoing subgraph requests. Recorded only when a new connection is established. - `router.http.client.tls_handshake_duration`: Histogram (ms) of the TLS handshake duration for outgoing subgraph requests. Recorded only for new TLS connections. - `router.http.client.time_to_first_request_byte`: Histogram (ms) of the time from the start of the HTTP attempt to writing the first request byte to the subgraph. It includes acquiring the connection: pool wait, or DNS, TCP connect, and TLS handshake for new connections. - `router.http.client.time_to_first_byte`: Histogram (ms) of the time from completing the request write to receiving the first response byte from the subgraph. ### Custom Attributes You can also add custom attributes to OTEL and Prometheus. Please refer to the [Custom Attributes](https://cosmo-docs.wundergraph.com/router/open-telemetry/custom-attributes) section. ### Subgraph errors When the router sends a request to your subgraphs and encounters a failure, we capture each subgraph error as individual OpenTelemetry (OTEL) events associated with the `Engine - Fetch` Span. These events include details such as the subgraph name, ID, error message, and extension code (if available). Additionally, we increment the `router.http.requests.error` metric. ### Prometheus To get a list of all Prometheus metrics, we advise navigating to the Prometheus endpoint [http://127.0.0.1:8088/metrics](http://127.0.0.1:8088/metrics). However, it’s important to initiate a request first; failing to do so will result in no request metrics being displayed. ### Exclude certain metrics and labels Sometimes it is useful to have the flexibility to exclude specific metrics or labels to reduce the load “cardinality” of your metrics server. You can do this easily by excluding them in the router config. We support a Go Regex string. You can test your Regex at [https://regex101.com/](https://regex101.com/). config.yaml ``` telemetry: # OpenTelemetry Metrics metrics: # Expose OpenTelemetry metrics as Prometheus metrics prometheus: exclude_metrics: - "^router_http_request_duration_milliseconds$" # Exclude the full histogram exclude_metric_labels: - "^wg_client_version$" ``` This excludes `router_http_request_duration_milliseconds` histogram and the label `wg_client_version` from all metrics. ### Exclude scope info You can exclude OpenTelemetry scope info from Prometheus labels as well, which can be useful in some cases where you want to reduce metrics cardinality. config.yaml ``` telemetry: # OpenTelemetry Metrics metrics: # Expose OpenTelemetry metrics as Prometheus metrics prometheus: exclude_scope_info: true ``` ### Make metrics accessible on all networks In container environments, it is necessary to expose your server on `0.0.0.0` to make the port accessible from outside. You can enable it by setting the following configuration. config.yaml ``` telemetry: metrics: prometheus: listen_addr: "0.0.0.0:8088" ``` Alternatively, you can use the environment variable. ``` PROMETHEUS_LISTEN_ADDR=0.0.0.0:8088 ``` ### Example Prometheus Queries Here you can see a few example queries to query useful information about your client traffic: - Router - Subgraph - Connection - Cosmo Streams #### Get request rate by operation name ``` sum by (wg_operation_name) (rate(router_http_requests_total{wg_subgraph_name="",wg_operation_name!=""}[5m])) ``` #### Get error rate by operation name ``` sum by (wg_operation_name) (rate(router_http_requests_error_total{wg_subgraph_name="",wg_operation_name!=""}[5m])) ``` #### 95th percentile latency by operation name ``` histogram_quantile(0.95, sum by (wg_operation_name, le) (rate(router_http_request_duration… ``` A GraphQL query can result in multiple requests to your subgraphs. Every request is annotated with the subgraph name and id. #### Get request rate by operation name ``` sum by (wg_operation_name) (rate(router_http_requests_total{wg_subgraph_name!="",wg_operation_name!=""}[5m])) ``` #### Get error rate by operation name ``` sum by (wg_operation_name) (rate(router_http_requests_error_total{wg_subgraph_name!="",wg_operation_name!=""}[5m])) ``` #### 95th percentile latency by operation name ``` histogram_quantile(0.95, sum by (wg_operation_name, le) (rate(router_http_request_duration_milliseconds_bucket{wg_subgraph_id!=""}[5m]))) ``` #### Get error rate by subgraph name ``` sum by (wg_subgraph_name, le) (rate(router_http_requests_error_total[5m])) ``` If you have enabled `connection_stats` you can use the following queries to get useful information about your connection pool. We use `server_address`, `server_port` and `subgraph` as dimensions. The `subgraph` dimension is used because it is possible for two subgraphs to have the same host and port in the routing url. #### Get Number Of Unused Connections In Pool An important caveat is that if you have a non global [`max_conns_per_host`](https://cosmo-docs.wundergraph.com/router/configuration#traffic-shaping) for a subgraph, you must apply the correct labels on `router_http_client_max_connections` to get the correct value. ``` router_http_client_max_connections - on() group_left router_http_client_active_connections{server_address="192.1.2.5",server_port="4001"} ``` #### Check if the connection pool is not exhausted for the last N minutes ``` ( max_over_time(router_http_client_max_connections[2m]) - on() group_left min_over_time(router_http_client_active_connections{server_address="192.1.2.5", server_port="4001"}[2m]) ) == 0 ``` #### Identify slow connection acquisitions You can combine the above metric with outliers in durations to acquire connections. Since this is a histogram, you can filter by buckets (buckets are broken down by “ms”). We also have the `reused` dimension, which points to connections that already have been established. It is worth noting this dimension when observing timings, as `reused` true connections will most likely be faster than its `reused` false connection equivalent. In the below example, we find connections that took less than 25ms to acquire (using the 25ms bucket). ``` router_http_client_connection_acquire_duration_bucket{wg_http_client_reused_connection="true",le="25.0"} ``` Cosmo Streams exposes only two core metrics. To get meaningful insights, you’ll need to filter them using the available attributes. The examples below show how to work with these metrics effectively. #### Get failed publishes for a message broker Let’s say we want to see any failed publishes to our kafka broker. We can use the following query, ``` router_streams_sent_messages_total{wg_provider_type="kafka",wg_provider_id="provider-id",wg_destination_name="employeeUpdated",wg_error_type!=""} ``` If we are using a provider like nats which could publish to the same destination through different methods, we could also filter using the stream operation name ``` router_streams_sent_messages_total{wg_provider_type="nats",wg_destination_name="employeeUpdatedMyNats.1",wg_error_type!="",wg_stream_operation_name="publish"} ``` #### Look at messaged received throughput You can use the rate function to understand the per-second average rate of messages received over the last 60 minutes ``` rate(router_streams_received_messages_total{wg_provider_type="kafka"}[60m]) ``` With proper attribute filtering as demonstrated above, you can make it specific for a messaging backend or a destination. ### Summary By collecting the metrics, you can find answers to the following questions: - What is the error/success rate of my router/subgraph or a specific operation? - How is the performance of my router/subgraph or a specific operation? - What is the average request/response size of a specific operation? - How much traffic went through a router/subgraph instance? - What’s the distribution of Queries / Mutations and Subscription requests? - What’s the resource utilization in terms of CPU / Memory of my router instance? - What’re the Go runtime information of my router instance?