Cache Warmer - WunderGraph

Cache Warming

Cache warming is particularly beneficial in scenarios such as e-commerce flash sales, live broadcasts, and major marketing events, where even minor delays can impact user experience and revenue.

How CDN Based Cache Warming Works

Cache warming operates by:

1. Identifying Slow Queries: Telemetry data is used to detect high-latency operations that could cause performance bottlenecks. The system prioritizes queries based on P90 latency measurements, ensuring that the slowest queries are targeted for warming.
2. Building a Manifest: The system prioritizes the slowest queries and then compiles them into a manifest for caching. This manifest is stored in the CDN and fetched whenever the router needs it.
3. Precomputing Query Plans: The cache warmer precomputes and stores query plans in the router, ensuring immediate availability during peak traffic periods. This precomputing occurs at the start of the router, as well as whenever the router restarts due to a configuration update triggered by a subgraph publish.

Key Characteristics of CDN Cache Warming

CDN based cache warming has several important characteristics that distinguish it from the in-memory fallback cache: Advantages:

• Works from the first router start: Unlike in-memory fallback, CDN cache warming provides immediate benefits from the moment the router starts for the first time, with no cold start period.
• Targets slow queries strategically: It focuses on a curated set of queries that have slow planning times (identified through telemetry) and queries that have been manually added, ensuring the most impactful optimizations.
• Catches infrequent but slow queries: Even less regularly occurring queries that are slow to plan will be precomputed on startup, preventing occasional latency spikes.

Tradeoffs:

• Limited coverage: Fast-to-plan queries that occur frequently are not included in the CDN cache manifest, as they don’t benefit significantly from precomputation.

Configuration & Customization

Enabling Cache Warming

• The feature is available to enterprise customers via the Cosmo interface.

• Organizations can activate it at the namespace level to target specific workloads.

• Users can configure the maximum number of operations for cache warming.

• Operations are managed using a LIFO (Last-In, First-Out) policy, ensuring the latest operation is added while the oldest is removed once the limit is reached.

Router Configuration

To enable the cache warmer in the router, add the following configuration to your router configuration file:

cache_warmup:
  enabled: true

telemetry:
  metrics:
    attributes:
      - key: "wg.operation.hash"
        value_from:
          context_field: operation_hash

Cache Warmer Optimization

Three settings shape warmup throughput: workers, items_per_second, and item_delay. The timeout bounds how long startup will wait for warmup to complete. Tune them to balance warm-up speed against system load and deploy time.

cache_warmup:
  enabled: true
  workers: 8
  items_per_second: 50
  item_delay: 0s
  timeout: 30s

Customization Options

Manually Prioritized Operations Customers can add operations to the cache manually, ensuring critical queries are always warmed. It can be added using wgc.

wgc router cache push <graph_name> -n <namespace_name> -f <path_to_file>

Manual Recompute from Studio

Users can manually recompute slow queries from the Cosmo Studio. Currently, recomputation only occurs when a manual operation is added or when the subgraph is published.

In-Memory Fallback Cache Warming

The in-memory fallback cache warming feature uses the slow plan cache to preserve query plans across hot config reloads and schema changes, reducing latency spikes during restarts.

How It Works

The in-memory fallback relies on the slow plan cache — a secondary, bounded cache that tracks queries whose planning time exceeds a configurable threshold (slow_plan_cache_threshold, default 100ms). During normal operation, this cache is populated in two ways:

1. On first plan: When a query is planned and its planning duration exceeds the threshold, the plan is stored in both the main cache and the slow plan cache.
2. On eviction: If the main TinyLFU cache evicts a plan that is in the slow plan cache, the query plan won’t be recomputed and would simply be served from the slow plan cache.

When the router reloads, the slow plan cache contents are used to rewarm the cache.

Key Characteristics of In-Memory Fallback

Advantages:

• Coverage of expensive queries: By default, queries with planning times above the threshold (100ms) are preserved and warmed on reload, protecting slow-to-plan queries from cold-start latency.
• Eliminates reload spikes for expensive queries: You won’t experience query planning spikes for queries above the threshold after configuration or schema reloads. Users can tune the threshold to cover more or fewer queries.

Tradeoffs:

• Cold start on first start: The first router start will experience normal cache warming latency, as there’s no existing cache to preserve.

Configuration

The in-memory fallback can be enabled as a fallback for the CDN cache warmer. To do this, ensure:

• Cache warming is enabled in the router configuration (cache_warmup.enabled: true)
• source.cdn.enabled is set to true (this is true by default and does not need to be explicitly specified)
• in_memory_fallback is set to true (default)

cache_warmup:
  enabled: true
  in_memory_fallback: true  # Enabled by default

source:
    cdn:
      enabled: true

To use the in-memory fallback as the primary source, ensure:

• Cache warming is enabled in the router configuration (cache_warmup.enabled: true)
• source.cdn.enabled is set to false (this needs to be specified explicitly, as the default is true)
• in_memory_fallback is set to true (default)

cache_warmup:
  enabled: true
  in_memory_fallback: true  # Enabled by default

source:
    cdn:
      enabled: false

Slow Plan Cache

When in-memory fallback is enabled, the cache the in memory fallback uses is the Slow Plan Cache. This is different from the main query plan cache which uses a TinyLFU (Least Frequently Used) eviction policy, which is optimized for frequently accessed items. However, this can cause problems for queries that are slow to plan but infrequently accessed — the LFU policy may evict them in favor of cheaper, more frequent queries. When an expensive query is evicted and re-requested, the router must re-plan it from scratch, causing a latency spike. The slow plan cache is a secondary cache that protects these slow-to-plan queries from eviction. It is automatically enabled when in_memory_fallback is set to true.

How It Works

1. When a query is planned for the first time, its planning duration is measured.
2. If the planning duration exceeds the configured threshold (slow_plan_cache_threshold, default 100ms), the query plan is stored in both the main cache and the slow plan cache.
3. If the main cache later evicts this plan (due to LFU pressure from more frequent queries), the OnEvict hook pushes it to the slow plan cache (if it meets the threshold).
4. On subsequent requests, if the plan is not found in the main cache, the router checks the slow plan cache before re-planning. If found, the plan is served immediately and re-inserted into the main cache.
5. During config reloads, slow plan cache entries are used as the warmup source, ensuring slow queries survive cache rebuilds.

Cache Size and Eviction

The slow plan cache has a configurable maximum size (slow_plan_cache_size, default 300). When the cache is full and a new expensive query needs to be added:

• The new query’s planning duration is compared to the shortest duration in the cache.
• If the new query is more expensive (took longer to plan), it replaces the least expensive entry.
• If the new query is cheaper or equal, it is not added. This ensures the cache always contains the most expensive queries.

Whenever an existing item in the cache is attempted to be added to the cache while full, we will not remove the entry and will only update its plan time duration if it was higher than the previous duration it took to plan. This way we only consider the worst case planning duration.

Configuration

The slow plan cache is configured through the engine configuration:

engine:
  slow_plan_cache_size: 300  # Maximum entries (default: 300)
  slow_plan_cache_threshold: 100ms    # Minimum planning time to qualify (default: 100ms)

cache_warmup:
  enabled: true
  in_memory_fallback: true  # Required to enable the slow plan cache

Tuning

You can tune the threshold and cache size to control warmup coverage:

• Lower threshold → more queries protected: Setting slow_plan_cache_threshold: 1ns captures all queries regardless of planning time. This gives you full “carry forward everything” behaviour similar to preserving the entire plan cache.
• Higher cache size → more entries held: Increase slow_plan_cache_size to hold more entries. For full coverage, set it to match or exceed execution_plan_cache_size.
• Tradeoff: Lower thresholds and larger cache sizes increase memory usage but provide broader warmup coverage.