# Monitoring

Interpret matchmaking queue activity, match throughput, ticket state, and connection health.

Source: https://birdhouse.tech/docs/monitoring/

BirdHouse separates matching rules from observation. Your game defines ticket constraints; monitoring shows the activity those rules produce.

## Queue and match activity

Read queue depth alongside creation rate, match rate, and wait time. A growing queue can indicate incompatible party sizes, restrictive rules, missing player traffic, or insufficient processing capacity.

| Signal | What to look for |
| --- | --- |
| Pending tickets | Tickets waiting in the queue; interpret alongside in-progress matching work. |
| Active and status counts | Retained tickets, including completed tickets during their retry window. |
| Created and matched ticket rates | Incoming demand versus successful matching throughput. |
| Published matches | Completed groups; one match can contain several tickets and players. |
| Match duration | Time from ticket creation to successful match publication. |
| Workers online | Approximate number of workers with live leases. |

Retained matched tickets are not searches still waiting to match. The service preserves their results for ten minutes so lost responses can be recovered.

## Live snapshots

The monitor exposes tRPC procedures at `/api/trpc`:

| Procedure | Purpose |
| --- | --- |
| `stats` | Latest snapshot with up to 120 history points; may be null before a sample is available. |
| `statsSubscription` | Current snapshot followed by live updates over server-sent events. |
| `tickets` | Read-only ticket search using a `query` string. |
| `ticket` | Read one retained ticket by `id`. |

The default fast refresh interval is one second. Status and metadata aggregates refresh on a slower, ten-second interval. Subscription buffering keeps the latest update for a stalled consumer rather than an unbounded backlog.

This is the monitoring transport, separate from the [matchmaking API](https://birdhouse.tech/docs/http-api/). An application integrating directly with tRPC should use the monitor's typed router contract instead of inventing REST paths for these procedures.

## Environment activity

The environment-aware service supplies status counts, lifetime counter totals, and sampled rates per canonical environment. Rates come from counter changes; they are not reconstructed from retained ticket documents.

Environment status aggregates have their own freshness timestamp. Different panels can therefore represent samples taken at slightly different times. Per-environment counters begin with the environment-aware release and cannot reconstruct earlier expired activity.

## Health endpoints

| Endpoint | Service | Behavior |
| --- | --- | --- |
| `GET /livez` | API service and worker | Responds when the process is alive; independent of the data-store connection. |
| `GET /readyz` | API service and worker | Returns `503` when the shared data-store connection is not ready. |
| `GET /metrics` | API service, worker, monitor | Prometheus metrics for the owning component. |
| `GET /healthz` | Monitor | Basic process response. |

A healthy process can be temporarily unready. An unavailable dependency should not be mistaken for a process that needs to restart.

## Metrics and freshness

Worker metrics include `matches_published_total`, `matched_players_total`, and `ticket_match_duration_seconds`. These are worker-local: aggregate rates and histogram buckets across workers.

Global queue and lifecycle gauges come from the monitor. Avoid summing duplicate samples of the same global quantity.

`workersOnline` is a lease-based approximation. After a hard worker failure, the value can remain stale for almost 41 seconds, including lease expiry, the slow discovery scan, and the next snapshot. Treat connection freshness and data freshness as separate signals.

## Investigate a waiting search

Retain the session's `TicketId`. Check its environment, party size, required metadata, and constraints. Verify that enough compatible tickets exist and that the search is still sending successful confirmations.

Use the read-only explorer to inspect the retained ticket. If it is absent after an outage, it may have expired. If it is matched, recover the shared result within the fixed retention window rather than creating a duplicate search.