Cluster Status Check Sequence

The Cluster Status page (/operate/cluster, sidebar Operate → Cluster) is the operator’s single pane for “is the OAP backend healthy and configured correctly?” It runs three independent checks in parallel — the Query and Admin OAP ports plus a Zipkin/OTLP trace-source probe — they do not block each other, and the page surfaces each pane’s result independently.

The panes are independent: a healthy :12800 with broken :17128 is a real and recoverable state (forgot to expose the admin port behind a Kubernetes Service), and Horizon makes that diagnosis obvious. The Zipkin/OTLP pane is informational for the trace menu — a red dot there is not a cluster-wide outage.

The page header carries a single refresh both button that re-runs every check immediately — use it after fixing a network rule or an OAP selector instead of waiting for the next poll.

Pane A — Query / GraphQL port (:12800)

Single GraphQL call fired every 30 seconds:

query {
  version
  getTimeInfo { timezone, currentTimestamp }
  checkHealth { score, details }
}

What the pane shows

Failure modes

• Hard fail (unreachable): GraphQL endpoint refused / timed out / 5xx. reachable: false. Whole UI shows a top-of-page “OAP unreachable” banner — query pages cannot render.
• Soft fail (degraded): score > 0 — OAP is up but degraded (storage lag, receiver backlog, internal queue depth). Shown as a yellow “degraded (score N)” chip; details from checkHealth.details.
• Soft fail (not started): score < 0 — OAP process is running but has not finished initialization yet. Shown as “not started”; usually transient during a rolling restart.

Poll cadence

• Stale-time: 20 s
• Refetch interval: 30 s

Pane B — Admin host (:17128)

Per-feature reachability probes. Horizon checks each admin feature by firing a safe GET at the real REST path that feature calls, and colors the row by whether the path responds. Health is the live probe, not config-presence: a build that serves the path reads as reachable even if its config dump looks unfamiliar, and a path that 404s reads as unreachable even when the module’s selector appears in the OAP config.

Every round starts with the admin host itself:

GET <adminUrl>/debugging/config/dump

If that call fails, the pane shows a red Admin host unreachable block with the error and the exact URL tried, and every feature row reads unreachable — one root cause, not five stacked failures. Fix the network / port exposure / SW_ADMIN_SERVER=default first; individual selectors are irrelevant until the host answers.

When the host answers, each feature is probed on its own path and gets its own row:

Row states

• reachable (green) — the GET got an HTTP answer other than 404 or a 5xx. An auth challenge (401) or a bad-request answer (400) still proves the route is served, so it counts as reachable.
• unreachable (red) — the path returned 404 (the route isn’t registered: selector off, renamed, or absent in this OAP build), returned a 5xx, or failed at the network level. The pages the feature gates show a warning banner naming the env-var to set.
• readonly · bundled (yellow) — the ui-management row only, when Horizon runs with templates in readonly mode. Horizon serves its bundled dashboard templates and never calls the template store, so the path is not probed. This is informational, not a failure — but it does keep the pane badge at “4/5 reachable” instead of “all reachable”.

The pane badge summarizes the rows: all reachable, N/M reachable when some rows are not live-reachable, or unreachable when the admin host itself is down.

The “selector detected” footnote

Under each row’s state chip there is a footnote — selector detected or selector not detected — reporting whether any key with that module’s prefix appears in OAP’s config dump, i.e. whether the running release advertises that selector. This is the old config-presence check, kept as an informational hint only: a custom build can be perfectly reachable with no selector detected, or advertise a selector whose endpoint still 404s. The reachable/unreachable chip is the verdict.

Timestamps and re-checking

The pane header and each row carry a checked … ago stamp — when the displayed probe round actually ran. It can lag the page’s own refresh slightly, because concurrent viewers share one probe round for up to 30 seconds. refresh both forces a fresh probe round immediately.

Poll cadence

• Refetch interval: 60 s
• Probe rounds are shared across viewers for up to 30 s; refresh both bypasses the sharing

Pane C — Zipkin / OTLP traces

A third pane probes OAP’s Zipkin v2 REST endpoint and reports reachability. It feeds only the Zipkin/OTLP trace menu — a red dot here is not a cluster-wide outage; the rest of the UI keeps working and only Zipkin/OTLP trace views are affected.

Reading the page during an incident

The triage flow during “Horizon shows banners I don’t understand”:

1. Is the Query pane green? If not, OAP itself is down / unreachable — fix OAP first, the rest is downstream.
2. Does the Admin pane say “Admin host unreachable”? The admin port isn’t answering at all — expose port 17128 and confirm SW_ADMIN_SERVER=default. Don’t chase individual selectors yet.
3. Admin host up but a row reads unreachable? That one feature’s endpoint isn’t served — usually its selector is off. The row’s Gates column says what breaks and which env-var enables it (e.g. SW_INSPECT=default); set it on OAP, restart, then hit refresh both.
4. Is the health score > 0? OAP is up but degraded — pull details from checkHealth (visible in the Query pane) and triage on the OAP side.