Search logs, traces and metrics
This page helps you find out why an app is slow or failing: compare projects, follow one request through your services, search logs and see CPU use. For one project, open the project and choose Monitoring. For all projects, choose Monitoring in the sidebar.
These views need the full install profile (sudo platformctl provision --profile full) and services that use the Cairn SDK (see Monitor real users in the browser for browsers). Without it the views say so. Errors, uptime checks, alerts and SLOs still work.
Find a slow request
- Open the project and choose Monitoring, view Traces.
- Set Time range to Last 6 hours and, if you have several, pick the Environment.
- In the endpoints table, find the route with the highest p95 (the time 95% of requests finish within). Choose Traces on its row.
- Open a trace. The waterfall shows each step (span) and how long it took. Choose Logs for this trace to see the log lines from that request.
The time range, environment and open trace are kept in the page address, so you can share the link.
The views
| View | What it shows |
|---|---|
| Overview | Health; requests, error rate, p95 and Apdex (a 0 to 1 score of how many requests were fast enough); active alerts, open errors and uptime status; charts with deploy markers; SLO table; links to errors, uptime, alerts and releases |
| Logs | Filters (service, level, text, trace id) or a query; Live tail; a Trace button on lines that carry a trace id |
| Traces | Service map (choose a service to filter), services table, endpoints table (top 100 by request rate); trace search by service, span name, minimum duration and status, or a query; trace waterfall with span details |
| Users | Page speed per page for real users (p75 of LCP, INP, CLS, FCP, TTFB), coloured good, needs improvement or poor; session replays; heatmaps |
| Resources | CPU and memory by pod (top 8) and container restarts for the project's environments; flame graph per service and profile type (CPU, allocated or in-use memory and objects) |
Errors and Uptime are covered in Track and fix errors and Check that your site is up.
All projects
Sidebar Monitoring lists one row per project you can access: health, requests per second, error rate, p95, active alerts, open errors, uptime status and SLOs (total and breaching). Sort by any column. Choose a project to open its Monitoring overview with the same time range.
Cairn admins and DevOps users see every project. Everyone else sees projects they are a member of.
Health
Health is worked out the same way everywhere:
| Health | When |
|---|---|
| Critical | A critical alert is open, error rate is 5% or more, or uptime checks report a major outage |
| Degraded | Any alert is open, error rate is 1% or more, an SLO is breaching, or uptime checks report a partial outage or degraded |
| No data | No error rate, no latency and no uptime status |
| Healthy | Otherwise |
Logs and live tail
Filters build the query for you: service, environment (from the environment picker), level (matched case-insensitively against the usual level fields), text in the line, and a trace id (16 to 32 hex characters). You can also write the whole query yourself (LogQL). A metric query such as rate(...) returns series instead of lines, and the view says so.
Live tail checks every 5 seconds for newer lines and adds them on top, up to 1000 lines. The first load shows up to 300 lines.
Traces
Search returns up to 50 traces in the portal. The waterfall only shows spans from this project's services. Spans from other projects in the same request are left out and counted (N spans from other projects hidden). A trace with no spans from this project shows "Trace not found". The service map labels callers or callees outside the project as (other project); databases and messaging systems keep their names.
Profiles
Profiles cover at most just under 24 hours. With Last 7 days the view shows Range shortened to 23 h 59 min. The service list uses the same range, so with Last 15 minutes only services profiled in the last 15 minutes are listed.
Troubleshooting
| Message | What to do |
|---|---|
Observability backends are not installed (full install profile) |
Install the full profile |
<Backend> backend is unavailable |
The API cannot reach the backend. Check the pods in the monitoring namespace and the OBSERVABILITY_*_URL settings |
<Backend> backend is busy; retry shortly |
Retry |
<Backend> query timed out; narrow the time range or the query |
Shorten the range or simplify the query. Raise OBSERVABILITY_QUERY_TIMEOUT_MS if needed |
<Backend> response is too large; narrow the query |
Narrow the query, or raise OBSERVABILITY_MAX_RESPONSE_BYTES |
<Backend> backend rejected the query; check its syntax |
Fix the query |
<Backend> backend error |
See the API log ([observability] …) |
invalid since (use e.g. 15m, 1h, 24h, 7d), start must be before end, invalid step |
Fix the parameter |
Any error with code: "invalid_query" |
The query cannot be limited to this project. Simplify it |
Too many observability queries. Retry in a minute. |
Rate limit. See Rate limit |
Trace not found |
The trace does not exist, or has no spans from this project |
| Numbers show but with a red banner | Metrics were unavailable; counts from the database are still shown |
Empty tables usually mean no data: no SDK telemetry from your services, no browser monitoring (page speed), or no running environment (pod metrics).
How it works
The views query the full profile's telemetry backends through the API: Prometheus (metrics), Loki (logs), Tempo (traces) and Pyroscope (profiles). The full profile sets PLATFORM_OBSERVABILITY on the API; without it the query endpoints answer 503 observability_disabled.
Metric definitions
- RED metrics (rate, errors, duration) come from Tempo span metrics for server and consumer spans: request rate, error rate (spans with error status, %), and p50, p95 and p99 latency.
- Apdex uses T = 512 ms (satisfied up to 512 ms, tolerating up to 4T = 2048 ms), shown as Apdex (T = 512 ms). It is computed from latency histogram buckets, so T has to be a bucket boundary: 0.512 s and 2.048 s are default Tempo span-metrics buckets, 0.5 s is not. The summary returns it as
red.apdexThresholdMs. - Page speed uses one Loki query per web vital.
Project scoping
Every request checks that you are at least a viewer of the project. Cairn admins and DevOps users have access to all projects; agent tokens need projects:read. Then:
- Every PromQL, LogQL and TraceQL query, typed or built from filters, is rewritten on the server so that each selector is limited to the project. A query that tries to select another project, or that the server cannot safely rewrite, is rejected with 400 and
code: "invalid_query". The rewritten query is returned in the response'squeryfield. - Trace detail returns only this project's spans.
- Pod metrics use the namespaces of the project's own environments, never a namespace from the request.
- Profile selectors are built only from validated parameters.
Queries are limited to 4000 characters. Profile queries are limited to 86 340 seconds because Pyroscope refuses queries longer than its max_query_length of one day; the response marks a shortened range with range.clamped: true.
Time ranges and limits
Every endpoint accepts since (15m, 1h, 24h, 7d, …; default 1h) or start and end (Unix seconds, milliseconds or nanoseconds, or RFC 3339). An end more than 5 minutes in the future is treated as now. Ranges longer than the maximum keep their end and are shortened (range.clamped: true). The views then show a notice such as Range shortened to 3 d. Trims under 1% (the 24 h to 23 h 59 min profile trim) are not shown.
| Variable | Default (allowed) | Meaning |
|---|---|---|
OBSERVABILITY_MAX_RANGE_HOURS |
168 (1–2160) |
Longest range a query may cover |
OBSERVABILITY_MAX_POINTS |
1100 (50–11000) |
Points per series; the step is raised to fit |
OBSERVABILITY_MAX_SERIES |
500 (10–5000) |
Series returned by metric queries (truncated: true beyond) |
OBSERVABILITY_MAX_LOG_LINES |
1000 (10–5000) |
Upper bound for the log limit (default 200 via the API) |
OBSERVABILITY_MAX_TRACES |
100 (5–500) |
Upper bound for the trace search limit (default 20 via the API) |
OBSERVABILITY_MAX_TRACE_SPANS |
5000 (100–50000) |
Spans returned for one trace (truncated: true beyond) |
OBSERVABILITY_MAX_RESPONSE_BYTES |
16777216 (1–64 MiB) |
Larger backend responses are refused |
OBSERVABILITY_QUERY_TIMEOUT_MS |
10000 (1000–30000) |
Backend request timeout |
OBSERVABILITY_RATE_LIMIT |
600 (10–10000) |
Requests per minute per user or agent token |
OBSERVABILITY_FLEET_MAX_PROJECTS |
200 (1–2000) |
Projects in the all-projects list (truncated: true beyond) |
OBSERVABILITY_LOKI_SCOPE |
stream |
stream: the project id is a Loki stream label. metadata: it is structured metadata |
OBSERVABILITY_PROMETHEUS_URL |
PROMETHEUS_URL or http://kube-prometheus-kube-prome-prometheus.monitoring.svc:9090 |
Prometheus |
OBSERVABILITY_LOKI_URL |
LOKI_URL or http://loki.monitoring.svc:3100 |
Loki |
OBSERVABILITY_TEMPO_URL |
TEMPO_URL or http://tempo.monitoring.svc:3200 |
Tempo |
OBSERVABILITY_PYROSCOPE_URL |
PYROSCOPE_URL or http://pyroscope.monitoring.svc:4040 |
Pyroscope |
Values outside the allowed range are clamped to it.
Rate limit
One per-user limit (default 600 requests per minute, OBSERVABILITY_RATE_LIMIT) covers every observability endpoint except status, including the all-projects list. With Redis it is shared by all API replicas. Typical portal cost per panel load: overview 2, services and endpoints 3, trace search 1, logs 1 (plus 1 every 5 seconds during live tail), pod metrics 1, profiles 2, page speed 5. Over the limit the API answers 429 and the view shows the error. Raise the variable if many people share one agent token or account.
Error status codes
| Message | Status |
|---|---|
| Backends not installed | 503 |
| Backend unavailable, response too large, other backend error | 502 |
| Backend busy (it returned 429 or 503) | 503 |
| Query timed out | 504 |
Query rejected, invalid parameter, invalid_query |
400 |
| Rate limit | 429 |
| Trace not found | 404 |
Overview with telemetryError (red banner) |
200 |
For developers / API
All endpoints need authentication. Project endpoints need project role viewer or higher. All except status share the rate limit.
| Method | Path | Purpose |
|---|---|---|
GET |
/api/observability/status |
Backends installed? Current maxRangeHours, maxLogLines, maxTraces |
GET |
/api/observability/fleet?since= |
Health, request rate, error rate, p95, alerts, open errors, uptime and SLOs per accessible project |
GET |
/api/projects/<id>/observability/summary?since=&environment= |
RED + Apdex, alert incidents, open errors, uptime status, SLOs, health |
GET |
/api/projects/<id>/observability/services?since=&environment= |
Services with request rate, error rate, p95 |
GET |
/api/projects/<id>/observability/endpoints?since=&service=&environment= |
Top 100 endpoints (span names) with rate, errors, p95 |
GET |
/api/projects/<id>/observability/red?since=&service=&environment=&step= |
RED time series |
GET |
/api/projects/<id>/observability/service-graph?since=&environment= |
Service map nodes and edges |
GET |
/api/projects/<id>/observability/infra?since=&environment=&step= |
Pod CPU and memory series, restarts |
GET |
/api/projects/<id>/observability/query/prom?query=&start=&end=&step=&instant=1 |
Scoped PromQL (range, or instant with instant=1) |
GET |
/api/projects/<id>/observability/query/loki?query=&limit=&direction=&after= |
Scoped LogQL. Without query, built from service, environment, level, search, traceId. after is a nanosecond cursor for live tail |
GET |
/api/projects/<id>/observability/query/tempo?q=&limit= |
Scoped TraceQL search. Without q, built from service, name, environment, minDuration, maxDuration, status (error, ok, unset) |
GET |
/api/projects/<id>/observability/traces/<traceId> |
One trace, this project's spans only (spans, hiddenSpans, truncated) |
GET |
/api/projects/<id>/observability/profiles/meta?since= |
Profiled services and profile types (default range 24 h, clamped under 1 day) |
GET |
/api/projects/<id>/observability/query/pyroscope?profileType=&service=&environment=&since=&maxNodes= |
Flame graph (maxNodes default 1024, max 8192; range clamped under 1 day) |
curl -G "$API/api/projects/$PROJECT/observability/query/prom" \
-H "Authorization: Bearer $TOKEN" \
--data-urlencode 'query=sum by (service) (rate(traces_spanmetrics_calls_total[5m]))' \
--data-urlencode 'since=6h'