Set up a project
Connections belong to the workspace. Each project then says which connection answers its metrics, logs and traces, and how its environments map onto the labels in your backend. This needs the Developer settings permission and lives in the project's Settings → Observability.
Pick the connections
| Signal | Answered by | Used for |
|---|---|---|
| Metrics | A Prometheus connection, or a Grafana connection with a Prometheus data source | Metric panels on releases and incidents |
| Logs | A Loki connection, or a Grafana connection with a Loki data source | Log lines on error events, traces, replays and incidents |
| Traces | A Tempo connection, or a Grafana connection with a Tempo data source | Backend traces on error events, traces and replays |
Leave a signal empty to turn its panels off. One Grafana connection can answer all three.
Map the environments
Guidely's environments (production, staging and any you added) need to find the right series, streams and services in your backend. For each environment, set:
| Setting | Example | Meaning |
|---|---|---|
| PromQL matcher | {service="checkout",env="prod"} | Label matchers, braces included, put into the metric panels' queries. Empty matches every series. |
| LogQL selector | {service_name="checkout",deployment_environment="production"} | The Loki stream selector for this environment's logs. Logs panels need one. |
| Tempo service | checkout | The service.name to search traces by, for panels that list recent or slow traces. Looking up a trace by its id doesn't need it. |
The All other environments row (*) applies to every environment that has no row of its own, which is handy when one set of selectors covers everything, or as a fallback while you fill in the others.
Use the label names your stack actually has: with the OpenTelemetry Collector's Loki exporter or Grafana Alloy that's usually service_name; with Promtail on Kubernetes it may be app, namespace or container. Grafana's Explore shows which labels exist.
Metric panels
Releases and incidents show a few metric panels, each a PromQL query over the time around the event. A project starts with three panels for services instrumented with OpenTelemetry's HTTP conventions:
| Panel | Query |
|---|---|
| Requests / s | sum(rate(http_server_request_duration_seconds_count{{selector}}[{{interval}}])) |
| 5xx responses (%) | 100 * sum(rate(http_server_request_duration_seconds_count{{{matchers}}http_response_status_code=~"5.."}[{{interval}}])) / sum(rate(http_server_request_duration_seconds_count{{selector}}[{{interval}}])) |
| p95 latency (ms) | 1000 * histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{{selector}}[{{interval}}]))) |
The Prometheus client preset swaps in the metrics of the Prometheus client libraries: http_requests_total (with a status label) and http_request_duration_seconds_bucket. Or write your own: up to 12 panels, each with a title, a query and a unit (count, rate, percent, ms, s or bytes).
Three placeholders are filled in before a query is sent:
| Placeholder | Becomes | Example |
|---|---|---|
{{selector}} | The environment's PromQL matcher, braces included, or nothing when it's empty | up{{selector}} → up{service="checkout",env="prod"} |
{{matchers}} | The same matchers without the braces and with a trailing comma, or nothing, for adding your own | up{{{matchers}}job="api"} → up{service="checkout",env="prod",job="api"} |
{{interval}} | A rate window that fits the chart: four times the step between points, at least 60 s | [{{interval}}] → [120s] for a 1-hour chart, [480s] for 4 hours |
The step is chosen to give about 120 points (at least 15 s). Each panel shows at most 12 series, so aggregate (sum by (…)) rather than charting every pod.
Matching log lines to traces
On error events, traces and replays, Guidely asks Loki for the lines that mention the trace ids it has (up to 50 at a time), using the environment's LogQL selector. Finding a trace in the logs says how:
- Anywhere in the line (the default):
{selector} |~ "(?i)<id>|<id>…". Works whenever the id appears anywhere in the text, which covers JSON logs ("trace_id":"4bf9…") and logfmt (trace_id=4bf9…). - trace_id label:
{selector} | trace_id=~"(?i)<id>|<id>…". For logs wheretrace_idis a label or structured metadata (Loki 3 with OpenTelemetry logs) and the line itself doesn't contain the id.
Incident panels show the lines in a window instead, optionally only errors (lines matching error, exception, fatal, panic or critical). You get the most recent lines (200 by default, at most
- in time order, each cut at 4,000 characters; levels come from a
level,detected_levelorseveritylabel, or from the line.
For the ids to match in the first place, see Trace ids that match.
Traces
A backend trace is looked up in Tempo by its id (/api/traces/<id>), the same id as the Guidely error or trace it belongs to. Spans show their service, name, kind, status, timing and attributes. When there is no id, trace lists search Tempo for the environment's Tempo service, optionally only traces slower than a duration.