Agent Analytics

Agent Analytics shows how your own AI agents and chatbots are doing: who talks to them, whether people get what they came for, where they get frustrated, what they ask about, and whether they come back. Your app sends each turn of a conversation (the person's messages, the agent's replies, tool calls) and Guidely does the rest. Open it from Analyze → Agent analytics.

How it differs from Scout

Scout is Guidely's own AI support agent in the messenger. Agent Analytics is for the agents you built (an in-app copilot, a support bot, a Slack assistant). Scout appears there too, as a built-in agent with a "Built in" badge, so you can compare them in one place: its messenger conversations are mirrored automatically (nothing to install), with Scout's own classification (topic, resolved, handed to the team, rating) mapped across.

Sending turns

A turn is one message in a conversation: { conversationId, agentId, role, text, … }, with role user, assistant, tool or system. The agent is created the first time its agentId appears (name from agentName, editable later); the conversation is keyed by your conversationId.

From the browser (the SDK adds the visitor's anonymous id, user id, account and session for you, so turns link to session replays):

guidely.agentConversation({ conversationId: thread.id, agentId: "support-bot", agentName: "Support bot", role: "user", text: message });
guidely.agentConversation({
  conversationId: thread.id,
  agentId: "support-bot",
  role: "assistant",
  text: reply.text,
  latencyMs: reply.latencyMs,
  toolCalls: [{ name: "lookup_order", status: "ok", latencyMs: 180 }],
});

See sdk.md for retries, consent and the client-side agents.redact hook.

From your servers with a secret key (Agent analytics → Agents → REST API → Create key, scope agents:write):

curl -X POST https://api.guidely.app/api/v1/agent-events \
  -H "Authorization: Bearer $GUIDELY_AGENTS_KEY" -H "Content-Type: application/json" \
  -d '{ "environment": "production", "turns": [ { "conversationId": "c_123", "agentId": "support-bot",
        "turnId": "c_123:1", "role": "user", "text": "Where is my order?", "userId": "user_42" } ] }'

Full field list and responses: api.md. Things worth knowing:

• Idempotent: a turn sent twice with the same turnId is stored once (the SDK sets one for you). Without a turnId, a stable one is derived from the role, occurredAt, text and tool names, so send occurredAt (or a turnId) when the same words can legitimately repeat.
• Tool calls: only name, status (ok / error) and latencyMs. Arguments and results are never sent or stored, even if you include them.
• Outcome and task success: put outcome (success, failure, escalated, abandoned) and taskSuccess on the turn that settles them; the latest reported outcome wins. Anything you report stands over Guidely's own judgement.
• People: userId is your identify() id and accountId your account id. Guidely links the conversation to the end user with that userId (else anonymousId) in the same project; it never creates end users from turns and never matches across projects. Like identify(), these ids are not verified.
• Limits: 100 turns and 1 MB per request, 20,000 characters per turn, 50 agents per project (turns for more are rejected), 2,000 turns per conversation.
• Usage: each new turn counts as one product analytics event on your plan. Past this month's allowance turns are dropped (overLimit in the response) like other analytics events. Scout's mirrored conversations don't count. Turning off analytics for the project (Settings → Privacy) stores nothing.

Redaction

Each agent has server-side redaction (Agents tab → Redaction, needs Developer settings), applied to text and string metadata before anything is stored:

With text storage off, Guidely keeps no text but still records roles, timings, tool calls, outcomes and the frustration signals (rephrases are compared through keyed hashes of the words, which can't be read back); topics, sentiment and summaries need text, so those conversations aren't classified.

Custom patterns are JavaScript regular expressions, checked when saved so they can't stall the server on hostile input: no backreferences, no nested repetition ((a+)+) or repeated alternatives ((a|b)+, use [ab]+), at most six unbounded repetitions, they must not match empty text, and they are timed against adversarial inputs. All rules run on pieces of at most 1,000 characters (split at whitespace), which bounds the work per match. Use Try it in the dialog to preview the result. Changes apply to new turns; stored turns keep the redaction they had. Changes are audit-logged (patterns by name only).

Redact in your app as well for anything that must not leave the browser or your servers: the SDK's agents.redact hook runs before a turn is sent.

Signals

Conversations are counted in the period they started, per environment. Users are counted by end user, else userId, else anonymousId.

Classification and what is sent to Anthropic

When a conversation of one of your agents has been quiet for 30 minutes (and again after new turns), an AI model reads it and records a topic (reusing the project's agent topics; at most 40 are created, then "Other"), the person's sentiment (positive / neutral / negative), task success (only when your agent didn't report it) and a one-sentence summary written without personal details.

What leaves Guidely: the conversation's stored text, already redacted (each turn cut to 2,000 characters, 40,000 in all, at most 200 turns), tool names and whether they failed, rephrase markers, the agent's name, the outcome or task success you reported, and the project's agent topic names. It goes to the Anthropic API with the project's own key when one is set (Scout settings → AI key), else the server's. Metadata, ids, user and account details are not sent. The transcript is treated as data to classify, never as instructions.

Classification is on by default. Turn it off in Agents → AI classification (needs Project settings); conversations then have no topic, sentiment or summary. Scout's conversations are never read by this classifier: Scout classifies its own conversations (reports.md) and those results are mirrored (rating 4–5 → positive, 3 → neutral, 1–2 → negative).

Dashboard

• Overview: conversations, users and accounts, average turns, task success rate, frustration rate, average latency, tool calls and tool error rate, with the previous period; conversations and frustrated conversations by day; each agent's figures; top topics. Filter by agent (Scout included), date range and environment.
• Conversations: filter by agent, topic, sentiment, outcome, frustrated, user (end user id, userId or anonymous id) and account.
• Users & accounts: the top 50 people and accounts by conversations, when they last used an agent, their success rate and frustration.
• Retention: weekly cohorts by the week people first used an agent (the selected one, or any), and the share who used any agent again in each later week (weeks start Monday, UTC).
• Topics & sentiment: per topic, conversations, success rate, frustration and the sentiment split.
• Agents: every agent with Scout's badge, redaction, AI classification, install snippets and the REST keys.

A conversation shows the transcript with roles (Scout's teammate replies as "Teammate"), tool calls (failed ones in red), latencies, rephrase markers and reported outcomes, its classification and where each verdict came from, and links to the end user's profile and the account's conversations.

Replay links

When a conversation carries a sessionId (the SDK sends it) and that session was recorded, each turn has Watch replay, which opens the recording at the turn's time: offset = turn time − recording start (never negative). Replay links are only shown to members with View session replays.

Access

Everyone in the workspace (viewer role and up) can see Agent Analytics, transcripts included, as with the Inbox. Redaction, agent names and deleting agents need Developer settings; the classification switch needs Project settings; deleting a conversation needs Delete end-user data. All changes are audit-logged. AI assistants can read it through the MCP tools get_agent_analytics_summary and list_agent_conversations.

Privacy, retention and deletion

• Conversations follow the project's event retention (default 365 days): ones whose last turn is older are deleted nightly with their turns.
• Deleting an end user (Users → Delete user data) deletes their agent conversations, including ones only linked by userId or anonymous id.
• Delete a single conversation from its page, or an agent with all its conversations from the Agents tab. Scout's mirrored conversations go when the Scout conversation is deleted in the Inbox (or by retention).
• Turn text is what your redaction leaves; prefer redacting in your app too, and turn text storage off for agents whose conversations should never be kept.

Workers

All are bounded per pass and safe to run on several workers (rows are claimed atomically or written idempotently).