Copilot Studio Conversation Analytics | Day 76

Day 76 of 365 Days of Copilot Studio explores Copilot Studio conversation analytics. Publishing an agent is only the beginning. You also need to understand whether people use it, where conversations stop, and which answers need improvement.

In Day 75, we investigated knowledge-source problems. Today, we look for patterns across conversations so that the next fix addresses a recurring user need.

1. Start With a Question You Can Investigate

Before opening a dashboard, choose a question: are employees finding the policy they need, are common requests reaching the right support team, or are users leaving before getting an answer? A clear question makes the numbers useful.

• Usage: are people starting conversations, and when?
• Outcomes: how do those interactions end?
• Experience: what do users tell you about the answers?

Choose a reporting period and record the agent version or recent changes. Otherwise, a shift in traffic can be mistaken for an improvement in quality.

2. Find the Analytics Area

Open Copilot Studio and select your agent. Look for its Analytics or Monitor area; the label can differ by experience. Microsoft's current monitoring documentation explains the available views. Check the selected reporting period before interpreting a chart.

Pay attention to the unit being counted. Sessions and conversations are not always interchangeable: one conversation can include multiple analytics sessions. Use the dashboard's own definitions and denominator when comparing rates.

3. Understand How Sessions End

Outcome	How to interpret it	What to investigate
Resolved	Resolution can be confirmed or implied. It does not independently prove answer accuracy.	Does a sample of answers match approved information?
Escalated	An escalation path was triggered; that alone does not prove a successful human handoff.	Was human help appropriate, and did the support process work?
Abandoned	An engaged session ended without a resolved or escalated outcome.	Was the next step unclear, repetitive, or unhelpful?

Unengaged sessions are a separate consideration. Avoid treating every visit as a completed attempt to solve a problem.

4. Read Resolution Rates in Context

Illustrative calculation: if a defined group contains 100 eligible sessions and 80 are resolved, its resolution rate is 80%. This is a teaching example; check the actual dashboard definition before reproducing a calculation.

A percentage without its sample size can mislead. Eight resolved sessions out of ten and 800 out of 1,000 produce the same rate, but represent very different levels of evidence. Also consider the kinds of requests: a week of simple questions is not directly comparable with a week of complex exceptions.

5. Investigate Escalations and Unexpected Endings

Group recurring problems by user intent. For example, an onboarding agent might answer general questions well but repeatedly struggle with requests for a particular department. That pattern gives you a narrower investigation than “improve the agent.”

• Missing knowledge: the approved answer is absent or hard to retrieve.
• Complex questions: the request needs clarification or several steps.
• Unsupported requests: the agent cannot perform the requested action.
• Unexpected endings: users may not understand what to do next.

Keep appropriate escalation paths. A request needing human approval should reach a person; reducing escalation at any cost is not a useful quality target.

6. Combine Usage Trends With Feedback

Review conversation volume over time alongside outcomes. A spike could follow a launch announcement, an internal campaign, or a business event. Note that context before attributing it to an agent change.

CSAT reflects survey responses, not the views of every user. Treat feedback as another signal alongside session outcomes.

Read the available comments for concrete problems. “The answer linked to last year's policy” suggests a specific content check. A low rating without detail needs further investigation before you decide which component to change.

7. Build a Continuous Improvement Loop

1. Review: capture a baseline for a defined period.
2. Find: select one recurring question or unsuccessful path.
3. Improve: update the relevant knowledge, instructions, or conversation flow.
4. Test: repeat the original question and realistic variations.
5. Measure: compare a similar period after release, including sample size and feedback.

Keep a short change log with the problem, owner, change date, test cases, and observed result. If the numbers improve but the original failed question still produces a wrong answer, the work is not finished.

Day 76 Practice

Choose one agent and one reporting period. Write down its session volume, outcome breakdown, and one recurring issue. Propose a targeted improvement and three questions you will use to verify it. Review only conversation information you are authorized to access, and avoid copying personal details into your notes.

Today's takeaway: measure conversations, understand outcomes, and use the evidence to choose the next improvement. Continue exploring the 365 Days of Copilot Studio series.