Understanding Conversation Metrics in Copilot Studio | Day 77

Day 77 of 365 Days of Copilot Studio focuses on understanding conversation metrics. In Day 76, we built a conversation-analytics review process. Today, we examine the numbers behind that review so a dashboard can guide a useful decision.

The lesson covers usage, engagement, resolution, escalation, abandonment, satisfaction, and duration. The hero is a conceptual illustration. All worked calculations are teaching examples, not telemetry from an actual nextM365 agent.

1. Start With the Unit, Period, and Business Question

Before interpreting a number, write down what it counts and which interactions it includes. A count of sessions is different from a count of people. A percentage for engaged sessions is different from a percentage for all sessions. A trend covering a launch campaign is different from an ordinary operational week.

Microsoft distinguishes conversations from analytics sessions: one conversation can contain multiple sessions. Its Monitor overview also explains that analytics timestamps use UTC and test-panel activity does not populate Monitor.

1. Choose the agent and the task you expect it to support.
2. Open its Monitor or Analytics experience and select a reporting period.
3. Record the unit, filters, timezone, and relevant release or campaign dates.
4. Read the chart’s definition before deciding which denominator to use.

For example, an IT support agent might be assessed on whether people receive a usable route to restore access. A high interaction count matters less if those people repeatedly ask the same unresolved question.

2. Total Sessions: How Much Activity Is Taking Place?

Total sessions help you understand activity over time. Use them to spot demand, compare the reporting scope, and decide whether there is enough evidence for a useful review. Do not label a session count as the number of unique users or successful tasks.

Illustrative count: 1,250 total sessions tells you that the selected group contains 1,250 recorded sessions. It does not say that 1,250 different people used the agent, or that 1,250 issues were resolved.

• More sessions after an announcement may reflect increased awareness.
• Repeated sessions around one request may indicate persistent friction.
• Lower volume could reflect a quieter business period or an entry point that users struggle to find.

Keep volume beside every percentage. A rate based on ten sessions can move sharply when one outcome changes; a larger group provides a different level of evidence.

3. Engaged Sessions: Are Interactions Progressing?

Engagement helps distinguish activity that progresses into a substantive interaction from activity that remains unengaged. Microsoft’s engagement classification depends on orchestration mode and the topics, knowledge, plans, or tools involved; use its current engagement definitions.

For a teaching group of 1,250 total sessions with 950 engaged sessions, the engagement proportion is 950 ÷ 1,250 × 100 = 76%. The remaining 300 sessions are unengaged within this example.

This is a signal to investigate, not a measure of answer quality. If engagement is unexpectedly low, review the greeting, the entry point, and whether users understand the supported tasks. Compare similar reporting groups before changing the agent.

4. Read the Four Outcome Categories Together

Category	Interpretation	What to check
Resolved	A successful session ending, classified as confirmed or implied.	Whether representative answers address the task using approved information.
Escalated	An escalation or transfer action ended the analytics session.	Whether help was appropriate and the receiving support process worked.
Abandoned	An engaged session ended by timeout without resolution or escalation.	The point where guidance, an action, or the next step stopped helping.
Unengaged	The session did not progress into engagement.	Entry-point relevance and clarity about the agent’s capabilities.

The conversation-outcome documentation describes the product’s classification rules. The table is a starting point for review, not an independent verification of task completion.

An agent can end a session with a plausible but outdated answer. It can also escalate a sensitive request correctly. Sample the affected interactions before labeling either outcome a success or a failure.

5. Use a Consistent Cohort for Outcome Rates

The following is a single, internally consistent illustrative dataset. It combines the lesson’s session-count themes into one worked calculation with a clearly stated eligible group.

Measure	Example count	Calculation	Result
Total sessions	1,250	All sessions in this teaching group	1,250
Engaged sessions	950	950 ÷ 1,250 × 100	76% of total sessions
Unengaged sessions	300	300 ÷ 1,250 × 100	24% of total sessions
Resolved sessions	720	720 ÷ 950 × 100	75.8% of engaged sessions
Escalated sessions	120	120 ÷ 950 × 100	12.6% of engaged sessions
Abandoned sessions	110	110 ÷ 950 × 100	11.6% of engaged sessions

The engaged outcomes reconcile: 720 + 120 + 110 = 950. Adding the 300 unengaged sessions gives the total of 1,250. Rounding explains minor differences when percentage totals are displayed with limited decimal places.

Outcome rate = outcome count / selected eligible session count × 100
Resolved example: 720 / 950 × 100 ≈ 75.8%

If you use all 1,250 sessions instead, 720 resolved sessions represent 57.6% of that total. Both calculations describe this example, but they answer different questions. Label the denominator and use the dashboard’s definition when reproducing its metric.

6. Resolution Rate: Did the Intended Task Progress?

A separate small example is 80 resolved sessions out of 100 eligible sessions, giving an 80% resolution rate. That is a different group from the 950 engaged sessions above; do not mix the two examples.

Review the resolved count alongside the kinds of requests being handled. Straightforward policy questions and unusual approval exceptions create different workloads. A higher rate could reflect easier questions rather than improved handling of the original problem.

Use representative questions to verify correctness. For a policy agent, check that the answer points to the approved current policy. For a support agent, check that the next step is usable and permitted. Keep the outcome metric alongside those checks.

7. Escalation Rate: When Is Human Help Appropriate?

For another independent teaching example, 15 escalated sessions out of 100 eligible sessions gives a 15% escalation rate. The percentage alone does not explain why those handoffs occurred.

• Complex request: clarification or specialist judgment may be needed.
• Missing knowledge: the approved answer may be absent or difficult to retrieve.
• Unsupported action: the agent may be unable to complete the requested task.
• Intended boundary: human approval or a user’s request for help may justify escalation.

Look for repeated unnecessary handoffs and verify that the intended support destination works. Removing a handoff merely to lower the metric can leave users without a way to finish their task.

8. Abandonment: Investigate the Point of Friction

Unexpected endings deserve investigation, but a chart cannot tell you the user’s reason for leaving. The lesson highlights possible causes such as an unhelpful answer, a long interaction, or disengagement. Treat these as hypotheses to test.

1. Choose an authorized sample from the relevant group.
2. Identify the intended task and the last useful step.
3. Check for repeated clarification, missing content, failed actions, or an unclear next step.
4. Write a reproducible question and the behavior you expect instead.

Use a bounded sample and record how you selected it. A few reviewed interactions can reveal a concrete defect, but they do not establish what every user experienced. Microsoft describes the supported session and question drill-down views for deeper investigation.

9. CSAT: Separate an Average Rating From a Percentage

Microsoft defines CSAT as an average score out of five from responding sessions. See its satisfaction documentation for the current display and categories.

Illustrative calculation: if ten ratings sum to 43, their average is 43 ÷ 10 = 4.3 out of 5. Dividing 4.3 by 5 gives 86% of the scale maximum; it does not mean that 86% of respondents, or all users, were satisfied.

A satisfaction percentage needs a separate rule defining which ratings count as satisfied and the number of responses meeting that rule. Keep the response count alongside either measure. A score based on a handful of respondents does not describe the views of people who did not answer the survey.

Read comments for specific repair ideas. “The link is out of date” points toward content maintenance. “It keeps repeating the same question” suggests a flow or clarification issue. The required change is not always an instruction rewrite.

10. Duration: Is the Interaction Useful or Just Short?

If your report or authorized session data includes duration, use it to investigate the effort required by a task. A short interaction can mean a quick useful answer or an early exit. A longer interaction can mean a complicated task or repeated friction.

Illustrative average: five durations of 120, 180, 240, 300, and 510 seconds sum to 1,350 seconds. Their average is 270 seconds, or 4 minutes 30 seconds. The middle value is 240 seconds; the long final interaction pulls the average upward.

Keep the units visible when exporting or calculating. Also check whether the duration includes waiting or inactivity. Compare similar tasks and outcomes before describing a shorter average as an efficiency improvement.

11. Look Beyond One Metric

Pattern	Possible interpretation	Next investigation
More activity with more repeated questions	Demand increased, but a recurring need may remain unmet.	Group repeated intents and review coverage.
High resolution with poor feedback	An ending may look successful while the experience remains unhelpful.	Review answer quality and the actual next step.
More escalation for approval requests	A supported human boundary may be functioning as intended.	Check the receiving approval process.
Longer duration with repeated clarification	The flow may be asking for information inefficiently.	Repeat the question and inspect the clarification steps.
Shorter duration with more abandonment	Users may be leaving sooner rather than completing tasks faster.	Inspect unfinished interactions before claiming an improvement.

These are investigation prompts, not conclusions. Check the reporting period, sample size, request mix, and recent releases before deciding what changed.

12. Turn the Review Into One Testable Action

Suppose employees repeatedly ask where to find a departmental onboarding checklist. A review shows a generic answer, repeated clarification, and no usable departmental link. The next action could be a content correction and a clearer question about the department.

1. Record the metric pattern and the observed issue separately.
2. Assign the correction to the content or application owner.
3. Test a named department, a paraphrased request, and a request missing the department.
4. Retest a previously working question to check for regressions.
5. Review a comparable period after release, using the same units and definitions.

If the rate improves but the original failure question still receives an unusable answer, keep the issue open. Use functional tests to establish behavior and published-agent data to understand operational usage.

13. Build a Metric Dictionary Before Exporting Data

A metric dictionary is a short agreement about what your report means. It prevents a reviewer from comparing an engaged-session rate in one report with a total-session rate in another. Keep it beside the spreadsheet or review notes rather than relying on memory.

Field	What to record	Illustrative entry
Business question	The task or behavior you are investigating.	Are employees finding the correct onboarding checklist?
Unit	Sessions, conversations, users, responses, or seconds.	Engaged analytics sessions.
Population	The included group and relevant filters.	Sessions in the chosen reporting period; exclusions documented.
Numerator	The event or outcome being counted.	Resolved sessions.
Denominator	The eligible group used for the rate.	All engaged sessions in that same group.
Period and timezone	Start, end, and boundary convention.	A complete seven-day period, with the report timezone stated.
Context	Releases, campaigns, or request changes.	A departmental checklist was updated during the period.
Limitation	What the measure cannot establish.	A resolved outcome does not independently verify answer accuracy.

When two reports disagree, compare their dictionaries first. Different scopes can produce different valid results. Do not change a formula just to make a manual report match a dashboard whose definition you have not checked.

14. Reproduce the Worked Example in a Spreadsheet

Enter the following teaching values in column B. These formulas describe the example cohort in this article; they are not a promise that every product dashboard uses the same eligible group.

Cell	Input	Value
B2	Total sessions	1,250
B3	Engaged sessions	950
B4	Resolved sessions	720
B5	Escalated sessions	120
B6	Abandoned sessions	110
B7	Unengaged sessions	300

Engagement proportion: =IF(B2=0,"",B3/B2)
Resolution proportion: =IF(B3=0,"",B4/B3)
Escalation proportion: =IF(B3=0,"",B5/B3)
Abandonment proportion: =IF(B3=0,"",B6/B3)
Engaged outcomes reconcile: =B4+B5+B6=B3
Total sessions reconcile: =B3+B7=B2

Format the proportion cells as percentages. Do not also multiply the formula by 100 when using percentage formatting: the stored value 0.76 should display as 76%, rather than 7,600%.

The IF expressions leave a rate blank when the denominator is zero. Label that result as unavailable or not applicable in your report; it does not mean that the rate was 0%. Likewise, an unavailable feedback field should not be silently converted to a zero rating.

The reconciliation checks should return TRUE for this teaching dataset. If a real export does not reconcile, first check its scope, filters, category definitions, and update timing. Preserve the raw values while you investigate rather than forcing the totals to agree.

15. Distinguish Percentage Points From Relative Change

Illustrative before-and-after comparison: the baseline contains 720 resolved sessions out of 950 eligible sessions. A follow-up contains 800 resolved sessions out of 1,000 eligible sessions. The resolution rates are approximately 75.8% and 80%.

Baseline rate = 720 / 950 ≈ 75.7895%
Follow-up rate = 800 / 1000 = 80%
Percentage-point change = 80 - 75.7895 ≈ +4.21 points
Relative change = (80 - 75.7895) / 75.7895 × 100 ≈ +5.56%

“Up 4.21 percentage points” and “up about 5.56% relative to the baseline” describe different calculations. State which one you mean, and retain the underlying counts. The difference between 720 and 800 resolved sessions also reflects the changed group size.

This comparison alone does not establish that an agent change caused the increase. Check the types of requests, campaign activity, reporting boundaries, and known failure questions before attributing the change to a release.

16. Check Request Mix Before Judging the Overall Rate

An overall rate can decline even when individual groups are stable or improving. Consider this illustrative request mix, using two groups with the same definitions in both periods.

Group	Baseline resolved / eligible	Baseline rate	Follow-up resolved / eligible	Follow-up rate
Routine questions	720 / 800	90%	180 / 200	90%
Complex requests	100 / 200	50%	480 / 800	60%
Combined	820 / 1,000	82%	660 / 1,000	66%

Routine handling stayed at 90%, while complex-request handling increased from 50% to 60%. The overall rate still fell because the follow-up contained many more complex requests. Declaring that the agent became worse based on the combined rate alone would overlook that change in workload.

Where your authorized data supports it, compare relevant request groups separately and state how the overall mix changed. Avoid splitting a small sample into so many groups that every percentage becomes unstable.

Aggregate counts before calculating an overall outcome rate. The simple average of 90% and 50% is 70%, but the baseline overall rate is 820 ÷ 1,000 = 82% because the groups have different sizes.

The same principle applies to combining survey averages. If one group has 50 ratings totaling 240 points and another has five ratings totaling 10, their combined mean is 250 ÷ 55 ≈ 4.55. Averaging the two group means, 4.8 and 2.0, would incorrectly give 3.4.

17. Troubleshoot a Confusing Report and Write an Honest Summary

Symptom	First checks	Useful next action
Your manual rate differs from the dashboard.	Denominator, filters, units, rounding, and reporting period.	Write both definitions and compare the included groups.
Outcome counts do not add up.	Whether the categories cover the same population and update period.	Retain the discrepancy and resolve it before publishing a reconciled total.
A trend changes sharply after a campaign.	Volume and the mix of incoming requests.	Separate demand changes from evidence about answer handling.
CSAT rises while the original issue persists.	Response count, respondent mix, and the affected task.	Repeat the known failure question and inspect relevant feedback.
Average duration falls alongside early exits.	Outcomes and representative unfinished interactions.	Check whether the flow became clearer or users stopped sooner.

Use a concise review note that keeps observation, interpretation, and action distinct:

Observation: State the counts, rate, group, and reporting period.
Interpretation: Describe a plausible explanation and its limits.
Evidence: Record the relevant test result or de-identified interaction.
Action: Name one focused change and its owner.
Verification: Repeat realistic questions and compare a defined follow-up.
Unknowns: State what you could not establish from the available data.

A useful report can conclude that the evidence is insufficient for a broad claim. The next step might be a focused test or a better-defined comparison, rather than a larger agent rewrite.

Day 77 Practice: Build a Metrics Dictionary

Choose one agent and one reporting period. Create a short dictionary for five metrics, then investigate one pattern.

Metric name and business question:
Unit and reporting period:
Numerator and denominator, if a rate:
Filters and sample size:
Observed value:
What this metric does not establish:
Related feedback or reviewed session:
Proposed change and owner:
Test questions and follow-up comparison:

• Distinguish sessions, conversations, and users.
• Record total, engaged, and outcome counts for a consistent group.
• Verify the arithmetic and label each denominator.
• Keep the CSAT response count and duration units visible.
• Review one relevant pattern using authorized information.
• Propose one change with realistic test questions and a follow-up period.

Today's takeaway: metrics work better together. Understand the count, the scope, and the limitation of each number, then connect the evidence to a specific improvement. Revisit Day 76's analytics review process or explore the 365 Days of Copilot Studio series.