Measure Copilot Studio Agent Performance with Response Analysis
Diesen Beitrag auf Deutsch lesen
Use the M365 Agents SDK, Python DataFrames and Gradio to measure latency, inspect planner activity and visualize Copilot Studio agent execution.
TL;DR
The Response Analysis tool uses the M365 Agents SDK to send test utterances to a Copilot Studio agent, capture message and event Activities, calculate latency statistics with Python DataFrames, and stream metrics and planner data to a Gradio dashboard. The implementation is limited to single-session analysis and requires secure token handling.
Original by Vineet Kaul, on The Custom Engine. Read the original
This is our own summary, not a republication or full translation.
Governance takeaway
- Makers: Run single-session tests with the Response Analysis tool to isolate telemetry and refine agent instructions using measured behavior.
- Admins/CoE: Verify environment prerequisites before setup and account for the tool's single-session limitation when planning testing workflows.
- Security/Compliance: Keep access tokens out of source control, use environment variables or secure credential stores, and implement token refresh logic.
Frequently asked questions
How does the M365 Agents SDK measure Copilot Studio response latency?
It records timestamps for message Activities and calculates elapsed time, then aggregates mean, median, standard deviation, maximum, and correlation metrics.
What does the Response Analysis tool capture from Copilot Studio Activity events?
It records planner steps, tool calls, and execution details from event Activities, while message Activities provide final response text.
How should Copilot Studio access tokens be stored when using the analysis tool?
Store tokens in environment variables or secure credential stores, never source control, and implement proper token refresh logic.