Same score can hide opposite Copilot Studio outcomes
Diesen Beitrag auf Deutsch lesen
A governed Copilot Studio design uses a curated SharePoint library, named owners, Power Automate lifecycle controls, and repeated evaluations.
TL;DR
Use two SharePoint libraries so a Copilot Studio agent can read only approved files in the Curated Library. A named owner controls a Yes/No approval field, while Power Automate copies or removes files and monthly audits flag outdated, orphaned, and departed-owner content. Repeat evaluations because one unchanged agent moved from 24 to 31 passes out of 43.
Original by Karl-Johan Spiik, on Karl-Johan Spiik, Microsoft MVP. Read the original
This is our own summary, not a republication or full translation.
Governance takeaway
- Makers: Keep the agent limited to the Curated Library and treat refusal as the correct response when an approved source cannot support an answer.
- Admins/CoE: Use the owned Yes/No approval field, Power Automate boundary flow, and monthly Teams audit cards to identify stale, orphaned, and departed-owner content.
- Security/Compliance: Require named accountability and content lifecycle checks because approval alone can still admit superseded, draft, or contradictory material.
- Leadership/Business: Review which questions pass, not only the overall score, because equal scores can represent opposite quality outcomes.
Frequently asked questions
Why can a good evaluation result still wave through wrong agent answers?
Because the general quality grader measures relevance, coverage and grounding, and does not compare the response with the expected answer.
What did comparing two test runs with almost the same score show?
The first run scored 7 out of 20 although four documents should never have been approved and three correct ones were missing. A second run scored about 6 and refused drafts correctly.
Why does a control run matter when testing a Copilot Studio agent?
A control run repeats the evaluation without changing the agent. On a production agent the score moved from 24 to 31 out of 43 without touching anything.