Post

Copilot Studio Model Assessment: test agents before a model switch

Diesen Beitrag auf Deutsch lesen

Model Assessment in Copilot Agent Kit compares an agent across models using shared evaluation checks, model history and reported Copilot Credits before a model change.

Schematic view of the steps described by The Custom Engine (Microsoft Copilot Studio CAT team blog), not a screenshot.

TL;DR

Model Assessment in Copilot Agent Kit tests a Copilot Studio agent on evaluation test sets before a model switch. The example compares GPT-6 Luna, 14 passed checks, with GPT-6.1 Sol, 12, recommending Luna at Medium confidence while both runs stay Critical. Credits appear per run and per test case, a partial flag blocks savings claims, and AI analysis consumed around 120-150 Copilot Credits per assessment.

Original by Ravikiran Patil, on The Custom Engine (Microsoft Copilot Studio CAT team blog). Read the original

This is our own summary, not a republication or full translation.

Governance takeaway

  • Makers: Capture a baseline assessment before any model change, keep instructions, knowledge, tools and test sets stable, and reproduce changed checks in a development or test environment, because a switch made through Model Assessment is saved to the source agent and the Kit path publishes it.
  • Admins/CoE: Budget around 120-150 Copilot Credits per assessment for AI analysis, note that Kit tests report no credits, and resolve partial consumption reporting before any cost-based model decision.
  • Leadership/Business: Review the model recommendation together with its confidence level and the PDF export, and remember both example runs stayed Critical, so a preferred model still leaves failures to fix.

Frequently asked questions

How do I set up Copilot Studio evaluations for Model Assessment?

Create evaluation test sets whose names include MODELCHECK, keep them active, and capture a baseline first. The tool sits under Advanced Testing in Copilot Agent Kit.

What does a Model Assessment cost in Copilot Credits?

Observed AI analysis used around 120-150 Copilot Credits per assessment, separate from the answering and scoring credits the test cases report. Kit tests report no credits.

Why does Model Assessment show a recommendation when the run status is Critical?

The recommendation compares the two selected runs, while Critical is each run's own status. In the example the preferred model still had four failing classification checks.

Summary by KMUpower, licensed under CC BY 4.0 . Rights to the original article remain with its author.

Privacy Policy · Legal notice

Microsoft and the Power Platform product names (for example Power Apps, Power Automate, Copilot Studio) are trademarks of the Microsoft group of companies. This is an independent website. It is not operated, sponsored or endorsed by Microsoft.