Gate Copilot Studio agent merges with automated Azure DevOps evaluations

Schematic view of the steps described by The Custom Engine, not a screenshot.

TL;DR

The Copilot Studio Evaluation API runs test sets against draft agents and returns metric-level results. The EvalGateADO pipeline packs the PR source with pac, imports it into a dedicated CI environment, evaluates the draft, and blocks the Azure DevOps merge when the configurable pass-rate threshold is not met.

Original by Adi Leibowitz, on The Custom Engine. Read the original

This is our own summary, not a republication or full translation.

Governance takeaway

• Makers: Commit the agent source and test sets with each change, because the pipeline evaluates the PR branch rather than uncommitted sandbox changes.
• Admins/CoE: Provide a dedicated CI environment and configure the pipeline import process, so automated tests run against a consistent runtime target.
• Security/Compliance: Protect the delegated authentication refresh token in Key Vault and monitor its inactivity-based expiry, because the Evaluation API has no app-only authentication path.
• Leadership/Business: Require the evaluation gate before merging agent changes, so quality failures are identified during review instead of after production release.