Copilot Evals: test one real task before you rewrite your prompts again

You improve an AI instruction, try it once, and get a better answer. Did the change help, or did this run simply go your way? That is the everyday problem behind an intimidating word: evaluation. A useful evaluation is a task you can repeat and a result you can judge.
GitHub announced Copilot Evals in public preview on November 17. It lets users create tests for skills, custom agents and MCP tools, inspect checks and compare runs after a customization changes. Each run requires approval. GitHub lists the preview for Copilot Individual, Business and Enterprise, with installation routes for VS Code, Copilot CLI and the Copilot app. Release announcement
The interesting part for a small project is not a giant leaderboard. It is the opportunity to stop judging your setup by whichever answer you happened to see last.
Choose a task with a visible answer
Here is our suggested rehearsal. Give an assistant a short, invented project note containing three tasks, one deadline and one deliberately unresolved decision. Ask it to produce a handoff for tomorrow.
Before running it, write down what must survive: all three tasks, the exact deadline and the fact that the decision is still open. Also write down what must not appear: an invented owner, a made-up deadline or a claim that the unresolved decision has already been settled.
That small example tests a real annoyance: a tidy summary that quietly changes the meaning. It is easier to judge than a broad instruction such as make this better. Use synthetic notes rather than private customer material for the first experiment.
Change one thing, then compare
Run the same task several times before editing your instructions. Keep the inputs and checks fixed. Then make one targeted change, such as explicitly telling the agent to preserve unresolved questions, and repeat the task.
Count the failures that matter to you, not just how pleasant the wording sounds. A shorter handoff that loses the deadline is not an improvement. Nor is a perfect-looking answer worth much if the assistant used a tool you did not intend to involve.
These are suggested test-design habits, not reported results from our own benchmark. A handful of runs can reveal a problem; it cannot establish that an assistant will behave perfectly on every future task.
Keep the test after the fix
GitHub says the preview uses the local evaluation engine Vally and supports asking in natural language or using /eval. Its announcement positions comparisons as a way to catch regressions when prompts, tools or other customizations change. What the preview does
Save the original task and its checks alongside the project. Next time you change your instructions, you have something better than memory to compare against. Add another test when a real failure teaches you something new, rather than building a huge suite before you know what breaks.
If your assistant also acts on websites, pair this with our browser-use rehearsal. For broader safeguards, see the seven failure checks. The aim is not a flattering score. It is fewer surprises on work you actually care about.
Announcement checked November 17, 2026. We have not hands-on tested this preview; linked product documentation could not be retrieved during this research pass.