In March 2026 work started on a test system for the Peakboard Designer, a Windows Presentation Foundation (WPF) application. An AI agent, Claude Code, operates the user interface (UI) through Windows UI Automation like a human tester, judges the result and reports every bug it finds. Everything that has to be repeated runs without a language model. A separate runner executes JSON test scripts.
The Peakboard Designer is the editor for boards, the displays that Peakboard shows on a screen. It has a canvas, a toolbox with the controls, property dialogs and its own project file format. The screenshot below shows a board with six stations on the canvas and the properties of a text control on the right.
The test system is internal tooling and not part of the product. The Designer is the application under test.
The idea
Agentic means that a language model calls a tool, reads the result and decides the next step on its own. A tool is a function that another program offers to the model, for instance a click on an element or reading a value. The model does everything that needs understanding. It explores an unknown dialog, chooses the next click, checks that a value is correct and writes the bug report. A runner without a model does everything that has to be repeated in the same way every time, for instance a nightly regression suite.
The code of the system never calls a model. Only Claude Code contains a model. It runs as a separate process with a prompt and a set of tools. The rest is .NET, Markdown and JSON.
The diagram below shows the parts and the connections between them.
A tester writes a request in Slack. A bot then starts Claude Code with the prompt for testing and the text of the request. A card on a Kanban board, a task board with columns, starts Claude Code as well. A run is one such start of Claude Code, from the first step to the result line that ends every run. Claude Code starts the MCP server as a child process, reads its notes and drives the application through the tools of the server.
When a flow has to be repeated later, Claude Code writes a JSON test script and starts the runner with it. The report and the findings, the reported problems of the application, go to Slack. The agent adds what it has learned to its notes.
The MCP server
The Model Context Protocol (MCP) is the open standard a model uses to call tools in another program. The server is a .NET 8 console process. Claude Code starts it as a child process and sends JSON-RPC messages over stdin. The answers return over stdout, one message per line.
The server implements three methods, initialize, tools/list and tools/call. The message loop is hand-written without a software development kit (SDK), because the protocol is small.
The tools use Windows UI Automation, the interface of Windows through which a program reads and operates the user interface of another program. Every window, button and text box is an element with properties, and the elements form a tree, called the UI tree in this article. The server uses System.Windows.Automation, the managed UI Automation API of .NET. An environment variable switches to the library FlaUI instead, which uses the newer UI Automation interface based on the Component Object Model (COM).
The server offers more than 30 tools in four groups:
- Reading tools for the UI tree, one element, its text or a property, and the list of open windows
- Action tools for click, double click, right click, hover, typing, key strokes, list selection and drag and drop
- Assertion tools for exists, visible, enabled, value and text
- Tools for waiting, screenshots, video recording, building the application and committing to Git
The element search tries the AutomationId first. This is a fixed ID that the developer sets on a control. If it is missing, only the Name property remains.
The model never sees screenshots and never sees the whole UI tree. That is the central decision of the design. Both would fill the context of the model with data it does not need for the next step. Instead, a click can return a diff of the tree, the elements that appeared, disappeared or changed, at most 30 of each. Every click also returns a report of the windows that opened or closed. The following JSON shows the answer to a click on the File menu:
{ "success": true, "automationId": "MainMenu_File", "executionMs": 2840,
"diff": { "appeared": ["FileMenu_New", "FileMenu_Open"], "disappeared": [], "changed": [] },
"activeDialog": null, "windows": { "count": 1, "opened": [], "closed": [] } }The field executionMs holds the response time in milliseconds. The value is high because the diff was switched on for this click. Under diff are the AutomationIds of the two menu items that appeared after the click. The field activeDialog is null because the click opened no dialog.
The diff needs two snapshots of the tree, one before and one after the click. That raises the response time from about 200 to about 3000 milliseconds. So the diff is off by default. The model enables it with a parameter when it needs the changes of a click. The tool description tells the model both times, so that it enables the diff only when needed.
The window report is part of every answer, because the model must never miss a new dialog. Full trees and screenshots go to disk. The tool returns the file path, never the content.
The agent is a set of Markdown files
There is no AI code in the repository. The agent is Claude Code with Markdown prompts. A role is a Markdown file with instructions, called a command in Claude Code. When the prompt starts with the name of the file, for instance /test for the test role, Claude Code loads the file as the instructions for the run. Further roles write a concept, implement a task or write documentation.
The test prompt has about 250 lines. It contains two principles, five rules and the format of the result line. The principles override every other rule. The details live in reference files, one per topic, for instance how to use the notes, bug hunting, live testing or script authoring. Claude Code reads a reference only when the task needs it.
The first principle is live first. Live means that the agent drives the application directly through the MCP tools and checks the result after every step itself. That is the default. The agent writes a JSON test script only when a flow has to run again later, for a nightly run, a regression suite or a permanent reproduction of a bug.
The second principle is memory first. The agent reads its notes before the first click. When a step fails, it searches the notes for the symptom instead of repeating the call.
The test prompt has five rules:
- A round trip for every value (set a non-default value, save, reopen, read it back)
- No guessed AutomationIds, only IDs from the current UI tree
- A lookup in the notes on every failure instead of a retry
- A fix in the source for a missing AutomationId, no workaround with name or coordinate
- A finding for every bug in the application, even when the test passes
The first rule comes from a bug. A port number in a connection dialog returned to its default value after saving and reopening the file. The test found the bug only because it used a non-default value, saved, reopened and checked the stored value. If one of these four steps is missing, the bug stays invisible.
Memory, feedback and glossary
Claude Code starts every run without memory of earlier runs. Without notes the agent would have to find every known problem of the application again on every run. The notes are Markdown files in the repository, around a thousand at the moment.
A table of contents, called the router, lists about 30 topics with one line each. A topic file lists its notes with one line each and links to related topics. The agent reads the router, opens one to three topics and follows at most one link. When something fails, it searches the notes by symptom, for instance “value reverts”, not by the name of the feature.
There are three kinds of notes. An error note describes a failed step, its cause and the step that works instead. A workflow note holds a verified sequence of steps. A verify note holds the result of checking a feature or a task.
One example is an error note about a combo box. Pressing Enter there closed the whole dialog, because WPF sends the Enter key to the default button of the dialog. The note names the step that works: press the Down key to accept the autocompletion and then click the OK button.
Besides the notes the agent reads two more sources, the feedback and the glossary. The feedback contains two dozen rules from testers and developers. A tester adds a rule in Slack with a message that starts with “remember”. Every rule has a scope, for instance test or documentation. Every run reads the rules, applies them and lists the numbers of the applied rules in its report.
The glossary lists the allowed terms with a definition and the wrong terms with their replacement, per language. Every role reads it before it writes text for people, for instance a bug report or a help article.
The script runner
The runner is a console program that executes JSON test scripts. It uses the same element search and the same click code as the MCP server. So a click from the model and a click from a script run through the same code.
A script has a setup that brings the application into its start state, the steps, a teardown that cleans up afterwards, variables and imports. There are twelve step types, for instance action, assert, wait_for, store, if, for_each and run for a sub script. A schema file defines the format. An unknown step type is rejected when the script is loaded.
An action step may declare the elements that must appear or disappear in the tree. Without that declaration, a double click on a toolbox icon counts as passed even when nothing happens. With it, the step fails when nothing appears or disappears.
After every step the runner runs seven state checks, called test oracles in the testing literature. A state check looks at the state of the application, independent of the step itself. The seven checks detect the following cases:
- The process of the application has exited
- A window does not answer a message within a timeout
- A new window appeared, reported by a WinEvent hook (a notification from Windows for every new window), even when it closed again
- A new window appeared, found by polling the window list at regular intervals when the hook fails
- A label with an error message is visible in the window
- The log file of the application has new error lines
- The Windows application event log has new entries from the application
A crash dialog is recognized by its content, not by its title. The WPF dialog for an unhandled exception carries the normal title of the application. So the runner reads the text of a new window. It searches for an exception type, a stack trace line or the two buttons to continue or quit.
The runner writes an HTML report with a table of steps and screenshots, a GIF from the screenshots and one row per run in a SQLite database. Every finding gets a fingerprint, a hash over the name of the state check, the cleaned window title and the first stack trace line. The same fingerprint means the same finding. So a crash that ten scripts find is one problem, not ten.
There is also a random walk, a script of operations picked at random. An operations file lists four operations that each return the application to its start state, so they can follow each other in any order, for instance create a text block and undo. A generator picks 40 of them at random and writes a normal script. A seed, the start value for the random numbers, fixes the selection. Three walks run every night. The same seed produces the same script, so a crash from a nightly run is reproducible.
Missing AutomationIds
UI Automation finds an element through its AutomationId, which the developer sets in the Extensible Application Markup Language (XAML). Without it, only the name or a coordinate remains. Both are unreliable, because the name changes with the language or the content and the coordinate with the window size. The test prompt forbids both and treats a missing ID as a bug in the application.
A second agent, called a subagent, adds the missing IDs. The main agent starts it with a task, and when the task is done the subagent returns with a result. A tool of the MCP server returns the task text ready-made. The subagent has no UI tools, only file tools. It finds the element in the XAML or C# source, adds the AutomationId with a name built from area, control type and purpose, and changes nothing else.
The subagent fixes at most 50 elements per run. After it returns, the main agent builds the application with the build tool, checks the new IDs in the UI tree of the new build, writes a note and commits.
Slack and the Kanban board
A bot connects Slack to the test machine. It is a .NET service without a model and uses the SlackNet library in Socket Mode. In this mode the bot opens the connection to Slack itself, so it needs no public endpoint. It checks the sender against a list of allowed users and takes the first word of the message as the command. Then it starts Claude Code in headless mode, that is without an interactive terminal, with the command and the request text, and reads the output as a JSON stream.
From the stream the bot reads the session ID, the key under which Claude Code stores the run. Every four seconds it posts the new tool calls and messages of the agent to Slack. The last line of a run is the result line with the path of the HTML report or an error text. The bot uploads the report and the GIF and stores the session ID per channel.
A later message in the same channel continues the same session. For this the bot starts Claude Code with the stored session ID and the resume flag. So a tester can ask a question about step 12 of the report and get an answer with the full context of the run.
Only one run happens at a time because only one instance of the application can run on the desktop. Further requests wait in a queue. The bot runs as a scheduled task at logon, not as a Windows service, because UI Automation needs a logged-in user with a visible desktop.
In July 2026 the Kanban board became the second way to start a run. A card on the board is a task. For every card the board starts Claude Code in a role, first to develop and then in a second run to test. The test run does not rely on checks of the development run. One line in the test prompt says that a card that has just been developed counts as untested.
Limits
Four places are kept simple on purpose:
- In live mode no state check runs. Only the agent decides whether there is a bug.
- The random walk picks from only four operations. So it covers a small part of the Designer.
- The Slack bot keeps the session per channel, not per thread, and only in memory. After a restart every channel starts again.
- The notes are text files. The agent finds them with the router and grep, without embeddings, fine-tuning or a second model call.
A second article describes a smaller version of this system, with code to try out.
That’s it.
If you have questions feel free to contact me.
Links:
Model Context Protocol →
Claude Code, run programmatically →
Claude Code, skills and commands →
Claude Code, subagents →
UI Automation overview →
FlaUI →
SlackNet →