Recently I collected the AI terms from my posts about CardsToLoop, the Peakboard Designer test system and the self-healing loop. This first part explains the terms around models, prompts, tools and agents, each in a few sentences and with a sample of 5 to 25 lines. Most C# samples come from the CardRunner of CardsToLoop and use Microsoft.Extensions.AI.
Every entry starts with a definition sentence. Where a prompt, an answer or a sequence of steps shows the term better than code, the sample is text. The glossary is not a tutorial. The posts about CardsToLoop with Claude Code, CardRunner, the Peakboard Designer test system and the self-healing loop for a new Peakboard runtime show the terms in use. The second part covers retrieval, evaluation and the frameworks.
The three groups of this part and their terms:
- Models
- Prompts
- Tools and agents
The two groups of the second part and their terms:
- Retrieval and evaluation
- Frameworks and tools
Models
Large language model (LLM) and generative AI
A large language model (LLM) is a neural network trained on text that predicts the next token of a given text. An answer is a long chain of such predictions. Generative AI is the general term for models that produce text, code or images instead of a classification. The following lines show two prompts and typical answers:
Prompt: Which WPF class shows a modal dialog with Yes and No buttons?
Answer: MessageBox. The call MessageBox.Show(text, caption, MessageBoxButton.YesNo)
returns MessageBoxResult.Yes or MessageBoxResult.No.
Prompt: Write the C# for the delete button.
Answer: var result = MessageBox.Show($"Delete {item.Description}?", "Delete item",
MessageBoxButton.YesNo, MessageBoxImage.Question);
if (result != MessageBoxResult.Yes) return;The model writes code the same way as prose, token by token, without a compiler in the loop. The code above is right because the model has seen many Windows Presentation Foundation (WPF) dialogs, not because it checked anything. The model has no memory between two calls. The second prompt works only because the first exchange is sent again with it (see the entry about the system prompt and the user message below).
Token and context window
A token is a piece of text of about four characters, a short word or a part of a longer one. The model reads and writes tokens, and the API bills them. The context window is the number of tokens a model reads in one call, 1 million for the current Claude models. The following lines show a sentence as tokens and the numbers of one run of CardRunner:
"The Delete selected button removes the row immediately."
The | Delete | selected | button | removes | the | row | immediately | .
about 10 tokens, the exact split depends on the model
Run of the delete card: round 1 reads 3,270 input tokens, round 20 reads 19,360
20 rounds together: 254,762 input tokens, 4,306 output tokens, 59 seconds
Claude Sonnet 5.5: 2 dollars per million input tokens, 10 dollars per million output tokensThe input of a round is the whole conversation so far, so the count grows with every round. Twenty rounds read the card and the earlier tool results 20 times. The following class of CardRunner runs below the loop and counts the tokens of every round:
public sealed class RoundLogger(IChatClient inner, long maxTokens) : DelegatingChatClient(inner)
{
public int Rounds { get; private set; }
public long InputTokens { get; private set; }
public long OutputTokens { get; private set; }
public override async Task<ChatResponse> GetResponseAsync(IEnumerable<ChatMessage> messages, ChatOptions? options = null,
CancellationToken cancellationToken = default)
{
Rounds++;
var response = await base.GetResponseAsync(messages, options, cancellationToken);
InputTokens += response.Usage?.InputTokenCount ?? 0;
OutputTokens += response.Usage?.OutputTokenCount ?? 0;
if (InputTokens + OutputTokens > maxTokens)
{
throw new InvalidOperationException($"Token limit of {maxTokens} reached after round {Rounds}.");
}
return response;
}
}Every response carries the token counts of its call in the property Usage. The class sums them and stops the run at 400,000 tokens, so a run that never ends leaves a failed card. CardRunner also cuts long tool results (see the entry about context engineering).
Multimodal
A multimodal model takes images, and sometimes audio or documents, as input besides text. For a desktop application that means the model can read a screenshot. The following code sends the screenshot of a dialog with a question:
var image = await File.ReadAllBytesAsync("after.png");
var message = new ChatMessage(ChatRole.User,
[
new TextContent("Does the dialog name the item to delete? Answer with yes or no and the name."),
new DataContent(image, "image/png"),
]);
var response = await client.GetResponseAsync([message]);The class DataContent carries the bytes and the media type, and one message can hold several texts and images. An image costs tokens like text. The documentation of Claude gives the formula width times height divided by 750, so a screenshot of 1,280 by 800 pixels costs about 1,400 tokens.
The self-healing loop for the new Peakboard runtime gives Claude Code three images per board this way, the reference, the actual screenshot and the diff image. CardRunner takes screenshots as evidence but reads values through Windows UI Automation. A number read from an element is exact. A number read from an image is not.
Hallucination
A hallucination is an answer that sounds right and is wrong, for instance a parameter that does not exist. The model predicts text that sounds right, and without a source it does the same for a missing fact. The following lines show the pattern:
Question: Which parameter of MessageBox.Show in WPF sets the default button?
Answer: The parameter defaultButton of type MessageBoxDefaultButton.
Check: MessageBoxDefaultButton belongs to Windows Forms. WPF takes a
MessageBoxResult as the fifth parameter, named defaultResult.The fix is context. A tool that reads the documentation or the source gives the model the fact, as in CardRunner. The right excerpt in the prompt does the same, see the entry about retrieval in part 2. A verifying step finds the rest, in CardsToLoop the live check of every acceptance criterion.
An agent hallucinates tools as well, a name that is not in its list. The loop class of Microsoft.Extensions.AI returns such a call to the model as an error result by default, so the model corrects itself. The option TerminateOnUnknownCalls ends the run instead.
Deterministic and non-deterministic
A deterministic program gives the same output for the same input every time. A language model does not. The same prompt gets a different answer on the next call. The following lines show the same prompt twice and a test step that stays the same:
Prompt, twice: Name a WPF control for a list of rows with columns.
Answer 1: DataGrid.
Answer 2: The DataGrid control, or a ListView with a GridView as its view.
Script step, the same on every run:
{ "action": "click", "window": "Invoice Editor", "target": "DeleteItemButton" }Both answers are right, and a test that compares them with a fixed string fails every second time. So the Peakboard Designer test system lets the model explore once and write a script in JavaScript Object Notation (JSON). A runner without a model repeats the script. The tools, the runner and the script are deterministic, only the model is not. Earlier models had a temperature parameter to reduce the variation. The Claude 5 models reject it.
Prompts
System prompt and user message
The system prompt is the instruction that applies to the whole conversation. The user message is the task. Both are messages with a role, and the answer comes back as a message with the role assistant. The following code builds the two messages of CardRunner, continues the conversation and lists the roles:
var messages = new List<ChatMessage>
{
new(ChatRole.System, File.ReadAllText("instructions.md")),
new(ChatRole.User, "THE CARD\n--------\n" + card.Text),
};
var response = await client.GetResponseAsync(messages);
messages.AddRange(response.Messages);
messages.Add(new ChatMessage(ChatRole.User, "Which file did you change?"));
response = await client.GetResponseAsync(messages);
messages.AddRange(response.Messages);
foreach (var message in messages)
{
Console.WriteLine($"{message.Role,-9} {message.Text.Length,6} characters");
}The output has five lines, system, user, assistant, user, assistant. The list is the whole memory of the conversation, and every call sends the complete list. The second question works only because the first answer is in the list. CardRunner keeps the system prompt in a file next to the exe, 44 lines, and sends the card as the user message. Tool results get a fourth role, tool, see the next group.
Prompt engineering
Prompt engineering is the writing of instructions so that the model does the task reliably, with rules, a fixed answer format and examples. The following lines show a weak instruction and the lines of the CardRunner prompt that replace it:
Weak: Implement the card and make sure it works.
Better:
4. Start the app again and verify every acceptance criterion of the card live
through the ui_* tools: change values, click, read the results back.
Take an after screenshot. A build that compiles proves nothing.
THE LAST LINE of your final message must be exactly one of:
RESULT: done
RESULT: failed: <one-line reason>The better version names the tools, the order of the steps and the sentence that matters most, “A build that compiles proves nothing”. The last line gives the runner a fixed format to read the result. The following method of CardRunner reads it:
private static readonly Regex _resultLine = new(@"^RESULT: (?<outcome>done|failed)(: (?<reason>.*))?\s*$", RegexOptions.Multiline);
public static RunResult Parse(string text)
{
var matches = _resultLine.Matches(text);
if (matches.Count == 0)
{
return new RunResult(false, "no RESULT line in the final answer");
}
var last = matches[^1];
var done = last.Groups["outcome"].Value == "done";
return new RunResult(done, done ? "" : last.Groups["reason"].Value.Trim());
}A regular expression reads the result, so no code has to understand prose. The last match counts, because the model sometimes quotes the format before it answers. A final answer without the line counts as failed. The prompt and this method are a contract, and a change on one side needs a change on the other.
Context engineering
Context engineering is the choice of what the model sees in a call, how much and in what order. A tool result stays in the context of every later round, so a long result costs tokens until the run ends. The following lines show how the build tool of CardRunner answers:
var relevant = lines.Where(line => line.Contains("error", StringComparison.OrdinalIgnoreCase)
|| line.Contains("warning", StringComparison.OrdinalIgnoreCase));
var summary = exitCode == 0 ? "Build succeeded." : $"Build failed with exit code {exitCode}.";
return summary + "\n" + string.Join("\n", relevant.Distinct().Take(BuildOutputLines));The tool returns a summary line and the errors and warnings, not the full output of the build. The full output would cost tokens in every later round, and the model needs only the lines that name a problem. The following class, a shortened version of the LoggedFunction of CardRunner, wraps every tool and cuts every result:
public sealed class CutFunction(AIFunction inner) : DelegatingAIFunction(inner)
{
private const int MaxResultLength = 4000;
protected override async ValueTask<object?> InvokeCoreAsync(AIFunctionArguments arguments, CancellationToken cancellationToken)
{
var text = (await base.InvokeCoreAsync(arguments, cancellationToken))?.ToString() ?? "";
if (text.Length > MaxResultLength)
{
text = text[..MaxResultLength] + $"\n[cut after {MaxResultLength} of {text.Length} characters]";
}
return text;
}
}The class DelegatingAIFunction passes every call to the inner tool, so one line in the runner wraps the whole tool list. The cut names itself, so the model can ask for less next time, for instance the elements of one window instead of all windows. In the Peakboard Designer test system a click returns a diff of the UI tree instead of the tree.
Structured output
Structured output is an answer in a fixed JSON format instead of prose, so that code reads it without parsing text. The library builds a JSON schema from a C# type and asks the model to fill it. The following code asks for a verdict with two fields:
public sealed record Verdict(
[property: Description("true when the difference between the images is a bug")] bool Bug,
[property: Description("One sentence that names the difference")] string Reason);
var response = await client.GetResponseAsync<Verdict>(
"Compare reference.png and actual.png. Is the difference a bug?");
if (!response.TryGetResult(out var verdict))
{
throw new InvalidOperationException("The answer was not the expected JSON: " + response.Text);
}
Console.WriteLine($"{verdict.Bug}: {verdict.Reason}");The library builds a schema from the record, with the two descriptions, and sends it with the request. The model answers with a JSON object with the fields bug and reason, and the library deserializes it into the record. TryGetResult returns false when the answer is not valid JSON for the type, the property Result throws instead. The RESULT line of CardRunner is the simpler form of the same idea, one fixed line at the end of a free text.
Tools and agents
Tool calling
Tool calling, also called function calling, lets the model ask the program to run a function. A tool has a name, a description and typed parameters, and the model sees all three. The following code turns a C# method into a tool and prints what the model sees:
public sealed class FileTools(string repoRoot)
{
[Description("Reads a text file. The path is relative to the repository root and must lie under src/.")]
public string ReadFile([Description("File path relative to the repository root")] string path)
{
return File.ReadAllText(Path.GetFullPath(path, repoRoot));
}
}
var readFile = AIFunctionFactory.Create(new FileTools(repoRoot).ReadFile, "read_file");
Console.WriteLine($"{readFile.Name}: {readFile.Description}\n{readFile.JsonSchema}");
var options = new ChatOptions { Tools = [readFile] };The output is the name, the description and the following schema:
{
"type": "object",
"properties": {
"path": {
"description": "File path relative to the repository root",
"type": "string"
}
},
"required": ["path"]
}The model does not run the function. It answers with a call, the name read_file, a call ID and the argument path as JSON. The program runs the method and sends the result back as a message with the role tool and the same call ID.
The two Description attributes are the only place where the model learns when to call the tool and what the parameter means. Tool integration is the broader term for connecting functions and systems to a model this way.
Model Context Protocol (MCP)
The Model Context Protocol (MCP) is an open standard for tool calling across processes. An MCP server offers tools, and a client lists and calls them with JSON-RPC messages over standard input and output or over HTTP. The following code is the server side, one of the eight tools of the UiMcpServer of CardsToLoop and the host that offers them:
[McpServerToolType]
public sealed class UiTools
{
[McpServerTool(Name = "ui_get_text")]
[Description("Reads an element's text: Value pattern, then Text pattern, then the Name property.")]
public static string GetText(
[Description("Title fragment of the window")] string windowTitle,
[Description("AutomationId or Name of the element")] string target)
{
var element = FindTarget(FindWindow(windowTitle), target);
if (element.TryGetCurrentPattern(ValuePattern.Pattern, out var value))
{
return ((ValuePattern)value).Current.Value;
}
return element.Current.Name;
}
}
var builder = Host.CreateApplicationBuilder(args);
builder.Logging.ClearProviders();
builder.Services.AddMcpServer().WithStdioServerTransport().WithTools<UiTools>();
await builder.Build().RunAsync();Two attributes mark a class and a method as tools, the Description attributes are the same as for a local tool. Standard output carries the JSON-RPC stream, so the server must not write anything else to it, not even a log line. The following code connects CardRunner to this server:
var transport = new StdioClientTransport(new StdioClientTransportOptions
{
Name = "ui",
Command = "src/UiMcpServer/bin/Debug/net8.0-windows/UiMcpServer.exe",
});
await using var mcpClient = await McpClient.CreateAsync(transport);
var tools = await mcpClient.ListToolsAsync();The client starts the server as a child process. The eight tools arrive as AIFunction objects and go into the same list as the local tools, so the model does not see a difference. Claude Code uses the same server through the same protocol (see the entry about Claude Code in part 2).
Agent and agent loop
An agent is a language model that calls tools in a loop until the task is done. Agentic AI and agentic coding name the same thing, the second for changes to code. The loop is short, the following code is a plain version of it:
while (true)
{
var response = await client.GetResponseAsync(messages, options);
messages.AddRange(response.Messages);
var calls = response.Messages.SelectMany(message => message.Contents).OfType<FunctionCallContent>().ToList();
if (calls.Count == 0)
{
return response.Text;
}
foreach (var call in calls)
{
var result = await tools[call.Name].InvokeAsync(new AIFunctionArguments(call.Arguments));
messages.Add(new ChatMessage(ChatRole.Tool, [new FunctionResultContent(call.CallId, result)]));
}
}One pass through the loop is a round. The model asks for tools, the program runs them and appends the results, and the model decides the next step. Microsoft.Extensions.AI contains this loop, the following three lines add it to a client:
var client = chatClient.AsBuilder()
.UseFunctionInvocation(configure: loop => loop.MaximumIterationsPerRequest = 60)
.Build();
var response = await client.GetResponseAsync(messages, new ChatOptions { Tools = tools });The call GetResponseAsync then runs until the model answers with text or the limit of rounds is reached, CardRunner uses 60. When a tool throws, the loop sends the error message to the model as the result, so the model can try again. Claude Code is the same loop with its own tools.
Guardrails
Guardrails are the fixed limits around an agent, in code, not in the prompt. The prompt says what the model should do. The guardrail decides what it can do. The following code checks every path the model passes to a file tool of CardRunner:
private string Resolve(string path, string[] roots)
{
var fullPath = Path.GetFullPath(path, repoRoot);
var allowed = roots.Select(root => Path.GetFullPath(root, repoRoot))
.Any(rootPath => fullPath.StartsWith(rootPath + Path.DirectorySeparatorChar, StringComparison.OrdinalIgnoreCase));
if (!allowed)
{
throw new ArgumentException($"{path} is outside the allowed folders ({string.Join(", ", roots)}).");
}
return fullPath;
}The method resolves the path first, so a path with two dots that leaves the repository fails the same way. The exception goes to the model as the tool result, so it reads the reason and picks another path.
CardRunner has four such limits. The paths stay under src, the tool list has no shell, a run ends after 60 rounds and after 400,000 tokens. The Peakboard Designer test system adds state checks after every step of a script, called invariants or test oracles. Examples are no new window and no error dialog.
Human in the loop
Human in the loop means that a person decides at a defined point of an automated run, with an approval, an answer or a correction. The following class asks before the commit tool of CardRunner runs:
public sealed class ApprovedFunction(AIFunction inner) : DelegatingAIFunction(inner)
{
protected override async ValueTask<object?> InvokeCoreAsync(AIFunctionArguments arguments, CancellationToken cancellationToken)
{
Console.Write($"{Name} {arguments["message"]}\nRun it? [y/n] ");
if (Console.ReadLine() != "y")
{
return "The user declined the commit. Say what is missing in the RESULT line.";
}
return await base.InvokeCoreAsync(arguments, cancellationToken);
}
}A declined call returns a text, and the model reads it like any other tool result. CardRunner does not have this class and commits on its own. Microsoft.Extensions.AI has a class for the same idea, ApprovalRequiredAIFunction. The loop then returns the call as a request instead of running it, and the program answers with an approval in the next call. The entry about the Microsoft Agent Framework in part 2 shows it.
In the self-healing loop a tester adds a comment to a board and restarts it, and the comment goes into the prompt. The Peakboard Designer test system takes rules from testers in Slack and applies them on every run.
Agent memory
Agent memory is what an agent keeps between runs, as files it reads at the start and writes at the end. Without it every run starts without knowledge and finds every known problem again. The following lines show a router file and one topic file:
MEMORY.md
- [Dialogs](dialogs.md): Enter in a combo box, default buttons, dialogs of the main window
- [Grid](grid.md): row selection, cell values, the German number format
dialogs.md
## Enter closes the dialog
Symptom: the dialog closed on Enter in the combo box, the value was lost
Cause: WPF sends Enter to the default button of the dialog
Works: press Down to accept the autocompletion, then click the OK buttonThe agent reads the router, opens one or two topics and follows the links it needs. A learning loop is the step at the end of a run that writes the new notes.
In the Peakboard Designer test system the notes are around a thousand Markdown files in the repository. The agent finds them with the router and grep, without embeddings or a database. Two more files work the same way there, the feedback with the rules of the testers and a glossary of the allowed terms.
Agent orchestration, subagents and session continuation
Agent orchestration is the coordination of several runs over one process, with an order, dependencies and the points where a person decides. A subagent is a second agent that the first one starts with a task. Session continuation resumes a run with its full context. The following lines show the runs for one card on a Kanban board:
card-17 concept run 1 claude -p "/concept card-17" concept.md on the card
card-17 develop run 2 claude -p "/develop card-17" commit on branch card-17
card-17 test run 3 claude -p "/test card-17" report with one finding
card-17 question "Is the discount a percentage or an amount?" waits for an answer
card-17 develop run 4 claude --resume <session of run 2> "..." continues with the answerEvery run is one start of Claude Code with a role, a Markdown file with instructions. The board starts the runs one after another and shows the results on the card. The resume flag continues the session of run 2, so run 4 knows what it changed. The Peakboard Designer test system starts a subagent for missing AutomationIds, with file tools only, so the main run keeps its context small.
ChatOps
ChatOps means that a chat tool, here Slack, is the user interface of the automation. A message starts a run, the bot posts the progress and the result, and a later message in the same channel continues the session. The following lines show such an exchange:
tester @testbot test the connection dialog of the new data source
testbot run started, session 7c1e
testbot ui_click MainMenu_File, ui_find_elements ... (the new tool calls, every 4 seconds)
testbot run finished, 1 finding, report.html and run.gif attached
tester @testbot what did step 12 check?
testbot Step 12 reopened the dialog and read the port back, it showed 1883 as entered.
tester @testbot remember: [test] never navigate with Tab, fix the AutomationId insteadIn the Peakboard Designer test system the bot is a .NET service without a model. It checks the sender against a list, starts Claude Code in headless mode and reads its output as a JSON stream. The following code is the core of such a bot:
var startInfo = new ProcessStartInfo("claude", $"-p \"{request}\" --output-format stream-json --verbose")
{
RedirectStandardOutput = true,
UseShellExecute = false,
};
using var process = Process.Start(startInfo)!;
while (await process.StandardOutput.ReadLineAsync() is { } line)
{
var json = JsonDocument.Parse(line).RootElement;
switch (json.GetProperty("type").GetString())
{
case "system":
await PostAsync(channel, $"run started, session {json.GetProperty("session_id").GetString()}");
break;
case "result":
await PostAsync(channel, json.GetProperty("result").GetString() ?? "");
break;
}
}Every line of the stream is one JSON object with a type. The first one carries the session ID. The lines in between are the messages of the model and the tool results, and the last one is the final answer.
The bot keeps the session ID per channel, and a later message continues the run with the resume flag. A message that starts with remember adds a rule to a feedback file, and every later run applies the rules in it.
Self-healing loop
A self-healing loop is a test loop in which the agent that finds a bug also fixes it, and the loop tests again, up to a fixed number of attempts. In the loop for the new web runtime of Peakboard the test compares two screenshots of the same board, the reference from the production runtime on WPF and the actual image from the web runtime. Claude Code judges the images and fixes the source of the web runtime when it finds a bug. The following method tests one board:
enum Verdict { Ok, Fixed }
enum BoardResult { Passed, PassedAfterFix, Failed }
async Task<BoardResult> TestBoardAsync(string board, string sessionId)
{
var repaired = false;
for (var attempt = 1; attempt <= 3; attempt++)
{
BuildWebRuntime();
var reference = RenderWithWpf(board);
var actual = RenderWithWeb(board);
var diff = DiffImage(reference, actual);
// Claude Code reads the three images. It answers Ok, or it fixes the source and answers Fixed.
var verdict = await AskClaudeCodeAsync(sessionId, reference, actual, diff);
if (verdict == Verdict.Ok)
{
return repaired ? BoardResult.PassedAfterFix : BoardResult.Passed;
}
repaired = true;
}
return BoardResult.Failed;
}The method returns one of three results. Passed means the first comparison was fine. PassedAfterFix means at least one Fixed came before the Ok, and the caller commits the changed source on the test branch. Failed means three attempts without an Ok, the source stays as Claude Code left it.
In every case the caller writes the result next to the board, and the page for the testers shows the board with both images. A tester can add a comment and start the board again. The session ID keeps every attempt in the same conversation, so Claude Code does not read the code base again. The loop itself has no model call, the only one is inside AskClaudeCodeAsync.
That’s it.
If you have questions feel free to contact me.
Links:
CardsToLoop on GitHub →
Microsoft.Extensions.AI →
Model Context Protocol →
Anthropic C# SDK →
Claude Code, run programmatically →