AI terms for .NET developers, part 2: retrieval, evaluation and frameworks

In October 2026 I collected the AI terms from my posts about CardsToLoop, the Peakboard Designer test system and the self-healing loop. The first part covered models, prompts, tools and agents. This second part explains the terms around retrieval, evaluation and the frameworks, each with a sample of 5 to 25 lines in C# or text.

Every entry starts with a definition sentence, then the sample, then what it shows. The glossary is not a tutorial. The samples around the judge come from the self-healing loop for a new Peakboard runtime, the framework samples run the card of CardRunner. The first part explains token, tool calling, agent loop and the other terms these entries build on.

The two groups of this part and their terms:

• Retrieval and evaluation
  • Embeddings
  • Retrieval-augmented generation and agentic retrieval
  • LLM as judge
  • Visual regression
  • Evals
• Frameworks and tools
  • Claude Code
  • Microsoft.Extensions.AI
  • Microsoft Agent Framework
  • Semantic Kernel
  • Microsoft Foundry
  • Git workflow
  • UI Automation and Playwright

The three groups of the first part and their terms:

• Models
  • Large language model and generative AI
  • Token and context window
  • Multimodal
  • Hallucination
  • Deterministic and non-deterministic
• Prompts
  • System prompt and user message
  • Prompt engineering
  • Context engineering
  • Structured output
• Tools and agents
  • Tool calling
  • Model Context Protocol
  • Agent and agent loop
  • Guardrails
  • Human in the loop
  • Agent memory
  • Agent orchestration, subagents and session continuation
  • ChatOps
  • Self-healing loop

Retrieval and evaluation

Embeddings

An embedding is a vector of numbers that encodes the meaning of a text, computed by an embedding model. Two texts with a similar meaning get vectors that point in a similar direction, and the cosine similarity measures that. The following code compares three sentences:

IEmbeddingGenerator<string, Embedding<float>> generator =
    new OpenAIClient(openAiKey).GetEmbeddingClient("text-embedding-3-small").AsIEmbeddingGenerator();

var embeddings = await generator.GenerateAsync(
[
    "The total ignores the discount.",
    "The sum does not apply the rebate.",
    "The window title is wrong.",
]);
Console.WriteLine($"{Similarity(embeddings[0], embeddings[1]):F2} {Similarity(embeddings[0], embeddings[2]):F2}");

static float Similarity(Embedding<float> a, Embedding<float> b)
{
    return TensorPrimitives.CosineSimilarity(a.Vector.Span, b.Vector.Span);
}

The program prints about 0.55 and 0.18. The first two sentences share no word except “the” and still get the clearly higher value. The third scores low against both, 0.18 and 0.21.

An embedding model reads a text like a language model: it splits the text into tokens and computes a vector for every token that takes the surrounding words into account. Instead of predicting the next token, it combines these vectors into a single vector of fixed length. In training, the model sees millions of text pairs that belong together, such as a question and its answer. It learns to give their vectors a similar direction and the vectors of unrelated texts different ones. A single number of the vector has no meaning of its own, only the direction of the whole vector counts.

The interface IEmbeddingGenerator is the second interface of Microsoft.Extensions.AI, next to IChatClient. Anthropic offers no embedding model, so the sample uses “text-embedding-3-small” of OpenAI. OpenAI released the model in January 2024 together with the larger “text-embedding-3-large”. It reads up to 8,191 tokens per text and returns a vector of 1,536 numbers. According to the price list, OpenAI bills $0.02 per million tokens, so a help file of 100,000 words costs about a quarter of a cent. The method CosineSimilarity of the class TensorPrimitives compares two vectors without a loop.

Retrieval-augmented generation (RAG) and agentic retrieval

Retrieval-augmented generation (RAG) puts the excerpts of a document collection that are most similar to the question into the prompt before the model answers. The documents are split into chunks of a few hundred words, every chunk gets an embedding, and a question finds its chunks by similarity. The following code builds the chunks of a help file once and answers a question from the three best ones:

// Once: one chunk per paragraph of the help, each with its embedding
var chunks = File.ReadAllText("help.md").Split("\n\n", StringSplitOptions.RemoveEmptyEntries);
var embeddings = await generator.GenerateAsync(chunks);

// Per question: the three most similar chunks go into the prompt
var questionEmbedding = await generator.GenerateAsync(question);
var excerpts = chunks.Zip(embeddings)
    .OrderByDescending(pair => Similarity(pair.Second, questionEmbedding))
    .Take(3)
    .Select(pair => pair.First);
var prompt = "Answer only from the following excerpts of the help.\n\n"
    + string.Join("\n---\n", excerpts) + "\n\nQuestion: " + question;
var response = await chatClient.GetResponseAsync(prompt);

The model answers from the excerpts, so the help can change without a change to the model. The split into chunks decides the quality, a paragraph is the simplest unit, a section with its heading is often the better one. A vector database replaces the list and the sort for a large collection.

A small collection needs no embeddings. A help file of 100,000 words has about 130,000 tokens and fits into the context window of current models as a whole. With prompt caching, the provider keeps the processed start of a prompt, and the next request with the same start does not compute it again. Anthropic keeps this cache for five minutes after the last hit and bills it at a tenth of the normal input price. The whole text still sits in every prompt, the answer takes longer than with three excerpts, and in a very long context the model misses details more easily.

Embeddings pay off when the collection no longer fits into the context window or when many questions come in and the tokens add up. With RAG, a question carries three excerpts with about 2,000 tokens instead of 130,000, and the embeddings of the help are computed only once.

Agentic retrieval needs no chunks and no embeddings. The model itself decides what to search for and calls tools for it, as Claude Code does with Glob to find files, Grep to search their content and Read to open them. In its simplest form, the model picks sections from the table of contents, and only these go into the prompt. It can search several times and refine the search after each result. For code and notes this often finds more than RAG. A search for the class name InvoiceCalculator finds exactly the files that contain it, while an embedding only finds texts with a similar meaning.

LLM as judge

LLM as judge means that a large language model (LLM) grades an output against written rules, where a fixed check cannot. A pixel count cannot tell anti-aliasing from a wrong status color. A model with the rules can. The following lines show the rules of the judge in the self-healing loop, shortened and with a JSON answer instead of the RESULT line:

You compare two screenshots of the same board, reference.png from the WPF runtime
and actual.png from the web runtime. diff.png marks the differing pixels.
Acceptable: anti-aliasing, a slightly different font, an offset of one or two pixels.
A bug: wrong colors, wrong or missing texts and values, missing elements,
a clearly wrong position or size, even on a small area.
Charts: type, values and colors have to match, labels may move.
Answer with one JSON object: {"bug": true or false, "reason": "<one sentence>"}

The rules name what is acceptable and what is a bug, with the order of the images and a fixed answer format. The following class sends the rules and the three images and reads the verdict:

public sealed class Judge(IChatClient client)
{
    private static readonly string _rules = File.ReadAllText("judge-rules.md");

    public async Task<Verdict> JudgeAsync(string referencePath, string actualPath, string diffPath)
    {
        var message = new ChatMessage(ChatRole.User,
        [
            new TextContent(_rules),
            new DataContent(await File.ReadAllBytesAsync(referencePath), "image/png"),
            new DataContent(await File.ReadAllBytesAsync(actualPath), "image/png"),
            new DataContent(await File.ReadAllBytesAsync(diffPath), "image/png"),
        ]);
        var response = await client.GetResponseAsync<Verdict>([message]);
        return response.Result;
    }
}

The record Verdict with the fields Bug and Reason is the one from the entry about structured output in part 1. The judge gets its own prompt and its own call, separate from the agent that writes the code. So the rules of the judge do not mix with the instructions for the fix. In the self-healing loop the judge is Claude Code, which reads the three images with its own Read tool.

Visual regression

Visual regression testing compares a screenshot with a reference image and reports the differing pixels. In the self-healing loop the reference image comes from the Windows Presentation Foundation (WPF) runtime and the actual image from the web runtime. The following code marks the differing pixels, counts them and saves the diff image:

var diff = new Bitmap(reference.Width, reference.Height);
var differing = 0;
for (var y = 0; y < reference.Height; y++)
{
    for (var x = 0; x < reference.Width; x++)
    {
        if (Distance(reference.GetPixel(x, y), actual.GetPixel(x, y)) > Tolerance)
        {
            diff.SetPixel(x, y, Color.Red);
            differing++;
        }
    }
}
diff.Save("diff.png");
Console.WriteLine($"{100.0 * differing / (reference.Width * reference.Height):F2} percent of the pixels differ");

static double Distance(Color a, Color b)
{
    return Math.Sqrt(Math.Pow(a.R - b.R, 2) + Math.Pow(a.G - b.G, 2) + Math.Pow(a.B - b.B, 2));
}

The method Distance is the length of the vector between the two colors, with red, green and blue as the three axes. A pixel counts as different above a tolerance. The percentage goes to the log and decides nothing. The judge decides, with the diff image as the third input. A bug can be a few hundred wrong pixels, and a chart with many thousand different pixels can be fine.

Evals

Evals measure how often a model solves a task correctly, for example how often the judge gives the right verdict. A unit test expects the same result on every run. A model can answer the same input differently, so an eval runs many cases with a known answer and counts how many the model gets right. The result is a pass rate, not a single pass or fail. The eval for the judge above uses screenshot pairs that a tester has already checked by hand. Every case is one line of JSON with the three image paths and the verdict of the tester:

{"reference":"cases/status-bar/reference.png","actual":"cases/status-bar/actual.png","diff":"cases/status-bar/diff.png","expectedBug":true}

The flag expectedBug holds the verdict of the tester. A line with true marks a bug. A line with false is a case where the images differ and the tester accepted the difference, for example anti-aliasing. The following test with xUnit, the test framework of .NET, runs 20 such cases through the judge and counts the cases where the judge agrees with the tester:

public sealed record EvalCase(string Reference, string Actual, string Diff, bool ExpectedBug);

[Fact]
public async Task The_judge_agrees_with_the_tester_in_at_least_18_of_20_cases()
{
    var judge = new Judge(new AnthropicClient().AsIChatClient("claude-sonnet-5-5"));
    var cases = File.ReadLines("cases.jsonl").Select(line => JsonSerializer.Deserialize<EvalCase>(line)!).ToList();
    var agreed = 0;
    foreach (var evalCase in cases)
    {
        var verdict = await judge.JudgeAsync(evalCase.Reference, evalCase.Actual, evalCase.Diff);
        agreed += verdict.Bug == evalCase.ExpectedBug ? 1 : 0;
    }
    Assert.True(agreed >= 18, $"{agreed} of {cases.Count} cases agreed with the tester.");
}

The test creates the class Judge from the entry above with a Claude client. The test passes at 18 of 20, not only at 20 of 20. The model is non-deterministic and some cases are borderline, for example an offset of three pixels, so one or two different verdicts are normal. The eval pays off when the judge prompt changes. The new prompt runs against the same 20 cases, and a worse prompt shows up as a lower number, for example 15 of 20, before the self-healing loop uses it. Every verdict of the judge that a tester corrects becomes a new line in cases.jsonl.

Evals are worth the effort for every step where a model decides and no fixed check can tell right from wrong. Examples are the verdict of the judge, the category of a support ticket or a value read from an invoice. They run before every change to the prompt and before a switch to another model or model version. In a loop or a pipeline where nobody reads the single answer, they also run on a schedule. A one-off prompt or an output that a person checks every time needs no eval, and deterministic code gets a normal unit test.

Frameworks and tools

Claude Code

Claude Code is the command line tool of Anthropic that runs Claude as an agent in a repository. It brings tools for files, a shell and servers of the Model Context Protocol (MCP). In headless mode it runs without the terminal dialog until the task is done. The following lines show how the PowerShell runner of CardsToLoop starts it and reads the result:

$output = claude -p $prompt --mcp-config .mcp.json --strict-mcp-config 2>&1
$output | Set-Content $logPath -Encoding UTF8

$resultLine = @($output | Where-Object { $_ -match "^RESULT:" }) | Select-Object -Last 1
if ($resultLine -match "^RESULT: done") { Set-CardState "Done" } else { Set-CardState "Failed" }

The flag -p passes the prompt and prints the last message. The strict flag loads no server besides the ones in the named file. The following file names the UiMcpServer:

{
  "mcpServers": {
    "ui": {
      "command": "src/UiMcpServer/bin/Debug/net8.0-windows/UiMcpServer.exe",
      "args": []
    }
  }
}

Claude Code starts the server as a child process and lists its tools, the same way CardRunner does with the MCP client. With the flag –output-format stream-json the output comes as one JSON object per line, with the session ID. The Slack bot and the Kanban board use the ID to continue a run (see the entry about ChatOps in part 1).

Microsoft.Extensions.AI

Microsoft.Extensions.AI is a library of Microsoft with one interface for chat models, IChatClient, one class for tools, AIFunction, and the loop. The provider is one line. The following code creates the same kind of client for Claude and for an OpenAI model, then builds the client of CardRunner with its own middleware:

IChatClient claude = new AnthropicClient().AsIChatClient("claude-sonnet-5-5");
IChatClient openAi = new OpenAIClient(openAiKey).GetChatClient(openAiModel).AsIChatClient();

var client = new RoundLogger(claude, maxTokens: 400_000).AsBuilder()
    .UseFunctionInvocation(configure: loop => loop.MaximumIterationsPerRequest = 60)
    .Build();
var response = await client.GetResponseAsync("What is a token?");
Console.WriteLine(response.Text);

The method AsIChatClient comes with the software development kit (SDK) of the provider, here the Anthropic C# SDK and the package Microsoft.Extensions.AI.OpenAI. Everything after the first line is the same for both.

The class RoundLogger from the entry about tokens in part 1 derives from DelegatingChatClient, the base class for custom middleware. The builder puts the loop above it, so the logger sees every round. The library brings more middleware, for logging, caching and OpenTelemetry. CardRunner uses version 10.11.0.

Microsoft Agent Framework

The Microsoft Agent Framework is a library of Microsoft for agents in .NET and Python. It builds on Microsoft.Extensions.AI, which provides the client and the loop, and adds three things on top. An agent is an object that holds the instructions and the tools. A session holds the conversation and can be saved and loaded again. A tool can wait for a person to approve it before it runs. Microsoft names the framework the successor of Semantic Kernel and AutoGen. The following code runs the CardRunner prompt as an agent, asks before the commit and saves the session:

IChatClient chatClient = new AnthropicClient().AsIChatClient("claude-sonnet-5-5");
var commit = new ApprovalRequiredAIFunction(AIFunctionFactory.Create(localTools.GitCommitAsync, "git_commit"));
AIAgent agent = new ChatClientAgent(chatClient, instructions: File.ReadAllText("instructions.md"), tools: [.. uiTools, commit]);
AgentSession session = await agent.CreateSessionAsync();
var response = await agent.RunAsync("THE CARD\n--------\n" + card.Text, session);

var request = response.Messages.SelectMany(message => message.Contents).OfType<ToolApprovalRequestContent>().FirstOrDefault();
if (request is not null)
{
    var call = (FunctionCallContent)request.ToolCall;
    Console.Write($"{call.Name} {call.Arguments?["message"]}\nRun it? [y/n] ");
    var approval = request.CreateResponse(approved: Console.ReadLine() == "y");
    response = await agent.RunAsync(new ChatMessage(ChatRole.User, [approval]), session);
}

var checkpoint = await agent.SerializeSessionAsync(session);
File.WriteAllText("session.json", checkpoint.GetRawText());

The class ChatClientAgent is the agent. It holds the system prompt from the file instructions.md and the tools. The session holds the messages of the conversation, so the second call of RunAsync continues where the first one stopped. The wrapper ApprovalRequiredAIFunction marks the commit tool as a tool that needs an approval. When the model wants to commit, the framework does not run the tool. The first run ends with a ToolApprovalRequestContent instead, which holds the planned call with the commit message. The code shows the message, reads y or n from the console and sends the answer as the next message of the same session. With y the framework runs the commit, with n the model gets the refusal and continues without it.

The method SerializeSessionAsync turns the session into JSON, and the file session.json is the checkpoint. A later process, even after a restart, loads the file and continues the card at the same point. The hand-written loop of CardRunner keeps the conversation only in memory and cannot do that. The sample uses version 1.25.0 of the package Microsoft.Agents.AI.

Semantic Kernel

Semantic Kernel is the older SDK of Microsoft for LLM applications in .NET and Python. A kernel holds the model and the plugins, and a plugin is a class whose methods the model can call. The following code registers a plugin and lets the model use it:

public sealed class InvoicePlugin
{
    [KernelFunction("read_items")]
    [Description("Returns the items of the current invoice, one per line: name, quantity, price.")]
    public string ReadItems()
    {
        return "Gadget, 0, 100.00\nWidget, 2, 25.00";
    }
}

var builder = Kernel.CreateBuilder();
builder.Services.AddSingleton(chatClient.AsChatCompletionService());
builder.Plugins.AddFromType<InvoicePlugin>("invoice");
var kernel = builder.Build();

var settings = new PromptExecutionSettings { FunctionChoiceBehavior = FunctionChoiceBehavior.Auto() };
var result = await kernel.InvokePromptAsync("Which items have a quantity of zero?", new KernelArguments(settings));

The attribute KernelFunction marks a method of the plugin as a tool, like the Description attribute in Microsoft.Extensions.AI, and the model sees it as invoice-read_items. The setting FunctionChoiceBehavior.Auto lets the model decide whether it calls the plugin, the kernel runs the loop. The method AsChatCompletionService wraps any IChatClient, so the Anthropic client works here as well. The sample uses version 1.81.0.

Microsoft Foundry

Microsoft Foundry, until November 2025 Azure AI Foundry, is the Azure platform that hosts models of several vendors behind one endpoint. Billing and access control come from the Azure subscription. Claude runs there as well. The following code creates the Anthropic client for Foundry, with an API key and with a token:

var credentials = new AnthropicFoundryApiKeyCredentials(apiKey, "my-resource");
IChatClient chatClient = new AnthropicFoundryClient(credentials).AsIChatClient("claude-sonnet-5-5");
var response = await chatClient.GetResponseAsync("What is a token?");
Console.WriteLine(response.Text);

// The same client with a token from Microsoft Entra ID instead of the key
var entraCredentials = new AnthropicFoundryIdentityTokenCredentials(new DefaultAzureCredential(), "my-resource");
IChatClient entraClient = new AnthropicFoundryClient(entraCredentials).AsIChatClient("claude-sonnet-5-5");

The package Anthropic.Foundry replaces the client. The rest of the code stays. The class DefaultAzureCredential from the package Azure.Identity takes the token from the signed-in developer, a managed identity or an environment variable. No key lies in the code or the configuration. The resource name is part of the endpoint of the Foundry resource.

The Foundry client is an IChatClient like any other, so it also works as the model of the agent in the entry about the Microsoft Agent Framework above. Foundry also has its own Foundry Agent Service, which stores the agent and its conversation in Azure instead of a local session.json.

Foundry fits a company that already runs on Azure. Claude then comes with the Azure subscription, access goes through Entra ID and Azure roles, and the models of several vendors sit in one account. For a small project or a company without Azure, the Anthropic API is simpler, and new models and beta features usually arrive there first.

Git workflow

An agent changes files. A Git workflow makes every one of these changes visible and easy to undo. After each task the agent commits its change. The commit message says what it changed and how it checked the change in the running application, for example that the delete dialog appears and No keeps the total at 88,80. A developer then sees the change in the log and can undo it with git revert. In CardRunner the model cannot call Git directly. It has no shell, only one tool for the commit, and this tool runs fixed commands:

[Description("Stages the changed files under src/InvoiceApp and commits them. The first line of the message is 'card: <title>', the body says what changed and how it was verified. Returns the commit hash.")]
public async Task<string> GitCommitAsync([Description("The complete commit message, first line and body")] string message)
{
    var messageFile = Path.GetTempFileName();
    File.WriteAllText(messageFile, message);
    try
    {
        await RunAsync("git", $"add {AppFolder}", GitTimeoutMs);
        await RunAsync("git", $"commit -F \"{messageFile}\"", GitTimeoutMs);
        var (_, hash) = await RunAsync("git", "rev-parse --short HEAD", GitTimeoutMs);
        return $"committed {hash.Trim()}";
    }
    finally
    {
        File.Delete(messageFile);
    }
}

The tool does three things. It adds only the files in src/InvoiceApp, so no log file and no other file ends up in the commit. It passes the message through a temporary file, because quotes and line breaks in a message cause errors on the command line. Finally, it returns the short commit hash to the model. The message itself comes from the model. The description of the tool tells it to start with “card:” and the title of the card. The following lines show the log of CardsToLoop after two runs, with the newest commit at the top:

17b20d5 Revert the input checks of the .NET run so main keeps all cards open
dc9c0cb Add the log of the validate card run from 20.10.2026 with the .NET runner
495c995 (tag: validate-card-dotnet-done) card: Validate quantity and price before adding an item
bdd5c43 Revert the delete dialog of the .NET run so main keeps all cards open
696299a Add the log of the delete card run from 20.10.2026 with the .NET runner
a8d6528 (tag: delete-card-dotnet-done) card: Deleting an item should ask first

Each run leaves three commits. Read from the bottom, the first is the fix of the model, with the card title and a tag. The second adds the log of the run. The third is a revert that takes the fix out of main again. So main stays at the old state, and the same card can run again in the next test. The tag keeps the fix, and git checkout delete-card-dotnet-done shows it at any time. The self-healing loop commits only on its own test branch and never pushes.

UI Automation and Playwright

Windows UI Automation and Playwright are the interfaces through which the tools of an agent click and read. UI Automation reports the controls of a window to other programs. Playwright is a library of Microsoft for browser automation. The following code clicks a button and reads a field with each of them:

// Windows UI Automation: click a button of a WPF window and read a text box, both by AutomationId
var window = AutomationElement.RootElement.FindFirst(TreeScope.Children,
    new PropertyCondition(AutomationElement.NameProperty, "Invoice Editor"));
var button = window.FindFirst(TreeScope.Descendants,
    new PropertyCondition(AutomationElement.AutomationIdProperty, "DeleteItemButton"));
((InvokePattern)button.GetCurrentPattern(InvokePattern.Pattern)).Invoke();
var discount = window.FindFirst(TreeScope.Descendants,
    new PropertyCondition(AutomationElement.AutomationIdProperty, "DiscountTextBox"));
var discountText = ((ValuePattern)discount.GetCurrentPattern(ValuePattern.Pattern)).Current.Value;

// Playwright: the same in the browser, the button by role and name, the field by its test id
var page = await browser.NewPageAsync();
await page.GotoAsync("http://localhost:5000/board/1");
await page.GetByRole(AriaRole.Button, new() { Name = "Delete selected" }).ClickAsync();
var discountInBrowser = await page.GetByTestId("discount").InputValueAsync();

An AutomationId is a fixed name the developer sets on a control, a test ID is the same thing in HTML. Playwright finds an element by role and name as well. A pattern, here Invoke and Value, is the set of actions a control supports. A button has no Value pattern and a text box no Invoke pattern.

Both interfaces work without a screenshot and read values exactly. The MCP servers of CardsToLoop and the Peakboard Designer test system wrap UI Automation. The control tests of the self-healing loop use Playwright.

That’s it.

If you have questions feel free to contact me.

Links:

CardsToLoop on GitHub →
Microsoft.Extensions.AI →
Microsoft Agent Framework →
Semantic Kernel →
Claude on Microsoft Foundry →
Playwright for .NET →