Multi-LLM Deliberation in .NET: Architecture and Measurements

One LLM in a chat gives one answer. That same model is the author of the decision, its critic, and the party that decides the answer is good. On a team those are three different roles, and in an LLM pipeline it pays to separate them too: let several independent models work on the same problem and let the disagreement between them follow a predefined protocol.

We build this in the open-source project Delibera — a .NET framework for structured deliberation on top of any LLM provider. What follows is not a tour of the project. It is what the measurements on real runs showed: where a council of models wins, where it loses, and which defects are visible only by measuring rather than by testing.

What a round is

A debate protocol in .NET is not "ask everyone and count the votes". A round is made of fixed phases, and each phase has its own job. Delibera ships three strategies, and they differ not in prompt wording but in what the participants actually argue about:

The difference between them is not cosmetic. In StandardDebate the members write a first analysis, critique each other, and revise the first analysis; the Chairman assembles the verdict. In CritiqueDebate a member never writes an analysis at all — it defends a position, and the outcome depends on whether that position survives the attack. In ConsensusDebate the members diverge first, then look for common ground, and only then formulate a consensus.

Every strategy is an implementation of IDebateStrategy. That is what lets you write your own: the contract is exactly the same as the built-in ones, and everything else in the pipeline — Knowledge Keeper, compression, Operator, cache — applies to it unchanged.

Architecture: who owns what

A council is assembled with a fluent builder, and the model providers sit behind interfaces:

using Delibera.Core.Council;
using Delibera.Core.Providers;

using var factory = new ProviderFactory();
var ollama = factory.CreateOllama("http://localhost:11434");

var result = await new CouncilBuilder()
    .AddMember("llama3.2:3b", ollama, "Analyst")
    .AddMember("qwen2.5:7b", ollama, "Strategist")
    .SetChairman(Chairman.CreateStandard("qwen2.5:7b", ollama))
    .WithStandardDebate()
    .WithSystemPrompt("You are a software architecture expert.")
    .WithUserPrompt("Microservices vs Monolith for a 5-person startup?")
    .WithMaxRounds(4)
    .SaveResultTo("./deliberation.md")
    .Build()
    .ExecuteAsync();

Console.WriteLine(result.FinalVerdict);

The decisive choice is Microsoft.Extensions.AI as the seam between providers and the pipeline. Adding a model must not force a rewrite of the decision logic, so the layer that talks to LLMs sits behind an interface and the concrete providers (Ollama, cloud APIs) are its implementations.

The core layers look like this:

Delibera.Core
├── Council/              ← CouncilBuilder, CouncilExecutor, Chairman, KnowledgeKeeper, Operator
├── Debate/               ← StandardDebate, CritiqueDebate, ConsensusDebate
├── Orchestration/        ← IDebateOrchestrator, LocalDebateOrchestrator, DebateRoundEvent
├── Cache/                ← IDebateCache, InMemoryDebateCache, FileDebateCache, DebateCacheKeyGenerator
├── Compression/          ← Semantic / Deduplication / Summarization / Hybrid
├── Chunking/             ← AutoChunker, AutoChunkingOrchestrator, AutoChunkingOptions
└── Interfaces/           ← ILLMProvider, IRagProvider, IContextCompressor, IMcpClient

The same layer contains one thing that is rarely done right: a debate is an executable strategy, not a request. IDebateOrchestrator separates how a round is executed from the round itself. The local orchestrator runs the council in-process, the Redis-backed one spreads rounds across workers over Streams, and both look identical to the calling code.

Chairman: synthesis, not vote counting

Majority voting over a debate is a trap. It optimises for agreement rather than correctness: three participants backed by the same model under three names produce a unanimous and entirely useless verdict.

So the final verdict is written by the Chairman — a separate model that reads the whole transcript and formulates the answer. That costs more than counting votes, and it is also the only place where "the participants disagreed" can appear at all, instead of "the majority won".

Voting has not gone away: Majority, BordaCount and Weighted engines exist. They are just an explicitly chosen layer rather than a substitute for synthesis.

A side effect of synthesis is a risk we hit in the measurements and that deserves to be stated plainly: a cheap roster can invent unanimity. In one run the Chairman wrote that "all three participants recognised the N+1 query pattern as the dominant scalability defect" — something the transcript did not contain, because the participants had been discussing different things. Verdicts are now checked against the transcript, and when no support is found, that shows up in the result rather than as trust in the wording.

Knowledge Keeper: retrieval on every round

Knowledge is not handed to the council as one blob up front. The Knowledge Keeper is a participant with access to a vector store that runs targeted queries against the corpus as the debate proceeds and returns what it found into the round that needs it.

using Delibera.Core.Council;

var kkMember = new CouncilMember("llama3.2:3b", ollama, "Knowledge Keeper");
var keeper = new KnowledgeKeeper(rag, kkMember, "my_knowledge");
await keeper.IndexFileAsync("./docs/architecture.md");

There are two stores, for two operating conditions: QdrantRagProvider (Qdrant over gRPC) and PgVectorRagProvider (PostgreSQL with the pgvector extension). Embeddings go through IEmbeddingProvider; in our measurements that was OllamaEmbeddingProvider over nomic-embed-text.

The measurements show a stable pattern: three corpus queries per debate, regardless of topic. Retrieval does not grow with the length of the discussion — only the cost of the debate itself does.

Retrieval quality was checked separately: a probe for "which failures are transient and retryable?" returned two hits with a top score of 0.738, and the verdicts in those debates cited specific documents with line ranges. A model that cannot point at a source shows up as a missing citation.

AutoChunking deserves its own mention: large documents are automatically split on semantic boundaries into chunks sized for the smallest model's context window and distributed across rounds progressively — by the final round every model has seen the whole document. That is a discipline of its own, and we wrote about it separately in «Chunking Is Not a Strategy: How to Choose Retrieval Units».

Context compression: 11.7–12.1%

Between rounds the context is compressed: previous answers take up more and more space while the novelty in them drops. Four strategies are implemented:

using Delibera.Core.Compression;

var result = await new CouncilBuilder()
    .AddMember("llama3.2:3b", ollama, "Analyst")
    .AddMember("qwen2.5:7b", ollama, "Strategist")
    .SetChairman(Chairman.CreateStandard("qwen2.5:7b", ollama))
    .WithCompression(CompressionStrategy.Hybrid,
        llmProvider: ollama,
        modelName: "llama3.2:3b",
        embeddingProvider: embeddings)
    .WithCompressionOptions(new CompressionOptions { TargetRatio = 0.5 })
    .WithCompressionCache()
    .WithUserPrompt("Analyze our architecture options...")
    .WithMaxRounds(4)
    .Build()
    .ExecuteAsync();

Console.WriteLine(result.TokenStats?.ToSummary());

The number worth remembering: 11.7–12.1% of prompt tokens saved. The project docs used to promise 30–70%. That figure was never measured — the compressor was configured but never invoked, and ten runs honestly reported "0.00% saved at 0 ms overhead". We fixed it and rewrote the number in the docs to the measured one.

The lesson for design: context compression is a useful but small effect. It will not rescue a budget. If the budget and the latency are tight, two other levers work: fewer rounds and the cache.

Operator: tools inside the debate

The Operator is a micro-agent that connects the council to external tools through MCP (Model Context Protocol). A participant can ask it for an external action at any moment by putting a marker into its message:

[[OPERATOR: open https://modelcontextprotocol.io and summarize what MCP is]]

The Operator then interprets the request with its own, cheaper model, picks the right MCP tool, calls it, interprets the result and returns a short answer that lands in the next round.

var council = new CouncilBuilder()
    .AddMember("llama3.2:3b", ollama, "Optimist")
    .AddMember("qwen2.5:7b",  ollama, "Skeptic")
    .SetChairman(Chairman.CreateStandard("qwen2.5:7b", ollama))
    .WithOperator("llama3.2:3b", ollama, servers, reuseCompression: true)
    .WithStandardDebate()
    .WithUserPrompt("Research the latest .NET 10 features and prepare a short summary.")
    .WithMaxRounds(4)
    .Build()
    .ExecuteAsync();

One detail deserves emphasis: the Operator compresses the tool result with the same compressor the council uses (reuseCompression: true) — otherwise a single browser answer can eat the whole round's context.

The measurements put it at 1–3 delegated tasks per debate — the count grows exactly as much as the question requires an outside look, and not on its own.

What the measurements showed

Every number below comes from real debates against Ollama Cloud, not from estimates. The full method and the caveats are in the "Measured Behaviour" section of the project site.

This table has to be read together with two caveats.

First: a debate is not fast. 154–209 s is the honest cost of the question. If that does not fit the budget, cut rounds, shrink the roster or turn on the cache — do not look for "fast multi-agent".

Second, and it is the more awkward one: roster composition matters more than the framework. The spread between the members' answers is 1.23–1.56× out of a possible 3×, because a round lasts as long as its slowest participant. And 68.8% of the output coming from one model means a "council of three" can easily be one voice, multiplied three times.

Defects that measurement found and testing did not

The most useful output of working on the project is the list of bugs that passed the tests and were found only by measuring real behaviour. Some of them disabled headline features completely.

The general conclusion from that list: a test with a fake does not test an integration. Fakes reproduce neither Npgsql's type mapping, nor an endpoint's behaviour, nor the real asynchrony of a round. Measurement on a live stack found what no unit test could.

When a council of models is not what you need

The honest half of the article. A multi-model council is an expensive instrument, and it loses to a single strong prompt when:

• the task is unambiguous and fits in the context window — a debate adds no meaning, only time;
• the question is factual rather than evaluative — "which .NET version supports field" needs no three dissenting opinions;
• the budget is hard-capped: 154–209 s and a full roster cost money that is cheaper spent on a stronger single model;
• you are not going to read the transcript. If a human still decides, and that human will not read the analysis, the debate becomes expensive decoration.

A council earns its cost where verifiability matters: competing hypotheses that have to be weighed, citations to sources, a decision you will later have to defend. That is exactly where "all participants agreed" is valuable — because you can see who disagreed with whom.

An implementation checklist

If you are building this yourself, the order is roughly:

1. Fix the round protocol before choosing models. Standard, Critique or Consensus decides which models you need in the first place.
2. Lock the roster and measure output balance. A model that contributes 68.8% of the text makes the council pointless.
3. Time it honestly and decide up front whether 154–209 s fits.
4. Turn on the cache. IDebateCache returns in 0.0 s against a full run for a repeated question.
5. Attach the Knowledge Keeper if you have a corpus: three queries per debate is cheap, and grounding changes verdict quality.
6. Check the verdict against the transcript. Text you cannot support is a hallucination with good formatting.
7. Leave room for your own strategy by implementing IDebateStrategy.

In this series: Next — Delibera: a field note on the shape of the product

Sources

• Delibera repository — source, docs, changelog
• Project site — the "Measured Behaviour" section with the full method and caveats
• Delibera.Core on NuGet — package, versions, changelog inside
• Model Context Protocol — the protocol specification behind the Operator
• Microsoft.Extensions.AI — the seam between providers and the pipeline

If you are designing a multi-model pipeline for your team, start with the round protocol and the roster rather than with model selection. Discuss an AI pipeline architecture → techbuzzz.ru